dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: dohertj2/ScadaBridge:9b67c7cb4b4041934ab4a07d1abc40223880df05
Branches Tags
dohertj2/ScadaBridge:main
...
compare: dohertj2/ScadaBridge:09989fdf32e9e5bee2b379d027fbed55dfe70465
Branches Tags
dohertj2/ScadaBridge:main

18 Commits
9b67c7cb4b ... 09989fdf32

Author SHA1 Message Date
Joseph Doherty 09989fdf32 deps(localdb)+docs: consume LocalDb 0.3.3 (FK-parent registration guard + atomic per-entry apply, arch-review P4-7); Low-wave integration docs — register rows 103 closed / 104 auto_vacuum legacy files / 105 FK-parent constraint; CLAUDE.md bulk-deploy up-to-date filter + --force, 0.3.3 line, infra/LDAP correction (P8-7); plan doc status 2026-08-20 08:56:50 -04:00
Joseph Doherty bbc1b55cc4 test(siteeventlog): make P4-5's ExecuteAsync-yield test deterministic — park the purge on the isActiveNode gate instead of racing the pool thread 
The T4 test asserted 'reflection Invoke returned AND the row is still present', but the
yielded continuation runs RunPurge on a pool thread and can delete the row before the
test thread reaches its assert — lost 2/2 full-suite sweeps on the integration branch,
green in isolation (row-40 class, introduced by this wave). The purge now blocks inside
the isActiveNode check (consulted before any delete) until the test releases it: fixed
code returns from Invoke while the gate is held; unfixed code blocks on the gate itself
and the bounded wait fails. Proof of failure re-run: with Task.Yield() removed the test
fails with the intended message after the 10 s bound; 3/3 green with it restored.
 2026-08-20 08:56:50 -04:00
Joseph Doherty 57c8973b0d fix(centralui)+test(centralui): Low-wave integration — T12 Topology race fixture gains T13's IStaleInstanceProbe dependency; T11 sandbox-run continuation skips StateHasChanged after Dispose 
Cross-task regression caught by the combined sweep: TopologyLiveUpdatesTimerDisposeRaceTests
(T12) builds a real DeploymentService, whose constructor grew an IStaleInstanceProbe
parameter in T13 (P7-6 up-to-date pre-filter) — 4 DI-resolution failures on the
integration branch that neither task saw in isolation. Registered a substitute, matching
TopologyPageTests.

Bundle review Minor: ApiMethodForm/SharedScriptForm copied TemplateScriptDialog's sandbox
pattern exactly, whose finally re-renders unconditionally; the plan asked the continuation
to be guarded on disposal, so a _disposed flag now suppresses the post-cancel
StateHasChanged (the reference dialog keeps its original shape — out of T11's scope).
 2026-08-20 08:42:56 -04:00
Joseph Doherty 3bf1537ea0 docs(docker)+docs(docker-env2)+infra: close P8-7 + P8-10 — LDAP doc drift + env2 gRPC port/teardown parity 
P8-7: docker/README.md and docker-env2/README.md still listed the retired
scadabridge-ldap container. Verified against the running config
(central-node-*/appsettings.Central.json): both rigs actually bind to the
local redundant GLAuth pair (zb-ldap-primary :3893 + zb-ldap-backup :3894 on
host.docker.internal, source of truth scadaproj/infra/glauth-redundant), not
the shared 10.100.0.35 GLAuth the infra/docker-compose.yml retirement comment
claimed — that binding is itself stale, its serviceaccount password having
been rotated away by SEC-36. Corrected all three docs to name the actual
config each rig uses.

P8-10: docker-env2 mapped no host port for the central gRPC control plane
(docker/ maps 9013/9014->8083); added 9113/9114->8083 on both env2 centrals,
matching the free 91XX slots, plus the README port table. teardown.sh's
"preserved in node directories" data note predates the named-volume
migration; rewrote it honestly and added an explicit `docker volume rm`
recipe scoped to the env2 volume prefix, clearly labeled destructive,
mirroring docker/teardown.sh's existing pattern.

Gate: bash docker-env2/deploy.sh from this worktree — both centrals + both
site-x nodes recreated and came up with the new 9113/9114 ports open. The
simultaneous force-recreate split both pairs into independent 1-node
clusters (expected: BootstrapGuard is default-OFF on docker-env2, per
CLAUDE.md); a single restart of the b-side of each pair joined it to its
already-formed peer and both pairs converged (central: memberCount=2,
leader=central-a; site-x: node-b Welcome + singleton Younger). docker/ (the
primary rig) was not touched.
 2026-08-19 16:41:04 -04:00
Joseph Doherty 428b8db024 test(dcl+communication+storeandforward+kpihistory+host): close register row-40-class sleeps + row-32 provider leak (arch-review 2026-08-19 P8-8 + P8-9) 
P8-8: replace the remaining wall-clock sleep-then-assert instances outside
Communication.Tests' Medium-wave sweep with the same row-40 house pattern —
continuous polling (AssertHoldsFor) for ABSENCE assertions instead of a single
end-of-window sample, which cannot distinguish "stayed quiet throughout" from
"hadn't happened yet when the one sample landed" and cannot catch a value that
flips wrong and back before that sample.

- DataConnectionActorBatchTests: QualityCounters_ArePushedOnTransitionOnly_AndCoalesced
  (:206/:214 class) converted to AssertHoldsFor; ResolutionProbes_BackOff...
  (:181/:184, a genuine real-time backoff-RATE assertion, not a sync-primitive
  stand-in) converted to AssertRoundsWithinWindow, which enforces the ceiling
  continuously (fails fast on a no-backoff regression) while the floor still
  necessarily waits out the window.
- SiteAlarmLiveCacheServiceTests (:452-454, :470-472): converted to the
  project's own AssertHoldsFor (mirrors SiteAlarmAggregatorActorTests', same
  assembly).
- StoreAndForwardServiceTests (:891-892): async AssertHoldsForAsync variant
  (the class runs on the async test thread).
- KpiHistoryRecorderActorTests (:1043-1049): two skip-path FoldCount==0
  absence checks converted.

Each helper is a private per-file copy, matching the Medium-wave T17
precedent (SiteAlarmAggregatorActorTests.AssertHoldsFor) rather than a shared
TestSupport utility. Proof of the mechanism: a standalone (non-repo) harness
run 100 trials of a real background thread flipping a boolean true then back
to false inside a 300ms window — Thread.Sleep(300)+one sample missed the
violation 100/100 times (false pass), continuous 20ms polling caught it
100/100 times. Most of the actual conversions here guard monotonic counters
(their single-sample variant would eventually see the final value too), so
each converted test carries an inline honesty note on what specifically the
conversion buys for that assertion (fail-fast vs. true flip-back detection).

P8-9: ScadaBridgeTelemetryTests.EmitHelpers_DoNotThrow called
SetQueueDepthProvider(() => 5) against the process-global row-32 provider
slot with no clear and no [Collection] — a landmine for any future
same-process reader of scadabridge.store_and_forward.queue.depth. Fixed with
compare-and-clear in a finally via the existing (production, untouched)
ClearQueueDepthProvider. Searched this assembly and the whole test tree for a
second same-process reader that asserts on the value: none exists today
(MetricsEndpointTests scrapes /metrics but only asserts on the response
shape, and the only value-asserting test, QueueDepthGaugeTests, lives in
StoreAndForward.Tests — a separate test-host process the leak cannot reach),
so no [Collection] was added; the test's doc comment records the search and
flags it for re-check if that changes.

Test-only change, zero production edits. Full suites green: DCL.Tests
342/342, Communication.Tests 729/729, StoreAndForward.Tests 135/135,
KpiHistory.Tests 54/54, Host.Tests 522/522. Each converted test additionally
looped 20x in isolation with zero flakes.
 2026-08-19 16:41:04 -04:00
Joseph Doherty b34ec10f8b fix(deploymentmanager): bulk site deploy skips up-to-date instances and un-poisons the shared DbContext (arch-review Low P7-5 + P7-6) 
P7-6 (behaviour fix, plan decision 1): DeploySiteAsync redeployed EVERY
deployable instance unconditionally. A redeploy is not idempotent at the site —
it resets static attribute overrides and clears native alarm state by design — so
one site-wide push destroyed live operational state on every unchanged instance
in the batch, which is normally most of them. Bulk deploy now pre-filters with
IStaleInstanceProbe (the same seam the bundle importer and Deployments page use;
its memo timing was fixed by the Medium wave's P7-4): an instance whose freshly
computed revision hash still matches its DeployedConfigSnapshot is reported in the
existing additive SkippedInstances list with reason "up-to-date" and never
flattened, staged or sent. The filter is deliberately conservative — every
uncertainty deploys: only an Enabled instance qualifies, a missing snapshot
disqualifies, a prior record that ShouldQuerySiteBeforeRedeploy flags goes through
prepare so the reconcile that HEALS it still runs, and a null/faulting probe means
"indeterminate". New additive MgmtDeploySiteCommand.Force (default false = the new
filtered behaviour) plus CLI `deploy site --force` restores the unconditional push
for the case central cannot see (restored volume, rebuilt node). Composes with the
Medium wave's P7-2: Disabled wins — force suppresses the up-to-date filter only,
never the Disabled skip.

P7-5 (verified, then fixed): the cascade REPRODUCES. All three bulk phases share
one scoped DbContext, and a failed SaveChanges does not untrack the offending
entity — EF re-attempts it on the next save, including MarkDeploymentFailedAsync's
cleanup write. Measured on a real SQLite-backed ScadaBridgeDbContext: a permanent
save failure on instance 1's post-success side effects failed instances 2, 3 and 4
and left all three deployment records stranded InProgress. MarkDeploymentFailedAsync
now detaches the pending changes (new IDeploymentManagerRepository.DetachPendingChanges)
when the cleanup save fails and retries the Failed write once on the clean tracker,
so later instances save cleanly and the record reaches a terminal status. The
DeployFailed audit row moved outside the status-write try (best-effort, must not be
lost because the status write failed).

Tests: 5 new DeploySiteAsyncTests (skip + reason + no site round-trip; --force
deploys; Disabled skipped under both force values; the three not-enough-to-skip
guards), 2 new IntegrationTests over a real EF context (the poisoned-context
cascade, and the narrow EF tracking fact the fix rests on), CLI --force flag
parse test, and a MgmtDeploySite wire-default test. Proof of failure: both P7-6
behaviour tests and both P7-5 assertions fail against the unfixed code.
 2026-08-19 16:41:04 -04:00
Joseph Doherty 6d0d8b6258 fix(centralui): close register row 103 — take-once session/timer discipline for TransportImport + Topology (P6-6 lesser instances) 
Mirrors the Medium wave's DebugView P6-6 fix (_sessionLock + _disposed,
publish-then-recheck-disposal under one lock) for the two TTL-bounded/
unbounded lesser instances the Medium wave deferred:

- TransportImport.razor.cs: TryLoadAsync awaits IBundleImporter.LoadAsync and
  used to publish the result into _session unconditionally. A circuit torn
  down mid-await left Dispose() with nothing to release (session still null),
  so the freshly-opened server-side BundleSession stayed registered, stranding
  one of MaxConcurrentImportSessions (register row 43) for the full session
  TTL. Now publishes and re-checks disposal atomically under _sessionLock, so
  exactly one side (TryLoadAsync or Dispose) abandons the session.

- Topology.razor: OnInitializedAsync awaits LoadDataAsync before calling
  StartLiveUpdatesTimer. A circuit torn down mid-await left Dispose() with no
  timer to stop yet, so the timer StartLiveUpdatesTimer creates once the await
  resumes was never disposed — it ticked against a torn-down component every
  15s until process restart. StartLiveUpdatesTimer/StopLiveUpdatesTimer/
  Dispose now create/replace/take the timer atomically under _timerLock
  together with a _disposed check, so a late-arriving (re)start refuses to
  create a timer once disposed.

Tests (bUnit, both proven to fail against the unfixed code):
- TransportImportSessionDisposeRaceTests: dispose-during-load abandons the
  just-opened session immediately; dispose-after-successful-load abandons
  exactly once.
- TopologyLiveUpdatesTimerDisposeRaceTests: a late StartLiveUpdatesTimer call
  after Dispose leaves no live timer; Dispose stops the timer exactly once
  (idempotent double-dispose); OnLiveUpdatesToggled(off) + Dispose leaves no
  live timer. All assert on deterministic field/call-order state, no sleeps.

Register row 103 stays open for integration to close per the plan's shared
constraints (register edits happen at integration only).
 2026-08-19 16:41:04 -04:00
Joseph Doherty f1ef95f4da fix(centralui): P6-12 — dispose sandbox-run CTS in ApiMethodForm + SharedScriptForm 
Neither Test Run form implemented IDisposable, so navigating away mid-run left
_runCts uncancelled: the sandbox kept the run alive until its own internal
script timeout, and the eventual continuation touched a torn-down component's
_running/_runResult fields. TemplateScriptDialog already got this right — both
forms now mirror it exactly: @implements IDisposable plus a Dispose() that
cancels + disposes _runCts.

ScriptAnalysisService.RunInSandboxAsync is marked virtual (test-only seam,
mirroring the existing virtual on DebugStreamService.StartStreamAsync/
StopStream) so a TaskCompletionSource-gated stub can hold a run open
deterministically. New bUnit tests (ScriptFormSandboxDisposalTests) prove, for
both forms: disposing mid-run cancels the token the sandbox observed, and the
cancelled continuation resolves with no unhandled exception and no _runResult/
_running mutation. Proof of failure: reverting only the two .razor changes
(keeping the virtual seam) reproduces all 4 failures — the token is never
cancelled and the no-op assertions time out.
 2026-08-19 16:41:04 -04:00
Joseph Doherty 21d50d4406 fix(centralui): P6-11 — request-generation guard on Deployments page 
Three fetch initiators (search debounce, paging, push/coalesce reload) could
each have a LoadDataAsync flight in flight at once; completion order was not
call order, so a slow, stale flight could resolve after a newer one and
clobber the grid the operator was already looking at. Add a monotonic
_requestGeneration counter, bumped at fetch start and checked at apply —
last-write-wins by generation, not by whichever await resolves first.

Separately, the filter-debounce timer wrote `_currentPage = 1` directly on
its raw System.Threading.Timer callback thread — a Blazor threading
violation, since component state may only be mutated on the renderer's
sync context — and that write was not covered by the generation guard at
all. LoadDataAsync now takes a resetToFirstPage flag, resolves the target
page into a local, and applies it only behind the same generation check as
every other piece of state, via a new DispatchFilterReloadAsync that
marshals through InvokeAsync like the existing push-reload path.

Tests (DeploymentsRequestGenerationTests, bUnit): a delayed stale flight
arriving after a newer one does not overwrite the grid; a stale
filter-debounce flight does not reset the page after a newer navigation.
Both driven deterministically via TaskCompletionSource-backed repository
stubs. Verified to fail against the unfixed code (swapped via
`git show HEAD:<path>`, not `git stash` — repo-wide and unsafe across the
parallel worktree wave).
 2026-08-19 16:41:04 -04:00
Joseph Doherty 7ad1872454 fix(management): strict enum parsing for DataType/ElementDataType/AlarmTriggerType (P6-10) 
ManagementActor's ParseDataType/ParseElementDataType (bare Enum.TryParse) and
the two AlarmTriggerType parse sites in HandleAddAlarm/HandleUpdateAlarm (raw
Enum.Parse) all admitted ordinals ("2"), undefined values ("99" — a
deploy-time bomb that saves fine and only detonates at deploy), and
comma-joined name lists (bitwise-ORed together regardless of [Flags], e.g.
"String,Int32" silently becoming DateType.DateTime). A typo against the raw
Enum.Parse sites also surfaced as a masked "An internal error occurred"
rather than a curated field-naming error.

Adds ParseEnumStrict<TEnum>, mirroring the existing ParseDeploymentStatus
guard (names only, defined values only, comma-token rejection per the
Medium-wave T8 nuance / AuditQueryParamParsers.ParseEnumListStrict pattern),
and routes all four call sites through it.

Tests: 22 new ManagementActorTests cases per parse site (ordinal, undefined
numeric, comma-joined, and typo all rejected with the curated field-naming
error; every defined name still parses through to the service layer).
Proof of failure recorded in each rejection test's doc comment; verified
live by stashing the ManagementActor.cs fix — exactly the 14 new rejection
tests fail pre-fix (13 acceptance tests pass unchanged), confirming the
defect and the fix.
 2026-08-19 16:41:04 -04:00
Joseph Doherty 88fbc9e351 fix(auditlog): P5-6 — deterministic synthetic ReconciliationAbandoned EventId 
SiteAuditReconciliationActor minted a fresh random EventId for the synthetic
ReconciliationAbandoned row every time a permanently-abandoned row was
re-abandoned. Because the reconciliation cursor is held at `since` by ANY
co-batch unresolved row (see PullSiteAsync's hasUnresolvedFailure gate), a
poisoned row's per-EventId attempt counter restarts every time it reappears
in a stuck batch, so it gets re-abandoned roughly every
MaxPermanentInsertAttempts ticks — minting an unbounded number of duplicate
central audit rows for the SAME lost event.

Fix: DeriveAbandonmentEventId(Guid lostEventId) derives the synthetic
EventId deterministically (SHA-256 over a fixed namespace GUID + the lost
row's bytes, RFC4122-v5-shaped). Re-abandonment of the same lost row now
always produces the same synthetic EventId, so
IAuditLogRepository.InsertIfNotExistsAsync's existing idempotency collapses
every repeat into the first insert. Also adds an in-memory _abandonedEventIds
set as a CHEAP SHORT-CIRCUIT ONLY (skips the wasted ~25 min retry-then-abandon
cycle for a row already known abandoned this process's lifetime) — it is not
the correctness mechanism; it resets on singleton restart/failover, at which
point the deterministic id alone still keeps re-abandonment idempotent.

Tests (SiteAuditReconciliationActorTests):
- ReAbandonment_SameLostEventId_YieldsExactlyOneCentralRow: end-to-end repro
  of the defect trigger (poison row + a rotating-EventId co-batch row that
  never itself reaches the threshold, holding the cursor forever) across two
  full abandon cycles. Proof of failure: run against the unfixed source
  (git stash of the source file only, pure-function tests temporarily
  #if false'd to keep the file compiling), the test failed with exactly the
  predicted defect shape — 2 distinct synthetic rows for one lost EventId.
  Passes post-fix (1 row).
- DeriveAbandonmentEventId_DifferentLostRows_YieldDistinctSyntheticIds and
  DeriveAbandonmentEventId_IsPureAndStableAcrossIndependentInvocations:
  pure-function coverage of the derivation directly, including a pinned
  golden-value regression guard proving it is a function of the input GUID's
  value alone (no clock/randomness/process state).
 2026-08-19 16:41:04 -04:00
Joseph Doherty abbbcd1a8d fix(kpihistory)+fix(notificationoutbox)+fix(sitecallaudit)+fix(host): P5-5 — rate-fold interval triple made self-consistent + cross-validated 
Defect: the hourly KPI rollup fold (KpiHistoryRepository.FoldHourlyRollupsAsync)
sums every Rate-classified sample recorded in an hour, correct only if each
sample is a non-overlapping delta covering exactly one recorder tick
(KpiHistoryOptions.SampleInterval). That invariant was an unenforced silent
assumption across three independently-owned config keys — SampleInterval,
NotificationOutboxOptions.DeliveredKpiWindow, SiteCallAuditOptions.KpiInterval —
so widening DeliveredKpiWindow to e.g. 15 min would silently over-count the
rollup ~15x.

Robust fix: NotificationOutboxKpiSampleSource and SiteCallAuditKpiSampleSource
now rescale their Rate-classified samples (deliveredLastInterval,
failedLastInterval) by SampleInterval / (DeliveredKpiWindow or KpiInterval)
before recording, so the rollup's per-hour Sum is correct regardless of how
the other two keys are tuned — the live dashboard tile's own cutoff/window is
untouched. Both sources gain a project reference to KpiHistory to read
SampleInterval; no cycle (KpiHistory references only Commons).

Eager cross-validation (defense-in-depth): KpiRateFoldIntervalCrossValidator
(Host, the one composition root referencing all three component projects)
fails boot naming all three config keys when they diverge, protecting the
separate (still-real) invariant that the recorded/charted KPI-history sample
equals the live tile at the same instant — which the rescale trades away
whenever the scale factor isn't 1. Rides KpiHistoryOptions' existing
ValidateOnStart trigger.

Also: NotificationOutboxOptions had no registered validator at all before this
(the gap this task's "whichever options class owns DeliveredKpiWindow" file
scope pointed at) — added NotificationOutboxOptionsValidator mirroring
SiteCallAuditOptionsValidator's positive-duration/count convention, wired with
ValidateOnStart.

Tests: new rescale tests in both sample-source suites prove the fold is
numerically correct at a non-default SampleInterval (30/40 counted over a
15/20-minute window correctly divides down to a 1-minute-tick delta); new
KpiRateFoldIntervalCrossValidatorTests prove divergent config fails with all
three keys named; new NotificationOutboxOptionsValidatorTests. Every new test
proven to fail against the unfixed computation first (temporarily
short-circuited the scale factor / cross-check to the pre-fix behavior,
confirmed failure, restored) — not via git stash (repo-wide, shared across
concurrent worktrees this session) but via file-copy swap.

Full suites green: Host.Tests (527), ConfigurationDatabase.Tests (401),
KpiHistory.Tests (54), NotificationOutbox.Tests (163),
SiteCallAudit.Tests (67).
 2026-08-19 16:41:04 -04:00
Joseph Doherty 1cea01d924 fix(notifications): list sort index + CAS operator Retry/Discard (P5-3, P5-4) 
P5-3: add IX_Notifications_Created_NotificationId (CreatedAt DESC,
NotificationId DESC) — the Central UI Notification Outbox list page sorts on
exactly that pair, but no existing index leads unconditionally on CreatedAt,
so an unfiltered/lightly filtered page fell back to a full clustered-index
scan + sort on every COUNT + page request. Mirrors the Medium wave's
IX_SiteCalls_Created_TrackedOperationId migration + SQL twin pattern
faithfully, including the handwritten docs/plans/sql script.

P5-4: operator Retry/Discard was read-check-write keyed on NotificationId
only — a detached PipeTo task interleaving with the dispatcher's own write
(or a second concurrent operator action) could land between the read and
the write and get silently clobbered by the unconditional UpdateAsync,
stamping a Delivered row Discarded with stale fields. Adds
INotificationOutboxRepository.TryUpdateFromParkedAsync, a single-statement
compare-and-swap (WHERE Status == Parked) mirroring
SecuredWriteRepository.TryMarkApprovedAsync's CAS shape. NotificationOutboxActor's
RetryAsync/DiscardAsync now use the CAS write and, on a lost race, re-read
to report "notification already progressed to <Status>" instead of silently
overwriting whatever is there now.

Proof of failure: reverting only the actor's CAS usage while keeping the
repository addition reproduces the silent-overwrite bug — 7 tests fail,
including two new deterministic (non-sleep) race-proof tests that mock the
exact GetByIdAsync -> lost-CAS -> re-read sequence the finding describes.
P5-3 is perf-only; verified via migration/SQL-twin parity instead (no
runtime proof-of-failure is meaningful for an index).
 2026-08-19 16:41:04 -04:00
Joseph Doherty 11675285af fix(siteeventlog)+fix(host): purge startup yield + auto_vacuum reality (arch-review 2026-08-19 P4-5 + P4-6) 
P4-5: EventLogPurgeService.ExecuteAsync now yields (Task.Yield()) before running the
initial RunPurge(), making "the purge never runs synchronously on the caller's thread"
a property of the method itself rather than an artifact of .NET 8+'s BackgroundService
scheduling (verified empirically not to reproduce the literal "StartAsync blocks" symptom
on this repo's net10.0 target — a 1.5s synchronous Thread.Sleep in ExecuteAsync still lets
StartAsync return in ~2ms). Proven instead via a reflection-invoked ExecuteAsync test that
bypasses BackgroundService's Task.Run wrapper, which is timing-independent. Also: a
throwing isActive predicate now fails CLOSED (logged as inactive, at Warning) instead of
silently claiming "active" — the predicate only affects the tick's log line since P4-4,
but a throw must not masquerade as a successful check.

P4-6: PRAGMA auto_vacuum = INCREMENTAL only takes effect on a table-less database, and
LocalDbSchema.EnsureCreated (inside SqliteLocalDb's constructor) always creates LocalDb's
own bookkeeping tables before SiteEventLogSchema.Apply ever gets a connection — so the
pragma there had never once taken effect, in any real call path. Real fix per plan
decision 2: SiteLocalDbSetup.EnsureAutoVacuumOnFreshFile pre-creates a genuinely fresh
LocalDb file with the pragma set, called from SiteServiceRegistration.Configure BEFORE
AddZbLocalDb's lazy singleton ever constructs SqliteLocalDb — no LocalDb API change
needed. Only helps NEW site databases; an existing file (every node on the docker rig
today) is left untouched rather than VACUUM'd, per the plan's explicit rejection of an
unconditional VACUUM against a live ~1 GB file. The now-dead pragma call in
SiteEventLogSchema.Apply is removed and its comment corrected.

Tests proven to fail against the unfixed code (git-diff-and-restore, not stash — this repo
shares one stash ref across all worktrees): ExecuteAsync_YieldsBeforeRunningTheInitial-
PurgeSynchronously and RunPurge_ActiveNodeCheckThrows_TickLogsAsInactive_NotActive fail
without the EventLogPurgeService fix; SiteLocalDbAutoVacuumTests fails to compile without
EnsureAutoVacuumOnFreshFile. SiteEventLogging.Tests 112/112 and Host.Tests 526/526 green
with the fix applied.
 2026-08-19 16:41:04 -04:00
Joseph Doherty 99da32d635 fix(communication): P3-4 open-attempt deadline + P3-5 duplicate-subscribe cap order 
P3-4 — SiteAlarmAggregatorActor pinned _openInFlight forever on a stream open
that produced none of its three peer-driven outcomes. SiteStreamGrpcClient gives
up on response headers after ConnectedHeaderTimeout (10s) and then treats the
first received EVENT as its fallback connected signal, so an alarm-only site
that stays quiet supplies neither — leaving _streamDown=true AND _openInFlight=
true, which makes OnReconcileTick's reopen guard (_streamDown && !_openInFlight)
skip every tick until the site's 4h max stream lifetime ends the call.

Fix: bound one open attempt with a generation-fenced single-shot deadline
(GrpcAlarmStreamOpenTimedOut, ctor-injected, production default 30s — above the
10s header timeout, below the 60s reconcile interval). Firing it releases the
flag and nothing else: no retry-budget spend, no node flip, no liveness
republish, and the abandoned subscription is left running so a merely-slow
connect still comes up normally. Self-healing rather than reachability analysis.
The deadline is cancelled on connect / error / graceful completion.

P3-5 — SiteStreamGrpcServer cancelled the incumbent stream on a duplicate
correlation id and only THEN checked the concurrency cap. Checking Count >= max
after the removal is right at exactly the cap but wrong above it, and above it
is reachable because the cap check is documented check-then-act: two subscribes
that both clear it before either registers overshoot by one. In that window a
routine reconnect killed a healthy stream AND was refused with
ResourceExhausted, leaving the correlation with ZERO streams until its next
retry.

Fix: claim the removal, consult the cap only for a genuinely NEW correlation id,
then cancel. A replacement is net-neutral on the count so it never consults the
cap; a refused new subscriber cancelled nothing (TryRemove found nothing), so
the incumbent survives. Newest-wins is unchanged — a fresh authenticated
subscribe still displaces a dead/wedged incumbent.

Tests (both shown failing against the unfixed code):
- OpenAttemptThatNeverResolves_ReleasesTheInFlightGuard_SoTheTickCanReopen —
  fails (reopen never happens) with the deadline arming disabled.
- OpenDeadline_DoesNotFire_OnAStreamThatConnects_NorFlipNodeOrBurnBudget —
  inertness lock on every healthy path.
- AtCapOvershoot_DuplicateReconnect_IsNotRefused_AndLeavesExactlyOneLiveStream —
  fails with ResourceExhausted against the old ordering; the overshoot is driven
  deterministically by parking both openers on the ServerCallContext token read,
  which sits between the cap check and the _activeStreams insert.
- AtCap_AGenuinelyNewSubscriber_IsStillRefused_AndTheIncumbentSurvives.

The P3-4 test's doc comment carries the honesty note: the aggregator-side
starvation reproduces exactly, but how often the >10s header delay that gets you
there occurs in production is NOT demonstrated — the site flushes headers
explicitly as soon as its relay is attached, so the budget has to be consumed by
connect + handshake + server scheduling instead.
 2026-08-19 16:41:03 -04:00
Joseph Doherty 6823e7e3fa fix(dcl): P2-5 seed probe-resolved tags; P2-6 close the Connecting/Reconnecting AdapterDisconnected gap 
P2-5 — the tag-resolution PROBE was the only subscribe path that did not seed.
HandleSubscribe carries seeds back on SubscribeCompleted and ReSubscribeAll re-seeds
after a reconnect, because a fresh advise only yields a value on the source's next
CHANGE. HandleRetryTagResolution did neither, so a tag that resolved LATE (device still
booting, item created after deployment) and then never changed had no value path at all.
OPC UA masks it with the initial data-change notification on subscription creation; an
MxGateway advise can legitimately return VT_EMPTY, so a static tag stayed Uncertain for
the life of the connection. New SeedNewlyResolvedTags reads the round's newly resolved
tags and delivers them through the normal generation-guarded TagValueReceived path —
directly rather than on the completion message (the probe only covers tags already
registered in _instancesByTag, so the initial subscribe's ordering hazard does not
apply), and as its own fire-and-forget task so the seed reads never delay the
BatchSubscribeCompleted that re-arms the retry timer.

P2-6 — only the Connected state handled AdapterDisconnected; in Connecting and
Reconnecting it fell through to Unhandled. A connect that succeeds and drops again before
its piped ConnectResult is applied therefore delivered the disconnect FIRST, where it was
discarded, and ConnectResult(true) then declared the connection up on a dead adapter. An
adapter raises Disconnected at most once per successful connect (OpcUaDataConnection's
once-only _disconnectFired, reset only by the next successful ConnectAsync), so a
connection with no subscriptions — nothing to fail at connection level and drive the
reconnect machine — sat Connected on a dead adapter until the process restarted. Both
states now record the disconnect in _disconnectDuringConnectAttempt (cleared at the top of
HandleAttemptConnect, so it only ever refers to the attempt in flight) and both result
handlers demote such a "success" to the ordinary connect-FAILURE tail — counting toward
the failover budget and re-arming the retry timer. A flag rather than Stash: the stash has
no selective drop, so a stashed disconnect would be replayed after the next successful
connect and cost a spurious reconnect cycle.

AdapterDisconnected is now stamped with the adapter generation (hooked via a
per-adapter closure instead of a method group; SwapAdapter hooks AFTER the bump), and
IsStaleDisconnect drops a notification from a swapped-out adapter in all three states —
the disconnect twin of IsStaleConnectResult. SwapAdapter's unhook cannot stop an event
already executing on a background thread, so without the fence an endpoint the operator
has just reconfigured AWAY from could knock the healthy replacement into a reconnect.

Tests: DataConnectionProbeSeedTests + DataConnectionDisconnectRaceTests (4 new). The
race is reproduced by ORDERING, not timing — the connect is held on a gate, Disconnected
is raised while the actor sits idle, and a GetHealthReport round trip is the mailbox
barrier; no sleeps. All 4 were proved to fail against the pre-fix behaviour via a
targeted revert of the four behaviour sites (the stale-generation test cannot be proved
by a plain revert — the record carried no generation before this change, so the test
would not compile; noted in its doc comment). Full DCL suite 346/346 green.
 2026-08-19 16:41:03 -04:00
Joseph Doherty 65b3c23239 perf(siteruntime): projected single-row deployed-config reads (P1-4) 
HandleEnsureInstanceStarted, HandleDeploymentStateQuery, and HandleEnable each
called GetAllDeployedConfigsAsync — materializing every deployed config,
including every ConfigJson blob — just to FirstOrDefault one row. A
volume-restore reconcile healing hundreds of instances turned that into
hundreds of full-table reads.

Add SiteStorageService.GetDeployedConfigAsync(name) (single-row, full
projection incl. ConfigJson, for callers that (re)start an Instance Actor)
and GetDeployedConfigMetadataAsync(name) (single-row, ConfigJson-free
projection, for callers that only need DeploymentId/RevisionHash/IsEnabled).
Switch the three call sites; leave decisions/semantics identical.

HandleEnsureInstanceStarted gets its own _deployedConfigLoader seam,
deliberately separate from the existing _configLoader (still used by the
boot-time load, which legitimately needs every row) so a test overriding one
cannot accidentally gate the other. Generation fencing (StoreGenerationAtRead,
Medium wave T1) is untouched: the generation is still captured on the actor
thread before the read dispatches.

Tests: new SiteStorageServiceTests lock in that both projections return the
same values GetAllDeployedConfigsAsync().FirstOrDefault(...) did. The three
row-101 ensure-started race tests (ReconcileStartsInstanceTests) are ported
to the new seam and still pass, proving generation fencing survives the
projection. Proof of failure: since this is a pure I/O-cost fix with no
observable behavior change, the new/changed tests were confirmed to fail to
even compile against the unfixed code (stash/unstash) — GetDeployedConfigAsync,
GetDeployedConfigMetadataAsync, and the deployedConfigLoader ctor param don't
exist there.
 2026-08-19 16:41:03 -04:00
Joseph Doherty 942acc37b4 docs(plans): low-findings remediation plan — 21 review Lows + register row 103 as 5 phases / 15 model-matched tasks 2026-08-19 16:01:03 -04:00
85 changed files with 8251 additions and 262 deletions
Expand all files Collapse all files
CLAUDE.md
+3 -2
View File
@@ -12,7 +12,7 @@ Layout is self-describing (`ls`, plus each directory's own README). Only the non
- `docker/` — the primary 8-node cluster topology (2 central + 3 sites). Rebuild + redeploy with `bash docker/deploy.sh`.
- `docker-env2/` — minimal second cluster (2 central + 1 site × 2 nodes) that runs **concurrently** with `docker/` on host ports 91XX; built for testing Transport (#24) against two real environments. Rebuild with `bash docker-env2/deploy.sh`.
- `infra/` — Docker Compose for local test services (MS SQL, OPC UA, SMTP, REST API, Traefik). **LDAP is no longer started here** — dev/test LDAP is the shared GLAuth on `10.100.0.35:3893` (source of truth: `scadaproj/infra/glauth/`), and the same applies to `docs/test_infra/test_infra.md`, which still reads as if it were a local container.
- `infra/` — Docker Compose for local test services (MS SQL, OPC UA, SMTP, REST API). Traefik is **not** an `infra/` service — it is per-rig (`docker/docker-compose.yml`). **LDAP is no longer started here** — both rigs (`docker/`, `docker-env2/`) bind the local **redundant GLAuth pair** (`zb-ldap-primary` :3893 + `zb-ldap-backup` :3894 on `host.docker.internal`; source of truth `scadaproj/infra/glauth-redundant/`, verified against each central node's `appsettings.Central.json` — arch-review 2026-08-19 P8-7), NOT the shared `10.100.0.35:3893` GLAuth, whose serviceaccount binding was rotated away by SEC-36. `docs/test_infra/test_infra.md` still reads as if LDAP were a local container.
- `deprecated/` — retired docs/notes kept only for reference; not current.
## Sister Projects
@@ -93,6 +93,7 @@ spec for each is `docs/requirements/Component-<Name>.md`, and `README.md` carrie
- **CDC registration is conditional, and both directions self-heal at boot** (arch-review WP1.3 + WP3.3, `Host/SiteLocalDbSetup.cs`). Capture triggers are installed only when this node has replication configured — `PeerAddress` **or** `ApiKey`, an OR because only the dialling half sets `PeerAddress` while the passive half carries the key alone. An unreplicated node calls **`DeregisterReplicated`** on all ten tables at boot, dropping triggers an earlier build left behind and pruning their oplog/row-version rows (idempotent; logs once at Information when something was actually cleaned). A replicating node registers with **`baselineExistingRows: true`**, which seeds `__localdb_row_version` for pre-existing rows at the LWW floor (HLC `0`, this node's id) and flags a snapshot resync — so turning replication ON for a site that has been running without it now converges on the rows already in the file instead of only on writes made after the restart. Deregistration must be symmetric (the handshake compares registered-table digests fail-closed), so replication is a both-nodes-together change in either direction. LocalDb 0.2.0 also makes backlog depth O(1) and drops the unused `__localdb_oplog_hlc` index; the on-disk bookkeeping schema goes to v2, upgraded in place on open, with no wire change (0.1.x peers still sync).
- **LocalDb 0.3.0 (2026-08-15) makes that snapshot flag PER-TABLE** — the residual 0.2.1 explicitly deferred. On-disk bookkeeping schema goes to **v3** (a new `__localdb_snapshot_state` table, upgraded in place on open); still wire-compatible, and `lib_schema_version` deliberately stays `1` because the handshake compares it fail-closed for *equality*. The extension is negotiated by **capability** (`Handshake.supports_partial_snapshot`, `SnapshotBegin.tables` where **empty == all tables**), so a peer that does not advertise it is served a full snapshot — a scoped one would make it advance `last_applied_remote_seq` past deltas it never received. For the same reason **a partial snapshot moves no watermark on either side**; the overlapping deltas are still pumped and absorbed by LWW. Snapshot debt now clears on the receiver's new `SnapshotAck` rather than on enqueue, so a crash mid-snapshot re-streams the same scope. **`SiteLocalDbSetup.cs` needed no restructuring** — it already calls `RegisterReplicated` once per table in a loop, which is exactly what per-table flagging keys off. The consequence to know: **adding an eleventh table to `ReplicatedTables` on an already-replicating site now snapshots that one table**, where through 0.2.x the same edit re-streamed all eleven in full in both directions. First-boot behaviour is unchanged (all ten seed at once ⇒ the flagged set is every registered table ⇒ an ordinary full snapshot), and so is upgrading the rig in place (the ten tables are already ledgered, nothing seeds, nothing is flagged). A future "register the whole list in one call" API would silently give the coarse behaviour back.
- **LocalDb 0.3.2 (2026-08-19, ScadaBridge arch-review P4-1 + P4-3; wire-compatible, no schema change):** (1) the sync pump now **faults the live session when a cap-prune invalidates its position** (horizon check on the oplog's min seq before each read batch + on the batch head after it; throws the new public `LocalDbOplogGapException` through the 0.3.1 fault-teardown) — previously a slow-but-alive peer silently skipped the pruned gap and the pair diverged until the next reconnect; (2) **transport keepalives both sides** — client `SocketsHttpHandler` pings via new validated `LocalDb:Replication:KeepAlivePingDelay`/`KeepAlivePingTimeout` (defaults 60s/20s, `Always` policy), and ScadaBridge's site gRPC listener sets Kestrel h2 keepalive (`Host/SiteKestrelSetup.cs` — server-wide, also governs the metrics port); (3) the server's single-stream guard is now **newest-wins with an awaited handover** — a fresh authenticated `Sync` cancels a dead/wedged incumbent and *waits* for it to finish (`TakeoverTimeout` 10s) before starting, so a half-open link can no longer wedge the pair behind `AlreadyExists` while the zombie side reports `LocalDbReplicationConnected=true`.
- **LocalDb 0.3.3 (2026-08-20, ScadaBridge arch-review P4-7; wire-compatible, no schema change):** two latent `LwwApplier` hazards closed before any table could hit them. (1) **`RegisterReplicated` now REFUSES a table that is the parent of a foreign key with a row-mutating `ON DELETE` action (`CASCADE`/`SET NULL`/`SET DEFAULT`) from any other table** (`LocalDbRegistrationException` naming `child.column ON DELETE <action>`; `RESTRICT`/`NO ACTION` still allowed) — the applier's `INSERT OR REPLACE` runs with `foreign_keys=ON`, so the implicit delete fired those actions while CDC capture was suppressed, i.e. child rows changed on one node and never replicated. Registration-time only: SQLite can't add an FK to an existing table, so a later-created child is caught at the next boot's registration. None of ScadaBridge's ten tables has an FK (verified on the live rig file: zero `REFERENCES` across 29 `CREATE TABLE`s) — **keep it that way; a cascading-FK parent can never be one of `ReplicatedTables`** (register row 105). (2) The per-entry apply (delete-or-upsert + row-version ledger write) is now **atomic under a `SAVEPOINT`** — a fault between the two statements used to leave a deleted row with a live ledger entry, which `SnapshotStreamer` warns-and-skips forever; poison-catch semantics unchanged (narrow `SqliteException`, per-entry dead-letter, batch continues). Cost: two extra in-memory statements per applied entry on a backlog drain.
- **LocalDb 0.2.1** (two adversarial-review findings against 0.2.0, wire-compatible, no schema change) fixes: (1) `DeregisterReplicated` was deleting the HLC clock's only durable crash anchor along with the table rows — an ungraceful exit (crash, SIGKILL, power loss) on a fully-deregistered file left nothing to recover from, so the reopened clock could restart from wall-clock time and re-issue stamps below HLCs peers already hold (silently discarded by LWW); the current clock is now flushed inside the deregistration transaction, before the prune. (2) The sync inbox was an **unbounded** channel, so inbound memory was a function of what the peer sends — a mutual full snapshot (both sides of a pair baseline unconditionally when replication is enabled late) buffered whole, in memory, on both nodes at once; it is now **bounded at 64 messages** with wait-mode backpressure (unrelated to, and not to be confused with, the rig's separate `MaxBatchSize: 64` DB-read-page pin above), which required moving the snapshot to run alongside the receive loop (not inside the pre-loop handshake) so the inbox is drained for the whole session and can't deadlock itself.
- All timestamps are UTC throughout the system.
- Inter-cluster communication uses **three** transports, not two — **all cross-cluster command/control and data now rides gRPC** after the ClusterClient→gRPC migration's Phase 4 (`docs/plans/2026-07-22-clusterclient-to-grpc-plan.md`) deleted Akka `ClusterClient`/`ClusterClientReceptionist`: (1) **gRPC command/control** — site→central over the central-hosted `CentralControlService` (`GrpcCentralTransport`, sticky central-a→central-b channel pair; deployments/notifications/health/heartbeat/audit-ingest/reconcile), and central→site over the site-hosted `SiteCommandService` (`GrpcSiteTransport`, per-site NodeA→NodeB channel pair; the 28 lifecycle/OPC-UA/query/parked/route/failover commands); (2) **gRPC** server-streaming for real-time data (attribute values, alarm states, `SiteStreamService`); and (3) **plain token-gated HTTP** for the deployment config itself — notify-and-fetch, the site pulls the config from `DeploymentConfigEndpoints` (`ManagementService/DeploymentConfigEndpoints.cs`) with an `X-Deployment-Token` header, `AllowAnonymous` with the per-deployment token as the entire security boundary. The gRPC boundary is per-site PSK-authenticated (`ControlPlaneAuthInterceptor`, unchanged). There is **no receptionist registration** — discovery is by dialling configured endpoints; central builds one `SitePairChannelProvider` per site (addresses from `Site.GrpcNodeAAddress`/`GrpcNodeBAddress`, refreshed from the DB every 60s and on admin changes), sites dial `ScadaBridge:Communication:CentralGrpcEndpoints` (both central nodes, h2c on `CentralGrpcPort` 8083, **NOT** via Traefik). **Discovery is asymmetric by design:** central discovers site gRPC addresses from the *database* (refreshable at runtime), sites discover central from *appsettings* (`CentralGrpcEndpoints`, static — restart required; `StartupValidator` requires a Site node to list at least one). `Akka.Cluster.Tools` stays for ClusterSingleton; only the ClusterClient part is gone. **Central never buffers for an unreachable site** — the send fails with the caller's Ask/deadline timing out; a `ConnectionStateChanged` mechanism built for this was deleted as dead code.
@@ -125,7 +126,7 @@ spec for each is `docs/requirements/Component-<Name>.md`, and `README.md` carrie
- Flattened configs include a revision hash for staleness detection.
- Deployment identity: unique deployment ID + revision hash for idempotency.
- Per-instance operation lock covers all mutating commands (deploy, disable, enable, delete).
- **Bulk site deployment (`DeploymentService.DeploySiteAsync`, arch-review WP2.5)** redeploys every out-of-date instance at a site through one flatten session (a template chain shared by N instances is walked once) in three phases — serial Prepare, bounded-parallel Send (`SiteDeploymentMaxParallelism`, default 4), serial Finalize — still **not all-or-nothing**: any one instance's failure doesn't block the rest. **Disabled instances are SKIPPED by bulk deploy as of 2026-08-19 (arch-review P7-2)** — they are reported in the summary's additive `SkippedInstances` list, never flattened/sent/re-enabled (single-instance deploy remains the explicit operator path that deploys+enables a Disabled instance); Prepare's per-instance reconcile also runs under a short `SiteDeploymentReconcileTimeout` (5s) with a once-per-batch unreachable-site circuit-breaker, so a down site no longer costs 30s × N serial before the first send (P7-3). The `PendingDeployment` staging row (5-minute TTL) is written per-instance immediately before that instance's round-trip in the Send phase, not upfront in Prepare, so a large batch's tail instances don't have their TTL clock already half-spent by the time they're reached.
- **Bulk site deployment (`DeploymentService.DeploySiteAsync`, arch-review WP2.5)** redeploys every out-of-date instance at a site through one flatten session (a template chain shared by N instances is walked once) in three phases — serial Prepare, bounded-parallel Send (`SiteDeploymentMaxParallelism`, default 4), serial Finalize — still **not all-or-nothing**: any one instance's failure doesn't block the rest. **Disabled instances are SKIPPED by bulk deploy as of 2026-08-19 (arch-review P7-2)** — they are reported in the summary's additive `SkippedInstances` list, never flattened/sent/re-enabled (single-instance deploy remains the explicit operator path that deploys+enables a Disabled instance); **Up-to-date instances are ALSO skipped as of 2026-08-20 (arch-review P7-6)** — a redeploy is not idempotent at the site (it resets static overrides and clears native alarm state), so `DeploySiteAsync` pre-filters through `IStaleInstanceProbe` (trustworthy since P7-4 fixed its memo timing) and reports unchanged instances in `SkippedInstances` with reason `up-to-date`; the filter is conservative (every uncertainty deploys: missing snapshot, `ShouldQuerySiteBeforeRedeploy`, null/faulting probe). Additive `MgmtDeploySiteCommand.Force` / CLI `deploy site --force` restores the unconditional push for what central cannot see (restored volume, rebuilt node) — **Disabled still wins under `--force`**. Also P7-5: the three phases share one scoped DbContext and a failed `SaveChanges` leaves the entity tracked, so one permanent failure used to cascade to every later instance and strand records `InProgress`; `MarkDeploymentFailedAsync` now detaches pending changes and retries the `Failed` write once on the clean tracker; Prepare's per-instance reconcile also runs under a short `SiteDeploymentReconcileTimeout` (5s) with a once-per-batch unreachable-site circuit-breaker, so a down site no longer costs 30s × N serial before the first send (P7-3). The `PendingDeployment` staging row (5-minute TTL) is written per-instance immediately before that instance's round-trip in the Send phase, not upfront in Prepare, so a large batch's tail instances don't have their TTL clock already half-spent by the time they're reached.
- Site-side apply is all-or-nothing per instance.
- System-wide artifact version skew across sites is supported.
- Last-write-wins for concurrent template editing (no optimistic concurrency on templates).
Directory.Packages.props
+3 -3
View File
@@ -109,9 +109,9 @@
<PackageVersion Include="ZB.MOM.WW.Secrets.Ui" Version="0.6.2" />
<PackageVersion Include="ZB.MOM.WW.Secrets.Replicator.SqlServer" Version="0.6.2" />
<PackageVersion Include="ZB.MOM.WW.Secrets.Replicator.Grpc" Version="0.6.2" />
<PackageVersion Include="ZB.MOM.WW.LocalDb" Version="0.3.2" />
<PackageVersion Include="ZB.MOM.WW.LocalDb.Replication" Version="0.3.2" />
<PackageVersion Include="ZB.MOM.WW.LocalDb.Contracts" Version="0.3.2" />
<PackageVersion Include="ZB.MOM.WW.LocalDb" Version="0.3.3" />
<PackageVersion Include="ZB.MOM.WW.LocalDb.Replication" Version="0.3.3" />
<PackageVersion Include="ZB.MOM.WW.LocalDb.Contracts" Version="0.3.3" />
</ItemGroup>
<!--
docker-env2/README.md
+21 -5
View File
@@ -41,8 +41,8 @@ Env2 host ports are the primary's ports + 100. Both stacks can run simultaneousl
| Node | Container Name | Host Web | Host Akka | Host gRPC | Internal |
|------|----------------|----------|-----------|-----------|----------|
| Traefik LB | `scadabridge-env2-traefik` | 9100 | — | — | 80 (proxy), 8080 (dashboard host:8181) |
| Central A | `scadabridge-env2-central-a` | 9101 | 9111 | — | 5000 (web), 8081 (Akka) |
| Central B | `scadabridge-env2-central-b` | 9102 | 9112 | — | 5000 (web), 8081 (Akka) |
| Central A | `scadabridge-env2-central-a` | 9101 | 9111 | 9113 | 5000 (web), 8081 (Akka), 8083 (gRPC) |
| Central B | `scadabridge-env2-central-b` | 9102 | 9112 | 9114 | 5000 (web), 8081 (Akka), 8083 (gRPC) |
| Site-X A | `scadabridge-env2-site-x-a` | — | 9121 | 9123 | 8082 (Akka), 8083 (gRPC) |
| Site-X B | `scadabridge-env2-site-x-b` | — | 9122 | 9124 | 8082 (Akka), 8083 (gRPC) |
@@ -53,11 +53,17 @@ Env2 attaches to the existing `scadabridge-net` Docker bridge network and reuses
| Service | Container | What env2 uses it for |
|---------|-----------|-----------------------|
| MS SQL | `scadabridge-mssql` | Env2-specific databases `ScadaBridgeConfig2` / `ScadaBridgeMachineData2` |
| LDAP | `scadabridge-ldap` | Authentication (same test users) |
| SMTP | `scadabridge-smtp` | Notification capture in Mailpit (env2 emails distinguishable by `FromAddress`) |
| OPC UA | `scadabridge-opcua` | Simulated tags for site-x data connections |
| REST API | `scadabridge-restapi` | External REST API testing |
LDAP is **not** one of the reused `infra/` containers (that stack's `ldap` service is
retired/commented out). Env2's central nodes bind to the same local redundant GLAuth pair
as `docker/` — `zb-ldap-primary` on `host.docker.internal:3893` with `zb-ldap-backup` on
`host.docker.internal:3894` as `FallbackServers` (source of truth:
`scadaproj/infra/glauth-redundant`); same test users, same bind credentials as the primary
stack.
## Commands
### First-Time Setup
@@ -108,7 +114,16 @@ Both UIs are now reachable:
bash docker-env2/teardown.sh
```
Containers stop, volumes (data + logs) preserved. To also drop the env2 databases:
Containers stop; node data lives in named Docker volumes (`scadabridge-env2-*-data`) and
logs in node directories, both left in place. To also remove the persistent data:
```bash
# Destructive: deletes all env2 SQLite data (site config cache, S&F buffers, LocalDb).
docker volume rm $(docker volume ls -q --filter name=scadabridge-env2-)
rm -rf docker-env2/*/logs
```
To also drop the env2 databases:
```bash
docker exec scadabridge-mssql /opt/mssql-tools18/bin/sqlcmd \
@@ -134,7 +149,8 @@ docker logs -f scadabridge-env2-central-a
## Test Users
Same as primary (env2 shares LDAP). See `infra/glauth/config.toml` and primary `docker/README.md`.
Same as primary (env2 shares the local redundant GLAuth pair, `scadaproj/infra/glauth-redundant`).
See "Shared Infrastructure" above and primary `docker/README.md`.
## Transport Testing Workflow
docker-env2/docker-compose.yml
+2
View File
@@ -57,6 +57,7 @@ services:
ports:
- "9101:5000" # Web UI + Inbound API
- "9111:8081" # Akka remoting
- "9113:8083" # gRPC control plane (CentralControlService) — mirrors docker/'s 9013
volumes:
- ./central-node-a/appsettings.Central.json:/app/appsettings.Central.json:ro
- central-node-a-data:/app/data
@@ -113,6 +114,7 @@ services:
ports:
- "9102:5000" # Web UI + Inbound API
- "9112:8081" # Akka remoting
- "9114:8083" # gRPC control plane (CentralControlService) — mirrors docker/'s 9014
volumes:
- ./central-node-b/appsettings.Central.json:/app/appsettings.Central.json:ro
- central-node-b-data:/app/data
docker-env2/teardown.sh
+5 -1
View File
@@ -10,7 +10,11 @@ docker compose -f "$SCRIPT_DIR/docker-compose.yml" down
echo ""
echo "Teardown complete."
echo "Site data (SQLite DBs) and logs are preserved in node directories."
echo "Node data (SQLite DBs) is preserved in named docker volumes; logs in node directories."
echo ""
echo "To also remove persistent data:"
echo " docker volume rm \$(docker volume ls -q --filter name=scadabridge-env2-)"
echo " rm -rf $SCRIPT_DIR/*/logs"
echo ""
echo "Env2 databases (ScadaBridgeConfig2 / ScadaBridgeMachineData2) remain on"
echo "the shared scadabridge-mssql. To drop them:"
docker/README.md
+11 -2
View File
@@ -71,11 +71,18 @@ Port block pattern: `90X1`/`90X2` (Akka), `90X3`/`90X4` (gRPC) where X = 0 (cent
| Service | Container Name | Host Port | Purpose |
|---------|---------------|-----------|---------|
| MS SQL 2022 | `scadabridge-mssql` | 1433 | Configuration and machine data databases |
| LDAP (GLAuth) | `scadabridge-ldap` | 3893 | Authentication with test users |
| SMTP (Mailpit) | `scadabridge-smtp` | 1025 / 8025 | Email capture (SMTP / web UI) |
| OPC UA | `scadabridge-opcua` | 50000 / 8080 | Simulated OPC UA server (protocol / web UI) |
| REST API | `scadabridge-restapi` | 5200 | External REST API for integration testing |
`infra/docker-compose.yml` no longer runs an LDAP container (its `ldap` service is commented
out, kept only for rollback). The central nodes bind to the local redundant GLAuth pair
instead — `zb-ldap-primary` on `host.docker.internal:3893` with `zb-ldap-backup` on
`host.docker.internal:3894` as `FallbackServers` (source of truth:
`scadaproj/infra/glauth-redundant`; see each central node's `appsettings.Central.json`). This
replaces the previously-used shared dev GLAuth at `10.100.0.35:3893` — its serviceaccount
password was rotated away (SEC-36) and is no longer valid for this rig.
All containers communicate over the shared `scadabridge-net` Docker bridge network using container names as hostnames.
## Directory Structure
@@ -152,7 +159,9 @@ If you add a site to the rig, add its key in both places or its streams will fai
Start infrastructure services first, then build and deploy the application:
```bash
# 1. Start test infrastructure (MS SQL, LDAP, SMTP, OPC UA)
# 1. Start test infrastructure (MS SQL, SMTP, OPC UA — LDAP is the local
# redundant GLAuth pair on the Docker host, not started here; see
# "Infrastructure Services" above)
cd infra && docker compose up -d && cd ..
# 2. Build and deploy all 8 ScadaBridge nodes
docs/plans/2026-07-08-deferred-work-register.md
+3 -1
View File
@@ -95,7 +95,9 @@ All 7 fix-now items landed via PLAN-04/05/06/07/08 (verified in review 08 round
| 100 | ~~**`SiteScriptCompileCache.Clear()` is process-global — test classes wipe each other's entries**~~ **RESOLVED 2026-08-17 (merge `8a9a8d29`, agent commit `84b405ff`) — premise corrected: there were ZERO production callers of `Clear()`** (the only production entry point is `ScriptCompilationService.CompileCore` → `GetOrAdd`; `Clear()` existed solely for tests), so neither register option was taken as written — `Clear()` was **deleted outright** and the cache became an instance-scoped `sealed class` owned by `ScriptCompilationService`. The N1 "one compile per body per process" property now rests on `AddSiteRuntime` registering that service as a **DI singleton** — a lifetime that became load-bearing, so new `SiteRuntimeServiceRegistrationTests` pins it (a Scoped/Transient downgrade would reintroduce WP3.1's recompile storms with no behavioral test failing). The 37 test sites doing `new ScriptCompilationService(logger)` each silently get their own cache — isolation for the whole assembly, not just the row-91 classes. Row-91 workarounds RETIRED, not preserved: serialized collection removed from all three classes, exact `Hits`/`Count` assertions restored, guid key token gone, `ForeignEntrySlack` 512→8; new `SeparateCaches_DoNotShareEntriesOrCounters` pins the scoping. SiteRuntime.Tests green three consecutive full-parallel runs. | Found by the 2026-08-17 row-91 fix session | Closed 2026-08-17. | — |
| 101 | ~~**`ApplyingArtifacts_RemovedDataConnection_StopsLiveDclConnection` flaked once — suspected overlay-sweep read racing the artifact-apply write**~~ **RESOLVED 2026-08-17 (`0d3a90a9`) — hypothesis VERIFIED, and the production consequence was WORSE than the row guessed.** The race is real: the snapshot is read off the actor thread but compared against the overlay only when the piped message lands, so a read starting before an artifact apply's store writes can land after `HandleApplyArtifactDataConnectionsToDcl` populated the overlay — its stale (even empty) snapshot then "proves" the just-applied names disappeared. Overlay iteration order explains the exact fingerprint (`PlcA` removed first). Two corrections to the row's guesses: (1) it is NOT "self-healing at the next tick" — the removal evicts the hash slot the overlay's push is gated on, so the next fresh sweep clears the tombstone but re-creates nothing; the stopped connection stays dead **until an instance redeploy**; (2) the mid-write "momentarily not seeing the row" framing was the lesser half — the whole-snapshot staleness (read-before-write, land-after-apply) is the dominant window. Fix: `_definitionsWriteGeneration`, bumped on the actor thread by every artifact apply and captured at every snapshot-read dispatch (boot load + sweep); a result carrying a stale generation is discarded WHOLE — additions included, since a stale row pushed at a live connection would reconfigure it backward and flip-flop with the next fresh read. Deterministic repro fails before the fix (`StaleSweepSnapshot_RacingAnArtifactApply_DoesNotRemoveJustAppliedConnections`); `FreshEmptySweepSnapshot_StillRemovesDisappearedConnections` pins that row-90's legitimate disappearance still fires. SiteRuntime.Tests 647/647; rig redeployed, zero spurious removals/discards at boot, all rig-probe-1 tags Good. Memo §10 records the mechanism. **Postscript 2026-08-18 (merge `505f3569`, agent commit `f02e1166`): extended to the WHOLE snapshot.** The instance-config half had the same TOCTOU, and all three hazards were PROVEN real pre-fix: a stale sweep restarts an instance **backward** (`TryRestartStaleInstance` feeds the snapshot's old `ConfigJson` to `HandleDeploy` — a literal old-config redeploy; pipeline ownership only covers in-flight deploys and the old revision was never claimed), a read-pre-delete/disable **resurrects** a just-stopped instance (delete even clears the sweep claim), and the boot path skews `_deployedInstanceNames`. Fix: one `_storeWriteGeneration` over both tables, bumped at BOTH ends of every actor-thread config mutation (deploy persist, enable, disable, delete, rollback, artifact apply) — both ends because `ApplyDeployment`/`HandleDisable`/`HandleDelete` advance actor state BEFORE the off-thread write, so a completion-only bump (the row-101 original argument, which holds only for the artifact apply) leaves a window. Stale sweep discarded whole; stale boot load re-dispatched, capped at 5 consecutive (then applied with a Warning). CDC/`SiteReconciliationActor` writes deliberately never bump — discovering them is the sweep's purpose. 3 new fail-before tests + 2 fresh-generation guards pinning that row-72/78 legitimate repairs survive; SiteRuntime.Tests 652/652 x3; full suite green (one unrelated Communication.Tests gauge flake → row 102); rig probe: disable/enable/redeploy churn over multiple sweep ticks produced zero self-heal events on either site-a node, tags Good. Memo §11. | Observed by the 2026-08-17 row-100 fix session | Closed 2026-08-17; whole-snapshot extension 2026-08-18. | — |
| 102 | ~~**`SiteStreamSubscribersActiveGauge_GoesToOneOnConnect_AndBackToZeroOnCancel` is delta-hardened but still flakes — the baseline can't survive other contributors CLOSING**~~ **RESOLVED 2026-08-18 via the preferred option (row-96 pattern).** The survey found the gauge moves ONLY inside the real `SiteStreamGrpcServer.SubscribeInstance` handler, and exactly two test classes drive it: `SiteStreamGrpcServerTests` (the gauge test's own class) and `CleanupVerificationTests` — `GrpcOptionsWiringTests` instantiates the server but never opens a stream, and the client/aggregator suites subscribe against fakes, so none of those can perturb the counter. Both drivers now share the `[Collection("SiteStreamSubscriberGauge")]` xunit collection (comments on both classes + at the baseline poll explain why the relative read alone was insufficient), so no other contributor can close a stream between the baseline capture and the assertions. Verified 4/4 consecutive full-suite runs green (717/717), durations unchanged (22–25 s vs the prior 24 s) — the serialization is free. Original evidence: the test polls `baseline + 1`, but the process-global gauge's baseline is captured ONCE; capture while parallel classes' streams are up → they close → the gauge settles at this test's own 1 and `baseline+1` is unreachable forever — a fixed-baseline delta only survives contributors that hold constant or increase. | Observed 2026-08-18 (one full-suite failure, `Expected: 4, Actual: 1`; 3/3 green in isolation); fixed same day | Closed 2026-08-18. | — |
| 103 | **Two lesser TTL-bounded circuit-teardown-vs-async-setup races remain in CentralUI** (arch-review 2026-08-19 P6-6's optional tail, deliberately deferred by the fix). The permanent debug-stream leak was closed at `DebugView.razor` + `DebugStreamHub` (dispose-vs-connect take-once pattern, commit `11e623c2`), but two same-family instances with a *different shape* were left: (1) `TransportImport.razor.cs TryLoadAsync` (~:244) — a circuit torn down mid-load strands 1 of the 8 `MaxConcurrentImportSessions` slots until the 30-min `BundleSession` TTL reaps it (partially defeats row 43); (2) `Topology.razor` (~:201-214) — timer created before a disposal re-check. Both are TTL/lifetime-bounded (no permanent leak), which is why they missed the Medium bar. Fix is the same post-await `_disposed` re-check + take-once teardown, but routed through `BundleSession.ReleaseSession`/the wizard-step lifecycle rather than a session-stop call. | Arch-review 2026-08-19 P6-6 remediation (T10 explicitly scoped them out) | Open. | Next CentralUI hardening pass, or opportunistically when either file is next touched. |
| 103 | ~~**Two lesser TTL-bounded circuit-teardown-vs-async-setup races remain in CentralUI**~~ **RESOLVED 2026-08-20 (Low-wave T12, agent commit `c8452136`).** Both instances closed with the Medium-wave T10 take-once discipline: `TransportImport.razor.cs` re-checks disposal after `TryLoadAsync`'s await and releases the import slot through exactly-once ownership (dispose-during-start frees the `BundleSession` slot immediately instead of stranding it for the 30-min TTL), and `Topology.razor` creates its timer under the same post-await re-check so a disposed page leaves no live timer (asserted event-style, not by sleep). Original note: (arch-review 2026-08-19 P6-6's optional tail, deliberately deferred by the fix). The permanent debug-stream leak was closed at `DebugView.razor` + `DebugStreamHub` (dispose-vs-connect take-once pattern, commit `11e623c2`), but two same-family instances with a *different shape* were left: (1) `TransportImport.razor.cs TryLoadAsync` (~:244) — a circuit torn down mid-load strands 1 of the 8 `MaxConcurrentImportSessions` slots until the 30-min `BundleSession` TTL reaps it (partially defeats row 43); (2) `Topology.razor` (~:201-214) — timer created before a disposal re-check. Both are TTL/lifetime-bounded (no permanent leak), which is why they missed the Medium bar. Fix is the same post-await `_disposed` re-check + take-once teardown, but routed through `BundleSession.ReleaseSession`/the wizard-step lifecycle rather than a session-stop call. | Arch-review 2026-08-19 P6-6 remediation (T10 explicitly scoped them out) | Open. | Next CentralUI hardening pass, or opportunistically when either file is next touched. |
| 104 | **Existing site LocalDb files stay `auto_vacuum=NONE`; `PRAGMA incremental_vacuum` after a cap trim is a no-op on them** | Arch-review 2026-08-19 P4-6 (Low-wave T4, `47f03fe6`) | `auto_vacuum` is only honoured on a table-less database, and `LocalDbSchema.EnsureCreated` had always created LocalDb's bookkeeping tables before `SiteEventLogSchema.Apply` ran — so the pragma had never once taken effect on any real file. The fix (`SiteLocalDbSetup.EnsureAutoVacuumOnFreshFile`, run before `AddZbLocalDb`'s lazy singleton constructs `SqliteLocalDb`) only helps **fresh** files; an unconditional `VACUUM` against a live ~1 GB replicated file was explicitly rejected (long exclusive lock on the active node; the rewrite is CDC-invisible but blocks the sync pump). Every node on the docker rig today predates the fix, so freed `site_events` pages there are reused, not returned to the OS — the storage cap (row 81) still bounds the data volume, only the file size stops shrinking. | Operator-scheduled `VACUUM` in a maintenance window (both nodes of a pair stopped together), or when a site's file is next recreated. |
| 105 | **Design constraint (not a defect): a LocalDb-replicated table must never be the parent of a row-mutating `ON DELETE` foreign key (`CASCADE`/`SET NULL`/`SET DEFAULT`) from another table** | Arch-review 2026-08-19 P4-7 → LocalDb 0.3.3 (scadaproj `122940c` + review follow-up `9663279`, Low-wave T5) | `LwwApplier` upserts with `INSERT OR REPLACE` under `foreign_keys=ON`, so the implicit delete fires the child action while CDC capture is suppressed — the child rows change on the applying node only and the pair diverges silently. 0.3.3 makes `RegisterReplicated` refuse such a parent at registration (clear exception naming the child FK), which is a BOOT failure for the site node, so this is enforced, not advisory. None of the ten `SiteLocalDbSetup.ReplicatedTables` has any FK today (zero `REFERENCES` on the live rig file). Model cross-table cleanup as explicit application deletes on each table (as the 3-table deploy cascade already does), never as SQLite FK actions. | Whenever a schema change adds a foreign key between site tables, or an eleventh table joins `ReplicatedTables`. |
| 68 | ~~**`deploy.sh` run from a git worktree silently rehomes every node's bind-mount data dir**~~ **RESOLVED 2026-08-15 (guard added in the same close-out).** The compose files' bind mounts are relative (`./site-a-node-a/data`), so a deploy from a worktree checkout mounts the worktree's copies — the rig keeps running and looks healthy, but its SQLite stores split from the main checkout's and everything written there is orphaned by the next deploy from main. Bit the rows-46–53 close-out live: the seeding agent's `rig-probe-1` deploy persisted into worktree-homed dirs and "vanished" after the orchestrator's redeploy (no tombstone, no product bug — LWW/tombstone forensics ruled out deletion; the row sat in `.claude/worktrees/…/docker/site-a-node-a/data/site-localdb.db` the whole time). Both `docker/deploy.sh` and `docker-env2/deploy.sh` now refuse to run from a `.claude/worktrees/` path (override: `SCADABRIDGE_ALLOW_WORKTREE_DEPLOY=1`). | Found during the rows-46–53 close-out | Closed 2026-08-15. | — |
## Resolved (verified against the code 2026-07-10)
docs/plans/2026-08-19-low-findings-remediation-plan.md
+368
View File
@@ -0,0 +1,368 @@
# Low-Findings Remediation Plan — 2026-08-19 Arch Review
> **Status 2026-08-20: EXECUTED — all 15 tasks landed** (14 via dedicated `low/T*` worktrees cherry-picked onto
> `low/integration`, T5 as LocalDb **0.3.3** in scadaproj `122940c` + review follow-up `9663279`, published to the
> Gitea feed and pinned here). Honest outcomes of the two *plausible-needs-verification* items: **P3-4 reproduced**
> (deterministically, by construction; how often the >10s header path occurs in production is NOT demonstrated —
> noted in the test's doc comment) and **P7-5 reproduced** (4-instance cascade measured on a real SQLite-backed
> `ScadaBridgeDbContext`). Plan decisions 1–3 were all taken as written (P7-6 behaviour fix + `--force`; P4-6 real
> fix via `SiteLocalDbSetup.EnsureAutoVacuumOnFreshFile` — fresh files only, register row 104; P4-7 as 0.3.3, with
> the FK-parent constraint recorded as register row 105). Integration caught one cross-task regression (T12's
> Topology fixture vs. T13's new `IStaleInstanceProbe` ctor dependency) and one bundle-review Minor (T11's copied
> pattern re-rendered after Dispose) — both fixed in `57c8973b`. Register row 103 closed.
Scope: the **21 Low** findings left open after the High wave (`c4d3c8b8..d18f344e`) and the
Medium/Med-High wave (`d18f344e..9b67c7cb`, deployed to the rig), **plus register row 103**
(the two TTL-bounded lesser instances of P6-6, deferred by the Medium wave to "next
CentralUI hardening pass" — that pass is Phase 4 here). Baseline: `main @ 9b67c7cb`.
Structured as **five phases, one per subsystem section**. All 15 tasks are file-disjoint
across the whole plan, so phases may run concurrently under the usual rolling dispatch —
the phase grouping is thematic (one reviewer lens per phase at integration), not a
dependency ordering. The one true ordering constraint: T5 (LocalDb 0.3.3) must publish
before the pin bump lands on main.
**Model assignment rule** (same as the Medium wave): **Opus** for tasks whose defect is a
lifecycle/ordering race, a state-machine gap, or a cross-repo library invariant — anywhere
the fix can be subtly wrong while tests pass. **Sonnet** for pattern-mirroring, mechanical
guards, index/migration work, docs, and test hygiene where the correct pattern already
exists in-repo and the task is to copy it faithfully.
Execution model (unchanged): one dedicated **manual git worktree** per task (memory rule:
`isolation:worktree` is broken here), model-matched subagent, **pathspec commits**,
cherry-pick onto main, combined build + affected suites, `git diff` review, single merge
session.
## Shared constraints (every task)
- **Proof of failure:** every new test must be shown to fail against the unfixed code
(stash/unstash or targeted revert) before the fix lands. Where the defect is
timing-dependent and won't reproduce at test sizes, do what T1 did in the Medium wave:
inject a modeled delay to prove the mechanism, plus a timing-independent invariant test,
and record the honesty note in the test's doc comment.
- **No new sleep-then-assert** (register row 40): `AssertHoldsFor` / event-based waits;
Watch/Stop/ExpectTerminated on any ticker ≤1s (rows 82/95).
- **EF migrations:** build first, never `--no-build`; every migration ships the
handwritten-SQL twin under `docs/plans/sql/` with parity (Medium-wave T7 precedent).
- **Additive-only** message/DTO/proto evolution; no proto changes are expected in this
plan at all.
- **Register and CLAUDE.md edits happen at integration only**, never by task agents
(file contention). Integration closes row 103 (T12) and applies the P7-6 doc decision.
- Verifier confidence is quoted from the review: findings marked *plausible-needs-
verification* (P3-4, P7-5) require the agent to **confirm the defect first** and report
honestly if it does not reproduce — a guard test still lands, but the register note
must say what was and wasn't proven.
## Decisions made in this plan (owner may override)
1. **P7-6 takes the behavior fix, not the docs fix:** bulk `DeploySiteAsync` gains an
up-to-date pre-filter via `IStaleInstanceProbe` (skipped instances reported in the
existing `SkippedInstances` list with reason "up-to-date"), plus an explicit `Force`
flag (management command + CLI `deploy site --force`) that restores today's
unconditional behavior. Rationale: an unconditional redeploy **wipes static overrides
and native alarm state on unchanged instances** — that destructive side effect, not
the wording, is the hazard; and CLAUDE.md already promises "out-of-date" semantics.
2. **P4-6 takes the real fix, not the comment deletion:** run the `auto_vacuum`
pragma before `LocalDbSchema.EnsureCreated` via a documented pre-create hook/ordering
in `SiteLocalDbSetup`, with `VACUUM` fallback explicitly NOT attempted (too costly on
a 1 GB live file); if ordering proves impossible without a LocalDb API change, fall
back to deleting the misleading comment + register row, and say so.
3. **P4-7 ships as LocalDb 0.3.3** (wire-compatible, no schema change), published to the
Gitea feed and pinned here in the same integration session — same flow as 0.3.2.
---
## Phase 1 — Site Runtime / DCL / Communication (streams)
### T1 (Sonnet) — P1-4: projected single-row reads in DeploymentManagerActor
**Files:** `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs`
(:1573, :2061, :1489), `Services/SiteStorageService.cs` (:84), its interface, tests.
`HandleEnsureInstanceStarted`, `HandleDeploymentStateQuery`, and `HandleEnable` each call
`GetAllDeployedConfigsAsync` — materializing ALL configs **including ConfigJson blobs** —
to `FirstOrDefault` one row. A volume-restore reconcile healing hundreds of instances does
hundreds of full-table reads; the sweep tick needs only `(name, revision, is_enabled)`.
**Fix:** add `GetDeployedConfigAsync(name)` (single row) and a metadata-only projection
for the sweep; switch the three call sites; leave semantics identical (T1 of the Medium
wave already threads `StoreGenerationAtRead` here — do not disturb the generation fencing;
capture the generation at the same point the full-table read did). Tests: projection
returns identical decisions to the old path; generation capture still fences (existing
tests keep passing).
### T2 (Opus) — P2-5 + P2-6: DataConnectionActor probe seeding + Connecting-state gap
**File:** `src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Actors/DataConnectionActor.cs`
(+ `Adapters/OpcUaDataConnection.cs` read-only for the `_disconnectFired` trace), tests.
- **P2-5:** initial subscribe (:984) and reconnect resubscribe (:2329-2344) call
`SeedTagsAsync`; the resolution-probe path (:1975-1988) does not. OPC UA masks it via
the initial notification; MxGateway advise can be `VT_EMPTY`, so a late-resolving
static tag stays Uncertain forever. **Fix:** seed newly probe-resolved tags — same
call the other two paths make.
- **P2-6:** `Connecting` has no `AdapterDisconnected` case → Unhandled. A connect-then-
immediate-drop can enqueue the disconnect before `ConnectResult(true)`; the adapter's
once-only `_disconnectFired` guard (OpcUaDataConnection.cs:486) means no later
`Disconnected` ever arrives, so a zero-subscription connection sits Connected on a dead
adapter. **Fix:** stash (or handle) `AdapterDisconnected` in Connecting so it is
replayed/processed after the state transition; respect the Medium-wave T3
`AdapterGeneration` fencing (a stale generation's disconnect must still be dropped).
Opus because the fix sits inside the Become/Stash machine next to fresh generation
logic. Tests: ordered enqueue disconnect-before-ConnectResult ends Disconnected/
reconnecting, not Connected; stale-generation disconnect still ignored.
### T3 (Opus) — P3-4 + P3-5: stream open-attempt deadline + duplicate-subscribe cap order
**Files:** `src/ZB.MOM.WW.ScadaBridge.Communication/Actors/SiteAlarmAggregatorActor.cs`
(:800, :444), `Grpc/SiteStreamGrpcServer.cs` (:368-385), tests.
- **P3-4 (plausible-needs-verification):** if response headers take >10s
(`ConnectedHeaderTimeout`) and an alarm-only site stays quiet,
`GrpcAlarmStreamConnected` never fires; `_streamDown` stays true but `_openInFlight`
stays pinned, so `OnReconcileTick`'s reopen is skipped until the 4h stream lifetime
cycles. Mitigated by the server header flush — **verify first** whether the starvation
is still reachable; regardless, add an **open-attempt deadline** that clears
`_openInFlight` so reconcile can retry (self-healing beats reachability analysis).
- **P3-5:** duplicate-subscribe cancels the incumbent stream (:368-379) **then** checks
the cap (:385): at cap, a reconnect kills the healthy old stream and refuses the
replacement (`ResourceExhausted`) → zero streams until retry. **Fix:** treat the
duplicate case as a replacement (exclude it from the cap check) or check the cap before
cancelling; must not regress the newest-wins semantics. Tests: at-cap reconnect ends
with exactly one live stream; cap still refuses a genuinely new subscriber.
## Phase 2 — LocalDb / Site Event Log
### T4 (Sonnet) — P4-5 + P4-6: purge startup yield + auto_vacuum reality
**Files:** `src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/EventLogPurgeService.cs`
(:169, :208-212, :516-519), `SiteEventLogSchema.cs` (:42-46),
`src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbSetup.cs`, tests.
- **P4-5:** `ExecuteAsync` runs the initial `RunPurge()` synchronously before its first
await, so `StartAsync` blocks every later hosted service (including `AkkaHostedService`)
behind the purge — ~17s ceiling-limited, ~285s unpaced worst case; a throwing gate is
also treated as active (:208-212). **Fix:** `await Task.Yield()` first (and make the
gate's throw path fail closed with a logged fallback, not "active"). Test: hosted-service
start returns promptly with a slow purge stubbed in.
- **P4-6:** `auto_vacuum=INCREMENTAL` is only honored on a table-less database, and
`LocalDbSchema.EnsureCreated` runs first — so the consolidated file is permanently
`auto_vacuum=NONE` and the per-trim `PRAGMA incremental_vacuum` is a silent no-op while
the adjacent comment claims otherwise (the standalone `auditlog.db` does it right,
`SqliteAuditWriter.cs:154-157`). **Fix per plan decision 2:** issue the pragma before
any table exists (pre-`EnsureCreated` ordering in `SiteLocalDbSetup`); if that requires
a LocalDb API change, do NOT make one in this task — delete the trap comment, log a
one-time Information note that vacuum is unavailable on legacy files, and report for a
register row. Note either way: existing rig files already have tables, so the pragma
only helps fresh files — document that in the code comment honestly. Test: fresh file
reports `PRAGMA auto_vacuum = 2`.
### T5 (Opus) — P4-7: LwwApplier latent hazards → LocalDb 0.3.3 (cross-repo)
**Files (scadaproj):** `~/Desktop/scadaproj/ZB.MOM.WW.LocalDb/src/.../Internal/LwwApplier.cs`
(:95-99, :273), `SqliteLocalDb.cs` (RegisterReplicated), `Directory.Build.props`
(0.3.2 → 0.3.3), tests. **Files (here, at integration):** `Directory.Packages.props` pin.
- **Hazard 1:** `INSERT OR REPLACE`'s implicit delete fires FK `ON DELETE CASCADE` with
capture suppressed (`foreign_keys=ON`) → silent divergence for any future FK'd
replicated table (none of the ten today — verified by the review). **Fix:**
`RegisterReplicated` gains an FK guard — refuse (clear exception) to register a table
that is the parent of an FK with cascading delete from another table, and document the
constraint in the LocalDb README.
- **Hazard 2:** the tombstone apply path runs `DeleteAsync` + `WriteRowVersionAsync` as
two statements under a poison catch — a failure between them leaves a deleted row with
a stale live ledger (SnapshotStreamer warns+skips). **Fix:** make the pair atomic
(single transaction), preserving the poison-catch semantics around the transaction.
Wire-compatible, no schema change, no ScadaBridge code change. Publish 0.3.3 via
`build/push.sh` (env-held `GITEA_NUGET_SOURCE`/`GITEA_NUGET_KEY`), bump the three-package
pin here, restore/build 0/0. Opus: replication-library invariants where a wrong fix
diverges pairs silently. Tests: FK-parent registration throws; injected failure between
delete and ledger write leaves both or neither. Integration also updates the umbrella
`scadaproj/CLAUDE.md` index (0.3.3 entry) per the cross-repo propagation rule.
## Phase 3 — Central Outbox / Audit / KPI
### T6 (Sonnet) — P5-3 + P5-4: Notifications list index + operator-path status guard
**Files:** `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/NotificationOutboxRepository.cs`
(:192-203, :280-294), a new EF migration + `docs/plans/sql/` twin,
`src/ZB.MOM.WW.ScadaBridge.NotificationOutbox/NotificationOutboxActor.cs`
(:1088-1129, :1172-1213), tests.
- **P5-3:** the list page sorts `CreatedAt DESC, NotificationId DESC` but every index
leads elsewhere → the default page is a 2× whole-table scan/sort (COUNT + page).
**Fix:** add `IX_Notifications_Created_NotificationId (CreatedAt DESC, NotificationId
DESC)` — exact mirror of the Medium wave's `IX_SiteCalls_Created_TrackedOperationId`
(T7), including the VarChar-typed `FromSqlRaw` parameter check and the handwritten SQL.
- **P5-4:** operator Retry/Discard is read-check-write keyed on `NotificationId` only;
detached `PipeTo` tasks interleaving with the dispatcher can stamp a `Delivered` row
`Discarded` with stale fields. **Fix:** add a `Status == Parked` predicate to the
operator-path `ExecuteUpdate` (conditional update, rows-affected checked — the CAS
shape T-P6-1 used); surface "already progressed" to the caller instead of silent
overwrite. Tests: migration parity; Retry/Discard against a row that just went
Delivered is a no-op with the honest reply.
### T7 (Sonnet) — P5-5: cross-validate the rate-fold interval triple
**Files:** `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/KpiHistoryRepository.cs`
(:169-172), the options classes' validators (`KpiHistoryOptions`, `SiteCallAuditOptions`,
the outbox `DeliveredKpiWindow` owner), tests.
Rate classification silently assumes `DeliveredKpiWindow` (1 min) ==
`SiteCallAuditOptions.KpiInterval` (1 min) == `KpiHistoryOptions.SampleInterval` (60s);
widening `DeliveredKpiWindow` to 15 min makes rollups over-count ~15×. **Fix (pick the
robust half):** compute the fold against `SampleInterval` directly where possible, and
add eager cross-validation (ValidateOnStart, the §1.5 pattern) that fails boot with a
message naming all three keys when they diverge — do not leave a validator-only fix if
the computation can simply stop depending on the equality. Tests: divergent config fails
validation with the three keys named; fold correct at a non-default SampleInterval.
### T8 (Sonnet) — P5-6: deterministic synthetic ReconciliationAbandoned rows
**File:** `src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/SiteAuditReconciliationActor.cs`
(:312-336, :362-364), tests.
After `MaxPermanentInsertAttempts` removal, the cursor held at `since` by a co-batch
unresolved row re-pulls the abandoned row; its attempt counter restarts and a **new**
synthetic `ReconciliationAbandoned` row is minted with a fresh `EventId` every 5-tick
cycle. **Fix:** make the synthetic `EventId` deterministic — derived from the lost row's
`EventId` (e.g. namespaced GUID) — so re-abandonment is idempotent under the existing
insert-if-not-exists ingest; keep an in-memory abandoned-set only as a cheap short-circuit,
never as the correctness mechanism (it dies with the singleton). Tests: same lost EventId
abandoned twice yields one row; different lost rows yield distinct synthetic ids.
## Phase 4 — Central UI / Management / Deployment (the "CentralUI hardening pass")
### T9 (Sonnet) — P6-10: strict enum parsing in ManagementActor (row-88 class)
**File:** `src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementActor.cs`
(:2922, :2937, :2854, :2867 — alarm trigger type, ParseDataType, ParseElementDataType),
tests.
Raw `Enum.Parse` accepts ordinals ("2") and undefined values ("99" → deploy-time bomb);
a typo surfaces as a masked "internal error". **Fix:** apply the `ParseDeploymentStatus`
guard pattern (:2745) to all three parse sites — names only, defined values only, honest
error naming the field and the valid set. Mind the Medium-wave T8 nuance: `Enum.TryParse`
accepts comma-joined name lists and bare ordinals; the guard must reject both (reuse or
mirror `ParseEnumListStrict`'s comma-token rejection from Commons where a single value is
expected). Tests: ordinal, undefined, comma-joined, and typo inputs all rejected with the
field-naming error; every defined name accepted.
### T10 (Sonnet) — P6-11: request-generation guard on Deployments page
**File:** `src/ZB.MOM.WW.ScadaBridge.CentralUI/.../Deployments.razor` (:428, :500-552),
bUnit tests.
Three fetch initiators (search, paging, timer) interleave at awaits and a stale flight
overwrites a newer grid; `_currentPage = 1` is also written on the Timer thread (:428).
**Fix:** a monotonic request generation captured at fetch start and checked at apply
(last-write-wins by generation, not by completion order); marshal the timer's state
writes through `InvokeAsync`. Tests: delayed stale response arriving after a newer one
does not overwrite; timer refresh does not reset the page mid-search.
### T11 (Sonnet) — P6-12: dispose sandbox-run CTS in the two script forms
**Files:** `src/ZB.MOM.WW.ScadaBridge.CentralUI/.../ApiMethodForm.razor` (:233),
`SharedScriptForm.razor`, bUnit tests.
Neither form implements `IDisposable`, so an abandoned sandbox run parks a thread on
`GetAwaiter().GetResult()` until the script timeout and its continuation touches disposed
state. `TemplateScriptDialog.razor:354-358` does it right. **Fix:** copy that pattern
exactly — `IDisposable`, cancel+dispose `_runCts`, guard the continuation on disposal.
Tests: disposing the form mid-run cancels the sandbox run; continuation after dispose is
a no-op.
### T12 (Sonnet) — Register row 103: TransportImport slot strand + Topology timer
**Files:** `src/ZB.MOM.WW.ScadaBridge.CentralUI/.../TransportImport.razor.cs` (:244),
`Topology.razor` (:201-214), bUnit tests.
The two TTL-bounded lesser instances of the P6-6 class, deferred by the Medium wave:
`TryLoadAsync` abandoned mid-start strands 1 of 8 import session slots for 30 min
(partially defeating row 43), and `Topology.razor`'s timer survives disposal in the same
window shape. **Fix:** apply the Medium wave T10 pattern (`DebugStreamHub`'s take-once
`_sessionLock`/`TakeSession()`) — post-await disposal re-check with exactly-once cleanup
ownership; timer created under the same discipline. Closing row 103 happens at
integration. Tests: dispose-during-start releases the import slot immediately; disposed
Topology page leaves no live timer (Watch/ExpectTerminated-style assertion, not sleep).
### T13 (Opus) — P7-5 + P7-6: bulk-deploy DbContext poisoning + out-of-date pre-filter
**Files:** `src/ZB.MOM.WW.ScadaBridge.TemplateEngine/.../DeploymentService.cs`
(:677-683, :804), `StaleInstanceProbe`/`IStaleInstanceProbe` call path, the bulk-deploy
management command + `SiteDeploymentSummary` (additive), CLI `deploy site` (`--force`),
tests.
- **P7-5 (plausible-needs-verification):** the three bulk phases share one scoped
DbContext; a persistent `SaveChanges` failure on one tracked entity rethrows on every
later save including `MarkDeploymentFailedAsync` cleanup — later instances fail and
records stay `InProgress` (locks are released; reconcile self-heals next deploy).
**Verify first** with a forced save failure; **fix:** detach the failed entries on
cleanup-save failure so subsequent instances save cleanly.
- **P7-6 (plan decision 1):** implement the up-to-date pre-filter using
`IStaleInstanceProbe` (its memo timing was fixed by Medium-wave P7-4 —
`FlattenSession.VisitedTemplateVersions` records at the right point, so the probe is
trustworthy here); up-to-date instances land in the existing additive
`SkippedInstances` list with reason "up-to-date"; `Force = true` (management command,
additive; CLI `--force`) restores unconditional behavior. Must compose with the P7-2
Disabled-skip: Disabled wins (never deployed even under `--force` at bulk scope —
single-instance deploy remains the explicit re-enable path). Opus: deployment-semantics
change composing three prior fixes. Tests: unchanged instance skipped with reason and
its static overrides untouched; `--force` deploys it; Disabled skipped under both;
poisoned entity no longer cascades.
## Phase 5 — Tests / Infra / Docs
### T14 (Sonnet) — P8-8 + P8-9: residual test-health instances
**Files:** `tests/.../DataConnectionLayer.Tests/Actors/DataConnectionActorBatchTests.cs`
(:181-184, :206, :214), `SiteAlarmLiveCacheServiceTests.cs` (:453-454, :471-472),
`StoreAndForwardServiceTests.cs` (:891-892), `KpiHistoryRecorderActorTests.cs`
(:1043-1049), `tests/.../Host.Tests/ScadaBridgeTelemetryTests.cs` (:30).
- **P8-8:** the remaining row-40-class wall-clock sleeps outside Communication.Tests —
apply the Medium-wave T17 treatment (`AssertHoldsFor` for absence assertions,
event/condition waits for presence; no widened tolerances that mask real regressions).
- **P8-9:** `SetQueueDepthProvider(() => 5)` mutates the process-global row-32 slot with
no clear and no `[Collection]` — a cross-assembly landmine. **Fix:** compare-and-clear
in `finally` (restore prior provider), plus a collection if any second reader appears.
Proof-of-failure here = demonstrate the pre-fix flake mechanism (e.g. run the sleep
assertion under load) or, where impractical, the honesty-note pattern.
### T15 (Sonnet) — P8-7 + P8-10: docs drift + env2 parity tail
**Files:** `docker/README.md` (:74, :155), `docker-env2/README.md` (:56),
`docker-env2/docker-compose.yml` (:34 area), `docker-env2/teardown.sh` (:13, :163-171),
`CLAUDE.md` (:15, at integration if contended), `infra/` compose comment.
- **P8-7:** both rig READMEs still list the retired `scadabridge-ldap` container; the
infra compose retirement comment is true only of env2; CLAUDE.md names Traefik as an
`infra/` service (it is per-rig, `docker/docker-compose.yml:411`) and points at
`scadaproj/infra/glauth` where `docker/` actually uses `glauth-redundant`. **Fix:**
correct all five spots to match the running rigs (LDAP = shared GLAuth
`10.100.0.35:3893`; name the actual glauth config each rig uses).
- **P8-10:** env2 maps no host port for the central gRPC control plane (docker/ maps
9013/9014→8083; 9113/9114 are free — use them), and `teardown.sh` still claims data is
"preserved in node directories" — untrue since the named-volume migration — with no
volume-rm recipe. **Fix:** add the port mappings + rewrite teardown's data note with an
explicit `docker volume rm` recipe (destructive step clearly labeled). Verify env2
still boots: `bash docker-env2/deploy.sh` gate (it runs concurrently with docker/ —
do not touch docker/ ports).
---
## Integration (single session, after all phases report)
1. Cherry-pick order: T5 (LocalDb 0.3.3 publish + pin) first or last-but-isolated;
everything else in phase order. Interim build every ~6 picks.
2. Combined `dotnet build ZB.MOM.WW.ScadaBridge.slnx` 0/0, then the full sweep
(known flakes: `SiteConnectionUpGauge_*`, DCL027, IO-errno under load — rerun in
isolation before calling regression).
3. Register: close **row 103** (T12); add rows for anything a task honestly deferred
(P4-6 fallback branch, P3-4 if starvation didn't reproduce); new row for the P4-7
FK-guard constraint ("replicated tables must not be cascade-delete FK parents").
4. CLAUDE.md: bulk-deploy bullet gains the out-of-date filter + `--force` sentence
(P7-6); LocalDb bullet gains the 0.3.3 line; fix the :15 Traefik/glauth drift if T15
left it for integration. Umbrella `scadaproj/CLAUDE.md`: 0.3.3 entry.
5. `git diff` review, single descriptive merge-session commit set; push and rig redeploy
only on explicit request.
docs/plans/sql/AddNotificationsListSortIndex.sql
+51
View File
@@ -0,0 +1,51 @@
-- arch-review 2026-08-19 P5-3 (register row-35 class): the Notifications list
-- page's sort key gets its own index.
--
-- NotificationOutboxRepository.QueryAsync orders (CreatedAt DESC, NotificationId
-- DESC) (NotificationId breaks CreatedAt ties for deterministic OFFSET paging).
-- None of the existing Notifications indexes lead unconditionally on CreatedAt:
-- - IX_Notifications_Status_NextAttemptAt requires an equality on Status first.
-- - IX_Notifications_SourceSiteId_CreatedAt requires an equality on SourceSiteId first.
-- - IX_Notifications_Delivered is filtered to Status = 'Delivered' only.
-- So an unfiltered (or lightly filtered) page fell back to a full clustered-index
-- scan + sort. IX_Notifications_Created_NotificationId (CreatedAt DESC,
-- NotificationId DESC) makes a forward scan of the index deliver rows in the
-- requested order directly, so a page reads a Top-N off the index instead of
-- sorting the whole table on every COUNT + page request.
--
-- Notifications is small enough today that a plain (offline) index build is
-- fine. On an Enterprise edition instance where the table has grown large and
-- the Central UI must stay responsive, add WITH (ONLINE = ON) to the CREATE
-- INDEX below.
--
-- Run with QUOTED_IDENTIFIER ON. Several tables in this database carry filtered
-- indexes and persisted computed columns, and SQL Server refuses DDL/DML against
-- them with error 1934 when the session has QUOTED_IDENTIFIER OFF -- which is
-- exactly what `sqlcmd` gives you by default (pass -I, or keep this header). The
-- SETs sit in their own batch so they are in force when the statements after the
-- GO parse.
SET QUOTED_IDENTIFIER ON;
SET ANSI_NULLS ON;
GO
BEGIN TRANSACTION;
IF NOT EXISTS (
SELECT * FROM [__EFMigrationsHistory]
WHERE [MigrationId] = N'20260819201210_AddNotificationsListSortIndex'
)
BEGIN
EXEC(N'CREATE INDEX [IX_Notifications_Created_NotificationId]
ON [Notifications] ([CreatedAt] DESC, [NotificationId] DESC)');
END;
IF NOT EXISTS (
SELECT * FROM [__EFMigrationsHistory]
WHERE [MigrationId] = N'20260819201210_AddNotificationsListSortIndex'
)
BEGIN
INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion])
VALUES (N'20260819201210_AddNotificationsListSortIndex', N'10.0.7');
END;
COMMIT;
GO
infra/docker-compose.yml
+10 -5
View File
@@ -41,11 +41,16 @@ services:
- scadabridge-net
restart: unless-stopped
# RETIRED 2026-06-04: superseded by the shared dev GLAuth on 10.100.0.35:3893
# (scadaproj/infra/glauth/). The central nodes now bind there (see
# docker/ + docker-env2 central-node appsettings: Ldap:Server=10.100.0.35).
# Kept here, commented, for rollback — uncomment + `docker compose up -d ldap`
# and revert the central-node Server back to "scadabridge-ldap".
# RETIRED 2026-06-04: originally superseded by the shared dev GLAuth on
# 10.100.0.35:3893 (scadaproj/infra/glauth/). That binding is itself now stale —
# the shared GLAuth's serviceaccount password was rotated (SEC-36) and is no
# longer valid, so both docker/ and docker-env2 central nodes now bind to the
# local redundant GLAuth pair instead: zb-ldap-primary on
# host.docker.internal:3893 with zb-ldap-backup on host.docker.internal:3894 as
# FallbackServers (source of truth: scadaproj/infra/glauth-redundant; see each
# central node's appsettings.Central.json). Kept here, commented, for rollback —
# uncomment + `docker compose up -d ldap` and point the central-node Server back
# at "scadabridge-ldap".
# ldap:
# image: glauth/glauth:latest
# container_name: scadabridge-ldap
src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/SiteAuditReconciliationActor.cs
+149 -41
View File
@@ -1,3 +1,4 @@
using System.Security.Cryptography;
using System.Text.Json;
using Akka.Actor;
using Microsoft.Extensions.DependencyInjection;
@@ -125,6 +126,34 @@ public class SiteAuditReconciliationActor : ReceiveActor
/// </summary>
private readonly Dictionary<Guid, int> _failedInsertAttempts = new();
/// <summary>
/// EventIds already permanently abandoned earlier in this process's
/// lifetime — a cheap short-circuit ONLY, not a correctness mechanism.
/// </summary>
/// <remarks>
/// <para>
/// The cursor can be held at <c>since</c> indefinitely by an unrelated
/// co-batch row that keeps failing (see <c>hasUnresolvedFailure</c> in
/// <see cref="PullSiteAsync"/>), which means an already-abandoned row can
/// keep reappearing in the pulled batch tick after tick. Without this
/// set, each reappearance would re-run the full
/// <see cref="MaxPermanentInsertAttempts"/>-attempt retry cycle (since
/// <see cref="_failedInsertAttempts"/> was cleared when the row was first
/// abandoned) purely to re-derive the same conclusion. Checking this set
/// first skips straight to "already resolved" instead of burning another
/// ~25 minutes per reappearance.
/// </para>
/// <para>
/// This is deliberately NOT what makes re-abandonment safe —
/// <see cref="DeriveAbandonmentEventId"/>'s determinism is. This set is
/// in-memory only and is empty again after a singleton restart/failover,
/// at which point the full retry-then-abandon cycle simply runs again;
/// the deterministic EventId still makes that re-run's synthetic insert
/// collapse into the SAME row instead of minting a duplicate.
/// </para>
/// </remarks>
private readonly HashSet<Guid> _abandonedEventIds = new();
/// <summary>
/// Number of consecutive central-insert failures before a row
/// is permanently abandoned with a Critical log entry and the cursor is
@@ -133,6 +162,15 @@ public class SiteAuditReconciliationActor : ReceiveActor
/// </summary>
private const int MaxPermanentInsertAttempts = 5;
/// <summary>
/// Fixed namespace GUID mixed into <see cref="DeriveAbandonmentEventId"/>'s
/// hash input. Any fixed value works — it exists only to separate this
/// derivation's ID space from other GUID-from-GUID derivations that might
/// ever reuse the same technique elsewhere in the codebase.
/// </summary>
private static readonly Guid AbandonmentEventIdNamespace =
new("f2b6a6d1-6e2b-4b8b-9b3e-9a6b6a2c1d4e");
private ICancelable? _timer;
/// <summary>
@@ -285,54 +323,70 @@ public class SiteAuditReconciliationActor : ReceiveActor
foreach (var evt in response.Events)
{
var advanceForThisRow = false;
try
if (_abandonedEventIds.Contains(evt.EventId))
{
// Idempotent repository write: duplicate EventIds (from a
// concurrent push, or a retry of this very pull) collapse to
// a no-op on InsertIfNotExistsAsync.
// IngestedAtUtc is a DetailsJson field on the canonical record —
// stamp it via the projection helper.
var ingested = AuditRowProjection.WithIngestedAtUtc(evt, nowUtc);
await repository.InsertIfNotExistsAsync(ingested).ConfigureAwait(false);
_failedInsertAttempts.Remove(evt.EventId);
// Cheap short-circuit ONLY (see the field's doc comment): this row was
// already permanently abandoned earlier in this process's lifetime and
// keeps reappearing because an unrelated co-batch row is holding the
// cursor at `since`. Skip straight to "resolved" instead of re-running
// the whole MaxPermanentInsertAttempts retry cycle — the insert is known
// to fail, and DeriveAbandonmentEventId would make a re-run's synthetic
// row collapse into the existing one anyway.
advanceForThisRow = true;
}
catch (Exception ex)
else
{
// Per-row catch so one bad event does not abandon
// the rest of the batch. Track the failure count per EventId —
// below MaxPermanentInsertAttempts the cursor is HELD BACK so
// the next tick re-pulls and retries; at the threshold the row
// is permanently abandoned (LogCritical + cursor advances past)
// to keep a truly broken row from blocking all subsequent
// progress for the site.
var attempts = _failedInsertAttempts.GetValueOrDefault(evt.EventId) + 1;
_failedInsertAttempts[evt.EventId] = attempts;
if (attempts >= MaxPermanentInsertAttempts)
try
{
_logger.LogCritical(
ex,
"Permanently abandoning AuditEvent {EventId} from site {SiteId} after {Attempts} consecutive insert failures; cursor will advance past it.",
evt.EventId,
site.SiteId,
attempts);
// Idempotent repository write: duplicate EventIds (from a
// concurrent push, or a retry of this very pull) collapse to
// a no-op on InsertIfNotExistsAsync.
// IngestedAtUtc is a DetailsJson field on the canonical record —
// stamp it via the projection helper.
var ingested = AuditRowProjection.WithIngestedAtUtc(evt, nowUtc);
await repository.InsertIfNotExistsAsync(ingested).ConfigureAwait(false);
_failedInsertAttempts.Remove(evt.EventId);
advanceForThisRow = true;
await EmitAbandonmentRecordAsync(repository, evt, site.SiteId, ex, nowUtc)
.ConfigureAwait(false);
}
else
catch (Exception ex)
{
_logger.LogError(
ex,
"Reconciliation ingest failed for AuditEvent {EventId} from site {SiteId} (attempt {Attempts}/{Max}); cursor held back for retry.",
evt.EventId,
site.SiteId,
attempts,
MaxPermanentInsertAttempts);
hasUnresolvedFailure = true;
// Per-row catch so one bad event does not abandon
// the rest of the batch. Track the failure count per EventId —
// below MaxPermanentInsertAttempts the cursor is HELD BACK so
// the next tick re-pulls and retries; at the threshold the row
// is permanently abandoned (LogCritical + cursor advances past)
// to keep a truly broken row from blocking all subsequent
// progress for the site.
var attempts = _failedInsertAttempts.GetValueOrDefault(evt.EventId) + 1;
_failedInsertAttempts[evt.EventId] = attempts;
if (attempts >= MaxPermanentInsertAttempts)
{
_logger.LogCritical(
ex,
"Permanently abandoning AuditEvent {EventId} from site {SiteId} after {Attempts} consecutive insert failures; cursor will advance past it.",
evt.EventId,
site.SiteId,
attempts);
_failedInsertAttempts.Remove(evt.EventId);
_abandonedEventIds.Add(evt.EventId);
advanceForThisRow = true;
await EmitAbandonmentRecordAsync(repository, evt, site.SiteId, ex, nowUtc)
.ConfigureAwait(false);
}
else
{
_logger.LogError(
ex,
"Reconciliation ingest failed for AuditEvent {EventId} from site {SiteId} (attempt {Attempts}/{Max}); cursor held back for retry.",
evt.EventId,
site.SiteId,
attempts,
MaxPermanentInsertAttempts);
hasUnresolvedFailure = true;
}
}
}
@@ -372,14 +426,27 @@ public class SiteAuditReconciliationActor : ReceiveActor
/// row when a pulled event is permanently abandoned — so the loss is
/// queryable in the Audit Log rather than living only in the Critical log line.
/// The synthetic row is built through the same
/// <see cref="ScadaBridgeAuditEventFactory"/> the ingest path uses (fresh
/// <c>EventId</c>, <c>Actor="system"</c>, channel preserved from the lost row's
/// <see cref="ScadaBridgeAuditEventFactory"/> the ingest path uses (a
/// <see cref="DeriveAbandonmentEventId"/>-derived <c>EventId</c>,
/// <c>Actor="system"</c>, channel preserved from the lost row's
/// <c>Category</c> where parseable) and carries the abandoned <c>EventId</c>,
/// the source site, and the final error in <c>Extra</c>. It runs in its own
/// try/catch: the synthetic row is small and well-formed, so the row-specific
/// fault that killed the original will not recur; if it somehow does, log and
/// continue — the abandonment cursor must never be blocked a second time.
/// </summary>
/// <remarks>
/// The synthetic <c>EventId</c> is deterministic (derived from
/// <c>lost.EventId</c>), not fresh, specifically so this call is idempotent:
/// a co-batch row that never resolves can hold the reconciliation cursor at
/// <c>since</c> indefinitely, causing the SAME lost row to be re-pulled and
/// re-abandoned on a later tick (its <see cref="_failedInsertAttempts"/> entry
/// was cleared on the first abandonment, so it restarts). Every re-abandonment
/// of the same lost row now derives the same synthetic id, so
/// <see cref="IAuditLogRepository.InsertIfNotExistsAsync"/> collapses every
/// repeat into the first insert instead of minting an unbounded number of
/// duplicate rows for one lost event.
/// </remarks>
private async Task EmitAbandonmentRecordAsync(
IAuditLogRepository repository,
ZB.MOM.WW.Audit.AuditEvent lost,
@@ -403,6 +470,7 @@ public class SiteAuditReconciliationActor : ReceiveActor
});
var synthetic = ScadaBridgeAuditEventFactory.Create(
eventId: DeriveAbandonmentEventId(lost.EventId),
channel: channel,
kind: AuditKind.ReconciliationAbandoned,
status: AuditStatus.Failed,
@@ -428,6 +496,46 @@ public class SiteAuditReconciliationActor : ReceiveActor
}
}
/// <summary>
/// Derives a deterministic synthetic <see cref="ZB.MOM.WW.Audit.AuditEvent.EventId"/>
/// for the <see cref="AuditKind.ReconciliationAbandoned"/> row minted when
/// <paramref name="lostEventId"/> is permanently abandoned.
/// </summary>
/// <remarks>
/// <para>
/// Pure function of <paramref name="lostEventId"/> alone — no clock, no
/// randomness, no actor state. Same input always produces the same output,
/// including across a process restart, which is what makes re-abandonment
/// idempotent under <see cref="IAuditLogRepository.InsertIfNotExistsAsync"/>
/// even after a singleton failover wipes every in-memory tracking
/// dictionary this actor keeps (<see cref="_failedInsertAttempts"/>,
/// <see cref="_abandonedEventIds"/>, the cursors themselves).
/// </para>
/// <para>
/// Built as a name-based (RFC 4122 "version 5"-shaped) GUID: SHA-256 over
/// a fixed namespace GUID concatenated with <paramref name="lostEventId"/>'s
/// bytes, truncated to 16 bytes with the version/variant bits set so the
/// result is a well-formed GUID. This is not a cryptographic use of
/// SHA-256 — collision resistance against an adversarial input is not the
/// goal, only a stable, well-distributed GUID-to-GUID mapping so distinct
/// lost rows get distinct synthetic ids.
/// </para>
/// </remarks>
internal static Guid DeriveAbandonmentEventId(Guid lostEventId)
{
Span<byte> input = stackalloc byte[32];
AbandonmentEventIdNamespace.TryWriteBytes(input[..16]);
lostEventId.TryWriteBytes(input[16..]);
Span<byte> hash = stackalloc byte[32];
SHA256.HashData(input, hash);
var guidBytes = hash[..16].ToArray();
guidBytes[6] = (byte)((guidBytes[6] & 0x0F) | 0x50); // version 5
guidBytes[8] = (byte)((guidBytes[8] & 0x3F) | 0x80); // RFC 4122 variant
return new Guid(guidBytes);
}
/// <summary>
/// Flips the per-site stalled flag based on whether this tick drained the
/// queue. A "draining" cycle is one where the server reported no more rows
src/ZB.MOM.WW.ScadaBridge.CLI/Commands/DeployCommands.cs
+12 -1
View File
@@ -55,14 +55,25 @@ public static class DeployCommands
private static Command BuildSite(Option<string> urlOption, Option<string> formatOption, Option<string> usernameOption, Option<string> passwordOption)
{
var siteIdOption = new Option<int>("--site-id") { Description = "Target site ID", Required = true };
// A flag, not a value option: absent means the default (skip unchanged
// instances), present means redeploy them. Disabled instances are skipped
// either way — force is not an override for that.
var forceOption = new Option<bool>("--force")
{
Description =
"Redeploy instances whose config is unchanged (default: skip them — a redeploy " +
"resets static overrides and native alarm state at the site)",
};
var cmd = new Command("site") { Description = "Deploy every deployable instance at a site" };
cmd.Add(siteIdOption);
cmd.Add(forceOption);
cmd.SetAction(async (ParseResult result) =>
{
var siteId = result.GetValue(siteIdOption);
var force = result.GetValue(forceOption);
return await CommandHelpers.ExecuteCommandAsync(
result, urlOption, formatOption, usernameOption, passwordOption,
new MgmtDeploySiteCommand(siteId),
new MgmtDeploySiteCommand(siteId, force),
// A bulk deploy is N instance round-trips; the default 30 s client
// timeout would abandon the request while the server is still
// applying. Matches the server-side long-running Ask window.
src/ZB.MOM.WW.ScadaBridge.CLI/README.md
+18 -2
View File
@@ -974,19 +974,35 @@ Site round-trips run with bounded concurrency
`SiteDeploymentTimeoutPerInstance` (default 120 s), so one wedged instance cannot stall
the batch. The client waits up to 5 minutes.
Two classes of instance are **skipped** — not flattened, not sent, not changed in any
way — and reported in `skippedInstances` with a reason instead of counting as a success
or a failure:
- **Disabled** instances. Deploy is a legal transition out of `Disabled` and re-enables
the instance, so a site-wide push would silently restart equipment an operator
deliberately stopped. Deploy them individually to re-enable.
- **Up-to-date** instances — those whose freshly flattened config still matches the
revision the site is running. A redeploy is destructive at the site (it resets static
attribute overrides and clears native alarm state), so pushing one to an unchanged
instance loses live state for no benefit. `--force` deploys them anyway; use it when
central's recorded state cannot be trusted, e.g. after a site's data volume was
restored or a node rebuilt. `--force` does **not** override the Disabled skip.
`--site-id` is **required** — unlike `deploy artifacts`, omitting it does not mean
fleet-wide.
```sh
scadabridge --url <url> deploy site --site-id <int>
scadabridge --url <url> deploy site --site-id <int> [--force]
```
| Option | Required | Description |
|--------|----------|-------------|
| `--site-id` | yes | Target site ID. A site-scoped (non-Administrator) Deployer must supply an in-scope site. |
| `--force` | no | Redeploy instances whose config is unchanged (default: skip them). Disabled instances are still skipped. |
Output is a per-instance result matrix (`instanceId`, `uniqueName`, `deploymentId`,
`success`, `errorMessage`) plus `successCount` / `failureCount`.
`success`, `errorMessage`) plus `successCount` / `failureCount`, and `skippedInstances`
(`instanceId`, `uniqueName`, `reason`).
#### `deploy artifacts`
src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Deployment/Deployments.razor
+107 -9
View File
@@ -17,7 +17,7 @@
aria-label="@(_autoRefresh ? "Pause auto-refresh" : "Resume auto-refresh")">
@(_autoRefresh ? "⏸ Pause updates" : "▶ Resume updates")
</button>
<button class="btn btn-outline-secondary" @onclick="LoadDataAsync" aria-label="Refresh deployments">Refresh</button>
<button class="btn btn-outline-secondary" @onclick="() => LoadDataAsync()" aria-label="Refresh deployments">Refresh</button>
</div>
</div>
@@ -259,6 +259,19 @@
private int _totalPages;
private const int PageSize = 25;
// ── Request-generation guard (P6-11) ────────────────────────────────────────
// Three independent initiators can each start a LoadDataAsync flight: a search
// debounce, a page click, and the push/coalesce reload. Every flight awaits its
// own round-trip to central MS SQL, so COMPLETION order is not CALL order — a
// slow, stale flight that started first can resolve after a faster, newer one
// and clobber the grid the operator is already looking at. _requestGeneration
// is bumped synchronously at the top of every LoadDataAsync call (call order is
// still reliable — increments never interleave on the renderer's dispatcher),
// and a flight discards its own results at apply time if a newer flight has
// since started. Last-write-wins by GENERATION, not by whichever await
// resolves first.
private long _requestGeneration;
// ── Filters ───────────────────────────────────────────────────────────────
private DeploymentStatus? _statusFilter;
@@ -423,10 +436,48 @@
if (_disposed) return;
}
// A changed search term redefines the result set, so the operator's current
// page number no longer refers to anything — restart at page 1.
_currentPage = 1;
_ = DispatchReloadAsync();
// This callback runs on the raw System.Threading.Timer thread — it is NOT
// the renderer's synchronization context. _currentPage is component state,
// so its reset to page 1 must be marshaled through InvokeAsync the same as
// the reload itself (P6-11); writing it here directly was a Blazor
// threading violation that could also race a same-tick page click on the
// renderer thread.
_ = DispatchFilterReloadAsync();
}
/// <summary>
/// Marshals the filter-debounce trailing edge onto the renderer's dispatcher.
/// The page-1 reset MUST happen inside <c>InvokeAsync</c>, and behind the SAME
/// request-generation guard as the rest of the applied state (P6-11) —
/// <see cref="OnFilterDebounceElapsed"/> runs on a raw
/// <see cref="System.Threading.Timer"/> callback thread, not the renderer's
/// synchronization context, and a slow search flight can resolve after the
/// operator has already navigated to a different page by other means. Passing
/// <c>resetToFirstPage: true</c> defers the actual <c>_currentPage = 1</c>
/// write into <see cref="LoadDataAsync"/>'s apply step, so a stale flight here
/// is discarded exactly like a stale flight on any other initiator.
/// </summary>
private async Task DispatchFilterReloadAsync()
{
if (_disposed) return;
try
{
await InvokeAsync(async () =>
{
if (_disposed) return;
// A changed search term redefines the result set, so the
// operator's current page number no longer refers to anything —
// restart at page 1 (applied only if this flight is still current;
// see the resetToFirstPage remarks on LoadDataAsync).
await LoadDataAsync(resetToFirstPage: true);
StateHasChanged();
});
}
catch (ObjectDisposedException)
{
// Component disposed between the guard and the dispatch — ignore.
}
}
/// <summary>
@@ -497,8 +548,27 @@
await LoadDataAsync();
}
private async Task LoadDataAsync()
/// <param name="resetToFirstPage">
/// True when this flight represents a redefinition of the result set (a new
/// search term) rather than a plain reload of whatever page is currently on
/// screen. The target page is resolved into a LOCAL <c>pageToQuery</c> rather
/// than writing <c>_currentPage</c> up front, so the page number — like every
/// other piece of applied state — only lands once this flight has been
/// confirmed to still be the newest one (see the generation guard below). A
/// stale, slow search flight that resolves after a newer page click must not
/// yank the operator back to page 1 out from under a navigation they already
/// completed.
/// </param>
private async Task LoadDataAsync(bool resetToFirstPage = false)
{
// Captured before any await — see the request-generation guard note on
// _requestGeneration above. Bumping it here (rather than at apply time)
// means it also marks THIS flight as the newest one in flight, so an
// earlier flight still awaiting its own round-trip will find itself
// stale when it reaches the check below.
var generation = ++_requestGeneration;
var pageToQuery = resetToFirstPage ? 1 : _currentPage;
_loading = _rows.Count == 0; // Only show loading on first load
_errorMessage = null;
try
@@ -520,16 +590,29 @@
SiteIdScope: siteScope);
var page = await DeploymentManagerRepository
.QueryDeploymentListPageAsync(filter, _currentPage, PageSize);
.QueryDeploymentListPageAsync(filter, pageToQuery, PageSize);
// The tile counts deliberately ignore the status filter (see
// GetDeploymentStatusCountsAsync) so each tile keeps showing its own
// total while it is the one selected.
_counts = await DeploymentManagerRepository.GetDeploymentStatusCountsAsync(filter);
var counts = await DeploymentManagerRepository.GetDeploymentStatusCountsAsync(filter);
// A newer flight has started since this one began (its generation is
// now the current one) — this flight's results are stale. Discard
// them rather than overwrite whatever the newer flight already
// applied (or is about to apply). This guards _currentPage exactly
// like every other applied field — a stale reset-to-page-1 flight
// must not clobber a page the operator has since navigated to.
if (generation != _requestGeneration)
{
return;
}
_currentPage = pageToQuery;
_rows = page.Rows;
_totalCount = page.TotalCount;
_totalPages = page.PageCount(PageSize);
_counts = counts;
// A concurrent retention purge, or a filter that narrowed the set, can
// leave the operator past the end. Fall back to the last real page and
@@ -539,6 +622,12 @@
_currentPage = _totalPages;
var reread = await DeploymentManagerRepository
.QueryDeploymentListPageAsync(filter, _currentPage, PageSize);
if (generation != _requestGeneration)
{
return;
}
_rows = reread.Rows;
_totalCount = reread.TotalCount;
_totalPages = reread.PageCount(PageSize);
@@ -546,9 +635,18 @@
}
catch (Exception ex)
{
if (generation != _requestGeneration)
{
return;
}
_errorMessage = $"Failed to load deployments: {ex.Message}";
}
_loading = false;
if (generation == _requestGeneration)
{
_loading = false;
}
}
private async Task GoToPageAsync(int page)
src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Deployment/Topology.razor
+76 -10
View File
@@ -160,6 +160,21 @@
private Timer? _liveUpdatesTimer;
private static readonly TimeSpan LiveUpdatesInterval = TimeSpan.FromSeconds(15);
/// <summary>
/// Guards <see cref="_liveUpdatesTimer"/> together with <see cref="_disposed"/>
/// (register row 103, the P6-6 lesser instance — mirrors <c>DebugView</c>'s
/// <c>_sessionLock</c>). <see cref="OnInitializedAsync"/> awaits
/// <see cref="LoadDataAsync"/> before calling <see cref="StartLiveUpdatesTimer"/>; the
/// circuit can be torn down inside that window, and <c>Dispose</c> may run on a
/// non-UI thread, so creating/replacing the timer and observing disposal must be one
/// atomic step — otherwise a timer created after <c>Dispose</c> already ran is never
/// disposed and ticks forever against a torn-down component.
/// </summary>
private readonly object _timerLock = new();
/// <summary>Set once <see cref="Dispose"/> has run; checked under <see cref="_timerLock"/>.</summary>
private volatile bool _disposed;
private TreeView<TopoNode> _tree = default!;
private object? _selectedKey;
private const string SelectedKeyStorage = "topology-tree-selected";
@@ -194,23 +209,51 @@
StartLiveUpdatesTimer();
}
/// <summary>
/// (Re)creates the live-update timer, or leaves none running when disposed / live
/// updates are off. Register row 103: creation and disposal of any timer this method
/// is replacing both happen under <see cref="_timerLock"/> together with the
/// <see cref="_disposed"/> check, so a call that resumes after <see cref="Dispose"/>
/// already ran refuses to create a new timer instead of leaking one.
/// </summary>
private void StartLiveUpdatesTimer()
{
_liveUpdatesTimer?.Dispose();
if (!_liveUpdates) return;
_liveUpdatesTimer = new Timer(_ =>
Timer? old;
lock (_timerLock)
{
InvokeAsync(async () =>
old = _liveUpdatesTimer;
_liveUpdatesTimer = null;
if (!_disposed && _liveUpdates)
{
_liveUpdatesTimer = new Timer(
OnLiveUpdatesTick, null, LiveUpdatesInterval, LiveUpdatesInterval);
}
}
old?.Dispose();
}
private void OnLiveUpdatesTick(object? state)
{
if (_disposed) return;
InvokeAsync(async () =>
{
if (_disposed || !_liveUpdates) return;
try
{
if (!_liveUpdates) return;
// Staleness is deliberately OFF the live loop: it only changes when a
// template/instance is edited or redeployed, and recomputing it (a
// full re-flatten per deployed instance) every 15s is the page's
// dominant cost. The poll refreshes only the cheap deployed state.
await LoadDeployedStateAsync();
if (_disposed) return;
StateHasChanged();
});
}, null, LiveUpdatesInterval, LiveUpdatesInterval);
}
catch (ObjectDisposedException)
{
// Component disposed between the guard and the dispatch — ignore
// (mirrors DebugView's SafeInvokeAsync / CentralUI-009).
}
});
}
private void OnLiveUpdatesToggled(ChangeEventArgs e)
@@ -222,14 +265,37 @@
}
else
{
_liveUpdatesTimer?.Dispose();
_liveUpdatesTimer = null;
StopLiveUpdatesTimer();
}
}
private void StopLiveUpdatesTimer()
{
Timer? timer;
lock (_timerLock)
{
timer = _liveUpdatesTimer;
_liveUpdatesTimer = null;
}
timer?.Dispose();
}
/// <summary>
/// Register row 103: the flag is set and the timer taken under the same lock
/// <see cref="StartLiveUpdatesTimer"/> creates through, so a still-pending
/// <see cref="OnInitializedAsync"/> either hands its new timer over here (never
/// created) or this method has nothing left to dispose — never both create AND leak.
/// </summary>
public void Dispose()
{
_liveUpdatesTimer?.Dispose();
Timer? timer;
lock (_timerLock)
{
_disposed = true;
timer = _liveUpdatesTimer;
_liveUpdatesTimer = null;
}
timer?.Dispose();
}
protected override async Task OnAfterRenderAsync(bool firstRender)
src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Design/ApiMethodForm.razor
+13 -1
View File
@@ -11,6 +11,7 @@
@inject IInboundApiKeyAdmin ApiKeyAdmin
@inject ScriptAnalysis.ScriptAnalysisService AnalysisService
@inject NavigationManager NavigationManager
@implements IDisposable
<div class="container-fluid mt-3">
<button class="btn btn-outline-secondary btn-sm mb-2" @onclick="GoBack">&larr; Back</button>
@@ -231,6 +232,7 @@
private Dictionary<string, object?> _paramValues = new();
private ScriptAnalysis.SandboxRunResult? _runResult;
private CancellationTokenSource? _runCts;
private bool _disposed;
protected override async Task OnInitializedAsync()
{
@@ -377,6 +379,15 @@
private void GoBack() => NavigationManager.NavigateTo("/design/external-systems");
/// <summary>Cancels an in-flight Test Run when the component is torn down.</summary>
public void Dispose()
{
_disposed = true;
_runCts?.Cancel();
_runCts?.Dispose();
_runCts = null;
}
private void ToggleTestRunPanel() => _showTestRun = !_showTestRun;
private async Task RunInSandboxAsync()
@@ -413,7 +424,8 @@
finally
{
_running = false;
StateHasChanged();
// The run may have been cancelled BY Dispose — the component is gone, don't re-render it.
if (!_disposed) StateHasChanged();
}
}
src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Design/SharedScriptForm.razor
+13 -1
View File
@@ -11,6 +11,7 @@
@inject ScriptAnalysis.ScriptAnalysisService AnalysisService
@inject AuthenticationStateProvider AuthStateProvider
@inject NavigationManager NavigationManager
@implements IDisposable
<div class="container-fluid mt-3">
<div class="d-flex align-items-center mb-3">
@@ -201,6 +202,7 @@
private Dictionary<string, object?> _paramValues = new();
private ScriptAnalysis.SandboxRunResult? _runResult;
private CancellationTokenSource? _runCts;
private bool _disposed;
// CentralUI-024: delegates to the shared helper so the claim type stays
// resolved through JwtTokenService rather than a duplicated magic string.
@@ -241,6 +243,15 @@
NavigationManager.NavigateTo("/design/shared-scripts");
}
/// <summary>Cancels an in-flight Test Run when the component is torn down.</summary>
public void Dispose()
{
_disposed = true;
_runCts?.Cancel();
_runCts?.Dispose();
_runCts = null;
}
private void CheckCompilation()
{
var syntaxError = ValidateSyntaxLocally(_formCode);
@@ -335,7 +346,8 @@
finally
{
_running = false;
StateHasChanged();
// The run may have been cancelled BY Dispose — the component is gone, don't re-render it.
if (!_disposed) StateHasChanged();
}
}
src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Design/TransportImport.razor.cs
+55 -4
View File
@@ -110,6 +110,20 @@ public partial class TransportImport : ComponentBase, IDisposable
private BundleSession? _session;
private bool _uploadInProgress;
/// <summary>
/// Guards <see cref="_session"/> together with <see cref="_disposed"/> (register row
/// 103, the P6-6 lesser instance — mirrors <c>DebugView</c>'s <c>_sessionLock</c>).
/// <see cref="TryLoadAsync"/> awaits <see cref="IBundleImporter.LoadAsync"/>, and the
/// circuit can be torn down inside that window; publishing the freshly-opened session
/// and observing disposal must be one atomic step so exactly one side (this method or
/// <see cref="Dispose"/>) releases it — otherwise the session is neither published nor
/// abandoned and strands one of <c>MaxConcurrentImportSessions</c> for the full TTL.
/// </summary>
private readonly object _sessionLock = new();
/// <summary>Set once <see cref="Dispose"/> has run; checked under <see cref="_sessionLock"/>.</summary>
private volatile bool _disposed;
// Staging directory for in-flight bundle uploads. Lives under the system
// temp directory rather than wwwroot/ because the file is never served to
// a browser — it is only read by the in-process IBundleImporter.
@@ -241,8 +255,25 @@ public partial class TransportImport : ComponentBase, IDisposable
using var stream = new FileStream(
_bundleTempPath, FileMode.Open, FileAccess.Read, FileShare.Read,
bufferSize: 81920, useAsync: true);
_session = await BundleImporter.LoadAsync(
var loaded = await BundleImporter.LoadAsync(
stream, passphrase, CancellationToken.None, ClientAddress.ClientAddress);
// Register row 103 (P6-6 lesser instance): the component may have been
// disposed while LoadAsync was in flight (navigation away mid-upload).
// Publish and re-check disposal atomically under _sessionLock so exactly
// one side tears the freshly-opened session down.
bool abandoned;
lock (_sessionLock)
{
abandoned = _disposed;
if (!abandoned) _session = loaded;
}
if (abandoned)
{
try { BundleImporter.Abandon(loaded.SessionId); } catch { /* best-effort */ }
return;
}
_errorMessage = null;
}
catch (ArgumentException)
@@ -921,8 +952,12 @@ public partial class TransportImport : ComponentBase, IDisposable
/// </summary>
private void ReleaseSession()
{
var session = _session;
_session = null;
BundleSession? session;
lock (_sessionLock)
{
session = _session;
_session = null;
}
if (session is null) return;
try
{
@@ -938,10 +973,26 @@ public partial class TransportImport : ComponentBase, IDisposable
/// Ensures neither the staged temp file nor the server-side bundle session
/// survives circuit teardown. Blazor invokes Dispose when the user navigates
/// away or the circuit ends, so an abandoned wizard cleans up automatically.
/// <para>
/// Register row 103: the flag is set and the session taken under the same lock
/// <see cref="TryLoadAsync"/> publishes through, so a <see cref="TryLoadAsync"/> still
/// awaiting <see cref="IBundleImporter.LoadAsync"/> either hands the session over here
/// or abandons it itself — never neither (mirrors <c>DebugView</c>'s P6-6 fix).
/// </para>
/// </summary>
public void Dispose()
{
ReleaseSession();
BundleSession? session;
lock (_sessionLock)
{
_disposed = true;
session = _session;
_session = null;
}
if (session is not null)
{
try { BundleImporter.Abandon(session.SessionId); } catch { /* best-effort */ }
}
DeleteBundleTempFile();
}
}
src/ZB.MOM.WW.ScadaBridge.CentralUI/ScriptAnalysis/ScriptAnalysisService.cs
+1 -1
View File
@@ -242,7 +242,7 @@ public class ScriptAnalysisService
/// <param name="request">The sandbox run request including code, kind, parameters, and optional instance binding.</param>
/// <param name="ct">Cancellation token for the caller; combined with an internal timeout CTS.</param>
/// <returns>A <see cref="SandboxRunResult"/> describing the outcome, return value, console output, and any error.</returns>
public async Task<SandboxRunResult> RunInSandboxAsync(SandboxRunRequest request, CancellationToken ct)
public virtual async Task<SandboxRunResult> RunInSandboxAsync(SandboxRunRequest request, CancellationToken ct)
{
if (string.IsNullOrWhiteSpace(request.Code))
{
src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/IDeploymentManagerRepository.cs
+22
View File
@@ -355,4 +355,26 @@ public interface IDeploymentManagerRepository
/// <param name="cancellationToken">A cancellation token that can be used to cancel the operation.</param>
/// <returns>A task representing the number of entities saved.</returns>
Task<int> SaveChangesAsync(CancellationToken cancellationToken = default);
/// <summary>
/// Abandons every pending (added / modified / deleted) change on the unit of
/// work, leaving the underlying store untouched, and returns how many entries
/// were dropped.
///
/// <para>
/// <b>Recovery only.</b> A <see cref="SaveChangesAsync"/> failure does NOT
/// untrack the entity that caused it: it stays in the change set and is
/// re-attempted by the NEXT save, so one unsavable entity poisons every
/// later save made through the same unit of work. That matters most on the
/// bulk site-deploy path, where all three phases share ONE scoped context
/// across every instance in the batch — a persistent failure on instance 1
/// would otherwise fail instance 2..N's status writes as well, including the
/// cleanup write that exists precisely to stop records being stranded
/// <c>InProgress</c>. Call this only after a save has already failed and its
/// pending work has been given up on; anything not yet committed is lost by
/// design.
/// </para>
/// </summary>
/// <returns>The number of tracked entries detached (0 when nothing was pending).</returns>
int DetachPendingChanges();
}
src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/INotificationOutboxRepository.cs
+38
View File
@@ -1,4 +1,5 @@
using ZB.MOM.WW.ScadaBridge.Commons.Entities.Notifications;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Notifications;
namespace ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
@@ -83,6 +84,43 @@ public interface INotificationOutboxRepository
/// </returns>
Task<bool> UpdateAsync(Notification n, CancellationToken cancellationToken = default);
/// <summary>
/// Compare-and-swap variant of <see cref="UpdateAsync"/> for the operator
/// Retry/Discard one-shots: persists <paramref name="n"/>'s delivery-state
/// columns (same narrow column set as <see cref="UpdateAsync"/>) ONLY if the
/// row's current <c>Status</c> in the database is still
/// <see cref="NotificationStatus.Parked"/>. Commits internally — this call
/// is its own transaction.
/// </summary>
/// <remarks>
/// <b>Why this exists (arch-review 2026-08-19 P5-4).</b> The operator retry/
/// discard handlers are read-check-write across two round trips:
/// <see cref="GetByIdAsync"/> reads the row, the handler checks its
/// in-memory <c>Status</c>, then a write persists the transition. A detached
/// <c>PipeTo</c> task can interleave with the dispatcher's own write (or a
/// second concurrent operator action) between those two round trips, so the
/// in-memory check is a fast-path rejection, not a correctness guard — a
/// blind <see cref="UpdateAsync"/> in that window can stamp a row that has
/// already moved on (e.g. the dispatcher delivered it) with the operator's
/// stale fields, silently overwriting a <c>Delivered</c> row as
/// <c>Discarded</c>. The <c>WHERE Status = Parked</c> predicate below makes
/// the <c>Parked-&gt;(Pending|Discarded)</c> transition atomic at the row
/// level, mirroring <c>SecuredWriteRepository.TryMarkApprovedAsync</c>'s
/// single-statement compare-and-swap. A row that has already progressed
/// (dispatcher won the race, or a second operator action won it) and a row
/// that has vanished (retention purge) both report zero rows affected;
/// callers cannot tell those apart from the return value alone and must
/// re-read to report which honestly.
/// </remarks>
/// <param name="n">The notification whose delivery state should be persisted, if still Parked.</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>
/// <see langword="true"/> when a row with that <c>NotificationId</c> was
/// found with <c>Status == Parked</c> and its delivery state was persisted;
/// <see langword="false"/> otherwise (not found, or no longer Parked).
/// </returns>
Task<bool> TryUpdateFromParkedAsync(Notification n, CancellationToken cancellationToken = default);
/// <summary>
/// Returns a page of notifications matching <paramref name="filter"/>, ordered by
/// <c>CreatedAt</c> descending, together with the total matching count.
src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Management/DeploymentCommands.cs
+10 -1
View File
@@ -21,7 +21,16 @@ public record MgmtDeployArtifactsCommand(int? SiteId = null);
/// </para>
/// </summary>
/// <param name="SiteId">Database id of the target site. Required — this command is never fleet-wide.</param>
public record MgmtDeploySiteCommand(int SiteId);
/// <param name="Force">
/// Deploy every deployable instance even when its flattened config already matches
/// the revision the site is running. Default <see langword="false"/>: an unchanged
/// instance is skipped and reported in the summary's <c>SkippedInstances</c>,
/// because a redeploy resets static attribute overrides and native alarm state at
/// the site. Set it when central's recorded state cannot be trusted — a site whose
/// data volume was restored or whose node was rebuilt looks up-to-date from central
/// and needs the push anyway. Never overrides the Disabled skip.
/// </param>
public record MgmtDeploySiteCommand(int SiteId, bool Force = false);
public record QueryDeploymentsCommand(int? InstanceId = null, string? Status = null, int Page = 1, int PageSize = 50);
/// <summary>
src/ZB.MOM.WW.ScadaBridge.Communication/Actors/SiteAlarmAggregatorActor.cs
+80 -2
View File
@@ -65,6 +65,15 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
private const string ReconcileTimerKey = "alarm-reconcile";
private const string PublishTimerKey = "alarm-publish-coalesce";
private const string SeedRetryTimerKey = "alarm-seed-retry";
private const string OpenDeadlineTimerKey = "alarm-grpc-open-deadline";
/// <summary>
/// Production default for the stream open-attempt deadline (arch-review 2026-08-19 P3-4).
/// Comfortably above <c>SiteStreamGrpcClient.ConnectedHeaderTimeout</c> (10 s) so a
/// normally-slow connect is never pre-empted, and comfortably below the 60 s reconcile
/// interval so the tick that follows always finds the flag released.
/// </summary>
internal static readonly TimeSpan DefaultOpenAttemptTimeout = TimeSpan.FromSeconds(30);
/// <summary>True while a coalesced publish is armed (dirty deltas awaiting one tick). Actor-thread only.</summary>
private bool _publishPending;
@@ -179,10 +188,38 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
/// <summary>
/// True between issuing a stream open and its first outcome (connected / error /
/// completed), so a reconcile tick never stacks a second open on an in-flight one.
/// Actor-thread only.
/// Bounded by <see cref="_openAttemptTimeout"/> — see there for why an unbounded flag
/// was a starvation hazard. Actor-thread only.
/// </summary>
private bool _openInFlight;
/// <summary>
/// Deadline on ONE stream open attempt, after which <see cref="_openInFlight"/> is
/// released so <see cref="OnReconcileTick"/> can reopen (arch-review 2026-08-19 P3-4).
/// <para>
/// The three outcomes that clear the flag — connected, error, graceful completion — are
/// all events the peer has to produce. A stream that produces NONE of them pins the flag:
/// the client gives up on response headers after
/// <c>SiteStreamGrpcClient.ConnectedHeaderTimeout</c> (10 s) and then falls back to the
/// first received event as its connected signal, so an alarm-only site that is quiet
/// supplies neither. In that state <c>_streamDown</c> is still true (nothing ever
/// reported connected) but <c>_openInFlight</c> is also true, and the tick's reopen guard
/// — <c>_streamDown &amp;&amp; !_openInFlight</c> — is skipped until the site's 4 h max
/// stream lifetime finally ends the call. The deadline makes the flag self-healing
/// instead of relying on that analysis staying true.
/// </para>
/// <para>
/// Firing it is deliberately CHEAP: it releases the flag and nothing else. The retry
/// budget is untouched (this is not a fault), the node is not flipped, liveness is not
/// republished (<c>_streamDown</c> was never cleared), and the abandoned subscription is
/// left running — if it was merely slow and connects late, <c>GrpcAlarmStreamConnected</c>
/// still matches the current generation and brings it up normally; if a reconcile tick
/// reopens first, <see cref="OpenGrpcStream"/> cancels its token source and fences its
/// generation out.
/// </para>
/// </summary>
private readonly TimeSpan _openAttemptTimeout;
/// <summary>Consecutive whole-fan-out failures, driving the seed-retry backoff. Actor-thread only.</summary>
private int _consecutiveSeedFailures;
@@ -246,6 +283,11 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
/// N per-site aggregators on one central node do not fan out in lockstep (production 0.2 =
/// up to +20%). Zero disables jitter, which is what tests want for determinism.
/// </param>
/// <param name="openAttemptTimeout">
/// Deadline on one gRPC stream open attempt (P3-4); <c>null</c> takes
/// <see cref="DefaultOpenAttemptTimeout"/> (30 s) and <see cref="TimeSpan.Zero"/> or less
/// disables the deadline. See <see cref="_openAttemptTimeout"/>.
/// </param>
public SiteAlarmAggregatorActor(
string siteIdentifier,
string correlationId,
@@ -258,7 +300,8 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
TimeSpan publishCoalesce,
TimeSpan reconnectDelay,
TimeSpan stabilityWindow,
double reconcileJitterFraction = 0.0)
double reconcileJitterFraction = 0.0,
TimeSpan? openAttemptTimeout = null)
{
_siteIdentifier = siteIdentifier;
_correlationId = correlationId;
@@ -272,6 +315,7 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
_reconnectDelay = reconnectDelay;
_stabilityWindow = stabilityWindow;
_reconcileJitterFraction = Math.Clamp(reconcileJitterFraction, 0.0, 1.0);
_openAttemptTimeout = openAttemptTimeout ?? DefaultOpenAttemptTimeout;
// Live delta from the site-wide alarm stream (marshalled in via Self.Tell).
// A received delta must NOT reset the retry budget (a flapping stream that
@@ -306,6 +350,23 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
OnStreamConnected();
});
// The open attempt produced none of its three outcomes inside the deadline (P3-4):
// release the in-flight flag so the reconcile tick's reopen guard can run again.
// Generation-fenced like every other stream callback, and a no-op once the attempt
// has already resolved (a late timer racing its own stream's connect).
Receive<GrpcAlarmStreamOpenTimedOut>(msg =>
{
if (_stopped) return;
if (msg.Generation != _streamGeneration) return;
if (!_openInFlight) return;
_openInFlight = false;
_log.Warning(
"Site-alarm gRPC stream open for {0} did not report connected, error or completion within {1}; " +
"releasing the in-flight guard so the next reconcile tick can reopen",
_siteIdentifier, _openAttemptTimeout);
});
// Coalesced-publish tick: one publish for a batch of dirtying deltas (N6).
Receive<PublishCoalesced>(_ =>
{
@@ -799,6 +860,15 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
_openInFlight = true;
var generation = ++_streamGeneration;
// Bound this attempt (P3-4). Single-shot on a shared key, so a re-open replaces the
// previous attempt's deadline rather than stacking one.
if (_openAttemptTimeout > TimeSpan.Zero)
{
Timers.StartSingleTimer(
OpenDeadlineTimerKey, new GrpcAlarmStreamOpenTimedOut(generation), _openAttemptTimeout);
}
var client = _grpcFactory.GetOrCreate(_siteIdentifier, endpoint);
var self = Self;
var ct = _grpcCts.Token;
@@ -834,6 +904,7 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
/// </summary>
private void OnStreamConnected()
{
Timers.Cancel(OpenDeadlineTimerKey);
_openInFlight = false;
var wasDown = _streamDown;
@@ -858,6 +929,7 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
// Stream failed before the stability window — retry budget NOT recovered.
Timers.Cancel(StabilityTimerKey);
Timers.Cancel(OpenDeadlineTimerKey);
_openInFlight = false;
var wasLive = !_streamDown;
@@ -920,6 +992,7 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
// The stream is gone, so its armed stability timer must not later "recover" a
// budget that its successor has since spent.
Timers.Cancel(StabilityTimerKey);
Timers.Cancel(OpenDeadlineTimerKey);
_log.Info("Site-alarm gRPC stream for {0} completed gracefully (server end of stream); " +
"reopening on the next reconcile tick", _siteIdentifier);
@@ -1005,3 +1078,8 @@ internal sealed record ReconnectAlarmStream;
/// <summary>Internal: the current site-alarm gRPC stream has stayed up long enough to recover the retry budget.</summary>
internal sealed record GrpcAlarmStreamStable;
/// <summary>Internal: a stream OPEN attempt produced none of connected/error/completed inside its
/// deadline (P3-4), stamped with the generation it was armed for so a late timer from a superseded
/// attempt is ignored. Releases the in-flight guard only — it is not a fault.</summary>
internal sealed record GrpcAlarmStreamOpenTimedOut(int Generation);
src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcServer.cs
+30 -10
View File
@@ -353,8 +353,35 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase
StatusCode.InvalidArgument, "correlation_id is missing or not a valid identifier"));
}
// Duplicate prevention -- cancel existing stream for this correlationId.
// Duplicate prevention -- REPLACE the existing stream for this correlationId.
//
// The removal is claimed FIRST, but nothing is cancelled until the concurrency cap
// has had its say, and a REPLACEMENT never consults the cap at all (arch-review
// 2026-08-19 P3-5). Both halves matter:
//
// * A replacement is net-neutral on the stream count (one entry out, one back in),
// so refusing it can only ever destroy capacity, never protect it. The old code
// checked `Count >= max` after the removal, which is correct at exactly the cap
// but NOT above it — and above it is reachable, because the check below is
// deliberately check-then-act (see the note there). In that window a routine
// reconnect cancelled a healthy incumbent and was then refused with
// ResourceExhausted, leaving the correlation with ZERO streams until its next
// retry. Excluding the replacement from the cap removes that outcome entirely.
// * A genuinely NEW subscriber that the cap refuses must leave the existing streams
// untouched, which is automatic here: TryRemove found nothing, so nothing was
// cancelled before the throw.
//
// Newest-wins is preserved exactly: a fresh subscribe on a live correlation id
// always displaces the incumbent, whether that incumbent is healthy or wedged.
var replacingExistingStream = _activeStreams.TryRemove(correlationId, out var existingEntry);
// Max concurrent streams, applied only to a genuinely new correlation id.
// Deliberately check-then-act: two concurrent subscribes at the boundary can
// both pass, overshooting the cap by at most (concurrent subscribers - 1).
// Bounded and benign — not worth a lock on this path.
if (!replacingExistingStream && _activeStreams.Count >= _maxConcurrentStreams)
throw new RpcException(new GrpcStatus(StatusCode.ResourceExhausted, "Max concurrent streams reached"));
// CANCEL ONLY, never Dispose. The replaced stream's CTS belongs to its own
// handler's `using var streamCts`, which is still running and still reads
// `streamCts.Token` (at the pump call below, and previously at the
@@ -365,11 +392,11 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase
// pre-existing hazard (the same Dispose + the same first-token-read relationship
// exist unchanged before R2). Cancellation alone is what replacement needs; the
// owning handler's `using` still disposes it exactly once on every exit path.
if (_activeStreams.TryRemove(correlationId, out var existingEntry))
if (replacingExistingStream)
{
try
{
existingEntry.Cts.Cancel();
existingEntry!.Cts.Cancel();
}
catch (ObjectDisposedException)
{
@@ -378,13 +405,6 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase
}
}
// Check max concurrent streams after duplicate removal.
// Deliberately check-then-act: two concurrent subscribes at the boundary can
// both pass, overshooting the cap by at most (concurrent subscribers - 1).
// Bounded and benign — not worth a lock on this path.
if (_activeStreams.Count >= _maxConcurrentStreams)
throw new RpcException(new GrpcStatus(StatusCode.ResourceExhausted, "Max concurrent streams reached"));
using var streamCts = CancellationTokenSource.CreateLinkedTokenSource(context.CancellationToken);
// Session timeout (design doc "gRPC Connection Keepalive": 4-hour third layer
// of dead-client detection) — forces a long-lived zombie stream to terminate
src/ZB.MOM.WW.ScadaBridge.Communication/SiteAlarmLiveCacheService.cs
+5 -1
View File
@@ -245,7 +245,11 @@ public sealed class SiteAlarmLiveCacheService : ISiteAlarmLiveCache
_options.LiveAlarmCachePublishCoalesce,
TimeSpan.FromSeconds(5), // reconnect delay — former ReconnectDelay static default
TimeSpan.FromSeconds(60), // stability window — former StabilityWindow static default
_options.LiveAlarmCacheReconcileJitterFraction));
_options.LiveAlarmCacheReconcileJitterFraction,
// Open-attempt deadline (P3-4). Passed explicitly rather than defaulted:
// Props.Create takes an expression tree, which may not omit an optional
// argument (CS0854).
SiteAlarmAggregatorActor.DefaultOpenAttemptTimeout));
entry.Actor = system.ActorOf(props, $"site-alarm-aggregator-{entry.SiteId}-{Guid.NewGuid():N}");
src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Configurations/NotificationOutboxConfiguration.cs
+18
View File
@@ -83,5 +83,23 @@ public class NotificationOutboxConfiguration : IEntityTypeConfiguration<Notifica
builder.HasIndex(n => n.DeliveredAt)
.HasDatabaseName("IX_Notifications_Delivered")
.HasFilter("[Status] = 'Delivered'");
// Created_NotificationId backs the Central UI Notification Outbox list
// page's sort key — NotificationOutboxRepository.QueryAsync orders
// (CreatedAt DESC, NotificationId DESC) (NotificationId breaks CreatedAt
// ties for deterministic OFFSET paging). None of the indexes above lead
// unconditionally on CreatedAt: Status_NextAttemptAt requires an equality
// on Status first, SourceSiteId_CreatedAt requires an equality on
// SourceSiteId first, and IX_Notifications_Delivered is filtered to
// Status='Delivered' only. So an unfiltered (or lightly filtered) page
// fell back to a full clustered-index scan + sort on every COUNT + page
// request (arch-review 2026-08-19 P5-3, register row-35 class). With both
// key columns descending, a forward scan of this index already delivers
// rows in the requested order. Exact mirror of
// SiteCallEntityTypeConfiguration's IX_SiteCalls_Created_TrackedOperationId
// (2026-08-19, arch-review P5-2).
builder.HasIndex(n => new { n.CreatedAt, n.NotificationId })
.IsDescending(true, true)
.HasDatabaseName("IX_Notifications_Created_NotificationId");
}
}
src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/20260819201210_AddNotificationsListSortIndex.Designer.cs
Generated
+2131
View File
File diff suppressed because it is too large Load Diff
src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/20260819201210_AddNotificationsListSortIndex.cs
+46
View File
@@ -0,0 +1,46 @@
using Microsoft.EntityFrameworkCore.Migrations;
#nullable disable
namespace ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Migrations
{
/// <summary>
/// Adds <c>IX_Notifications_Created_NotificationId (CreatedAt DESC,
/// NotificationId DESC)</c> — arch-review 2026-08-19 P5-3, register
/// row-35 class.
/// </summary>
/// <remarks>
/// <see cref="ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Repositories.NotificationOutboxRepository.QueryAsync"/>
/// orders <c>(CreatedAt DESC, NotificationId DESC)</c> for the Central UI
/// Notification Outbox list page. None of the existing Notifications indexes
/// lead unconditionally on <c>CreatedAt</c> — <c>IX_Notifications_Status_NextAttemptAt</c>
/// requires an equality on <c>Status</c> first, <c>IX_Notifications_SourceSiteId_CreatedAt</c>
/// requires an equality on <c>SourceSiteId</c> first, and
/// <c>IX_Notifications_Delivered</c> is filtered to <c>Status = 'Delivered'</c>
/// only — so an unfiltered (or lightly filtered) list page fell back to a full
/// clustered-index scan + sort on every COUNT + page request. With both key
/// columns descending, a forward scan of this index already delivers rows in
/// the requested order. See <c>docs/plans/sql/AddNotificationsListSortIndex.sql</c>
/// for the handwritten production-parity script.
/// </remarks>
public partial class AddNotificationsListSortIndex : Migration
{
/// <inheritdoc />
protected override void Up(MigrationBuilder migrationBuilder)
{
migrationBuilder.CreateIndex(
name: "IX_Notifications_Created_NotificationId",
table: "Notifications",
columns: new[] { "CreatedAt", "NotificationId" },
descending: new bool[0]);
}
/// <inheritdoc />
protected override void Down(MigrationBuilder migrationBuilder)
{
migrationBuilder.DropIndex(
name: "IX_Notifications_Created_NotificationId",
table: "Notifications");
}
}
}
src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/ScadaBridgeDbContextModelSnapshot.cs
+4
View File
@@ -913,6 +913,10 @@ namespace ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Migrations
.HasDatabaseName("IX_Notifications_Delivered")
.HasFilter("[Status] = 'Delivered'");
b.HasIndex("CreatedAt", "NotificationId")
.IsDescending()
.HasDatabaseName("IX_Notifications_Created_NotificationId");
b.HasIndex("SourceSiteId", "CreatedAt");
b.HasIndex("Status", "NextAttemptAt");
src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/DeploymentManagerRepository.cs
+18
View File
@@ -551,6 +551,24 @@ public class DeploymentManagerRepository : IDeploymentManagerRepository
return await _dbContext.SaveChangesAsync(cancellationToken);
}
/// <inheritdoc />
public int DetachPendingChanges()
{
// ChangeTracker.Entries() is enumerated into a list FIRST: setting State
// mutates the tracker, and mutating it while enumerating it throws.
// Unchanged/Detached entries are left alone — they carry no pending write,
// so keeping them tracked preserves identity resolution for anything the
// caller still holds a reference to.
var pending = _dbContext.ChangeTracker.Entries()
.Where(e => e.State is EntityState.Added or EntityState.Modified or EntityState.Deleted)
.ToList();
foreach (var entry in pending)
entry.State = EntityState.Detached;
return pending.Count;
}
/// <summary>
/// Flat shape of the deployment-record ⋈ instance join used to compose the list
/// query. A named type rather than an anonymous one so
src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/KpiHistoryRepository.cs
+13
View File
@@ -166,6 +166,19 @@ public sealed class KpiHistoryRepository : IKpiHistoryRepository
// Per-metric aggregation intent: Rate metrics sum the hour (last-value would
// discard 59 of 60 intra-hour deltas); Gauge metrics keep the last (latest
// timestamp) value in the hour. Min/Max/Count preserve the fold's fidelity.
//
// This Sum is only correct if every sample in the group is a non-overlapping
// delta covering exactly one recorder tick (KpiHistoryOptions.SampleInterval).
// That invariant used to be an unenforced assumption across three
// independently-owned config keys (KpiHistoryOptions.SampleInterval,
// NotificationOutboxOptions.DeliveredKpiWindow, SiteCallAuditOptions.KpiInterval)
// — widening one silently over/under-counted the rollup (arch-review P5-5, Low).
// As of P5-5, NotificationOutboxKpiSampleSource / SiteCallAuditKpiSampleSource
// rescale their Rate-classified samples onto SampleInterval before recording, so
// this Sum is correct here regardless of how the other two keys are tuned; a
// boot-time cross-validator (Host's KpiRateFoldIntervalCrossValidator) still
// enforces all three being equal so the recorded/charted samples never silently
// diverge from what the live dashboard tile shows at the same instant.
var aggregation = KpiMetricAggregationCatalog.Resolve(key.Source, key.Metric);
var value = aggregation == KpiRollupAggregation.Rate
? group.Sum(s => s.Value)
src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/NotificationOutboxRepository.cs
+42
View File
@@ -205,6 +205,48 @@ VALUES
return rowsAffected > 0;
}
/// <inheritdoc />
public async Task<bool> TryUpdateFromParkedAsync(Notification n, CancellationToken cancellationToken = default)
{
ArgumentNullException.ThrowIfNull(n);
// Same seven mutable delivery-state columns as UpdateAsync, but the WHERE
// clause below ALSO requires the row's current Status to still be Parked
// -- a single-statement compare-and-swap (arch-review 2026-08-19 P5-4).
// Two round trips (GetByIdAsync then this write) with a status check in
// between is not a correctness guard by itself: a detached PipeTo task
// can race the dispatcher's own write (or a second concurrent operator
// action) in that window. Mirrors SecuredWriteRepository.TryMarkApprovedAsync's
// conditional ExecuteUpdate/ExecuteSql pattern, expressed via LINQ
// ExecuteUpdateAsync (like UpdateAsync above) so this stays portable
// across the SQL Server production provider and the SQLite provider the
// repository tests run against.
var status = n.Status;
var retryCount = n.RetryCount;
var lastError = n.LastError;
var resolvedTargets = n.ResolvedTargets;
var lastAttemptAt = n.LastAttemptAt;
var nextAttemptAt = n.NextAttemptAt;
var deliveredAt = n.DeliveredAt;
var notificationId = n.NotificationId;
var rowsAffected = await _context.Notifications
.Where(row => row.NotificationId == notificationId && row.Status == NotificationStatus.Parked)
.ExecuteUpdateAsync(
setters => setters
.SetProperty(row => row.Status, status)
.SetProperty(row => row.RetryCount, retryCount)
.SetProperty(row => row.LastError, lastError)
.SetProperty(row => row.ResolvedTargets, resolvedTargets)
.SetProperty(row => row.LastAttemptAt, lastAttemptAt)
.SetProperty(row => row.NextAttemptAt, nextAttemptAt)
.SetProperty(row => row.DeliveredAt, deliveredAt),
cancellationToken);
return rowsAffected > 0;
}
/// <inheritdoc />
public async Task<Notification?> GetByIdAsync(string notificationId, CancellationToken cancellationToken = default)
{
src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Actors/DataConnectionActor.cs
+264 -11
View File
@@ -234,6 +234,51 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers
/// </summary>
private IActorRef _self = null!;
/// <summary>
/// The delegate currently hooked to <c>_adapter.Disconnected</c>. It closes over the
/// <see cref="_adapterGeneration"/> in force when it was hooked, so the resulting
/// <see cref="AdapterDisconnected"/> carries the generation of the adapter that raised it —
/// a method group could not. Kept in a field so <see cref="SwapAdapter"/> and
/// <see cref="PostStop"/> unhook exactly the instance they hooked.
/// </summary>
private Action? _disconnectedHandler;
/// <summary>
/// Arch-review 2026-08-19 <b>P2-6</b>. Set when an <see cref="AdapterDisconnected"/> for the
/// CURRENT adapter generation is received while the actor is still Connecting/Reconnecting —
/// i.e. the adapter connected and dropped again before the piped
/// <see cref="ConnectResult"/> for that attempt reached the mailbox.
///
/// <para>
/// Neither state used to handle the message at all, so it fell through to
/// <c>Unhandled</c> and the subsequent <c>ConnectResult(true)</c> declared the connection up
/// on an adapter that was already dead. Nothing recovers from there: an adapter fires
/// <c>Disconnected</c> at most once per successful connect (<c>OpcUaDataConnection</c>'s
/// once-only <c>_disconnectFired</c> guard, reset only by the next successful
/// <c>ConnectAsync</c>), so a connection with zero subscriptions — nothing to fail at
/// connection level and drive the reconnect machine — sat Connected on a dead adapter until
/// the process restarted.
/// </para>
///
/// <para>
/// It is cleared at the top of <see cref="HandleAttemptConnect"/>, so it only ever refers to
/// the attempt currently in flight: a disconnect belonging to a session that ended BEFORE
/// this attempt began (e.g. the adapter's <c>Disconnected</c> trailing a connection-level
/// subscribe failure that already drove the actor into Reconnecting) is discarded rather than
/// replayed against the fresh session. That is also why the flag is used instead of
/// <c>Stash</c>: the stash has no selective drop, so a stashed disconnect would be replayed
/// after the next successful connect and cost a spurious reconnect cycle.
/// </para>
///
/// <para>
/// Residual (accepted, and in the safe direction): a disconnect raised by the PREVIOUS
/// session but delivered after the next attempt has already cleared the flag is attributed to
/// the new attempt, costing one extra reconnect cycle. Being wedged Connected on a dead
/// adapter is the failure this trades against.
/// </para>
/// </summary>
private bool _disconnectDuringConnectAttempt;
/// <summary>
/// Initializes the data connection actor with its adapter and configuration.
/// </summary>
@@ -283,23 +328,43 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers
_self = Self;
// Listen for unexpected adapter disconnections
_adapter.Disconnected += OnAdapterDisconnected;
HookAdapterDisconnected();
BecomeConnecting();
}
private void OnAdapterDisconnected()
/// <summary>
/// Subscribes to the CURRENT adapter's <c>Disconnected</c> event with a delegate that closes
/// over the generation in force right now, so the marshalled
/// <see cref="AdapterDisconnected"/> identifies the adapter that raised it. Callers must
/// have bumped <see cref="_adapterGeneration"/> for the new adapter first
/// (see <see cref="SwapAdapter"/>).
/// </summary>
private void HookAdapterDisconnected()
{
// Marshal the event onto the actor's message loop using captured _self reference.
// This runs on a background thread (gRPC stream reader), so Self would throw.
_self.Tell(new AdapterDisconnected());
var generation = _adapterGeneration;
var self = _self;
// Marshal the event onto the actor's message loop using the captured _self reference.
// This runs on a background thread (keep-alive / gRPC stream reader), so Self would throw.
_disconnectedHandler = () => self.Tell(new AdapterDisconnected(generation));
_adapter.Disconnected += _disconnectedHandler;
}
/// <summary>Unhooks the delegate installed by <see cref="HookAdapterDisconnected"/>, if any.</summary>
private void UnhookAdapterDisconnected()
{
if (_disconnectedHandler is null)
return;
_adapter.Disconnected -= _disconnectedHandler;
_disconnectedHandler = null;
}
/// <inheritdoc />
protected override void PostStop()
{
_log.Info("DataConnectionActor [{0}] stopping — disposing adapter", _connectionName);
_adapter.Disconnected -= OnAdapterDisconnected;
UnhookAdapterDisconnected();
// S9: fire-and-forget, but observe a faulted dispose so a failing adapter
// teardown is logged rather than swallowed into an unobserved task exception.
_adapter.DisposeAsync().AsTask().ContinueWith(
@@ -340,6 +405,14 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers
case ConnectResult result:
HandleConnectResult(result);
break;
case AdapterDisconnected disconnected:
// P2-6 — the adapter connected and dropped again before this attempt's
// ConnectResult was applied. Remember it (see _disconnectDuringConnectAttempt)
// rather than acting now: acting would put the actor into Reconnecting and the
// in-flight ConnectResult(true) would then be applied THERE, re-subscribing and
// declaring the same dead adapter Connected anyway.
NotePendingDisconnect(disconnected);
break;
case SubscribeTagsRequest:
case WriteTagRequest:
case WriteTagBatchRequest:
@@ -467,8 +540,13 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers
case QualityFlushTick:
FlushQualityCountersIfDirty();
break;
case AdapterDisconnected:
HandleDisconnect();
case AdapterDisconnected disconnected:
// P2-6 — fence the disconnect on its adapter generation. The outgoing adapter's
// event can fire concurrently with SwapAdapter's unhook, so a Disconnected from
// the endpoint we have just failed over (or reconfigured) AWAY from could arrive
// here and knock the healthy replacement back into a reconnect cycle.
if (!IsStaleDisconnect(disconnected))
HandleDisconnect();
break;
case ReconfigureConnectionCommand rc:
HandleReconfigure(rc);
@@ -600,14 +678,17 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers
/// </summary>
private void SwapAdapter(IDictionary<string, string> newConfig)
{
_adapter.Disconnected -= OnAdapterDisconnected;
UnhookAdapterDisconnected();
_ = _adapter.DisposeAsync().AsTask();
_adapter = _factory.Create(_protocolType, newConfig);
_connectionDetails = newConfig;
_adapter.Disconnected += OnAdapterDisconnected;
_adapterGeneration++;
// Hooked AFTER the bump so the new adapter's handler stamps the NEW generation; the
// outgoing adapter's handler was captured with the old one and is now recognised as
// stale by IsStaleDisconnect even if it fires concurrently with the unhook above.
HookAdapterDisconnected();
}
/// <summary>
@@ -702,6 +783,12 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers
case ConnectResult result:
HandleReconnectResult(result);
break;
case AdapterDisconnected disconnected:
// P2-6 — same gap as Connecting: a reconnect that succeeds and drops again before
// its ConnectResult is applied would otherwise be declared Connected on a dead
// adapter that will never raise Disconnected a second time.
NotePendingDisconnect(disconnected);
break;
case SubscribeTagsRequest:
case WriteTagRequest:
case WriteTagBatchRequest:
@@ -777,6 +864,10 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers
private void HandleAttemptConnect()
{
_log.Debug("[{0}] Attempting connection...", _connectionName);
// P2-6 — scope the pending-disconnect flag to THIS attempt. Anything noted before now
// belongs to a session that is already over and must not be replayed against the one
// about to start.
_disconnectDuringConnectAttempt = false;
var self = Self;
// Stamp the generation this attempt belongs to, exactly as every other piped
// completion does. A failover or an in-place reconfigure landing while ConnectAsync
@@ -827,6 +918,75 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers
return true;
}
/// <summary>
/// True when this <see cref="AdapterDisconnected"/> was raised by an adapter that has since
/// been swapped out (failover or in-place reconfigure), in which case it is dropped — the
/// disconnect twin of <see cref="IsStaleConnectResult"/> (arch-review 2026-08-19 P2-6).
/// <see cref="SwapAdapter"/> unhooks the outgoing adapter, but the event can already be
/// executing on a background thread when it does, so the message can still be delivered
/// after the swap.
/// </summary>
private bool IsStaleDisconnect(AdapterDisconnected disconnected)
{
if (disconnected.AdapterGeneration == _adapterGeneration)
return false;
_log.Debug(
"[{0}] Dropping disconnect notification from adapter generation {1} (current {2}) — "
+ "the adapter that raised it has been replaced",
_connectionName, disconnected.AdapterGeneration, _adapterGeneration);
return true;
}
/// <summary>
/// Records a current-generation disconnect that arrived while a connect attempt was still in
/// flight, so <see cref="HandleConnectResult"/> / <see cref="HandleReconnectResult"/> can
/// refuse to declare the connection up. See <see cref="_disconnectDuringConnectAttempt"/>.
/// </summary>
private void NotePendingDisconnect(AdapterDisconnected disconnected)
{
if (IsStaleDisconnect(disconnected))
return;
_log.Warning(
"[{0}] Adapter reported a disconnect while a connect attempt was still in flight — "
+ "the attempt's result will be treated as a failed connection", _connectionName);
_disconnectDuringConnectAttempt = true;
}
/// <summary>
/// Reads and clears <see cref="_disconnectDuringConnectAttempt"/>, logging the demotion. True
/// means the successful connect result being applied belongs to a session the adapter has
/// already reported as dropped.
/// </summary>
private bool ConsumeDisconnectDuringConnectAttempt()
{
if (!_disconnectDuringConnectAttempt)
return false;
_disconnectDuringConnectAttempt = false;
_log.Warning(
"[{0}] Connect succeeded but the adapter had already reported the session lost — "
+ "treating the attempt as a failed connect", _connectionName);
return true;
}
/// <summary>
/// Applies a demoted connect success (see <see cref="ConsumeDisconnectDuringConnectAttempt"/>)
/// exactly as a connect FAILURE: the session the "success" describes is already gone, so it
/// must count toward the failover budget and re-arm the retry timer rather than declare the
/// connection up. Verbatim the tail of both result handlers' failure branches, which is the
/// point — a flapping endpoint that drops inside every connect window still reaches its
/// backup instead of retrying the same dead primary forever.
/// </summary>
private void DemoteConnectAttemptToFailure()
{
_consecutiveFailures++;
CountFailureAndMaybeFailover(
"adapter reported the session lost before the connect result was applied");
Timers.StartSingleTimer("reconnect", new AttemptConnect(), _options.ReconnectInterval);
}
private void HandleConnectResult(ConnectResult result)
{
if (IsStaleConnectResult(result))
@@ -834,6 +994,16 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers
if (result.Success)
{
// P2-6 — a disconnect that raced ahead of this result means the session this
// "success" describes is already gone; going Connected here would strand the
// connection on a dead adapter that never raises Disconnected again. Stay in
// Connecting and retry, exactly as a failed connect would.
if (ConsumeDisconnectDuringConnectAttempt())
{
DemoteConnectAttemptToFailure();
return;
}
_log.Info("[{0}] Connection established", _connectionName);
BecomeConnected();
}
@@ -858,6 +1028,15 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers
if (result.Success)
{
// P2-6 — same demotion as the initial connect: the reconnect landed but the adapter
// has already reported the new session lost, so re-subscribing and going Connected
// would wedge on a dead adapter. Stay in Reconnecting and retry.
if (ConsumeDisconnectDuringConnectAttempt())
{
DemoteConnectAttemptToFailure();
return;
}
_log.Info("[{0}] Reconnected successfully on {1} endpoint", _connectionName, _activeEndpoint);
_consecutiveFailures = 0;
@@ -2131,10 +2310,79 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers
}));
}
// NOTE: `self` is the Self reference captured on the actor thread above — this runs on
// a background task, where Akka's Self property is invalid.
SeedNewlyResolvedTags(self, adapter, generation, rows);
return new BatchSubscribeCompleted(rows, generation, BatchSubscribeSource.ResolutionProbe);
}).PipeTo(self);
}
/// <summary>
/// Reads the current value of every tag this probe round has just resolved and delivers it
/// through the normal generation-guarded <see cref="TagValueReceived"/> path — the same seed
/// the initial subscribe (<see cref="HandleSubscribe"/>) and the reconnect re-subscribe
/// (<see cref="ReSubscribeAll"/>) already perform, which the resolution probe was missing
/// (arch-review 2026-08-19 <b>P2-5</b>).
///
/// <para>
/// A newly advised subscription only starts producing values on the source's next CHANGE.
/// OPC UA masks that with an initial data-change notification on subscription creation, but
/// an MxGateway advise can legitimately return <c>VT_EMPTY</c>, so a tag that resolved LATE
/// (device still booting, item created after deployment) and then never changes had no value
/// path at all — its attribute stayed Uncertain for the life of the connection, healed only
/// by an unrelated reconnect.
/// </para>
///
/// <para>
/// Unlike <see cref="HandleSubscribe"/>, the seeds are delivered directly rather than carried
/// back on the completion message: the probe only ever covers tags already present in
/// <see cref="_subscriptionsByInstance"/> / <see cref="_instancesByTag"/>, so the
/// registration-ordering hazard that forces the initial subscribe to defer does not exist
/// here. It runs as its own fire-and-forget task (mirroring the reconnect re-seed) so the
/// seed reads — bounded by <c>SeedOverallTimeout</c>, not by the subscribe — never delay the
/// <see cref="BatchSubscribeCompleted"/> that carries the round's resolution bookkeeping and
/// re-arms the retry timer. Values that arrive after a failover or a drop to Reconnecting are
/// discarded by the generation guard / state handler exactly like any other tag value.
/// </para>
/// </summary>
/// <param name="self">
/// The actor's own reference, captured on the actor thread by the caller — this method is
/// invoked from a background task, where Akka's <c>Self</c> property is invalid.
/// </param>
/// <param name="adapter">The adapter the probe round ran against.</param>
/// <param name="generation">The adapter generation the probe round was stamped with.</param>
/// <param name="rows">The probe round's per-tag subscribe results.</param>
private void SeedNewlyResolvedTags(
IActorRef self, IDataConnection adapter, int generation, IReadOnlyList<SubscribeTagResult> rows)
{
var resolved = new List<string>(rows.Count);
foreach (var row in rows)
{
if (row is { Success: true, SubscriptionId: not null })
resolved.Add(row.TagPath);
}
if (resolved.Count == 0)
return;
_ = Task.Run(async () =>
{
try
{
var seeds = await SeedTagsAsync(adapter, resolved);
foreach (var seed in seeds)
self.Tell(new TagValueReceived(seed.TagPath, seed.Value, generation));
}
catch (Exception ex)
{
// Fire-and-forget: guarantee a trace rather than an unobserved task fault.
// SeedTagsAsync already swallows per-read errors, so reaching here is unexpected.
_log.Warning("[{0}] Tag-resolution re-seed task faulted: {1}", _connectionName, ex.Message);
}
});
}
/// <summary>
/// Arms the single-shot tag-resolution retry timer at the current backoff interval
/// when unresolved tags remain, and cancels it when none do. Single-shot (rescheduled
@@ -2961,7 +3209,12 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers
/// rather than applied to its replacement (see <see cref="IsStaleConnectResult"/>).
/// </summary>
internal record ConnectResult(bool Success, string? Error, int AdapterGeneration);
internal record AdapterDisconnected;
/// <summary>
/// The adapter's <c>Disconnected</c> event marshalled onto the actor thread, stamped with the
/// <see cref="_adapterGeneration"/> the raising adapter was hooked at so a notification from a
/// swapped-out adapter can be dropped (<see cref="IsStaleDisconnect"/>).
/// </summary>
internal record AdapterDisconnected(int AdapterGeneration);
internal record TagValueReceived(string TagPath, TagValue Value, int AdapterGeneration);
internal record RetryTagResolution;
src/ZB.MOM.WW.ScadaBridge.DeploymentManager/DeploymentService.cs
+220 -22
View File
@@ -5,6 +5,7 @@ using ZB.MOM.WW.ScadaBridge.Commons.Entities.Deployment;
using ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Transport;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Deployment;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Lifecycle;
using ZB.MOM.WW.ScadaBridge.Commons.Observability;
@@ -45,6 +46,7 @@ public class DeploymentService
private readonly DiffService _diffService;
private readonly RevisionHashService _revisionHashService;
private readonly IDeploymentStatusNotifier _statusNotifier;
private readonly IStaleInstanceProbe _staleInstanceProbe;
private readonly DeploymentManagerOptions _options;
private readonly CommunicationOptions _commOptions;
private readonly ILogger<DeploymentService> _logger;
@@ -64,6 +66,15 @@ public class DeploymentService
internal const string DisabledSkipReason =
"Disabled — bulk deploy preserves disabled state; deploy individually to re-enable";
/// <summary>
/// Reason reported on every <see cref="SkippedInstanceResult"/> a bulk site
/// deploy produces for an instance whose flattened config already matches the
/// revision the site is running. Suppressed by the <c>force</c> flag.
/// </summary>
internal const string UpToDateSkipReason =
"up-to-date — the flattened config still matches the deployed revision; " +
"redeploy with force to push it anyway";
/// <summary>Rows deleted per terminal-record purge batch.</summary>
private const int TerminalPurgeBatchSize = 500;
@@ -98,6 +109,11 @@ public class DeploymentService
/// staleness hash from the (List-normalized) deserialized snapshot — see I-1.
/// </param>
/// <param name="statusNotifier">Notifier for pushing deployment status changes to the UI.</param>
/// <param name="staleInstanceProbe">
/// Current-revision-hash probe backing the bulk deploy's up-to-date pre-filter
/// (see <see cref="DeploySiteAsync"/>). The same seam the bundle importer and
/// the Deployments page use, so all three agree on what "stale" means.
/// </param>
/// <param name="options">Deployment manager configuration options.</param>
/// <param name="communicationOptions">
/// Central-site communication options. The notify-and-fetch deploy path reads
@@ -117,6 +133,7 @@ public class DeploymentService
DiffService diffService,
RevisionHashService revisionHashService,
IDeploymentStatusNotifier statusNotifier,
IStaleInstanceProbe staleInstanceProbe,
IOptions<DeploymentManagerOptions> options,
IOptions<CommunicationOptions> communicationOptions,
ILogger<DeploymentService> logger)
@@ -130,6 +147,7 @@ public class DeploymentService
_diffService = diffService;
_revisionHashService = revisionHashService;
_statusNotifier = statusNotifier;
_staleInstanceProbe = staleInstanceProbe;
_options = options.Value;
_commOptions = communicationOptions.Value;
_logger = logger;
@@ -681,6 +699,21 @@ public class DeploymentService
/// A fault DURING cleanup is logged loudly and swallowed — it must not mask
/// the original error the caller is about to report.
/// </para>
///
/// <para>
/// <b>The cleanup save is un-poisoned before it is given up on (P7-5).</b> A
/// failed <c>SaveChanges</c> does not untrack the entity that caused it: EF
/// keeps it in the change set and re-attempts it on the NEXT save. Every phase
/// of a bulk site deploy shares ONE scoped <c>DbContext</c> across every
/// instance in the batch, so a persistent failure on one instance would
/// otherwise fail instance 2..N's status writes too — including their own
/// cleanup writes, which is exactly the write that stops a record being
/// stranded <c>InProgress</c>. On a cleanup-save failure the pending changes
/// are therefore DETACHED (<see cref="IDeploymentManagerRepository.DetachPendingChanges"/>)
/// and the Failed write is retried ONCE on the now-clean tracker: if the poison
/// came from an earlier instance this record still reaches its terminal status,
/// and either way the next instance in the batch saves cleanly.
/// </para>
/// </summary>
private async Task MarkDeploymentFailedAsync(
DeploymentRecord record,
@@ -697,25 +730,57 @@ public class DeploymentService
try
{
await _repository.UpdateDeploymentRecordAsync(record, CancellationToken.None);
// Note: if the staging SaveChangesAsync was interrupted, an Added
// PendingDeployment may still be tracked and will be committed by this
// cleanup save. That row is orphaned (no RefreshDeploymentCommand was
// sent, so no site holds its token) and is removed by TTL purge /
// superseded by the next deploy -- harmless.
await _repository.SaveChangesAsync(CancellationToken.None);
NotifyStatusChange(record);
await _auditService.LogAsync(user, "DeployFailed", "Instance", instance.Id.ToString(),
instance.UniqueName, new { DeploymentId = deploymentId, Error = cause.Message },
CancellationToken.None);
await PersistFailedStatusAsync();
}
catch (Exception cleanupEx)
{
// Give up on whatever is pending — it has already failed to save at
// least once — so the shared change tracker stops re-attempting it.
var detached = _repository.DetachPendingChanges();
_logger.LogError(cleanupEx,
"Failed to persist Failed status for deployment {DeploymentId} of instance {Instance} " +
"after deployment error: {Error}",
deploymentId, instance.UniqueName, cause.Message);
"after deployment error: {Error}. Detached {DetachedCount} pending change(s) so the " +
"shared unit of work does not carry the failure into later writes",
deploymentId, instance.UniqueName, cause.Message, detached);
if (detached > 0)
{
try
{
await PersistFailedStatusAsync();
}
catch (Exception retryEx)
{
_logger.LogError(retryEx,
"Retrying the Failed status write for deployment {DeploymentId} of instance " +
"{Instance} on a cleaned unit of work also failed; the record remains " +
"InProgress and must be reconciled (the next deploy of this instance heals it)",
deploymentId, instance.UniqueName);
}
}
}
// Best-effort and deliberately OUTSIDE the status-write try: an audit row
// for a failed deployment is worth writing even when the status write
// itself could not be persisted, and an audit failure must never mask the
// deployment error the caller is about to report.
await TryLogAuditAsync(user, "DeployFailed", "Instance", instance.Id.ToString(),
instance.UniqueName, new { DeploymentId = deploymentId, Error = cause.Message },
CancellationToken.None);
// Writes the terminal Failed status and pushes the status change.
//
// Note: if the staging SaveChangesAsync was interrupted, an Added
// PendingDeployment may still be tracked and will be committed by this
// cleanup save. That row is orphaned (no RefreshDeploymentCommand was
// sent, so no site holds its token) and is removed by TTL purge /
// superseded by the next deploy -- harmless.
async Task PersistFailedStatusAsync()
{
await _repository.UpdateDeploymentRecordAsync(record, CancellationToken.None);
await _repository.SaveChangesAsync(CancellationToken.None);
NotifyStatusChange(record);
}
}
@@ -845,18 +910,36 @@ public class DeploymentService
/// <para>
/// <b>Disabled instances are skipped entirely</b> and reported in
/// <see cref="SiteDeploymentSummary.SkippedInstances"/> — see the filter below
/// for why. <b>Reconciliation is bounded per batch</b> by
/// for why. <b>Up-to-date instances are skipped too</b> unless
/// <paramref name="force"/> is set, for the reason documented on
/// <see cref="IsUpToDateAsync"/>: a redeploy is destructive to site-local
/// state, so pushing one to an instance whose config has not changed loses
/// data for no benefit. <b>Reconciliation is bounded per batch</b> by
/// <see cref="ReconcileUnderBatchGateAsync"/>, so an unreachable site costs the
/// batch one short timeout rather than one 30-second query per instance.
/// </para>
///
/// <para>
/// The two filters compose in one direction only: <b>Disabled wins</b>. A
/// Disabled instance is never deployed at bulk scope, force or not — force
/// suppresses the up-to-date filter, never the Disabled one.
/// </para>
/// </summary>
/// <param name="siteId">The database ID of the site whose instances are deployed.</param>
/// <param name="user">The username initiating the deployment, recorded on every record and audit row.</param>
/// <param name="force">
/// Deploy every deployable instance even when its config is unchanged — the
/// pre-2026-08 behaviour. The operator escape hatch for the case central's
/// recorded state cannot see: a site that lost its config (restored volume,
/// rebuilt node) looks up-to-date from here and needs the push anyway. Has no
/// effect on Disabled instances.
/// </param>
/// <param name="cancellationToken">Cancellation token for the operation.</param>
/// <returns>A per-instance result matrix, or a failure result when the site itself cannot be resolved.</returns>
public async Task<Result<SiteDeploymentSummary>> DeploySiteAsync(
int siteId,
string user,
bool force = false,
CancellationToken cancellationToken = default)
{
var site = await _siteRepository.GetSiteByIdAsync(siteId, cancellationToken);
@@ -874,16 +957,33 @@ public class DeploymentService
// stopped. They are reported as skipped rows instead — no flatten, no send,
// no state change — and remain individually deployable, which is the
// explicit "yes, re-enable this one" action.
//
// Unchanged instances are then filtered out for the mirror-image reason:
// see IsUpToDateAsync.
var deployable = new List<Instance>(instances.Count);
var skipped = new List<SkippedInstanceResult>();
var disabledCount = 0;
foreach (var instance in instances)
{
cancellationToken.ThrowIfCancellationRequested();
if (instance.State == InstanceState.Disabled)
{
skipped.Add(new SkippedInstanceResult(instance.Id, instance.UniqueName, DisabledSkipReason));
disabledCount++;
}
else if (!force && await IsUpToDateAsync(instance, cancellationToken))
{
skipped.Add(new SkippedInstanceResult(instance.Id, instance.UniqueName, UpToDateSkipReason));
}
else
{
deployable.Add(instance);
}
}
var upToDateCount = skipped.Count - disabledCount;
// ---- Phase 1: prepare, serially, on ONE shared flatten session. ----
var session = _flatteningPipeline.CreateSession();
var reconcileGate = new BulkReconcileGate(_options.SiteDeploymentReconcileTimeout);
@@ -988,14 +1088,110 @@ public class DeploymentService
_logger.LogInformation(
"Bulk deployment of site {SiteId} complete: {SuccessCount} succeeded, {FailureCount} failed, " +
"{SkippedCount} skipped as Disabled " +
"{DisabledCount} skipped as Disabled, {UpToDateCount} skipped as up-to-date (force={Force}) " +
"({ChainLoads} template-chain load(s) across {InstanceCount} instance(s))",
site.SiteIdentifier, summary.SuccessCount, summary.FailureCount, skipped.Count,
session.ChainLoads, deployable.Count);
site.SiteIdentifier, summary.SuccessCount, summary.FailureCount, disabledCount, upToDateCount,
force, session.ChainLoads, deployable.Count);
return Result<SiteDeploymentSummary>.Success(summary);
}
/// <summary>
/// True when a bulk site deploy should leave this instance alone because the
/// site is already running the config it would be sent.
///
/// <para>
/// <b>Why an unconditional bulk redeploy is the wrong default.</b> A redeploy
/// is NOT idempotent at the site: applying a config resets that instance's
/// static attribute overrides and clears its native alarm state, both by
/// design. A site-wide push therefore destroyed live operational state on every
/// instance it touched, including the ones whose configuration had not changed
/// at all — the usual case, since the operator is normally pushing one edited
/// template. Skipping them costs nothing (the site is already running that
/// revision) and preserves the state.
/// </para>
///
/// <para>
/// <b>The test is deliberately conservative.</b> Every uncertainty resolves to
/// "deploy it", so this filter can only ever fail to skip — it can never skip
/// an instance that needed the push:
/// </para>
/// <list type="bullet">
/// <item>
/// Only an <see cref="InstanceState.Enabled"/> instance can be up to date.
/// A <see cref="InstanceState.NotDeployed"/> one is not running the config
/// no matter what a leftover snapshot row says.
/// </item>
/// <item>
/// No <see cref="DeployedConfigSnapshot"/> means central has never seen a
/// successful apply — nothing to compare against.
/// </item>
/// <item>
/// A prior record that <see cref="ShouldQuerySiteBeforeRedeploy"/> flags
/// (stuck <c>InProgress</c>, or timeout-<c>Failed</c>) must go through
/// prepare, because that is where the query-before-redeploy reconciliation
/// that HEALS such a record runs. Skipping here would strand it forever.
/// </item>
/// <item>
/// The probe returning <see langword="null"/> (unresolvable template chain)
/// or throwing means staleness is indeterminate.
/// </item>
/// </list>
///
/// <para>
/// The comparison uses the snapshot's own stored <c>RevisionHash</c> rather
/// than <see cref="GetDeploymentComparisonAsync"/>'s recomputed-from-normalized
/// hash: the normalization exists to suppress a SPURIOUS stale flag (I-1), and
/// treating a spuriously-stale instance as stale here just deploys it, which is
/// the old behaviour and always safe.
/// </para>
///
/// <para>
/// <b>Cost.</b> The probe re-flattens on its own session rather than the
/// batch's, so a fully-stale site pays one extra flatten per instance before
/// prepare re-flattens it for real. That is bounded by the probe's watermark
/// memo (unchanged graph ⇒ no flatten at all on the second and later calls, and
/// the Deployments page's staleness sweep warms the same memo), and it buys
/// back far more than it costs on the common mostly-unchanged site, which now
/// sends no site round-trips at all instead of one per instance.
/// </para>
/// </summary>
/// <param name="instance">The candidate instance.</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns><see langword="true"/> only when the instance is provably unchanged.</returns>
private async Task<bool> IsUpToDateAsync(Instance instance, CancellationToken cancellationToken)
{
if (instance.State != InstanceState.Enabled)
return false;
var snapshot = await _repository.GetDeployedSnapshotByInstanceIdAsync(instance.Id, cancellationToken);
if (snapshot == null)
return false;
var prior = await _repository.GetCurrentDeploymentStatusAsync(instance.Id, cancellationToken);
if (prior != null && ShouldQuerySiteBeforeRedeploy(prior))
return false;
string? currentHash;
try
{
currentHash = await _staleInstanceProbe.GetCurrentRevisionHashAsync(instance.Id, cancellationToken);
}
catch (Exception ex) when (ex is not OperationCanceledException)
{
// Staleness is an optimisation, never a gate: an unexpected probe fault
// falls back to deploying the instance, which is what this method's
// absence would have done.
_logger.LogWarning(ex,
"Could not determine whether instance {Instance} is up to date; deploying it",
instance.UniqueName);
return false;
}
return currentHash != null
&& string.Equals(currentHash, snapshot.RevisionHash, StringComparison.Ordinal);
}
/// <summary>
/// Unwinds the prepared deployments a bulk deploy never finalised, after the
/// operation escaped early (cancellation, or a fault in phase 3's own
@@ -2085,11 +2281,13 @@ public record SkippedInstanceResult(
/// <param name="SuccessCount">Number of instances the site confirmed.</param>
/// <param name="FailureCount">Number of instances that failed for any reason.</param>
/// <param name="SkippedInstances">
/// Instances the batch deliberately did not attempt — currently the
/// Instances the batch deliberately did not attempt: the
/// <see cref="InstanceState.Disabled"/> ones, which a bulk push must not silently
/// re-enable. They are neither successes nor failures, so they are counted in
/// neither total. Additive and nullable so an older client deserialising this
/// summary is unaffected.
/// re-enable, and the up-to-date ones, which a bulk push must not needlessly reset
/// (a redeploy clears static overrides and native alarm state at the site). Each
/// row carries the reason. They are neither successes nor failures, so they are
/// counted in neither total. Additive and nullable so an older client deserialising
/// this summary is unaffected.
/// </param>
public record SiteDeploymentSummary(
string SiteIdentifier,
src/ZB.MOM.WW.ScadaBridge.Host/KpiRateFoldIntervalCrossValidator.cs
+87
View File
@@ -0,0 +1,87 @@
using Microsoft.Extensions.Options;
using ZB.MOM.WW.Configuration;
using ZB.MOM.WW.ScadaBridge.KpiHistory;
using ZB.MOM.WW.ScadaBridge.NotificationOutbox;
using ZB.MOM.WW.ScadaBridge.SiteCallAudit;
namespace ZB.MOM.WW.ScadaBridge.Host;
/// <summary>
/// Boot-time cross-validation of the "rate-fold interval triple" (arch-review P5-5,
/// Low). <see cref="KpiHistoryOptions.SampleInterval"/>,
/// <see cref="NotificationOutboxOptions.DeliveredKpiWindow"/>, and
/// <see cref="SiteCallAuditOptions.KpiInterval"/> are three independently-owned config
/// keys, bound in three different component projects with no compile-time relationship
/// to one another.
/// </summary>
/// <remarks>
/// <para>
/// <b>What no longer depends on this equality.</b> <c>NotificationOutboxKpiSampleSource</c>
/// and <c>SiteCallAuditKpiSampleSource</c> now rescale their Rate-classified KPI-history
/// samples (deliveredLastInterval, failedLastInterval) by
/// <c>SampleInterval / (DeliveredKpiWindow or KpiInterval)</c> before recording them, so
/// <c>KpiHistoryRepository.FoldHourlyRollupsAsync</c>'s per-hour <c>Sum</c> over those
/// samples is correct for <em>any</em> value of the other two keys — a widened
/// <c>DeliveredKpiWindow</c> can no longer silently over-count the hourly rollup the way
/// it did before P5-5.
/// </para>
/// <para>
/// <b>What still depends on it.</b> The rescale trades exact per-sample fidelity for
/// rollup correctness: whenever the scale factor is not 1, the recorded/charted
/// deliveredLastInterval/failedLastInterval sample no longer numerically equals what the
/// live Health-dashboard tile shows at the same instant (the live tile computation is
/// untouched and still uses the wider/narrower window directly) — a documented invariant
/// both sample sources' XML docs otherwise promise. This validator keeps that promise true
/// by construction: it fails the host at boot, naming all three configuration keys, unless
/// they are all equal (scale factor exactly 1 on both sides).
/// </para>
/// <para>
/// Registered against <c>IValidateOptions&lt;KpiHistoryOptions&gt;</c> — that options type
/// already calls <c>ValidateOnStart()</c> (see <c>KpiHistory.ServiceCollectionExtensions</c>),
/// so this cross-validator rides the same eager trigger without either
/// <see cref="NotificationOutboxOptions"/> or <see cref="SiteCallAuditOptions"/> needing
/// their own <c>ValidateOnStart()</c> call for this specific check to fire. It lives in Host
/// — the one composition root that references all three component projects — rather than in
/// any of the three (which deliberately do not reference each other).
/// </para>
/// </remarks>
public sealed class KpiRateFoldIntervalCrossValidator : OptionsValidatorBase<KpiHistoryOptions>
{
private readonly IOptions<NotificationOutboxOptions> _notificationOutboxOptions;
private readonly IOptions<SiteCallAuditOptions> _siteCallAuditOptions;
/// <summary>
/// Creates the validator with the two sibling options it cross-checks against
/// <see cref="KpiHistoryOptions"/>.
/// </summary>
public KpiRateFoldIntervalCrossValidator(
IOptions<NotificationOutboxOptions> notificationOutboxOptions,
IOptions<SiteCallAuditOptions> siteCallAuditOptions)
{
ArgumentNullException.ThrowIfNull(notificationOutboxOptions);
ArgumentNullException.ThrowIfNull(siteCallAuditOptions);
_notificationOutboxOptions = notificationOutboxOptions;
_siteCallAuditOptions = siteCallAuditOptions;
}
/// <inheritdoc />
protected override void Validate(ValidationBuilder builder, KpiHistoryOptions options)
{
var sampleInterval = options.SampleInterval;
var deliveredKpiWindow = _notificationOutboxOptions.Value.DeliveredKpiWindow;
var kpiInterval = _siteCallAuditOptions.Value.KpiInterval;
builder.RequireThat(
deliveredKpiWindow == sampleInterval && kpiInterval == sampleInterval,
"KPI rate-fold interval mismatch: " +
$"ScadaBridge:KpiHistory:{nameof(KpiHistoryOptions.SampleInterval)} ({sampleInterval}), " +
$"ScadaBridge:NotificationOutbox:{nameof(NotificationOutboxOptions.DeliveredKpiWindow)} " +
$"({deliveredKpiWindow}), and " +
$"ScadaBridge:SiteCallAudit:{nameof(SiteCallAuditOptions.KpiInterval)} ({kpiInterval}) " +
"must all be equal. The KPI-history sample sources rescale their delivered/failed-last-interval " +
"counts onto SampleInterval before recording so the hourly rollup fold's sum stays correct " +
"regardless of this divergence, but a mismatch here still means the recorded/charted KPI-history " +
"values would silently stop matching the live Health-dashboard tile at the same instant.");
}
}
src/ZB.MOM.WW.ScadaBridge.Host/Program.cs
+6
View File
@@ -346,6 +346,12 @@ try
// the KpiHistoryRecorderActor cluster singleton (started in
// AkkaHostedService). Observability/best-effort: NOT readiness-gated.
builder.Services.AddKpiHistory(builder.Configuration);
// P5-5 (arch-review Low): cross-validates the rate-fold interval triple
// (KpiHistoryOptions.SampleInterval, NotificationOutboxOptions.DeliveredKpiWindow,
// SiteCallAuditOptions.KpiInterval) at boot. Rides KpiHistoryOptions' existing
// ValidateOnStart trigger set by AddKpiHistory above.
builder.Services.TryAddEnumerable(
ServiceDescriptor.Singleton<IValidateOptions<KpiHistoryOptions>, KpiRateFoldIntervalCrossValidator>());
// The Audit Log's IKpiSampleSource, enumerated by the recorder registered just
// above. Central-only (its IAuditLogRepository is central-only); split out of
// AddAuditLog by deferred register row 50 for the same reason as
src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbSetup.cs
+66
View File
@@ -1,3 +1,4 @@
using Microsoft.Data.Sqlite;
using Microsoft.Extensions.Configuration;
using ZB.MOM.WW.LocalDb;
using ZB.MOM.WW.ScadaBridge.SiteEventLogging;
@@ -234,4 +235,69 @@ public static class SiteLocalDbSetup
return !string.IsNullOrWhiteSpace(section["PeerAddress"])
|| !string.IsNullOrWhiteSpace(section["ApiKey"]);
}
/// <summary>
/// Sets <c>auto_vacuum = INCREMENTAL</c> on the consolidated LocalDb file — arch-review
/// 2026-08-19 P4-6 — but ONLY when that file does not exist yet.
/// </summary>
/// <remarks>
/// <para>
/// <b>The defect this closes.</b> SQLite only honors a change to <c>auto_vacuum</c> while
/// a database is still completely table-less; once one table exists the pragma is a
/// silent no-op. <c>SqliteLocalDb</c>'s constructor runs <c>LocalDbSchema.EnsureCreated</c>
/// on its own master connection before <see cref="OnReady"/> — or anything else — ever
/// gets a connection to the file, so the pragma <see cref="SiteEventLogSchema.Apply"/>
/// used to issue had never once taken effect, in any real call path in this repo: by the
/// time it ran, LocalDb's own bookkeeping tables already existed.
/// </para>
/// <para>
/// <b>The fix, and why it needs no LocalDb API change.</b> <c>AddZbLocalDb</c> registers
/// <see cref="ILocalDb"/> as a LAZY singleton — nothing touches the file until it is
/// first resolved. This method is called from <c>SiteServiceRegistration.Configure</c>,
/// synchronously, BEFORE <c>services.AddZbLocalDb(...)</c> is even reached in that method
/// — i.e. well before the host is built and anything could resolve <see cref="ILocalDb"/>
/// — so it is free to open its own short-lived connection to the not-yet-existing file and
/// set the pragma there. Empirically verified (not merely inferred from the pragma's
/// documented "unless a subsequent operation modifies the database" caveat): opening a
/// connection to a fresh path and executing this single PRAGMA is sufficient by itself —
/// Microsoft.Data.Sqlite persists it to the file header on close, with no throwaway
/// CREATE/DROP TABLE and no VACUUM required. A LATER session — <c>SqliteLocalDb</c>'s own
/// constructor, opening the SAME path — sees the flag already baked in and every table it
/// (and everyone downstream) subsequently creates lives under incremental-vacuum mode.
/// </para>
/// <para>
/// <b>Only helps FRESH files — say so honestly, always.</b> A file that already exists
/// already has tables, so this method deliberately does nothing to it: retrofitting
/// auto_vacuum onto a populated database needs SQLite's own <c>VACUUM</c> rebuild, which
/// plan decision 2 (2026-08-19) explicitly rejects running unconditionally against a live
/// ~1 GB file on every boot. Consequence: every node on the docker rig today, and any
/// site database created before this fix shipped, stays permanently
/// <c>auto_vacuum=NONE</c> — <c>PRAGMA incremental_vacuum</c>, the per-trim call in
/// <see cref="ZB.MOM.WW.ScadaBridge.SiteEventLogging.EventLogPurgeService.PurgeByStorageCap"/>,
/// remains a harmless no-op there until an operator opts into a one-time offline VACUUM.
/// Only a NEWLY created site database benefits, from the moment this ships onward.
/// </para>
/// </remarks>
/// <param name="config">Configuration, read for <c>LocalDb:Path</c> only.</param>
internal static void EnsureAutoVacuumOnFreshFile(IConfiguration config)
{
var path = config["LocalDb:Path"];
if (string.IsNullOrWhiteSpace(path))
return; // AddZbLocalDb's own options validation reports a missing path loudly; nothing to pre-create here.
if (File.Exists(path))
return; // Already has tables (or is at least a pre-existing file) — the pragma cannot help it now.
var directory = Path.GetDirectoryName(path);
if (!string.IsNullOrEmpty(directory))
Directory.CreateDirectory(directory);
var connectionString = new SqliteConnectionStringBuilder { DataSource = path }.ToString();
using var connection = new SqliteConnection(connectionString);
connection.Open();
using var pragmaCmd = connection.CreateCommand();
pragmaCmd.CommandText = "PRAGMA auto_vacuum = INCREMENTAL";
pragmaCmd.ExecuteNonQuery();
}
}
src/ZB.MOM.WW.ScadaBridge.Host/SiteServiceRegistration.cs
+10
View File
@@ -91,6 +91,16 @@ public static class SiteServiceRegistration
// failure, and the default site path is the relative "./data/site-localdb.db".
// The shim is gone; SiteLocalDbDirectoryTests still pins the outcome here, since
// what this host needs is the guarantee, not any particular owner of it.
//
// arch-review 2026-08-19 P4-6: MUST run before AddZbLocalDb — its singleton
// factory is lazy, but the ordering still has to be enforced here rather than
// relied upon, because it is what makes "PRAGMA auto_vacuum = INCREMENTAL" take
// effect at all. SQLite only honors that pragma on a table-less database, and
// AddZbLocalDb's own SqliteLocalDb constructor creates LocalDb's bookkeeping
// tables the moment it is first resolved — see
// SiteLocalDbSetup.EnsureAutoVacuumOnFreshFile for the full mechanism, and note
// it only helps a file that does not exist yet.
SiteLocalDbSetup.EnsureAutoVacuumOnFreshFile(config);
services.AddZbLocalDb(config, db => SiteLocalDbSetup.OnReady(db, config));
// The replication engine, likewise unconditional but INERT by default: with no
src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementActor.cs
+57 -11
View File
@@ -943,12 +943,20 @@ public class ManagementActor : ReceiveActor
/// their scope, and every instance the batch touches belongs to that site by
/// construction — so the per-instance check would be redundant.
/// </para>
///
/// <para>
/// <c>Force</c> is relayed verbatim: unset (the default) the batch skips
/// instances whose config already matches the deployed revision, set it
/// redeploys them anyway. It carries no extra authorization — the command's
/// own Deployer + site scope already gates the whole operation, and forcing
/// only widens it to instances this same caller could deploy one at a time.
/// </para>
/// </summary>
private static async Task<object?> HandleDeploySite(IServiceProvider sp, MgmtDeploySiteCommand cmd, AuthenticatedUser user)
{
EnforceSiteScope(user, cmd.SiteId);
var svc = sp.GetRequiredService<DeploymentService>();
var result = await svc.DeploySiteAsync(cmd.SiteId, user.Username);
var result = await svc.DeploySiteAsync(cmd.SiteId, user.Username, cmd.Force);
return result.IsSuccess
? result.Value
: throw new ManagementCommandException(result.Error);
@@ -2931,11 +2939,7 @@ public class ManagementActor : ReceiveActor
/// <see cref="ArgumentException"/> be masked as a generic internal error.
/// </summary>
private static Commons.Types.Enums.DataType ParseDataType(string? dataType)
{
if (!Enum.TryParse<Commons.Types.Enums.DataType>(dataType, ignoreCase: true, out var parsed))
throw new ManagementCommandException($"Unrecognised data type '{dataType}'.");
return parsed;
}
=> ParseEnumStrict<Commons.Types.Enums.DataType>("data type", dataType);
/// <summary>
/// Parses an optional element data type token. Returns null when the token is
@@ -2945,9 +2949,51 @@ public class ManagementActor : ReceiveActor
private static Commons.Types.Enums.DataType? ParseElementDataType(string? elementDataType)
{
if (string.IsNullOrWhiteSpace(elementDataType)) return null;
if (!Enum.TryParse<Commons.Types.Enums.DataType>(elementDataType, ignoreCase: true, out var parsed))
throw new ManagementCommandException($"Unrecognised element type '{elementDataType}'.");
return parsed;
return ParseEnumStrict<Commons.Types.Enums.DataType>("element type", elementDataType);
}
/// <summary>
/// Parses a single required enum token by NAME ONLY, rejecting ordinals,
/// comma-joined name lists, and defined-but-out-of-range composites; throws a
/// curated <see cref="ManagementCommandException"/> naming the field and the
/// valid value set on any rejection.
/// </summary>
/// <remarks>
/// Register row 88 pattern (see <see cref="ParseDeploymentStatus"/>). Plain
/// <c>Enum.TryParse</c>/<c>Enum.Parse</c> admits three traps a wire-contract
/// field must not tolerate:
/// <list type="number">
/// <item><b>Ordinals.</b> <c>"2"</c> parses to whatever member sits at that
/// position (e.g. <c>DataType.Float</c>) — a caller-facing field is a
/// name-based contract, never a positional one.</item>
/// <item><b>Undefined ordinals.</b> <c>"99"</c> parses to an undefined enum
/// value that saves without error and only detonates at deploy time.</item>
/// <item><b>Comma-joined tokens.</b> <c>Enum.TryParse</c> bitwise-ORs a
/// comma-separated string's pieces together regardless of whether the enum
/// is <c>[Flags]</c> — <c>"String,Int32"</c> (4|1) silently becomes
/// <c>DateTime</c> (5) for a dense 0-based enum. Any token containing a
/// comma is rejected outright, before <c>TryParse</c> ever sees it.</item>
/// </list>
/// Mirrors <see cref="Commons.Types.Audit.AuditQueryParamParsers.ParseEnumListStrict{TEnum}"/>,
/// adapted to a single required value and <see cref="ManagementCommandException"/>.
/// </remarks>
/// <exception cref="ManagementCommandException">The value is not a defined member name of <typeparamref name="TEnum"/>.</exception>
private static TEnum ParseEnumStrict<TEnum>(string fieldName, string? raw)
where TEnum : struct, Enum
{
var trimmed = raw?.Trim() ?? string.Empty;
var looksNumeric = trimmed.Length > 0
&& (char.IsAsciiDigit(trimmed[0]) || trimmed[0] == '+' || trimmed[0] == '-');
if (!looksNumeric
&& !trimmed.Contains(',')
&& Enum.TryParse<TEnum>(trimmed, ignoreCase: true, out var parsed)
&& Enum.IsDefined(parsed))
{
return parsed;
}
throw new ManagementCommandException(
$"Unrecognised {fieldName} '{raw}'. Valid values: {string.Join(", ", Enum.GetNames<TEnum>())}.");
}
/// <summary>
@@ -3000,7 +3046,7 @@ public class ManagementActor : ReceiveActor
var svc = sp.GetRequiredService<TemplateService>();
var alarm = new TemplateAlarm(cmd.Name)
{
TriggerType = Enum.Parse<ZB.MOM.WW.ScadaBridge.Commons.Types.Enums.AlarmTriggerType>(cmd.TriggerType, ignoreCase: true),
TriggerType = ParseEnumStrict<ZB.MOM.WW.ScadaBridge.Commons.Types.Enums.AlarmTriggerType>("alarm trigger type", cmd.TriggerType),
PriorityLevel = cmd.PriorityLevel,
Description = cmd.Description,
TriggerConfiguration = cmd.TriggerConfiguration,
@@ -3015,7 +3061,7 @@ public class ManagementActor : ReceiveActor
var svc = sp.GetRequiredService<TemplateService>();
var alarm = new TemplateAlarm(cmd.Name)
{
TriggerType = Enum.Parse<ZB.MOM.WW.ScadaBridge.Commons.Types.Enums.AlarmTriggerType>(cmd.TriggerType, ignoreCase: true),
TriggerType = ParseEnumStrict<ZB.MOM.WW.ScadaBridge.Commons.Types.Enums.AlarmTriggerType>("alarm trigger type", cmd.TriggerType),
PriorityLevel = cmd.PriorityLevel,
Description = cmd.Description,
TriggerConfiguration = cmd.TriggerConfiguration,
src/ZB.MOM.WW.ScadaBridge.NotificationOutbox/Kpi/NotificationOutboxKpiSampleSource.cs
+66 -8
View File
@@ -4,6 +4,7 @@ using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Kpi;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Kpi;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Notifications;
using ZB.MOM.WW.ScadaBridge.KpiHistory;
namespace ZB.MOM.WW.ScadaBridge.NotificationOutbox.Kpi;
@@ -20,9 +21,12 @@ namespace ZB.MOM.WW.ScadaBridge.NotificationOutbox.Kpi;
/// <c>capturedAtUtc</c> rather than wall-clock <c>now</c>: the stuck cutoff is
/// <c>capturedAtUtc - <see cref="NotificationOutboxOptions.StuckAgeThreshold"/></c> and the
/// delivered window is <c>capturedAtUtc - <see cref="NotificationOutboxOptions.DeliveredKpiWindow"/></c>.
/// The COUNT metrics (queueDepth, stuckCount, parkedCount, deliveredLastInterval) equal the
/// live tile at the same instant; the <c>oldestPendingAgeSeconds</c> metric is computed against
/// the repository's internal clock and may differ from the live tile by the query-execution latency.
/// The COUNT metrics (queueDepth, stuckCount, parkedCount) equal the live tile at the same
/// instant; deliveredLastInterval is rescaled onto <c>KpiHistoryOptions.SampleInterval</c>
/// before being recorded (see the rescale note below) and equals the live tile only when
/// that rescale factor is 1, which the boot-time cross-validator guarantees; the
/// <c>oldestPendingAgeSeconds</c> metric is computed against the repository's internal clock
/// and may differ from the live tile by the query-execution latency.
/// </para>
/// <para>
/// Emits Global (<c>ScopeKey == null</c>), per-Site (<c>ScopeKey == SourceSiteId</c>), and
@@ -31,6 +35,28 @@ namespace ZB.MOM.WW.ScadaBridge.NotificationOutbox.Kpi;
/// <c>oldestPendingAgeSeconds</c> via <see cref="TimeSpan.TotalSeconds"/>; that one metric is
/// omitted when the age is <c>null</c>.
/// </para>
/// <para>
/// <b>deliveredLastInterval rescale (P5-5, arch-review Low).</b>
/// <see cref="KpiMetricAggregationCatalog"/> classifies this metric as a
/// <see cref="KpiRollupAggregation.Rate"/>: the central hourly rollup fold sums every
/// sample recorded in the hour, which is only correct if each sample's value is a
/// non-overlapping delta covering exactly one recorder tick
/// (<c>KpiHistoryOptions.SampleInterval</c>). The live cutoff this class queries the
/// repository with is still anchored on the independently-configurable
/// <see cref="NotificationOutboxOptions.DeliveredKpiWindow"/> — unchanged, so the live
/// dashboard tile keeps its own smoothing window — but the COUNT that gets written into
/// <see cref="KpiSample.Value"/> is rescaled by
/// <c>SampleInterval / DeliveredKpiWindow</c> before it is recorded, so the rollup's sum
/// is correct for the hour regardless of how <c>DeliveredKpiWindow</c> is tuned (e.g.
/// widened to smooth a noisy live tile). At the documented defaults (both 1 minute) the
/// scale factor is exactly 1 and this is a no-op. A configuration where the scale factor
/// is not 1 — i.e. where the live tile's window no longer equals the recorder's sampling
/// cadence — still boots and rolls up correctly, but the recorded/charted
/// deliveredLastInterval sample then no longer numerically equals the live tile at the
/// same instant (only the <em>rollup total</em> is guaranteed correct); the boot-time
/// cross-validator (Host's <c>KpiRateFoldIntervalCrossValidator</c>) enforces the equality
/// so that divergence can never actually happen in a config that boots.
/// </para>
/// </remarks>
public sealed class NotificationOutboxKpiSampleSource : IKpiSampleSource
{
@@ -44,23 +70,51 @@ public sealed class NotificationOutboxKpiSampleSource : IKpiSampleSource
private readonly INotificationOutboxRepository _repository;
private readonly NotificationOutboxOptions _options;
private readonly KpiHistoryOptions _kpiHistoryOptions;
/// <summary>
/// Creates the sample source.
/// </summary>
/// <param name="repository">Outbox repository providing the KPI computation methods.</param>
/// <param name="options">Outbox options carrying the stuck-age and delivered-window cutoffs.</param>
/// <param name="kpiHistoryOptions">
/// KPI History options carrying <see cref="KpiHistoryOptions.SampleInterval"/> — the
/// recorder's sampling cadence the deliveredLastInterval sample is rescaled onto (P5-5),
/// so the hourly rollup fold's per-sample sum stays correct independent of
/// <see cref="NotificationOutboxOptions.DeliveredKpiWindow"/>.
/// </param>
public NotificationOutboxKpiSampleSource(
INotificationOutboxRepository repository,
IOptions<NotificationOutboxOptions> options)
IOptions<NotificationOutboxOptions> options,
IOptions<KpiHistoryOptions> kpiHistoryOptions)
{
ArgumentNullException.ThrowIfNull(repository);
ArgumentNullException.ThrowIfNull(options);
ArgumentNullException.ThrowIfNull(kpiHistoryOptions);
_repository = repository;
_options = options.Value;
_kpiHistoryOptions = kpiHistoryOptions.Value;
}
/// <summary>
/// Rescale factor applied to the raw <c>deliveredLastInterval</c> count before it is
/// recorded, so the value written into <see cref="KpiSample.Value"/> represents a
/// non-overlapping per-<see cref="KpiHistoryOptions.SampleInterval"/> delta regardless of
/// how wide <see cref="NotificationOutboxOptions.DeliveredKpiWindow"/> is configured — the
/// hourly rollup fold (<c>KpiHistoryRepository.FoldHourlyRollupsAsync</c>) sums this value
/// across the hour's samples, which is only correct for non-overlapping deltas. Exactly 1
/// at the documented defaults (both 1 minute). Guards against a non-positive
/// <see cref="NotificationOutboxOptions.DeliveredKpiWindow"/> (validated &gt; 0 at boot by
/// <see cref="NotificationOutboxOptionsValidator"/>, but this defends the arithmetic even
/// if that validation is ever bypassed) by falling back to the unscaled 1:1 factor rather
/// than dividing by zero.
/// </summary>
private double DeliveredIntervalScaleFactor =>
_options.DeliveredKpiWindow > TimeSpan.Zero
? _kpiHistoryOptions.SampleInterval.TotalSeconds / _options.DeliveredKpiWindow.TotalSeconds
: 1.0;
/// <inheritdoc />
public string Source => KpiSources.NotificationOutbox;
@@ -75,13 +129,14 @@ public sealed class NotificationOutboxKpiSampleSource : IKpiSampleSource
var deliveredSince = capturedAt - _options.DeliveredKpiWindow;
var samples = new List<KpiSample>();
var scaleFactor = DeliveredIntervalScaleFactor;
var global = await _repository.ComputeKpisAsync(stuckCutoff, deliveredSince, cancellationToken)
.ConfigureAwait(false);
AddSnapshot(
samples, capturedAtUtc, KpiScopes.Global, scopeKey: null,
global.QueueDepth, global.StuckCount, global.ParkedCount,
global.DeliveredLastInterval, global.OldestPendingAge);
global.DeliveredLastInterval * scaleFactor, global.OldestPendingAge);
var perSite = await _repository.ComputePerSiteKpisAsync(stuckCutoff, deliveredSince, cancellationToken)
.ConfigureAwait(false);
@@ -90,7 +145,7 @@ public sealed class NotificationOutboxKpiSampleSource : IKpiSampleSource
AddSnapshot(
samples, capturedAtUtc, KpiScopes.Site, site.SourceSiteId,
site.QueueDepth, site.StuckCount, site.ParkedCount,
site.DeliveredLastInterval, site.OldestPendingAge);
site.DeliveredLastInterval * scaleFactor, site.OldestPendingAge);
}
var perNode = await _repository.ComputePerNodeKpisAsync(stuckCutoff, deliveredSince, cancellationToken)
@@ -100,7 +155,7 @@ public sealed class NotificationOutboxKpiSampleSource : IKpiSampleSource
AddSnapshot(
samples, capturedAtUtc, KpiScopes.Node, node.SourceNode,
node.QueueDepth, node.StuckCount, node.ParkedCount,
node.DeliveredLastInterval, node.OldestPendingAge);
node.DeliveredLastInterval * scaleFactor, node.OldestPendingAge);
}
return samples;
@@ -109,6 +164,9 @@ public sealed class NotificationOutboxKpiSampleSource : IKpiSampleSource
/// <summary>
/// Appends the five outbox metrics for one snapshot at the given scope, omitting
/// <c>oldestPendingAgeSeconds</c> when <paramref name="oldestPendingAge"/> is <c>null</c>.
/// <paramref name="deliveredLastInterval"/> is already rescaled by
/// <see cref="DeliveredIntervalScaleFactor"/> (a <c>double</c>, not the raw repository
/// <c>int</c> count) by the caller.
/// </summary>
private void AddSnapshot(
List<KpiSample> samples,
@@ -118,7 +176,7 @@ public sealed class NotificationOutboxKpiSampleSource : IKpiSampleSource
int queueDepth,
int stuckCount,
int parkedCount,
int deliveredLastInterval,
double deliveredLastInterval,
TimeSpan? oldestPendingAge)
{
samples.Add(Sample(capturedAtUtc, MetricQueueDepth, scope, scopeKey, queueDepth));
src/ZB.MOM.WW.ScadaBridge.NotificationOutbox/NotificationOutboxActor.cs
+31 -11
View File
@@ -1109,14 +1109,25 @@ public class NotificationOutboxActor : ReceiveActor, IWithTimers
notification.NextAttemptAt = null;
notification.LastError = null;
// Zero rows updated means the row was purged between the read above and
// this write. Answer the operator honestly instead of reporting a
// re-queue that never happened — and emit no un-park audit row, because
// there is nothing to attribute it to.
if (!await repository.UpdateAsync(notification))
// Compare-and-swap on Status == Parked (arch-review 2026-08-19 P5-4). The
// read above and this write are two separate round trips; a detached
// PipeTo task can race the dispatcher's own write (or a second
// concurrent operator action) in between, so the in-memory check above
// is a fast-path rejection, not the correctness guard — a blind
// UpdateAsync here could stamp a row that has already moved on (e.g. the
// dispatcher delivered it) back to Pending with stale fields. A lost CAS
// means either the row vanished (purge) or it progressed past Parked
// under us; re-read to report which honestly instead of silently
// overwriting whatever is there now, and emit no un-park audit row in
// either case, because there is nothing to attribute it to.
if (!await repository.TryUpdateFromParkedAsync(notification))
{
var current = await repository.GetByIdAsync(request.NotificationId);
return new RetryNotificationResponse(
request.CorrelationId, Success: false, ErrorMessage: "notification not found");
request.CorrelationId, Success: false,
ErrorMessage: current is null
? "notification not found"
: $"notification already progressed to {current.Status}; retry not applied");
}
// Operator re-queued a parked notification. Emit a Submitted NotifyDeliver
@@ -1190,13 +1201,22 @@ public class NotificationOutboxActor : ReceiveActor, IWithTimers
notification.Status = NotificationStatus.Discarded;
// Same vanished-row honesty as the retry path: a purge between the read
// and the write leaves nothing to discard, so report not-found rather
// than success (and skip the terminal audit row below).
if (!await repository.UpdateAsync(notification))
// Same compare-and-swap on Status == Parked as the retry path (arch-review
// 2026-08-19 P5-4): a purge between the read and the write leaves nothing
// to discard, and a dispatcher/other-operator write landing in that same
// window (e.g. a delivery succeeding) must not be silently clobbered by a
// blind write of this stale in-memory copy. Re-read on a lost CAS to
// report which case it was, and skip the terminal audit row below either
// way — there is nothing to attribute a discard of a row that was never
// actually discarded.
if (!await repository.TryUpdateFromParkedAsync(notification))
{
var current = await repository.GetByIdAsync(request.NotificationId);
return new DiscardNotificationResponse(
request.CorrelationId, Success: false, ErrorMessage: "notification not found");
request.CorrelationId, Success: false,
ErrorMessage: current is null
? "notification not found"
: $"notification already progressed to {current.Status}; discard not applied");
}
// A manual discard is the OTHER code path that produces
src/ZB.MOM.WW.ScadaBridge.NotificationOutbox/NotificationOutboxOptionsValidator.cs
+58
View File
@@ -0,0 +1,58 @@
using ZB.MOM.WW.Configuration;
namespace ZB.MOM.WW.ScadaBridge.NotificationOutbox;
/// <summary>
/// Validates <see cref="NotificationOutboxOptions"/> at host startup (arch-review
/// §1.5 eager-options-validation pattern — this options class had no registered
/// validator before P5-5). The durations feed the dispatch sweep, stuck-call
/// detection, terminal-row purge, and the delivered-throughput KPI window; a
/// zero/negative cadence would spin an Akka scheduler, and a non-positive
/// <see cref="NotificationOutboxOptions.DeliveredKpiWindow"/> would collapse the
/// KPI window and — because <see cref="Kpi.NotificationOutboxKpiSampleSource"/>
/// divides by it to rescale the KPI-history sample onto
/// <c>KpiHistoryOptions.SampleInterval</c> — divide by zero.
/// <see cref="NotificationOutboxOptions.MaxParallelDeliveries"/> is deliberately
/// validated on its BASE property, not the clamped
/// <see cref="NotificationOutboxOptions.ResolvedMaxParallelDeliveries"/>, so a
/// misconfigured non-positive value fails fast instead of silently clamping to 1
/// (mirrors <c>SiteCallAuditOptionsValidator</c>'s cadence-field convention).
/// </summary>
public sealed class NotificationOutboxOptionsValidator : OptionsValidatorBase<NotificationOutboxOptions>
{
/// <inheritdoc />
protected override void Validate(ValidationBuilder builder, NotificationOutboxOptions options)
{
builder.RequireThat(options.DispatchInterval > TimeSpan.Zero,
$"ScadaBridge:NotificationOutbox:{nameof(NotificationOutboxOptions.DispatchInterval)} " +
$"({options.DispatchInterval}) must be a positive duration; it is the dispatch-sweep cadence.");
builder.RequireThat(options.DispatchBatchSize > 0,
$"ScadaBridge:NotificationOutbox:{nameof(NotificationOutboxOptions.DispatchBatchSize)} " +
$"({options.DispatchBatchSize}) must be positive; it is the max rows claimed per sweep.");
builder.RequireThat(options.StuckAgeThreshold > TimeSpan.Zero,
$"ScadaBridge:NotificationOutbox:{nameof(NotificationOutboxOptions.StuckAgeThreshold)} " +
$"({options.StuckAgeThreshold}) must be a positive duration; it is the age past which a " +
"pending/retrying notification is flagged stuck.");
builder.RequireThat(options.TerminalRetention > TimeSpan.Zero,
$"ScadaBridge:NotificationOutbox:{nameof(NotificationOutboxOptions.TerminalRetention)} " +
$"({options.TerminalRetention}) must be a positive duration; it is the terminal-row retention window.");
builder.RequireThat(options.PurgeInterval > TimeSpan.Zero,
$"ScadaBridge:NotificationOutbox:{nameof(NotificationOutboxOptions.PurgeInterval)} " +
$"({options.PurgeInterval}) must be a positive duration; it is the terminal-row purge-sweep cadence.");
builder.RequireThat(options.DeliveredKpiWindow > TimeSpan.Zero,
$"ScadaBridge:NotificationOutbox:{nameof(NotificationOutboxOptions.DeliveredKpiWindow)} " +
$"({options.DeliveredKpiWindow}) must be a positive duration; it is the trailing window for the " +
"delivered-throughput KPI and the divisor the KPI-history sample source rescales onto " +
"KpiHistoryOptions.SampleInterval.");
builder.RequireThat(options.MaxParallelDeliveries > 0,
$"ScadaBridge:NotificationOutbox:{nameof(NotificationOutboxOptions.MaxParallelDeliveries)} " +
$"({options.MaxParallelDeliveries}) must be positive; a non-positive value silently clamps to 1 " +
$"via {nameof(NotificationOutboxOptions.ResolvedMaxParallelDeliveries)}, hiding the misconfiguration.");
}
}
src/ZB.MOM.WW.ScadaBridge.NotificationOutbox/ServiceCollectionExtensions.cs
+9 -1
View File
@@ -56,8 +56,16 @@ public static class ServiceCollectionExtensions
{
ArgumentNullException.ThrowIfNull(services);
// Eager startup validation (arch-review §1.5 / P5-5): this options class had no
// registered validator before — ValidateOnStart makes a bad
// ScadaBridge:NotificationOutbox section fail fast at boot with a clear,
// key-naming message instead of spinning a scheduler or, for DeliveredKpiWindow,
// dividing by zero in NotificationOutboxKpiSampleSource's KpiHistory rescale.
services.AddOptions<NotificationOutboxOptions>()
.BindConfiguration(OptionsSection);
.BindConfiguration(OptionsSection)
.ValidateOnStart();
services.TryAddEnumerable(
ServiceDescriptor.Singleton<IValidateOptions<NotificationOutboxOptions>, NotificationOutboxOptionsValidator>());
// Bind + validate SmsOptions (ScadaBridge:Sms): the SMS adapter's body cap and
// per-request timeout fallback. ValidateOnStart surfaces a bad option at host
src/ZB.MOM.WW.ScadaBridge.NotificationOutbox/ZB.MOM.WW.ScadaBridge.NotificationOutbox.csproj
+5
View File
@@ -21,6 +21,11 @@
(ISmtpClientWrapper, SmtpPermanentException, SmtpTlsModeParser,
OAuth2TokenService). -->
<ProjectReference Include="../ZB.MOM.WW.ScadaBridge.NotificationService/ZB.MOM.WW.ScadaBridge.NotificationService.csproj" />
<!-- The KPI sample source computes deliveredLastInterval against
KpiHistoryOptions.SampleInterval directly (P5-5), not the independently
configurable DeliveredKpiWindow, so the central hourly rollup fold's
per-sample sum is correct regardless of how DeliveredKpiWindow is tuned. -->
<ProjectReference Include="../ZB.MOM.WW.ScadaBridge.KpiHistory/ZB.MOM.WW.ScadaBridge.KpiHistory.csproj" />
</ItemGroup>
<ItemGroup>
src/ZB.MOM.WW.ScadaBridge.SiteCallAudit/Kpi/SiteCallAuditKpiSampleSource.cs
+61 -8
View File
@@ -4,6 +4,7 @@ using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Kpi;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Audit;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Kpi;
using ZB.MOM.WW.ScadaBridge.KpiHistory;
namespace ZB.MOM.WW.ScadaBridge.SiteCallAudit.Kpi;
@@ -21,8 +22,10 @@ namespace ZB.MOM.WW.ScadaBridge.SiteCallAudit.Kpi;
/// live <c>SiteCallAuditActor</c> KPI handlers derive them
/// (<c>stuckCutoff = capturedAtUtc - StuckAgeThreshold</c>,
/// <c>intervalSince = capturedAtUtc - KpiInterval</c>). The COUNT metrics
/// (buffered, parked, failedLastInterval, deliveredLastInterval, stuck) equal the
/// live tile at the same instant; the <c>oldestPendingAgeSeconds</c> metric is
/// (buffered, parked, stuck) equal the live tile at the same instant;
/// failedLastInterval and deliveredLastInterval are rescaled before recording (see
/// below) and equal the live tile only when that rescale factor is 1, which the
/// boot-time cross-validator guarantees; the <c>oldestPendingAgeSeconds</c> metric is
/// computed against the repository's internal clock and may differ from the live
/// tile by the query-execution latency. The recorder's <c>capturedAtUtc</c> is
/// the single anchor for both cutoffs.
@@ -32,6 +35,25 @@ namespace ZB.MOM.WW.ScadaBridge.SiteCallAudit.Kpi;
/// each sampling pass resolves a fresh repository scope, mirroring the actor's
/// scope-per-message repository access.
/// </para>
/// <para>
/// <b>failedLastInterval / deliveredLastInterval rescale (P5-5, arch-review Low).</b>
/// <see cref="KpiMetricAggregationCatalog"/> classifies both as
/// <see cref="KpiRollupAggregation.Rate"/>: the central hourly rollup fold sums every
/// sample recorded in the hour, which is only correct if each sample's value is a
/// non-overlapping delta covering exactly one recorder tick
/// (<c>KpiHistoryOptions.SampleInterval</c>). The live cutoff this class queries the
/// repository with is still anchored on the independently-configurable
/// <see cref="SiteCallAuditOptions.KpiInterval"/> — unchanged, so the live dashboard
/// tile keeps its own smoothing window — but the COUNTS that get written into
/// <see cref="KpiSample.Value"/> are rescaled by <c>SampleInterval / KpiInterval</c>
/// before they are recorded, so the rollup's sum is correct for the hour regardless
/// of how <c>KpiInterval</c> is tuned. At the documented defaults (both 1 minute) the
/// scale factor is exactly 1 and this is a no-op. The boot-time cross-validator
/// (Host's <c>KpiRateFoldIntervalCrossValidator</c>) enforces
/// <c>KpiInterval == SampleInterval</c> (alongside the outbox's
/// <c>DeliveredKpiWindow</c>) so a config where the recorded/charted samples no
/// longer equal the live tile can never actually boot.
/// </para>
/// </remarks>
public sealed class SiteCallAuditKpiSampleSource : IKpiSampleSource
{
@@ -48,6 +70,7 @@ public sealed class SiteCallAuditKpiSampleSource : IKpiSampleSource
private readonly ISiteCallAuditRepository _repository;
private readonly SiteCallAuditOptions _options;
private readonly KpiHistoryOptions _kpiHistoryOptions;
/// <summary>
/// Creates the sample source over the central <c>SiteCalls</c> repository and
@@ -56,17 +79,43 @@ public sealed class SiteCallAuditKpiSampleSource : IKpiSampleSource
/// </summary>
/// <param name="repository">The central <c>SiteCalls</c> operational-state repository.</param>
/// <param name="options">Site Call Audit windowing options (stuck-age + KPI interval).</param>
/// <param name="kpiHistoryOptions">
/// KPI History options carrying <see cref="KpiHistoryOptions.SampleInterval"/> — the
/// recorder's sampling cadence the failedLastInterval/deliveredLastInterval samples are
/// rescaled onto (P5-5), so the hourly rollup fold's per-sample sum stays correct
/// independent of <see cref="SiteCallAuditOptions.KpiInterval"/>.
/// </param>
public SiteCallAuditKpiSampleSource(
ISiteCallAuditRepository repository,
IOptions<SiteCallAuditOptions> options)
IOptions<SiteCallAuditOptions> options,
IOptions<KpiHistoryOptions> kpiHistoryOptions)
{
ArgumentNullException.ThrowIfNull(repository);
ArgumentNullException.ThrowIfNull(options);
ArgumentNullException.ThrowIfNull(kpiHistoryOptions);
_repository = repository;
_options = options.Value;
_kpiHistoryOptions = kpiHistoryOptions.Value;
}
/// <summary>
/// Rescale factor applied to the raw <c>failedLastInterval</c>/<c>deliveredLastInterval</c>
/// counts before they are recorded, so the values written into
/// <see cref="KpiSample.Value"/> represent non-overlapping
/// per-<see cref="KpiHistoryOptions.SampleInterval"/> deltas regardless of how wide
/// <see cref="SiteCallAuditOptions.KpiInterval"/> is configured. Exactly 1 at the
/// documented defaults (both 1 minute). Guards against a non-positive
/// <see cref="SiteCallAuditOptions.KpiInterval"/> (validated &gt; 0 at boot by
/// <see cref="SiteCallAuditOptionsValidator"/>, but this defends the arithmetic even if
/// that validation is ever bypassed) by falling back to the unscaled 1:1 factor rather
/// than dividing by zero.
/// </summary>
private double IntervalScaleFactor =>
_options.KpiInterval > TimeSpan.Zero
? _kpiHistoryOptions.SampleInterval.TotalSeconds / _options.KpiInterval.TotalSeconds
: 1.0;
/// <inheritdoc />
public string Source => KpiSources.SiteCallAudit;
@@ -90,12 +139,13 @@ public sealed class SiteCallAuditKpiSampleSource : IKpiSampleSource
.ConfigureAwait(false);
var samples = new List<KpiSample>();
var scaleFactor = IntervalScaleFactor;
// Global scope (null ScopeKey).
AddSnapshot(
samples, capturedAtUtc, KpiScopes.Global, scopeKey: null,
global.BufferedCount, global.ParkedCount,
global.FailedLastInterval, global.DeliveredLastInterval,
global.FailedLastInterval * scaleFactor, global.DeliveredLastInterval * scaleFactor,
global.OldestPendingAge, global.StuckCount);
// Per-site scope (ScopeKey = source site id).
@@ -104,7 +154,7 @@ public sealed class SiteCallAuditKpiSampleSource : IKpiSampleSource
AddSnapshot(
samples, capturedAtUtc, KpiScopes.Site, scopeKey: site.SourceSite,
site.BufferedCount, site.ParkedCount,
site.FailedLastInterval, site.DeliveredLastInterval,
site.FailedLastInterval * scaleFactor, site.DeliveredLastInterval * scaleFactor,
site.OldestPendingAge, site.StuckCount);
}
@@ -114,7 +164,7 @@ public sealed class SiteCallAuditKpiSampleSource : IKpiSampleSource
AddSnapshot(
samples, capturedAtUtc, KpiScopes.Node, scopeKey: node.SourceNode,
node.BufferedCount, node.ParkedCount,
node.FailedLastInterval, node.DeliveredLastInterval,
node.FailedLastInterval * scaleFactor, node.DeliveredLastInterval * scaleFactor,
node.OldestPendingAge, node.StuckCount);
}
@@ -125,6 +175,9 @@ public sealed class SiteCallAuditKpiSampleSource : IKpiSampleSource
/// Appends the six-metric catalog for one snapshot at the given scope. The
/// oldest-pending-age metric is omitted when the snapshot's age is
/// <c>null</c> (no non-terminal rows) rather than written as zero.
/// <paramref name="failedLastInterval"/>/<paramref name="deliveredLastInterval"/> are
/// already rescaled by <see cref="IntervalScaleFactor"/> (a <c>double</c>, not the raw
/// repository <c>int</c> counts) by the caller.
/// </summary>
private void AddSnapshot(
List<KpiSample> samples,
@@ -133,8 +186,8 @@ public sealed class SiteCallAuditKpiSampleSource : IKpiSampleSource
string? scopeKey,
int buffered,
int parked,
int failedLastInterval,
int deliveredLastInterval,
double failedLastInterval,
double deliveredLastInterval,
TimeSpan? oldestPendingAge,
int stuck)
{
src/ZB.MOM.WW.ScadaBridge.SiteCallAudit/ZB.MOM.WW.ScadaBridge.SiteCallAudit.csproj
+6
View File
@@ -39,6 +39,12 @@
SiteCallAudit. Preferred over moving the interfaces into Commons (Commons has no Akka /
Communication dependency and would have to carry a Communication-adjacent message). -->
<ProjectReference Include="../ZB.MOM.WW.ScadaBridge.AuditLog/ZB.MOM.WW.ScadaBridge.AuditLog.csproj" />
<!-- The KPI sample source computes deliveredLastInterval/failedLastInterval
against KpiHistoryOptions.SampleInterval directly (P5-5), not the
independently configurable KpiInterval, so the central hourly rollup
fold's per-sample sum is correct regardless of how KpiInterval is tuned.
No cycle: KpiHistory references only Commons. -->
<ProjectReference Include="../ZB.MOM.WW.ScadaBridge.KpiHistory/ZB.MOM.WW.ScadaBridge.KpiHistory.csproj" />
</ItemGroup>
<ItemGroup>
src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/EventLogPurgeService.cs
+35 -6
View File
@@ -164,6 +164,19 @@ public class EventLogPurgeService : BackgroundService
_options.RetentionDays, _options.MaxStorageMb, _options.PurgeInterval,
_options.MaxPurgedRowsPerTick > 0 ? _options.MaxPurgedRowsPerTick.ToString() : "unlimited");
// arch-review 2026-08-19 P4-5: yield BEFORE the initial (potentially ~17s
// ceiling-limited, ~285s unpaced worst case) RunPurge() rather than calling it
// as the first synchronous statement. .NET 8+'s BackgroundService already invokes
// ExecuteAsync via Task.Run, so a synchronous prelude here does not actually block
// IHostedService.StartAsync on this repo's net10.0 target (verified empirically —
// it did not reproduce). This Task.Yield() is still the correct fix rather than a
// no-op: it makes the "purge never blocks the calling thread" contract a property of
// THIS method, not an artifact of the host's scheduling internals — reflection-invoking
// ExecuteAsync directly (as EventLogPurgeServiceTests does, bypassing BackgroundService
// entirely) would otherwise still run the initial purge fully synchronously, and any
// future caller that awaits ExecuteAsync directly (not through StartAsync) would block.
await Task.Yield();
// Run an initial purge on startup
var ceilingHit = RunPurge();
@@ -213,9 +226,18 @@ public class EventLogPurgeService : BackgroundService
}
catch (Exception checkEx)
{
_logger.LogDebug(checkEx,
"Active-node check threw during purge tick; treating as active for logging purposes");
isActive = true;
// arch-review 2026-08-19 P4-5: fail CLOSED, not "active". This predicate no
// longer gates any work (P4-4 — both nodes purge their own file every tick
// regardless), so a wrong value here cannot skip a purge; but silently
// reporting "active" for a check that just threw is dishonest, and an
// operator correlating this line against the real active-node state would be
// misled into thinking the check succeeded. Warning (not Debug) because the
// check itself is broken and is worth noticing on its own.
_logger.LogWarning(checkEx,
"Active-node check threw during purge tick; logging this tick as inactive " +
"rather than claiming an active status the check could not confirm. The purge " +
"itself is unaffected — every node purges its own file regardless of this flag");
isActive = false;
}
_logger.LogDebug("Running event log purge tick (active={IsActive})", isActive);
@@ -513,9 +535,16 @@ public class EventLogPurgeService : BackgroundService
""";
var rows = cmd.ExecuteNonQuery();
// Reclaim free pages so page_count/freelist measurement reflects the
// delete. Effective because auto_vacuum = INCREMENTAL is set at schema
// creation; harmless otherwise.
// Reclaim free pages so page_count/freelist measurement reflects the delete.
// arch-review 2026-08-19 P4-6: this is effective ONLY on a consolidated
// LocalDb file that was FRESH when SiteLocalDbSetup.EnsureAutoVacuumOnFreshFile
// ran (Host/SiteLocalDbSetup.cs) — auto_vacuum can only be enabled before any
// table exists, and SqliteLocalDb's constructor creates LocalDb's own
// bookkeeping tables before this component ever gets a connection. Every file
// that predates that fix — including every node on the docker rig today —
// stays permanently auto_vacuum=NONE, and this PRAGMA is a harmless, silent
// no-op on it (retrofitting one requires an offline VACUUM, deliberately not
// attempted here against a live ~1 GB file — plan decision 2, 2026-08-19).
using var vacuumCmd = connection.CreateCommand();
vacuumCmd.CommandText = "PRAGMA incremental_vacuum";
vacuumCmd.ExecuteNonQuery();
src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogSchema.cs
+18 -9
View File
@@ -36,15 +36,24 @@ public static class SiteEventLogSchema
{
ArgumentNullException.ThrowIfNull(connection);
// auto_vacuum must be set before any table is created for it to take effect
// on a fresh database. With INCREMENTAL mode, PRAGMA incremental_vacuum can
// later reclaim free pages so the storage-cap purge can shrink the file.
using (var pragmaCmd = connection.CreateCommand())
{
pragmaCmd.CommandText = "PRAGMA auto_vacuum = INCREMENTAL";
pragmaCmd.ExecuteNonQuery();
}
// arch-review 2026-08-19 P4-6: this method used to also set
// "PRAGMA auto_vacuum = INCREMENTAL" here, on the theory that setting it right
// before CREATE TABLE would make it take effect on a fresh database. That was
// wrong in every real call path: SQLite only honors an auto_vacuum change on a
// database that has NO tables yet, and by the time this method runs — via the
// Host's AddZbLocalDb onReady callback (SiteLocalDbSetup.OnReady) OR via
// SiteEventLogger's own InitializeSchema fallback for directly-constructed
// instances — the consolidated file already holds LocalDb's own bookkeeping
// tables (SqliteLocalDb's constructor runs LocalDbSchema.EnsureCreated on its
// master connection before either caller ever gets a connection). So the pragma
// here had never once taken effect; it was a silent, permanent no-op. The real
// fix now lives in Host/SiteLocalDbSetup.cs
// (EnsureAutoVacuumOnFreshFile), which pre-creates a genuinely fresh LocalDb file
// with the pragma set BEFORE AddZbLocalDb's lazy singleton ever constructs
// SqliteLocalDb — and, being a Host-layer, one-time, pre-file-existence step, it
// helps ONLY brand-new site databases; every file created before this fix shipped
// stays auto_vacuum=NONE forever (see the PurgeByStorageCap remarks in
// EventLogPurgeService for the operational consequence).
using var cmd = connection.CreateCommand();
cmd.CommandText = """
CREATE TABLE IF NOT EXISTS site_events (
src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs
+36 -11
View File
@@ -82,6 +82,21 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
/// </summary>
private readonly Func<Task<List<DeployedInstance>>> _configLoader;
/// <summary>
/// Reads a single deployed config row by instance name. Defaults to
/// <see cref="SiteStorageService.GetDeployedConfigAsync"/>; overridable via the ctor so a
/// test can hold the read open mid-flight (row 101's ensure-started race tests).
/// </summary>
/// <remarks>
/// Register P1-4: the row-67 ensure-started heal used to read through
/// <see cref="_configLoader"/> — the same full-table-including-ConfigJson load the boot
/// sequence uses — just to <c>FirstOrDefault</c> one instance. A volume-restore reconcile
/// that raises hundreds of <see cref="EnsureInstanceStartedCommand"/>s in a row turned that
/// into hundreds of full-table reads. This seam is deliberately separate from
/// <see cref="_configLoader"/>: the two are read at different cardinalities (all rows for
/// boot, one row per heal) and a test overriding one must not accidentally gate the other.
/// </remarks>
private readonly Func<string, Task<DeployedInstance?>> _deployedConfigLoader;
/// <summary>
/// Reads the durable <c>data_connection_definitions</c> rows (deferred register rows 89 + 90).
/// Defaults to <see cref="SiteStorageService.GetAllDataConnectionDefinitionsAsync"/>;
/// overridable via the ctor so a test can drive a read failure.
@@ -276,6 +291,11 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
/// defaults to reading from <paramref name="storage"/>. Primarily for tests that need to
/// drive a read failure.
/// </param>
/// <param name="deployedConfigLoader">
/// Register P1-4: optional override for the single-row deployed-config read the row-67
/// ensure-started heal uses; defaults to reading from <paramref name="storage"/>. Primarily
/// for tests that need to hold the read open mid-flight to drive the row-101 race.
/// </param>
public DeploymentManagerActor(
SiteStorageService storage,
ScriptCompilationService compilationService,
@@ -292,7 +312,8 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
Func<Task<List<DeployedInstance>>>? configLoader = null,
ExternalSystemDefinitionCache? externalSystemCache = null,
ScriptExecutionScheduler? scriptScheduler = null,
Func<Task<List<StoredDataConnectionDefinition>>>? connectionDefinitionLoader = null)
Func<Task<List<StoredDataConnectionDefinition>>>? connectionDefinitionLoader = null,
Func<string, Task<DeployedInstance?>>? deployedConfigLoader = null)
{
_scriptScheduler = scriptScheduler;
_storage = storage;
@@ -311,6 +332,7 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
? TimeSpan.FromSeconds(options.InstanceSweepIntervalSeconds)
: TimeSpan.Zero;
_configLoader = configLoader ?? (() => _storage.GetAllDeployedConfigsAsync());
_deployedConfigLoader = deployedConfigLoader ?? (name => _storage.GetDeployedConfigAsync(name));
_connectionDefinitionLoader = connectionDefinitionLoader
?? (() => _storage.GetAllDataConnectionDefinitionsAsync());
_logger = logger;
@@ -1502,8 +1524,9 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
Task.Run(async () =>
{
await _storage.SetInstanceEnabledAsync(instanceName, true);
var configs = await _storage.GetAllDeployedConfigsAsync();
var config = configs.FirstOrDefault(c => c.InstanceUniqueName == instanceName);
// Row P1-4: a single-row read — this used to materialize every deployed config
// (including every ConfigJson blob) just to FirstOrDefault this one instance.
var config = await _storage.GetDeployedConfigAsync(instanceName);
return new EnableResult(command, config, null, sender);
}).ContinueWith(t =>
{
@@ -1595,12 +1618,11 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
var generationAtRead = _storeWriteGeneration;
Task.Run(async () =>
{
// Read through the same seam the boot load and the sweep read through
// (_configLoader defaults to _storage.GetAllDeployedConfigsAsync, so this is the
// identical production call), which lets a test hold the read open and land the
// result behind a committed disable/delete.
var configs = await _configLoader();
var config = configs.FirstOrDefault(c => c.InstanceUniqueName == instanceName);
// Register P1-4: read a single row through _deployedConfigLoader (defaults to
// _storage.GetDeployedConfigAsync — a WHERE instance_unique_name = @name read, not
// the full-table-including-ConfigJson load the boot sequence uses), which still lets
// a test hold the read open and land the result behind a committed disable/delete.
var config = await _deployedConfigLoader(instanceName);
return new EnsureInstanceStartedResult(command, config, null, generationAtRead);
}).ContinueWith(t => t.IsCompletedSuccessfully
? t.Result
@@ -2095,7 +2117,10 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
var sender = Sender;
var instanceName = request.InstanceUniqueName;
_storage.GetAllDeployedConfigsAsync().ContinueWith(t =>
// Row P1-4: a single-row, ConfigJson-free projection — this used to materialize every
// deployed config (including every ConfigJson blob) just to FirstOrDefault this one
// instance's DeploymentId/RevisionHash. Neither is ever read here.
_storage.GetDeployedConfigMetadataAsync(instanceName).ContinueWith(t =>
{
if (!t.IsCompletedSuccessfully)
{
@@ -2110,7 +2135,7 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
request.CorrelationId, instanceName, false, null, null, DateTimeOffset.UtcNow);
}
var config = t.Result.FirstOrDefault(c => c.InstanceUniqueName == instanceName);
var config = t.Result;
return config == null
? new DeploymentStateQueryResponse(
request.CorrelationId, instanceName, false, null, null, DateTimeOffset.UtcNow)
src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs
+99
View File
@@ -108,6 +108,89 @@ public class SiteStorageService
return results;
}
/// <summary>
/// Returns the single deployed instance configuration for <paramref name="instanceUniqueName"/>,
/// or <see langword="null"/> if no row exists for it.
/// </summary>
/// <remarks>
/// Register P1-4: <see cref="GetAllDeployedConfigsAsync"/> plus a <c>FirstOrDefault</c> was
/// materializing every deployed config — including every <c>ConfigJson</c> blob — just to find
/// one row. A volume-restore reconcile healing hundreds of instances turned that into hundreds
/// of full-table reads. Callers that need the full row (including <c>ConfigJson</c>) for exactly
/// one instance — e.g. to (re)start its Instance Actor — should use this instead. Callers that
/// only need identity/staleness metadata should use <see cref="GetDeployedConfigMetadataAsync"/>,
/// which never touches the <c>config_json</c> column.
/// </remarks>
/// <param name="instanceUniqueName">The unique name of the instance.</param>
/// <returns>
/// A task that resolves to the deployed configuration row, or <see langword="null"/> if none
/// exists for the instance.
/// </returns>
public async Task<DeployedInstance?> GetDeployedConfigAsync(string instanceUniqueName)
{
await using var connection = OpenConnection();
await using var command = connection.CreateCommand();
command.CommandText = @"
SELECT instance_unique_name, config_json, deployment_id, revision_hash, is_enabled, deployed_at
FROM deployed_configurations
WHERE instance_unique_name = @name
LIMIT 1";
command.Parameters.AddWithValue("@name", instanceUniqueName);
await using var reader = await command.ExecuteReaderAsync();
if (!await reader.ReadAsync())
return null;
return new DeployedInstance
{
InstanceUniqueName = reader.GetString(0),
ConfigJson = reader.GetString(1),
DeploymentId = reader.GetString(2),
RevisionHash = reader.GetString(3),
IsEnabled = reader.GetInt64(4) != 0,
DeployedAt = reader.GetString(5)
};
}
/// <summary>
/// Returns the deployment-identity metadata for <paramref name="instanceUniqueName"/> —
/// <c>(name, deployment id, revision hash, is_enabled)</c> — without reading the
/// <c>config_json</c> column, or <see langword="null"/> if no row exists for it.
/// </summary>
/// <remarks>
/// Register P1-4: for a caller that only needs staleness/identity metadata (e.g. the
/// central deployment-state query, which reports only <c>DeploymentId</c>/<c>RevisionHash</c>),
/// reading the <c>config_json</c> blob is pure waste. This projection is that lighter read.
/// </remarks>
/// <param name="instanceUniqueName">The unique name of the instance.</param>
/// <returns>
/// A task that resolves to the metadata row, or <see langword="null"/> if none exists for the
/// instance.
/// </returns>
public async Task<DeployedConfigMetadata?> GetDeployedConfigMetadataAsync(string instanceUniqueName)
{
await using var connection = OpenConnection();
await using var command = connection.CreateCommand();
command.CommandText = @"
SELECT instance_unique_name, deployment_id, revision_hash, is_enabled
FROM deployed_configurations
WHERE instance_unique_name = @name
LIMIT 1";
command.Parameters.AddWithValue("@name", instanceUniqueName);
await using var reader = await command.ExecuteReaderAsync();
if (!await reader.ReadAsync())
return null;
return new DeployedConfigMetadata(
InstanceUniqueName: reader.GetString(0),
DeploymentId: reader.GetString(1),
RevisionHash: reader.GetString(2),
IsEnabled: reader.GetInt64(3) != 0);
}
/// <summary>
/// Stores or updates a deployed instance configuration. Uses UPSERT semantics.
/// </summary>
@@ -939,6 +1022,22 @@ public class DeployedInstance
public string DeployedAt { get; init; } = string.Empty;
}
/// <summary>
/// Deployment-identity metadata for one deployed instance — the projection of
/// <see cref="DeployedInstance"/> that omits <c>ConfigJson</c>. Returned by
/// <see cref="SiteStorageService.GetDeployedConfigMetadataAsync"/> for callers that only need to
/// answer "is this instance deployed, and at what revision" (register P1-4).
/// </summary>
/// <param name="InstanceUniqueName">The unique name of the instance.</param>
/// <param name="DeploymentId">The unique deployment identifier.</param>
/// <param name="RevisionHash">The configuration revision hash for staleness detection.</param>
/// <param name="IsEnabled">Whether the instance is enabled.</param>
public record DeployedConfigMetadata(
string InstanceUniqueName,
string DeploymentId,
string RevisionHash,
bool IsEnabled);
/// <summary>
/// Represents a shared script stored locally in SQLite.
/// </summary>
tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Central/SiteAuditReconciliationActorTests.cs
+182
View File
@@ -176,6 +176,69 @@ public class SiteAuditReconciliationActorTests : TestKit, IClassFixture<MsSqlMig
Task.FromResult<IReadOnlyList<string>>(Array.Empty<string>());
}
/// <summary>
/// Recording stub whose <see cref="InsertIfNotExistsAsync"/> throws for EVERY
/// row except a synthetic <see cref="AuditKind.ReconciliationAbandoned"/> row
/// (identified by its <c>Action</c> suffix, mirroring <see cref="PoisonInsertRepo"/>'s
/// assertion pattern). Used to model a site whose backlog contains one row that
/// NEVER inserts (the eventual permanent-abandon target) alongside a co-batch
/// row that also never inserts but changes EventId every pull — i.e. it never
/// individually reaches <c>MaxPermanentInsertAttempts</c>, so it keeps
/// <c>hasUnresolvedFailure</c> true and holds the cursor at <c>since</c> forever
/// (P5-6's defect trigger). Dedupes inserted rows on <see cref="AuditEvent.EventId"/>
/// exactly like the real <c>InsertIfNotExistsAsync</c>, so a test asserting on
/// <see cref="Inserted"/>'s count is exercising the same idempotency contract
/// production relies on.
/// </summary>
private sealed class AlwaysFailUnlessSyntheticRepo : IAuditLogRepository
{
public List<AuditEvent> Inserted { get; } = new();
private readonly HashSet<Guid> _seen = new();
public Task InsertIfNotExistsAsync(AuditEvent evt, CancellationToken ct = default)
{
if (!evt.Action.EndsWith(".ReconciliationAbandoned", StringComparison.Ordinal))
{
throw new InvalidOperationException("simulated permanent insert failure");
}
if (_seen.Add(evt.EventId))
{
Inserted.Add(evt);
}
return Task.CompletedTask;
}
public Task<IReadOnlyList<AuditEvent>> QueryAsync(
AuditLogQueryFilter filter, AuditLogPaging paging, CancellationToken ct = default) =>
Task.FromResult<IReadOnlyList<AuditEvent>>(Inserted);
public Task<long> SwitchOutPartitionAsync(DateTime monthBoundary, TimeSpan? commandTimeout = null, CancellationToken ct = default) =>
Task.FromResult(0L);
public Task<long> PurgeChannelOlderThanAsync(
string channel, DateTime threshold, int batchSize, TimeSpan? commandTimeout = null, CancellationToken ct = default) =>
Task.FromResult(0L);
public Task<long> BackfillSourceNodeAsync(
string sentinel, DateTime before, int batchSize, CancellationToken ct = default) =>
Task.FromResult(0L);
public Task<IReadOnlyList<DateTime>> GetPartitionBoundariesOlderThanAsync(
DateTime threshold, CancellationToken ct = default) =>
Task.FromResult<IReadOnlyList<DateTime>>(Array.Empty<DateTime>());
public Task<ZB.MOM.WW.ScadaBridge.Commons.Types.AuditLogKpiSnapshot> GetKpiSnapshotAsync(
TimeSpan window, DateTime? nowUtc = null, CancellationToken ct = default) =>
Task.FromResult(new ZB.MOM.WW.ScadaBridge.Commons.Types.AuditLogKpiSnapshot(0L, 0L, 0L, nowUtc ?? DateTime.UtcNow));
public Task<IReadOnlyList<ExecutionTreeNode>> GetExecutionTreeAsync(
Guid executionId, CancellationToken ct = default) =>
Task.FromResult<IReadOnlyList<ExecutionTreeNode>>(Array.Empty<ExecutionTreeNode>());
public Task<IReadOnlyList<string>> GetDistinctSourceNodesAsync(CancellationToken ct = default) =>
Task.FromResult<IReadOnlyList<string>>(Array.Empty<string>());
}
/// <summary>
/// In-memory enumerator returning a static list of sites.
/// </summary>
@@ -727,4 +790,123 @@ public class SiteAuditReconciliationActorTests : TestKit, IClassFixture<MsSqlMig
StopReconciliationActor(actor);
}
// ---------------------------------------------------------------------
// 10. P5-6: deterministic synthetic ReconciliationAbandoned EventId
// ---------------------------------------------------------------------
/// <summary>
/// Reproduces the P5-6 defect end to end: a permanently-poisoned row sits
/// in a batch alongside a co-batch row that ALSO fails on every pull but
/// carries a fresh <see cref="AuditEvent.EventId"/> each time (modelling a
/// site that keeps producing new events with transient insert failures).
/// Per <c>PullSiteAsync</c>'s <c>hasUnresolvedFailure</c> rule the rotating
/// row never individually reaches <c>MaxPermanentInsertAttempts</c>, so it
/// holds the reconciliation cursor at <c>since</c> forever — the poison row
/// keeps being re-pulled with a reset attempt counter (its
/// <c>_failedInsertAttempts</c> entry was cleared by its first abandonment)
/// and is permanently abandoned again roughly every
/// <c>MaxPermanentInsertAttempts</c> ticks. Ten ticks reproduces two full
/// abandon cycles for the SAME poisoned EventId. Pre-fix, each cycle mints a
/// fresh random synthetic <c>EventId</c>, so two DISTINCT rows land in the
/// repository. Post-fix, <c>DeriveAbandonmentEventId</c> derives the same id
/// both times and <c>InsertIfNotExistsAsync</c>'s idempotency collapses the
/// second insert into a no-op, leaving exactly one row.
/// </summary>
[Fact]
public void ReAbandonment_SameLostEventId_YieldsExactlyOneCentralRow()
{
var siteId = "siteReabandon";
var poisonTime = new DateTime(2026, 5, 20, 10, 5, 0, DateTimeKind.Utc);
var poison = NewEvent(siteId, poisonTime);
// 10 scripted responses: enough for exactly two 5-attempt abandon
// cycles of the poison row (the cursor never advances between them,
// so the SAME poison row + a freshly-guid'd rotating sibling are
// re-served every tick). Once exhausted, ScriptedPullClient returns
// empty responses, so no further cycles occur — the test's eventual
// row count is bounded and deterministic.
var responses = Enumerable.Range(0, 10)
.Select(i => new PullAuditEventsResponse(
new[] { poison, NewEvent(siteId, poisonTime.AddSeconds(i + 1)) },
MoreAvailable: false))
.ToArray();
var client = new ScriptedPullClient().Script(siteId, responses);
var sites = new StaticEnumerator(new SiteEntry(siteId, "http://x:8083"));
var repo = new AlwaysFailUnlessSyntheticRepo();
var actor = CreateActor(sites, client, repo, FastTickOptions());
// Wait for all 10 scripted pulls to have been consumed, i.e. both
// abandon cycles have run to completion.
AwaitAssert(
() => Assert.True(client.Calls.Count >= 10, $"expected >= 10 pull calls, got {client.Calls.Count}"),
duration: TimeSpan.FromSeconds(10),
interval: TimeSpan.FromMilliseconds(100));
// Give the actor's mailbox a chance to fully drain the 10th tick's
// processing (the call count above only proves PullAsync was invoked,
// not that the resulting insert(s) have landed yet) before reading
// the repository state.
AwaitAssert(
() => Assert.True(
repo.Inserted.Any(e => e.Action.EndsWith(".ReconciliationAbandoned", StringComparison.Ordinal)),
"expected at least one synthetic ReconciliationAbandoned row"),
duration: TimeSpan.FromSeconds(5),
interval: TimeSpan.FromMilliseconds(100));
StopReconciliationActor(actor);
var synthetic = repo.Inserted
.Where(e => e.Action.EndsWith(".ReconciliationAbandoned", StringComparison.Ordinal))
.ToList();
// The defect: pre-fix this is 2 (a fresh random EventId per abandon
// cycle defeats InsertIfNotExistsAsync's dedup). Post-fix it is 1.
Assert.Single(synthetic);
Assert.Contains(poison.EventId.ToString(), synthetic[0].DetailsJson);
}
/// <summary>
/// Pure-function coverage of <see cref="SiteAuditReconciliationActor.DeriveAbandonmentEventId"/>
/// directly: distinct lost rows must derive distinct synthetic ids (so two
/// different permanently-abandoned rows don't collide into one central row).
/// </summary>
[Fact]
public void DeriveAbandonmentEventId_DifferentLostRows_YieldDistinctSyntheticIds()
{
var lostA = Guid.NewGuid();
var lostB = Guid.NewGuid();
var derivedA = SiteAuditReconciliationActor.DeriveAbandonmentEventId(lostA);
var derivedB = SiteAuditReconciliationActor.DeriveAbandonmentEventId(lostB);
Assert.NotEqual(derivedA, derivedB);
Assert.NotEqual(lostA, derivedA);
Assert.NotEqual(lostB, derivedB);
}
/// <summary>
/// Pure-function coverage proving the derivation depends ONLY on the lost
/// row's <see cref="AuditEvent.EventId"/> VALUE — not on object identity, not
/// on any clock/random/process state. Two independently-parsed <see cref="Guid"/>
/// values that are equal by value must derive the identical synthetic id, and
/// the exact value is pinned as a regression guard: since the function has no
/// external input besides <paramref name="lostEventId"/>, this literal must
/// reproduce identically in any process — this run, CI, or a reconciliation
/// singleton after a failover restart — which is the "stable across process
/// restarts" contract the fix depends on for idempotent re-abandonment.
/// </summary>
[Fact]
public void DeriveAbandonmentEventId_IsPureAndStableAcrossIndependentInvocations()
{
var lostEventId = Guid.Parse("6b2f1e3a-8c9d-4a2b-9e3f-1a2b3c4d5e6f");
var roundTripped = Guid.Parse(lostEventId.ToString("D"));
var first = SiteAuditReconciliationActor.DeriveAbandonmentEventId(lostEventId);
var second = SiteAuditReconciliationActor.DeriveAbandonmentEventId(roundTripped);
Assert.Equal(first, second);
Assert.Equal(Guid.Parse("36baac53-f4c6-835e-9a77-d0b350a4e0d9"), first);
}
}
tests/ZB.MOM.WW.ScadaBridge.CLI.Tests/CommandTreeTests.cs
+22
View File
@@ -157,6 +157,28 @@ public class CommandTreeTests
Assert.Empty(site.Parse(["--site-id", "3"]).Errors);
}
[Fact]
public void DeploySite_HasForceFlag_DefaultingToTheNonDestructiveBehaviour()
{
// P7-6: bulk deploy skips instances whose config is unchanged, because a
// redeploy resets their static overrides and native alarm state at the
// site. --force is the operator escape hatch (restored volume, rebuilt
// node) and must be a FLAG — absent means skip-unchanged, which is the
// safe default a scripted `deploy site` keeps getting.
var deploy = DeployCommands.Build(Url, Format, Username, Password);
var site = deploy.Subcommands.Single(c => c.Name == "site");
var forceOption = (Option<bool>)site.Options.Single(o => o.Name == "--force");
var withoutForce = site.Parse(["--site-id", "3"]);
Assert.Empty(withoutForce.Errors);
Assert.False(withoutForce.GetValue(forceOption));
// A bare --force (no value) must parse: it is a flag, not a value option.
var withForce = site.Parse(["--site-id", "3", "--force"]);
Assert.Empty(withForce.Errors);
Assert.True(withForce.GetValue(forceOption));
}
[Fact]
public void InstanceNativeAlarmSource_HasSetAndClear()
{
tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Deployment/DeploymentsRequestGenerationTests.cs
+254
View File
@@ -0,0 +1,254 @@
using System.Reflection;
using System.Security.Claims;
using Bunit;
using Microsoft.AspNetCore.Components.Authorization;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging.Abstractions;
using NSubstitute;
using ZB.MOM.WW.ScadaBridge.CentralUI.Auth;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Deployment;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
using ZB.MOM.WW.ScadaBridge.DeploymentManager;
using DeploymentsPage = ZB.MOM.WW.ScadaBridge.CentralUI.Components.Pages.Deployment.Deployments;
namespace ZB.MOM.WW.ScadaBridge.CentralUI.Tests.Deployment;
/// <summary>
/// Regression tests for arch-review 2026-08-19 P6-11. Three independent initiators —
/// search, paging, and the push/coalesce reload — can each have a
/// <c>LoadDataAsync</c> flight in flight on the Deployment Status page at once. Every
/// flight awaits its own round-trip to central MS SQL, so COMPLETION order is not
/// CALL order: a slow flight that started first can resolve LAST and overwrite a
/// newer grid the operator is already looking at.
///
/// <para>
/// Separately, the filter-debounce timer used to write <c>_currentPage = 1</c>
/// directly on its raw <see cref="System.Threading.Timer"/> callback thread — a
/// Blazor threading violation, since component state may only be mutated on the
/// renderer's synchronization context — and that write was NOT covered by the
/// request-generation guard at all, so even a correctly-marshaled version of it
/// could still reset the operator's page out from under a newer navigation. The fix
/// resolves the target page into a local at fetch start and applies it only behind
/// the same generation check as every other piece of state
/// (<c>LoadDataAsync(resetToFirstPage:)</c>), so a stale filter-debounce flight is
/// discarded exactly like a stale flight on any other initiator.
/// </para>
///
/// <para>
/// Both races are driven deterministically with <see cref="TaskCompletionSource{T}"/>
/// -backed repository stubs — never sleep-then-assert timing. Both tests reference
/// methods/fields the fix introduces (<c>_requestGeneration</c>,
/// <c>DispatchFilterReloadAsync</c>), so they fail — by compile error or reflection
/// NRE — against the unfixed source, in addition to failing behaviourally once a
/// same-shaped unfixed build is exercised; verified via a saved pre-fix copy of
/// <c>Deployments.razor</c>, not <c>git stash</c> (repo-wide, unsafe to use in this
/// worktree wave).
/// </para>
/// </summary>
public class DeploymentsRequestGenerationTests : BunitContext
{
private IDeploymentManagerRepository _deployRepo = null!;
private DeploymentStatusNotifier _notifier = null!;
/// <summary>
/// One controllable response per call to <c>QueryDeploymentListPageAsync</c>, in
/// call order. A queued <see cref="TaskCompletionSource{T}"/> lets the test choose
/// exactly when each flight's fetch resolves, regardless of when it started.
/// </summary>
private readonly List<TaskCompletionSource<DeploymentListPage>> _pageResponses = new();
private int _pageCallIndex;
private void RegisterServices()
{
_deployRepo = Substitute.For<IDeploymentManagerRepository>();
_notifier = new DeploymentStatusNotifier(NullLogger<DeploymentStatusNotifier>.Instance);
_deployRepo.QueryDeploymentListPageAsync(
Arg.Any<DeploymentListFilter>(), Arg.Any<int>(), Arg.Any<int>(), Arg.Any<CancellationToken>())
.Returns(_ =>
{
var index = Interlocked.Increment(ref _pageCallIndex) - 1;
return index < _pageResponses.Count
? _pageResponses[index].Task
: Task.FromResult(DeploymentListPage.Empty);
});
_deployRepo.GetDeploymentStatusCountsAsync(
Arg.Any<DeploymentListFilter>(), Arg.Any<CancellationToken>())
.Returns(DeploymentStatusCounts.Empty);
Services.AddSingleton(_deployRepo);
Services.AddSingleton<IDeploymentStatusNotifier>(_notifier);
var identity = new ClaimsIdentity(
new[] { new Claim(ClaimTypes.Name, "deployer") }, "TestCookie");
var stubAuth = new StubAuthStateProvider(
new AuthenticationState(new ClaimsPrincipal(identity)));
Services.AddSingleton<AuthenticationStateProvider>(stubAuth);
Services.AddScoped(_ => new SiteScopeService(stubAuth));
}
private sealed class StubAuthStateProvider : AuthenticationStateProvider
{
private readonly AuthenticationState _state;
public StubAuthStateProvider(AuthenticationState state) => _state = state;
public override Task<AuthenticationState> GetAuthenticationStateAsync()
=> Task.FromResult(_state);
}
private static TaskCompletionSource<DeploymentListPage> Gate() =>
new(TaskCreationOptions.RunContinuationsAsynchronously);
private static TaskCompletionSource<DeploymentListPage> CompletedGate(DeploymentListPage page)
{
var tcs = Gate();
tcs.SetResult(page);
return tcs;
}
private static DeploymentListRow Row(int id, string instance) =>
new(
Id: id,
DeploymentId: $"dep-{id:D4}",
InstanceId: id,
InstanceUniqueName: instance,
SiteId: 1,
Status: DeploymentStatus.Success,
RevisionHash: $"rev{id:D4}hash",
DeployedBy: "deployer",
DeployedAt: DateTimeOffset.UtcNow.AddMinutes(-id),
CompletedAt: DateTimeOffset.UtcNow,
ErrorMessage: null);
/// <summary>
/// Invokes a private async method on the component via the renderer's dispatcher
/// (mirroring the pattern in <c>DebugViewConnectDisposeRaceTests</c>) and hands
/// back the resulting <see cref="Task"/> without awaiting it, so the caller can
/// act while the flight is still in progress.
/// </summary>
private static async Task<Task> InvokePrivateAsync(
IRenderedComponent<DeploymentsPage> cut, string methodName, params object?[] args)
{
var method = typeof(DeploymentsPage).GetMethod(methodName, BindingFlags.Instance | BindingFlags.NonPublic)!;
Task? task = null;
await cut.InvokeAsync(() => { task = (Task)method.Invoke(cut.Instance, args)!; });
return task!;
}
private static IReadOnlyList<DeploymentListRow> GetRows(DeploymentsPage page) =>
(IReadOnlyList<DeploymentListRow>)typeof(DeploymentsPage)
.GetField("_rows", BindingFlags.Instance | BindingFlags.NonPublic)!
.GetValue(page)!;
private static int GetCurrentPage(DeploymentsPage page) =>
(int)typeof(DeploymentsPage)
.GetField("_currentPage", BindingFlags.Instance | BindingFlags.NonPublic)!
.GetValue(page)!;
[Fact]
public async Task StaleFlightArrivingAfterANewerOne_DoesNotOverwriteTheGrid()
{
RegisterServices();
var initialRows = Enumerable.Range(1, 25).Select(i => Row(i, $"Init-{i}")).ToList();
_pageResponses.Add(CompletedGate(new DeploymentListPage(initialRows, TotalCount: 100))); // call #1: initial render
var stale = Gate(); // call #2: page-2 flight — held open, the SLOW, STALE flight
_pageResponses.Add(stale);
// call #3: status-filter flight — resolves immediately, the FAST, NEWER flight
_pageResponses.Add(CompletedGate(
new DeploymentListPage(new[] { Row(999, "TileFilteredFreshRow") }, TotalCount: 1)));
var cut = Render<DeploymentsPage>();
cut.WaitForAssertion(() => Assert.Contains(GetRows(cut.Instance), r => r.InstanceUniqueName == "Init-1"));
// Flight A starts FIRST (a page click) and blocks awaiting `stale`.
var flightA = await InvokePrivateAsync(cut, "GoToPageAsync", 2);
Assert.False(flightA.IsCompleted);
// Flight B starts SECOND (a status-tile click) while A is still in flight, and
// resolves immediately — it is the newer, correct result.
var flightB = await InvokePrivateAsync(cut, "SetStatusFilterAsync", DeploymentStatus.Failed);
await flightB.WaitAsync(TimeSpan.FromSeconds(5));
Assert.Contains(GetRows(cut.Instance), r => r.InstanceUniqueName == "TileFilteredFreshRow");
// Now let the STALE flight (page 2) resolve, arriving AFTER the newer one has
// already applied. Awaiting flightA guarantees its continuation — including
// the apply-time generation check under test — has fully run before we
// assert, rather than racing bUnit's render pump.
stale.SetResult(new DeploymentListPage(new[] { Row(2, "PageTwoStaleRow") }, TotalCount: 100));
await flightA.WaitAsync(TimeSpan.FromSeconds(5));
// Pre-fix, flight A's late arrival clobbers _rows with the stale page-2 data.
// Post-fix, its generation is no longer current, so it discards its own
// results and the newer grid survives untouched.
Assert.DoesNotContain(GetRows(cut.Instance), r => r.InstanceUniqueName == "PageTwoStaleRow");
Assert.Contains(GetRows(cut.Instance), r => r.InstanceUniqueName == "TileFilteredFreshRow");
}
/// <summary>
/// The "does the timer refresh reset the page mid-search" half of P6-11. The
/// filter-debounce ("timer") reload fires first, as if 500ms had elapsed after a
/// keystroke, and is held open. While it is still in flight the operator
/// navigates to page 2 by other means (a fresh, faster flight). The debounce
/// flight — which intends to reset to page 1 — must not resolve later and
/// clobber a page the operator has since, and more recently, navigated to.
/// <para>
/// This is driven through <c>DispatchFilterReloadAsync</c> directly (rather than
/// the outer <c>OnFilterDebounceElapsed</c> timer callback, which is fire-and-
/// forget and gives no <see cref="Task"/> handle to await) so completion of the
/// apply step under test — including the generation check — can be awaited
/// deterministically instead of racing bUnit's render pump.
/// </para>
/// </summary>
[Fact]
public async Task StaleFilterDebounceFlight_DoesNotResetThePageAfterANewerNavigation()
{
RegisterServices();
var initialRows = Enumerable.Range(1, 25).Select(i => Row(i, $"Init-{i}")).ToList();
_pageResponses.Add(CompletedGate(new DeploymentListPage(initialRows, TotalCount: 100))); // call #1: initial render
var staleSearch = Gate(); // call #2: the debounce-triggered ("timer") flight — held open, STALE
_pageResponses.Add(staleSearch);
// call #3: page-2 click — resolves immediately, NEWER.
_pageResponses.Add(CompletedGate(
new DeploymentListPage(new[] { Row(2, "PageTwoFreshRow") }, TotalCount: 100)));
var cut = Render<DeploymentsPage>();
cut.WaitForAssertion(() => Assert.Contains(GetRows(cut.Instance), r => r.InstanceUniqueName == "Init-1"));
// The filter-debounce ("timer") reload starts FIRST and blocks on `staleSearch`.
var staleFlight = await InvokePrivateAsync(cut, "DispatchFilterReloadAsync");
Assert.False(staleFlight.IsCompleted);
// While it is still in flight, the operator navigates to page 2 — a fresh
// action that resolves immediately.
var nav = await InvokePrivateAsync(cut, "GoToPageAsync", 2);
await nav.WaitAsync(TimeSpan.FromSeconds(5));
Assert.Equal(2, GetCurrentPage(cut.Instance));
Assert.Contains(GetRows(cut.Instance), r => r.InstanceUniqueName == "PageTwoFreshRow");
// Now let the STALE debounce flight resolve, arriving after the navigation
// already applied. Awaiting staleFlight guarantees its continuation —
// including the apply-time generation check under test — has fully run
// before we assert, rather than racing bUnit's render pump.
staleSearch.SetResult(
new DeploymentListPage(new[] { Row(1, "SearchPageOneStaleRow") }, TotalCount: 1));
await staleFlight.WaitAsync(TimeSpan.FromSeconds(5));
// Pre-fix, the stale flight unconditionally wrote _currentPage = 1 up front
// and then overwrote the grid with page-1 data once it resolved — resetting
// the operator's page mid an already-completed navigation. Post-fix, its
// generation is no longer current, so both the page and the grid are left
// untouched.
Assert.Equal(2, GetCurrentPage(cut.Instance));
Assert.Contains(GetRows(cut.Instance), r => r.InstanceUniqueName == "PageTwoFreshRow");
Assert.DoesNotContain(GetRows(cut.Instance), r => r.InstanceUniqueName == "SearchPageOneStaleRow");
}
}
tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Design/ScriptFormSandboxDisposalTests.cs
+286
View File
@@ -0,0 +1,286 @@
using System.Reflection;
using System.Security.Claims;
using Bunit;
using Microsoft.AspNetCore.Components.Authorization;
using Microsoft.Extensions.Caching.Memory;
using Microsoft.Extensions.DependencyInjection;
using NSubstitute;
using ZB.MOM.WW.ScadaBridge.CentralUI.ScriptAnalysis;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Security;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services;
using ZB.MOM.WW.ScadaBridge.Security;
using ZB.MOM.WW.ScadaBridge.TemplateEngine;
using ApiMethodFormPage = ZB.MOM.WW.ScadaBridge.CentralUI.Components.Pages.Design.ApiMethodForm;
using SharedScriptFormPage = ZB.MOM.WW.ScadaBridge.CentralUI.Components.Pages.Design.SharedScriptForm;
namespace ZB.MOM.WW.ScadaBridge.CentralUI.Tests.Design;
/// <summary>
/// Regression tests for arch-review P6-12. <c>ApiMethodForm</c> and
/// <c>SharedScriptForm</c> run a Test Run sandbox script via
/// <c>AnalysisService.RunInSandboxAsync(request, _runCts.Token)</c> but — unlike
/// <c>TemplateScriptDialog</c>, which cancels/disposes <c>_runCts</c> from an
/// <see cref="IDisposable"/> <c>Dispose()</c> — neither form implemented
/// <see cref="IDisposable"/> at all. An abandoned run's <c>_runCts</c> was never
/// cancelled, so the sandbox kept the run alive until its own internal script
/// timeout; when it finally completed, the continuation touched a torn-down
/// component's <c>_running</c>/<c>_runResult</c> fields. The fix mirrors
/// <c>TemplateScriptDialog</c> exactly: <c>@implements IDisposable</c> plus a
/// <c>Dispose()</c> that cancels + disposes <c>_runCts</c>.
/// <para>
/// <see cref="GatedScriptAnalysisService"/> holds a run open on a
/// <see cref="TaskCompletionSource{T}"/> gate so the dispose-mid-run window is
/// deterministic (never <c>Task.Delay</c> ordering) — same pattern as
/// <c>DebugViewConnectDisposeRaceTests.GatedDebugStreamService</c>. Overriding
/// requires <c>ScriptAnalysisService.RunInSandboxAsync</c> to be <c>virtual</c> —
/// a test-only seam added alongside this fix (mirroring the pre-existing
/// <c>virtual</c> on <c>DebugStreamService.StartStreamAsync</c>/<c>StopStream</c>),
/// not itself part of the P6-12 defect or its fix.
/// </para>
/// <para>
/// <b>Proof of failure:</b> stashing only the two <c>.razor</c> changes (leaving
/// the <c>virtual</c> seam in place) and re-running
/// <see cref="ApiMethodFormSandboxDisposalTests.Dispose_MidRun_CancelsTheSandboxRunToken"/>
/// and
/// <see cref="SharedScriptFormSandboxDisposalTests.Dispose_MidRun_CancelsTheSandboxRunToken"/>
/// fails both: with no <see cref="IDisposable"/> on the form, <c>(cut.Instance as
/// IDisposable)</c> is <c>null</c>, so the disposal no-ops and the token the
/// sandbox observed is never cancelled.
/// </para>
/// </summary>
public class ApiMethodFormSandboxDisposalTests : BunitContext
{
private (IRenderedComponent<ApiMethodFormPage> Cut, GatedScriptAnalysisService Sandbox) RenderPage()
{
// ApiMethodForm mounts a MonacoEditor unconditionally; let bUnit answer its
// JS interop calls loosely (same convention as DebugView's disposal tests).
JSInterop.Mode = JSRuntimeMode.Loose;
var repo = Substitute.For<IInboundApiRepository>();
Services.AddSingleton(repo);
var keyAdmin = Substitute.For<IInboundApiKeyAdmin>();
keyAdmin.ListAsync(Arg.Any<CancellationToken>())
.Returns(Array.Empty<InboundApiKeyInfo>());
Services.AddSingleton(keyAdmin);
var sandbox = new GatedScriptAnalysisService();
Services.AddSingleton<ScriptAnalysisService>(sandbox);
var claims = new[]
{
new Claim(JwtTokenService.UsernameClaimType, "tester"),
new Claim(JwtTokenService.RoleClaimType, "Designer"),
};
var user = new ClaimsPrincipal(new ClaimsIdentity(claims, "TestAuth"));
Services.AddSingleton<AuthenticationStateProvider>(new TestAuthStateProvider(user));
Services.AddAuthorizationCore();
return (Render<ApiMethodFormPage>(), sandbox);
}
/// <summary>
/// Starts <c>RunInSandboxAsync()</c> on the renderer's dispatcher (where it really
/// runs) and hands back its task without awaiting it, so the test can act inside the
/// run window.
/// </summary>
private static async Task<Task> StartRunAsync(IRenderedComponent<ApiMethodFormPage> cut)
{
var run = typeof(ApiMethodFormPage).GetMethod(
"RunInSandboxAsync", BindingFlags.Instance | BindingFlags.NonPublic)!;
Task? task = null;
await cut.InvokeAsync(() => { task = (Task)run.Invoke(cut.Instance, null)!; });
return task!;
}
private static object? GetField(ApiMethodFormPage page, string name) =>
typeof(ApiMethodFormPage).GetField(name, BindingFlags.Instance | BindingFlags.NonPublic)!
.GetValue(page);
[Fact]
public async Task Dispose_MidRun_CancelsTheSandboxRunToken()
{
var (cut, sandbox) = RenderPage();
var run = await StartRunAsync(cut);
await sandbox.Entered.WaitAsync(TimeSpan.FromSeconds(5));
Assert.False(sandbox.LastToken.IsCancellationRequested,
"the run must still be in flight, unabandoned, before dispose");
// The user navigates away while the run is still in flight. Dispose runs off
// the renderer's thread, exactly as a circuit teardown can.
(cut.Instance as IDisposable)?.Dispose();
Assert.True(sandbox.LastToken.IsCancellationRequested,
"Dispose() must cancel _runCts so the sandbox stops the abandoned run " +
"instead of parking it until the script timeout");
Assert.Null(GetField(cut.Instance, "_runCts"));
// Unblock the gated run so nothing is left hanging past this test.
sandbox.ReleaseRun();
await run.WaitAsync(TimeSpan.FromSeconds(5));
}
[Fact]
public async Task Dispose_MidRun_ContinuationIsANoOp()
{
var (cut, sandbox) = RenderPage();
var run = await StartRunAsync(cut);
await sandbox.Entered.WaitAsync(TimeSpan.FromSeconds(5));
(cut.Instance as IDisposable)?.Dispose();
try
{
// The cancelled run's continuation must resolve on its own (no
// Task.Delay-based waiting) and must not surface an unhandled exception —
// RunInSandboxAsync's own catch(OperationCanceledException) swallows it.
var ex = await Record.ExceptionAsync(() => run.WaitAsync(TimeSpan.FromSeconds(5)));
Assert.Null(ex);
// No state mutation: the disposed run must not have been (mis)recorded as
// an "Unexpected: ..." runtime error — it stays exactly as it started.
Assert.Null(GetField(cut.Instance, "_runResult"));
Assert.False((bool)GetField(cut.Instance, "_running")!);
}
finally
{
// Safety net so an unfixed run (never cancelled) doesn't hang past this test.
sandbox.ReleaseRun();
}
}
}
public class SharedScriptFormSandboxDisposalTests : BunitContext
{
private (IRenderedComponent<SharedScriptFormPage> Cut, GatedScriptAnalysisService Sandbox) RenderPage()
{
// SharedScriptForm mounts a MonacoEditor unconditionally; let bUnit answer its
// JS interop calls loosely (same convention as DebugView's disposal tests).
JSInterop.Mode = JSRuntimeMode.Loose;
var repo = Substitute.For<ITemplateEngineRepository>();
var audit = Substitute.For<IAuditService>();
Services.AddSingleton(repo);
Services.AddSingleton(audit);
Services.AddScoped<SharedScriptService>();
var sandbox = new GatedScriptAnalysisService();
Services.AddSingleton<ScriptAnalysisService>(sandbox);
var claims = new[]
{
new Claim(JwtTokenService.UsernameClaimType, "tester"),
new Claim(JwtTokenService.RoleClaimType, "Designer"),
};
var user = new ClaimsPrincipal(new ClaimsIdentity(claims, "TestAuth"));
Services.AddSingleton<AuthenticationStateProvider>(new TestAuthStateProvider(user));
Services.AddAuthorizationCore();
return (Render<SharedScriptFormPage>(), sandbox);
}
private static async Task<Task> StartRunAsync(IRenderedComponent<SharedScriptFormPage> cut)
{
var run = typeof(SharedScriptFormPage).GetMethod(
"RunInSandboxAsync", BindingFlags.Instance | BindingFlags.NonPublic)!;
Task? task = null;
await cut.InvokeAsync(() => { task = (Task)run.Invoke(cut.Instance, null)!; });
return task!;
}
private static object? GetField(SharedScriptFormPage page, string name) =>
typeof(SharedScriptFormPage).GetField(name, BindingFlags.Instance | BindingFlags.NonPublic)!
.GetValue(page);
[Fact]
public async Task Dispose_MidRun_CancelsTheSandboxRunToken()
{
var (cut, sandbox) = RenderPage();
var run = await StartRunAsync(cut);
await sandbox.Entered.WaitAsync(TimeSpan.FromSeconds(5));
Assert.False(sandbox.LastToken.IsCancellationRequested,
"the run must still be in flight, unabandoned, before dispose");
(cut.Instance as IDisposable)?.Dispose();
Assert.True(sandbox.LastToken.IsCancellationRequested,
"Dispose() must cancel _runCts so the sandbox stops the abandoned run " +
"instead of parking it until the script timeout");
Assert.Null(GetField(cut.Instance, "_runCts"));
sandbox.ReleaseRun();
await run.WaitAsync(TimeSpan.FromSeconds(5));
}
[Fact]
public async Task Dispose_MidRun_ContinuationIsANoOp()
{
var (cut, sandbox) = RenderPage();
var run = await StartRunAsync(cut);
await sandbox.Entered.WaitAsync(TimeSpan.FromSeconds(5));
(cut.Instance as IDisposable)?.Dispose();
try
{
var ex = await Record.ExceptionAsync(() => run.WaitAsync(TimeSpan.FromSeconds(5)));
Assert.Null(ex);
Assert.Null(GetField(cut.Instance, "_runResult"));
Assert.False((bool)GetField(cut.Instance, "_running")!);
}
finally
{
sandbox.ReleaseRun();
}
}
}
/// <summary>
/// A <see cref="ScriptAnalysisService"/> whose sandbox run can be held open on demand
/// via a <see cref="TaskCompletionSource{T}"/> gate, so the dispose-mid-run window is
/// deterministic across both script forms' tests. Overriding
/// <see cref="ScriptAnalysisService.RunInSandboxAsync"/> requires it to be
/// <c>virtual</c> — see the class-level remarks on
/// <see cref="ApiMethodFormSandboxDisposalTests"/>.
/// </summary>
internal sealed class GatedScriptAnalysisService : ScriptAnalysisService
{
private readonly TaskCompletionSource<SandboxRunResult> _gate =
new(TaskCreationOptions.RunContinuationsAsynchronously);
private readonly TaskCompletionSource _entered =
new(TaskCreationOptions.RunContinuationsAsynchronously);
public GatedScriptAnalysisService()
: base(
Substitute.For<ISharedScriptCatalog>(),
new MemoryCache(new MemoryCacheOptions()),
new ServiceCollection().BuildServiceProvider())
{
}
/// <summary>Completes once <see cref="RunInSandboxAsync"/> has been entered.</summary>
public Task Entered => _entered.Task;
/// <summary>The cancellation token supplied to the most recent call.</summary>
public CancellationToken LastToken { get; private set; }
/// <summary>Completes the held-open run with a successful, inert result.</summary>
public void ReleaseRun() => _gate.TrySetResult(
new SandboxRunResult(true, "1", "int", "", null, SandboxErrorKind.None, 1, null));
public override async Task<SandboxRunResult> RunInSandboxAsync(SandboxRunRequest request, CancellationToken ct)
{
LastToken = ct;
_entered.TrySetResult();
using var reg = ct.Register(() => _gate.TrySetCanceled(ct));
return await _gate.Task;
}
}
tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Pages/Design/TransportImportSessionDisposeRaceTests.cs
+227
View File
@@ -0,0 +1,227 @@
using System.Reflection;
using System.Security.Claims;
using Bunit;
using Microsoft.AspNetCore.Components.Authorization;
using Microsoft.EntityFrameworkCore;
using Microsoft.EntityFrameworkCore.Diagnostics;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Options;
using ZB.MOM.WW.ScadaBridge.CentralUI.Auth;
using ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Transport;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Transport;
using ZB.MOM.WW.ScadaBridge.ConfigurationDatabase;
using ZB.MOM.WW.ScadaBridge.Security;
using ZB.MOM.WW.ScadaBridge.Transport;
using ZB.MOM.WW.ScadaBridge.Transport.Import;
using NSubstitute;
using TransportImportPage = ZB.MOM.WW.ScadaBridge.CentralUI.Components.Pages.Design.TransportImport;
namespace ZB.MOM.WW.ScadaBridge.CentralUI.Tests.Pages.Design;
/// <summary>
/// Regression tests for register row 103 (the P6-6 lesser instance in
/// <c>TransportImport</c>). <c>TryLoadAsync</c> awaits
/// <see cref="IBundleImporter.LoadAsync"/>, then used to publish the result into
/// <c>_session</c> unconditionally. If the circuit was torn down inside that await
/// (user navigates away mid-upload), <c>Dispose</c> would have already run, found
/// <c>_session</c> still null, and released nothing — so the freshly-opened server-side
/// <c>BundleSession</c> stayed registered, stranding one of
/// <c>TransportOptions.MaxConcurrentImportSessions</c> (register row 43) for the full
/// session TTL. The fix publishes the session and re-checks disposal under one lock
/// (<c>_sessionLock</c>), mirroring <c>DebugView</c>'s P6-6 fix, so exactly one side
/// (this method or <c>Dispose</c>) abandons it.
/// </summary>
public class TransportImportSessionDisposeRaceTests : BunitContext
{
/// <summary>
/// A real <see cref="BundleSessionStore"/>-backed <see cref="IBundleImporter"/> whose
/// <see cref="LoadAsync"/> can be held open on demand, so the dispose-during-load
/// window is deterministic rather than timing-dependent. Mirrors
/// <c>DebugViewConnectDisposeRaceTests.GatedDebugStreamService</c>.
/// </summary>
private sealed class GatedImporter : IBundleImporter
{
private readonly TaskCompletionSource _gate = new(TaskCreationOptions.RunContinuationsAsynchronously);
private readonly TaskCompletionSource _entered = new(TaskCreationOptions.RunContinuationsAsynchronously);
private readonly List<Guid> _abandoned = new();
public BundleSessionStore Store { get; } = new(
Options.Create(new TransportOptions()), TimeProvider.System);
/// <summary>Completes once <see cref="LoadAsync"/> has been entered.</summary>
public Task Entered => _entered.Task;
/// <summary>Session ids passed to <see cref="Abandon"/>, in call order.</summary>
public IReadOnlyList<Guid> Abandoned
{
get { lock (_abandoned) { return _abandoned.ToList(); } }
}
public void ReleaseLoad() => _gate.TrySetResult();
public async Task<BundleSession> LoadAsync(
Stream bundleStream, string? passphrase, CancellationToken ct = default, string? clientKey = null)
{
_entered.TrySetResult();
await _gate.Task;
var session = new BundleSession
{
SessionId = Guid.NewGuid(),
Manifest = new BundleManifest(
BundleFormatVersion: 1,
SchemaVersion: "1.0",
CreatedAtUtc: DateTimeOffset.UtcNow,
SourceEnvironment: "prod-cluster",
ExportedBy: "bob",
ScadaBridgeVersion: "1.0.0",
ContentHash: "sha256:0000",
Encryption: null,
Summary: new BundleSummary(0, 0, 0, 0, 0, 0, 0, 0),
Contents: Array.Empty<ManifestContentEntry>()),
// Non-zero bytes so a zeroing assertion (if ever added) is meaningful.
DecryptedContent = new byte[] { 0x11, 0x22, 0x33, 0x44 },
ExpiresAt = DateTimeOffset.UtcNow.AddMinutes(30),
};
return Store.Open(session);
}
public Task<ImportPreview> PreviewAsync(Guid sessionId, CancellationToken ct = default)
=> throw new NotSupportedException("Not exercised by these tests.");
public Task<ImportResult> ApplyAsync(
Guid sessionId,
IReadOnlyList<ImportResolution> resolutions,
string user,
CancellationToken ct = default,
BundleNameMap? nameMap = null)
=> throw new NotSupportedException("Not exercised by these tests.");
public void Abandon(Guid sessionId)
{
lock (_abandoned) { _abandoned.Add(sessionId); }
var session = Store.Get(sessionId);
if (session is { DecryptedContent.Length: > 0 })
{
Array.Clear(session.DecryptedContent, 0, session.DecryptedContent.Length);
}
Store.Remove(sessionId);
}
}
private readonly GatedImporter _importer = new();
private readonly ISiteRepository _siteRepo = Substitute.For<ISiteRepository>();
private const string TestClientIp = "203.0.113.7";
public TransportImportSessionDisposeRaceTests()
{
JSInterop.Mode = JSRuntimeMode.Loose;
Services.AddSingleton<IBundleImporter>(_importer);
Services.AddSingleton(Substitute.For<IAuditService>());
Services.AddSingleton(new CircuitClientAddressProvider(TestClientIp));
_siteRepo.GetAllSitesAsync(Arg.Any<CancellationToken>()).Returns(Array.Empty<Site>());
Services.AddSingleton(_siteRepo);
Services.AddSingleton<IOptions<TransportOptions>>(
Options.Create(new TransportOptions
{
MaxBundleSizeMb = 10,
MaxUnlockAttemptsPerSession = 3,
}));
var dbOptions = new DbContextOptionsBuilder<ScadaBridgeDbContext>()
.UseSqlite("DataSource=:memory:")
.ConfigureWarnings(w => w.Ignore(RelationalEventId.AmbientTransactionWarning))
.Options;
var dbContext = new ScadaBridgeDbContext(dbOptions);
dbContext.Database.OpenConnection();
dbContext.Database.EnsureCreated();
Services.AddSingleton(dbContext);
var claims = new List<Claim> { new(JwtTokenService.UsernameClaimType, "alice") };
var principal = new ClaimsPrincipal(new ClaimsIdentity(claims, "TestAuth"));
Services.AddSingleton<AuthenticationStateProvider>(new TestAuthStateProvider(principal));
Services.AddAuthorizationCore();
}
private static void SetField(object obj, string name, object? value) =>
obj.GetType().GetField(name, BindingFlags.Instance | BindingFlags.NonPublic | BindingFlags.Public)!
.SetValue(obj, value);
private static T GetField<T>(object obj, string name) =>
(T)obj.GetType().GetField(name, BindingFlags.Instance | BindingFlags.NonPublic | BindingFlags.Public)!
.GetValue(obj)!;
/// <summary>Stages a temp bundle file and points the wizard's <c>_bundleTempPath</c> at it.</summary>
private static string StageBundleFile()
{
var dir = Path.Combine(Path.GetTempPath(), "scadabridge-transport-staging");
Directory.CreateDirectory(dir);
var path = Path.Combine(dir, $"test-{Guid.NewGuid():N}.scadabundle");
File.WriteAllBytes(path, new byte[] { 0x01, 0x02, 0x03 });
return path;
}
/// <summary>
/// Starts <c>TryLoadAsync</c> on the renderer's dispatcher (where it really runs) and
/// hands back its task without awaiting it, so the test can act inside the load window.
/// Mirrors <c>DebugViewConnectDisposeRaceTests.StartConnectAsync</c>.
/// </summary>
private static async Task<Task> StartTryLoadAsync(IRenderedComponent<TransportImportPage> cut)
{
var method = typeof(TransportImportPage).GetMethod(
"TryLoadAsync", BindingFlags.Instance | BindingFlags.NonPublic)!;
Task? task = null;
await cut.InvokeAsync(() => { task = (Task)method.Invoke(cut.Instance, new object?[] { null })!; });
return task!;
}
[Fact]
public async Task TryLoadAsync_DisposedMidLoad_AbandonsTheJustOpenedSession()
{
var cut = Render<TransportImportPage>();
SetField(cut.Instance, "_bundleTempPath", StageBundleFile());
var tryLoad = await StartTryLoadAsync(cut);
await _importer.Entered.WaitAsync(TimeSpan.FromSeconds(5));
// The user navigates away while LoadAsync is still in flight. Dispose runs off
// the renderer's thread, exactly as a circuit teardown can.
cut.Instance.Dispose();
_importer.ReleaseLoad();
await tryLoad.WaitAsync(TimeSpan.FromSeconds(10));
Assert.Single(_importer.Abandoned);
Assert.Null(GetField<object?>(cut.Instance, "_session"));
// The store must not still hold the session either — Abandon actually released it.
Assert.Null(_importer.Store.Get(_importer.Abandoned[0]));
}
[Fact]
public async Task TryLoadAsync_ThenDispose_AbandonsTheSessionExactlyOnce()
{
var cut = Render<TransportImportPage>();
SetField(cut.Instance, "_bundleTempPath", StageBundleFile());
// Normal path: the load completes while the component is very much alive.
_importer.ReleaseLoad();
var tryLoad = await StartTryLoadAsync(cut);
await tryLoad.WaitAsync(TimeSpan.FromSeconds(10));
Assert.Empty(_importer.Abandoned);
Assert.NotNull(GetField<object?>(cut.Instance, "_session"));
cut.Instance.Dispose();
Assert.Single(_importer.Abandoned);
// A second Dispose (bUnit tears the component down again at test end) must not
// abandon an already-released session a second time.
cut.Instance.Dispose();
Assert.Single(_importer.Abandoned);
}
}
tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/TopologyLiveUpdatesTimerDisposeRaceTests.cs
+196
View File
@@ -0,0 +1,196 @@
using System.Reflection;
using System.Security.Claims;
using Bunit;
using Microsoft.AspNetCore.Components.Authorization;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Options;
using NSubstitute;
using ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances;
using ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites;
using ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services;
using ZB.MOM.WW.ScadaBridge.Commons.Entities.Deployment;
using ZB.MOM.WW.ScadaBridge.Communication;
using ZB.MOM.WW.ScadaBridge.DeploymentManager;
using ZB.MOM.WW.ScadaBridge.Security;
using ZB.MOM.WW.ScadaBridge.TemplateEngine.Services;
using TopologyPage = ZB.MOM.WW.ScadaBridge.CentralUI.Components.Pages.Deployment.Topology;
namespace ZB.MOM.WW.ScadaBridge.CentralUI.Tests;
/// <summary>
/// Regression tests for register row 103 (the P6-6 lesser instance in
/// <c>Topology</c>). <c>OnInitializedAsync</c> awaits <c>LoadDataAsync</c> and only then
/// calls <c>StartLiveUpdatesTimer</c>. If the circuit was torn down inside that await
/// (user navigates away during the initial page load), the pre-fix <c>Dispose</c> would
/// have already run and found <c>_liveUpdatesTimer</c> still null — a no-op — so the
/// timer <c>StartLiveUpdatesTimer</c> creates once the await resumes is never disposed:
/// it ticks against a torn-down component every 15s until process restart. The fix
/// (mirroring <c>DebugView</c>'s P6-6 fix and <c>TransportImport</c>'s row-103 sibling
/// fix) creates/replaces the timer and observes disposal atomically under
/// <c>_timerLock</c>, so a (re)start that resumes after <c>Dispose</c> already ran
/// refuses to create a new timer instead of leaking one.
///
/// <para>
/// The race is reproduced deterministically by driving the exact call ORDER the async
/// gap allows — <c>Dispose()</c> then a late-arriving <c>StartLiveUpdatesTimer()</c> —
/// rather than by gating I/O and sleeping: <c>StartLiveUpdatesTimer</c> itself is
/// synchronous, so once we control ordering there is nothing left to wait on. This is
/// the "deterministic observation, never sleep-then-assert" style the shared plan
/// constraints require: the assertion is a direct field-state check, not a timer tick.
/// </para>
/// </summary>
public class TopologyLiveUpdatesTimerDisposeRaceTests : BunitContext
{
private readonly ITemplateEngineRepository _repo = Substitute.For<ITemplateEngineRepository>();
private readonly ISiteRepository _siteRepo = Substitute.For<ISiteRepository>();
private readonly IDeploymentManagerRepository _deployRepo = Substitute.For<IDeploymentManagerRepository>();
private readonly IAuditService _audit = Substitute.For<IAuditService>();
public TopologyLiveUpdatesTimerDisposeRaceTests()
{
Services.AddSingleton(_repo);
Services.AddSingleton(_siteRepo);
Services.AddSingleton(_deployRepo);
Services.AddSingleton(Substitute.For<ZB.MOM.WW.ScadaBridge.DeploymentManager.IFlatteningPipeline>());
// DeploymentService takes IStaleInstanceProbe since the Low-wave P7-6 pre-filter (T13).
Services.AddSingleton(Substitute.For<ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Transport.IStaleInstanceProbe>());
Services.AddSingleton(_audit);
var comms = new CommunicationService(
Options.Create(new CommunicationOptions()),
NullLogger<CommunicationService>.Instance);
Services.AddSingleton(comms);
Services.AddSingleton(new OperationLockManager());
Services.AddSingleton(Options.Create(new DeploymentManagerOptions
{
OperationLockTimeout = TimeSpan.FromSeconds(5)
}));
Services.AddSingleton(Options.Create(new CommunicationOptions()));
Services.AddSingleton<ILogger<DeploymentService>>(NullLogger<DeploymentService>.Instance);
Services.AddScoped<ZB.MOM.WW.ScadaBridge.TemplateEngine.Flattening.DiffService>();
Services.AddScoped<ZB.MOM.WW.ScadaBridge.TemplateEngine.Flattening.RevisionHashService>();
Services.AddSingleton<IDeploymentStatusNotifier>(
new DeploymentStatusNotifier(NullLogger<DeploymentStatusNotifier>.Instance));
Services.AddScoped<DeploymentService>();
Services.AddScoped<AreaService>();
Services.AddScoped<InstanceService>();
AddTestAuth();
Services.AddScoped<ZB.MOM.WW.ScadaBridge.CentralUI.Components.Shared.IDialogService,
ZB.MOM.WW.ScadaBridge.CentralUI.Components.Shared.DialogService>();
Services.AddScoped<ZB.MOM.WW.ScadaBridge.CentralUI.Auth.SiteScopeService>();
JSInterop.Setup<string?>("treeviewStorage.load", _ => true).SetResult(null);
JSInterop.SetupVoid("treeviewStorage.save", _ => true);
SeedEmptyRepos();
}
private void AddTestAuth()
{
var claims = new[]
{
new Claim(JwtTokenService.UsernameClaimType, "tester"),
new Claim(JwtTokenService.RoleClaimType, "Deployer")
};
var user = new ClaimsPrincipal(new ClaimsIdentity(claims, "TestAuth"));
Services.AddSingleton<AuthenticationStateProvider>(new TestAuthStateProvider(user));
Services.AddAuthorizationCore();
}
private void SeedEmptyRepos()
{
_siteRepo.GetAllSitesAsync(Arg.Any<CancellationToken>())
.Returns(Task.FromResult<IReadOnlyList<Site>>(new List<Site>()));
_repo.GetAllTemplatesAsync(Arg.Any<CancellationToken>())
.Returns(Task.FromResult<IReadOnlyList<Template>>(new List<Template>()));
_repo.GetAllInstancesAsync(Arg.Any<CancellationToken>())
.Returns(Task.FromResult<IReadOnlyList<Instance>>(new List<Instance>()));
_repo.GetAreasBySiteIdAsync(Arg.Any<int>(), Arg.Any<CancellationToken>())
.Returns(Task.FromResult<IReadOnlyList<Area>>(new List<Area>()));
}
private static void InvokePrivate(object obj, string name) =>
obj.GetType().GetMethod(name, BindingFlags.Instance | BindingFlags.NonPublic)!
.Invoke(obj, null);
private static T? GetField<T>(object obj, string name) =>
(T?)obj.GetType().GetField(name, BindingFlags.Instance | BindingFlags.NonPublic)!.GetValue(obj);
[Fact]
public void Render_StartsALiveTimer_Sanity()
{
// Baseline: the normal init path (fast/empty repos complete synchronously) leaves
// a running timer, so the race tests below start from a known non-null state.
var cut = Render<TopologyPage>();
Assert.NotNull(GetField<Timer?>(cut.Instance, "_liveUpdatesTimer"));
cut.Instance.Dispose();
}
[Fact]
public void StartLiveUpdatesTimer_ResumingAfterDispose_LeavesNoLiveTimer()
{
// Simulates: OnInitializedAsync is suspended awaiting LoadDataAsync; the user
// navigates away, so Dispose() runs (and, since no timer exists yet, has nothing
// to stop); the await then resumes and calls StartLiveUpdatesTimer() — the exact
// shape register row 103 flags for Topology (the P6-6 lesser instance).
var cut = Render<TopologyPage>();
// The normal init path already started a timer; dispose it first so the field is
// null going into the race exactly as it would be if OnInitializedAsync's await
// was still pending when Dispose ran (no timer created yet).
cut.Instance.Dispose();
Assert.Null(GetField<Timer?>(cut.Instance, "_liveUpdatesTimer"));
// The late-arriving continuation's call, happening strictly AFTER Dispose().
InvokePrivate(cut.Instance, "StartLiveUpdatesTimer");
// Post-fix: refuses to create a timer once disposed — nothing is leaked.
// Pre-fix: this assignment is unconditional, so the field would be non-null here
// (a live Timer nothing will ever dispose).
Assert.Null(GetField<Timer?>(cut.Instance, "_liveUpdatesTimer"));
}
[Fact]
public void Dispose_StopsTheLiveTimer_ExactlyOnce()
{
var cut = Render<TopologyPage>();
Assert.NotNull(GetField<Timer?>(cut.Instance, "_liveUpdatesTimer"));
cut.Instance.Dispose();
Assert.Null(GetField<Timer?>(cut.Instance, "_liveUpdatesTimer"));
// A second Dispose (bUnit tears the component down again at test end) must not
// throw (e.g. from double-disposing the same Timer) and must leave the slot null.
var ex = Record.Exception(() => cut.Instance.Dispose());
Assert.Null(ex);
Assert.Null(GetField<Timer?>(cut.Instance, "_liveUpdatesTimer"));
}
[Fact]
public void OnLiveUpdatesToggled_Off_ThenDispose_LeavesNoLiveTimer()
{
var cut = Render<TopologyPage>();
Assert.NotNull(GetField<Timer?>(cut.Instance, "_liveUpdatesTimer"));
// Turn live updates off via the toolbar handler (StopLiveUpdatesTimer path).
var toggleMethod = typeof(TopologyPage).GetMethod(
"OnLiveUpdatesToggled", BindingFlags.Instance | BindingFlags.NonPublic)!;
var changeArgsType = typeof(Microsoft.AspNetCore.Components.ChangeEventArgs);
var changeArgs = (Microsoft.AspNetCore.Components.ChangeEventArgs)Activator.CreateInstance(changeArgsType)!;
changeArgs.Value = false;
toggleMethod.Invoke(cut.Instance, new object?[] { changeArgs });
Assert.Null(GetField<Timer?>(cut.Instance, "_liveUpdatesTimer"));
var ex = Record.Exception(() => cut.Instance.Dispose());
Assert.Null(ex);
Assert.Null(GetField<Timer?>(cut.Instance, "_liveUpdatesTimer"));
}
}
tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/TopologyPageTests.cs
+6
View File
@@ -13,6 +13,7 @@ using ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites;
using ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Transport;
using ZB.MOM.WW.ScadaBridge.Commons.Types;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Flattening;
@@ -74,6 +75,11 @@ public class TopologyPageTests : BunitContext
Services.AddSingleton<ZB.MOM.WW.ScadaBridge.DeploymentManager.IDeploymentStatusNotifier>(
new ZB.MOM.WW.ScadaBridge.DeploymentManager.DeploymentStatusNotifier(
NullLogger<ZB.MOM.WW.ScadaBridge.DeploymentManager.DeploymentStatusNotifier>.Instance));
// P7-6: the bulk site deploy's up-to-date pre-filter asks the staleness
// probe for each instance's current revision hash, so DeploymentService now
// depends on it too. The page under test never bulk-deploys; a substitute
// keeps the DI graph resolvable without dragging in the flattening stack.
Services.AddSingleton(Substitute.For<IStaleInstanceProbe>());
Services.AddScoped<DeploymentService>();
Services.AddScoped<AreaService>();
Services.AddScoped<InstanceService>();
tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/SiteAlarmAggregatorActorTests.cs
+99 -2
View File
@@ -166,7 +166,8 @@ public class SiteAlarmAggregatorActorTests : TestKit
}
private (IActorRef Actor, SeedStub Seed, PublishSink Sink, MockSiteAlarmStreamClientFactory Factory) CreateActor(
TimeSpan? reconcileInterval = null, TimeSpan? publishCoalesce = null, bool autoConnect = true)
TimeSpan? reconcileInterval = null, TimeSpan? publishCoalesce = null, bool autoConnect = true,
TimeSpan? openAttemptTimeout = null)
{
var seed = new SeedStub();
var sink = new PublishSink();
@@ -178,7 +179,11 @@ public class SiteAlarmAggregatorActorTests : TestKit
publishCoalesce ?? TimeSpan.Zero,
TestReconnectDelay,
TestStabilityWindow,
0.0)); // no reconcile jitter in tests — determinism
0.0, // no reconcile jitter in tests — determinism
// Production default (30 s) unless a test overrides it: every other test in this
// class finishes far inside it, so the open-attempt deadline stays dormant and
// their behaviour is unchanged.
openAttemptTimeout ?? SiteAlarmAggregatorActor.DefaultOpenAttemptTimeout));
var actor = Sys.ActorOf(props);
return (actor, seed, sink, factory);
@@ -849,6 +854,98 @@ public class SiteAlarmAggregatorActorTests : TestKit
AwaitCondition(() => seed.CallCount == 2, TimeSpan.FromSeconds(3));
}
// ── P3-4: the stream open attempt is deadline-bounded ───────────────────────
/// <summary>
/// Proof of the P3-4 starvation and of its fix.
/// <para>
/// An open attempt is cleared by exactly three peer-produced outcomes: connected, error,
/// or graceful completion. A stream that produces NONE of them pins <c>_openInFlight</c>.
/// That is reachable: <c>SiteStreamGrpcClient</c> gives up on response headers after
/// <c>ConnectedHeaderTimeout</c> (10 s) and then treats the FIRST RECEIVED EVENT as its
/// fallback connected signal — which an alarm-only site that happens to be quiet never
/// sends (proven independently by
/// <c>SiteStreamGrpcClientTests.ConsumeStream_HeaderTimeoutOnADeadSite_NeverReportsConnected</c>).
/// The aggregator then sits with <c>_streamDown == true</c> AND <c>_openInFlight == true</c>,
/// and <see cref="SiteAlarmAggregatorActor.OnReconcileTick"/>'s reopen guard
/// (<c>_streamDown &amp;&amp; !_openInFlight</c>) is skipped every tick until the site's
/// 4 h max stream lifetime finally ends the call.
/// </para>
/// <para>
/// <b>Honesty note.</b> The mock reproduces the aggregator-side state machine exactly and
/// deterministically (that is what <c>autoConnect: false</c> models). What it does NOT
/// prove is how often the >10 s header delay that gets you there actually happens in
/// production: the site flushes response headers explicitly as soon as its relay is
/// attached (<c>SiteStreamGrpcServer.RunSubscriptionStreamAsync</c>), so the 10 s budget
/// has to be consumed by channel connect + HTTP/2 handshake + server scheduling instead —
/// plausible under a WAN stall or a GC/thread-pool-starved site node, but not something
/// this suite can demonstrate. The deadline is therefore a self-healing guard rather than
/// a fix for a measured production incident.
/// </para>
/// </summary>
[Fact]
public void OpenAttemptThatNeverResolves_ReleasesTheInFlightGuard_SoTheTickCanReopen()
{
// 250 ms deadline; the reconcile timer is parked 10 minutes away so the ONLY tick is
// the one this test issues by hand.
var (actor, seed, sink, factory) = CreateActor(
reconcileInterval: TimeSpan.FromMinutes(10),
autoConnect: false,
openAttemptTimeout: TimeSpan.FromMilliseconds(250));
AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3));
AwaitCondition(() => factory.ClientFor(GrpcNodeA).Subs.Count == 1, TimeSpan.FromSeconds(3));
seed.CompleteNext();
// The stream never reports connected, never faults, never completes — the quiet-site
// header-timeout shape. Liveness is (correctly) false the whole time.
AssertHoldsFor(TimeSpan.FromMilliseconds(300),
(() => !sink.StreamLive, "a stream that never connected must never read as live"));
// Pre-fix this tick found _openInFlight still true and skipped the reopen, leaving
// node A on one subscription forever. Post-fix the deadline has released the flag.
actor.Tell(new RunReconcile());
AwaitCondition(() => factory.ClientFor(GrpcNodeA).Subs.Count == 2, TimeSpan.FromSeconds(5));
// And the reopen is a real one: its connect brings the feed up and earns a re-seed.
factory.ClientFor(GrpcNodeA).Subs[1].OnConnected!();
AwaitCondition(() => seed.CallCount == 2, TimeSpan.FromSeconds(3));
seed.CompleteNext();
AwaitCondition(() => sink.StreamLive, TimeSpan.FromSeconds(3));
}
[Fact]
public void OpenDeadline_DoesNotFire_OnAStreamThatConnects_NorFlipNodeOrBurnBudget()
{
// The deadline must be inert on every healthy path: a connect cancels it, so no
// spurious reopen, no node flip, and no retry-budget spend can be attributed to it.
// Timing-independent in the sense that matters — the assertions below would fail for
// ANY firing of the timer, and the window is 8x the deadline.
var (actor, seed, sink, factory) = CreateActor(
reconcileInterval: TimeSpan.FromMinutes(10),
autoConnect: true,
openAttemptTimeout: TimeSpan.FromMilliseconds(100));
AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3));
AwaitCondition(() => factory.ClientFor(GrpcNodeA).Subs.Count == 1, TimeSpan.FromSeconds(3));
seed.CompleteNext();
AwaitCondition(() => sink.StreamLive, TimeSpan.FromSeconds(3));
AssertHoldsFor(TimeSpan.FromMilliseconds(800),
(() => sink.StreamLive, "a connected stream must stay live across the open deadline"),
(() => factory.ClientFor(GrpcNodeA).Subs.Count == 1, "no second open on node A"),
(() => factory.ClientFor(GrpcNodeB).Subs.Count == 0, "no node flip"),
(() => seed.CallCount == 1, "no extra fan-out"));
// A reconcile tick on a live stream still must not reopen. (First tick is consumed by
// the initial seed's skip flag; the second is the real fan-out.)
actor.Tell(new RunReconcile());
actor.Tell(new RunReconcile());
AwaitCondition(() => seed.CallCount == 2, TimeSpan.FromSeconds(3)); // tick fan-out only
seed.CompleteNext();
Assert.Single(factory.ClientFor(GrpcNodeA).Subs);
}
[Fact]
public void FailedSeed_IsRetried_WithBackoff_BeforeTheNextReconcile()
{
tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/SiteStreamGrpcServerTests.cs
+150
View File
@@ -8,6 +8,7 @@ using Microsoft.Extensions.Logging.Abstractions;
using NSubstitute;
using ZB.MOM.WW.ScadaBridge.Commons.Observability;
using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
using GrpcStatus = Grpc.Core.Status;
namespace ZB.MOM.WW.ScadaBridge.Communication.Tests.Grpc;
@@ -794,6 +795,155 @@ public class SiteStreamGrpcServerTests : TestKit
}
}
// ── P3-5: the concurrency cap must never refuse a REPLACEMENT ───────────────
/// <summary>
/// A <see cref="ServerCallContext"/> whose <see cref="ServerCallContext.CancellationToken"/>
/// read runs a caller-supplied hook. That read is the first statement after the concurrency
/// cap check and before the <c>_activeStreams</c> insert in
/// <c>SiteStreamGrpcServer.RunSubscriptionStreamAsync</c>, so parking there is how a test
/// deterministically drives two subscribes into the cap's documented check-then-act window.
/// </summary>
private sealed class GatedServerCallContext : ServerCallContext
{
private readonly CancellationToken _token;
private readonly Action _onCancellationTokenRead;
private readonly Metadata _trailers = new();
private readonly Dictionary<object, object> _userState = new();
public GatedServerCallContext(CancellationToken token, Action onCancellationTokenRead)
{
_token = token;
_onCancellationTokenRead = onCancellationTokenRead;
}
protected override string MethodCore => "/sitestream.SiteStreamService/SubscribeInstance";
protected override string HostCore => "localhost";
protected override string PeerCore => "ipv4:127.0.0.1:0";
protected override DateTime DeadlineCore => DateTime.MaxValue;
protected override Metadata RequestHeadersCore => Metadata.Empty;
protected override Metadata ResponseTrailersCore => _trailers;
protected override GrpcStatus StatusCore { get; set; }
protected override WriteOptions? WriteOptionsCore { get; set; }
protected override AuthContext AuthContextCore =>
new(null, new Dictionary<string, List<AuthProperty>>());
protected override IDictionary<object, object> UserStateCore => _userState;
protected override CancellationToken CancellationTokenCore
{
get
{
_onCancellationTokenRead();
return _token;
}
}
protected override ContextPropagationToken CreatePropagationTokenCore(
ContextPropagationOptions? options) => throw new NotSupportedException();
protected override Task WriteResponseHeadersAsyncCore(Metadata responseHeaders) =>
Task.CompletedTask;
}
[Fact]
public async Task AtCapOvershoot_DuplicateReconnect_IsNotRefused_AndLeavesExactlyOneLiveStream()
{
// P3-5. The duplicate-replacement path cancels the incumbent; the cap check then had
// to decide whether the REPLACEMENT may take its place. Checking `Count >= max` after
// the removal is right at exactly the cap but wrong above it — and above it is
// reachable, because the cap check is documented check-then-act: two subscribes that
// both clear it before either registers overshoot by one. In that state a routine
// reconnect killed a healthy stream AND was refused with ResourceExhausted, leaving
// the correlation with ZERO streams until its next retry.
//
// The overshoot is produced DETERMINISTICALLY here (no sleeps, no racing threads left
// to luck): both openers park on the CancellationToken read, which sits between the
// cap check and the _activeStreams insert, so both are past the cap with nothing
// registered. Releasing the gate then lands two entries under a cap of one.
var server = CreateServer(maxStreams: 1);
server.SetReady(Sys);
using var gate = new ManualResetEventSlim(false);
using var parked = new CountdownEvent(2);
void Park()
{
parked.Signal();
Assert.True(gate.Wait(TimeSpan.FromSeconds(15)), "gate was never released");
}
using var ctsA = new CancellationTokenSource();
using var ctsB = new CancellationTokenSource();
var streamA = Task.Run(() => server.SubscribeInstance(
MakeRequest("corr-a"),
Substitute.For<IServerStreamWriter<SiteStreamEvent>>(),
new GatedServerCallContext(ctsA.Token, Park)));
var streamB = Task.Run(() => server.SubscribeInstance(
MakeRequest("corr-b"),
Substitute.For<IServerStreamWriter<SiteStreamEvent>>(),
new GatedServerCallContext(ctsB.Token, Park)));
Assert.True(parked.Wait(TimeSpan.FromSeconds(15)), "both openers should reach the token read");
gate.Set();
// Both registered: the cap of 1 is overshot to 2, exactly as its comment allows.
await WaitForConditionAsync(() => server.ActiveStreamCount == 2);
// Now "corr-a" reconnects. This is a REPLACEMENT — net-neutral on the stream count —
// so it must be accepted. Pre-fix it threw ResourceExhausted *after* cancelling the
// incumbent, so "corr-a" was left with no stream at all.
using var ctsA2 = new CancellationTokenSource();
var streamA2 = Task.Run(() => server.SubscribeInstance(
MakeRequest("corr-a"),
Substitute.For<IServerStreamWriter<SiteStreamEvent>>(),
CreateMockContext(ctsA2.Token)));
// The incumbent unwinds through its normal cancellation path...
await streamA;
Assert.False(streamA2.IsFaulted,
"the replacement must not be refused: " + streamA2.Exception?.GetBaseException().Message);
// ...and exactly one stream is live for "corr-a" (plus the untouched "corr-b").
await WaitForConditionAsync(() => server.ActiveStreamCount == 2);
Assert.False(streamA2.IsCompleted, "the replacement stream should still be running");
ctsA2.Cancel();
ctsB.Cancel();
await streamA2;
await streamB;
await WaitForConditionAsync(() => server.ActiveStreamCount == 0);
}
[Fact]
public async Task AtCap_AGenuinelyNewSubscriber_IsStillRefused_AndTheIncumbentSurvives()
{
// The other half of P3-5: excluding replacements from the cap must NOT weaken the cap
// for a new correlation id, and a refusal must leave the existing streams untouched
// (TryRemove found nothing, so nothing was cancelled before the throw).
var server = CreateServer(maxStreams: 1);
server.SetReady(Sys);
using var cts1 = new CancellationTokenSource();
var stream1 = Task.Run(() => server.SubscribeInstance(
MakeRequest("corr-keep"),
Substitute.For<IServerStreamWriter<SiteStreamEvent>>(),
CreateMockContext(cts1.Token)));
await WaitForConditionAsync(() => server.ActiveStreamCount == 1);
var ex = await Assert.ThrowsAsync<RpcException>(() => server.SubscribeInstance(
MakeRequest("corr-new"),
Substitute.For<IServerStreamWriter<SiteStreamEvent>>(),
CreateMockContext()));
Assert.Equal(StatusCode.ResourceExhausted, ex.StatusCode);
// The incumbent is untouched by the refusal.
Assert.Equal(1, server.ActiveStreamCount);
Assert.False(stream1.IsCompleted, "a refused NEW subscriber must not disturb the incumbent");
cts1.Cancel();
await stream1;
}
/// <summary>Total events carried across all captured frames (unpacking batch frames).</summary>
private static int CountEvents(List<SiteStreamEvent> capture)
{
tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/SiteAlarmLiveCacheServiceTests.cs
+28 -4
View File
@@ -136,6 +136,30 @@ public class SiteAlarmLiveCacheServiceTests : TestKit
return service;
}
/// <summary>
/// Row-40 house pattern for an ABSENCE assertion (mirrors the Medium-wave
/// <c>SiteAlarmAggregatorActorTests.AssertHoldsFor</c> treatment in this same test
/// project): proves every check in <paramref name="checks"/> keeps holding throughout
/// <paramref name="window"/> by polling, rather than a single blind
/// <c>Thread.Sleep(window)</c> followed by one end-of-window sample. A bare
/// sleep-then-assert cannot distinguish "the mechanism correctly stayed quiet" from "the
/// mechanism just hadn't run yet when the one sample was taken", and it can't catch a
/// value that flips wrong and then flips back before that single sample. This fails
/// immediately the first tick any check goes false, and only returns once the full
/// window has elapsed with every check still true.
/// </summary>
private static void AssertHoldsFor(TimeSpan window, params (Func<bool> Holds, string Because)[] checks)
{
var deadline = DateTime.UtcNow + window;
while (true)
{
foreach (var (holds, because) in checks)
Assert.True(holds(), because);
if (DateTime.UtcNow >= deadline) return;
Thread.Sleep(20);
}
}
[Fact]
public void Seed_FanOut_Publishes_The_Instances_That_Failed_To_Answer()
{
@@ -450,8 +474,8 @@ public class SiteAlarmLiveCacheServiceTests : TestKit
sub.Dispose();
AwaitAssert(() => Assert.False(service.IsLive(SiteId)), TimeSpan.FromSeconds(10));
var startsAfterStop = factory.GetOrCreateCount;
Thread.Sleep(300);
Assert.Equal(startsAfterStop, factory.GetOrCreateCount); // no self-heal restart fired
AssertHoldsFor(TimeSpan.FromMilliseconds(300),
(() => factory.GetOrCreateCount == startsAfterStop, "no self-heal restart fired"));
}
// ── WP2.3: IsLive tracks the STREAM, not merely "a snapshot was published once" ──
@@ -468,8 +492,8 @@ public class SiteAlarmLiveCacheServiceTests : TestKit
// the cache is not live — the page must keep polling. Pre-fix IsLive flipped true
// here and the page grafted a never-updating snapshot over fresh poll data.
AwaitCondition(() => service.GetCurrentAlarms(SiteId) is not null, TimeSpan.FromSeconds(5));
Thread.Sleep(400);
Assert.False(service.IsLive(SiteId));
AssertHoldsFor(TimeSpan.FromMilliseconds(400),
(() => !service.IsLive(SiteId), "must stay not-live until the site accepts the stream"));
// The site accepts it → live.
AwaitCondition(() => factory.Client.Connects.Count >= 1, TimeSpan.FromSeconds(5));
tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/RepositoryCoverageTests.cs
+148
View File
@@ -573,6 +573,154 @@ public class NotificationOutboxRepositoryTests : IDisposable
Assert.False(await _repository.UpdateAsync(loaded));
}
[Fact]
public async Task TryUpdateFromParkedAsync_StatusIsParked_UpdatesAndReturnsTrue()
{
var id = Guid.NewGuid().ToString();
_context.Notifications.Add(MakeNotification(id, NotificationStatus.Parked));
await _context.SaveChangesAsync();
_context.ChangeTracker.Clear();
var loaded = await _repository.GetByIdAsync(id);
loaded!.Status = NotificationStatus.Pending;
loaded.RetryCount = 0;
loaded.NextAttemptAt = null;
loaded.LastError = null;
Assert.True(await _repository.TryUpdateFromParkedAsync(loaded));
_context.ChangeTracker.Clear();
var reloaded = await _context.Notifications.FindAsync(id);
Assert.Equal(NotificationStatus.Pending, reloaded!.Status);
}
/// <summary>
/// Direct CAS-miss: the row is currently Delivered (never Parked in this
/// scenario), so the conditional WHERE Status == Parked matches nothing and
/// the row is left completely untouched — arch-review 2026-08-19 P5-4.
/// </summary>
[Fact]
public async Task TryUpdateFromParkedAsync_StatusIsNotParked_ReturnsFalse_AndLeavesRowUnchanged()
{
var id = Guid.NewGuid().ToString();
var deliveredAt = new DateTimeOffset(2026, 5, 19, 9, 0, 0, TimeSpan.Zero);
_context.Notifications.Add(MakeNotification(
id, NotificationStatus.Delivered, deliveredAt: deliveredAt));
await _context.SaveChangesAsync();
_context.ChangeTracker.Clear();
var staleDiscard = MakeNotification(id, NotificationStatus.Discarded);
Assert.False(await _repository.TryUpdateFromParkedAsync(staleDiscard));
_context.ChangeTracker.Clear();
var reloaded = await _context.Notifications.FindAsync(id);
Assert.Equal(NotificationStatus.Delivered, reloaded!.Status);
Assert.Equal(deliveredAt, reloaded.DeliveredAt);
}
[Fact]
public async Task TryUpdateFromParkedAsync_RowPurgedBeforeWrite_ReturnsFalse()
{
var id = Guid.NewGuid().ToString();
_context.Notifications.Add(MakeNotification(id, NotificationStatus.Parked));
await _context.SaveChangesAsync();
_context.ChangeTracker.Clear();
var loaded = await _repository.GetByIdAsync(id);
Assert.NotNull(loaded);
await _context.Notifications.Where(n => n.NotificationId == id).ExecuteDeleteAsync();
_context.ChangeTracker.Clear();
loaded!.Status = NotificationStatus.Discarded;
Assert.False(await _repository.TryUpdateFromParkedAsync(loaded));
}
/// <summary>
/// Deterministic (non-sleep) proof of the arch-review 2026-08-19 P5-4 race:
/// the operator reads a Parked row, but before its write lands the
/// dispatcher (or a concurrent second operator action) has already moved
/// the row to Delivered. <see cref="TryUpdateFromParkedAsync"/>'s CAS must
/// refuse the stale write and leave the Delivered row exactly as it is.
/// The companion fact below proves the general-purpose
/// <see cref="NotificationOutboxRepository.UpdateAsync"/> — the method the
/// operator path used PRE-FIX — has no such guard, which is the mechanism
/// the finding describes.
/// </summary>
[Fact]
public async Task TryUpdateFromParkedAsync_ConcurrentlyDelivered_ReturnsFalse_AndDoesNotOverwrite()
{
var id = Guid.NewGuid().ToString();
_context.Notifications.Add(MakeNotification(id, NotificationStatus.Parked));
await _context.SaveChangesAsync();
_context.ChangeTracker.Clear();
// The operator's read (this snapshot is carried forward stale below).
var operatorRead = await _repository.GetByIdAsync(id);
Assert.NotNull(operatorRead);
// The dispatcher's own write lands first, in the window between the
// operator's read and write.
var deliveredAt = new DateTimeOffset(2026, 5, 19, 9, 0, 0, TimeSpan.Zero);
var dispatcherRead = await _repository.GetByIdAsync(id);
dispatcherRead!.Status = NotificationStatus.Delivered;
dispatcherRead.DeliveredAt = deliveredAt;
Assert.True(await _repository.UpdateAsync(dispatcherRead));
_context.ChangeTracker.Clear();
// The operator's stale in-memory copy still says Parked; it attempts a Discard.
operatorRead!.Status = NotificationStatus.Discarded;
Assert.False(await _repository.TryUpdateFromParkedAsync(operatorRead));
_context.ChangeTracker.Clear();
var reloaded = await _context.Notifications.FindAsync(id);
Assert.Equal(NotificationStatus.Delivered, reloaded!.Status);
Assert.Equal(deliveredAt, reloaded.DeliveredAt);
}
/// <summary>
/// Companion to <see cref="TryUpdateFromParkedAsync_ConcurrentlyDelivered_ReturnsFalse_AndDoesNotOverwrite"/>:
/// runs the IDENTICAL interleaving through the unconditional
/// <see cref="NotificationOutboxRepository.UpdateAsync"/> instead — the
/// method the operator Retry/Discard path called pre-fix — and shows it
/// silently clobbers the Delivered row. This documents the exact defect
/// arch-review 2026-08-19 P5-4 fixed by routing the operator path through
/// <see cref="TryUpdateFromParkedAsync"/>: UpdateAsync itself is unchanged
/// (the dispatcher still needs an unconditional write), so this scenario
/// remains "true" by design — it is the operator actor's call-site choice
/// that was the bug, proven fixed by the actor-level tests in
/// NotificationOutbox.Tests (Discard_ConcurrentlyDelivered_...).
/// </summary>
[Fact]
public async Task UpdateAsync_Unconditional_WouldOverwriteConcurrentlyDeliveredRow_DemonstratingP5_4Mechanism()
{
var id = Guid.NewGuid().ToString();
_context.Notifications.Add(MakeNotification(id, NotificationStatus.Parked));
await _context.SaveChangesAsync();
_context.ChangeTracker.Clear();
var operatorRead = await _repository.GetByIdAsync(id);
Assert.NotNull(operatorRead);
var deliveredAt = new DateTimeOffset(2026, 5, 19, 9, 0, 0, TimeSpan.Zero);
var dispatcherRead = await _repository.GetByIdAsync(id);
dispatcherRead!.Status = NotificationStatus.Delivered;
dispatcherRead.DeliveredAt = deliveredAt;
Assert.True(await _repository.UpdateAsync(dispatcherRead));
_context.ChangeTracker.Clear();
operatorRead!.Status = NotificationStatus.Discarded;
// The unconditional UPDATE has no status guard, so it "succeeds" here —
// this is the bug. It clobbers the Delivered row to Discarded and drops
// DeliveredAt, which is exactly the silent-overwrite hazard P5-4 closes
// by moving the operator path to TryUpdateFromParkedAsync.
Assert.True(await _repository.UpdateAsync(operatorRead));
_context.ChangeTracker.Clear();
var reloaded = await _context.Notifications.FindAsync(id);
Assert.Equal(NotificationStatus.Discarded, reloaded!.Status);
Assert.Null(reloaded.DeliveredAt);
}
[Fact]
public async Task QueryAsync_AppliesFilters_OrdersByCreatedAtDescending_AndPaginates()
{
tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/Actors/DataConnectionActorBatchTests.cs
+71 -7
View File
@@ -177,11 +177,73 @@ public class DataConnectionActorBatchTests : TestKit
ExpectMsg<TagValueUpdate>(u => u.Quality == QualityCode.Bad, TimeSpan.FromSeconds(5));
ExpectMsg<SubscribeTagsResponse>(TimeSpan.FromSeconds(5));
// This is a genuine real-time rate assertion (how many retry rounds a backoff
// schedule produces inside a fixed wall-clock window), not a synchronization
// primitive standing in for "wait until X happens" — proving the floor (>= 2
// rounds) is unavoidably tied to actually waiting out the window. What the old
// blind Thread.Sleep(2000) + single end-of-window sample could NOT catch is a
// regression to full-rate (no-backoff) retrying: it would still land somewhere
// in [2, 8] by luck if the window elapsed while under load, or it would blow
// past 8 well before the 2s mark and nobody would know until the final sample.
// AssertRoundsWithinWindow polls throughout and fails immediately the first time
// the ceiling is breached, so a no-backoff regression is caught fast instead of
// being masked by (or racing) the fixed window.
var initialCalls = adapter.SubscribeBatches.Count;
Thread.Sleep(2000);
var rounds = adapter.SubscribeBatches.Count - initialCalls;
AssertRoundsWithinWindow(TimeSpan.FromSeconds(2),
() => adapter.SubscribeBatches.Count - initialCalls,
minInclusive: 2, maxInclusive: 8,
context: "dead-device probe backoff (floor 150ms, ceiling 600ms)");
}
Assert.InRange(rounds, 2, 8);
/// <summary>
/// Row-40 house pattern adapted for a RATE assertion: proves a monotonically-increasing
/// count stays within <c>[minInclusive, maxInclusive]</c> across the whole
/// <paramref name="window"/>, rather than a single blind sleep-then-sample at the end.
/// The upper bound is enforced continuously (fails fast the instant it is exceeded,
/// well before the window elapses); the lower bound can only be confirmed once the
/// full window has elapsed, since proving "at least N happened by wall-clock time T"
/// requires actually waiting until T.
/// </summary>
private static void AssertRoundsWithinWindow(
TimeSpan window, Func<int> currentRounds, int minInclusive, int maxInclusive, string context)
{
var deadline = DateTime.UtcNow + window;
while (true)
{
var rounds = currentRounds();
Assert.True(rounds <= maxInclusive,
$"{context}: {rounds} rounds already fired, exceeding the ceiling of {maxInclusive} " +
"before the window even elapsed — backoff is not slowing retries down.");
if (DateTime.UtcNow >= deadline)
{
Assert.True(rounds >= minInclusive,
$"{context}: only {rounds} rounds fired in the window (need >= {minInclusive}).");
return;
}
Thread.Sleep(20);
}
}
/// <summary>
/// Row-40 house pattern for an ABSENCE assertion (mirrors the Medium-wave
/// <c>SiteAlarmAggregatorActorTests.AssertHoldsFor</c> treatment): proves
/// <paramref name="holds"/> keeps being true for the whole <paramref name="window"/> by
/// polling, rather than a single blind <c>Thread.Sleep(window)</c> followed by one
/// end-of-window sample. A bare sleep-then-assert cannot distinguish "the mechanism
/// correctly stayed quiet" from "the mechanism just hadn't run yet when the one sample
/// was taken", and it can't catch a value that flips wrong and then flips back before
/// that single sample. This fails immediately the first time <paramref name="holds"/>
/// goes false, and only returns once the full window has elapsed with it still true.
/// </summary>
private static void AssertHoldsFor(TimeSpan window, Func<bool> holds, string because)
{
var deadline = DateTime.UtcNow + window;
while (true)
{
Assert.True(holds(), because);
if (DateTime.UtcNow >= deadline) return;
Thread.Sleep(20);
}
}
[Fact]
@@ -203,16 +265,18 @@ public class DataConnectionActorBatchTests : TestKit
for (var i = 0; i < 20; i++)
adapter.ValueCallback!("tag1", new TagValue(i, QualityCode.Good, DateTimeOffset.UtcNow));
Thread.Sleep(400);
Assert.DoesNotContain(_health.ReceivedCalls(), c => c.GetMethodInfo().Name == "UpdateTagQuality");
AssertHoldsFor(TimeSpan.FromMilliseconds(400),
() => !_health.ReceivedCalls().Any(c => c.GetMethodInfo().Name == "UpdateTagQuality"),
"20 unchanged-quality repeats must never push UpdateTagQuality");
// One genuine transition → exactly one coalesced push.
adapter.ValueCallback!("tag1", new TagValue(1, QualityCode.Bad, DateTimeOffset.UtcNow));
AwaitCondition(
() => _health.ReceivedCalls().Count(c => c.GetMethodInfo().Name == "UpdateTagQuality") == 1,
TimeSpan.FromSeconds(3));
Thread.Sleep(300);
Assert.Single(_health.ReceivedCalls(), c => c.GetMethodInfo().Name == "UpdateTagQuality");
AssertHoldsFor(TimeSpan.FromMilliseconds(300),
() => _health.ReceivedCalls().Count(c => c.GetMethodInfo().Name == "UpdateTagQuality") == 1,
"the coalesced push must not repeat");
}
[Fact]
tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/Actors/DataConnectionDisconnectRaceTests.cs
+242
View File
@@ -0,0 +1,242 @@
using Akka.Actor;
using Akka.TestKit.Xunit2;
using NSubstitute;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Protocol;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.DataConnection;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
using ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Actors;
using ZB.MOM.WW.ScadaBridge.HealthMonitoring;
namespace ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests.Actors;
/// <summary>
/// Arch-review 2026-08-19 <b>P2-6</b> — the <c>AdapterDisconnected</c> gap in the
/// Connecting/Reconnecting states, and the generation fence that keeps a swapped-out adapter's
/// disconnect from disturbing its replacement.
///
/// <para>
/// <b>The gap these pin.</b> Only the Connected state handled <c>AdapterDisconnected</c>; in
/// Connecting and Reconnecting it fell through to <c>Unhandled</c>. A connect that succeeds and
/// drops again before its piped <c>ConnectResult</c> is applied therefore delivered the disconnect
/// FIRST, where it was discarded, and the <c>ConnectResult(true)</c> that followed declared the
/// connection up on an adapter that was already dead. Nothing recovers from there: an adapter
/// raises <c>Disconnected</c> at most once per successful connect (<c>OpcUaDataConnection</c>'s
/// once-only <c>_disconnectFired</c> guard, reset only by the next successful <c>ConnectAsync</c>),
/// so on a connection with no subscriptions — nothing to fail at connection level and drive the
/// reconnect machine — the actor sat Connected on a dead adapter for the life of the process.
/// </para>
///
/// <para>
/// The fix must not weaken the Medium-wave generation fencing: a disconnect raised by an adapter
/// that has since been swapped out (failover, or the row-97 in-place reconfigure) must still be
/// dropped, or the endpoint the operator has just corrected AWAY from could knock the healthy
/// replacement back into a reconnect cycle.
/// </para>
///
/// <para>
/// <b>Determinism note.</b> The race is reproduced by ORDERING, not by timing: the adapter's
/// connect is held on a gate, the <c>Disconnected</c> event is raised while the actor sits idle in
/// Connecting, and a <c>GetHealthReport</c> round trip is used as a mailbox barrier proving the
/// disconnect has been processed before the connect result is released. No sleeps.
/// </para>
/// </summary>
[Collection("DataConnectionManagerActor")]
public class DataConnectionDisconnectRaceTests : TestKit
{
private const string FirstEndpoint = "opc.tcp://first-plc:4840";
private const string SecondEndpoint = "opc.tcp://second-plc:4840";
private readonly ISiteHealthCollector _health = Substitute.For<ISiteHealthCollector>();
private readonly IDataConnectionFactory _factory = Substitute.For<IDataConnectionFactory>();
private readonly DataConnectionOptions _options = new()
{
ReconnectInterval = TimeSpan.FromMilliseconds(100),
TagResolutionRetryInterval = TimeSpan.FromMilliseconds(200),
WriteTimeout = TimeSpan.FromSeconds(5),
SeedReadMaxAttempts = 1,
SeedReadRetryDelay = TimeSpan.FromMilliseconds(10)
};
private static Dictionary<string, string> Config(string endpoint) =>
new() { ["Endpoint"] = endpoint };
private IActorRef CreateConnectionActor(string name, IDataConnection adapter) =>
Sys.ActorOf(Props.Create(() =>
new DataConnectionActor(
name, adapter, _options, _health, _factory, "OpcUa",
Config(FirstEndpoint), null, 3)), name);
private static IDataConnection Adapter(Task connect)
{
var adapter = Substitute.For<IDataConnection>();
adapter.ConnectAsync(Arg.Any<IDictionary<string, string>>(), Arg.Any<CancellationToken>())
.Returns(connect);
adapter.Status.Returns(ConnectionHealth.Connected);
adapter.SubscribeAsync(Arg.Any<string>(), Arg.Any<SubscriptionCallback>(), Arg.Any<CancellationToken>())
.Returns("sub-1");
adapter.ReadAsync(Arg.Any<string>(), Arg.Any<CancellationToken>())
.Returns(new ReadResult(false, null, null));
return adapter;
}
private int HealthTransitions(string connectionName, ConnectionHealth health) =>
_health.ReceivedCalls().Count(c =>
c.GetMethodInfo().Name == nameof(ISiteHealthCollector.UpdateConnectionHealth) &&
c.GetArguments() is [string name, ConnectionHealth reported, ..] &&
name == connectionName && reported == health);
/// <summary>
/// Mailbox barrier: the reply to a message sent AFTER <paramref name="actor"/> was told
/// something proves the earlier message has already been processed (Akka mailboxes are FIFO
/// per sender pair, and every state of this actor answers <c>GetHealthReport</c>). Used
/// instead of a settle delay so the ordering the defect depends on is asserted, not waited on.
/// </summary>
private void Barrier(IActorRef actor)
{
var probe = CreateTestProbe();
probe.Send(actor, new DataConnectionActor.GetHealthReport());
probe.ExpectMsg<DataConnectionHealthReport>(TimeSpan.FromSeconds(5));
}
/// <summary>
/// The headline case: the adapter connects and drops again before the actor has applied the
/// connect result. The actor must NOT settle in Connected — the adapter will never raise
/// <c>Disconnected</c> again, so Connected would be terminal — and must retry instead.
///
/// <para><b>Fails before the fix:</b> the disconnect was Unhandled in Connecting, the actor
/// went straight to Connected and never reconnected — exactly one <c>ConnectAsync</c> for the
/// life of the actor.</para>
/// </summary>
[Fact]
public void DisconnectBeforeConnectResult_RetriesInsteadOfGoingConnected()
{
const string name = "disconnect-before-connect-result";
var firstConnect = new TaskCompletionSource();
// Attempt #2 onward never completes, so "the actor did not go Connected on the dead
// session" is a stable assertion rather than a race against the next retry tick.
var laterConnects = new TaskCompletionSource();
var connects = 0;
var adapter = Substitute.For<IDataConnection>();
adapter.ConnectAsync(Arg.Any<IDictionary<string, string>>(), Arg.Any<CancellationToken>())
.Returns(_ => Interlocked.Increment(ref connects) == 1 ? firstConnect.Task : laterConnects.Task);
adapter.Status.Returns(ConnectionHealth.Connected);
adapter.ReadAsync(Arg.Any<string>(), Arg.Any<CancellationToken>())
.Returns(new ReadResult(false, null, null));
var actor = CreateConnectionActor(name, adapter);
// The actor is in Connecting with ConnectAsync in flight on the gate.
AwaitCondition(() => Volatile.Read(ref connects) >= 1,
TimeSpan.FromSeconds(5), TimeSpan.FromMilliseconds(10));
// The link drops. This is the real ordering: the adapter's keep-alive thread raises
// Disconnected while the connect's piped result is still on its way.
adapter.Disconnected += Raise.Event<Action>();
Barrier(actor);
// Only now does the (already obsolete) success land.
firstConnect.SetResult();
// The retry cycle runs, rather than the actor parking Connected on a dead adapter.
AwaitCondition(() => Volatile.Read(ref connects) >= 2,
TimeSpan.FromSeconds(5), TimeSpan.FromMilliseconds(10));
Assert.Equal(0, HealthTransitions(name, ConnectionHealth.Connected));
}
/// <summary>
/// The Reconnecting twin of the case above — the same gap existed there, and a reconnect is
/// where a flapping endpoint spends most of its time.
///
/// <para><b>Fails before the fix:</b> the disconnect was Unhandled in Reconnecting and the
/// stale success re-subscribed and declared the connection up on the dead adapter.</para>
/// </summary>
[Fact]
public void DisconnectBeforeReconnectResult_StaysReconnecting()
{
const string name = "disconnect-before-reconnect-result";
var firstConnect = new TaskCompletionSource();
var secondConnect = new TaskCompletionSource();
// Attempt #3 onward never completes, so the actor parks in Reconnecting and the
// "did not go Connected on the dead session" assertion is stable rather than a race
// against the next retry tick.
var laterConnects = new TaskCompletionSource();
var connects = 0;
var adapter = Substitute.For<IDataConnection>();
adapter.ConnectAsync(Arg.Any<IDictionary<string, string>>(), Arg.Any<CancellationToken>())
.Returns(_ => Interlocked.Increment(ref connects) switch
{
1 => firstConnect.Task,
2 => secondConnect.Task,
_ => laterConnects.Task
});
adapter.Status.Returns(ConnectionHealth.Connected);
adapter.ReadAsync(Arg.Any<string>(), Arg.Any<CancellationToken>())
.Returns(new ReadResult(false, null, null));
var actor = CreateConnectionActor(name, adapter);
firstConnect.SetResult();
AwaitCondition(() => HealthTransitions(name, ConnectionHealth.Connected) >= 1,
TimeSpan.FromSeconds(5), TimeSpan.FromMilliseconds(10));
// Drop while Connected → Reconnecting, and the retry timer launches attempt #2.
adapter.Disconnected += Raise.Event<Action>();
AwaitCondition(() => Volatile.Read(ref connects) >= 2,
TimeSpan.FromSeconds(5), TimeSpan.FromMilliseconds(10));
// The reconnect drops again before its result is applied.
adapter.Disconnected += Raise.Event<Action>();
Barrier(actor);
secondConnect.SetResult();
// The demoted success re-arms the retry timer instead of declaring the link up again.
AwaitCondition(() => Volatile.Read(ref connects) >= 3,
TimeSpan.FromSeconds(5), TimeSpan.FromMilliseconds(10));
Assert.Equal(1, HealthTransitions(name, ConnectionHealth.Connected));
}
/// <summary>
/// The fencing guard rail: a disconnect stamped with a superseded adapter generation must
/// still be dropped. <c>SwapAdapter</c> unhooks the outgoing adapter, but the event can
/// already be running on a background thread when it does, so the message can outlive the
/// swap — and acting on it would take a healthy, freshly reconfigured connection down.
///
/// <para>
/// <b>Proof of failure.</b> This one cannot be shown against the pre-fix tree by stashing:
/// <c>AdapterDisconnected</c> carried no generation at all before this change, so the test
/// would not compile. It was proved by a targeted revert instead — restoring
/// <c>case AdapterDisconnected: HandleDisconnect(); break;</c> in the Connected state while
/// keeping the stamped record — which makes the assertion below fail with 2 Disconnected
/// transitions instead of 1.
/// </para>
/// </summary>
[Fact]
public void StaleGenerationDisconnect_IsIgnored()
{
const string name = "stale-generation-disconnect";
var first = Adapter(Task.CompletedTask);
var second = Adapter(Task.CompletedTask);
_factory.Create("OpcUa", Arg.Is<IDictionary<string, string>>(d => d["Endpoint"] == SecondEndpoint))
.Returns(second);
var actor = CreateConnectionActor(name, first);
AwaitCondition(() => HealthTransitions(name, ConnectionHealth.Connected) >= 1,
TimeSpan.FromSeconds(5), TimeSpan.FromMilliseconds(10));
// In-place reconfigure: swaps the adapter and bumps the generation 0 → 1. The single
// Disconnected transition it publishes on the way through Reconnecting is the baseline.
actor.Tell(new ReconfigureConnectionCommand(name, "OpcUa", Config(SecondEndpoint), null, 3));
AwaitCondition(() => HealthTransitions(name, ConnectionHealth.Connected) >= 2,
TimeSpan.FromSeconds(5), TimeSpan.FromMilliseconds(10));
Assert.Equal(1, HealthTransitions(name, ConnectionHealth.Disconnected));
// The endpoint we have just moved away from finally reports its loss.
actor.Tell(new DataConnectionActor.AdapterDisconnected(0));
Barrier(actor);
Assert.Equal(1, HealthTransitions(name, ConnectionHealth.Disconnected));
Assert.Equal(2, HealthTransitions(name, ConnectionHealth.Connected));
}
}
tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/Actors/DataConnectionProbeSeedTests.cs
+106
View File
@@ -0,0 +1,106 @@
using Akka.Actor;
using Akka.TestKit.Xunit2;
using NSubstitute;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Protocol;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.DataConnection;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
using ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Actors;
using ZB.MOM.WW.ScadaBridge.HealthMonitoring;
namespace ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests.Actors;
/// <summary>
/// Arch-review 2026-08-19 <b>P2-5</b> — the tag-resolution PROBE path did not seed the tags it
/// resolved.
///
/// <para>
/// <b>The gap this pins.</b> Both other subscribe paths read each newly advised tag's current
/// value and deliver it (<c>HandleSubscribe</c> carries the seeds back on <c>SubscribeCompleted</c>;
/// <c>ReSubscribeAll</c> re-seeds after a reconnect), because a fresh advise only produces a value
/// on the source's next CHANGE. <c>HandleRetryTagResolution</c> did not. OPC UA masks that with an
/// initial data-change notification on subscription creation, but an MxGateway advise can
/// legitimately return <c>VT_EMPTY</c> — so a STATIC tag that resolved LATE (device still booting,
/// item created after deployment) had no value path at all and its attribute stayed Uncertain for
/// the life of the connection.
/// </para>
/// </summary>
[Collection("DataConnectionManagerActor")]
public class DataConnectionProbeSeedTests : TestKit
{
private const string Tag = "plant/static/setpoint";
private readonly ISiteHealthCollector _health = Substitute.For<ISiteHealthCollector>();
private readonly IDataConnectionFactory _factory = Substitute.For<IDataConnectionFactory>();
private readonly DataConnectionOptions _options = new()
{
ReconnectInterval = TimeSpan.FromMilliseconds(100),
TagResolutionRetryInterval = TimeSpan.FromMilliseconds(150),
WriteTimeout = TimeSpan.FromSeconds(5),
SeedReadMaxAttempts = 1,
SeedReadRetryDelay = TimeSpan.FromMilliseconds(10)
};
private IActorRef CreateConnectionActor(string name, IDataConnection adapter) =>
Sys.ActorOf(Props.Create(() =>
new DataConnectionActor(name, adapter, _options, _health, _factory, "OpcUa")), name);
private bool HasEnteredConnectedState(string connectionName) =>
_health.ReceivedCalls().Any(c =>
c.GetMethodInfo().Name == nameof(ISiteHealthCollector.UpdateConnectionHealth) &&
c.GetArguments() is [string name, ConnectionHealth.Connected, ..] &&
name == connectionName);
/// <summary>
/// A tag that fails to resolve on the initial subscribe, resolves on the first probe round,
/// and then never changes again must still reach its subscriber with a Good value.
///
/// <para>
/// The adapter here models exactly the MxGateway shape the finding calls out: the advise
/// succeeds on the retry but the subscription callback is NEVER invoked (a <c>VT_EMPTY</c>
/// advise on a static item), so the seed read is the tag's only possible value source.
/// </para>
///
/// <para><b>Fails before the fix:</b> the probe resolved the tag and stopped there — the
/// subscriber received only the Bad-quality resolution-failure push from the initial
/// subscribe, and this fish for a Good value timed out.</para>
/// </summary>
[Fact]
public void ProbeResolvedTag_IsSeeded_SoAStaticTagLeavesUncertain()
{
const string name = "probe-seed";
var subscribeCalls = 0;
var adapter = Substitute.For<IDataConnection>();
adapter.ConnectAsync(Arg.Any<IDictionary<string, string>>(), Arg.Any<CancellationToken>())
.Returns(Task.CompletedTask);
adapter.Status.Returns(ConnectionHealth.Connected);
adapter.SubscribeAsync(Tag, Arg.Any<SubscriptionCallback>(), Arg.Any<CancellationToken>())
.Returns(_ => Interlocked.Increment(ref subscribeCalls) == 1
// Initial subscribe: the device is still booting — a genuine (non
// connection-level) resolution failure, so the tag lands in _unresolvedTags.
? Task.FromException<string>(new KeyNotFoundException("node not found yet"))
// Probe round: the advise succeeds. The callback is deliberately never invoked.
: Task.FromResult("sub-1"));
adapter.ReadAsync(Tag, Arg.Any<CancellationToken>())
.Returns(new ReadResult(true, new TagValue(42, QualityCode.Good, DateTimeOffset.UtcNow), null));
var actor = CreateConnectionActor(name, adapter);
AwaitCondition(() => HasEnteredConnectedState(name),
TimeSpan.FromSeconds(5), TimeSpan.FromMilliseconds(10));
var subscriber = CreateTestProbe();
subscriber.Send(actor, new SubscribeTagsRequest("c1", "inst1", name, [Tag], DateTimeOffset.UtcNow));
// The initial subscribe fails: Bad-quality push, then the ack.
subscriber.FishForMessage<SubscribeTagsResponse>(_ => true, TimeSpan.FromSeconds(5));
// The probe round resolves the tag; the seed read is the only thing that can now
// deliver a value, because the advise callback never fires for this static item.
var seeded = subscriber.FishForMessage<TagValueUpdate>(
m => m.TagPath == Tag && m.Quality == QualityCode.Good, TimeSpan.FromSeconds(10));
Assert.Equal(42, seeded.Value);
Assert.True(Volatile.Read(ref subscribeCalls) >= 2, "the probe round must have re-subscribed the tag");
}
}
tests/ZB.MOM.WW.ScadaBridge.DeploymentManager.Tests/DeploySiteAsyncTests.cs
+201 -1
View File
@@ -10,6 +10,7 @@ using ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances;
using ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Transport;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Deployment;
using ZB.MOM.WW.ScadaBridge.Commons.Types;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
@@ -34,6 +35,7 @@ public class DeploySiteAsyncTests : TestKit
private readonly IFlatteningPipeline _pipeline = Substitute.For<IFlatteningPipeline>();
private readonly ISiteRepository _siteRepo = Substitute.For<ISiteRepository>();
private readonly IAuditService _audit = Substitute.For<IAuditService>();
private readonly IStaleInstanceProbe _staleProbe = Substitute.For<IStaleInstanceProbe>();
private readonly OperationLockManager _lockManager = new();
public DeploySiteAsyncTests()
@@ -99,6 +101,7 @@ public class DeploySiteAsyncTests : TestKit
new DiffService(),
new RevisionHashService(),
new DeploymentStatusNotifier(NullLogger<DeploymentStatusNotifier>.Instance),
_staleProbe,
options,
Options.Create(new CommunicationOptions
{
@@ -302,7 +305,7 @@ public class DeploySiteAsyncTests : TestKit
var service = CreateService(commActor, maxParallelism: 2);
await Assert.ThrowsAnyAsync<OperationCanceledException>(
() => service.DeploySiteAsync(SiteId, "admin", cts.Token));
() => service.DeploySiteAsync(SiteId, "admin", cancellationToken: cts.Token));
Assert.Equal(0, _lockManager.TrackedLockCount);
@@ -425,6 +428,203 @@ public class DeploySiteAsyncTests : TestKit
Assert.Equal(0, result.Value.FailureCount);
}
/// <summary>
/// Marks an instance as one a bulk deploy must treat as up to date: it is
/// <c>Enabled</c>, central holds a deployed snapshot for it, and the staleness
/// probe recomputes the very hash that snapshot recorded.
/// </summary>
/// <param name="instance">The instance to make up-to-date.</param>
private void ArrangeUpToDate(Instance instance)
{
var hash = $"sha256:{instance.Id}";
instance.State = InstanceState.Enabled;
_repo.GetDeployedSnapshotByInstanceIdAsync(instance.Id, Arg.Any<CancellationToken>())
.Returns(new DeployedConfigSnapshot($"prior-{instance.Id}", hash, "{}") { InstanceId = instance.Id });
_repo.GetCurrentDeploymentStatusAsync(instance.Id, Arg.Any<CancellationToken>())
.Returns(new DeploymentRecord($"prior-{instance.Id}", "admin")
{
InstanceId = instance.Id,
Status = DeploymentStatus.Success,
RevisionHash = hash,
});
_staleProbe.GetCurrentRevisionHashAsync(instance.Id, Arg.Any<CancellationToken>())
.Returns(hash);
}
/// <summary>
/// P7-6: a bulk site deploy must not redeploy instances whose configuration has
/// not changed.
///
/// <para>
/// A redeploy is NOT a no-op at the site: applying a config RESETS that
/// instance's static attribute overrides and CLEARS its native alarm state,
/// both by design. Pushing one site-wide therefore destroyed live operational
/// state on every instance in the batch, including the majority whose config
/// the operator had not touched at all.
/// </para>
///
/// <para>
/// The central-side proof that the site state survives is that the site is
/// never told to apply anything: no <c>RefreshDeploymentCommand</c> is sent
/// for a skipped instance, so nothing at the site runs the reset. The
/// assertions below therefore pin the whole chain — no flatten, no deployment
/// record, no staged config, no command — for the unchanged pair.
/// </para>
/// </summary>
[Fact]
public async Task DeploySiteAsync_SkipsUpToDateInstances_SoTheirSiteStateIsNeverReset()
{
var instances = ArrangeInstances(4);
ArrangeUpToDate(instances.Single(i => i.Id == 2));
ArrangeUpToDate(instances.Single(i => i.Id == 3));
var applied = new ConcurrentBag<string>();
var commActor = Sys.ActorOf(Props.Create(() => new RecordingSiteActor(applied)));
var result = await CreateService(commActor, maxParallelism: 4).DeploySiteAsync(SiteId, "admin");
Assert.True(result.IsSuccess);
var summary = result.Value;
// Only the two genuinely-stale instances were deployed.
Assert.Equal(2, summary.InstanceResults.Count);
Assert.Equal(2, summary.SuccessCount);
Assert.Equal(0, summary.FailureCount);
Assert.Equal(new[] { "Inst-01", "Inst-04" }, summary.InstanceResults.Select(r => r.UniqueName).Order().ToArray());
// The unchanged pair is reported as skipped, with the reason the operator needs.
Assert.NotNull(summary.SkippedInstances);
var skipped = summary.SkippedInstances;
Assert.Equal(new[] { "Inst-02", "Inst-03" }, skipped.Select(s => s.UniqueName).Order().ToArray());
Assert.All(skipped, s => Assert.Contains("up-to-date", s.Reason));
// ...and nothing reached the site for them, which is what leaves their
// static overrides and native alarm state alone.
Assert.Equal(new[] { "Inst-01", "Inst-04" }, applied.Order().ToArray());
foreach (var id in new[] { 2, 3 })
{
await _pipeline.DidNotReceive().FlattenAndValidateAsync(
id, Arg.Any<CancellationToken>(), Arg.Any<bool>(), Arg.Any<FlattenSession?>());
await _repo.DidNotReceive().AddDeploymentRecordAsync(
Arg.Is<DeploymentRecord>(r => r.InstanceId == id), Arg.Any<CancellationToken>());
await _repo.DidNotReceive().AddPendingDeploymentAsync(
Arg.Is<PendingDeployment>(p => p.InstanceId == id), Arg.Any<CancellationToken>());
}
}
/// <summary>
/// P7-6: <c>force</c> restores the unconditional behaviour for operators whose
/// site state central cannot see — a restored data volume or a rebuilt node
/// looks perfectly up to date from central and still needs the push.
/// </summary>
[Fact]
public async Task DeploySiteAsync_Force_RedeploysUpToDateInstances()
{
var instances = ArrangeInstances(3);
ArrangeUpToDate(instances.Single(i => i.Id == 2));
var applied = new ConcurrentBag<string>();
var commActor = Sys.ActorOf(Props.Create(() => new RecordingSiteActor(applied)));
var result = await CreateService(commActor, maxParallelism: 3)
.DeploySiteAsync(SiteId, "admin", force: true);
Assert.True(result.IsSuccess);
Assert.Equal(3, result.Value.SuccessCount);
Assert.Empty(result.Value.SkippedInstances!);
Assert.Contains("Inst-02", applied);
}
/// <summary>
/// P7-6 composed with P7-2: <c>force</c> suppresses the up-to-date filter and
/// ONLY that filter. A Disabled instance is never deployed at bulk scope,
/// because forcing a bulk push is an operator saying "the site's config may be
/// wrong", not "restart the equipment I stopped" — the single-instance deploy
/// remains the explicit re-enable action.
/// </summary>
[Theory]
[InlineData(false)]
[InlineData(true)]
public async Task DeploySiteAsync_DisabledInstancesAreSkipped_RegardlessOfForce(bool force)
{
var instances = ArrangeInstances(3, disabledIds: [2]);
var applied = new ConcurrentBag<string>();
var commActor = Sys.ActorOf(Props.Create(() => new RecordingSiteActor(applied)));
var result = await CreateService(commActor, maxParallelism: 3)
.DeploySiteAsync(SiteId, "admin", force: force);
Assert.True(result.IsSuccess);
Assert.DoesNotContain("Inst-02", applied);
Assert.Equal(InstanceState.Disabled, instances.Single(i => i.Id == 2).State);
var skipped = Assert.Single(result.Value.SkippedInstances!);
Assert.Equal("Inst-02", skipped.UniqueName);
Assert.Contains("deploy individually to re-enable", skipped.Reason);
}
/// <summary>
/// The up-to-date filter is deliberately conservative: every uncertainty
/// resolves to "deploy it". These are the three cases where a matching hash is
/// NOT sufficient — each one would be a silently skipped instance that actually
/// needed the deploy.
/// </summary>
[Fact]
public async Task DeploySiteAsync_MatchingHashIsNotEnough_WhenTheInstanceStillNeedsDeploying()
{
var instances = ArrangeInstances(3);
// 1: NotDeployed. A leftover snapshot says nothing about an instance that
// is not running — it must be deployed to become Enabled.
ArrangeUpToDate(instances.Single(i => i.Id == 1));
instances.Single(i => i.Id == 1).State = InstanceState.NotDeployed;
// 2: prior record stuck InProgress. Prepare is where query-before-redeploy
// reconciliation HEALS such a record; skipping here strands it forever.
ArrangeUpToDate(instances.Single(i => i.Id == 2));
_repo.GetCurrentDeploymentStatusAsync(2, Arg.Any<CancellationToken>())
.Returns(new DeploymentRecord("prior-2", "admin")
{
InstanceId = 2,
Status = DeploymentStatus.InProgress,
RevisionHash = "sha256:2",
});
// 3: staleness indeterminate — the probe cannot compute a hash.
ArrangeUpToDate(instances.Single(i => i.Id == 3));
_staleProbe.GetCurrentRevisionHashAsync(3, Arg.Any<CancellationToken>()).Returns((string?)null);
var applied = new ConcurrentBag<string>();
var commActor = Sys.ActorOf(Props.Create(() => new RecordingSiteActor(applied)));
var result = await CreateService(commActor, maxParallelism: 3).DeploySiteAsync(SiteId, "admin");
Assert.True(result.IsSuccess);
Assert.Empty(result.Value.SkippedInstances!);
Assert.Equal(new[] { "Inst-01", "Inst-02", "Inst-03" }, applied.Order().ToArray());
}
/// <summary>Answers every deploy with Success, recording which instances the site was asked to apply.</summary>
private sealed class RecordingSiteActor : ReceiveActor
{
public RecordingSiteActor(ConcurrentBag<string> applied)
{
Receive<SiteEnvelope>(env =>
{
if (env.Message is not RefreshDeploymentCommand cmd)
return;
applied.Add(cmd.InstanceUniqueName);
Sender.Tell(new DeploymentStatusResponse(
cmd.DeploymentId, cmd.InstanceUniqueName,
DeploymentStatus.Success, null, DateTimeOffset.UtcNow));
});
}
}
/// <summary>
/// Answers <c>RefreshDeploymentCommand</c> normally but NEVER answers a
/// <c>DeploymentStateQueryRequest</c>, counting how many arrived — a site that
tests/ZB.MOM.WW.ScadaBridge.DeploymentManager.Tests/DeploymentServiceTests.cs
+9
View File
@@ -9,6 +9,7 @@ using ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances;
using ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Transport;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Deployment;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Lifecycle;
using ZB.MOM.WW.ScadaBridge.Commons.Observability;
@@ -54,6 +55,7 @@ public class DeploymentServiceTests : TestKit
new DiffService(),
new RevisionHashService(),
new DeploymentStatusNotifier(NullLogger<DeploymentStatusNotifier>.Instance),
Substitute.For<IStaleInstanceProbe>(),
options,
DeployCommOptions(),
NullLogger<DeploymentService>.Instance);
@@ -165,6 +167,7 @@ public class DeploymentServiceTests : TestKit
new DiffService(),
new RevisionHashService(),
new DeploymentStatusNotifier(NullLogger<DeploymentStatusNotifier>.Instance),
Substitute.For<IStaleInstanceProbe>(),
Options.Create(new DeploymentManagerOptions { OperationLockTimeout = TimeSpan.FromSeconds(5) }),
DeployCommOptions(),
NullLogger<DeploymentService>.Instance);
@@ -385,6 +388,7 @@ public class DeploymentServiceTests : TestKit
new DiffService(),
new RevisionHashService(),
new DeploymentStatusNotifier(NullLogger<DeploymentStatusNotifier>.Instance),
Substitute.For<IStaleInstanceProbe>(),
Options.Create(new DeploymentManagerOptions { OperationLockTimeout = TimeSpan.FromSeconds(5) }),
DeployCommOptions(),
NullLogger<DeploymentService>.Instance);
@@ -1153,6 +1157,7 @@ public class DeploymentServiceTests : TestKit
new DiffService(),
new RevisionHashService(),
new DeploymentStatusNotifier(NullLogger<DeploymentStatusNotifier>.Instance),
Substitute.For<IStaleInstanceProbe>(),
Options.Create(new DeploymentManagerOptions { OperationLockTimeout = TimeSpan.FromSeconds(5) }),
DeployCommOptions(centralFetchBaseUrl),
NullLogger<DeploymentService>.Instance);
@@ -1743,6 +1748,7 @@ public class DeploymentServiceTests : TestKit
new DiffService(),
new RevisionHashService(),
new DeploymentStatusNotifier(NullLogger<DeploymentStatusNotifier>.Instance),
Substitute.For<IStaleInstanceProbe>(),
Options.Create(new DeploymentManagerOptions
{
OperationLockTimeout = TimeSpan.FromSeconds(5),
@@ -1801,6 +1807,7 @@ public class DeploymentServiceTests : TestKit
new DiffService(),
new RevisionHashService(),
new DeploymentStatusNotifier(NullLogger<DeploymentStatusNotifier>.Instance),
Substitute.For<IStaleInstanceProbe>(),
Options.Create(new DeploymentManagerOptions
{
OperationLockTimeout = TimeSpan.FromSeconds(5),
@@ -1851,6 +1858,7 @@ public class DeploymentServiceTests : TestKit
new DiffService(),
new RevisionHashService(),
new DeploymentStatusNotifier(NullLogger<DeploymentStatusNotifier>.Instance),
Substitute.For<IStaleInstanceProbe>(),
Options.Create(new DeploymentManagerOptions
{
OperationLockTimeout = TimeSpan.FromSeconds(5),
@@ -1898,6 +1906,7 @@ public class DeploymentServiceTests : TestKit
new DiffService(),
new RevisionHashService(),
new DeploymentStatusNotifier(NullLogger<DeploymentStatusNotifier>.Instance),
Substitute.For<IStaleInstanceProbe>(),
Options.Create(new DeploymentManagerOptions
{
OperationLockTimeout = TimeSpan.FromSeconds(5),
tests/ZB.MOM.WW.ScadaBridge.DeploymentManager.Tests/DeploymentStatusNotifierTests.cs
+3 -1
View File
@@ -7,6 +7,7 @@ using ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances;
using ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Transport;
using ZB.MOM.WW.ScadaBridge.Commons.Types;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Flattening;
@@ -60,7 +61,8 @@ public class DeploymentStatusNotifierTests : TestKit
_service = new DeploymentService(
_repo, siteRepo, _pipeline, _comms, _lockManager, _audit,
new DiffService(), new RevisionHashService(), _notifier, options,
new DiffService(), new RevisionHashService(), _notifier,
Substitute.For<IStaleInstanceProbe>(), options,
// Notify-and-fetch (Task 6): a non-empty CentralFetchBaseUrl lets the
// deploy path reach the send (and, here, the catch block) rather than
// tripping the fail-fast configuration guard.
tests/ZB.MOM.WW.ScadaBridge.Host.Tests/KpiRateFoldIntervalCrossValidatorTests.cs
+92
View File
@@ -0,0 +1,92 @@
using Microsoft.Extensions.Options;
using ZB.MOM.WW.ScadaBridge.KpiHistory;
using ZB.MOM.WW.ScadaBridge.NotificationOutbox;
using ZB.MOM.WW.ScadaBridge.SiteCallAudit;
namespace ZB.MOM.WW.ScadaBridge.Host.Tests;
/// <summary>
/// P5-5 (arch-review Low): <see cref="KpiRateFoldIntervalCrossValidator"/> is the
/// boot-time guardrail for the "rate-fold interval triple" —
/// <see cref="KpiHistoryOptions.SampleInterval"/>,
/// <see cref="NotificationOutboxOptions.DeliveredKpiWindow"/>, and
/// <see cref="SiteCallAuditOptions.KpiInterval"/>. The sample sources now rescale
/// their Rate-classified samples onto <c>SampleInterval</c>, so the hourly rollup
/// fold's correctness no longer depends on this equality — but the
/// recorded/charted sample no longer equals the live dashboard tile unless it
/// holds, so this validator still fails the host at boot on any divergence,
/// naming all three configuration keys.
/// </summary>
public class KpiRateFoldIntervalCrossValidatorTests
{
private static ValidateOptionsResult Validate(
KpiHistoryOptions kpiHistoryOptions,
NotificationOutboxOptions notificationOutboxOptions,
SiteCallAuditOptions siteCallAuditOptions)
{
var validator = new KpiRateFoldIntervalCrossValidator(
Options.Create(notificationOutboxOptions),
Options.Create(siteCallAuditOptions));
return validator.Validate(Options.DefaultName, kpiHistoryOptions);
}
[Fact]
public void AllThreeEqual_AtDocumentedDefaults_IsValid()
{
var result = Validate(
new KpiHistoryOptions(), new NotificationOutboxOptions(), new SiteCallAuditOptions());
Assert.True(result.Succeeded, result.FailureMessage);
}
[Fact]
public void AllThreeEqual_AtANonDefaultCommonValue_IsValid()
{
var common = TimeSpan.FromMinutes(5);
var result = Validate(
new KpiHistoryOptions { SampleInterval = common },
new NotificationOutboxOptions { DeliveredKpiWindow = common },
new SiteCallAuditOptions { KpiInterval = common });
Assert.True(result.Succeeded, result.FailureMessage);
}
[Fact]
public void DeliveredKpiWindowDiverges_FailsAndNamesAllThreeKeys()
{
var result = Validate(
new KpiHistoryOptions { SampleInterval = TimeSpan.FromMinutes(1) },
new NotificationOutboxOptions { DeliveredKpiWindow = TimeSpan.FromMinutes(15) },
new SiteCallAuditOptions { KpiInterval = TimeSpan.FromMinutes(1) });
Assert.True(result.Failed);
Assert.Contains("ScadaBridge:KpiHistory:SampleInterval", result.FailureMessage);
Assert.Contains("ScadaBridge:NotificationOutbox:DeliveredKpiWindow", result.FailureMessage);
Assert.Contains("ScadaBridge:SiteCallAudit:KpiInterval", result.FailureMessage);
}
[Fact]
public void KpiIntervalDiverges_FailsAndNamesAllThreeKeys()
{
var result = Validate(
new KpiHistoryOptions { SampleInterval = TimeSpan.FromMinutes(1) },
new NotificationOutboxOptions { DeliveredKpiWindow = TimeSpan.FromMinutes(1) },
new SiteCallAuditOptions { KpiInterval = TimeSpan.FromMinutes(20) });
Assert.True(result.Failed);
Assert.Contains("ScadaBridge:KpiHistory:SampleInterval", result.FailureMessage);
Assert.Contains("ScadaBridge:NotificationOutbox:DeliveredKpiWindow", result.FailureMessage);
Assert.Contains("ScadaBridge:SiteCallAudit:KpiInterval", result.FailureMessage);
}
[Fact]
public void BothDeliveredKpiWindowAndKpiIntervalDiverge_Fails()
{
var result = Validate(
new KpiHistoryOptions { SampleInterval = TimeSpan.FromMinutes(1) },
new NotificationOutboxOptions { DeliveredKpiWindow = TimeSpan.FromMinutes(15) },
new SiteCallAuditOptions { KpiInterval = TimeSpan.FromMinutes(20) });
Assert.True(result.Failed);
}
}
tests/ZB.MOM.WW.ScadaBridge.Host.Tests/ScadaBridgeTelemetryTests.cs
+41 -2
View File
@@ -15,9 +15,37 @@ public class ScadaBridgeTelemetryTests
Assert.Equal("ZB.MOM.WW.ScadaBridge", ScadaBridgeTelemetry.MeterName);
}
/// <summary>
/// Register row 32 class: <c>SetQueueDepthProvider</c> writes the SAME process-global
/// slot <c>QueueDepthGaugeCollection</c> (StoreAndForward.Tests) exists to serialize
/// access to — this test was mutating it with no clear and no <c>[Collection]</c>,
/// leaving a stray <c>() =&gt; 5</c> provider registered in the process for the rest of
/// this assembly's run. Since the meter/instrument state is a single static field on
/// <c>ScadaBridgeTelemetry</c> (Commons), the leak is scoped to whatever process this
/// assembly's test host runs in — it cannot reach StoreAndForward.Tests (a separate
/// assembly with its own test-host process), but it IS visible within Host.Tests, whose
/// <c>MetricsEndpointTests</c> boots a real in-process host (<c>WebApplicationFactory</c>
/// shares this AppDomain, not a child process) and scrapes <c>/metrics</c>, which
/// enumerates every registered gauge including this one.
///
/// Honesty note on what was and wasn't proven: a search of this assembly (and the whole
/// test tree) found no test today that asserts on the VALUE of
/// <c>scadabridge.store_and_forward.queue.depth</c> from Host.Tests or any other
/// same-process reader — <c>MetricsEndpointTests</c> only asserts the endpoint returns
/// Prometheus-shaped text, not this series' content, and the only test that DOES assert
/// on the value (<c>QueueDepthGaugeTests</c>) lives in StoreAndForward.Tests, a distinct
/// test-host process the leak cannot reach. So there is no currently-reproducible flake
/// to demonstrate here (no fork of this test with the leak left in has an assertion to
/// break) — the defect is a landmine for a FUTURE same-process reader, not a live one.
/// The fix (compare-and-clear in <c>finally</c>) closes the landmine regardless; a
/// <c>[Collection]</c> is intentionally not added since no second same-process reader of
/// this slot exists to serialize against as of this writing — re-run this search before
/// removing the guard below if that changes.
/// </summary>
[Fact]
public void EmitHelpers_DoNotThrow()
{
Func<long> queueDepthProvider = () => 5;
var ex = Record.Exception(() =>
{
ScadaBridgeTelemetry.RecordDeploymentApplied();
@@ -27,9 +55,20 @@ public class ScadaBridgeTelemetryTests
ScadaBridgeTelemetry.LiveAlarmAggregatorStarted();
ScadaBridgeTelemetry.LiveAlarmAggregatorStopped();
ScadaBridgeTelemetry.RecordLiveAlarmStreamReconnect();
ScadaBridgeTelemetry.SetQueueDepthProvider(() => 5);
ScadaBridgeTelemetry.SetQueueDepthProvider(queueDepthProvider);
});
Assert.Null(ex);
try
{
Assert.Null(ex);
}
finally
{
// Compare-and-clear: only removes OUR delegate from the process-global slot, so
// a legitimate concurrent registrant (a real StoreAndForwardService started
// elsewhere in this process) is never stomped — mirrors
// ScadaBridgeTelemetry.ClearQueueDepthProvider's own doc'd contract.
ScadaBridgeTelemetry.ClearQueueDepthProvider(queueDepthProvider);
}
}
}
tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbAutoVacuumTests.cs
+171
View File
@@ -0,0 +1,171 @@
using Microsoft.Data.Sqlite;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using ZB.MOM.WW.LocalDb;
namespace ZB.MOM.WW.ScadaBridge.Host.Tests;
/// <summary>
/// Arch-review 2026-08-19 P4-6 — <c>PRAGMA auto_vacuum = INCREMENTAL</c> only takes effect on
/// a table-less database, and <c>SqliteLocalDb</c>'s constructor creates LocalDb's own
/// bookkeeping tables before anything else ever gets a connection to the consolidated site
/// file. <see cref="SiteLocalDbSetup.EnsureAutoVacuumOnFreshFile"/> is the fix: it pre-creates
/// a genuinely fresh file with the pragma set BEFORE <c>AddZbLocalDb</c>'s lazy singleton ever
/// constructs <c>SqliteLocalDb</c>.
/// </summary>
/// <remarks>
/// These tests exercise <see cref="SiteLocalDbSetup.EnsureAutoVacuumOnFreshFile"/> directly
/// (it is <c>internal</c>, visible here via <c>InternalsVisibleTo</c>) rather than through the
/// full <see cref="SiteServiceRegistration"/> composition root — the mechanism under test is a
/// pure ordering/pragma concern that does not depend on the rest of the site graph, and
/// <see cref="SiteLocalDbWiringTests"/> already proves the composition root wires
/// <c>SiteLocalDbSetup</c> in at all.
/// </remarks>
public class SiteLocalDbAutoVacuumTests
{
[Fact]
public void FreshFile_ReportsIncrementalAutoVacuum()
{
// The plan's own acceptance test: a fresh file reports PRAGMA auto_vacuum = 2
// (INCREMENTAL) after the pre-create step runs.
var path = TempDbPath();
try
{
var config = ConfigFor(path);
SiteLocalDbSetup.EnsureAutoVacuumOnFreshFile(config);
Assert.Equal(2L, ReadAutoVacuum(path));
}
finally
{
Cleanup(path);
}
}
[Fact]
public void FreshFile_AutoVacuumSurvives_LocalDbCreatingItsOwnBookkeepingTablesAfterward()
{
// The real end-to-end claim: the pre-create step has to run BEFORE AddZbLocalDb ever
// constructs SqliteLocalDb (which creates LocalDb's own tables first, unconditionally,
// in its constructor) for the ordering fix to mean anything. This proves the flag
// baked in by EnsureAutoVacuumOnFreshFile survives that later table creation — the
// exact sequence SiteServiceRegistration.Configure now runs in production.
var path = TempDbPath();
try
{
var config = ConfigFor(path);
SiteLocalDbSetup.EnsureAutoVacuumOnFreshFile(config);
var services = new ServiceCollection();
services.AddZbLocalDb(config, db => SiteLocalDbSetup.OnReady(db, config));
using var provider = services.BuildServiceProvider();
using var scope = provider.CreateScope();
// Resolving ILocalDb is what triggers SqliteLocalDb's constructor (LocalDbSchema
// .EnsureCreated) and then OnReady (SiteEventLogSchema.Apply, etc.) — i.e. every
// real table this file will ever hold. This config carries no
// LocalDb:Replication section, so OnReady takes the deregistration branch and
// db.ReplicatedTables stays empty — check the raw file instead to confirm the
// table genuinely exists.
var db = scope.ServiceProvider.GetRequiredService<ILocalDb>();
Assert.NotNull(db);
Assert.Contains("site_events", TableNames(path));
Assert.Equal(2L, ReadAutoVacuum(path));
}
finally
{
SqliteConnection.ClearAllPools();
Cleanup(path);
}
}
[Fact]
public void ExistingFile_IsLeftAlone_AutoVacuumStaysNone()
{
// Honesty requirement (plan decision 2): the fix only helps a file that does not exist
// yet. A file that already has tables — every node on the docker rig today, and any
// site database created before this fix shipped — is untouched, and stays
// auto_vacuum = NONE forever short of an offline VACUUM this task deliberately does
// not attempt.
var path = TempDbPath();
try
{
// Pre-populate the file exactly like a real (pre-fix) LocalDb file: at least one
// table exists before EnsureAutoVacuumOnFreshFile ever runs.
using (var seed = new SqliteConnection(new SqliteConnectionStringBuilder { DataSource = path }.ToString()))
{
seed.Open();
using var cmd = seed.CreateCommand();
cmd.CommandText = "CREATE TABLE legacy_marker (id INTEGER PRIMARY KEY)";
cmd.ExecuteNonQuery();
}
Assert.Equal(0L, ReadAutoVacuum(path)); // NONE, the SQLite default.
var config = ConfigFor(path);
SiteLocalDbSetup.EnsureAutoVacuumOnFreshFile(config);
// Still NONE — the method must not attempt VACUUM against a populated file.
Assert.Equal(0L, ReadAutoVacuum(path));
}
finally
{
SqliteConnection.ClearAllPools();
Cleanup(path);
}
}
[Fact]
public void MissingLocalDbPath_DoesNotThrow()
{
// AddZbLocalDb's own options validation is the authoritative place a missing path is
// reported; this pre-create step must defer to it rather than throwing its own,
// differently-worded error first.
var config = new ConfigurationBuilder().AddInMemoryCollection(
new Dictionary<string, string?>()).Build();
var ex = Record.Exception(() => SiteLocalDbSetup.EnsureAutoVacuumOnFreshFile(config));
Assert.Null(ex);
}
private static string TempDbPath() =>
Path.Combine(Path.GetTempPath(), $"scadabridge_autovacuum_{Guid.NewGuid():N}.db");
private static IConfiguration ConfigFor(string path) =>
new ConfigurationBuilder()
.AddInMemoryCollection(new Dictionary<string, string?> { ["LocalDb:Path"] = path })
.Build();
private static long ReadAutoVacuum(string path)
{
using var connection = new SqliteConnection(new SqliteConnectionStringBuilder { DataSource = path }.ToString());
connection.Open();
using var cmd = connection.CreateCommand();
cmd.CommandText = "PRAGMA auto_vacuum";
return Convert.ToInt64(cmd.ExecuteScalar());
}
private static HashSet<string> TableNames(string path)
{
using var connection = new SqliteConnection(new SqliteConnectionStringBuilder { DataSource = path }.ToString());
connection.Open();
using var cmd = connection.CreateCommand();
cmd.CommandText = "SELECT name FROM sqlite_master WHERE type = 'table'";
using var reader = cmd.ExecuteReader();
var names = new HashSet<string>(StringComparer.Ordinal);
while (reader.Read()) names.Add(reader.GetString(0));
return names;
}
private static void Cleanup(string path)
{
foreach (var suffix in new[] { "", "-wal", "-shm" })
{
try { File.Delete(path + suffix); } catch { /* best effort */ }
}
}
}
tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/BulkDeployPoisonedContextTests.cs
+370
View File
@@ -0,0 +1,370 @@
using Akka.Actor;
using Akka.TestKit.Xunit2;
using Microsoft.AspNetCore.DataProtection;
using Microsoft.Data.Sqlite;
using Microsoft.EntityFrameworkCore;
using Microsoft.EntityFrameworkCore.Diagnostics;
using Microsoft.EntityFrameworkCore.Storage.ValueConversion;
using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Options;
using NSubstitute;
using ZB.MOM.WW.ScadaBridge.Commons.Entities.Deployment;
using ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances;
using ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites;
using ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Transport;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Deployment;
using ZB.MOM.WW.ScadaBridge.Commons.Types;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Flattening;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Templates;
using ZB.MOM.WW.ScadaBridge.Communication;
using ZB.MOM.WW.ScadaBridge.ConfigurationDatabase;
using ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Repositories;
using ZB.MOM.WW.ScadaBridge.DeploymentManager;
using ZB.MOM.WW.ScadaBridge.TemplateEngine.Flattening;
namespace ZB.MOM.WW.ScadaBridge.IntegrationTests;
/// <summary>
/// P7-5: a bulk site deploy runs all three phases over ONE scoped
/// <c>DbContext</c>, so a persistent <c>SaveChanges</c> failure on a single
/// instance used to fail every instance that came after it.
///
/// <para>
/// <b>Why this needs a real EF context.</b> The mechanism is not in the service's
/// control flow, it is in EF's: a failed <c>SaveChanges</c> does NOT untrack the
/// entity that caused it. The entity stays in the change set and is re-attempted
/// by the very next save made through the same context — including the cleanup
/// save inside <c>MarkDeploymentFailedAsync</c>, whose whole job is to stop a
/// record being stranded <c>InProgress</c>. A mocked repository could not prove
/// that; a real <see cref="ScadaBridgeDbContext"/> over SQLite does.
/// </para>
///
/// <para>
/// <b>Honesty note on the injected failure.</b> The persistent failure itself is
/// injected by a save interceptor rather than provoked by a real constraint
/// violation, because the entity that must fail (the <see cref="Instance"/> row
/// the deploy path mutates) is created and mutated inside the service, where a
/// test cannot reach in and corrupt it mid-batch. The interceptor throws
/// <see cref="DbUpdateException"/> at exactly the point a constraint violation,
/// a concurrency conflict or a dropped connection would — everything after that,
/// including the cascade and the fix, is real EF behaviour.
/// </para>
/// </summary>
public sealed class BulkDeployPoisonedContextTests : TestKit, IDisposable
{
private const int InstanceCount = 4;
private readonly SqliteConnection _connection;
private readonly ScadaBridgeDbContext _context;
private readonly PoisonedEntityInterceptor _poison = new();
private int _siteId;
public BulkDeployPoisonedContextTests()
{
_connection = new SqliteConnection("DataSource=:memory:");
_connection.Open();
var options = new DbContextOptionsBuilder<ScadaBridgeDbContext>()
.UseSqlite(_connection)
.ConfigureWarnings(w => w.Ignore(RelationalEventId.PendingModelChangesWarning))
.AddInterceptors(_poison)
.Options;
_context = new SqliteContext(options);
_context.Database.EnsureCreated();
Seed();
}
/// <summary>
/// The whole finding in one run: instance 1's post-success persistence fails
/// permanently (best-effort, so the deploy still reports Success), which leaves
/// its <see cref="Instance"/> row poisoning the shared change tracker. Every
/// later instance's status write then rides on that same change set.
///
/// <para>
/// Pre-fix this cascaded: instances 2, 3 and 4 all failed, and their records
/// stayed <c>InProgress</c> because the cleanup save re-attempted — and
/// re-failed on — the same poisoned entity. Post-fix the cleanup detaches the
/// pending work, so instance 2 is the only casualty (correctly recorded
/// <c>Failed</c>, not stranded), and 3 and 4 save cleanly.
/// </para>
/// </summary>
[Fact]
public async Task BulkDeploy_PersistentSaveFailureOnOneInstance_DoesNotCascadeToTheRest()
{
_poison.PoisonInstanceNamed("Inst-01");
var service = CreateService();
var result = await service.DeploySiteAsync(_siteId, "admin");
Assert.True(result.IsSuccess);
var summary = result.Value;
Assert.Equal(InstanceCount, summary.InstanceResults.Count);
// Instances 3 and 4 are downstream of the poison and must be unaffected:
// this is the assertion that fails outright without the detach.
foreach (var name in new[] { "Inst-03", "Inst-04" })
{
var row = Assert.Single(summary.InstanceResults, r => r.UniqueName == name);
Assert.True(row.Success, $"{name} failed with: {row.ErrorMessage}");
}
// Instance 2 is the one collateral casualty — its own status write shared
// the poisoned change set — but it must be recorded Failed, never left
// InProgress, which is what the retry on the cleaned tracker buys.
var stranded = await NoLongerInProgressAsync();
Assert.Empty(stranded);
// ...and the fix is not "swallow everything": the poisoned instance's own
// side effects genuinely did not persist, so its Instance row is still
// NotDeployed even though the site confirmed the apply. That divergence is
// the pre-existing best-effort contract, logged for operator reconciliation.
await using var verify = NewContext();
Assert.Equal(
InstanceState.NotDeployed,
verify.Instances.Single(i => i.UniqueName == "Inst-01").State);
}
/// <summary>
/// The narrow EF fact the fix rests on, pinned on its own so a future reader
/// does not have to re-derive it: a failed save leaves its entity tracked and
/// the NEXT save re-attempts it, and
/// <c>IDeploymentManagerRepository.DetachPendingChanges</c> is what clears it.
/// </summary>
[Fact]
public async Task FailedSave_KeepsTheEntityTracked_UntilPendingChangesAreDetached()
{
var repository = new DeploymentManagerRepository(_context);
var instance = _context.Instances.Single(i => i.UniqueName == "Inst-01");
_poison.PoisonInstanceNamed("Inst-01");
instance.State = InstanceState.Enabled;
await repository.UpdateInstanceAsync(instance);
await Assert.ThrowsAsync<DbUpdateException>(() => repository.SaveChangesAsync());
// An unrelated, perfectly valid write now fails too — the poisoned entity
// is still in the change set.
var record = new DeploymentRecord("dep-unrelated", "admin")
{
InstanceId = _context.Instances.Single(i => i.UniqueName == "Inst-02").Id,
Status = DeploymentStatus.Failed,
RevisionHash = "sha256:x",
DeployedAt = DateTimeOffset.UtcNow,
};
await repository.AddDeploymentRecordAsync(record);
await Assert.ThrowsAsync<DbUpdateException>(() => repository.SaveChangesAsync());
// Detaching the pending work un-poisons the unit of work; the same write
// then succeeds.
Assert.True(repository.DetachPendingChanges() > 0);
await repository.AddDeploymentRecordAsync(record);
Assert.Equal(1, await repository.SaveChangesAsync());
}
/// <summary>Deployment records left non-terminal after the batch — the stranded-InProgress check.</summary>
private async Task<List<string>> NoLongerInProgressAsync()
{
await using var verify = NewContext();
return await verify.DeploymentRecords
.Where(r => r.Status == DeploymentStatus.InProgress || r.Status == DeploymentStatus.Pending)
.Select(r => r.DeploymentId)
.ToListAsync();
}
private DeploymentService CreateService()
{
var repository = new DeploymentManagerRepository(_context);
var siteRepository = new SiteRepository(_context);
var pipeline = Substitute.For<IFlatteningPipeline>();
pipeline.CreateSession().Returns(_ => new FlattenSession(new TemplateGraphWatermark()));
pipeline
.FlattenAndValidateAsync(
Arg.Any<int>(), Arg.Any<CancellationToken>(), Arg.Any<bool>(), Arg.Any<FlattenSession?>())
.Returns(ci => Result<FlatteningPipelineResult>.Success(
new FlatteningPipelineResult(
new FlattenedConfiguration { InstanceUniqueName = $"Inst-{ci.ArgAt<int>(0):00}" },
$"sha256:{ci.ArgAt<int>(0)}",
ValidationResult.Success())));
var comms = new CommunicationService(
Options.Create(new CommunicationOptions { DeploymentTimeout = TimeSpan.FromSeconds(10) }),
NullLogger<CommunicationService>.Instance);
comms.SetCommunicationActor(Sys.ActorOf(Props.Create(() => new AcceptingSiteActor())));
// Null probe result = "staleness indeterminate", so the up-to-date
// pre-filter never skips: this test is about the persistence cascade and
// needs every instance to actually run.
var probe = Substitute.For<IStaleInstanceProbe>();
probe.GetCurrentRevisionHashAsync(Arg.Any<int>(), Arg.Any<CancellationToken>()).Returns((string?)null);
return new DeploymentService(
repository, siteRepository, pipeline, comms, new OperationLockManager(),
Substitute.For<IAuditService>(),
new DiffService(),
new RevisionHashService(),
new DeploymentStatusNotifier(NullLogger<DeploymentStatusNotifier>.Instance),
probe,
Options.Create(new DeploymentManagerOptions
{
OperationLockTimeout = TimeSpan.FromSeconds(5),
SiteDeploymentMaxParallelism = 1,
SiteDeploymentTimeoutPerInstance = TimeSpan.FromSeconds(10),
SiteDeploymentReconcileTimeout = TimeSpan.FromSeconds(2),
}),
Options.Create(new CommunicationOptions
{
CentralFetchBaseUrl = "https://central.test:9000",
PendingDeploymentTtl = TimeSpan.FromMinutes(5),
}),
NullLogger<DeploymentService>.Instance);
}
private void Seed()
{
using var ctx = NewContext();
var site = new Site("Poison Site", "site-poison");
var template = new Template("T-Poison");
ctx.Sites.Add(site);
ctx.Templates.Add(template);
ctx.SaveChanges();
_siteId = site.Id;
for (var i = 1; i <= InstanceCount; i++)
{
ctx.Instances.Add(new Instance($"Inst-{i:00}")
{
SiteId = site.Id,
TemplateId = template.Id,
State = InstanceState.NotDeployed,
});
}
ctx.SaveChanges();
}
private ScadaBridgeDbContext NewContext() =>
new SqliteContext(new DbContextOptionsBuilder<ScadaBridgeDbContext>()
.UseSqlite(_connection)
.ConfigureWarnings(w => w.Ignore(RelationalEventId.PendingModelChangesWarning))
.Options);
public new void Dispose()
{
_context.Dispose();
_connection.Dispose();
base.Dispose();
}
/// <summary>
/// Fails every <c>SaveChanges</c> whose change set contains the named
/// <see cref="Instance"/> — a stand-in for a permanent constraint violation on
/// one tracked entity, raised at the point EF raises a real one.
/// </summary>
private sealed class PoisonedEntityInterceptor : SaveChangesInterceptor
{
private string? _poisonedUniqueName;
public void PoisonInstanceNamed(string uniqueName) => _poisonedUniqueName = uniqueName;
public override InterceptionResult<int> SavingChanges(
DbContextEventData eventData, InterceptionResult<int> result)
{
ThrowIfPoisoned(eventData);
return base.SavingChanges(eventData, result);
}
public override ValueTask<InterceptionResult<int>> SavingChangesAsync(
DbContextEventData eventData,
InterceptionResult<int> result,
CancellationToken cancellationToken = default)
{
ThrowIfPoisoned(eventData);
return base.SavingChangesAsync(eventData, result, cancellationToken);
}
private void ThrowIfPoisoned(DbContextEventData eventData)
{
if (_poisonedUniqueName is null || eventData.Context is null)
return;
var poisoned = eventData.Context.ChangeTracker.Entries<Instance>()
.Any(e => e.State != EntityState.Unchanged
&& e.State != EntityState.Detached
&& e.Entity.UniqueName == _poisonedUniqueName);
if (poisoned)
throw new DbUpdateException($"simulated persistent failure on {_poisonedUniqueName}");
}
}
/// <summary>Answers every deployment with Success, so only persistence can fail here.</summary>
private sealed class AcceptingSiteActor : ReceiveActor
{
public AcceptingSiteActor()
{
Receive<SiteEnvelope>(env =>
{
if (env.Message is not RefreshDeploymentCommand cmd)
return;
Sender.Tell(new DeploymentStatusResponse(
cmd.DeploymentId, cmd.InstanceUniqueName,
DeploymentStatus.Success, null, DateTimeOffset.UtcNow));
});
}
}
/// <summary>
/// SQLite-adapted context: SQLite has no rowversion and no native
/// DateTimeOffset. Mirrors the adaptation <c>DeployNotifyFetchTests</c> uses.
/// </summary>
private sealed class SqliteContext : ScadaBridgeDbContext
{
public SqliteContext(DbContextOptions<ScadaBridgeDbContext> options)
: base(options, new EphemeralDataProtectionProvider())
{
}
protected override void OnModelCreating(ModelBuilder modelBuilder)
{
base.OnModelCreating(modelBuilder);
modelBuilder.Entity<DeploymentRecord>(builder =>
{
builder.Property(d => d.RowVersion)
.IsRequired(false)
.IsConcurrencyToken(false)
.ValueGeneratedNever();
});
var converter = new ValueConverter<DateTimeOffset, string>(
v => v.UtcDateTime.ToString("o"),
v => DateTimeOffset.Parse(v));
var nullableConverter = new ValueConverter<DateTimeOffset?, string?>(
v => v.HasValue ? v.Value.UtcDateTime.ToString("o") : null,
v => v != null ? DateTimeOffset.Parse(v) : null);
foreach (var entityType in modelBuilder.Model.GetEntityTypes())
{
foreach (var property in entityType.GetProperties())
{
if (property.ClrType == typeof(DateTimeOffset))
{
property.SetValueConverter(converter);
property.SetColumnType("TEXT");
}
else if (property.ClrType == typeof(DateTimeOffset?))
{
property.SetValueConverter(nullableConverter);
property.SetColumnType("TEXT");
}
}
}
}
}
}
tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Grpc/SiteAlarmStreamEndToEndTests.cs
+4 -1
View File
@@ -224,7 +224,10 @@ public class SiteAlarmStreamEndToEndTests : TestKit
var aggregator = Sys.ActorOf(Props.Create(() => new SiteAlarmAggregatorActor(
"site-alpha", "e2e-parity", _ => Task.FromResult<IReadOnlyList<AlarmStateChanged>>(new[] { seedRow }),
sink.Publish, factory, "http://a:5100", "http://b:5100", TimeSpan.FromMinutes(10),
TimeSpan.Zero, TimeSpan.FromSeconds(5), TimeSpan.FromSeconds(60))));
TimeSpan.Zero, TimeSpan.FromSeconds(5), TimeSpan.FromSeconds(60),
// Props.Create is an expression tree, which may not omit optional arguments
// (CS0854) — jitter and the P3-4 open-attempt deadline are passed explicitly.
0.0, SiteAlarmAggregatorActor.DefaultOpenAttemptTimeout)));
await WaitForConditionAsync(() => sink.Latest is { Count: 1 }); // seed applied
aggregator.Tell(liveRow);
tests/ZB.MOM.WW.ScadaBridge.KpiHistory.Tests/KpiHistoryRecorderActorTests.cs
+26 -4
View File
@@ -497,6 +497,28 @@ public class KpiHistoryRecorderActorTests : TestKit
NullLogger<KpiHistoryRecorderActor>.Instance)));
}
/// <summary>
/// Row-40 house pattern for an ABSENCE assertion (mirrors the Medium-wave
/// <c>SiteAlarmAggregatorActorTests.AssertHoldsFor</c> treatment): proves
/// <paramref name="holds"/> keeps being true for the whole <paramref name="window"/> by
/// polling, rather than a single blind <c>Thread.Sleep(window)</c> followed by one
/// end-of-window sample. A bare sleep-then-assert cannot distinguish "the mechanism
/// correctly stayed quiet" from "the mechanism just hadn't run yet when the one sample
/// was taken", and it can't catch a value that flips wrong and then flips back before
/// that single sample. This fails immediately the first time <paramref name="holds"/>
/// goes false, and only returns once the full window has elapsed with it still true.
/// </summary>
private static void AssertHoldsFor(TimeSpan window, Func<bool> holds, string because)
{
var deadline = DateTime.UtcNow + window;
while (true)
{
Assert.True(holds(), because);
if (DateTime.UtcNow >= deadline) return;
Thread.Sleep(20);
}
}
[Fact]
public void SampleTick_WritesHealthySourceSamples_AndThrowingSourceDoesNotAbortTick()
{
@@ -1040,13 +1062,13 @@ public class KpiHistoryRecorderActorTests : TestKit
interval: TimeSpan.FromMilliseconds(50));
// …and then skip: no slices are ever folded.
Thread.Sleep(300);
Assert.Equal(0, repository.FoldCount);
AssertHoldsFor(TimeSpan.FromMilliseconds(300), () => repository.FoldCount == 0,
"watermark inside the lookback tail must skip the backfill entirely");
// A second BackfillTick is a no-op (the done-latch is set).
actor.Tell(KpiHistoryRecorderActor.BackfillTick.Instance);
Thread.Sleep(300);
Assert.Equal(0, repository.FoldCount);
AssertHoldsFor(TimeSpan.FromMilliseconds(300), () => repository.FoldCount == 0,
"the done-latch must make a second BackfillTick a no-op");
}
[Fact]
tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests/ManagementActorTests.cs
+249
View File
@@ -765,6 +765,255 @@ public class ManagementActorTests : TestKit, IDisposable
Assert.DoesNotContain("internal error", response.Error);
}
// ========================================================================
// P6-10 (register row-88 class): strict single-value enum parsing in
// ParseDataType / ParseElementDataType / the AlarmTriggerType parse sites.
// Raw Enum.Parse/TryParse admits ordinals ("2"), undefined values ("99" —
// a deploy-time bomb that saves fine and only detonates at deploy), and
// (Medium-wave T8 nuance) comma-joined name lists that TryParse/Parse
// bitwise-OR together regardless of [Flags]. ParseEnumStrict rejects all
// three, and — for AlarmTriggerType specifically — replaces a masked
// "An internal error occurred" (raw Enum.Parse's uncaught
// ArgumentException) with a curated, field-naming error.
// ========================================================================
[Theory]
[InlineData("2")] // ordinal for a DEFINED member (Float)
[InlineData("99")] // undefined ordinal — the deploy-time-bomb case
[InlineData("String,Int32")] // comma-joined names — TryParse ORs 4|1 into DateTime (5)
public void AddAttribute_WithNonNameDataTypeToken_ReturnsCuratedError(string token)
{
// PROOF OF FAILURE (pre-fix): the unguarded `Enum.TryParse` in the old
// ParseDataType has no numeric guard, no comma guard, and never checks
// Enum.IsDefined. Each of these tokens parsed successfully pre-fix (to
// Float, an undefined value, and DateTime respectively) and the command
// proceeded past the parser into TemplateService, failing later with
// "Template with ID 1 not found." instead of this curated message —
// verified by stashing this fix and re-running (see task report).
_services.AddScoped<TemplateService>();
var actor = CreateActor();
var envelope = Envelope(
new AddTemplateAttributeCommand(1, "Whatever", token, null, null, null, false, null),
"Designer");
actor.Tell(envelope);
var response = ExpectMsg<ManagementError>(TimeSpan.FromSeconds(5));
Assert.Equal("COMMAND_FAILED", response.ErrorCode);
Assert.Contains("Unrecognised data type", response.Error);
Assert.Contains(token, response.Error);
Assert.Contains("Valid values:", response.Error);
Assert.DoesNotContain("internal error", response.Error);
}
[Fact]
public void AddAttribute_WithTypoDataType_ReturnsCuratedErrorNamingValidValues()
{
// The pre-fix ParseDataType already curated a plain typo (that part of
// MV-10 was correct), but its message named no valid set — this asserts
// the new "Valid values:" suffix, which is absent pre-fix.
_services.AddScoped<TemplateService>();
var actor = CreateActor();
var envelope = Envelope(
new AddTemplateAttributeCommand(1, "Whatever", "Sting", null, null, null, false, null),
"Designer");
actor.Tell(envelope);
var response = ExpectMsg<ManagementError>(TimeSpan.FromSeconds(5));
Assert.Contains("Unrecognised data type 'Sting'", response.Error);
Assert.Contains("Valid values:", response.Error);
Assert.Contains("Boolean", response.Error);
Assert.Contains("List", response.Error);
}
[Theory]
[InlineData("Boolean")]
[InlineData("Int32")]
[InlineData("Float")]
[InlineData("Double")]
[InlineData("String")]
[InlineData("DateTime")]
[InlineData("Binary")]
public void AddAttribute_WithDefinedDataTypeName_ParsesAndProceedsPastGuard(string name)
{
// No repository stub: GetTemplateByIdAsync returns null (NSubstitute
// default), so a token the guard ACCEPTS reaches AddAttributeAsync's
// "Template not found" failure. A token the guard REJECTS would never
// reach the service at all (see the tests above), so this proves every
// defined name still parses.
_services.AddScoped<TemplateService>();
var actor = CreateActor();
var envelope = Envelope(
new AddTemplateAttributeCommand(1, "Whatever", name, null, null, null, false, null),
"Designer");
actor.Tell(envelope);
var response = ExpectMsg<ManagementError>(TimeSpan.FromSeconds(5));
Assert.Contains("Template with ID 1 not found", response.Error);
}
[Fact]
public void AddAttribute_WithDefinedListDataTypeNameAndElementType_ParsesAndProceedsPastGuard()
{
_services.AddScoped<TemplateService>();
var actor = CreateActor();
var envelope = Envelope(
new AddTemplateAttributeCommand(1, "Whatever", "List", null, null, null, false, "String"),
"Designer");
actor.Tell(envelope);
var response = ExpectMsg<ManagementError>(TimeSpan.FromSeconds(5));
Assert.Contains("Template with ID 1 not found", response.Error);
}
[Fact]
public void UpdateAttribute_WithCommaJoinedDataType_ReturnsCuratedError()
{
_services.AddScoped<TemplateService>();
var actor = CreateActor();
var envelope = Envelope(
new UpdateTemplateAttributeCommand(5, "Count", "String,Int32", null, null, null, false, null),
"Designer");
actor.Tell(envelope);
var response = ExpectMsg<ManagementError>(TimeSpan.FromSeconds(5));
Assert.Contains("Unrecognised data type", response.Error);
Assert.Contains("Valid values:", response.Error);
}
[Theory]
[InlineData("2")] // ordinal for a DEFINED, valid element type (Float)
[InlineData("99")] // undefined ordinal
[InlineData("String,Int32")] // comma-joined — combines to DateTime (also a valid element type)
public void AddListAttribute_WithNonNameElementTypeToken_ReturnsCuratedError(string token)
{
// PROOF OF FAILURE (pre-fix): each of these tokens parsed successfully
// (Float / an undefined value that fails a LATER, differently-worded
// "requires a valid element type" check / DateTime) rather than being
// rejected at the parse boundary with this message.
_services.AddScoped<TemplateService>();
var actor = CreateActor();
var envelope = Envelope(
new AddTemplateAttributeCommand(1, "Tags", "List", null, null, null, false, token),
"Designer");
actor.Tell(envelope);
var response = ExpectMsg<ManagementError>(TimeSpan.FromSeconds(5));
Assert.Equal("COMMAND_FAILED", response.ErrorCode);
Assert.Contains("Unrecognised element type", response.Error);
Assert.Contains(token, response.Error);
Assert.Contains("Valid values:", response.Error);
}
[Fact]
public void AddListAttribute_WithTypoElementType_ReturnsCuratedErrorNamingValidValues()
{
_services.AddScoped<TemplateService>();
var actor = CreateActor();
var envelope = Envelope(
new AddTemplateAttributeCommand(1, "Tags", "List", null, null, null, false, "Sting"),
"Designer");
actor.Tell(envelope);
var response = ExpectMsg<ManagementError>(TimeSpan.FromSeconds(5));
Assert.Contains("Unrecognised element type 'Sting'", response.Error);
Assert.Contains("Valid values:", response.Error);
}
[Theory]
[InlineData("0")] // ordinal for a DEFINED member (ValueMatch)
[InlineData("99")] // undefined ordinal — the deploy-time-bomb case
[InlineData("ValueMatch,RangeViolation")] // comma-joined — Enum.Parse ORs 0|1 into RangeViolation
public void AddAlarm_WithNonNameTriggerTypeToken_ReturnsCuratedError(string token)
{
// PROOF OF FAILURE (pre-fix): AddAlarm/UpdateAlarm built TriggerType via
// raw `Enum.Parse`, which has none of ParseDeploymentStatus's guards.
// Each of these tokens parsed SILENTLY (to ValueMatch, an undefined
// value, and RangeViolation respectively — no exception at all) and the
// command proceeded into TemplateService, failing later with
// "Template with ID 1 not found." instead of this curated message.
_services.AddScoped<TemplateService>();
var actor = CreateActor();
var envelope = Envelope(
new AddTemplateAlarmCommand(1, "AlarmX", token, 5, null, null, false),
"Designer");
actor.Tell(envelope);
var response = ExpectMsg<ManagementError>(TimeSpan.FromSeconds(5));
Assert.Equal("COMMAND_FAILED", response.ErrorCode);
Assert.Contains("Unrecognised alarm trigger type", response.Error);
Assert.Contains(token, response.Error);
Assert.Contains("Valid values:", response.Error);
}
[Fact]
public void AddAlarm_WithTypoTriggerType_ReturnsCuratedErrorInsteadOfMaskedInternalError()
{
// PROOF OF FAILURE (pre-fix): unlike ParseDataType (already curated by
// MV-10), the TriggerType sites used raw `Enum.Parse`. A typo's
// ArgumentException is uncaught by any handler-local guard and reaches
// MapFault as an unanticipated fault — masked to a generic
// "An internal error occurred (CorrelationId=...)" message, exactly the
// masking this task's defect description calls out. Post-fix,
// ParseEnumStrict raises a curated ManagementCommandException instead.
_services.AddScoped<TemplateService>();
var actor = CreateActor();
var envelope = Envelope(
new AddTemplateAlarmCommand(1, "AlarmX", "ValueMach", 5, null, null, false),
"Designer");
actor.Tell(envelope);
var response = ExpectMsg<ManagementError>(TimeSpan.FromSeconds(5));
Assert.Equal("COMMAND_FAILED", response.ErrorCode);
Assert.Contains("Unrecognised alarm trigger type 'ValueMach'", response.Error);
Assert.Contains("Valid values:", response.Error);
Assert.DoesNotContain("internal error", response.Error);
}
[Theory]
[InlineData("ValueMatch")]
[InlineData("RangeViolation")]
[InlineData("RateOfChange")]
[InlineData("HiLo")]
[InlineData("Expression")]
public void AddAlarm_WithDefinedTriggerTypeName_ParsesAndProceedsPastGuard(string name)
{
_services.AddScoped<TemplateService>();
var actor = CreateActor();
var envelope = Envelope(
new AddTemplateAlarmCommand(1, "AlarmX", name, 5, null, null, false),
"Designer");
actor.Tell(envelope);
var response = ExpectMsg<ManagementError>(TimeSpan.FromSeconds(5));
Assert.Contains("Template with ID 1 not found", response.Error);
}
[Fact]
public void UpdateAlarm_WithOrdinalTriggerType_ReturnsCuratedError()
{
_services.AddScoped<TemplateService>();
var actor = CreateActor();
var envelope = Envelope(
new UpdateTemplateAlarmCommand(5, "AlarmX", "0", 5, null, null, false),
"Designer");
actor.Tell(envelope);
var response = ExpectMsg<ManagementError>(TimeSpan.FromSeconds(5));
Assert.Contains("Unrecognised alarm trigger type", response.Error);
Assert.Contains("Valid values:", response.Error);
}
[Fact]
public void UpdateAttribute_ValidScalar_PersistsValueAndKeepsFixedFields()
{
tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests/ManagementEndpointsTests.cs
+22
View File
@@ -23,6 +23,28 @@ public class ManagementEndpointsTests
Assert.Equal("Desc", command.Description);
}
/// <summary>
/// P7-6: <c>MgmtDeploySiteCommand.Force</c> is an ADDITIVE field, so a client
/// that predates it — every CLI older than this change, and every scripted
/// HTTP caller — must keep working and must land on the SAFE default. Absent
/// means <c>false</c> means "skip instances whose config is unchanged", which
/// is the non-destructive behaviour; only an explicit <c>true</c> restores the
/// unconditional redeploy.
/// </summary>
[Theory]
[InlineData("""{ "command": "MgmtDeploySite", "payload": { "siteId": 7 } }""", false)]
[InlineData("""{ "command": "MgmtDeploySite", "payload": { "siteId": 7, "force": false } }""", false)]
[InlineData("""{ "command": "MgmtDeploySite", "payload": { "siteId": 7, "force": true } }""", true)]
public void ParseCommand_DeploySite_ForceDefaultsToTheNonDestructiveBehaviour(string json, bool expectedForce)
{
var result = ManagementEndpoints.ParseCommand(json);
Assert.True(result.Success);
var command = Assert.IsType<MgmtDeploySiteCommand>(result.Command);
Assert.Equal(7, command.SiteId);
Assert.Equal(expectedForce, command.Force);
}
[Fact]
public void ParseCommand_WithMissingPayload_DeserializesParameterlessCommand()
{
tests/ZB.MOM.WW.ScadaBridge.NotificationOutbox.Tests/Kpi/NotificationOutboxKpiSampleSourceTests.cs
+61 -1
View File
@@ -3,6 +3,7 @@ using ZB.MOM.WW.ScadaBridge.Commons.Entities.Kpi;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Kpi;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Notifications;
using ZB.MOM.WW.ScadaBridge.KpiHistory;
using ZB.MOM.WW.ScadaBridge.NotificationOutbox.Kpi;
namespace ZB.MOM.WW.ScadaBridge.NotificationOutbox.Tests.Kpi;
@@ -22,8 +23,28 @@ public class NotificationOutboxKpiSampleSourceTests
DeliveredKpiWindow = TimeSpan.FromMinutes(1),
};
// Matches Options.DeliveredKpiWindow (1 minute) so the default-config tests below see
// an identity rescale factor (1.0) and their pre-existing raw-count assertions hold.
private static readonly KpiHistoryOptions DefaultKpiHistoryOptions = new()
{
SampleInterval = TimeSpan.FromMinutes(1),
};
private static NotificationOutboxKpiSampleSource CreateSource(INotificationOutboxRepository repository) =>
new(repository, Microsoft.Extensions.Options.Options.Create(Options));
CreateSource(repository, DefaultKpiHistoryOptions);
private static NotificationOutboxKpiSampleSource CreateSource(
INotificationOutboxRepository repository, KpiHistoryOptions kpiHistoryOptions) =>
CreateSource(repository, Options, kpiHistoryOptions);
private static NotificationOutboxKpiSampleSource CreateSource(
INotificationOutboxRepository repository,
NotificationOutboxOptions options,
KpiHistoryOptions kpiHistoryOptions) =>
new(
repository,
Microsoft.Extensions.Options.Options.Create(options),
Microsoft.Extensions.Options.Options.Create(kpiHistoryOptions));
[Fact]
public void Source_IsNotificationOutbox()
@@ -131,6 +152,45 @@ public class NotificationOutboxKpiSampleSourceTests
AssertSample(samples, "oldestPendingAgeSeconds", KpiScopes.Node, "node-a", 60);
}
// P5-5 (arch-review Low): the KPI-history rollup fold sums every deliveredLastInterval
// sample recorded in an hour, which over-counts unless each sample is a non-overlapping
// delta covering exactly one recorder tick (KpiHistoryOptions.SampleInterval). Before the
// fix, this class recorded the RAW DeliveredKpiWindow-windowed count verbatim — with a
// 15-minute DeliveredKpiWindow sampled every 1-minute SampleInterval, 60 overlapping
// 15-minute counts summed to ~15x the true hourly total. Proven against the unfixed
// computation: temporarily short-circuiting DeliveredIntervalScaleFactor to always return
// 1.0 (i.e. "record the raw count verbatim", the pre-P5-5 behavior) makes this test fail
// (30 != 2); restoring the real scale factor passes it.
[Fact]
public async Task CollectAsync_RescalesDeliveredLastInterval_OntoKpiHistorySampleInterval()
{
var repository = Substitute.For<INotificationOutboxRepository>();
repository.ComputeKpisAsync(Arg.Any<DateTimeOffset>(), Arg.Any<DateTimeOffset>(), Arg.Any<CancellationToken>())
.Returns(new NotificationKpiSnapshot(
QueueDepth: 0, StuckCount: 0, ParkedCount: 0,
DeliveredLastInterval: 30, OldestPendingAge: null));
repository.ComputePerSiteKpisAsync(Arg.Any<DateTimeOffset>(), Arg.Any<DateTimeOffset>(), Arg.Any<CancellationToken>())
.Returns(Array.Empty<SiteNotificationKpiSnapshot>());
repository.ComputePerNodeKpisAsync(Arg.Any<DateTimeOffset>(), Arg.Any<DateTimeOffset>(), Arg.Any<CancellationToken>())
.Returns(Array.Empty<NodeNotificationKpiSnapshot>());
// DeliveredKpiWindow (15 min) is 15x KpiHistoryOptions.SampleInterval (1 min) — a
// non-default SampleInterval configuration deliberately chosen to prove the rescale
// math, not just the identity (factor == 1) default-config case the other tests use.
var options = new NotificationOutboxOptions { DeliveredKpiWindow = TimeSpan.FromMinutes(15) };
var kpiHistoryOptions = new KpiHistoryOptions { SampleInterval = TimeSpan.FromMinutes(1) };
var source = CreateSource(repository, options, kpiHistoryOptions);
var samples = await source.CollectAsync(CapturedAt);
var delivered = Assert.Single(samples, s => s.Metric == "deliveredLastInterval");
// 30 counted over a 15-minute window, rescaled onto a 1-minute recorder tick: 30 / 15 = 2.
// Summing 60 such 1-minute-tick samples across an hour therefore reconstructs the true
// hourly total instead of the ~15x over-count a raw (unscaled) 15-minute count would
// produce if simply summed 60 times.
Assert.Equal(2.0, delivered.Value, precision: 6);
}
[Fact]
public async Task CollectAsync_OmitsOldestPendingAge_WhenNull()
{
tests/ZB.MOM.WW.ScadaBridge.NotificationOutbox.Tests/NotificationOutboxActorQueryTests.cs
+84 -12
View File
@@ -258,7 +258,7 @@ public class NotificationOutboxActorQueryTests : TestKit
Assert.True(response.Success);
Assert.Null(response.ErrorMessage);
_repository.Received(1).UpdateAsync(
_repository.Received(1).TryUpdateFromParkedAsync(
Arg.Is<Notification>(n =>
n.NotificationId == row.NotificationId &&
n.Status == NotificationStatus.Pending &&
@@ -280,7 +280,7 @@ public class NotificationOutboxActorQueryTests : TestKit
var response = ExpectMsg<RetryNotificationResponse>();
Assert.False(response.Success);
Assert.NotNull(response.ErrorMessage);
_repository.DidNotReceive().UpdateAsync(Arg.Any<Notification>(), Arg.Any<CancellationToken>());
_repository.DidNotReceive().TryUpdateFromParkedAsync(Arg.Any<Notification>(), Arg.Any<CancellationToken>());
}
[Fact]
@@ -300,17 +300,19 @@ public class NotificationOutboxActorQueryTests : TestKit
/// <summary>
/// The row is read successfully but the retention purge deletes it before the
/// write lands, so the targeted UPDATE matches nothing. The operator must be
/// told the notification is gone — reporting a re-queue that never happened
/// is what the ExecuteUpdate rewrite silently introduced (the predecessor
/// threw DbUpdateConcurrencyException here).
/// write lands, so the CAS write matches nothing. The re-read that follows a
/// lost CAS (arch-review 2026-08-19 P5-4) finds nothing either, and the
/// operator must be told the notification is gone — reporting a re-queue
/// that never happened is what a blind ExecuteUpdate would silently do (the
/// pre-ExecuteUpdate predecessor threw DbUpdateConcurrencyException here).
/// </summary>
[Fact]
public void Retry_RowPurgedBeforeWrite_RepliesNotFound()
{
var row = MakeNotification(status: NotificationStatus.Parked, retryCount: 10, lastError: "gave up");
_repository.GetByIdAsync(row.NotificationId, Arg.Any<CancellationToken>()).Returns(row);
_repository.UpdateAsync(Arg.Any<Notification>(), Arg.Any<CancellationToken>()).Returns(false);
_repository.GetByIdAsync(row.NotificationId, Arg.Any<CancellationToken>())
.Returns(row, (Notification?)null);
_repository.TryUpdateFromParkedAsync(Arg.Any<Notification>(), Arg.Any<CancellationToken>()).Returns(false);
var actor = CreateActor();
actor.Tell(new RetryNotificationRequest("corr-vanished-retry", row.NotificationId), TestActor);
@@ -327,8 +329,9 @@ public class NotificationOutboxActorQueryTests : TestKit
public void Discard_RowPurgedBeforeWrite_RepliesNotFound()
{
var row = MakeNotification(status: NotificationStatus.Parked);
_repository.GetByIdAsync(row.NotificationId, Arg.Any<CancellationToken>()).Returns(row);
_repository.UpdateAsync(Arg.Any<Notification>(), Arg.Any<CancellationToken>()).Returns(false);
_repository.GetByIdAsync(row.NotificationId, Arg.Any<CancellationToken>())
.Returns(row, (Notification?)null);
_repository.TryUpdateFromParkedAsync(Arg.Any<Notification>(), Arg.Any<CancellationToken>()).Returns(false);
var actor = CreateActor();
actor.Tell(new DiscardNotificationRequest("corr-vanished-discard", row.NotificationId), TestActor);
@@ -340,6 +343,75 @@ public class NotificationOutboxActorQueryTests : TestKit
Assert.Contains("not found", response.ErrorMessage);
}
/// <summary>
/// PROOF OF FAILURE for arch-review 2026-08-19 P5-4. Models the race the
/// finding describes: the operator's read sees Parked, but by the time the
/// write lands the dispatcher has already delivered the row (a detached
/// PipeTo task interleaving with the dispatcher's own write). The mechanism
/// is proven deterministically via a mocked call sequence, not a wall-clock
/// sleep (per the "no new sleep-then-assert" constraint) — the first
/// <see cref="INotificationOutboxRepository.GetByIdAsync"/> call (the
/// operator's read) returns the Parked row, the CAS write reports a lost
/// race, and the SECOND <c>GetByIdAsync</c> call (the fix's re-read used to
/// build an honest reply) returns the row now Delivered.
/// <para>
/// Against the pre-fix actor — which calls the unconditional
/// <c>UpdateAsync</c> exactly once and never re-reads — this test fails:
/// the single-read code path only ever consumes the first queued value
/// (Parked), the in-memory check passes, <c>UpdateAsync</c> is stubbed
/// <c>Healthy()</c> (returns true), and the response comes back
/// <c>Success = true</c> — i.e. the Delivered row gets silently clobbered
/// to Discarded. Post-fix, the CAS is lost, the actor re-reads and reports
/// "already progressed to Delivered" instead of overwriting it.
/// </para>
/// </summary>
[Fact]
public void Discard_ConcurrentlyDelivered_RepliesAlreadyProgressed_DoesNotOverwrite()
{
var parkedRow = MakeNotification(status: NotificationStatus.Parked, retryCount: 5, lastError: "gave up");
var deliveredRow = MakeNotification(status: NotificationStatus.Delivered);
_repository.GetByIdAsync(parkedRow.NotificationId, Arg.Any<CancellationToken>())
.Returns(parkedRow, deliveredRow);
_repository.TryUpdateFromParkedAsync(Arg.Any<Notification>(), Arg.Any<CancellationToken>()).Returns(false);
var actor = CreateActor();
actor.Tell(new DiscardNotificationRequest("corr-race-discard", parkedRow.NotificationId), TestActor);
var response = ExpectMsg<DiscardNotificationResponse>();
Assert.Equal("corr-race-discard", response.CorrelationId);
Assert.False(response.Success);
Assert.NotNull(response.ErrorMessage);
Assert.Contains("already progressed", response.ErrorMessage);
Assert.Contains("Delivered", response.ErrorMessage);
// The critical invariant: the old unconditional UpdateAsync must NEVER
// be reached by the operator path — that call is what silently
// clobbered the Delivered row pre-fix.
_repository.DidNotReceive().UpdateAsync(Arg.Any<Notification>(), Arg.Any<CancellationToken>());
}
/// <summary>Retry half of <see cref="Discard_ConcurrentlyDelivered_RepliesAlreadyProgressed_DoesNotOverwrite"/>.</summary>
[Fact]
public void Retry_ConcurrentlyDelivered_RepliesAlreadyProgressed_DoesNotOverwrite()
{
var parkedRow = MakeNotification(status: NotificationStatus.Parked, retryCount: 5, lastError: "gave up");
var deliveredRow = MakeNotification(status: NotificationStatus.Delivered);
_repository.GetByIdAsync(parkedRow.NotificationId, Arg.Any<CancellationToken>())
.Returns(parkedRow, deliveredRow);
_repository.TryUpdateFromParkedAsync(Arg.Any<Notification>(), Arg.Any<CancellationToken>()).Returns(false);
var actor = CreateActor();
actor.Tell(new RetryNotificationRequest("corr-race-retry", parkedRow.NotificationId), TestActor);
var response = ExpectMsg<RetryNotificationResponse>();
Assert.Equal("corr-race-retry", response.CorrelationId);
Assert.False(response.Success);
Assert.NotNull(response.ErrorMessage);
Assert.Contains("already progressed", response.ErrorMessage);
Assert.Contains("Delivered", response.ErrorMessage);
_repository.DidNotReceive().UpdateAsync(Arg.Any<Notification>(), Arg.Any<CancellationToken>());
}
[Fact]
public void Discard_ParkedNotification_MarksDiscarded_AndSucceeds()
{
@@ -353,7 +425,7 @@ public class NotificationOutboxActorQueryTests : TestKit
Assert.Equal("corr-8", response.CorrelationId);
Assert.True(response.Success);
_repository.Received(1).UpdateAsync(
_repository.Received(1).TryUpdateFromParkedAsync(
Arg.Is<Notification>(n => n.Status == NotificationStatus.Discarded),
Arg.Any<CancellationToken>());
}
@@ -370,7 +442,7 @@ public class NotificationOutboxActorQueryTests : TestKit
var response = ExpectMsg<DiscardNotificationResponse>();
Assert.False(response.Success);
Assert.NotNull(response.ErrorMessage);
_repository.DidNotReceive().UpdateAsync(Arg.Any<Notification>(), Arg.Any<CancellationToken>());
_repository.DidNotReceive().TryUpdateFromParkedAsync(Arg.Any<Notification>(), Arg.Any<CancellationToken>());
}
[Fact]
tests/ZB.MOM.WW.ScadaBridge.NotificationOutbox.Tests/NotificationOutboxActorRetryEmissionTests.cs
+1 -1
View File
@@ -147,7 +147,7 @@ public class NotificationOutboxActorRetryEmissionTests : TestKit
ExpectMsg<RetryNotificationResponse>(r => r.Success);
AwaitAssert(() =>
{
_outboxRepository.Received(1).UpdateAsync(
_outboxRepository.Received(1).TryUpdateFromParkedAsync(
Arg.Is<Notification>(n => n.Status == NotificationStatus.Pending),
Arg.Any<CancellationToken>());
});
tests/ZB.MOM.WW.ScadaBridge.NotificationOutbox.Tests/NotificationOutboxOptionsValidatorTests.cs
+92
View File
@@ -0,0 +1,92 @@
using Microsoft.Extensions.Options;
namespace ZB.MOM.WW.ScadaBridge.NotificationOutbox.Tests;
/// <summary>
/// P5-5 (arch-review Low): <see cref="NotificationOutboxOptions"/> had no
/// registered validator before this — the dispatch/purge/KPI cadences and the
/// delivered-throughput KPI window (<see cref="NotificationOutboxOptions.DeliveredKpiWindow"/>)
/// could be zero/negative with no boot-time signal, which would spin an Akka
/// scheduler or — because <c>NotificationOutboxKpiSampleSource</c> divides by
/// <c>DeliveredKpiWindow</c> to rescale onto <c>KpiHistoryOptions.SampleInterval</c>
/// — divide by zero. <see cref="NotificationOutboxOptions.MaxParallelDeliveries"/>
/// is validated on its BASE property, not the clamped
/// <see cref="NotificationOutboxOptions.ResolvedMaxParallelDeliveries"/>, so a
/// misconfigured non-positive value fails fast instead of silently clamping to 1.
/// </summary>
public class NotificationOutboxOptionsValidatorTests
{
private static ValidateOptionsResult Validate(NotificationOutboxOptions options) =>
new NotificationOutboxOptionsValidator().Validate(Options.DefaultName, options);
[Fact]
public void DefaultOptions_AreValid()
{
var result = Validate(new NotificationOutboxOptions());
Assert.True(result.Succeeded, result.FailureMessage);
}
[Fact]
public void ZeroDispatchInterval_IsRejected()
{
var result = Validate(new NotificationOutboxOptions { DispatchInterval = TimeSpan.Zero });
Assert.True(result.Failed);
Assert.Contains("DispatchInterval", result.FailureMessage);
}
[Fact]
public void ZeroDispatchBatchSize_IsRejected()
{
var result = Validate(new NotificationOutboxOptions { DispatchBatchSize = 0 });
Assert.True(result.Failed);
Assert.Contains("DispatchBatchSize", result.FailureMessage);
}
[Fact]
public void ZeroStuckAgeThreshold_IsRejected()
{
var result = Validate(new NotificationOutboxOptions { StuckAgeThreshold = TimeSpan.Zero });
Assert.True(result.Failed);
Assert.Contains("StuckAgeThreshold", result.FailureMessage);
}
[Fact]
public void ZeroTerminalRetention_IsRejected()
{
var result = Validate(new NotificationOutboxOptions { TerminalRetention = TimeSpan.Zero });
Assert.True(result.Failed);
Assert.Contains("TerminalRetention", result.FailureMessage);
}
[Fact]
public void ZeroPurgeInterval_IsRejected()
{
var result = Validate(new NotificationOutboxOptions { PurgeInterval = TimeSpan.Zero });
Assert.True(result.Failed);
Assert.Contains("PurgeInterval", result.FailureMessage);
}
[Fact]
public void ZeroDeliveredKpiWindow_IsRejected()
{
var result = Validate(new NotificationOutboxOptions { DeliveredKpiWindow = TimeSpan.Zero });
Assert.True(result.Failed);
Assert.Contains("DeliveredKpiWindow", result.FailureMessage);
}
[Fact]
public void NegativeDeliveredKpiWindow_IsRejected()
{
var result = Validate(new NotificationOutboxOptions { DeliveredKpiWindow = TimeSpan.FromSeconds(-1) });
Assert.True(result.Failed);
Assert.Contains("DeliveredKpiWindow", result.FailureMessage);
}
[Fact]
public void ZeroMaxParallelDeliveries_IsRejected()
{
var result = Validate(new NotificationOutboxOptions { MaxParallelDeliveries = 0 });
Assert.True(result.Failed);
Assert.Contains("MaxParallelDeliveries", result.FailureMessage);
}
}
tests/ZB.MOM.WW.ScadaBridge.NotificationOutbox.Tests/OutboxRepositorySubstitute.cs
+13 -8
View File
@@ -8,14 +8,16 @@ namespace ZB.MOM.WW.ScadaBridge.NotificationOutbox.Tests;
/// Factory for the outbox repository substitute used across these tests.
/// </summary>
/// <remarks>
/// <see cref="INotificationOutboxRepository.UpdateAsync"/> returns whether the
/// targeted UPDATE actually matched a row — <c>false</c> is the "this
/// notification no longer exists" signal the operator retry/discard handlers
/// turn into a not-found reply. NSubstitute's default for <c>Task&lt;bool&gt;</c>
/// is <c>false</c>, i.e. the VANISHED-row answer, so an unconfigured substitute
/// would silently put every test on the failure path. Tests that want a healthy
/// store take one from here; the vanished-row tests configure
/// <c>Returns(false)</c> for themselves.
/// <see cref="INotificationOutboxRepository.UpdateAsync"/> and
/// <see cref="INotificationOutboxRepository.TryUpdateFromParkedAsync"/> both
/// return whether the targeted UPDATE actually matched a row — <c>false</c> is
/// the "this notification no longer exists (or, for the CAS variant, is no
/// longer Parked)" signal the operator retry/discard handlers turn into a
/// not-found / already-progressed reply. NSubstitute's default for
/// <c>Task&lt;bool&gt;</c> is <c>false</c>, i.e. the failure answer, so an
/// unconfigured substitute would silently put every test on the failure path.
/// Tests that want a healthy store take one from here; the vanished-row and
/// lost-CAS tests configure <c>Returns(false)</c> for themselves.
/// </remarks>
internal static class OutboxRepositorySubstitute
{
@@ -26,6 +28,9 @@ internal static class OutboxRepositorySubstitute
repository
.UpdateAsync(Arg.Any<Notification>(), Arg.Any<CancellationToken>())
.Returns(true);
repository
.TryUpdateFromParkedAsync(Arg.Any<Notification>(), Arg.Any<CancellationToken>())
.Returns(true);
return repository;
}
}
tests/ZB.MOM.WW.ScadaBridge.SiteCallAudit.Tests/Kpi/SiteCallAuditKpiSampleSourceTests.cs
+57 -2
View File
@@ -5,6 +5,7 @@ using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
using ZB.MOM.WW.ScadaBridge.Commons.Types;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Audit;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Kpi;
using ZB.MOM.WW.ScadaBridge.KpiHistory;
using ZB.MOM.WW.ScadaBridge.SiteCallAudit.Kpi;
namespace ZB.MOM.WW.ScadaBridge.SiteCallAudit.Tests.Kpi;
@@ -30,9 +31,21 @@ public class SiteCallAuditKpiSampleSourceTests
KpiInterval = kpiInterval ?? TimeSpan.FromMinutes(1),
};
// Matches Options().KpiInterval (1 minute) so the default-config tests below see an
// identity rescale factor (1.0) and their pre-existing raw-count assertions hold.
private static readonly KpiHistoryOptions DefaultKpiHistoryOptions = new()
{
SampleInterval = TimeSpan.FromMinutes(1),
};
private static SiteCallAuditKpiSampleSource CreateSource(
ISiteCallAuditRepository repo, SiteCallAuditOptions? options = null) =>
new(repo, OptionsWrap(options ?? Options()));
ISiteCallAuditRepository repo,
SiteCallAuditOptions? options = null,
KpiHistoryOptions? kpiHistoryOptions = null) =>
new(
repo,
OptionsWrap(options ?? Options()),
Microsoft.Extensions.Options.Options.Create(kpiHistoryOptions ?? DefaultKpiHistoryOptions));
private static IOptions<SiteCallAuditOptions> OptionsWrap(SiteCallAuditOptions o) =>
Microsoft.Extensions.Options.Options.Create(o);
@@ -205,6 +218,48 @@ public class SiteCallAuditKpiSampleSourceTests
Assert.Equal(3, repo.Calls.Count);
}
// ---------------------------------------------------------------------
// P5-5 (arch-review Low): failedLastInterval/deliveredLastInterval are rescaled onto
// KpiHistoryOptions.SampleInterval before recording, so the central hourly rollup fold's
// per-hour Sum stays correct regardless of how KpiInterval is tuned. Before the fix, this
// class recorded the RAW KpiInterval-windowed counts verbatim; with a 20-minute
// KpiInterval sampled every 1-minute SampleInterval, summing 60 overlapping 20-minute
// counts across an hour would over-count ~20x. Proven against the unfixed computation:
// temporarily short-circuiting IntervalScaleFactor to always return 1.0 (i.e. "record the
// raw count verbatim", the pre-P5-5 behavior) makes this test fail (40/1 != 2/0.05);
// restoring the real scale factor passes it.
// ---------------------------------------------------------------------
[Fact]
public async Task CollectAsync_RescalesFailedAndDeliveredLastInterval_OntoKpiHistorySampleInterval()
{
var repo = new StubRepo
{
Global = new SiteCallKpiSnapshot(
BufferedCount: 0,
ParkedCount: 0,
FailedLastInterval: 1,
DeliveredLastInterval: 40,
OldestPendingAge: null,
StuckCount: 0),
};
// KpiInterval (20 min) is 20x KpiHistoryOptions.SampleInterval (1 min) — a
// non-default SampleInterval configuration deliberately chosen to prove the rescale
// math, not just the identity (factor == 1) default-config case the other tests use.
var options = Options(kpiInterval: TimeSpan.FromMinutes(20));
var kpiHistoryOptions = new KpiHistoryOptions { SampleInterval = TimeSpan.FromMinutes(1) };
var samples = await CreateSource(repo, options, kpiHistoryOptions).CollectAsync(CapturedAt);
// 1 failure / 40 deliveries counted over a 20-minute window, rescaled onto a
// 1-minute recorder tick: divide by 20. Summing 60 such 1-minute-tick samples across
// an hour therefore reconstructs the true hourly totals instead of the ~20x
// over-count a raw (unscaled) 20-minute count would produce if simply summed 60 times.
AssertHas(samples, "failedLastInterval", KpiScopes.Global, null, 0.05);
AssertHas(samples, "deliveredLastInterval", KpiScopes.Global, null, 2.0);
}
// ---------------------------------------------------------------------
// 6. Multiple sites/nodes are all emitted with distinct ScopeKeys.
// ---------------------------------------------------------------------
tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/EventLogPurgeServiceTests.cs
+104
View File
@@ -1,3 +1,4 @@
using System.Reflection;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Options;
@@ -355,6 +356,109 @@ public class EventLogPurgeServiceTests : IDisposable
await purge.StopAsync(CancellationToken.None);
}
[Fact]
public async Task ExecuteAsync_YieldsBeforeRunningTheInitialPurgeSynchronously()
{
// arch-review 2026-08-19 P4-5 — timing-independent companion to
// StartAsync_DoesNotBlock_OnTheInitialPurge above.
//
// That test times BackgroundService.StartAsync as a whole and passes even on the
// UNFIXED code, because .NET 8+ already invokes ExecuteAsync via Task.Run — the
// synchronous prelude runs on a pool thread, not the caller's, so StartAsync itself
// never blocks regardless of whether ExecuteAsync yields first. Verified empirically
// against this repo's actual net10.0 target (a BackgroundService whose ExecuteAsync
// does Thread.Sleep(1500) before its first await still returns from StartAsync in
// ~2 ms). So P4-5's literal "StartAsync blocks" symptom does not reproduce here, and
// a wall-clock test cannot prove the fix.
//
// What CAN be proven without any timing is the ExecuteAsync-level contract itself:
// invoking the protected ExecuteAsync directly via reflection — bypassing
// BackgroundService's Task.Run wrapper entirely — still runs it synchronously, on
// the calling thread, up through its first genuinely-suspending await (that is what
// "async Task method" means; MethodInfo.Invoke does not change it). On the unfixed
// code ExecuteAsync's first statement is RunPurge() with no await before it, so
// Invoke would not return until the initial purge had fully run — deleting the
// expired row before this method's Assert ever executes. With Task.Yield() as the
// very first statement, Invoke returns as soon as MoveNext() hits the (always
// IsCompleted == false) yield point, before RunPurge has been called at all.
//
// Made deterministic at integration (2026-08-20): "Invoke returned, row still present"
// on its own is a RACE — the yielded continuation runs RunPurge on a pool thread and
// can delete the row before this thread reaches its assert (lost 2/2 full-suite
// sweeps, green in isolation). So the purge is parked on a gate instead: RunPurge
// consults the isActiveNode check BEFORE any delete, and the check blocks until the
// test releases it. Fixed code: Invoke returns while the gate is held (the continuation
// is the one waiting). Unfixed code: Invoke itself blocks on the gate, so the
// bounded wait below fails — no timing assumption in either direction.
InsertEventWithTimestamp(DateTimeOffset.UtcNow.AddDays(-31));
Assert.Equal(1, GetEventCount());
using var gate = new ManualResetEventSlim(initialState: false);
var purge = CreatePurgeService(isActiveNode: () => { gate.Wait(); return true; });
var executeAsync = typeof(EventLogPurgeService).GetMethod(
"ExecuteAsync", BindingFlags.NonPublic | BindingFlags.Instance);
Assert.NotNull(executeAsync);
using var cts = new CancellationTokenSource();
Task? executeTask = null;
var invoke = Task.Factory.StartNew(
() => executeTask = (Task)executeAsync!.Invoke(purge, [cts.Token])!,
TaskCreationOptions.LongRunning);
// With the gate held, Invoke can only return if ExecuteAsync yielded before RunPurge.
var returned = await Task.WhenAny(invoke, Task.Delay(TimeSpan.FromSeconds(10))) == invoke;
Assert.True(returned,
"ExecuteAsync did not return control before running the initial purge — " +
"it is running RunPurge synchronously ahead of its first await.");
Assert.Equal(1, GetEventCount());
// Release the purge; it still runs, just after control was yielded back.
gate.Set();
var deadline = DateTime.UtcNow.AddSeconds(5);
while (GetEventCount() != 0 && DateTime.UtcNow < deadline)
{
await Task.Delay(25);
}
Assert.Equal(0, GetEventCount());
await cts.CancelAsync();
try
{
await executeTask!;
}
catch (OperationCanceledException)
{
// Expected: the loop's Task.Delay observes the cancellation.
}
}
[Fact]
public void RunPurge_ActiveNodeCheckThrows_TickLogsAsInactive_NotActive()
{
// arch-review 2026-08-19 P4-5: a throwing isActive predicate must fail CLOSED
// (logged as inactive) rather than silently claiming "active". The predicate no
// longer gates any purge work (P4-4 — every node purges its own file every tick
// regardless), so a wrong value here cannot skip work; but an operator correlating
// this line against the real active-node state must not be misled into thinking the
// check actually succeeded and reported active.
var logger = new CapturingLogger();
var purge = CreatePurgeService(
isActiveNode: () => throw new InvalidOperationException("boom"),
logger: logger);
purge.RunPurge();
var warnLine = Assert.Single(
logger.Entries,
e => e.Message.Contains("Active-node check threw", StringComparison.Ordinal));
Assert.Equal(LogLevel.Warning, warnLine.Level);
var tickLine = Assert.Single(
logger.Entries,
e => e.Message.Contains("Running event log purge tick", StringComparison.Ordinal));
Assert.Contains("active=False", tickLine.Message, StringComparison.Ordinal);
}
[Fact]
public async Task PurgeByStorageCap_ConcurrentWritesDoNotCorruptConnection()
{
tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/ReconcileStartsInstanceTests.cs
+18 -13
View File
@@ -92,7 +92,8 @@ public class ReconcileStartsInstanceTests : TestKit, IDisposable
/// </summary>
private IActorRef CreateDeploymentManager(
IServiceProvider? serviceProvider = null,
Func<Task<List<DeployedInstance>>>? configLoader = null)
Func<Task<List<DeployedInstance>>>? configLoader = null,
Func<string, Task<DeployedInstance?>>? deployedConfigLoader = null)
{
var health = new DeploymentManagerStartupProbe();
var dm = ActorOf(Props.Create(() => new DeploymentManagerActor(
@@ -109,7 +110,11 @@ public class ReconcileStartsInstanceTests : TestKit, IDisposable
configFetcher: null,
// In position: an expression tree rejects a named argument that skips a parameter.
startupLoadRetryInterval: null,
configLoader: configLoader)));
configLoader: configLoader,
externalSystemCache: null,
scriptScheduler: null,
connectionDefinitionLoader: null,
deployedConfigLoader: deployedConfigLoader)));
AwaitCondition(() => health.ReportCount > 0, TimeSpan.FromSeconds(20),
TimeSpan.FromMilliseconds(25));
@@ -409,7 +414,7 @@ public class ReconcileStartsInstanceTests : TestKit, IDisposable
{
const string instanceName = "RacedDisablePump";
var loader = new GatedConfigLoader(_storage);
var dm = CreateDeploymentManager(configLoader: loader.LoadAsync);
var dm = CreateDeploymentManager(deployedConfigLoader: loader.LoadAsync);
await _storage.StoreDeployedConfigAsync(
instanceName, MakeConfigJson(instanceName), "dep-rd", "rev-rd", true);
@@ -451,7 +456,7 @@ public class ReconcileStartsInstanceTests : TestKit, IDisposable
{
const string instanceName = "RacedDeletePump";
var loader = new GatedConfigLoader(_storage);
var dm = CreateDeploymentManager(configLoader: loader.LoadAsync);
var dm = CreateDeploymentManager(deployedConfigLoader: loader.LoadAsync);
await _storage.StoreDeployedConfigAsync(
instanceName, MakeConfigJson(instanceName), "dep-rx", "rev-rx", true);
@@ -487,7 +492,7 @@ public class ReconcileStartsInstanceTests : TestKit, IDisposable
{
const string instanceName = "UnracedPump";
var loader = new GatedConfigLoader(_storage);
var dm = CreateDeploymentManager(configLoader: loader.LoadAsync);
var dm = CreateDeploymentManager(deployedConfigLoader: loader.LoadAsync);
await _storage.StoreDeployedConfigAsync(
instanceName, MakeConfigJson(instanceName), "dep-ok", "rev-ok", true);
@@ -510,15 +515,15 @@ public class ReconcileStartsInstanceTests : TestKit, IDisposable
.ResolveOne(TimeSpan.FromSeconds(5));
/// <summary>
/// A deployed-config loader that can be held open mid-read, so a test can commit a
/// lifecycle write while a heal's snapshot is provably in flight rather than hoping the
/// two race the right way round.
/// A single-instance deployed-config loader that can be held open mid-read, so a test can
/// commit a lifecycle write while a heal's snapshot is provably in flight rather than hoping
/// the two race the right way round.
///
/// <para>
/// Reads real storage every time and only parks AFTER the read, so the returned snapshot is
/// exactly what production would have seen at that instant. Inert until <see cref="Arm"/>
/// is called, which keeps the Deployment Manager's boot load (which runs through this same
/// seam) unblocked.
/// is called, which keeps the Deployment Manager's boot load — a separate seam
/// (<c>_configLoader</c>, register P1-4) untouched by this gate — unblocked.
/// </para>
/// </summary>
private sealed class GatedConfigLoader(SiteStorageService storage)
@@ -534,16 +539,16 @@ public class ReconcileStartsInstanceTests : TestKit, IDisposable
public Task WaitUntilParkedAsync() => _parked.Task.WaitAsync(TimeSpan.FromSeconds(10));
public async Task<List<DeployedInstance>> LoadAsync()
public async Task<DeployedInstance?> LoadAsync(string instanceName)
{
var configs = await storage.GetAllDeployedConfigsAsync();
var config = await storage.GetDeployedConfigAsync(instanceName);
var gate = _release;
if (gate != null)
{
_parked.TrySetResult();
await gate.Task;
}
return configs;
return config;
}
}
tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Persistence/SiteStorageServiceTests.cs
+74
View File
@@ -88,6 +88,80 @@ public class SiteStorageServiceTests : IAsyncLifetime, IDisposable
Assert.True(configs[0].IsEnabled);
}
// ── Register P1-4: projected single-row reads ──
//
// GetAllDeployedConfigsAsync + FirstOrDefault materializes every deployed config —
// including every ConfigJson blob — to find one row. These lock in that
// GetDeployedConfigAsync (single row, still carries ConfigJson) and
// GetDeployedConfigMetadataAsync (single row, deliberately WITHOUT ConfigJson) return
// exactly the same decision the old full-table-plus-FirstOrDefault path did — the
// defect here is pure I/O cost, not an observable behavior difference, so these tests
// pin the new projection's contract rather than reproduce a behavior bug.
[Fact]
public async Task GetDeployedConfigAsync_ReturnsSameRowAs_GetAllThenFirstOrDefault()
{
await _storage.StoreDeployedConfigAsync(
"Pump1", "{\"test\":true}", "dep-001", "sha256:abc", isEnabled: true);
await _storage.StoreDeployedConfigAsync(
"Pump2", "{\"other\":1}", "dep-002", "sha256:def", isEnabled: false);
var viaProjection = await _storage.GetDeployedConfigAsync("Pump1");
var viaFullScan = (await _storage.GetAllDeployedConfigsAsync())
.FirstOrDefault(c => c.InstanceUniqueName == "Pump1");
Assert.NotNull(viaProjection);
Assert.NotNull(viaFullScan);
Assert.Equal(viaFullScan.InstanceUniqueName, viaProjection.InstanceUniqueName);
Assert.Equal(viaFullScan.ConfigJson, viaProjection.ConfigJson);
Assert.Equal(viaFullScan.DeploymentId, viaProjection.DeploymentId);
Assert.Equal(viaFullScan.RevisionHash, viaProjection.RevisionHash);
Assert.Equal(viaFullScan.IsEnabled, viaProjection.IsEnabled);
Assert.Equal(viaFullScan.DeployedAt, viaProjection.DeployedAt);
}
[Fact]
public async Task GetDeployedConfigAsync_UnknownInstance_ReturnsNull()
{
await _storage.StoreDeployedConfigAsync(
"Pump1", "{}", "dep-001", "sha256:abc", isEnabled: true);
var result = await _storage.GetDeployedConfigAsync("NeverHeardOf");
Assert.Null(result);
}
[Fact]
public async Task GetDeployedConfigMetadataAsync_ReturnsSameIdentityAs_GetAllThenFirstOrDefault()
{
await _storage.StoreDeployedConfigAsync(
"Pump1", "{\"test\":true}", "dep-001", "sha256:abc", isEnabled: true);
await _storage.StoreDeployedConfigAsync(
"Pump2", "{\"other\":1}", "dep-002", "sha256:def", isEnabled: false);
var metadata = await _storage.GetDeployedConfigMetadataAsync("Pump2");
var viaFullScan = (await _storage.GetAllDeployedConfigsAsync())
.FirstOrDefault(c => c.InstanceUniqueName == "Pump2");
Assert.NotNull(metadata);
Assert.NotNull(viaFullScan);
Assert.Equal(viaFullScan.InstanceUniqueName, metadata.InstanceUniqueName);
Assert.Equal(viaFullScan.DeploymentId, metadata.DeploymentId);
Assert.Equal(viaFullScan.RevisionHash, metadata.RevisionHash);
Assert.Equal(viaFullScan.IsEnabled, metadata.IsEnabled);
}
[Fact]
public async Task GetDeployedConfigMetadataAsync_UnknownInstance_ReturnsNull()
{
await _storage.StoreDeployedConfigAsync(
"Pump1", "{}", "dep-001", "sha256:abc", isEnabled: true);
var result = await _storage.GetDeployedConfigMetadataAsync("NeverHeardOf");
Assert.Null(result);
}
[Fact]
public async Task StoreDeployedConfig_Upserts_OnConflict()
{
tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardServiceTests.cs
+30 -2
View File
@@ -786,6 +786,29 @@ public class StoreAndForwardServiceTests : IAsyncLifetime, IDisposable
while (Interlocked.CompareExchange(ref target, value, current) != current);
}
/// <summary>
/// Register row-40 house pattern for an ABSENCE assertion (mirrors the Medium-wave
/// <c>SiteAlarmAggregatorActorTests.AssertHoldsFor</c> treatment, ported here async since
/// this class's facts run on the async test thread): proves <paramref name="holds"/> keeps
/// being true for the whole <paramref name="window"/> by polling, rather than a single
/// blind <c>await Task.Delay(window)</c> followed by one end-of-window sample. A bare
/// delay-then-assert cannot distinguish "the mechanism correctly stayed quiet" from "the
/// mechanism just hadn't run yet when the one sample was taken", and it can't catch a
/// value that flips wrong and then flips back before that single sample. This fails
/// immediately the first time <paramref name="holds"/> goes false, and only returns once
/// the full window has elapsed with it still true throughout.
/// </summary>
private static async Task AssertHoldsForAsync(TimeSpan window, Func<bool> holds, string because)
{
var deadline = DateTime.UtcNow + window;
while (true)
{
Assert.True(holds(), because);
if (DateTime.UtcNow >= deadline) return;
await Task.Delay(20);
}
}
// ── Task 12: deferToSweep enqueue mode ──
/// <summary>
@@ -888,8 +911,13 @@ public class StoreAndForwardServiceTests : IAsyncLifetime, IDisposable
service.TriggerSweep(); // the clobbering kick
var stop = service.StopAsync();
await Task.Delay(300);
Assert.False(stop.IsCompleted); // pre-fix: StopAsync already returned (awaited the no-op)
// The real sweep is blocked on `release` behind `entered`, so `stop` is genuinely
// unable to complete during this window (not merely "hasn't yet") — proving it stays
// incomplete throughout, not just at one end-of-delay sample, is what actually
// distinguishes "waited for the real sweep" from "pre-fix: awaited the no-op and
// returned instantly, then this single sample raced whether that had happened yet".
await AssertHoldsForAsync(TimeSpan.FromMilliseconds(300), () => !stop.IsCompleted,
"pre-fix: StopAsync already returned (awaited the no-op)");
release.TrySetResult();
await stop.WaitAsync(TimeSpan.FromSeconds(5)); // drains the real sweep promptly once released