Register row 50. Row 45 (fixed 9e769735) was a live captive dependency —
NotificationOutboxActor cached delivery adapters resolved from an actor-lifetime
IServiceScope, so MaxParallelDeliveries concurrent deliveries shared one EF
DbContext and central-b logged EF's "second operation was started on this context
instance" continuously. Actors take the ROOT IServiceProvider rather than typed
services, which is invisible to build-time analysis, so the bug class had no
mechanical guard at all. This adds both of the register's mitigations; neither
subsumes the other.
1. ValidateScopes + ValidateOnBuild in Development, on BOTH branches, via
ServiceProviderValidation.Configure. WebApplication.CreateBuilder already sets
both flags to IsDevelopment(), so on paper this restates a default — it is
written out because that default is silent (a future move off CreateBuilder
drops it with no diagnostic) and because this host carries three
environment-ish signals (SCADABRIDGE_CONFIG, DOTNET_ENVIRONMENT,
ASPNETCORE_ENVIRONMENT), so "is this node in Development" is not obvious at the
call site. Production stays unvalidated deliberately: a validation failure is
fail-closed, and CI now proves what a production node will not check.
This surfaced two REAL latent violations on the SITE role — not captive
dependencies but unconstructible descriptors, which fail ValidateOnBuild for
the entire graph and are therefore what kept the site branch from being
validatable at all (SiteHealthEndpointTests had pinned UseEnvironment
("Production") to work around exactly this):
- ReconcileService (scoped, needs the central-only IDeploymentManagerRepository)
moved out of the shared AddCommunication() into a new AddCentralCommunication().
- AuditLogKpiSampleSource (scoped IKpiSampleSource, needs the central-only
IAuditLogRepository) moved out of the shared AddAuditLog() into a new
AddAuditLogCentralKpi().
Both were registered on sites under the argument "harmless — only the central
actor ever resolves it". True of the runtime, false of the container: a
registration nothing can build is what stops the validator being usable, and it
masks whatever real defect the validator would have found. Both are now
central-only and SiteHealthEndpointTests boots the site branch validated again.
2. CaptiveDependencyGuard: a reflection scan over every actor class in the shipped
assemblies (28 today; the reference graph is walked from the Host assembly
because referenced assemblies load lazily). Fails on any instance field — auto-
property backing fields included, they are ordinary private fields — typed as
IServiceScope, as a DbContext, or as a service the REAL composition roots
register Scoped. The scoped set is derived from the built Central and Site
containers rather than hand-listed, so a lifetime change automatically re-scopes
the check. IServiceProvider / IServiceScopeFactory fields are explicitly allowed:
that IS the correct per-operation-scope pattern.
No live violation exists — every actor opens a scope per operation. Two fields
are allowlisted with their safety argument rather than exempted silently:
AuditLogIngestActor._injectedRepository and SiteCallAuditActor._injectedRepository
are scoped repository types assigned ONLY by explicitly test-mode constructors,
null under production wiring (AkkaHostedService uses the IServiceProvider
overload in both cases), and SiteCallAuditActor additionally wraps its injected
instance in a SerializedRepository. The shape is one constructor call away from
being row 45 again, so it is recorded, not dismissed.
Tests: 10 new. Six permanent self-tests of the guard, including positive controls
that reproduce the pre-row-45 shape (an actor caching a scoped service, an
IServiceScope, a DbContext, and a scoped auto-property) and a negative control for
the allowed root-provider pattern — a reflection guard that matches nothing passes
just as green as one that works. Plus an actor-count floor with cross-assembly
spot-checks, an allowlist-staleness test that fails any entry no longer matching a
flagged field, and the two per-role scans. Verified non-vacuous by emptying the
allowlist: the Central scan then names both real fields. Two composition-root
build tests force both validation flags on per role regardless of environment.
Doc: REQ-HOST-11 in Component-Host.md records all three rules and their guards.
The per-IP unlock throttle's counters were a process-local ConcurrentDictionary
on each central node, so the effective fleet cap was N x
MaxUnlockAttemptsPerIpPerHour per client (failing over to the other node handed
the client a fresh budget) and a node restart cleared the window outright.
The window now lives in the central MS SQL BundleUnlockAttempt table — the HA
store both central nodes already share, so no new infrastructure, no new port,
no consensus protocol:
- Commons: BundleUnlockAttempt entity + IBundleUnlockAttemptRepository.
- ConfigurationDatabase: entity config (one index,
IX_BundleUnlockAttempt_Client_Attempted, serving both the windowed count and
the prune), repository (AsNoTracking count, ExecuteDeleteAsync prune),
DbSet, DI registration, migration AddBundleUnlockAttemptTable + the
hand-written idempotent production script under docs/plans/sql/.
- Transport: IBundleUnlockAttemptWindow seam + SqlBundleUnlockAttemptWindow
(singleton; resolves the scoped repository from its own scope per call, like
the central purge singletons). BundleUnlockRateLimiter keeps the policy and
gains TryRegisterAttemptAsync; BundleImporter.LoadAsync calls it.
Behaviour preserved from row 44: locked bundles spend no hourly budget (the
lockout check still runs first), throttle denials burn no per-bundle strike and
are NOT recorded so they cannot roll the window forward, unattributable clients
share the stricter <unattributed> bucket, and the budget counts attempts rather
than failures. Config keys and their semantics are unchanged, so the rig's 500
override stays valid.
Three deliberate trades, documented in both Transport docs:
- FAILS OPEN. A store error logs a Warning naming the client and cap, and
permits the attempt. This is a best-effort rate cap over an already
authenticated, Administrator-gated surface — the availability of legitimate
imports outranks the cap, and a DB blip must not make the import wizard
unusable. Falling back to the process-local window was considered and
rejected: it would still block a legitimate operator's 11th import.
- Approximate. Count-then-insert is not atomic across nodes, so simultaneous
attempts can overshoot by the number in flight — bounded, self-correcting
within the window, and not worth provider-specific raw SQL.
- Pruned opportunistically on the unlock path (CAS-claimed at most every 15
min per process, bounded to 1000 rows, retaining 2x the window, best-effort
try/catch) rather than by a new background service — Transport owns no
central scheduler. Same shape as DeploymentService's terminal-record sweep.
Tests: Transport.IntegrationTests 116 passed (+10 — two limiter instances over
one store enforcing a single combined window, restart durability, denials
burning no budget, sliding release, bucket isolation, store-failure fail-open,
caller-cancellation not swallowed, no-shared-window fallback, prune and its
negative control); ConfigurationDatabase 395 passed (+5).
Nothing counted open import sessions and nothing logged a rejection at
MaxConcurrentImportSessions (default 8), so the register-row-43 session leak
could only be inferred from rotating test failures.
- Gauge scadabridge.central.transport.import_sessions.open on the existing
ZB.MOM.WW.ScadaBridge meter — already in SiteServiceRegistration.ObservedMeters
(the ZbTelemetryOptions.Meters allowlist), so no telemetry config change is
needed and the series is not silently dropped. Provider-backed, mirroring the
StoreAndForward queue-depth gauge: registered by BundleSessionEvictionService
.StartAsync (Transport's only lifecycle seam) and identity-checked-cleared on
StopAsync so a late stop cannot clobber a newer registrant.
- IBundleSessionStore.OpenSessionCount counts only UNEXPIRED sessions. The raw
dictionary count would report the node as pinned at the cap for up to a
one-minute sweep interval after a burst had drained — the exact false positive
the instrument exists to rule out.
- BundleSessionStore.Open logs Warning with open count, cap and TTL when it
rejects. Fires only on a real cap hit: a store full of expired sessions is
reclaimed first and logs nothing.
- Docs: docs/components/Transport.md + docs/requirements/Component-Transport.md.
Tests: Transport 173 passed (+6 new — Warning content, two negative controls,
unexpired-only counting, gauge register/clear, no-clobber-on-late-stop).
ManagementService references Commons but not Transport, so MapFault could not
see BundleLockedException and the per-bundle 3-strike lockout reached the CLI
as "An internal error occurred (CorrelationId=...)" — the operator got a
correlation id instead of the reason, and no hint that further attempts on
that bundle would be refused.
Applies the pattern register row 44 established for BundleUnlockThrottledException:
- Relocate BundleLockedException from Transport/Import/ to
Commons/Types/Transport/, beside BundleUnlockThrottledException. Both
consumers (BundleImporter, CentralUI TransportImport) already carried the
Commons.Types.Transport using, so no call-site edits were needed.
- Add curated catch arms in ManagementActor.HandlePreviewBundle and
HandleImportBundle that rethrow as ManagementCommandException, so the
message survives MapFault's disclosure filter.
- Mirror the row-44 throttle fault-mapping test for both handlers.
- Document the relocation in docs/components/Transport.md and
docs/requirements/Component-Transport.md.
Tests: ManagementService 499 passed (+2 new), Transport 167 passed.
BundleUnlockRateLimiter was registered (ServiceCollectionExtensions), injected
into BundleImporter, unit-tested, documented on
TransportOptions.MaxUnlockAttemptsPerIpPerHour = 10 and pinned in
appsettings.Central.json — but _unlockRateLimiter was never invoked. A
brute-force passphrase attacker against a stolen bundle was bounded only by the
per-bundle 3-strike lockout, which a re-export side-steps (new ContentHash).
LoadAsync now registers every unlock attempt against the limiter. Ordering is
load-bearing: AFTER the per-bundle lockout check (hammering a locked bundle
spends no hourly budget) and BEFORE the decrypt (so a denial burns no per-bundle
strike — otherwise the throttle would be a way to lock a bundle someone is
legitimately importing). Over budget throws the new
BundleUnlockThrottledException, fail-closed with a clear message on both
surfaces; it lives in Commons because the management surface (Commons-only
references) must catch it to return a curated error rather than MapFault's
generic internal-error text.
Client key plumbing:
- Central UI: new scoped CircuitClientAddressProvider snapshots
Connection.RemoteIpAddress at circuit establishment — the same constraint and
the same pattern as CookieAuthenticationStateProvider, since HttpContext is
null by the time a circuit submits a passphrase.
- CLI/management: additive ManagementEnvelope.ClientIp, stamped by
ManagementEndpoints and threaded through DispatchCommand to the preview/import
bundle handlers.
- No attributable address => a single SHARED "<unattributed>" bucket, not a
bypass: unknown clients get a stricter budget, never none. Per-client
isolation is only real behind a ForwardedHeaders-trusted proxy, the same
caveat the login throttle carries (Component-Security.md).
Docs: Component-Transport.md + docs/components/Transport.md record the two-level
ordering, the client-key sourcing, the shared-bucket fallback and a
troubleshooting entry for the new error.
Tests: 6 importer tests (cap enforced; denial != passphrase failure, no strike
burned; per-client isolation; shared unattributed bucket; unencrypted bundles
consume no budget; a locked bundle spends no hourly budget) — 4 of the 6 fail
with the limiter call disabled, i.e. against the dead-code state. Plus
ManagementActor tests pinning the ClientIp forwarding and the curated
throttle error.
Register row 45. NotificationOutboxActor.ResolveAdapters() cached adapter
instances in an actor-lifetime IServiceScope (_adaptersScope), but both
EmailNotificationDeliveryAdapter and SmsNotificationDeliveryAdapter are
AddScoped and hold a scoped INotificationRepository over one EF DbContext.
MaxParallelDeliveries (default 4) then drove up to four concurrent deliveries
through that single adapter and single context, which central-b logged
continuously on the rig as "InvalidOperationException: A second operation was
started on this context instance before a previous operation completed". The
collided delivery's row is never written and the wasted attempt counts against
its retry budget, so sustained contention could park a legitimate notification.
The per-delivery scope DeliverGatedAsync already opened for the outbox
repository — added for exactly this hazard — did not cover the adapter.
_adaptersCache, _adaptersScope and ResolveAdapters() are deleted. Delivery
adapters are now resolved inside the per-delivery scope via a static
ResolveAdapter(IServiceProvider, NotificationType), preserving DI's last-wins
resolution semantics (the enumeration runs to completion rather than
short-circuiting) and the null-adapter park path. Nothing shared is lost: the
Email adapter builds a fresh SMTP client per attempt from an injected factory
and the SMS adapter draws pooled connections from the singleton
IHttpClientFactory, so no adapter carried state worth preserving across
deliveries. The Central UI's NotificationChannelCatalog is an independent
consumer of the same scoped registrations and is untouched, so the
adapter-gated Type selector still enumerates exactly the deliverable channels.
Tests: Dispatch_ResolvesAdaptersOnce_AcrossMultipleSweeps (which pinned the
superseded caching behaviour) is replaced by
ParallelDeliveries_EachGetTheirOwnAdapterAndDbContext — two parallel deliveries
through a scoped repository whose simulated DbContext throws EF's exact
second-operation exception on overlap, with a rendezvous that guarantees the
overlap window — and Dispatch_ResolvesAdapterPerDelivery_NotOncePerActorLifetime.
Both fail against the pre-fix actor (stash-verified: 1 of 2 rows persisted; 1
distinct adapter instead of 4) and pass after. Parallel-delivery
ordering/throughput semantics are unchanged.
Docs: Component-NotificationOutbox.md gains the scoped-adapter contract under
Delivery Adapters (resolve per delivery; adapters carry no cross-delivery
state; genuinely shared resources belong in a singleton/factory), and the
NotificationOutbox-006 finding is marked SUPERSEDED with the reasoning.
Solution build clean; NotificationOutbox (153), AuditLog (383), integration
NotificationOutbox flow (2) and CentralUI notification (14) suites green.
Only BundleImporter.ApplyAsync released an import session, so every path that
opened one and did NOT apply held one of MaxConcurrentImportSessions (8) — and
the DECRYPTED bundle content that T-007 deliberately zeroes on the apply paths —
for the full BundleSessionTtlMinutes (30). Eight abandonments locked out all
imports on the active node; observed live (R6: 3 back-to-back suite runs held 6
of 8 slots until central-b was restarted).
New IBundleImporter.Abandon(sessionId): the same zero-then-remove release
ApplyAsync performs, idempotent (double release / unknown id / expired id are
no-ops) and only ever given the random-GUID id its caller opened, so it can
never evict another circuit's session.
Wired into every abandonment path:
- Central UI wizard — ReleaseSession() from new upload, TryLoadAsync (before it
overwrites _session), Back-to-Upload / ResetSessionState, the 3-strike lockout
reset, and Dispose (navigation away / circuit teardown).
- CLI transport preview — a preview never applies, so nothing released it at
all; now released in a finally.
- CLI transport import — covers the pre-apply aborts (blocker rows, unresolved
site/connection mappings).
Class doc's "an abandoned wizard does not leak" claim is true again (it only
ever covered the staged plaintext temp file). Component-Transport.md +
docs/components/Transport.md now record the explicit-release contract and that
BundleSessionEvictionService is a TTL backstop, not an abandonment path.
Tests: 4 bUnit tests over a REAL BundleSessionStore (dispose / back-to-upload /
apply-then-dispose idempotency / concurrent-circuit isolation) — 3 of the 4 fail
against the pre-fix page; 4 importer tests (release + zeroing, idempotency,
cross-session isolation, and the slot-cap regression).
Root-caused from a full-suite-load-only failure of
GrpcStreamIntegrationTests.Pipeline_DuplicateCorrelationId_ReplacesStream:
ObjectDisposedException escaping SubscribeInstance.
SubscribeInstance's duplicate-prevention path cancelled AND disposed the
replaced stream's CancellationTokenSource. That CTS belongs to the replaced
handler's own `using var streamCts`, which is still running and still has to
read `streamCts.Token` — the Dispose raced that read. The race is pre-existing
and independent of R2: the same Dispose sat against the same first-token-read
when the handler still used `ReadAllAsync(streamCts.Token)`; it only ever
loses under enough scheduling pressure to land the replacement inside the
first stream's setup window.
Cancel only. The owning handler's `using` still disposes it exactly once on
every exit path, and cancellation is all replacement ever needed. The Cancel
is wrapped for the converse race (owner already finished and disposed),
mirroring CancelAllStreams().
Regression test makes the race deterministic by gating the first stream inside
its setup window — the _activeStreams entry is registered before Subscribe is
called, so the replacement always lands before the first stream reads its
token. Verified fail-before (ObjectDisposedException) / pass-after by
reinstating the Dispose as a negative control.
Every AttributeValueChanged/AlarmStateChanged rode its own gRPC message on
SiteStreamService; at target scale that is ~37.5k messages/s/site of pure
framing overhead. Coalesce them, additively, with no new RPC.
Wire (sitestream.proto, regenerated via docker/regen-proto.sh sitestream):
InstanceStreamRequest.batching_supported = 3
SiteStreamRequest.batching_supported = 2
SiteStreamEvent.batch = 4 (new oneof case)
SiteStreamEventBatch { repeated SiteStreamEvent events = 1 }
The proto3 default of batching_supported IS the negotiation, and it is
load-bearing: a batch frame reaches a pre-R2 central as EventOneofCase.None,
whose ConvertToDomainEvent returns null — the whole batch would vanish with
no error anywhere. An old central cannot set the flag so it never receives
one; an old site ignores the unknown request field and keeps sending
per-event frames, which the new client's ForEachEvent handles as the
single-event case. Both skew directions are covered by tests that go through
a real proto serialize/parse round-trip.
Server: SiteStreamEventBatcher, a per-subscriber pump replacing the handler's
await-foreach/WriteAsync loop and byte-identical to it at a cap of 1. It
never delays a lone event — it drains the already-queued backlog for free and
lingers only once a backlog is proven — emits a single-event buffer as a
plain frame, preserves order and per-event Timestamps exactly, and flushes
what is buffered when the send channel's writer completes. It sits DOWNSTREAM
of StreamRelayActor's bounded DropOldest channel, so it changes framing only
and does not move the burst ceiling (deferred register row 31, which lives in
the shared publish stage upstream of the BroadcastHub).
Client: sets the flag on both subscriptions and unpacks in order into the
existing per-event pipeline, so SiteAlarmAggregatorActor,
DebugStreamBridgeActor, consumer-keepalive/orphan logic,
reconnect-on-graceful-completion, generation fencing, the (siteId, endpoint)
factory key and IsLive semantics are untouched.
Options (validated): GrpcStreamBatchMaxEvents 100 (1 disables),
GrpcStreamBatchWindow 25 ms — validated strictly under 250 ms, the load
test's end-to-end P99 threshold. Measured worst case (trickle-with-backlog,
window-bound rather than cap-bound): P50 13.8 ms, P99 25.4 ms, max 25.8 ms;
cap-bound case sent 600 queued events in 6 frames.
Telemetry: histogram scadabridge.site.stream.batch_size tagged by stream
kind, recorded only on negotiated streams (per-event otherwise). It rides
ScadaBridgeTelemetry.MeterName, already in the ObservedMeters allowlist.
Docs: Component-Communication.md gains an Event Batching section; CLAUDE.md's
gRPC streaming bullet records the wire shape and the negotiation rationale.
Deferred flake-pattern sweep of tests/ for the class fixed in c4caebe9 and
cfa6acbf — a bounded wait on observable A followed by a bare assert on an
observable B that the product only reaches strictly after A. Three clear
instances, each reproduced deterministically by delaying only the later step
and each re-verified green with that same delay still injected.
AlarmOnTriggerRuns_ShedAtTheSameCap_WithAnAlarmScopedSiteEvent gated on the
rate-limited shed site event and then asserted the shed COUNT bare.
AlarmActor.ShedAlarmRun increments the counter and only then emits the event,
and the event fires on the first shed only — so the gate observed Flap(4)'s
shed and ordered nothing with respect to Flap(5)'s, which is a separate
mailbox message with no observable of its own (an alarm on-trigger run has no
Ask caller to reply to, unlike ScriptActor.ShedRun, whose sibling test is
correctly ordered by its ScriptCallResult and is left alone). Deferring
Flap(5) by 2 s failed it with "Expected: 2 / Actual: 1". The count is now
ACCUMULATED across polls rather than re-read, because
SiteHealthCollector.CollectReport DRAINS the interval counters — a poll loop
that simply re-read it would consume the first shed and never reach 2.
EndToEnd_GrpcStubError_RowStays_Pending_NextTick_Succeeds gated on the
central row arriving and then asserted bare that the site SQLite row had left
Pending. SiteAuditTelemetryActor pushes via IngestAuditEventsAsync (which is
what writes the central row) and calls MarkForwardedAsync only after parsing
the ack. Delaying just that post-push step failed it with
"Assert.DoesNotContain() Failure: Filter matched in collection".
PreSnapshotBuffer_IsCapped_DropsOldest_AndCountsTheDrops gated on
"Count >= cap" and then asserted "Count == cap + 1" bare — a gate strictly
weaker than the assertion it guards, so it ordered nothing with respect to
the last event of a FlushBuffer loop that delivers one at a time. Parking
that loop after its 19,999th delivery failed it with
"Expected: 20001 / Actual: 20000".
Also hardens GrpcCentralTransportTests.WaitUntil, which returned silently on
timeout; today's single caller re-asserts immediately, so this only sharpens
the message rather than fixing a live flake.
Cleared with evidence, not guessed: SiteAlarmLiveCacheService's LingerStop
removes the site entry inside one lock, so IsLive and GetCurrentAlarms flip
atomically; and SiteReconciliationActor walks response.Gap with a sequential
foreach in which the asserted "Gone" log precedes the awaited "Good" row, the
inverse of this class.
Test-only; every ordering named above is correct as written.
Sandbox_LongRunningScript_TimesOut and Sandbox_InfiniteLoop_CancelledByToken
armed a CancellationTokenSource for a fixed 100 ms / 500 ms and hoped it
expired while the script body was still running. That is a race between a
fixed timer and a fixed amount of work, not a synchronization, and it is
wrong in both directions.
Too slow a timer relative to the work and the script FINISHES first, so
nothing is cancelled and ThrowsAnyAsync fails with "No exception was thrown".
Measured on this machine: the bounded 100M-iteration loop takes 298 ms
against the 100 ms pin — a 3x margin that a faster host closes, and that any
move off the scripting default OptimizationLevel.Debug would close outright.
Reproduced deterministically by shrinking the loop to 1M iterations: it ran
in 76 ms and the test failed with exactly that message while all 27 siblings
passed.
Too fast a timer and the token is already cancelled before the body is
entered — Roslyn's runner throws OperationCanceledException up front
(verified with a pre-cancelled token) — so the test goes GREEN without the
script's own in-loop ThrowIfCancellationRequested ever being reached. That
vacuous pass is the worse half: it asserts nothing while looking healthy.
Both now cancel deterministically. The script invokes an Action handed in
through Parameters from inside its own loop; when that call returns the token
is already cancelled ON THE SAME THREAD, so the next in-loop check is
guaranteed to observe it, with ~10,000 checks still ahead of it. No wall
clock, no host-speed or scheduler dependence — and the 600 ms of sleeping
goes away. Sandbox_UncancelledScript_RunsToCompletion is added as the
negative control: the same script with the signal wired to a no-op must run
every iteration and return the closed-form sum, which is what establishes
that the sibling's OCE is caused by the cancellation.
Verified: the injection that killed the old test passes with the fix, and
suppressing the cancellation entirely still fails it with the identical
message, so the claim is unchanged in force. Test-only; the sandbox's
cancellation behaviour is correct as written.
All 14 pre-existing Playwright failures triage to selector drift, not rig data,
timing, or application defects — every one is a test-side assumption that two
merged UI refactors invalidated without updating this suite.
`9e243493` (2026-08-11 density sweep + Theme 0.4.1) regrouped page/dialog action
buttons into `btn-group btn-group-sm`, moving the size class off the button and
onto the group. Twelve selectors still pinned `.btn-sm` on the button itself:
ConfigurationAuditLog's Search (4 tests), the Topology Create/Move dialog footers
(3), the Notification Lists row Edit/Delete (2), and the Notification Report row
Retry/Discard (2). The same sweep re-cast SiteForm's per-node subsections from
bare `<h6>` headings to Bootstrap cards, so `h6:has-text('Node A')` matches
nothing (1). Note the swept surfaces are a subset — the recipient table on the
notification-list edit form, OffsetPager, DiffDialog and friends still carry
`.btn-sm` on the button, so the selectors are deliberately asymmetric; the
remaining `.btn-sm` selectors in this suite were each re-verified against current
markup rather than swept along.
`a506b19d` moved TemplateEdit's page-embedded modals onto the DialogService host.
The attribute modal is now the DialogHost modal (`modal fade show d-block`, title
in an h5), so the `:not(.fade)` guard written to exclude DialogHost — plus the
`h6.modal-title` assertion — excluded the very modal under test (2). TemplateEdit
renders no page-local modal any more, so the guard has nothing left to guard.
Comments at each site record why the size class is absent, so a future reader
does not "fix" the selectors back.
Verified: 14 failed / 158 passed baseline -> the 36 tests in the seven affected
classes all green; solution build 0 warnings / 0 errors.
The Deployment Status page client-materialized the whole deployment list. It read
EVERY DeploymentRecord — an insert-only table, one row per deploy attempt for the
retention window — plus EVERY Instance, then site-scoped, sorted, counted the four
status tiles and sliced a 25-row page in the Blazor circuit's memory. That ran on
first render AND on every IDeploymentStatusNotifier push, so the cost scaled with
the age of the system rather than the size of the page.
All four jobs move into SQL:
- `IDeploymentManagerRepository.QueryDeploymentListPageAsync(filter, page, size)`
returns one page of `DeploymentListRow` — DeploymentRecord INNER JOINed to
Instance, so the instance display name and site travel with the rows that need
them — plus the total count of the filtered set. The join is exact: the FK is
Restrict and DeleteInstanceAsync removes the records first, so no orphan exists.
- `GetDeploymentStatusCountsAsync(filter)` returns the tile counts from ONE grouped
aggregation, deliberately ignoring the filter's Status: the tiles are the status
BREAKDOWN of the filtered set, so honouring it would zero three of four tiles the
moment an operator clicked one.
- Site scoping runs in the query as `SiteIdScope` resolved through the record's
instance (DeploymentRecord has no SiteId of its own). An EMPTY grant stays a real
filter matching nothing, never "unconstrained".
- The now-callerless whole-table `GetAllDeploymentRecordsAsync` is deleted.
OFFSET paging, not the Audit Log's keyset cursor, and deliberately so: this page's
pager is numbered and jump-to-any-page, so it needs a page count, which only a total
can give it — a keyset cursor can express neither, and the total is required for the
tiles regardless. The deep-offset cost that pushes high-volume tables to keyset is
bounded here by the terminal-record retention purge, unlike the 365-day AuditLog.
This mirrors the Notification Outbox, offset-paged for the same reason. Ordering is
DeployedAt DESC, Id DESC — the Id tie-break is load-bearing, because DeployedAt ties
on rapid redeploys and an unstable sort key makes offset paging repeat or drop rows.
UI: the four status tiles become the status filter (click to apply, click again to
clear, aria-pressed, phrasing-only content so a <button> stays valid), plus a
free-text search matched DB-side against instance name, deployment id, revision hash
and initiating user. Search is TRAILING-edge debounced at 500ms — the same Timer +
lock + _disposed idiom as the existing leading-edge push coalescer, minus the leading
edge, because a search box must not query on the first keystroke. A filter change
resets to page 1; a page past the end falls back to the last real page. Bootstrap
only, existing PagerWindow pager retained.
The WP2.4 push coalescing is unchanged and still earns its keep: server paging shrank
what a reload costs, not how many arrive — it now bounds database round-trips rather
than table scans.
Tests: 19 new SQLite repository tests (paging slice + total, tie-break stability
across pages, page/size clamping, past-the-end, the joined projection, every filter
dimension incl. the empty-scope security case, and the grouped counts' status-blind
contract); 15 new bUnit page tests (page-1 request, server total drives the pager,
Next re-queries, tiles show server counts not page counts, tile filter + toggle,
system-wide vs site-scoped scope push, debounce collapses a keystroke burst to one
query, clear-filters, dispose with an armed timer). The two existing Deployments
suites re-point their reload assertions at the new query.
Doc: Component-CentralUI.md Deployment section — the "no server-side paging" known
residual is replaced by the shipped design.
The daily site_events retention purge (and the storage-cap trim) is CDC-captured
on a replication-enabled site node exactly like any other write — correct by
design, since LocalDb Phase 2 deliberately has no purge-exemption path — so the
backlog jumps by the deleted batch size at purge time. LocalDbOplogBacklog /
localdb_oplog_depth spike, drain, and an operator watching the gauge with no
context reads it as a replication fault.
Documentation + one log line, no behaviour change:
- topology-guide.md gains "Reading the replication backlog — the daily
site_events purge burst": when it fires (PurgeInterval 24h, anchored to the
active node's PROCESS START, not a wall-clock hour, so it moves after every
failover), where it shows (replicated nodes only — not rig site-b/site-c),
the healthy signature (LocalDbReplicationConnected stays true, backlog
returns to ~0) and what a genuine fault looks like instead.
- Component-SiteEventLogging.md Storage records the same under retention/purge;
Component-HealthMonitoring.md gains the two previously-undocumented
LocalDbReplicationConnected / LocalDbOplogBacklog metric rows carrying the
caveat, with cross-references both ways.
- EventLogPurgeService emits one Information line naming the row count and the
expected transient backlog when a purge deleted rows on a replication-enabled
node, so the spike is correlatable in the log. Replication-awareness comes in
as a Host-supplied SiteEventLogReplicationCheck delegate, mirroring the
existing SiteEventLogActiveNodeCheck seam: SiteLocalDbSetup.ReplicationIsConfigured
goes internal so the PeerAddress-OR-ApiKey rule stays in one place and
SiteEventLogging never learns to read LocalDb config. Unregistered ⇒ no note,
matching the default that replication is opt-in and off.
Both delete paths carry the note (a cap trim is usually the larger burst); the
predicate is try/caught since a log-wording check must never break the purge.
Tests: 5 new EventLogPurgeServiceTests cases (replicated logs it, unreplicated
does not, zero-rows does not, cap purge logs it, throwing predicate still purges
and swallows) via a local capturing ILogger. SiteEventLogging 81/81 green,
Host 490/490 green, full solution build clean (0 warnings).
Full-scale run executed on this machine: 10 sites x 500 instances x 75 tags =
375,000 live tag subscriptions, 37,518 updates/s achieved vs 37,500 nominal,
45,021,375 updates over a 20-minute steady-state window (M4 Pro, 14 cores, 48 GB,
with the 8-node docker rig still running so the figures are pessimistic).
11 clean passes, 1 pass with a caveat, 0 failures:
tag latency P50 0.88ms P99 4.57ms max 37.41ms (1.1M samples, end-to-end)
stream 900,675 delivered, 0 dropped at 100 live subscribers
health collect+ingest P99 0.31ms, 10/10 sites tracked
debug view P99 2.19ms, 264 completed, 0 timeouts
deploy 500 instances to a site in 2.6s
cpu 41% of ONE core = 2.9% of the box
memory working-set slope +8.83 MB/min
Three findings, reported rather than tuned away:
F1 (Low) 20 min with ZERO gen-2 collections cannot fully settle the leak
question; the heap demonstrably sawtooths but an uncompacted gen-2 makes a
positive slope ambiguous. The 1-hour run would settle it. Not tuned.
F2 (informational, by design) a deferred S&F backlog sits for one full
DefaultRetryInterval (28.9s measured) before anything drains --
EnqueueAsync(attemptImmediateDelivery:false) stamps LastAttemptAt. Easy to
misread as slow drainage, so drain is reported as two numbers: retry wait,
then 3,533 msg/s of actual capacity.
F3 (positive) slow-subscriber isolation is TOTAL: 4 healthy subscribers at
100.00% with zero drops while a peer lost 197,028/200,000 events entirely
within its own bounded channel. Mechanism recorded link by link.
Also records what the run does NOT prove (not clustered, not real-network, not
real OPC UA, not 1 hour) and the four WP-4 sub-criteria this harness does not
cover, so the evidence is not over-read.
Register rows 25 and 50 -> RESOLVED 2026-08-15; remediation execution-log
residual 7 -> resolved; phase-8-checklist WP-4 section replaced with the
measured numbers.
Standalone console harness under tests/ZB.MOM.WW.ScadaBridge.LoadHarness plus a
scaled-down Category=Performance smoke [Fact] in PerformanceTests. Deliberately an
Exe rather than an xunit suite: the Performance trait enables a filter but does not
exclude by default, so a 20-minute test would run on every 'dotnet test' of the slnx.
What is real: per-site ActorSystem + LocalDb SQLite file, the real DCL
(DataConnectionManagerActor/DataConnectionActor over a SimulatedDataConnection
registered through the documented DataConnectionFactory.RegisterAdapter seam), real
InstanceActors fed real TagValueUpdates, the real SiteStreamManager, real
StreamRelayActor + production-capacity bounded DropOldest channel, real
StoreAndForwardService/Storage, real SiteHealthCollector + CentralHealthAggregator.
Only the socket hops are stood in for.
Measures: end-to-end tag update latency (the emit instant rides
TagValueUpdate.Timestamp verbatim to the subscriber), instance ramp, memory
growth/CPU over a steady-state window, health report and debug view latency under
load, S&F concurrent buffering + drain throughput, and slow-subscriber isolation.
Closes arch-review remediation residual #1 (DCL unsubscribe-during-reconnect
count staleness).
DataConnectionActor tracked TotalSubscribedTags/ResolvedTags as two int fields
incremented and decremented at five independent sites. ReSubscribeAll clears the
very maps those decrements key off (_subscriptionIds, _unresolvedTags) while
deliberately preserving _subscriptionsByInstance, so an unsubscribe landing
inside a reconnect window matched NEITHER decrement branch: the total leaked +1
per subscribe/reconnect/unsubscribe churn cycle, permanently and cumulatively.
The 37f13e2e discard gate stopped the orphan-handle half of that race; it could
not stop the counters drifting, because they were state of their own.
Both counts are now DERIVED at report time from the authoritative per-tag
collections, which makes the drift unrepresentable rather than merely guarded:
total = _instancesByTag.Count (the per-tag counted set the residual
called for — distinct tags with at least
one subscribing instance)
resolved = _subscriptionIds.Count (tags for which the adapter holds a handle)
Two semantic corrections fall out of the derivation:
- A tag whose subscribe failed at CONNECTION level now counts toward the total.
It was excluded before, yet the reconnect re-subscribe re-issued it from
_subscriptionsByInstance and booked it as resolved — resolved above total, and
a total driven negative by the eventual unsubscribe.
- _tagSubscriberCount is deleted. It duplicated _instancesByTag exactly, so
HandleUnsubscribe's last-subscriber test is now "did UnindexTag drop the key?"
— still O(1), with no parallel count that can disagree about when a handle is
released. The subscribe-success promotion split (fresh vs. unresolved→resolved)
also goes: it existed only to pick which scalar to bump; set sizes get
DataConnectionLayer-020's double-count cases right for free.
Behavior is otherwise unchanged — same logging, same handle release, same
unresolved-tag probing, same in-flight-unsubscribe discard semantics (the long
comment block there is updated for the mechanics that changed).
Tests: five TagResolutionCounts_* cases in DataConnectionActorBatchTests
covering the churn repro (3 cycles), a shared tag losing one instance mid
reconnect, connection-level failure then recovery, plain subscribe/unsubscribe
cycles, and a completed reconnect re-subscribe. Verified failing against the
pre-fix actor (churn: total 1 not 0; connection-level: total 0 not 1) and
passing after. Full DCL suite 319/319; solution builds with 0 warnings.
Docs: Component-DataConnectionLayer.md health-reporting section describes the
derived counts; residuals register item 1 marked RESOLVED.
F1 (HIGH) DebugStreamBridgeActor: the 5-minute orphan net measured the MAILBOX
(SetReceiveTimeout), and once stream events were correctly marked
INotInfluenceReceiveTimeout nothing recurring reset it — the snapshot lands once
and GrpcStreamStable once — so every healthy session self-terminated at ~6 min
with a false "Site disconnected". Replaced with a periodic self-tick
(ConsumerLivenessCheckInterval, 30s) over a consumer-last-seen stamp renewed only
by DebugStreamConsumerAlive, which DebugStreamService Tells on a shared timer to
every session still in its registry (holding a session there IS "a consumer is
attached" — both the Blazor view and the SignalR hub release it on
dispose/disconnect, and it works headless). Reverting the wrapper was rejected: it
would restore the quiet-instance orphan bug.
F2 (MED) SiteStreamGrpcClient: the RpcException(Cancelled) filter now requires
cts.IsCancellationRequested. A peer-originated / channel-dispose Cancelled fired
none of onError/onCompleted/onConnected, leaving SiteAlarmAggregatorActor with
_streamDown=false forever (IsLive stuck true, reconcile reopen guard never fired).
F3 (MED) SiteStreamGrpcClient: a header TIMEOUT is no longer reported as
connected — that shape is exactly what an unreachable site produces, and it
cleared _streamDown, consumed _seedOnConnect and launched a full snapshot fan-out
at a dead site. AwaitHeadersAsync returns bool; the first received event is the
fallback connected signal, fired at most once from headers OR first event.
F4 (LOW-MED) SqliteAuditWriter.MarkReconciledUpToAsync: the blanket below-cursor
UPDATE retired late-stamped inserts that were never served (then age-purged —
silent loss). The flip is now bounded by insertion order: a Pending row retires
only if its rowid is at or below the high-water mark of rows this instance has
served from ReadPendingSinceAsync (clamped on purge, since SQLite reuses rowids);
Forwarded rows are exempt (central ACKed them over the push path). At-least-once
is unchanged.
F5 (LOW) Documented the liveness dependency (a served row never covered by a later
cursor stays Pending forever; PurgeExpiredAsync never purges Pending) in
ISiteAuditQueue + Component-AuditLog.md, and added a cheap site-health signal:
SiteAuditBacklogReporter logs a rate-limited warning when the existing
oldest-pending metric exceeds 24h.
F6 (MED) SiteAlarmAggregatorActor: _fanoutSinceLastTick was armed by the
reconcile's OWN fan-out, so steady state ran fan-out→skip→fan-out→skip — one
reconcile per 2x interval (120s), halving the not-reporting refresh and the alarm
reconcile backstop. The skip is now armed only by connect/failover-driven seeds
(initial, _seedOnConnect, and a re-seed queued behind one).
Tests: Communication.Tests 691 passed (+13), AuditLog.Tests 382 passed (+5).
Six adversarial-review findings, each verified against the code first.
F1 (HIGH) CLI HttpClient capped every call at min(30s, caller timeout),
silently truncating deploy site's 5-minute BulkDeployTimeout and the
5-minute bundle export/preview/import calls — which printed a fake
"504 Request timed out" while the server kept working. HttpClient.Timeout
is now Timeout.InfiniteTimeSpan (the per-call CTS is the single overall
deadline, connect included) with the connect phase bounded separately on
SocketsHttpHandler.ConnectTimeout. The env override is renamed to
SCADABRIDGE_HTTP_CONNECT_TIMEOUT_SECONDS to match its new meaning.
F2 (HIGH) StaleInstanceProbe's process-static memo served stale hashes
because nothing bumped the watermark on three paths:
(a) BundleImporter commits through the raw DbContext, so no import ever
moved the watermark — a second import overwriting the same template
could be OMITTED from ImportResult.StaleInstanceIds. It now bumps
once per apply ATTEMPT: after the commit, and after the rollback too
(the probe runs pre-commit, so a rolled-back attempt leaves memos for
state that never landed; bumping on both paths is the simplest
correct shape, versus threading transaction awareness through a
process-static cache).
(b) CollectWatermarkBumps' default: arm silently no-op'd, contradicting
its own doc. It now sets unattributed=true — an over-broad bump costs
extra work, a missed one produces stale work.
(c) DataConnection edits route through SiteRepository.SaveChangesAsync,
which had no watermark at all, yet Protocol/Primary+Backup config/
FailoverRetryCount are revision-hash inputs. It now bumps (BumpAll —
a connection has no owning template) after a commit that touched one.
F3 (MED) CLI TemplateTableProjection read child ARRAYS, but ListTemplates
now returns database-projected TemplateSummary rows, so template list
printed all zeros. It now prefers the *Count scalars and falls back to
array length (template get still returns full entities). --detail help
text and README corrected: a listing cannot yield definitions, so --detail
renders the raw summary payload and template get --id is the full dump.
F4 (MED) DeploySiteAsync staged every PendingDeployment in phase 1 against
a 5-min TTL while phase 2 reached them one batch at a time, so tail
instances' fetch tokens could expire before their command was sent.
Staging moved into phase 2, immediately before each send; prepare keeps
its flatten/validate/record work. The staging write is the phase's only
repository touch and is serialised behind a 1-permit semaphore, so the
non-thread-safe DbContext constraint holds and the sends stay concurrent.
F5 (MED, latent) DeploySiteAsync leaked every held operation lock if
cancelled — a wedged per-instance semaphore is permanent for the process.
Phase 2 no longer throws (cancellation is recorded as a per-instance
outcome so phase 3 still runs), and an escape from phase 1 or 3 now
unwinds every unfinalised entry: Failed status + lock release.
F6 (LOW) ScriptCompileVerdictCache's promotion wrote hot directly,
bypassing SegmentCapacity (true ceiling 3x against a documented 2x).
Promotion now goes through Store, keeping generational semantics; _hot
and _cold are volatile.
Tests: CLI 396, DeploymentManager 133, ManagementService 494,
TemplateEngine 478, ScriptAnalysis 60, Transport 157, Transport
integration 106, ConfigurationDatabase 366 — all green, 0 build warnings.
The F2/F4/F5 regression tests were each confirmed to FAIL with their fix
reverted.
Six adversarial-review findings in the central SQL/ingest layer.
F1 (AuditLogRepository.InsertChunkAsync) — the set-based ingest declared each
string parameter at its COLUMN width (Actor/Target 256, Action 64, Outcome 16,
Category 32, SourceNode 64), so SqlClient truncated an over-long value at bind
time and committed the mutilated row — silent, in an append-only store, with no
PayloadTruncated flag — while the per-row and reconciliation paths sent the same
value in full and let the server reject it with 2628. Bind at the value's own
length instead; explicit SqlDbType is kept (it fixes the VALUES constructor's
derived column types and datetime2 precision). Design: reject everywhere,
truncate nowhere — matching today's per-row behaviour.
F2 (SiteCallAuditRepository.UpsertAsync) — the single-statement upsert ran the
monotonic UPDATE first and INSERTed only if nothing matched. Two writers racing
the first packet of one TrackedOperationId (the cached dual-write and the
reconciliation pull carry DIFFERENT lifecycle states) both matched nothing, and
the loser then skipped its INSERT or swallowed a 2627 — dropping its
Status/RetryCount/HttpStatus/TerminalAtUtc. Legs swapped to
`IF NOT EXISTS … INSERT; UPDATE <monotonic>` — still one round trip, and the
loser's UPDATE now lands on the winner's row. The duplicate-key catch re-runs
the monotonic UPDATE for the same reason. Moved to raw SQL with explicitly-typed
parameters so the intricate rank predicate exists in exactly one place (an
untyped DateTime would bind as `datetime` and round the freshness tiebreaker).
F3 (docs/plans/sql/*.sql) — filtered-index DDL failed with error 1934 under the
documented `docker exec … sqlcmd` path, which defaults QUOTED_IDENTIFIER OFF;
once IX_Notifications_Delivered exists, QI-OFF DML on Notifications fails too.
All four scripts now open with `SET QUOTED_IDENTIFIER ON; SET ANSI_NULLS ON; GO`
(own batch, so it is in force when the next batch parses), and the migration
convention in Component-ConfigurationDatabase.md documents `sqlcmd -I`. Verified
live: the pre-fix script fails 1934 without -I, the fixed one applies.
F4 (SiteCallAuditActor) — the off-mailbox reconciliation/purge passes reuse the
injected repository, so tests drove one DbContext from the pass and a mailbox
handler concurrently. Serialized at the CALL via a private SerializedRepository
wrapper applied only by the test constructors, rather than running the pass
on-mailbox: production keeps its PipeTo shape untouched, and the existing
"a blocked drain does not stall ingest/query/KPI" regression tests stay
meaningful (they would have been invalidated by suspending the mailbox).
F5 (AuditLogIngestActor) — when the batch failed because the 20 s IngestBudget
expired, the per-row fallback reused the same expired token: N instant failures,
N counter bumps, zero accepted. The fallback now gets a fresh 5 s budget (inside
the 30 s outer Ask), and a blown budget bumps the failure counter ONCE for the
batch instead of once per row.
F6 (NotificationOutboxRepository.UpdateAsync) — ExecuteUpdate's row count was
discarded, so an operator Retry/Discard of a notification the retention purge had
already deleted reported success (the pre-ExecuteUpdate code threw
DbUpdateConcurrencyException). UpdateAsync now returns whether a row matched; the
operator one-shots answer "notification not found" and emit no audit row for the
action that did not happen, while the dispatcher logs a warning (its delivery
already happened; nothing to retry). GetByIdAsync switched to AsNoTracking since
the write is out-of-band.
Tests: 5 new SQL-backed regressions (over-long Target rejected on both paths +
boundary round-trip; concurrent first-write and already-created-by-another-writer
upserts; vanished-row UpdateAsync), a token-identity pin on the ingest fallback,
a repository-concurrency detector for the SiteCallAudit passes, and vanished-row
operator-path tests. The F1/F2/F4 regressions were each confirmed failing against
the pre-fix code. Suites: ConfigurationDatabase 369, AuditLog 378, SiteCallAudit
66, NotificationOutbox 152 — all green, solution builds with 0 warnings.
F1: deploy/wonder-app-vd03/appsettings.Site.json (outside git, WP1.2's
StartupValidator gate applies live on next install/upgrade) was missing the
now-required AuditLog:SiteWriter:DatabasePath, added pointing at
E:\ApiInstall\ScadaBridge\site\data\auditlog.db alongside the file's
existing SiteEventLog/LocalDb paths; scanned deploy/ for other Site-role
appsettings with the same gap (none) and confirmed wonder does not pin
LocalDb:Replication:MaxBatchSize (F2 doesn't apply there).
F2: re-pin an explicit LocalDb:Replication:MaxBatchSize=64 on docker/site-a
node-a and node-b. MaxBatchBytes (2 MB default) only bounds the wire
message via the per-message split in SyncSession.PumpLoopAsync;
MaxBatchSize separately bounds the DB read page in
OplogStore.ReadBatchAboveAsync/SnapshotStreamer, which materializes the
whole page into memory before that split runs. Left at the 500 default, a
reconnect drain of worst-case config_json rows could transiently allocate
~35 MB per read even though every wire message stayed within budget.
Updated the CLAUDE.md LocalDb bullet to stop implying the row cap is fully
redundant with the byte budget (topology-guide.md has no matching claim).
F3: StoreAndForwardService's observer-queue onDropped callback logged a
Warning per dropped item, flooding logs at sweep rate for a stuck observer
with a large queue. LogObserverQueueDrop now logs once immediately on the
first drop of an episode, then throttles to at most one rollup Warning per
minute while drops continue, reporting the count dropped since the last
log; the cumulative ObserverQueueDroppedCount counter is unaffected.
Extended StoreAndForwardServiceTests with
ObserverQueue_ManyDropsInOneEpisode_LogsExactlyOneWarning, which floods the
bounded queue and pins exactly one drop-related Warning log for the
episode via a small CapturingLogger test double.
dotnet build ZB.MOM.WW.ScadaBridge.slnx: 0 warnings, 0 errors.
dotnet test StoreAndForward.Tests: 134/134 passed.
dotnet test Host.Tests: 490/490 passed.
CachedDrain_OrphanRow_PastGrace_IsAbandoned_AndTheValidRowStillFlows gated on
IngestCachedTelemetryAsync being received once and then asserted, bare, that the
valid row had been marked Forwarded. The drain does that strictly AFTER the push
returns: OnCachedDrainAsync abandons the orphan (:351), pushes the batch (:366),
then parses the ack and marks the accepted ids (:380). Observing the push
therefore orders nothing with respect to the second MarkForwardedAsync — under a
loaded parallel run the post-push continuation can be scheduled after the poll
that saw the push, and the assertion fails fast with "Actually received no
matching calls" while the orphan's own earlier call is reported as the single
non-matching one.
Reproduced deterministically by delaying only the post-push step, which fails
exactly this test (11 siblings still pass) at ~1.3s into the assembly run —
matching the observed failure's fast-fail signature and pointing at line 430.
With the fix the same injected delay passes; suppressing the valid row's
MarkForwarded entirely still fails the test with the identical message, so the
claim (orphan abandoned in its own call, valid row pushed and marked, exactly
once each with exactly the same arguments) is unchanged in force.
Same unsynchronized-assertion class as c4caebe9, different actor. Test-only; the
drain's abandon/push/mark ordering is correct as written.
Both NotifyDispatcher_AuditWriter_Throws_DeliveryStillSucceeds and
NotificationDispatch_BrokenAuditWriter_StillTransitionsToDelivered read the
throwing writer's attempt counter with a bare Assert immediately after an
AwaitAssert on the Notifications row reaching Delivered. That assumes the audit
writes happen no later than the operational status write, which the dispatcher
deliberately does NOT guarantee: DeliverOneAsync persists the delivery state
first (NotificationOutboxActor.cs:657) and only then emits the Attempted
(:663) and terminal (:676) audit rows — audit is best-effort and must never
gate the user-facing action. Observing Delivered therefore establishes no
happens-before edge with the writer, and under a loaded full-solution parallel
run the continuation after the DB write can be scheduled after the poll that
saw Delivered, so the counter reads 0 and the test fails with "saw 0".
Reproduced deterministically by delaying only the post-update audit emission,
which yields both observed failure messages verbatim; with the fix in place the
same injected delay passes, and suppressing the emissions entirely still fails
both tests with the identical messages — the claims (delivery despite audit
failure, and attempts >= N) are unchanged in force, only the ordering
assumption is gone.
Test-only change; the update-then-audit ordering predates the remediation
(#23 M4) and is correct as written.
Implements WP3.2 stage (b) per docs/plans/2026-08-15-site-events-policy-design.md.
- Per-run instance-script Started/Completed Info site events are now off by
default (SiteRuntimeOptions.PerRunScriptEvents=false) instead of firing on
every run, closing the dominant site_events writer. Gated at the ScriptRunLauncher
call sites (moved there from ScriptExecutionActor by WP3.1). Error-level events
(timeout/failure/stuck-watchdog/recursion-limit) remain unconditional.
- ScriptRunSummaryRecorder accumulates per-(instance, script) run counters and a
new site-only ScriptRunSummaryFlushService emits one aggregate "script" Info
site event per ScriptRunSummaryIntervalSeconds (default 300s), top-50-script
breakdown with an "others" rollup, zero-activity intervals emit nothing.
- Per-script opt-in via PerRunScriptEventScripts ("Instance/Script" exact or
"Instance/*" wildcard), matched by the new pure ScriptRunEventPolicy. All three
options are read from IOptionsMonitor<SiteRuntimeOptions> per run, so the
policy is hot-togglable without a restart.
- Fixed the stale "event log is not replicated" comment at AkkaHostedService.cs
(~905): site_events IS registered in SiteLocalDbSetup.ReplicatedTables — the
singleton is what makes queries always hit the actively-written copy;
replication is what gives the singleton history to read after a failover
(memo Decision (b)). site_events replication itself is unchanged (still
registered) and already pinned by
tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbCdcRegistrationTests.cs.
- Updated Component-SiteEventLogging.md (Volume Policy section, corrected
Storage/replication rationale) and Component-SiteRuntime.md (Script Run
Launch + Error Handling sections).
Joseph Doherty