Root-caused from a full-suite-load-only failure of
GrpcStreamIntegrationTests.Pipeline_DuplicateCorrelationId_ReplacesStream:
ObjectDisposedException escaping SubscribeInstance.
SubscribeInstance's duplicate-prevention path cancelled AND disposed the
replaced stream's CancellationTokenSource. That CTS belongs to the replaced
handler's own `using var streamCts`, which is still running and still has to
read `streamCts.Token` — the Dispose raced that read. The race is pre-existing
and independent of R2: the same Dispose sat against the same first-token-read
when the handler still used `ReadAllAsync(streamCts.Token)`; it only ever
loses under enough scheduling pressure to land the replacement inside the
first stream's setup window.
Cancel only. The owning handler's `using` still disposes it exactly once on
every exit path, and cancellation is all replacement ever needed. The Cancel
is wrapped for the converse race (owner already finished and disposed),
mirroring CancelAllStreams().
Regression test makes the race deterministic by gating the first stream inside
its setup window — the _activeStreams entry is registered before Subscribe is
called, so the replacement always lands before the first stream reads its
token. Verified fail-before (ObjectDisposedException) / pass-after by
reinstating the Dispose as a negative control.
LocalDb 0.3.0 (scadaproj branch localdb-per-table-snapshot) narrows the
snapshot-resync flag from one per-database bit to one per table, closing the
residual 0.2.1 explicitly deferred. On-disk bookkeeping schema goes to v3
(__localdb_snapshot_state, upgraded in place on open); the wire stays compatible
with 0.1.x/0.2.x, negotiated by capability rather than by a lib_schema_version
bump — which is not available, since the handshake compares that field
fail-closed for equality.
SiteLocalDbSetup needed no restructuring: it already calls RegisterReplicated
once per table, which is exactly what per-table flagging keys off. The comment
now records that the loop shape is load-bearing, and what it buys — adding an
eleventh table to ReplicatedTables on an already-replicating site snapshots that
one table, where through 0.2.x the same edit re-streamed all eleven in full in
both directions. First boot is unchanged (all ten seed at once, so the flagged
set is every registered table and the library sends an ordinary full snapshot),
and upgrading the rig in place is a no-op (the ten are already ledgered, so
nothing seeds and nothing is flagged).
Suites: Host 490/490, SiteRuntime 604/604, StoreAndForward 134/134,
SiteEventLogging 76/76. Solution build clean.
topology-guide + CLAUDE.md LocalDb bullet updated; the umbrella scadaproj
CLAUDE.md travels in that repo's commit.
Every AttributeValueChanged/AlarmStateChanged rode its own gRPC message on
SiteStreamService; at target scale that is ~37.5k messages/s/site of pure
framing overhead. Coalesce them, additively, with no new RPC.
Wire (sitestream.proto, regenerated via docker/regen-proto.sh sitestream):
InstanceStreamRequest.batching_supported = 3
SiteStreamRequest.batching_supported = 2
SiteStreamEvent.batch = 4 (new oneof case)
SiteStreamEventBatch { repeated SiteStreamEvent events = 1 }
The proto3 default of batching_supported IS the negotiation, and it is
load-bearing: a batch frame reaches a pre-R2 central as EventOneofCase.None,
whose ConvertToDomainEvent returns null — the whole batch would vanish with
no error anywhere. An old central cannot set the flag so it never receives
one; an old site ignores the unknown request field and keeps sending
per-event frames, which the new client's ForEachEvent handles as the
single-event case. Both skew directions are covered by tests that go through
a real proto serialize/parse round-trip.
Server: SiteStreamEventBatcher, a per-subscriber pump replacing the handler's
await-foreach/WriteAsync loop and byte-identical to it at a cap of 1. It
never delays a lone event — it drains the already-queued backlog for free and
lingers only once a backlog is proven — emits a single-event buffer as a
plain frame, preserves order and per-event Timestamps exactly, and flushes
what is buffered when the send channel's writer completes. It sits DOWNSTREAM
of StreamRelayActor's bounded DropOldest channel, so it changes framing only
and does not move the burst ceiling (deferred register row 31, which lives in
the shared publish stage upstream of the BroadcastHub).
Client: sets the flag on both subscriptions and unpacks in order into the
existing per-event pipeline, so SiteAlarmAggregatorActor,
DebugStreamBridgeActor, consumer-keepalive/orphan logic,
reconnect-on-graceful-completion, generation fencing, the (siteId, endpoint)
factory key and IsLive semantics are untouched.
Options (validated): GrpcStreamBatchMaxEvents 100 (1 disables),
GrpcStreamBatchWindow 25 ms — validated strictly under 250 ms, the load
test's end-to-end P99 threshold. Measured worst case (trickle-with-backlog,
window-bound rather than cap-bound): P50 13.8 ms, P99 25.4 ms, max 25.8 ms;
cap-bound case sent 600 queued events in 6 frames.
Telemetry: histogram scadabridge.site.stream.batch_size tagged by stream
kind, recorded only on negotiated streams (per-event otherwise). It rides
ScadaBridgeTelemetry.MeterName, already in the ObservedMeters allowlist.
Docs: Component-Communication.md gains an Event Batching section; CLAUDE.md's
gRPC streaming bullet records the wire shape and the negotiation rationale.
The Deployment Status page client-materialized the whole deployment list. It read
EVERY DeploymentRecord — an insert-only table, one row per deploy attempt for the
retention window — plus EVERY Instance, then site-scoped, sorted, counted the four
status tiles and sliced a 25-row page in the Blazor circuit's memory. That ran on
first render AND on every IDeploymentStatusNotifier push, so the cost scaled with
the age of the system rather than the size of the page.
All four jobs move into SQL:
- `IDeploymentManagerRepository.QueryDeploymentListPageAsync(filter, page, size)`
returns one page of `DeploymentListRow` — DeploymentRecord INNER JOINed to
Instance, so the instance display name and site travel with the rows that need
them — plus the total count of the filtered set. The join is exact: the FK is
Restrict and DeleteInstanceAsync removes the records first, so no orphan exists.
- `GetDeploymentStatusCountsAsync(filter)` returns the tile counts from ONE grouped
aggregation, deliberately ignoring the filter's Status: the tiles are the status
BREAKDOWN of the filtered set, so honouring it would zero three of four tiles the
moment an operator clicked one.
- Site scoping runs in the query as `SiteIdScope` resolved through the record's
instance (DeploymentRecord has no SiteId of its own). An EMPTY grant stays a real
filter matching nothing, never "unconstrained".
- The now-callerless whole-table `GetAllDeploymentRecordsAsync` is deleted.
OFFSET paging, not the Audit Log's keyset cursor, and deliberately so: this page's
pager is numbered and jump-to-any-page, so it needs a page count, which only a total
can give it — a keyset cursor can express neither, and the total is required for the
tiles regardless. The deep-offset cost that pushes high-volume tables to keyset is
bounded here by the terminal-record retention purge, unlike the 365-day AuditLog.
This mirrors the Notification Outbox, offset-paged for the same reason. Ordering is
DeployedAt DESC, Id DESC — the Id tie-break is load-bearing, because DeployedAt ties
on rapid redeploys and an unstable sort key makes offset paging repeat or drop rows.
UI: the four status tiles become the status filter (click to apply, click again to
clear, aria-pressed, phrasing-only content so a <button> stays valid), plus a
free-text search matched DB-side against instance name, deployment id, revision hash
and initiating user. Search is TRAILING-edge debounced at 500ms — the same Timer +
lock + _disposed idiom as the existing leading-edge push coalescer, minus the leading
edge, because a search box must not query on the first keystroke. A filter change
resets to page 1; a page past the end falls back to the last real page. Bootstrap
only, existing PagerWindow pager retained.
The WP2.4 push coalescing is unchanged and still earns its keep: server paging shrank
what a reload costs, not how many arrive — it now bounds database round-trips rather
than table scans.
Tests: 19 new SQLite repository tests (paging slice + total, tie-break stability
across pages, page/size clamping, past-the-end, the joined projection, every filter
dimension incl. the empty-scope security case, and the grouped counts' status-blind
contract); 15 new bUnit page tests (page-1 request, server total drives the pager,
Next re-queries, tiles show server counts not page counts, tile filter + toggle,
system-wide vs site-scoped scope push, debounce collapses a keystroke burst to one
query, clear-filters, dispose with an armed timer). The two existing Deployments
suites re-point their reload assertions at the new query.
Doc: Component-CentralUI.md Deployment section — the "no server-side paging" known
residual is replaced by the shipped design.
The daily site_events retention purge (and the storage-cap trim) is CDC-captured
on a replication-enabled site node exactly like any other write — correct by
design, since LocalDb Phase 2 deliberately has no purge-exemption path — so the
backlog jumps by the deleted batch size at purge time. LocalDbOplogBacklog /
localdb_oplog_depth spike, drain, and an operator watching the gauge with no
context reads it as a replication fault.
Documentation + one log line, no behaviour change:
- topology-guide.md gains "Reading the replication backlog — the daily
site_events purge burst": when it fires (PurgeInterval 24h, anchored to the
active node's PROCESS START, not a wall-clock hour, so it moves after every
failover), where it shows (replicated nodes only — not rig site-b/site-c),
the healthy signature (LocalDbReplicationConnected stays true, backlog
returns to ~0) and what a genuine fault looks like instead.
- Component-SiteEventLogging.md Storage records the same under retention/purge;
Component-HealthMonitoring.md gains the two previously-undocumented
LocalDbReplicationConnected / LocalDbOplogBacklog metric rows carrying the
caveat, with cross-references both ways.
- EventLogPurgeService emits one Information line naming the row count and the
expected transient backlog when a purge deleted rows on a replication-enabled
node, so the spike is correlatable in the log. Replication-awareness comes in
as a Host-supplied SiteEventLogReplicationCheck delegate, mirroring the
existing SiteEventLogActiveNodeCheck seam: SiteLocalDbSetup.ReplicationIsConfigured
goes internal so the PeerAddress-OR-ApiKey rule stays in one place and
SiteEventLogging never learns to read LocalDb config. Unregistered ⇒ no note,
matching the default that replication is opt-in and off.
Both delete paths carry the note (a cap trim is usually the larger burst); the
predicate is try/caught since a log-wording check must never break the purge.
Tests: 5 new EventLogPurgeServiceTests cases (replicated logs it, unreplicated
does not, zero-rows does not, cap purge logs it, throwing predicate still purges
and swallows) via a local capturing ILogger. SiteEventLogging 81/81 green,
Host 490/490 green, full solution build clean (0 warnings).
Closes arch-review remediation residual #1 (DCL unsubscribe-during-reconnect
count staleness).
DataConnectionActor tracked TotalSubscribedTags/ResolvedTags as two int fields
incremented and decremented at five independent sites. ReSubscribeAll clears the
very maps those decrements key off (_subscriptionIds, _unresolvedTags) while
deliberately preserving _subscriptionsByInstance, so an unsubscribe landing
inside a reconnect window matched NEITHER decrement branch: the total leaked +1
per subscribe/reconnect/unsubscribe churn cycle, permanently and cumulatively.
The 37f13e2e discard gate stopped the orphan-handle half of that race; it could
not stop the counters drifting, because they were state of their own.
Both counts are now DERIVED at report time from the authoritative per-tag
collections, which makes the drift unrepresentable rather than merely guarded:
total = _instancesByTag.Count (the per-tag counted set the residual
called for — distinct tags with at least
one subscribing instance)
resolved = _subscriptionIds.Count (tags for which the adapter holds a handle)
Two semantic corrections fall out of the derivation:
- A tag whose subscribe failed at CONNECTION level now counts toward the total.
It was excluded before, yet the reconnect re-subscribe re-issued it from
_subscriptionsByInstance and booked it as resolved — resolved above total, and
a total driven negative by the eventual unsubscribe.
- _tagSubscriberCount is deleted. It duplicated _instancesByTag exactly, so
HandleUnsubscribe's last-subscriber test is now "did UnindexTag drop the key?"
— still O(1), with no parallel count that can disagree about when a handle is
released. The subscribe-success promotion split (fresh vs. unresolved→resolved)
also goes: it existed only to pick which scalar to bump; set sizes get
DataConnectionLayer-020's double-count cases right for free.
Behavior is otherwise unchanged — same logging, same handle release, same
unresolved-tag probing, same in-flight-unsubscribe discard semantics (the long
comment block there is updated for the mechanics that changed).
Tests: five TagResolutionCounts_* cases in DataConnectionActorBatchTests
covering the churn repro (3 cycles), a shared tag losing one instance mid
reconnect, connection-level failure then recovery, plain subscribe/unsubscribe
cycles, and a completed reconnect re-subscribe. Verified failing against the
pre-fix actor (churn: total 1 not 0; connection-level: total 0 not 1) and
passing after. Full DCL suite 319/319; solution builds with 0 warnings.
Docs: Component-DataConnectionLayer.md health-reporting section describes the
derived counts; residuals register item 1 marked RESOLVED.
F1 (HIGH) DebugStreamBridgeActor: the 5-minute orphan net measured the MAILBOX
(SetReceiveTimeout), and once stream events were correctly marked
INotInfluenceReceiveTimeout nothing recurring reset it — the snapshot lands once
and GrpcStreamStable once — so every healthy session self-terminated at ~6 min
with a false "Site disconnected". Replaced with a periodic self-tick
(ConsumerLivenessCheckInterval, 30s) over a consumer-last-seen stamp renewed only
by DebugStreamConsumerAlive, which DebugStreamService Tells on a shared timer to
every session still in its registry (holding a session there IS "a consumer is
attached" — both the Blazor view and the SignalR hub release it on
dispose/disconnect, and it works headless). Reverting the wrapper was rejected: it
would restore the quiet-instance orphan bug.
F2 (MED) SiteStreamGrpcClient: the RpcException(Cancelled) filter now requires
cts.IsCancellationRequested. A peer-originated / channel-dispose Cancelled fired
none of onError/onCompleted/onConnected, leaving SiteAlarmAggregatorActor with
_streamDown=false forever (IsLive stuck true, reconcile reopen guard never fired).
F3 (MED) SiteStreamGrpcClient: a header TIMEOUT is no longer reported as
connected — that shape is exactly what an unreachable site produces, and it
cleared _streamDown, consumed _seedOnConnect and launched a full snapshot fan-out
at a dead site. AwaitHeadersAsync returns bool; the first received event is the
fallback connected signal, fired at most once from headers OR first event.
F4 (LOW-MED) SqliteAuditWriter.MarkReconciledUpToAsync: the blanket below-cursor
UPDATE retired late-stamped inserts that were never served (then age-purged —
silent loss). The flip is now bounded by insertion order: a Pending row retires
only if its rowid is at or below the high-water mark of rows this instance has
served from ReadPendingSinceAsync (clamped on purge, since SQLite reuses rowids);
Forwarded rows are exempt (central ACKed them over the push path). At-least-once
is unchanged.
F5 (LOW) Documented the liveness dependency (a served row never covered by a later
cursor stays Pending forever; PurgeExpiredAsync never purges Pending) in
ISiteAuditQueue + Component-AuditLog.md, and added a cheap site-health signal:
SiteAuditBacklogReporter logs a rate-limited warning when the existing
oldest-pending metric exceeds 24h.
F6 (MED) SiteAlarmAggregatorActor: _fanoutSinceLastTick was armed by the
reconcile's OWN fan-out, so steady state ran fan-out→skip→fan-out→skip — one
reconcile per 2x interval (120s), halving the not-reporting refresh and the alarm
reconcile backstop. The skip is now armed only by connect/failover-driven seeds
(initial, _seedOnConnect, and a re-seed queued behind one).
Tests: Communication.Tests 691 passed (+13), AuditLog.Tests 382 passed (+5).
Six adversarial-review findings, each verified against the code first.
F1 (HIGH) CLI HttpClient capped every call at min(30s, caller timeout),
silently truncating deploy site's 5-minute BulkDeployTimeout and the
5-minute bundle export/preview/import calls — which printed a fake
"504 Request timed out" while the server kept working. HttpClient.Timeout
is now Timeout.InfiniteTimeSpan (the per-call CTS is the single overall
deadline, connect included) with the connect phase bounded separately on
SocketsHttpHandler.ConnectTimeout. The env override is renamed to
SCADABRIDGE_HTTP_CONNECT_TIMEOUT_SECONDS to match its new meaning.
F2 (HIGH) StaleInstanceProbe's process-static memo served stale hashes
because nothing bumped the watermark on three paths:
(a) BundleImporter commits through the raw DbContext, so no import ever
moved the watermark — a second import overwriting the same template
could be OMITTED from ImportResult.StaleInstanceIds. It now bumps
once per apply ATTEMPT: after the commit, and after the rollback too
(the probe runs pre-commit, so a rolled-back attempt leaves memos for
state that never landed; bumping on both paths is the simplest
correct shape, versus threading transaction awareness through a
process-static cache).
(b) CollectWatermarkBumps' default: arm silently no-op'd, contradicting
its own doc. It now sets unattributed=true — an over-broad bump costs
extra work, a missed one produces stale work.
(c) DataConnection edits route through SiteRepository.SaveChangesAsync,
which had no watermark at all, yet Protocol/Primary+Backup config/
FailoverRetryCount are revision-hash inputs. It now bumps (BumpAll —
a connection has no owning template) after a commit that touched one.
F3 (MED) CLI TemplateTableProjection read child ARRAYS, but ListTemplates
now returns database-projected TemplateSummary rows, so template list
printed all zeros. It now prefers the *Count scalars and falls back to
array length (template get still returns full entities). --detail help
text and README corrected: a listing cannot yield definitions, so --detail
renders the raw summary payload and template get --id is the full dump.
F4 (MED) DeploySiteAsync staged every PendingDeployment in phase 1 against
a 5-min TTL while phase 2 reached them one batch at a time, so tail
instances' fetch tokens could expire before their command was sent.
Staging moved into phase 2, immediately before each send; prepare keeps
its flatten/validate/record work. The staging write is the phase's only
repository touch and is serialised behind a 1-permit semaphore, so the
non-thread-safe DbContext constraint holds and the sends stay concurrent.
F5 (MED, latent) DeploySiteAsync leaked every held operation lock if
cancelled — a wedged per-instance semaphore is permanent for the process.
Phase 2 no longer throws (cancellation is recorded as a per-instance
outcome so phase 3 still runs), and an escape from phase 1 or 3 now
unwinds every unfinalised entry: Failed status + lock release.
F6 (LOW) ScriptCompileVerdictCache's promotion wrote hot directly,
bypassing SegmentCapacity (true ceiling 3x against a documented 2x).
Promotion now goes through Store, keeping generational semantics; _hot
and _cold are volatile.
Tests: CLI 396, DeploymentManager 133, ManagementService 494,
TemplateEngine 478, ScriptAnalysis 60, Transport 157, Transport
integration 106, ConfigurationDatabase 366 — all green, 0 build warnings.
The F2/F4/F5 regression tests were each confirmed to FAIL with their fix
reverted.
Six adversarial-review findings in the central SQL/ingest layer.
F1 (AuditLogRepository.InsertChunkAsync) — the set-based ingest declared each
string parameter at its COLUMN width (Actor/Target 256, Action 64, Outcome 16,
Category 32, SourceNode 64), so SqlClient truncated an over-long value at bind
time and committed the mutilated row — silent, in an append-only store, with no
PayloadTruncated flag — while the per-row and reconciliation paths sent the same
value in full and let the server reject it with 2628. Bind at the value's own
length instead; explicit SqlDbType is kept (it fixes the VALUES constructor's
derived column types and datetime2 precision). Design: reject everywhere,
truncate nowhere — matching today's per-row behaviour.
F2 (SiteCallAuditRepository.UpsertAsync) — the single-statement upsert ran the
monotonic UPDATE first and INSERTed only if nothing matched. Two writers racing
the first packet of one TrackedOperationId (the cached dual-write and the
reconciliation pull carry DIFFERENT lifecycle states) both matched nothing, and
the loser then skipped its INSERT or swallowed a 2627 — dropping its
Status/RetryCount/HttpStatus/TerminalAtUtc. Legs swapped to
`IF NOT EXISTS … INSERT; UPDATE <monotonic>` — still one round trip, and the
loser's UPDATE now lands on the winner's row. The duplicate-key catch re-runs
the monotonic UPDATE for the same reason. Moved to raw SQL with explicitly-typed
parameters so the intricate rank predicate exists in exactly one place (an
untyped DateTime would bind as `datetime` and round the freshness tiebreaker).
F3 (docs/plans/sql/*.sql) — filtered-index DDL failed with error 1934 under the
documented `docker exec … sqlcmd` path, which defaults QUOTED_IDENTIFIER OFF;
once IX_Notifications_Delivered exists, QI-OFF DML on Notifications fails too.
All four scripts now open with `SET QUOTED_IDENTIFIER ON; SET ANSI_NULLS ON; GO`
(own batch, so it is in force when the next batch parses), and the migration
convention in Component-ConfigurationDatabase.md documents `sqlcmd -I`. Verified
live: the pre-fix script fails 1934 without -I, the fixed one applies.
F4 (SiteCallAuditActor) — the off-mailbox reconciliation/purge passes reuse the
injected repository, so tests drove one DbContext from the pass and a mailbox
handler concurrently. Serialized at the CALL via a private SerializedRepository
wrapper applied only by the test constructors, rather than running the pass
on-mailbox: production keeps its PipeTo shape untouched, and the existing
"a blocked drain does not stall ingest/query/KPI" regression tests stay
meaningful (they would have been invalidated by suspending the mailbox).
F5 (AuditLogIngestActor) — when the batch failed because the 20 s IngestBudget
expired, the per-row fallback reused the same expired token: N instant failures,
N counter bumps, zero accepted. The fallback now gets a fresh 5 s budget (inside
the 30 s outer Ask), and a blown budget bumps the failure counter ONCE for the
batch instead of once per row.
F6 (NotificationOutboxRepository.UpdateAsync) — ExecuteUpdate's row count was
discarded, so an operator Retry/Discard of a notification the retention purge had
already deleted reported success (the pre-ExecuteUpdate code threw
DbUpdateConcurrencyException). UpdateAsync now returns whether a row matched; the
operator one-shots answer "notification not found" and emit no audit row for the
action that did not happen, while the dispatcher logs a warning (its delivery
already happened; nothing to retry). GetByIdAsync switched to AsNoTracking since
the write is out-of-band.
Tests: 5 new SQL-backed regressions (over-long Target rejected on both paths +
boundary round-trip; concurrent first-write and already-created-by-another-writer
upserts; vanished-row UpdateAsync), a token-identity pin on the ingest fallback,
a repository-concurrency detector for the SiteCallAudit passes, and vanished-row
operator-path tests. The F1/F2/F4 regressions were each confirmed failing against
the pre-fix code. Suites: ConfigurationDatabase 369, AuditLog 378, SiteCallAudit
66, NotificationOutbox 152 — all green, solution builds with 0 warnings.
F1: deploy/wonder-app-vd03/appsettings.Site.json (outside git, WP1.2's
StartupValidator gate applies live on next install/upgrade) was missing the
now-required AuditLog:SiteWriter:DatabasePath, added pointing at
E:\ApiInstall\ScadaBridge\site\data\auditlog.db alongside the file's
existing SiteEventLog/LocalDb paths; scanned deploy/ for other Site-role
appsettings with the same gap (none) and confirmed wonder does not pin
LocalDb:Replication:MaxBatchSize (F2 doesn't apply there).
F2: re-pin an explicit LocalDb:Replication:MaxBatchSize=64 on docker/site-a
node-a and node-b. MaxBatchBytes (2 MB default) only bounds the wire
message via the per-message split in SyncSession.PumpLoopAsync;
MaxBatchSize separately bounds the DB read page in
OplogStore.ReadBatchAboveAsync/SnapshotStreamer, which materializes the
whole page into memory before that split runs. Left at the 500 default, a
reconnect drain of worst-case config_json rows could transiently allocate
~35 MB per read even though every wire message stayed within budget.
Updated the CLAUDE.md LocalDb bullet to stop implying the row cap is fully
redundant with the byte budget (topology-guide.md has no matching claim).
F3: StoreAndForwardService's observer-queue onDropped callback logged a
Warning per dropped item, flooding logs at sweep rate for a stuck observer
with a large queue. LogObserverQueueDrop now logs once immediately on the
first drop of an episode, then throttles to at most one rollup Warning per
minute while drops continue, reporting the count dropped since the last
log; the cumulative ObserverQueueDroppedCount counter is unaffected.
Extended StoreAndForwardServiceTests with
ObserverQueue_ManyDropsInOneEpisode_LogsExactlyOneWarning, which floods the
bounded queue and pins exactly one drop-related Warning log for the
episode via a small CapturingLogger test double.
dotnet build ZB.MOM.WW.ScadaBridge.slnx: 0 warnings, 0 errors.
dotnet test StoreAndForward.Tests: 134/134 passed.
dotnet test Host.Tests: 490/490 passed.
Implements WP3.2 stage (b) per docs/plans/2026-08-15-site-events-policy-design.md.
- Per-run instance-script Started/Completed Info site events are now off by
default (SiteRuntimeOptions.PerRunScriptEvents=false) instead of firing on
every run, closing the dominant site_events writer. Gated at the ScriptRunLauncher
call sites (moved there from ScriptExecutionActor by WP3.1). Error-level events
(timeout/failure/stuck-watchdog/recursion-limit) remain unconditional.
- ScriptRunSummaryRecorder accumulates per-(instance, script) run counters and a
new site-only ScriptRunSummaryFlushService emits one aggregate "script" Info
site event per ScriptRunSummaryIntervalSeconds (default 300s), top-50-script
breakdown with an "others" rollup, zero-activity intervals emit nothing.
- Per-script opt-in via PerRunScriptEventScripts ("Instance/Script" exact or
"Instance/*" wildcard), matched by the new pure ScriptRunEventPolicy. All three
options are read from IOptionsMonitor<SiteRuntimeOptions> per run, so the
policy is hot-togglable without a restart.
- Fixed the stale "event log is not replicated" comment at AkkaHostedService.cs
(~905): site_events IS registered in SiteLocalDbSetup.ReplicatedTables — the
singleton is what makes queries always hit the actively-written copy;
replication is what gives the singleton history to read after a failover
(memo Decision (b)). site_events replication itself is unchanged (still
registered) and already pinned by
tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbCdcRegistrationTests.cs.
- Updated Component-SiteEventLogging.md (Volume Policy section, corrected
Storage/replication rationale) and Component-SiteRuntime.md (Script Run
Launch + Error Handling sections).
WP2.6 (arch-review remediation, cross-cutting misc):
- SiteExternalSystemRepository: name/ID-indexed ExternalSystemDefinitionCache replaces
the fetch-all + reverse-map scan on every by-ID/method lookup; loaded once per
redeploy, invalidated by DeploymentManagerActor after HandleDeployArtifacts applies
external-system changes. Static JsonSerializerOptions for method-list parsing.
- Inbound API: short-TTL ApiMethodCache fronts the per-request ApiMethod repository
fetch; invalidated by name via the existing ScriptArtifactChangeSubscriber/
IScriptArtifactChangeBus pipeline, self-healing via TTL for changes the bus
doesn't cover (e.g. Management API edits).
- StoreAndForward: the cached-call audit-observer queue — the one unbounded channel
left in the system — is now bounded (ObserverQueueCapacity, default 10,000) with
DropOldest overflow and a dropped-notification counter.
- SiteStreamManager: alarm state changes now travel a dedicated publish
source/broadcast hub, isolated from the (far higher-volume) attribute path, so an
attribute storm can no longer evict a pending alarm transition; the alarm hand-off
queue is bounded with a drop counter surfaced on the site health report
(SiteStreamAlarmDropCount via the new SiteStreamAlarmDropReporter), and publishing
is skipped entirely at zero subscribers on either path.
- CLI ManagementHttpClient: explicit 30s HttpClient.Timeout on the shared
construction (was the 100s framework default), overridable via
SCADABRIDGE_HTTP_TIMEOUT_SECONDS.
Deviation: the failback-probe heartbeat item is NOT included — its only viable
surface (CentralChannelProvider.cs / heartbeat consumers) lives entirely in the
Communication project, explicitly off-limits to this work package this phase.
Tests: SiteRuntime.Tests (550), InboundAPI.Tests (278), StoreAndForward.Tests (133),
CLI.Tests (390), HealthMonitoring.Tests (97) — all green after full solution build.
Closes WP1.2 of the arch-review remediation plan (finding #2, High):
SqliteAuditWriterOptions.DatabasePath defaulted to CWD-relative "auditlog.db",
which on the docker rig resolves onto the container's ephemeral overlayfs
(not the mounted /app/data volume), silently discarding the pending audit
forward-state backlog on every recreate; nothing in docker/ or docker-env2/
overrode it; FlushIntervalMs was validated but never read by the writer loop
(one commit per event even at trickle rate); and no PRAGMA synchronous was
set (SQLite's FULL default fsyncs every commit).
- DatabasePath now has no default (mirrors ZB.MOM.WW.LocalDb's LocalDbOptions.Path)
and is required pre-host for Site nodes only, via a new StartupValidator raw-config
check (top-level "AuditLog:SiteWriter:DatabasePath", NOT nested under ScadaBridge:
AddAuditLog binds that section off the configuration root). SqliteAuditWriterOptionsValidator
deliberately does NOT check DatabasePath itself, because AddAuditLog runs its
ValidateOnStart on both Central and Site composition roots but only Site nodes
ever resolve the writer — checking it there would fail Central's boot too.
- All 8 site-node appsettings under docker/ and docker-env2/ now set
AuditLog:SiteWriter:DatabasePath to /app/data/auditlog.db (mounted volume,
survives container recreate, same convention as LocalDb:Path); the local-dev
base appsettings.Site.json sets ./data/auditlog.db to match.
- The writer loop now honors FlushIntervalMs: after draining the immediately
available burst, it keeps the transaction open (bounded by FlushIntervalMs
from the first event) waiting for more trickle-rate events before committing,
instead of flushing (and fsyncing) per event.
- PRAGMA synchronous = NORMAL on the write connection — audit is best-effort by
design (CLAUDE.md: "Audit-write failure NEVER aborts the user-facing action"),
so NORMAL's narrower power-loss window is an acceptable trade for far fewer
fsyncs; WAL mode still guarantees no corruption.
- Tests: StartupValidator site-required/blank/central-exempt cases; writer
trickle-load single-transaction coalescing + beyond-interval separate-transaction
regression (new FlushCountForTests seam); options-validator doc updates reflecting
the moved responsibility. Full suite runs green: AuditLog.Tests 368/368,
Host.Tests 480/480.
One-time migration note: the existing container-local auditlog.db (wherever it
landed under CWD) is abandoned by this change, not migrated — already-forwarded
rows are safe centrally (AuditLog is the durable copy), and any still-Pending
rows on the abandoned path are lost once. This is the exact bug being fixed, not
a new loss: those rows were already living outside the mounted volume and would
not have survived the next container recreate regardless. Cross-reference
docs/known-issues/2026-07-20-cached-telemetry-drain-hot-loop.md, which this
placement bug caused.
SiteLocalDbSetup.OnReady registered all ten replicated tables
unconditionally, so a deliberately unreplicated site node (site-b and
site-c on the rig) carried the full 30-trigger CDC set forever. Every
write to those tables paid two extra INSERTs plus a json_object
serialization of the whole row, inside the caller's own transaction, and
appended to an oplog nothing ever drains. Arch-review finding #5 (High),
repo half; the library half — trigger cleanup API and O(1) backlog — is
WP3.3.
The ten RegisterReplicated calls are now behind a guard on whether the
node has LocalDb:Replication:PeerAddress OR LocalDb:Replication:ApiKey.
Either key counts, and the OR is load-bearing rather than defensive:
replication is one bidirectional stream that exactly one side dials, so
only the initiator sets PeerAddress. Verified against the rig — site-a
node-a has PeerAddress + ApiKey, site-a node-b (passive) has ApiKey
alone, site-b/site-c have no Replication section at all. Keying on
PeerAddress alone would have stripped capture from every passive node and
silently made each pair converge in one direction only.
The load-bearing ordering documented in the file is preserved: DDL still
precedes registration, and the legacy migrator still runs unconditionally
after it — an unreplicated node must still absorb its pre-Phase-1 files,
and it has no peer for those rows to be invisible to.
Known residual, documented in-file and in the topology guide: a database
file first created by an older build keeps its stale __localdb_* triggers.
The guard decides whether triggers are installed, not whether existing
ones are removed, and the library has no removal API until WP3.3. Moot on
the docker rig, where a schema-change redeploy recreates the volumes.
The inverse is also now documented: enabling replication on a site that
has run without it does not baseline existing rows, since CDC never
recorded them in __localdb_row_version and the snapshot resync streams
from that ledger.
Tests: new SiteLocalDbCdcRegistrationTests asserts trigger presence and
absence via sqlite_master across all four config shapes (none, ApiKey
only, PeerAddress + ApiKey, and the notification-table exclusion), plus
DDL-still-runs and migrator-still-runs on the unreplicated branch.
SiteLocalDbWiringTests and the integration site-pair harness now
configure an ApiKey — mirroring the rig's passive node — so their
registration and convergence assertions still describe a replicating
node. 483/483 Host.Tests pass; the 20 offline LocalDb convergence tests
still pass.
Joseph Doherty