Commit Graph

2424 Commits

Author SHA1 Message Date
Joseph Doherty 8abebdae33 docs(plans): target-scale load test harness design memo (WP-4 / register #25)
Records harness architecture, the real-vs-faked table with a justification per fake,
metric definitions, falsifiable pass/fail thresholds derived from the WP-4 acceptance
criteria, and the eight deviations from the WP-4 protocol with reasons — including the
1-hour to 20-minute sustained-window shortening (memory reported as a slope so a
shorter window still answers the leak question) and the four [xc-*] criteria this
harness does not cover.
2026-08-15 02:27:47 -04:00
Joseph Doherty 20f6b0b969 test(loadharness): target-scale load harness for WP-4 / register #25 + row 50
Standalone console harness under tests/ZB.MOM.WW.ScadaBridge.LoadHarness plus a
scaled-down Category=Performance smoke [Fact] in PerformanceTests. Deliberately an
Exe rather than an xunit suite: the Performance trait enables a filter but does not
exclude by default, so a 20-minute test would run on every 'dotnet test' of the slnx.

What is real: per-site ActorSystem + LocalDb SQLite file, the real DCL
(DataConnectionManagerActor/DataConnectionActor over a SimulatedDataConnection
registered through the documented DataConnectionFactory.RegisterAdapter seam), real
InstanceActors fed real TagValueUpdates, the real SiteStreamManager, real
StreamRelayActor + production-capacity bounded DropOldest channel, real
StoreAndForwardService/Storage, real SiteHealthCollector + CentralHealthAggregator.
Only the socket hops are stood in for.

Measures: end-to-end tag update latency (the emit instant rides
TagValueUpdate.Timestamp verbatim to the subscriber), instance ramp, memory
growth/CPU over a steady-state window, health report and debug view latency under
load, S&F concurrent buffering + drain throughput, and slow-subscriber isolation.
2026-08-15 02:23:04 -04:00
Joseph Doherty 986e6e7ad5 Merge branch 'arch-review-remediation' 2026-08-15 01:46:31 -04:00
Joseph Doherty 7804fe7958 docs: arch-review remediation — component docs sweep, execution log, residuals register
Final consistency sweep per plan §6: verified component docs against shipped
WP1-WP3 + adversarial-review-fix state, corrected drift found in SiteRuntime
(recursion-exempt run cap, stale ScriptExecutionActor/AlarmExecutionActor
references), TemplateEngine (BundleImporter watermark path), DeploymentManager
(phase-2 PendingDeployment staging), CentralUI (shared KPI cache, dedup'd alarm
poll, render coalescing), StoreAndForward (rate-limited drop logging), and
ConfigurationDatabase (documented DbContext-pooling non-adoption). Updated the
docs/components/ developer-reference set (SiteRuntime, SiteEventLogging,
InboundAPI) to drop the deleted per-run actor classes. Amended one known-issue
for the superseding MaxBatchSize:64 read-page pin. Added CLAUDE.md bullets for
stream graceful-completion reconnect, the required site audit DB path, honest
CLI HTTP timeouts, bulk DeploySiteAsync, and LocalDb 0.2.1. New execution log
records the phase→commit map, gate results, adversarial-review tally, the
three test-flake root causes, and the nine-item residuals register.
2026-08-15 01:30:59 -04:00
Joseph Doherty a9ca51e008 Merge branch 'worktree-agent-a465fb3cd6ec48cd3' into arch-review-remediation 2026-08-14 23:53:00 -04:00
Joseph Doherty fd5e023d08 fix(comms): review findings — consumer-based debug orphan net, foreign-cancel triad, honest onConnected, served-row-exact retirement, full-rate reconcile
F1 (HIGH) DebugStreamBridgeActor: the 5-minute orphan net measured the MAILBOX
(SetReceiveTimeout), and once stream events were correctly marked
INotInfluenceReceiveTimeout nothing recurring reset it — the snapshot lands once
and GrpcStreamStable once — so every healthy session self-terminated at ~6 min
with a false "Site disconnected". Replaced with a periodic self-tick
(ConsumerLivenessCheckInterval, 30s) over a consumer-last-seen stamp renewed only
by DebugStreamConsumerAlive, which DebugStreamService Tells on a shared timer to
every session still in its registry (holding a session there IS "a consumer is
attached" — both the Blazor view and the SignalR hub release it on
dispose/disconnect, and it works headless). Reverting the wrapper was rejected: it
would restore the quiet-instance orphan bug.

F2 (MED) SiteStreamGrpcClient: the RpcException(Cancelled) filter now requires
cts.IsCancellationRequested. A peer-originated / channel-dispose Cancelled fired
none of onError/onCompleted/onConnected, leaving SiteAlarmAggregatorActor with
_streamDown=false forever (IsLive stuck true, reconcile reopen guard never fired).

F3 (MED) SiteStreamGrpcClient: a header TIMEOUT is no longer reported as
connected — that shape is exactly what an unreachable site produces, and it
cleared _streamDown, consumed _seedOnConnect and launched a full snapshot fan-out
at a dead site. AwaitHeadersAsync returns bool; the first received event is the
fallback connected signal, fired at most once from headers OR first event.

F4 (LOW-MED) SqliteAuditWriter.MarkReconciledUpToAsync: the blanket below-cursor
UPDATE retired late-stamped inserts that were never served (then age-purged —
silent loss). The flip is now bounded by insertion order: a Pending row retires
only if its rowid is at or below the high-water mark of rows this instance has
served from ReadPendingSinceAsync (clamped on purge, since SQLite reuses rowids);
Forwarded rows are exempt (central ACKed them over the push path). At-least-once
is unchanged.

F5 (LOW) Documented the liveness dependency (a served row never covered by a later
cursor stays Pending forever; PurgeExpiredAsync never purges Pending) in
ISiteAuditQueue + Component-AuditLog.md, and added a cheap site-health signal:
SiteAuditBacklogReporter logs a rate-limited warning when the existing
oldest-pending metric exceeds 24h.

F6 (MED) SiteAlarmAggregatorActor: _fanoutSinceLastTick was armed by the
reconcile's OWN fan-out, so steady state ran fan-out→skip→fan-out→skip — one
reconcile per 2x interval (120s), halving the not-reporting refresh and the alarm
reconcile backstop. The skip is now armed only by connect/failover-driven seeds
(initial, _seedOnConnect, and a re-seed queued behind one).

Tests: Communication.Tests 691 passed (+13), AuditLog.Tests 382 passed (+5).
2026-08-14 23:52:25 -04:00
Joseph Doherty 11abda1d1f Merge branch 'worktree-agent-aaec6546913f0beae' into arch-review-remediation 2026-08-14 23:51:37 -04:00
Joseph Doherty e0e4b24679 fix(deploy+cli): review findings — honest CLI timeouts, watermark-complete staleness, phase-2 staging, lock-safe cancellation
Six adversarial-review findings, each verified against the code first.

F1 (HIGH) CLI HttpClient capped every call at min(30s, caller timeout),
silently truncating deploy site's 5-minute BulkDeployTimeout and the
5-minute bundle export/preview/import calls — which printed a fake
"504 Request timed out" while the server kept working. HttpClient.Timeout
is now Timeout.InfiniteTimeSpan (the per-call CTS is the single overall
deadline, connect included) with the connect phase bounded separately on
SocketsHttpHandler.ConnectTimeout. The env override is renamed to
SCADABRIDGE_HTTP_CONNECT_TIMEOUT_SECONDS to match its new meaning.

F2 (HIGH) StaleInstanceProbe's process-static memo served stale hashes
because nothing bumped the watermark on three paths:
  (a) BundleImporter commits through the raw DbContext, so no import ever
      moved the watermark — a second import overwriting the same template
      could be OMITTED from ImportResult.StaleInstanceIds. It now bumps
      once per apply ATTEMPT: after the commit, and after the rollback too
      (the probe runs pre-commit, so a rolled-back attempt leaves memos for
      state that never landed; bumping on both paths is the simplest
      correct shape, versus threading transaction awareness through a
      process-static cache).
  (b) CollectWatermarkBumps' default: arm silently no-op'd, contradicting
      its own doc. It now sets unattributed=true — an over-broad bump costs
      extra work, a missed one produces stale work.
  (c) DataConnection edits route through SiteRepository.SaveChangesAsync,
      which had no watermark at all, yet Protocol/Primary+Backup config/
      FailoverRetryCount are revision-hash inputs. It now bumps (BumpAll —
      a connection has no owning template) after a commit that touched one.

F3 (MED) CLI TemplateTableProjection read child ARRAYS, but ListTemplates
now returns database-projected TemplateSummary rows, so template list
printed all zeros. It now prefers the *Count scalars and falls back to
array length (template get still returns full entities). --detail help
text and README corrected: a listing cannot yield definitions, so --detail
renders the raw summary payload and template get --id is the full dump.

F4 (MED) DeploySiteAsync staged every PendingDeployment in phase 1 against
a 5-min TTL while phase 2 reached them one batch at a time, so tail
instances' fetch tokens could expire before their command was sent.
Staging moved into phase 2, immediately before each send; prepare keeps
its flatten/validate/record work. The staging write is the phase's only
repository touch and is serialised behind a 1-permit semaphore, so the
non-thread-safe DbContext constraint holds and the sends stay concurrent.

F5 (MED, latent) DeploySiteAsync leaked every held operation lock if
cancelled — a wedged per-instance semaphore is permanent for the process.
Phase 2 no longer throws (cancellation is recorded as a per-instance
outcome so phase 3 still runs), and an escape from phase 1 or 3 now
unwinds every unfinalised entry: Failed status + lock release.

F6 (LOW) ScriptCompileVerdictCache's promotion wrote hot directly,
bypassing SegmentCapacity (true ceiling 3x against a documented 2x).
Promotion now goes through Store, keeping generational semantics; _hot
and _cold are volatile.

Tests: CLI 396, DeploymentManager 133, ManagementService 494,
TemplateEngine 478, ScriptAnalysis 60, Transport 157, Transport
integration 106, ConfigurationDatabase 366 — all green, 0 build warnings.
The F2/F4/F5 regression tests were each confirmed to FAIL with their fix
reverted.
2026-08-14 23:51:04 -04:00
Joseph Doherty 09e350ab0d Merge branch 'worktree-agent-adf34e265d2dcae96' into arch-review-remediation 2026-08-14 23:47:00 -04:00
Joseph Doherty 5d075f1374 fix(central): review findings — no client-side audit truncation, insert-first upsert, QI-safe scripts, honest operator replies
Six adversarial-review findings in the central SQL/ingest layer.

F1 (AuditLogRepository.InsertChunkAsync) — the set-based ingest declared each
string parameter at its COLUMN width (Actor/Target 256, Action 64, Outcome 16,
Category 32, SourceNode 64), so SqlClient truncated an over-long value at bind
time and committed the mutilated row — silent, in an append-only store, with no
PayloadTruncated flag — while the per-row and reconciliation paths sent the same
value in full and let the server reject it with 2628. Bind at the value's own
length instead; explicit SqlDbType is kept (it fixes the VALUES constructor's
derived column types and datetime2 precision). Design: reject everywhere,
truncate nowhere — matching today's per-row behaviour.

F2 (SiteCallAuditRepository.UpsertAsync) — the single-statement upsert ran the
monotonic UPDATE first and INSERTed only if nothing matched. Two writers racing
the first packet of one TrackedOperationId (the cached dual-write and the
reconciliation pull carry DIFFERENT lifecycle states) both matched nothing, and
the loser then skipped its INSERT or swallowed a 2627 — dropping its
Status/RetryCount/HttpStatus/TerminalAtUtc. Legs swapped to
`IF NOT EXISTS … INSERT; UPDATE <monotonic>` — still one round trip, and the
loser's UPDATE now lands on the winner's row. The duplicate-key catch re-runs
the monotonic UPDATE for the same reason. Moved to raw SQL with explicitly-typed
parameters so the intricate rank predicate exists in exactly one place (an
untyped DateTime would bind as `datetime` and round the freshness tiebreaker).

F3 (docs/plans/sql/*.sql) — filtered-index DDL failed with error 1934 under the
documented `docker exec … sqlcmd` path, which defaults QUOTED_IDENTIFIER OFF;
once IX_Notifications_Delivered exists, QI-OFF DML on Notifications fails too.
All four scripts now open with `SET QUOTED_IDENTIFIER ON; SET ANSI_NULLS ON; GO`
(own batch, so it is in force when the next batch parses), and the migration
convention in Component-ConfigurationDatabase.md documents `sqlcmd -I`. Verified
live: the pre-fix script fails 1934 without -I, the fixed one applies.

F4 (SiteCallAuditActor) — the off-mailbox reconciliation/purge passes reuse the
injected repository, so tests drove one DbContext from the pass and a mailbox
handler concurrently. Serialized at the CALL via a private SerializedRepository
wrapper applied only by the test constructors, rather than running the pass
on-mailbox: production keeps its PipeTo shape untouched, and the existing
"a blocked drain does not stall ingest/query/KPI" regression tests stay
meaningful (they would have been invalidated by suspending the mailbox).

F5 (AuditLogIngestActor) — when the batch failed because the 20 s IngestBudget
expired, the per-row fallback reused the same expired token: N instant failures,
N counter bumps, zero accepted. The fallback now gets a fresh 5 s budget (inside
the 30 s outer Ask), and a blown budget bumps the failure counter ONCE for the
batch instead of once per row.

F6 (NotificationOutboxRepository.UpdateAsync) — ExecuteUpdate's row count was
discarded, so an operator Retry/Discard of a notification the retention purge had
already deleted reported success (the pre-ExecuteUpdate code threw
DbUpdateConcurrencyException). UpdateAsync now returns whether a row matched; the
operator one-shots answer "notification not found" and emit no audit row for the
action that did not happen, while the dispatcher logs a warning (its delivery
already happened; nothing to retry). GetByIdAsync switched to AsNoTracking since
the write is out-of-band.

Tests: 5 new SQL-backed regressions (over-long Target rejected on both paths +
boundary round-trip; concurrent first-write and already-created-by-another-writer
upserts; vanished-row UpdateAsync), a token-identity pin on the ingest fallback,
a repository-concurrency detector for the SiteCallAudit passes, and vanished-row
operator-path tests. The F1/F2/F4 regressions were each confirmed failing against
the pre-fix code. Suites: ConfigurationDatabase 369, AuditLog 378, SiteCallAudit
66, NotificationOutbox 152 — all green, solution builds with 0 warnings.
2026-08-14 23:46:28 -04:00
Joseph Doherty 95f091cff8 Merge branch 'worktree-agent-a6a0dffcb93fa5007' into arch-review-remediation 2026-08-14 23:43:03 -04:00
Joseph Doherty 950c54c5fc fix(runtime): review findings — recursion-safe run cap, atomic detach counter, summary edge cases, per-row event-log fallback 2026-08-14 23:42:29 -04:00
Joseph Doherty f689f495ef chore(deps): LocalDb 0.2.1 — HLC anchor flush on dereg, bounded sync inbox 2026-08-14 23:37:33 -04:00
Joseph Doherty 3913c6b2ac Merge branch 'worktree-agent-a71052048b4ee11d4' into arch-review-remediation 2026-08-14 23:32:36 -04:00
Joseph Doherty 56c99c92c3 fix(ops): wonder site config gains required audit DB path; explicit LocalDb read-page cap; rate-limited observer drop logging
F1: deploy/wonder-app-vd03/appsettings.Site.json (outside git, WP1.2's
StartupValidator gate applies live on next install/upgrade) was missing the
now-required AuditLog:SiteWriter:DatabasePath, added pointing at
E:\ApiInstall\ScadaBridge\site\data\auditlog.db alongside the file's
existing SiteEventLog/LocalDb paths; scanned deploy/ for other Site-role
appsettings with the same gap (none) and confirmed wonder does not pin
LocalDb:Replication:MaxBatchSize (F2 doesn't apply there).

F2: re-pin an explicit LocalDb:Replication:MaxBatchSize=64 on docker/site-a
node-a and node-b. MaxBatchBytes (2 MB default) only bounds the wire
message via the per-message split in SyncSession.PumpLoopAsync;
MaxBatchSize separately bounds the DB read page in
OplogStore.ReadBatchAboveAsync/SnapshotStreamer, which materializes the
whole page into memory before that split runs. Left at the 500 default, a
reconnect drain of worst-case config_json rows could transiently allocate
~35 MB per read even though every wire message stayed within budget.
Updated the CLAUDE.md LocalDb bullet to stop implying the row cap is fully
redundant with the byte budget (topology-guide.md has no matching claim).

F3: StoreAndForwardService's observer-queue onDropped callback logged a
Warning per dropped item, flooding logs at sweep rate for a stuck observer
with a large queue. LogObserverQueueDrop now logs once immediately on the
first drop of an episode, then throttles to at most one rollup Warning per
minute while drops continue, reporting the count dropped since the last
log; the cumulative ObserverQueueDroppedCount counter is unaffected.
Extended StoreAndForwardServiceTests with
ObserverQueue_ManyDropsInOneEpisode_LogsExactlyOneWarning, which floods the
bounded queue and pins exactly one drop-related Warning log for the
episode via a small CapturingLogger test double.

dotnet build ZB.MOM.WW.ScadaBridge.slnx: 0 warnings, 0 errors.
dotnet test StoreAndForward.Tests: 134/134 passed.
dotnet test Host.Tests: 490/490 passed.
2026-08-14 23:31:52 -04:00
Joseph Doherty cfa6acbf48 fix(test): order the cached-drain MarkForwarded assertions behind the push they follow
CachedDrain_OrphanRow_PastGrace_IsAbandoned_AndTheValidRowStillFlows gated on
IngestCachedTelemetryAsync being received once and then asserted, bare, that the
valid row had been marked Forwarded. The drain does that strictly AFTER the push
returns: OnCachedDrainAsync abandons the orphan (:351), pushes the batch (:366),
then parses the ack and marks the accepted ids (:380). Observing the push
therefore orders nothing with respect to the second MarkForwardedAsync — under a
loaded parallel run the post-push continuation can be scheduled after the poll
that saw the push, and the assertion fails fast with "Actually received no
matching calls" while the orphan's own earlier call is reported as the single
non-matching one.

Reproduced deterministically by delaying only the post-push step, which fails
exactly this test (11 siblings still pass) at ~1.3s into the assembly run —
matching the observed failure's fast-fail signature and pointing at line 430.
With the fix the same injected delay passes; suppressing the valid row's
MarkForwarded entirely still fails the test with the identical message, so the
claim (orphan abandoned in its own call, valid row pushed and marked, exactly
once each with exactly the same arguments) is unchanged in force.

Same unsynchronized-assertion class as c4caebe9, different actor. Test-only; the
drain's abandon/push/mark ordering is correct as written.
2026-08-14 23:28:31 -04:00
Joseph Doherty 49fb75c8ba Merge branch 'worktree-agent-a83bdadbbbbe18c6d' into arch-review-remediation 2026-08-14 23:26:24 -04:00
Joseph Doherty 37f13e2eaa fix(dcl): discard in-flight subscribe results for unsubscribed tags; release the orphaned handle 2026-08-14 23:25:48 -04:00
Joseph Doherty c4caebe9b4 fix(test): remove the unsynchronized audit-attempt assertion in the two dispatcher audit-safety tests
Both NotifyDispatcher_AuditWriter_Throws_DeliveryStillSucceeds and
NotificationDispatch_BrokenAuditWriter_StillTransitionsToDelivered read the
throwing writer's attempt counter with a bare Assert immediately after an
AwaitAssert on the Notifications row reaching Delivered. That assumes the audit
writes happen no later than the operational status write, which the dispatcher
deliberately does NOT guarantee: DeliverOneAsync persists the delivery state
first (NotificationOutboxActor.cs:657) and only then emits the Attempted
(:663) and terminal (:676) audit rows — audit is best-effort and must never
gate the user-facing action. Observing Delivered therefore establishes no
happens-before edge with the writer, and under a loaded full-solution parallel
run the continuation after the DB write can be scheduled after the poll that
saw Delivered, so the counter reads 0 and the test fails with "saw 0".

Reproduced deterministically by delaying only the post-update audit emission,
which yields both observed failure messages verbatim; with the fix in place the
same injected delay passes, and suppressing the emissions entirely still fails
both tests with the identical messages — the claims (delivery despite audit
failure, and attempts >= N) are unchanged in force, only the ordering
assumption is gone.

Test-only change; the update-then-audit ordering predates the remediation
(#23 M4) and is correct as written.
2026-08-14 23:12:06 -04:00
Joseph Doherty b1de9dfdd4 Merge branch 'worktree-agent-ada30dd5d1f68b742' into arch-review-remediation 2026-08-14 22:56:47 -04:00
Joseph Doherty c254d0740e perf(sitelog): sampled per-run events; interval run summaries; site_events replication policy pinned
Implements WP3.2 stage (b) per docs/plans/2026-08-15-site-events-policy-design.md.

- Per-run instance-script Started/Completed Info site events are now off by
  default (SiteRuntimeOptions.PerRunScriptEvents=false) instead of firing on
  every run, closing the dominant site_events writer. Gated at the ScriptRunLauncher
  call sites (moved there from ScriptExecutionActor by WP3.1). Error-level events
  (timeout/failure/stuck-watchdog/recursion-limit) remain unconditional.
- ScriptRunSummaryRecorder accumulates per-(instance, script) run counters and a
  new site-only ScriptRunSummaryFlushService emits one aggregate "script" Info
  site event per ScriptRunSummaryIntervalSeconds (default 300s), top-50-script
  breakdown with an "others" rollup, zero-activity intervals emit nothing.
- Per-script opt-in via PerRunScriptEventScripts ("Instance/Script" exact or
  "Instance/*" wildcard), matched by the new pure ScriptRunEventPolicy. All three
  options are read from IOptionsMonitor<SiteRuntimeOptions> per run, so the
  policy is hot-togglable without a restart.
- Fixed the stale "event log is not replicated" comment at AkkaHostedService.cs
  (~905): site_events IS registered in SiteLocalDbSetup.ReplicatedTables — the
  singleton is what makes queries always hit the actively-written copy;
  replication is what gives the singleton history to read after a failover
  (memo Decision (b)). site_events replication itself is unchanged (still
  registered) and already pinned by
  tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbCdcRegistrationTests.cs.
- Updated Component-SiteEventLogging.md (Volume Policy section, corrected
  Storage/replication rationale) and Component-SiteRuntime.md (Script Run
  Launch + Error Handling sections).
2026-08-14 22:56:08 -04:00
Joseph Doherty 799fd041ec Merge branch 'worktree-agent-a2b4268818a1b6201' into arch-review-remediation 2026-08-14 22:37:43 -04:00
Joseph Doherty c4fc1f8ecd perf(runtime): split trigger evals from the blocking pool; bounded, deadline-aware execution 2026-08-14 22:36:15 -04:00
Joseph Doherty cca7f1786d chore(deps): LocalDb 0.2.0 — dereg cleanup, late-opt-in baselining, byte-budget replication 2026-08-14 22:22:18 -04:00
Joseph Doherty 312216ff2b docs(plans): script pool split design — WP3.1a 2026-08-14 21:47:48 -04:00
Joseph Doherty 6cfb2dd858 docs(plans): site_events volume policy design — WP3.2a 2026-08-14 21:45:23 -04:00
Joseph Doherty a5882753dd perf(comms+audit): close phase-2 residuals — direct ingest path, monotonic timeouts, synthetic probe, not-reporting set, cursor-exact audit pull 2026-08-14 21:38:23 -04:00
Joseph Doherty 4cd1441984 Merge branch 'worktree-agent-aae48b78656e5a4e0' into arch-review-remediation
# Conflicts:
#	src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Repositories/SiteExternalSystemRepository.cs
2026-08-14 21:16:11 -04:00
Joseph Doherty edd925e277 Merge branch 'worktree-agent-a0098b40576d74cfd' into arch-review-remediation 2026-08-14 21:15:32 -04:00
Joseph Doherty d494911190 Merge branch 'worktree-agent-acc1e4b5202e79d46' into arch-review-remediation 2026-08-14 21:15:31 -04:00
Joseph Doherty b2181b80bd Merge branch 'worktree-agent-ae22af64445b321d4' into arch-review-remediation 2026-08-14 21:15:31 -04:00
Joseph Doherty f25b75a64d Merge branch 'worktree-agent-a7084b23177344196' into arch-review-remediation 2026-08-14 21:15:31 -04:00
Joseph Doherty d93f96d08a Merge branch 'worktree-agent-a3b474b485c0288de' into arch-review-remediation 2026-08-14 21:15:31 -04:00
Joseph Doherty 48b3c40a7f perf(deploy): flatten-session caching, bulk DeploySiteAsync, paged management queries 2026-08-14 21:14:22 -04:00
Joseph Doherty d15c5f02ea perf(dcl): batch subscribe/read/write seam, bounded reconnect, sharded subscriptions 2026-08-14 21:14:04 -04:00
Joseph Doherty 2ce0ad7ed1 perf(comms): alarms-only seed, capped buffers, at-least-once audit pull 2026-08-14 21:10:19 -04:00
Joseph Doherty 5db2a810c0 perf(central): set-based ingest, aligned partition purge, KPI query shapes, EF hygiene 2026-08-14 21:07:12 -04:00
Joseph Doherty a212283104 perf(misc): cached hot-path lookups, bounded observer queue, alarm-priority stream path
WP2.6 (arch-review remediation, cross-cutting misc):
- SiteExternalSystemRepository: name/ID-indexed ExternalSystemDefinitionCache replaces
  the fetch-all + reverse-map scan on every by-ID/method lookup; loaded once per
  redeploy, invalidated by DeploymentManagerActor after HandleDeployArtifacts applies
  external-system changes. Static JsonSerializerOptions for method-list parsing.
- Inbound API: short-TTL ApiMethodCache fronts the per-request ApiMethod repository
  fetch; invalidated by name via the existing ScriptArtifactChangeSubscriber/
  IScriptArtifactChangeBus pipeline, self-healing via TTL for changes the bus
  doesn't cover (e.g. Management API edits).
- StoreAndForward: the cached-call audit-observer queue — the one unbounded channel
  left in the system — is now bounded (ObserverQueueCapacity, default 10,000) with
  DropOldest overflow and a dropped-notification counter.
- SiteStreamManager: alarm state changes now travel a dedicated publish
  source/broadcast hub, isolated from the (far higher-volume) attribute path, so an
  attribute storm can no longer evict a pending alarm transition; the alarm hand-off
  queue is bounded with a drop counter surfaced on the site health report
  (SiteStreamAlarmDropCount via the new SiteStreamAlarmDropReporter), and publishing
  is skipped entirely at zero subscribers on either path.
- CLI ManagementHttpClient: explicit 30s HttpClient.Timeout on the shared
  construction (was the 100s framework default), overridable via
  SCADABRIDGE_HTTP_TIMEOUT_SECONDS.

Deviation: the failback-probe heartbeat item is NOT included — its only viable
surface (CentralChannelProvider.cs / heartbeat consumers) lives entirely in the
Communication project, explicitly off-limits to this work package this phase.

Tests: SiteRuntime.Tests (550), InboundAPI.Tests (278), StoreAndForward.Tests (133),
CLI.Tests (390), HealthMonitoring.Tests (97) — all green after full solution build.
2026-08-14 20:59:43 -04:00
Joseph Doherty 8c0b36b2fa perf(ui): shared KPI cache, live-cache-backed alarm summary, coalesced debug renders 2026-08-14 20:53:38 -04:00
Joseph Doherty 1040dc0fcc docs(plans): DCL batch seam design — WP2.1a 2026-08-14 20:30:35 -04:00
Joseph Doherty 70cf718949 Merge branch 'worktree-agent-ac58bf5116b74ff84' into arch-review-remediation 2026-08-14 20:14:13 -04:00
Joseph Doherty 628a14c299 Merge branch 'worktree-agent-a44b52ad62ac16433' into arch-review-remediation 2026-08-14 20:14:13 -04:00
Joseph Doherty c6e26a1a6f Merge branch 'worktree-agent-a7e4ca4e5668d81b3' into arch-review-remediation 2026-08-14 20:14:13 -04:00
Joseph Doherty c7a89cbe88 Merge branch 'worktree-agent-a95eaaa8a3181ad4c' into arch-review-remediation 2026-08-14 20:14:13 -04:00
Joseph Doherty da62e01105 Merge branch 'worktree-agent-af8a44154c4e2dbba' into arch-review-remediation 2026-08-14 20:14:13 -04:00
Joseph Doherty 663ce2f14f Merge branch 'worktree-agent-a27e5c199270a6d63' into arch-review-remediation 2026-08-14 20:14:13 -04:00
Joseph Doherty 87e25c790c Merge branch 'worktree-agent-a143c0cc0b4d07e76' into arch-review-remediation 2026-08-14 20:14:13 -04:00
Joseph Doherty 2e4e41a8f7 fix(auditlog): site audit DB onto the data volume; required path + soft flush
Closes WP1.2 of the arch-review remediation plan (finding #2, High):
SqliteAuditWriterOptions.DatabasePath defaulted to CWD-relative "auditlog.db",
which on the docker rig resolves onto the container's ephemeral overlayfs
(not the mounted /app/data volume), silently discarding the pending audit
forward-state backlog on every recreate; nothing in docker/ or docker-env2/
overrode it; FlushIntervalMs was validated but never read by the writer loop
(one commit per event even at trickle rate); and no PRAGMA synchronous was
set (SQLite's FULL default fsyncs every commit).

- DatabasePath now has no default (mirrors ZB.MOM.WW.LocalDb's LocalDbOptions.Path)
  and is required pre-host for Site nodes only, via a new StartupValidator raw-config
  check (top-level "AuditLog:SiteWriter:DatabasePath", NOT nested under ScadaBridge:
  AddAuditLog binds that section off the configuration root). SqliteAuditWriterOptionsValidator
  deliberately does NOT check DatabasePath itself, because AddAuditLog runs its
  ValidateOnStart on both Central and Site composition roots but only Site nodes
  ever resolve the writer — checking it there would fail Central's boot too.
- All 8 site-node appsettings under docker/ and docker-env2/ now set
  AuditLog:SiteWriter:DatabasePath to /app/data/auditlog.db (mounted volume,
  survives container recreate, same convention as LocalDb:Path); the local-dev
  base appsettings.Site.json sets ./data/auditlog.db to match.
- The writer loop now honors FlushIntervalMs: after draining the immediately
  available burst, it keeps the transaction open (bounded by FlushIntervalMs
  from the first event) waiting for more trickle-rate events before committing,
  instead of flushing (and fsyncing) per event.
- PRAGMA synchronous = NORMAL on the write connection — audit is best-effort by
  design (CLAUDE.md: "Audit-write failure NEVER aborts the user-facing action"),
  so NORMAL's narrower power-loss window is an acceptable trade for far fewer
  fsyncs; WAL mode still guarantees no corruption.
- Tests: StartupValidator site-required/blank/central-exempt cases; writer
  trickle-load single-transaction coalescing + beyond-interval separate-transaction
  regression (new FlushCountForTests seam); options-validator doc updates reflecting
  the moved responsibility. Full suite runs green: AuditLog.Tests 368/368,
  Host.Tests 480/480.

One-time migration note: the existing container-local auditlog.db (wherever it
landed under CWD) is abandoned by this change, not migrated — already-forwarded
rows are safe centrally (AuditLog is the durable copy), and any still-Pending
rows on the abandoned path are lost once. This is the exact bug being fixed, not
a new loss: those rows were already living outside the mounted volume and would
not have survived the next container recreate regardless. Cross-reference
docs/known-issues/2026-07-20-cached-telemetry-drain-hot-loop.md, which this
placement bug caused.
2026-08-14 20:13:31 -04:00
Joseph Doherty 7ebdcd370a perf(host): install CDC capture only when replication is configured
SiteLocalDbSetup.OnReady registered all ten replicated tables
unconditionally, so a deliberately unreplicated site node (site-b and
site-c on the rig) carried the full 30-trigger CDC set forever. Every
write to those tables paid two extra INSERTs plus a json_object
serialization of the whole row, inside the caller's own transaction, and
appended to an oplog nothing ever drains. Arch-review finding #5 (High),
repo half; the library half — trigger cleanup API and O(1) backlog — is
WP3.3.

The ten RegisterReplicated calls are now behind a guard on whether the
node has LocalDb:Replication:PeerAddress OR LocalDb:Replication:ApiKey.
Either key counts, and the OR is load-bearing rather than defensive:
replication is one bidirectional stream that exactly one side dials, so
only the initiator sets PeerAddress. Verified against the rig — site-a
node-a has PeerAddress + ApiKey, site-a node-b (passive) has ApiKey
alone, site-b/site-c have no Replication section at all. Keying on
PeerAddress alone would have stripped capture from every passive node and
silently made each pair converge in one direction only.

The load-bearing ordering documented in the file is preserved: DDL still
precedes registration, and the legacy migrator still runs unconditionally
after it — an unreplicated node must still absorb its pre-Phase-1 files,
and it has no peer for those rows to be invisible to.

Known residual, documented in-file and in the topology guide: a database
file first created by an older build keeps its stale __localdb_* triggers.
The guard decides whether triggers are installed, not whether existing
ones are removed, and the library has no removal API until WP3.3. Moot on
the docker rig, where a schema-change redeploy recreates the volumes.
The inverse is also now documented: enabling replication on a site that
has run without it does not baseline existing rows, since CDC never
recorded them in __localdb_row_version and the snapshot resync streams
from that ledger.

Tests: new SiteLocalDbCdcRegistrationTests asserts trigger presence and
absence via sqlite_master across all four config shapes (none, ApiKey
only, PeerAddress + ApiKey, and the notification-table exclusion), plus
DDL-still-runs and migrator-still-runs on the unreplicated branch.
SiteLocalDbWiringTests and the integration site-pair harness now
configure an ApiKey — mirroring the rig's passive node — so their
registration and convergence assertions still describe a replicating
node. 483/483 Host.Tests pass; the 20 offline LocalDb convergence tests
still pass.
2026-08-14 19:59:53 -04:00
Joseph Doherty 125055d998 perf(runtime): O(1) attribute resolution, precomputed types, coalesced static writes, shared JSON options 2026-08-14 19:57:26 -04:00