Commit Graph

1080 Commits

Author SHA1 Message Date
Joseph Doherty 1dd89d523b fix(config-db): EnableRetryOnFailure fleet-wide + execution-strategy wrap of the combined-telemetry dual-write
Installs SqlServerRetryingExecutionStrategy (5 retries, 30s cap) so transient SQL
faults retry transparently on every read-side path. Manual transactions aren't
auto-retried, so AuditLogIngestActor's idempotent dual-write is wrapped in an
explicit CreateExecutionStrategy().ExecuteAsync to become retriable. Verified
empirically that existing manual transactions (BundleImporter) are unaffected —
EF skips the strategy while a transaction is already active.
2026-07-09 08:03:30 -04:00
Joseph Doherty 6b06d1efcb fix(site-call-audit): same-rank freshness tiebreaker — retrying calls no longer freeze RetryCount/LastError at first write
Within an equal NON-terminal rank the newest UpdatedAtUtc wins, unfreezing the
Attempted-phase RetryCount/LastError/HttpStatus. Terminal ranks (>=3) are
excluded from the tiebreaker so terminal immutability is preserved (Delivered
never overwrites Parked); equal stamps stay idempotent, lower rank stays a no-op.
2026-07-09 07:56:39 -04:00
Joseph Doherty 1b53de1933 fix(kpi-history): backlogTotal trend records the real site-backlog aggregate instead of a hardwired zero 2026-07-09 07:51:55 -04:00
Joseph Doherty 5d0fcd4e66 perf(audit-log): bound GetExecutionTreeAsync edge scan to a root-anchored time window (partition elimination) 2026-07-09 07:10:35 -04:00
Joseph Doherty f1957606ff perf(site-call-audit): filtered IX_SiteCalls_NonTerminal index — KPI predicates seek the live queue instead of scanning 365 days 2026-07-09 07:06:50 -04:00
Joseph Doherty 7555e65746 fix(config-db): grant CREATE TABLE + scoped DELETE to scadabridge_audit_purger so a segregated maintenance principal can actually run the purge 2026-07-09 06:52:44 -04:00
Joseph Doherty 14c4df54f9 feat(audit-log): surface partition-purge failure as health event + counter 2026-07-09 06:49:28 -04:00
Joseph Doherty b59aa2d717 fix(audit-log): explicit long CommandTimeout on partition-switch and per-channel purge maintenance paths 2026-07-09 06:43:23 -04:00
Joseph Doherty 74aa872c07 feat(audit-log): daily site SQLite retention purge job (closes unbounded site DB growth)
SiteAuditRetentionService (site-only IHostedService) ticks PurgeExpiredAsync on
InitialDelay(5m)/PurgeInterval(24h, clamped >=1m), cutoff = UtcNow - RetentionDays.
Registered site-only in AddAuditLogHealthMetricsBridge; options bound from
AuditLog:SiteRetention. Per-tick failures swallowed. Added a test-only
PurgeIntervalOverride (mirrors SiteCallAuditOptions) so the ms-cadence tick test
can bypass the 1-min production clamp. Doc updated. (PLAN-04 Task 3, S1/U2)
2026-07-09 06:22:19 -04:00
Joseph Doherty b5271da5c8 feat(audit-log): SiteAuditRetentionOptions (7-day default, clamped 1-90)
RetentionDays clamped [1,90]; PurgeInterval clamped to >= 1 min (Timer spin
footgun); 5-min InitialDelay so a daily-recycled node still purges. (PLAN-04 Task 2)
2026-07-09 06:13:05 -04:00
Joseph Doherty 731ee69797 feat(audit-log): add site SQLite PurgeExpiredAsync honoring the ForwardState invariant
Retention purge deletes Forwarded/Reconciled rows older than the cutoff and
reclaims pages via incremental_vacuum; Pending rows are never purged on age
alone (would lose audit data that never reached central). Temp-table purge set
avoids the SQLite 999-param limit; runs under _writeLock. (PLAN-04 Task 1, S1/U2)
2026-07-09 06:11:54 -04:00
Joseph Doherty 9049690bdb fix(centralui-tests): register stub IBrowseService in SecuredWritesTests harness (PLAN-07 known-issue resolved)
The Secured Writes page embeds NodeBrowserDialog (@inject IBrowseService) for
the OPC UA tag picker; bUnit resolves the injected service at render time, so
three SecuredWritesTests failed with 'no registered service of type
IBrowseService'. Register a Substitute.For<IBrowseService>() singleton in the
fixture (mirrors NodeBrowserDialogSearchTests). Suite now 7/7 green.

Marks the PLAN-07 'Post-plan known issues' entry and the 00-MASTER-TRACKER
registry cross-reference RESOLVED.
2026-07-09 05:55:53 -04:00
Joseph Doherty c9ac075e39 fix(site-runtime,dcl): low-severity cleanups — observed adapter dispose (S9), alarm-filter overwrite warning (S10), bounded-channel comment (C2), invariant-culture condition fallback (C6) 2026-07-09 02:09:06 -04:00
Joseph Doherty 29ee9176a7 feat(site-runtime): wire ConfigFetchRetryCount into the standby replicated-config fetch (UA2) 2026-07-09 01:26:58 -04:00
Joseph Doherty c457e8f464 feat(site-runtime): cert-trust reconcile-on-join — singleton pushes trusted certs to (re)joining site nodes (UA1) 2026-07-09 01:22:13 -04:00
Joseph Doherty 39976772d9 feat(site-runtime): CertStoreActor exports trusted certs (thumbprint+DER) for node reconciliation (UA1 groundwork) 2026-07-09 01:15:51 -04:00
Joseph Doherty 2efe1a4202 feat(site-runtime): persist native-alarm display metadata so rehydrated conditions render fully (UA4) 2026-07-09 01:09:13 -04:00
Joseph Doherty 326d945d85 perf(site-runtime): coalesce native-alarm mirror persistence into batched flushes (P4) 2026-07-09 01:05:20 -04:00
Joseph Doherty d2feb92bfd fix(site-runtime): site SQLite runs WAL with an explicit busy-timeout floor (S8) — concurrent writers stop serializing on the rollback journal 2026-07-09 01:00:12 -04:00
Joseph Doherty d124025cc7 perf(site-runtime): per-attribute child routing in InstanceActor — attribute changes reach only interested children (P2) 2026-07-09 00:55:26 -04:00
Joseph Doherty 1673268aee perf(dcl): tag->instances reverse index replaces per-update all-instances scan (P3) 2026-07-09 00:48:22 -04:00
Joseph Doherty f294b46263 fix(dcl): capture adapter into locals before background tasks — no off-thread reads of the mutable _adapter field (S7) 2026-07-09 00:44:09 -04:00
Joseph Doherty 957db62df1 fix(site-runtime): deployment whose Instance Actor dies during init reports Failed, exactly-once reply (S6)
Success now requires BOTH persistence commit AND an InstanceActorInitialized
readiness signal from the actor's PreStart — persistence can commit before the
actor's async init has run or failed, so persistence alone must not report
Success. An actor that dies during init never signals readiness; the Terminated
fallback fails that deployment and rolls back the optimistic state. The
persisted-row rollback is deferred until the store commits so it cannot race the
optimistic write.

Deviation from plan Task 15: the plan's swallow-only guard did not handle the
persistence-first ordering (empirically the store commits before the Terminated
signal, so Success was reported for a dead actor). Added the readiness handshake
to make the join deterministic.
2026-07-09 00:33:40 -04:00
Joseph Doherty db5ad02cc9 test(site-runtime): harden scheduler OldestBusyAge gauge assertion — assert non-null, not strictly-positive (TickCount64 ~15ms granularity flakes under load) 2026-07-09 00:33:40 -04:00
Joseph Doherty 9343b4f30f fix(site-runtime): watch every Instance Actor; unexpected termination evicts the dead ref from routing (S6) 2026-07-09 00:20:12 -04:00
Joseph Doherty 2dabbbdde0 fix(site-runtime): retry the startup deployed-config load on SQLite failure (S5) — no more silent zero-instance node 2026-07-09 00:17:36 -04:00
Joseph Doherty fa556b4f8d fix(site-runtime): NativeAlarmActor retries a lost subscribe response, not only a failed one (S4) 2026-07-09 00:05:09 -04:00
Joseph Doherty feb6f34e90 fix(site-runtime): InstanceActor retries failed/lost tag subscriptions per connection (S4/UA6) — closes the unknown-connection ordering race 2026-07-09 00:03:44 -04:00
Joseph Doherty bd89c1474e perf(site-runtime): AlarmActor trigger expressions evaluate on the script scheduler (P1) 2026-07-08 23:59:54 -04:00
Joseph Doherty 5aee90d155 perf(site-runtime): ScriptActor trigger expressions evaluate on the script scheduler, coalesced, edge state actor-confined (P1) 2026-07-08 23:50:21 -04:00
Joseph Doherty 467edd74ac feat(site-runtime): stuck-script watchdog names the script holding a scheduler thread past timeout (S2) 2026-07-08 23:46:48 -04:00
Joseph Doherty dddafc025e feat(site-runtime): periodic scheduler-stats reporter feeding site health (S2/UA5) 2026-07-08 23:43:09 -04:00
Joseph Doherty 7d50555fd3 feat(health): additive script-scheduler stats on ISiteHealthCollector + SiteHealthReport (S2/UA5) 2026-07-08 23:37:50 -04:00
Joseph Doherty 474ec8a221 feat(site-runtime): script scheduler gauges — queue depth, busy threads, oldest-busy age (S2/UA5) 2026-07-08 23:35:09 -04:00
Joseph Doherty b83d1ed19c fix(site-runtime): site-side compile failure now rejects the deployment per spec (S3) — synchronous validation gate before actor creation
Validation runs on the actor thread (not off-thread as the plan sketched): piping
the verdict back to self reorders concurrent deploys relative to each other and to
delete/disable, breaking the redeploy-supersede FIFO ordering (SR020/SR029). A deploy
is an infrequent admin command, so a brief synchronous Roslyn compile is acceptable.
2026-07-08 23:33:48 -04:00
Joseph Doherty e20c329cad feat(site-runtime): pure deploy-time compile validator for flattened configs (S3 groundwork) 2026-07-08 23:21:42 -04:00
Joseph Doherty a8a01f026b fix(dcl): cancel old event loop + dispose old MxGateway client on reconnect (S1) 2026-07-08 23:19:11 -04:00
Joseph Doherty 11eb5157e6 fix(dcl): dispose+detach previous OPC UA client on reconnect (S1) — stops session leak and stale ConnectionLost flapping 2026-07-08 23:16:27 -04:00
Joseph Doherty ae94b22d7d test(reconciliation): pin inclusive (>=) cursor reads at the site (audit + site-calls); EventId/upsert dedup absorbs boundary duplicates 2026-07-08 21:58:57 -04:00
Joseph Doherty 846279f6a8 chore(communication): low-severity cleanup — eviction counting, shared ingest timeout, split app heartbeat interval, doc notes 2026-07-08 21:54:42 -04:00
Joseph Doherty 11c72ecf0f feat(health): oldest-parked-message age on the site health report (parked-retention aging signal) 2026-07-08 21:49:29 -04:00
Joseph Doherty 0321ad0c20 feat(store-and-forward): peer-join full-buffer resync — standby anti-entropy for the S&F buffer 2026-07-08 21:43:11 -04:00
Joseph Doherty 3eef9d1077 feat(store-and-forward): GetAllMessagesAsync/ReplaceAllAsync storage primitives for standby buffer resync 2026-07-08 21:22:25 -04:00
Joseph Doherty c6ea332c94 perf(store-and-forward): post audit-observer notifications to an ordered single-reader channel, latency-isolating the sweep 2026-07-08 21:19:59 -04:00
Joseph Doherty 5fe1b9747b perf(store-and-forward): epoch-ms due-check with additive column, backfill, and (status, due) index 2026-07-08 21:14:26 -04:00
Joseph Doherty 499a37ac78 perf(store-and-forward): enable WAL + per-connection busy_timeout on the S&F SQLite store 2026-07-08 21:01:53 -04:00
Joseph Doherty 7626d5ba34 test(communication): pin wire round-trip + type identity of cross-cluster message contracts 2026-07-08 21:00:02 -04:00
Joseph Doherty 4bdd5f0e4a fix(notifications): Notify.Send enqueues unbounded (maxRetries 0) and defers to sweep — no 30s script-thread block, no stranding 2026-07-08 20:56:09 -04:00
Joseph Doherty c3a8576863 fix(notifications): park corrupt buffered notification payloads instead of silently discarding as delivered (supersedes StoreAndForward-018) 2026-07-08 20:48:16 -04:00
Joseph Doherty 05a71a0260 feat(store-and-forward): deferToSweep enqueue mode with immediate sweep kick for latency-sensitive callers 2026-07-08 20:46:08 -04:00