Commit Graph

64 Commits

Author SHA1 Message Date
Joseph Doherty db5ad02cc9 test(site-runtime): harden scheduler OldestBusyAge gauge assertion — assert non-null, not strictly-positive (TickCount64 ~15ms granularity flakes under load) 2026-07-09 00:33:40 -04:00
Joseph Doherty 9343b4f30f fix(site-runtime): watch every Instance Actor; unexpected termination evicts the dead ref from routing (S6) 2026-07-09 00:20:12 -04:00
Joseph Doherty 2dabbbdde0 fix(site-runtime): retry the startup deployed-config load on SQLite failure (S5) — no more silent zero-instance node 2026-07-09 00:17:36 -04:00
Joseph Doherty fa556b4f8d fix(site-runtime): NativeAlarmActor retries a lost subscribe response, not only a failed one (S4) 2026-07-09 00:05:09 -04:00
Joseph Doherty feb6f34e90 fix(site-runtime): InstanceActor retries failed/lost tag subscriptions per connection (S4/UA6) — closes the unknown-connection ordering race 2026-07-09 00:03:44 -04:00
Joseph Doherty bd89c1474e perf(site-runtime): AlarmActor trigger expressions evaluate on the script scheduler (P1) 2026-07-08 23:59:54 -04:00
Joseph Doherty 5aee90d155 perf(site-runtime): ScriptActor trigger expressions evaluate on the script scheduler, coalesced, edge state actor-confined (P1) 2026-07-08 23:50:21 -04:00
Joseph Doherty 467edd74ac feat(site-runtime): stuck-script watchdog names the script holding a scheduler thread past timeout (S2) 2026-07-08 23:46:48 -04:00
Joseph Doherty dddafc025e feat(site-runtime): periodic scheduler-stats reporter feeding site health (S2/UA5) 2026-07-08 23:43:09 -04:00
Joseph Doherty 474ec8a221 feat(site-runtime): script scheduler gauges — queue depth, busy threads, oldest-busy age (S2/UA5) 2026-07-08 23:35:09 -04:00
Joseph Doherty b83d1ed19c fix(site-runtime): site-side compile failure now rejects the deployment per spec (S3) — synchronous validation gate before actor creation
Validation runs on the actor thread (not off-thread as the plan sketched): piping
the verdict back to self reorders concurrent deploys relative to each other and to
delete/disable, breaking the redeploy-supersede FIFO ordering (SR020/SR029). A deploy
is an infrequent admin command, so a brief synchronous Roslyn compile is acceptable.
2026-07-08 23:33:48 -04:00
Joseph Doherty e20c329cad feat(site-runtime): pure deploy-time compile validator for flattened configs (S3 groundwork) 2026-07-08 23:21:42 -04:00
Joseph Doherty ae94b22d7d test(reconciliation): pin inclusive (>=) cursor reads at the site (audit + site-calls); EventId/upsert dedup absorbs boundary duplicates 2026-07-08 21:58:57 -04:00
Joseph Doherty 0321ad0c20 feat(store-and-forward): peer-join full-buffer resync — standby anti-entropy for the S&F buffer 2026-07-08 21:43:11 -04:00
Joseph Doherty 4bdd5f0e4a fix(notifications): Notify.Send enqueues unbounded (maxRetries 0) and defers to sweep — no 30s script-thread block, no stranding 2026-07-08 20:56:09 -04:00
Joseph Doherty eb59c4244f feat(site): per-node startup reconciliation actor (self-heal missing/stale configs) 2026-06-26 16:44:31 -04:00
Joseph Doherty 3e64959582 refactor(deploy): retire cross-cluster DeployInstanceCommand wire path (config now fetched) 2026-06-26 14:32:47 -04:00
Joseph Doherty 5c2db9fe70 feat(site): replicate config by id + standby fetch (kills the intra-site frame trap) 2026-06-26 14:23:20 -04:00
Joseph Doherty 631ce5bfce feat(site): DeploymentManagerActor fetches config then applies (notify-and-fetch) 2026-06-26 13:58:37 -04:00
Joseph Doherty caaa855362 feat(site): older-write guard for replicated config writes
Add StoreDeployedConfigIfNewerAsync to SiteStorageService — an atomic
conditional upsert using SQLite's ON CONFLICT … WHERE clause so the
standby node only overwrites a deployed_configurations row when the
incoming deployed_at is strictly newer. The active-node path
(StoreDeployedConfigAsync) stays unguarded. Four deterministic tests
cover: no-row insert, older-overwrites, newer-is-noop, equal-is-noop;
all seed rows with explicit DateTimeOffset values via direct SQL to
avoid wall-clock timing dependencies.
2026-06-26 13:29:21 -04:00
Joseph Doherty 7a085444e4 feat(site): HTTP deployment-config fetcher + DI + options 2026-06-26 13:24:14 -04:00
Joseph Doherty 9ab1c00265 fix(review): remediate re-review findings — DCL-029/InboundAPI-031/SiteRuntime-032/StoreAndForward-028 + Low doc/test
Fixes the 8 findings from the 2026-06-24 re-review (commit c42bb485), with a
regression test per Medium finding:

- DataConnectionLayer-029 (Med): HandleAlarmSubscribeCompleted now mirrors the
  tag-path re-check — if a feed is already stored for the source, release the
  redundant just-created subscription instead of overwriting + leaking the first
  one (the double-subscribe window DCL-023 reopened). +regression test.
- InboundAPI-031 (Med): remove WaitForAttribute's local 5s grace backstop (tighter
  than the CommunicationService Ask's timeout+IntegrationTimeout round-trip budget,
  so a slow-but-valid timed-out 'false' got cancelled into a 500). Link only the
  client-abort + explicit caller tokens; the lower layer owns the backstop. +test.
- SiteRuntime-032 (Med): derive the deployed count from an authoritative set of
  deployed config names (HashSet) instead of a map-presence-gated int, so deleting
  a DISABLED instance decrements correctly (SiteRuntime-029's gate leaked it).
  +deploy->disable->delete regression test.
- StoreAndForward-028 (Med): reset _bufferedCount in StopAsync alongside the
  register-guard so a same-instance Stop->Start re-seeds from a clean base (no ~2N
  gauge double-count). +restart regression test.
- AuditLog-017 (Low): test the OnIngestAsync scope-resolution guard (actor survives,
  replies empty, counts the failure) — no longer unpinned.
- CentralUI-037 / ScriptAnalysis-009 / SiteRuntime-033 (Low): doc-comment + spec
  fixes (Database-throws in the inbound sandbox; baseReferences param wording;
  native-alarm cap return-to-normal + per-condition NativeAlarmDropped eviction).

Targeted suites green: SiteRuntime 5, StoreAndForward 6, InboundAPI 31,
DataConnectionLayer 10, AuditLog 5, ScriptAnalysis 40, CentralUI ScriptAnalysis 52.
2026-06-24 09:39:14 -04:00
Joseph Doherty fd618cf1dc fix(review): full code-review remediation — 5 High + Medium/Low across 16 modules
Remediation from the full per-module code review at 4307c381 (findings recorded
separately in code-reviews/).

Highs fixed:
- DeploymentManager-025/SiteRuntime-031: stop broadcasting notification lists + SMTP
  configs (incl. credentials) to sites; site purges already-persisted rows on apply
  (enforces the central-only delivery design; clears plaintext SMTP creds at rest).
- DataConnectionLayer-023: guard the native-alarm subscribe path against the
  mid-flight-unsubscribe adapter-feed leak (mirrors the DCL-021 tag-path fix).
- SiteEventLogging-024: normalize From/To query bounds to UTC (the -016 fix the
  audit trail claimed but never committed).
- KpiHistory-001: add an in-flight guard to the recorder sample tick.
- ScriptAnalysis-001: harden the trust analyzer's TPA-absent fallback (resolve
  forbidden anchors in the minimal reference set; warn on degraded mode) — anchors
  added to validation references only, never the compile gate.
(InboundAPI-026 left to the feat/ipsen-movein effort per owner decision.)

Medium/Low: DM-026 deterministic deploy-status tiebreaker; SR-027/028/029/030
native-alarm leak/phantom-active/delete-during-redeploy fixes; AL-013/014/016;
TE-024 (folder-mutation audit rows now persisted)/025; SF-025 gauge-provider
clear-on-stop; ESG-025/026; SEC-023/024/025; SCA-007/008/009; plus doc/test
accuracy COM-023/024, HOST-025/026, HM-024/025, NS-027/028.

Full-solution build 0 warnings; ~3560 tests across 18 touched suites green.
2026-06-20 17:55:12 -04:00
Joseph Doherty f4e03ce8f7 fix(site-runtime): normalize routed GetAttributes List values for cross-process transport (#162) 2026-06-19 01:49:38 -04:00
Joseph Doherty 2d139442ba fix(siteruntime): CertStoreActor — dispose listed certs + reject path-traversal thumbprints (T17) 2026-06-18 03:42:17 -04:00
Joseph Doherty c8d9303031 feat(siteruntime): per-node CertStore actor + trust broadcast to both site nodes (T17) 2026-06-18 03:13:48 -04:00
Joseph Doherty b5347faf44 fix(DV-2): clarify placeholder comment, stable MinValue timestamp, drain DCL probe in new tests
- Replace placeholder-loop comment with the double-render guard explanation
- Use _alarmTimestamps.GetValueOrDefault(binding, DateTimeOffset.MinValue) so the
  placeholder timestamp is stable/idempotent across snapshot calls (was UtcNow)
- Add dcl.ExpectMsg<SubscribeAlarmsRequest>() drain in Snapshot_QuietNativeBinding_EmitsPlaceholder
  and Snapshot_NativeBindingWithLiveCondition_NoPlaceholder to consume the DCL message
  the NativeAlarmActor sends at startup
2026-06-17 15:08:37 -04:00
Joseph Doherty 5d07ac24cb feat(debugview): DV-2 emit placeholder rows for quiet native alarm bindings
InstanceActor.BuildAlarmStatesSnapshot now adds an IsConfiguredPlaceholder
row per configured native source binding that currently has no live
condition, so the Debug View tree can show the binding node even when
quiet. A binding is "quiet" when no retained AlarmStateChanged carries its
NativeSourceCanonicalName (DV-1).

Kind derivation: reuses the exact nativeKind value already computed via
ResolveNativeKind(nativeSource.ConnectionName) at the NativeAlarmActor
creation site and stored in a new _nativeAlarmKinds dictionary -- the
accurate per-binding kind (NativeOpcUa vs NativeMxAccess), not the
NativeOpcUa default.

Tests: Snapshot_QuietNativeBinding_EmitsPlaceholder,
Snapshot_NativeBindingWithLiveCondition_NoPlaceholder.
2026-06-17 15:00:20 -04:00
Joseph Doherty 899ad6e106 feat(debugview): DV-1 native-binding linkage on AlarmStateChanged contract chain
Add two additive init-only fields to AlarmStateChanged so the Debug View can
nest live native conditions under their configured source-binding node:
  - NativeSourceCanonicalName (binding canonical name, e.g. "Motor1.MotorAlarms")
  - IsConfiguredPlaceholder (quiet-binding placeholder flag; default false)

Flow on BOTH cross-process paths:
  - Live: proto AlarmStateUpdate fields 22/23 -> StreamRelayActor packs ->
    SiteStreamGrpcClient unpacks (regenerated SiteStreamGrpc/Sitestream.cs).
  - Snapshot (Newtonsoft): record defaults carry through; no special handling.

NativeAlarmActor.Emit now stamps NativeSourceCanonicalName = _source.CanonicalName.
Additive-only: no existing positional constructor or wire frame changed.

Tests: StreamRelayActorTests round-trips both fields pack->unpack;
NativeAlarmActorTests asserts the emitted event carries the binding canonical name.
2026-06-17 14:52:03 -04:00
Joseph Doherty 8dcc55f633 test: address review feedback — cover WaitForAsync predicate overload; clarify ChildrenAccessor parity row; dedup object-method exclusion 2026-06-17 11:16:33 -04:00
Joseph Doherty b88f04ec2d fix(siteruntime): normalize routed WaitForAttribute response value for cross-process transport 2026-06-17 11:10:17 -04:00
Joseph Doherty adc8ee4afa test(scriptanalysis): parity test fails on any unmirrored runtime accessor method 2026-06-17 11:09:00 -04:00
Joseph Doherty 11534089b9 docs(siteruntime): mark WaitAsync deferred items implemented (§3/§4.2/§6) + fast-path throwing-predicate test 2026-06-17 09:15:42 -04:00
Joseph Doherty c482cac110 feat(siteruntime): unpack routed RouteToWaitForAttributeRequest into InstanceActor (spec §6 site half) 2026-06-17 09:10:08 -04:00
Joseph Doherty 61048a4ecf feat(siteruntime): WaitForAsync/WaitResult + quality-gated WaitAsync (spec §3, §4.2) 2026-06-17 09:05:12 -04:00
Joseph Doherty 04e97f4a87 fix(siteruntime): harden WaitAsync — no spurious match on quality republish, guard throwing predicate, Ask-timeout returns false 2026-06-17 08:44:03 -04:00
Joseph Doherty 75ffa09b8f feat(siteruntime): event-driven Attributes.WaitAsync attribute-change helper
Adds InstanceActor one-shot waiter registry (fast-path + change-match + scheduled
timeout self-eviction), threads per-script timeout token through ScriptRuntimeContext,
and exposes Attributes.WaitAsync(value|predicate, timeout). Replaces handshake busy-poll.
Implements spec docs/plans/2026-06-17-waitfor-attribute-change-helper-spec.md §3-§5;
§6 routed variant + WaitForAsync + quality-only mode deferred.
2026-06-17 08:25:06 -04:00
Joseph Doherty 20760014c2 feat(audit): M5.4 ParentExecutionId tag-cascade for alarm + nested calls (T4) 2026-06-16 21:42:14 -04:00
Joseph Doherty 069757209a fix(scriptanalysis): M3.6 — full-framework analysis refs close forbidden-type-in-allowed-ns blind spot; pin Process/Stopwatch; fix stale codec test; drop dead ContainsInCode 2026-06-16 20:00:28 -04:00
Joseph Doherty 64d6ac7288 refactor(siteruntime): M3.3 ValidateTrustModel delegates to shared ScriptAnalysis + compile-surface parity test 2026-06-16 19:37:50 -04:00
Joseph Doherty 5841cec958 feat(siteruntime): normalize old-form List static overrides to native JSON on load 2026-06-16 17:49:21 -04:00
Joseph Doherty 94be5e813b fix(siteruntime): decode List value to typed array before DCL write (OPC UA array write path) 2026-06-16 16:48:28 -04:00
Joseph Doherty ad6bfc8af9 fix(siteruntime): reject SetStaticAttribute with malformed list value (no silent poison persist) 2026-06-16 15:59:30 -04:00
Joseph Doherty 7f97780bb3 feat(siteruntime): decode static List attributes to typed lists in InstanceActor (load/override/set) 2026-06-16 15:52:29 -04:00
Joseph Doherty 96e817a7e1 fix(siteruntime): MV-8 review fixes (construct list inside try; dictionary attr lookup; test hygiene) 2026-06-16 15:48:25 -04:00
Joseph Doherty 4765706e94 feat(dcl): coerce OPC UA array reads to typed list attributes; Bad quality on element mismatch 2026-06-16 15:39:19 -04:00
Joseph Doherty a1d464b50d fix(siteruntime): encode list attribute writes via AttributeValueCodec (was .ToString())
Replace value?.ToString() with AttributeValueCodec.Encode(value) in
AttributeAccessor indexer set and SetAsync, so a List<string>{"a","b"}
encodes to ["a","b"] instead of the garbage ToString representation.
Add using ZB.MOM.WW.ScadaBridge.Commons.Types. Tests verify the codec
contract (list→JSON array, scalar passthrough, null); full round-trip
through the accessor is not viable without a live Akka ActorSystem —
noted in-test with explanation.
2026-06-16 15:38:00 -04:00
Joseph Doherty e2b31a9fd2 fix(siteruntime): M2.12 review nits — observe logger fault + meaningful source fallback (#25)
Replace bare task-discard with ContinueWith(OnlyOnFaulted|ExecuteSynchronously) so a
faulted ISiteEventLogger is logged and swallowed rather than going to the unobserved-task
firehose. Replace the "ScriptRuntimeContext" class-name fallback with the meaningful
"InstanceScript:{instanceName}" identifier (matching the site-event-log source convention).
Update the method doc-comment to state the best-effort contract explicitly. Pin the new
fallback value in the shape-precision test.
2026-06-16 06:26:00 -04:00
Joseph Doherty f08038db23 feat(siteruntime): M2.12 (#25) — emit script Error site event on recursion-limit violation
Inject ISiteEventLogger into ScriptRuntimeContext (additive optional ctor
param, defaulted null, all existing callers source-compatible). Add a single
private EmitRecursionLimitEventAsync helper that fires-and-forgets a
"script"/Error site event; called at both recursion guard sites (CallScript
at ~:332 and ScriptCallHelper.CallShared at ~:499). ScriptExecutionActor
threads the already-resolved siteEventLogger singleton into the context;
AlarmExecutionActor leaves it null (no siteEventLogger wired there).

Existing _logger.LogError + throw behaviour unchanged.

Tests: RecursionLimitSiteEventTests — 5 tests covering both CallScript and
CallShared (ISiteEventLogger.LogEventAsync called once with category "script",
severity "Error"; null logger path does not throw).
2026-06-16 06:20:58 -04:00
Joseph Doherty dbf44b9e10 fix(siteruntime): M2.11 — unknown-instance debug snapshot returns InstanceNotFound=true (#24)
RouteDebugSnapshot and RouteDebugViewSubscribe on DeploymentManagerActor
previously returned an empty DebugViewSnapshot for unknown instances,
indistinguishable from a deployed-but-empty instance. Callers had no way
to differentiate "not deployed here" from "deployed, no data yet."

Approach — additive field on existing message contract:
  Added `bool InstanceNotFound = false` as an optional trailing parameter
  to DebugViewSnapshot (Commons). All existing positional constructor calls
  and serialized wire frames are unaffected (default = false). A dedicated
  new message type was considered but rejected: the ClusterClient channel
  and DebugStreamService TCS are already typed on DebugViewSnapshot, and a
  second reply union would require wider changes for zero additive-safety
  gain.

Changes:
  - Commons/DebugViewSnapshot: add InstanceNotFound = false (additive)
  - DeploymentManagerActor: set InstanceNotFound=true in both unknown-
    instance branches (RouteDebugViewSubscribe, RouteDebugSnapshot)
  - DebugStreamBridgeActor: when snapshot.InstanceNotFound, forward it to
    _onEvent (resolves the TCS) then stop cleanly; no gRPC stream opened
  - DebugView.razor: check session.InitialSnapshot.InstanceNotFound after
    connect and show a clear "not deployed on this site" error toast
  - 3 new tests in DeploymentManagerActorTests covering: unknown→snapshot,
    unknown→subscribe, known-empty→InstanceNotFound stays false
2026-06-16 06:08:21 -04:00