Commit Graph

91 Commits

Author SHA1 Message Date
Joseph Doherty 1930f19b1a fix(test): relax exact compile-cache hit-count assertion to >= 1
SiteScriptCompileCache.Hits is a process-global counter; other test classes in
the assembly compile scripts concurrently (not in this serialized collection),
so an exact post-Clear count is non-deterministic under a full-assembly parallel
run. Assert.Same(r1,r2) remains the definitive proof of one shared Roslyn
compile. Integration-surfaced; belongs with plan R2-03 T5/T6.
2026-07-13 11:24:06 -04:00
Joseph Doherty ccee558ca7 merge(r2): r2-plan08
# Conflicts:
#	src/ZB.MOM.WW.ScadaBridge.Communication/CommunicationOptionsValidator.cs
2026-07-13 11:10:15 -04:00
Joseph Doherty 3ad5ad0d7e merge(r2): r2-plan03 2026-07-13 11:08:27 -04:00
Joseph Doherty 973e59de84 fix(options): bind + validate OperationTrackingOptions; register the site IOperationTrackingStore that two comments claimed AddSiteRuntime provides (plan R2-08 T8, arch-review 08r2 NF4)
VERIFY-THEN-FIX: the plan-authoring discovery is CONFIRMED. No code anywhere in
src/ registered IOperationTrackingStore in DI, yet AkkaHostedService and
AuditLog SCE both comment that AddSiteRuntime provides it. Every consumer
resolved it via GetService (null-tolerant) and silently ran degraded:
cached-drain scheduler never armed, PullSiteCalls reconciliation seam never
wired, Tracking.Status degraded to audit-only. This is a FUNCTIONAL FIX riding
the hygiene plan: AddSiteRuntime now registers the store (site-only), so
site-local cached-call tracking runs in its intended mode. OperationTrackingOptions
is now bound + eagerly validated in the Host site-options block.
2026-07-13 10:53:27 -04:00
Joseph Doherty 6e1ab62d27 feat(saf): resync ack confirmation + completed/ack-missing telemetry; doc resync rewrite (plan R2-02 T7) 2026-07-13 10:05:33 -04:00
Joseph Doherty 84bc2380f3 fix(saf): standby assembles chunked resync, applies atomically with re-check, acks; N5 race documented (plan R2-02 T6) 2026-07-13 10:00:47 -04:00
Joseph Doherty 3f937b399b fix(saf): chunk the resync snapshot to fit Akka remoting frames (additive protocol, byte-budgeted) (plan R2-02 T5) 2026-07-13 09:55:32 -04:00
Joseph Doherty 128d2bab73 perf(siteruntime): deploy gate + InstanceActor PreStart share one Roslyn compile via the compile cache (plan R2-03 T6) 2026-07-13 09:54:49 -04:00
Joseph Doherty 1cc05f132f feat(siteruntime): bounded process-wide compiled-script cache, keyed on code hash + globals type (plan R2-03 T5) 2026-07-13 09:52:53 -04:00
Joseph Doherty f3bec6922a fix(siteruntime): eagerly validate TagSubscribeRetryIntervalMs + StuckScriptGraceMs at boot (plan R2-03 T4) 2026-07-13 09:51:37 -04:00
Joseph Doherty 15018e4158 fix(siteruntime): faulted expression-eval task clears _evalInFlight instead of killing the AlarmActor trigger (plan R2-03 T3) 2026-07-13 09:50:40 -04:00
Joseph Doherty f149626a76 fix(siteruntime): faulted expression-eval task clears _evalInFlight instead of killing the ScriptActor trigger (plan R2-03 T2) 2026-07-13 09:48:46 -04:00
Joseph Doherty b14a4dbc6b feat(sitestream): add SubscribeSiteAlarms site-wide alarm-only broadcast (plan #10 T1)
Adds SiteStreamManager.SubscribeSiteAlarms(IActorRef) — same broadcast-hub
wiring as Subscribe(...) but with no per-instance filter and forwarding only
AlarmStateChanged events (AttributeValueChanged dropped). Returns a subscription
id torn down via the existing Unsubscribe, symmetric with the per-instance
subscribe. Backs the Task 2 SubscribeSite gRPC server handler.

Claude-Session: https://claude.ai/code/session_01MtdgwpEeCUn6cUA5f1LMPj
2026-07-10 11:39:42 -04:00
Joseph Doherty 22136d36d9 feat(testrun): route WaitForAttribute in the Test-Run sandbox (deferred #14)
Closes deferred-work register item #14 with full fidelity. The Central UI Test-Run
sandbox previously threw ScriptSandboxException for Attributes.WaitAsync/WaitForAsync;
now the value-equality forms route to the bound deployed instance over the same
cross-site path inbound Route.To().WaitForAttribute() uses.

- Commons: additive trailing RouteToWaitForAttributeRequest.RequireGoodQuality
  (default false; message-evolution safe) so quality-gated waits route too.
- SiteRuntime: DeploymentManagerActor's routed-wait handler now threads
  request.RequireGoodQuality into the InstanceActor WaitForAttributeRequest
  (previously hard-coded to default).
- CentralUI sandbox: ISandboxInstanceGateway.WaitForAttributeAsync +
  SandboxInstanceGateway impl (via CommunicationService.RouteToWaitForAttributeAsync);
  SandboxAttributeAccessor value-equality WaitAsync/WaitForAsync route through it
  (codec-encoded target, scope-resolved name). Predicate overloads stay unsupported
  (an in-process lambda can't be routed) and throw a clearer, dedicated message.
- Tests: sandbox accessor routing + predicate/unbound rejection (CentralUI);
  site-handler RequireGoodQuality threading (SiteRuntime). Register + the historical
  waitfor-deferred plan doc updated.

Full slnx build 0/0; CentralUI 59, SiteRuntime routed-wait 4, InboundAPI wait 32 green.

Claude-Session: https://claude.ai/code/session_01MtdgwpEeCUn6cUA5f1LMPj
2026-07-10 09:29:38 -04:00
Joseph Doherty d61881ffaf refactor(site-runtime): excise vestigial site-side notification-list surface — repo, DI registration, dead write paths (arch-review 08 §1.3/#23)
Claude-Session: https://claude.ai/code/session_01MtdgwpEeCUn6cUA5f1LMPj
2026-07-10 07:00:09 -04:00
Joseph Doherty 8b775f4ae1 feat(options): eager startup validation for site-pipeline options (SiteRuntime, S&F, SiteEventLog) (arch-review 08 §1.5)
Claude-Session: https://claude.ai/code/session_01MtdgwpEeCUn6cUA5f1LMPj
2026-07-10 06:53:30 -04:00
Joseph Doherty e3ef320359 feat(esg): TimeoutSeconds travels the artifact pipeline (additive contract) into site SQLite so site-side calls honor it
Claude-Session: https://claude.ai/code/session_01MtdgwpEeCUn6cUA5f1LMPj
2026-07-10 04:07:44 -04:00
Joseph Doherty a608d7a79b feat(site-call-audit): additive (UpdatedAtUtc, TrackedOperationId) keyset in the reconciliation pull contract
Site side: additive proto after_id field, ReadChangedSinceAsync gains an optional
afterId cursor and deterministic (UpdatedAtUtc, TrackedOperationId) ordering — a
batch fully inside one UpdatedAtUtc instant no longer re-reads the same page
forever. Absent/empty afterId preserves the exact legacy inclusive >= contract, so
an older central is unaffected. Regenerated the checked-in gRPC C#.
2026-07-09 08:31:42 -04:00
Joseph Doherty c9ac075e39 fix(site-runtime,dcl): low-severity cleanups — observed adapter dispose (S9), alarm-filter overwrite warning (S10), bounded-channel comment (C2), invariant-culture condition fallback (C6) 2026-07-09 02:09:06 -04:00
Joseph Doherty 29ee9176a7 feat(site-runtime): wire ConfigFetchRetryCount into the standby replicated-config fetch (UA2) 2026-07-09 01:26:58 -04:00
Joseph Doherty c457e8f464 feat(site-runtime): cert-trust reconcile-on-join — singleton pushes trusted certs to (re)joining site nodes (UA1) 2026-07-09 01:22:13 -04:00
Joseph Doherty 39976772d9 feat(site-runtime): CertStoreActor exports trusted certs (thumbprint+DER) for node reconciliation (UA1 groundwork) 2026-07-09 01:15:51 -04:00
Joseph Doherty 2efe1a4202 feat(site-runtime): persist native-alarm display metadata so rehydrated conditions render fully (UA4) 2026-07-09 01:09:13 -04:00
Joseph Doherty 326d945d85 perf(site-runtime): coalesce native-alarm mirror persistence into batched flushes (P4) 2026-07-09 01:05:20 -04:00
Joseph Doherty d2feb92bfd fix(site-runtime): site SQLite runs WAL with an explicit busy-timeout floor (S8) — concurrent writers stop serializing on the rollback journal 2026-07-09 01:00:12 -04:00
Joseph Doherty d124025cc7 perf(site-runtime): per-attribute child routing in InstanceActor — attribute changes reach only interested children (P2) 2026-07-09 00:55:26 -04:00
Joseph Doherty 957db62df1 fix(site-runtime): deployment whose Instance Actor dies during init reports Failed, exactly-once reply (S6)
Success now requires BOTH persistence commit AND an InstanceActorInitialized
readiness signal from the actor's PreStart — persistence can commit before the
actor's async init has run or failed, so persistence alone must not report
Success. An actor that dies during init never signals readiness; the Terminated
fallback fails that deployment and rolls back the optimistic state. The
persisted-row rollback is deferred until the store commits so it cannot race the
optimistic write.

Deviation from plan Task 15: the plan's swallow-only guard did not handle the
persistence-first ordering (empirically the store commits before the Terminated
signal, so Success was reported for a dead actor). Added the readiness handshake
to make the join deterministic.
2026-07-09 00:33:40 -04:00
Joseph Doherty db5ad02cc9 test(site-runtime): harden scheduler OldestBusyAge gauge assertion — assert non-null, not strictly-positive (TickCount64 ~15ms granularity flakes under load) 2026-07-09 00:33:40 -04:00
Joseph Doherty 9343b4f30f fix(site-runtime): watch every Instance Actor; unexpected termination evicts the dead ref from routing (S6) 2026-07-09 00:20:12 -04:00
Joseph Doherty 2dabbbdde0 fix(site-runtime): retry the startup deployed-config load on SQLite failure (S5) — no more silent zero-instance node 2026-07-09 00:17:36 -04:00
Joseph Doherty fa556b4f8d fix(site-runtime): NativeAlarmActor retries a lost subscribe response, not only a failed one (S4) 2026-07-09 00:05:09 -04:00
Joseph Doherty feb6f34e90 fix(site-runtime): InstanceActor retries failed/lost tag subscriptions per connection (S4/UA6) — closes the unknown-connection ordering race 2026-07-09 00:03:44 -04:00
Joseph Doherty bd89c1474e perf(site-runtime): AlarmActor trigger expressions evaluate on the script scheduler (P1) 2026-07-08 23:59:54 -04:00
Joseph Doherty 5aee90d155 perf(site-runtime): ScriptActor trigger expressions evaluate on the script scheduler, coalesced, edge state actor-confined (P1) 2026-07-08 23:50:21 -04:00
Joseph Doherty 467edd74ac feat(site-runtime): stuck-script watchdog names the script holding a scheduler thread past timeout (S2) 2026-07-08 23:46:48 -04:00
Joseph Doherty dddafc025e feat(site-runtime): periodic scheduler-stats reporter feeding site health (S2/UA5) 2026-07-08 23:43:09 -04:00
Joseph Doherty 474ec8a221 feat(site-runtime): script scheduler gauges — queue depth, busy threads, oldest-busy age (S2/UA5) 2026-07-08 23:35:09 -04:00
Joseph Doherty b83d1ed19c fix(site-runtime): site-side compile failure now rejects the deployment per spec (S3) — synchronous validation gate before actor creation
Validation runs on the actor thread (not off-thread as the plan sketched): piping
the verdict back to self reorders concurrent deploys relative to each other and to
delete/disable, breaking the redeploy-supersede FIFO ordering (SR020/SR029). A deploy
is an infrequent admin command, so a brief synchronous Roslyn compile is acceptable.
2026-07-08 23:33:48 -04:00
Joseph Doherty e20c329cad feat(site-runtime): pure deploy-time compile validator for flattened configs (S3 groundwork) 2026-07-08 23:21:42 -04:00
Joseph Doherty ae94b22d7d test(reconciliation): pin inclusive (>=) cursor reads at the site (audit + site-calls); EventId/upsert dedup absorbs boundary duplicates 2026-07-08 21:58:57 -04:00
Joseph Doherty 0321ad0c20 feat(store-and-forward): peer-join full-buffer resync — standby anti-entropy for the S&F buffer 2026-07-08 21:43:11 -04:00
Joseph Doherty 4bdd5f0e4a fix(notifications): Notify.Send enqueues unbounded (maxRetries 0) and defers to sweep — no 30s script-thread block, no stranding 2026-07-08 20:56:09 -04:00
Joseph Doherty eb59c4244f feat(site): per-node startup reconciliation actor (self-heal missing/stale configs) 2026-06-26 16:44:31 -04:00
Joseph Doherty 3e64959582 refactor(deploy): retire cross-cluster DeployInstanceCommand wire path (config now fetched) 2026-06-26 14:32:47 -04:00
Joseph Doherty 5c2db9fe70 feat(site): replicate config by id + standby fetch (kills the intra-site frame trap) 2026-06-26 14:23:20 -04:00
Joseph Doherty 631ce5bfce feat(site): DeploymentManagerActor fetches config then applies (notify-and-fetch) 2026-06-26 13:58:37 -04:00
Joseph Doherty caaa855362 feat(site): older-write guard for replicated config writes
Add StoreDeployedConfigIfNewerAsync to SiteStorageService — an atomic
conditional upsert using SQLite's ON CONFLICT … WHERE clause so the
standby node only overwrites a deployed_configurations row when the
incoming deployed_at is strictly newer. The active-node path
(StoreDeployedConfigAsync) stays unguarded. Four deterministic tests
cover: no-row insert, older-overwrites, newer-is-noop, equal-is-noop;
all seed rows with explicit DateTimeOffset values via direct SQL to
avoid wall-clock timing dependencies.
2026-06-26 13:29:21 -04:00
Joseph Doherty 7a085444e4 feat(site): HTTP deployment-config fetcher + DI + options 2026-06-26 13:24:14 -04:00
Joseph Doherty 9ab1c00265 fix(review): remediate re-review findings — DCL-029/InboundAPI-031/SiteRuntime-032/StoreAndForward-028 + Low doc/test
Fixes the 8 findings from the 2026-06-24 re-review (commit c42bb485), with a
regression test per Medium finding:

- DataConnectionLayer-029 (Med): HandleAlarmSubscribeCompleted now mirrors the
  tag-path re-check — if a feed is already stored for the source, release the
  redundant just-created subscription instead of overwriting + leaking the first
  one (the double-subscribe window DCL-023 reopened). +regression test.
- InboundAPI-031 (Med): remove WaitForAttribute's local 5s grace backstop (tighter
  than the CommunicationService Ask's timeout+IntegrationTimeout round-trip budget,
  so a slow-but-valid timed-out 'false' got cancelled into a 500). Link only the
  client-abort + explicit caller tokens; the lower layer owns the backstop. +test.
- SiteRuntime-032 (Med): derive the deployed count from an authoritative set of
  deployed config names (HashSet) instead of a map-presence-gated int, so deleting
  a DISABLED instance decrements correctly (SiteRuntime-029's gate leaked it).
  +deploy->disable->delete regression test.
- StoreAndForward-028 (Med): reset _bufferedCount in StopAsync alongside the
  register-guard so a same-instance Stop->Start re-seeds from a clean base (no ~2N
  gauge double-count). +restart regression test.
- AuditLog-017 (Low): test the OnIngestAsync scope-resolution guard (actor survives,
  replies empty, counts the failure) — no longer unpinned.
- CentralUI-037 / ScriptAnalysis-009 / SiteRuntime-033 (Low): doc-comment + spec
  fixes (Database-throws in the inbound sandbox; baseReferences param wording;
  native-alarm cap return-to-normal + per-condition NativeAlarmDropped eviction).

Targeted suites green: SiteRuntime 5, StoreAndForward 6, InboundAPI 31,
DataConnectionLayer 10, AuditLog 5, ScriptAnalysis 40, CentralUI ScriptAnalysis 52.
2026-06-24 09:39:14 -04:00
Joseph Doherty fd618cf1dc fix(review): full code-review remediation — 5 High + Medium/Low across 16 modules
Remediation from the full per-module code review at 4307c381 (findings recorded
separately in code-reviews/).

Highs fixed:
- DeploymentManager-025/SiteRuntime-031: stop broadcasting notification lists + SMTP
  configs (incl. credentials) to sites; site purges already-persisted rows on apply
  (enforces the central-only delivery design; clears plaintext SMTP creds at rest).
- DataConnectionLayer-023: guard the native-alarm subscribe path against the
  mid-flight-unsubscribe adapter-feed leak (mirrors the DCL-021 tag-path fix).
- SiteEventLogging-024: normalize From/To query bounds to UTC (the -016 fix the
  audit trail claimed but never committed).
- KpiHistory-001: add an in-flight guard to the recorder sample tick.
- ScriptAnalysis-001: harden the trust analyzer's TPA-absent fallback (resolve
  forbidden anchors in the minimal reference set; warn on degraded mode) — anchors
  added to validation references only, never the compile gate.
(InboundAPI-026 left to the feat/ipsen-movein effort per owner decision.)

Medium/Low: DM-026 deterministic deploy-status tiebreaker; SR-027/028/029/030
native-alarm leak/phantom-active/delete-during-redeploy fixes; AL-013/014/016;
TE-024 (folder-mutation audit rows now persisted)/025; SF-025 gauge-provider
clear-on-stop; ESG-025/026; SEC-023/024/025; SCA-007/008/009; plus doc/test
accuracy COM-023/024, HOST-025/026, HM-024/025, NS-027/028.

Full-solution build 0 warnings; ~3560 tests across 18 touched suites green.
2026-06-20 17:55:12 -04:00