From 0b201e410cb9ffd7d647c62efe798aaa8de3eac8 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Fri, 14 Aug 2026 19:23:56 -0400 Subject: [PATCH 01/31] =?UTF-8?q?docs(plans):=20arch-review=20remediation?= =?UTF-8?q?=20plan=20=E2=80=94=20multi-phase,=20subagent-executable=20prog?= =?UTF-8?q?ram=20for=20all=202026-08-14=20review=20findings?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- ...2026-08-14-arch-review-remediation-plan.md | 533 ++++++++++++++++++ 1 file changed, 533 insertions(+) create mode 100644 docs/plans/2026-08-14-arch-review-remediation-plan.md diff --git a/docs/plans/2026-08-14-arch-review-remediation-plan.md b/docs/plans/2026-08-14-arch-review-remediation-plan.md new file mode 100644 index 00000000..ffc42584 --- /dev/null +++ b/docs/plans/2026-08-14-arch-review-remediation-plan.md @@ -0,0 +1,533 @@ +# Architectural Review Remediation Plan + +**Date:** 2026-08-14 · **Status:** PROPOSED · **Source:** 2026-08-14 performance/architecture review +(eight-pass, all 27 projects + `ZB.MOM.WW.LocalDb`; report artifact +`https://claude.ai/code/artifact/8986cd89-6ab6-48aa-973f-119fddd5cafb`). + +This plan resolves **every** finding from the review — the ten ranked findings plus all +section-level Med/Low items — as an automated, multi-phase program executed by subagents, +parallel where file-ownership makes it safe. It is written so that each work package (WP) is a +self-contained subagent assignment: an executor agent should be able to complete a WP from its +spec here plus the code, without further orchestrator input. + +--- + +## 1. Execution model + +**Orchestrator:** the interactive Claude session (Fable) in this repo. It spawns one subagent per +work package, merges results, runs phase gates, and commits. + +**Isolation:** every code-writing package runs in its **own git worktree** +(`Agent` tool, `isolation: "worktree"`). Packages within a phase touch disjoint file sets (see +the conflict matrix, §7), so merges are clean; the orchestrator resolves any residual conflict +itself. Worktrees also make concurrent `dotnet build`/`test` safe (no shared `obj/`). + +**Per-package protocol.** Each executor agent must: +1. Read this plan's WP section and the referenced source files first; verify the finding still + reproduces in code (line numbers may have drifted — the *description* is authoritative). +2. Implement per the spec. Repo rules apply: edit in place, no backup copies; design doc + code + + tests + docker config travel together. +3. Build the touched projects and run the listed test suites in the worktree + (`dotnet build ZB.MOM.WW.ScadaBridge.slnx`, `dotnet test` filtered to relevant projects). +4. Commit on the worktree branch with the message given in the WP spec. +5. Report back: commit SHA, test summary (counts, any failures), and any spec deviations with + rationale. + +**Phase gates (orchestrator, serial).** After merging a phase's branches into the remediation +branch: full `dotnet build ZB.MOM.WW.ScadaBridge.slnx` + full `dotnet test` must be green; for +phases touching cluster runtime (2 and 3), rebuild the rig (`bash docker/deploy.sh`) and run the +phase's live probes (§6). A red gate stops the program — the orchestrator fixes or reverts the +offending package before proceeding. + +**Branching.** All work lands on `arch-review-remediation` (branched from `main` in Phase 0). +One commit per package (the worktree commit, merged `--no-ff`), plus one docs commit in Phase 4. +Cross-repo work in `~/Desktop/scadaproj` gets its own branch there (WP3.3). Nothing is pushed +without the user. + +### Model matrix + +| Model | Used for | Rationale | +|---|---|---| +| **fable** | Orchestration; design docs for structural seams (WP2.1a, WP3.1a, WP3.2a); final review synthesis | Hardest open design decisions and cross-package judgment | +| **opus** | Concurrency-sensitive implementation (streams, DCL, actors, ingest, UI coalescing, LocalDb library) | Subtle correctness work; failure modes are silent | +| **sonnet** | Mechanical/localized fixes, index migrations, schema tweaks, docs updates | Well-specified, verifiable by tests | +| **opus** (code-reviewer agents) | Phase 4 adversarial diff review | Independent verification pass | + +Approximate agent budget: Phase 1 = 7 executors; Phase 2 = 6 executors + 1 design; Phase 3 = +2 design + 3 executors; Phase 4 = 6 reviewers + 1 docs agent. ~26 agents total. + +--- + +## 2. Phase 0 — Preflight (orchestrator, serial, ~minutes) + +1. `git checkout -b arch-review-remediation` (current `main`; note the pre-existing dirty + `docs/requirements/Component-SiteRuntime.md` — stash or leave untouched, do not absorb it). +2. Baseline: full build + full test run; record pass/fail counts as the reference state. +3. Confirm the rig is deployable (`docker/deploy.sh` last known good) but do **not** redeploy yet. + +Gate: baseline green. If baseline has pre-existing failures, record them — packages are judged +against the baseline, not absolute green. + +--- + +## 3. Phase 1 — Quick wins (7 packages, ALL PARALLEL) + +Highest leverage per line changed. All packages are independent; the two silent-failure defects +(WP1.1, WP1.2) lead. + +### WP1.1 — Stream OK-completion reconnect `[opus]` +**Findings:** #1 (High) — streams silently die at the 4-hour max lifetime; OK completion is +invisible to reconnect logic. +**Files:** `Communication/Grpc/SiteStreamGrpcClient.cs`, `Communication/Actors/SiteAlarmAggregatorActor.cs`, +`Communication/Actors/DebugStreamBridgeActor.cs` (+ tests). +**Changes:** +- Add an `onCompleted` callback to the client subscribe surface (preferred over an error + sentinel — callers can distinguish graceful end from fault). Invoke it when the + `await foreach` ends without exception, including server OK completion at max lifetime and + graceful site shutdown. +- `SiteAlarmAggregatorActor`: on completion, `Tell` self a new `GrpcAlarmStreamCompleted(generation)` + message → set `_streamDown = true`, mark `IsLive = false`, and let the reconcile tick reopen + **without** consuming error-retry/backoff budget (completion is not a fault). Also observe the + `Task.Run` subscription task itself (`ContinueWith` faulted → route into the existing error + path) so no completion or fault is ever unobserved again. +- Same treatment for the debug stream path in `DebugStreamBridgeActor`. +- Generation-fence everything (the fencing pattern already exists in the actor — reuse it). +**Tests:** unit test on the client (fake stream that completes OK → `onCompleted` fires, `onError` +does not); actor test (stream completes OK → `IsLive` false → reopen on next reconcile tick → +backoff counter unchanged). Regression: error path still burns retry budget. +**Commit:** `fix(comms): reconnect on graceful stream completion — kills the 4h silent stream death` + +### WP1.2 — Site audit DB placement + flush `[sonnet]` +**Findings:** #2 (High) — `auditlog.db` on container overlayfs; dead `FlushIntervalMs`; fsync per event. +**Files:** `AuditLog/Site/SqliteAuditWriterOptions.cs`, `AuditLog/Site/SqliteAuditWriter.cs`, +all site-node `appsettings` under `docker/` and `docker-env2/` (+ tests). +**Changes:** +- Make `DatabasePath` **required on Site nodes** (drop the CWD-relative default; `ValidateOnStart`), + mirroring the `LocalDb:Path` precedent — a healthy-looking node silently writing to overlayfs is + the failure mode being closed. Update all rig site-node configs to `/app/data/auditlog.db`. + Update any tests that relied on the implicit default to pass an explicit temp path. +- Implement `FlushIntervalMs` as designed: the writer loop groups commits within the interval + (batch already drains up to 256 — add the time-based commit coalescing). Additionally set + `PRAGMA synchronous=NORMAL` on the audit connection — audit is best-effort by design + (CLAUDE.md: "audit-write failure NEVER aborts the user-facing action"), so NORMAL's + power-loss window is acceptable; say so in a code comment. +- One-time migration note in the commit message: the existing container-local file is abandoned; + already-forwarded rows are safe centrally, pending rows on the old path are lost once — accepted, + this is the bug being fixed. (Cross-reference `docs/known-issues/2026-07-20-cached-telemetry-drain-hot-loop.md`, + which this placement bug caused.) +**Tests:** options validation test (site role boots ⇒ path required); writer test asserting +multi-event single-transaction commit under trickle load. +**Commit:** `fix(auditlog): site audit DB onto the data volume; required path + soft flush` + +### WP1.3 — Conditional CDC registration `[opus]` +**Findings:** #5 (High), partial — non-replicating nodes pay full CDC capture forever. (The +library half — trigger cleanup API, O(1) backlog — is WP3.3.) +**Files:** `Host/SiteLocalDbSetup.cs` (+ tests). Read `~/Desktop/scadaproj/ZB.MOM.WW.LocalDb/` +and the site-a rig configs first to confirm the exact replication-config shape on both the active +and passive peer. +**Changes:** +- Skip all ten `RegisterReplicated` calls when the node has **no replication configured** + (neither `LocalDb:Replication:PeerAddress` nor `ApiKey` — verify the passive-side shape on the + rig before choosing the predicate; the guard must be true on BOTH members of a replicated pair + and false on site-b/site-c). +- Preserve the load-bearing ordering documented in the file: when registration is skipped, the + legacy migrator still runs (its rows simply don't replicate — consistent with an unreplicated node). +- **Known residual, documented in-file:** a database file that was registered under an older build + keeps its stale triggers until WP3.3 ships the library cleanup API; on the docker rig this is + moot (volumes are recreated on redeploy of a schema change) — state this in a comment and in the + commit message. +**Tests:** setup test with and without replication config, asserting trigger presence/absence via +`sqlite_master` (`name LIKE '%localdb%'` — confirm the library's trigger naming first). +**Commit:** `perf(host): install CDC capture only when replication is configured` + +### WP1.4 — SQL quick indexes + sliced purge `[sonnet]` +**Findings:** S&F sweep O(N) scan (High); audit KPI window key-lookup; Notification KPI +delivered-interval scan; unbounded terminal purge (all part of #7/#8 support). +**Files:** `StoreAndForward/StoreAndForwardSchema.cs` (+ its versioned-upgrade path), +`ConfigurationDatabase/Migrations/` (+ manual SQL script for production per repo convention), +`ConfigurationDatabase/Repositories/NotificationOutboxRepository.cs`. +**Changes:** +- `sf_messages`: add `(status, created_at)` index so the ordered due-scan terminates at `LIMIT`; + follow the schema's existing versioned-upgrade mechanism so existing site files get it. +- `IX_AuditLog_OccurredAtUtc` → add `INCLUDE (Status)` (EF migration; remember the repo gotcha — + **build first**, never `--no-build`, delete empty migrations if scaffolded). +- Notifications: filtered index `(DeliveredAt) WHERE Status = 'Delivered'` (or the enum's stored + value — check the column type) covering the delivered-last-interval KPI count. +- `DeleteTerminalOlderThanAsync`: slice into batched deletes — copy the slicing pattern used twice + elsewhere in the same layer (per-channel audit retention delete is one). +**Tests:** migration applies cleanly (dev auto-apply path); a query-shape test is not required — +Phase 4 live probes verify plans via `EXPLAIN`/showplan. +**Commit:** `perf(sql): sweep/KPI covering indexes + sliced notification terminal purge` + +### WP1.5 — Mechanical hot-path fixes `[sonnet]` +**Findings:** InstanceActor nits (Med); per-call `JsonSerializerOptions` ×4 (Med). +**Files:** `SiteRuntime/Actors/InstanceActor.cs`, `ExternalSystemGateway/ExternalSystemClient.cs`, +`SiteRuntime/Repositories/SiteExternalSystemRepository.cs` (options instance only — the query +redesign is WP2.6), `ManagementService/ManagementEndpoints.cs`, `CentralUI/.../ScriptAnalysisService.cs`. +**Changes:** +- `HandleSetStaticAttribute`: use the existing `_resolvedAttributeByName` `TryGetValue` instead of + the LINQ scan (InstanceActor.cs:373). +- `HandleTagValueUpdate`: precompute the parsed `DataType` into the resolved-attribute index entry + at config-apply time; stop re-`Enum.TryParse`-ing per update. Intern/cache the quality strings + (`Quality.ToString()` per update → static readonly map). +- Static-override writes: coalesce per instance on a short timer, reusing the batched-flush pattern + already present for native alarms in the same codebase (copy that shape, don't invent). +- One `static readonly JsonSerializerOptions` per assembly at the four verified sites. +**Tests:** existing InstanceActor suite must stay green; add a test pinning single-flush +coalescing of N rapid static writes. +**Commit:** `perf(runtime): O(1) attribute resolution, precomputed types, coalesced static writes, shared JSON options` + +### WP1.6 — Transport fail-fast `[sonnet]` +**Findings:** known-failure sends burn full Ask timeouts (Med, first half — the timeout stagger +and double-hop are WP2.2). +**Files:** `Communication/Grpc/GrpcSiteTransport.cs` (+ tests). +**Changes:** a send to a site with no configured channel replies `Status.Failure` (with a +descriptive exception) to the asker immediately instead of warn-and-drop. Audit every +warn-and-drop in the file for the same pattern. +**Tests:** Ask against an unconfigured site faults in <1s, not at the 30s timeout. +**Commit:** `fix(comms): fail known-dead sends immediately instead of burning Ask timeouts` + +### WP1.7 — Event-log writer batching + sliced retention `[sonnet]` +**Findings:** per-event transactions in `SiteEventLogger` (part of the site_events High — the +volume *policy* is WP3.2); unbatched `site_events` retention DELETE (Med). +**Files:** `SiteEventLogging/SiteEventLogger.cs`, `SiteEventLogging/EventLogPurgeService.cs` (+ tests). +**Changes:** +- Writer loop: drain the existing batching channel into **one transaction per drain** (up to 256), + exactly the `SqliteAuditWriter` shape — cite it in a comment. +- Retention purge: batch the DELETE at 1000 rows/iteration like the cap purge next to it. +**Tests:** writer test (N queued events → 1 transaction); purge test over >1000 expired rows. +**Commit:** `perf(sitelog): batched event-log commits and sliced retention purge` + +**Phase 1 gate:** merge all seven, full build + test, then rebuild the rig (`docker/deploy.sh` — +WP1.2 changed site configs) and run the Phase 1 live probes (§6). + +--- + +## 4. Phase 2 — Seam rework (6 packages, PARALLEL after WP2.1a) + +### WP2.1 — DCL batch seam `[design: fable → implement: opus]` *(largest package)* +**Findings:** #3 (High, four collapsed findings) + MxGateway 2-RPC subscribe / no-backoff retry +(Med) + quality-counter and alarm-fanout costs (Med). +**Stage a — design memo** (fable, ~1 page, committed as +`docs/plans/2026-08-XX-dcl-batch-seam-design.md`): the batch API shape on `IDataConnection` +(`SubscribeBatchAsync(IReadOnlyList)` returning per-tag results; true bulk +`ReadBatchAsync`/`WriteBatchAsync`), per-subscription monitored-item budget + sharding scheme, +reconnect chunking/stagger parameters (configurable; defaults sized for 37,500 tags), re-seed +deadline semantics, and what the MxGateway adapter can do **without** a cross-repo mxaccessgw +change (fallback: client-side chunked pipelining; note the cross-repo bulk-RPC follow-up if the +proto lacks one — check `~/Desktop/MxAccessGateway` first). +**Stage b — implementation** (opus): +- `RealOpcUaClient`: N×`AddItem` + one `ApplyChangesAsync` per batch; serialize all + `ApplyChangesAsync` on the shared Subscription behind a `SemaphoreSlim` (batching makes + contention rare; the lock makes it safe); shard monitored items across subscriptions above the + per-subscription budget (default ~5,000, configurable). +- `OpcUaDataConnection`: implement the batch interface; `ReadBatchAsync`/`WriteBatchAsync` become + single OPC UA service calls (the MxGateway adapter's `ReadBulk`/`WriteBulk` is the in-repo + reference). +- `DataConnectionActor`: `HandleSubscribe`/`ReSubscribeAll` pass whole tag lists; reconnect + becomes bounded chunked batches honoring the existing failover stagger; `SeedTagsAsync` becomes + chunked batch reads with bounded parallelism and an **overall** deadline (no more 30s-per-tag + serial worst case). Tag-resolution retry gains exponential backoff with a ceiling and batched + probes. Quality counters flush on transitions or a short timer instead of per-change; alarm + subscriber matching gets a source-prefix index and pushes the union filter to the gateway feed. +**Tests:** the existing DCL suite (Become/Stash lifecycle, generation fencing) must stay green; +new tests with a fake client: batch subscribe issues one apply per chunk; concurrent applies +serialize; re-seed respects the deadline; backoff caps. +**Commit:** `perf(dcl): batch subscribe/read/write seam, bounded reconnect, sharded subscriptions` + +### WP2.2 — Central ingest & SQL set-based rework `[opus]` +**Findings:** #6 (High, partition purge), #7 (High, row-at-a-time ingest), #8 (High, KPI scans — +query-shape half), KPI rollup N+1 / read-path hygiene / no pooling (Med). +**Files:** `AuditLog/.../AuditLogIngestActor.cs`, `ConfigurationDatabase/Repositories/AuditLogRepository.cs`, +`SiteCallAudit/.../SiteCallAuditActor.cs` + `SiteCallAuditRepository.cs`, +`ConfigurationDatabase/Repositories/NotificationOutboxRepository.cs`, `KpiHistoryRepository.cs`, +`ConfigurationDatabase/ServiceCollectionExtensions.cs`, gRPC ingest entry in Communication (+ migration + manual SQL script). +**Changes:** +- **Ingest batching:** one TVP (or multi-row `INSERT … SELECT … WHERE NOT EXISTS` batch) per + telemetry packet with per-row fallback on constraint violation; cached-telemetry entry = + one transaction, not ~5 round trips. +- **SiteCalls upsert:** single statement (`UPDATE … ; IF @@ROWCOUNT = 0 INSERT …` guarded for the + newer-status predicate, or `MERGE` with the usual HOLDLOCK hygiene). +- **Reconciliation off-mailbox:** the drain moves out of `ReceiveAsync` to the + `PipeTo` + in-flight-guard shape — `NotificationOutboxActor` next door is the in-repo reference; + ingest/queries/KPI asks must not queue behind post-outage catch-up. +- **Timeout stagger + hop removal:** inner budgets strictly less than outer (site-ask 25s → + ingest 20s → SQL 15s, or similar); the gRPC handler Asks the ingest singleton proxy directly, + removing the redundant intermediate actor hop. +- **Partition purge:** prefer making EventId uniqueness **partition-aligned** + (`UNIQUE (EventId, OccurredAtUtc)`) so `SWITCH` needs no index drop at all; EventId is globally + unique by construction (GUID minted at source), so the idempotency probe still seeks correctly — + document this reasoning in the migration. If alignment proves incompatible with the probe shape, + fall back to `ONLINE = ON` rebuild outside the switch transaction (Standard-edition caveat: check + target SQL edition; if ONLINE is unavailable, alignment is the only acceptable option). +- **KPI query shapes:** split the live-queue seeks from the delivered-interval count (filtered + index from WP1.4); rollup fold preloads existing rows per window into a dictionary (kill the + per-(series,hour) existence SELECT). +- **Hygiene:** `AsNoTracking` + keyset paging on the Notifications read path (mirror the two + sibling repos); targeted `ExecuteUpdate` for delivery-attempt status writes instead of detached + full-row `Update()`; `AddDbContextPool` — **first verify** the context registers no scoped + state/injected services that break pooling; if it does, fix or document why pooling is skipped. +**Tests:** ingest idempotency under batch (duplicate EventIds in one packet + across packets); +upsert newer-status semantics; reconciliation running while an ingest Ask completes promptly. +**Commit:** `perf(central): set-based ingest, aligned partition purge, KPI query shapes, EF hygiene` + +### WP2.3 — Wire efficiency `[opus]` +**Findings:** #10 (Med, alarm seed waste + per-attempt re-fan-out), unbounded pre-snapshot +buffers / lossy shared channel (Med), `PullAuditEvents` at-most-once (Low). +**Files:** `Communication/Actors/SiteAlarmAggregatorActor.cs`, `SiteAlarmLiveCacheService.cs`, +`SiteCommandDtoMapper.cs`, `DebugStreamBridgeActor.cs`, `Communication/Grpc/SiteStreamGrpcServer.cs`, +site audit queue (`ISiteAuditQueue` impl) (+ tests). +**Changes:** +- **Alarms-only snapshot:** the seed travels the `SiteCommandService` query surface + (`QueryReply.DebugViewSnapshot`), so add an additive alarms-only flag on the query DTO — verify + whether the vendored proto is involved at all; if it is, follow the documented manual + toggle-build-copy-untoggle regeneration, additive field numbers only. +- Seed **once per successful (re)connect** plus the existing reconcile as the diff/backstop — + not a full unconditional snapshot every 60s. Jitter the per-site aggregator timers so 10 sites + don't fan out in lockstep; give the seed-retry leg backoff. +- Cap the pre-snapshot buffers (drop-oldest + a dropped-count surfaced on the health report); + hard timer on snapshot arrival that fails the session rather than buffering forever; stream + events must not reset the receive-timeout that ends a stuck session. +- `SubscribeSite` gets its own channel (larger, or wait-mode) instead of sharing the debug view's + 1000-slot DropOldest — an alarm burst during a WAN stall must not silently drop transitions. + Batch multiple events per proto message where the contract allows (additive). +- `PullAuditEvents`: mirror `PullSiteCalls`' composite `(timestamp, id)` keyset cursor; flip rows + to `Reconciled` only when the **next** pull's cursor proves receipt (at-least-once restored). +**Tests:** cursor-proof flip semantics (fault between response and next pull → rows re-served); +seed-once behavior across a simulated reconnect; buffer cap + counter. +**Commit:** `perf(comms): alarms-only seed, capped buffers, at-least-once audit pull` + +### WP2.4 — Central UI performance `[opus]` +**Findings:** #8 (High, per-circuit KPI polling half), AlarmSummary/Health/DebugView per-circuit +costs (High), push-triggered full reloads / serial Asks / no virtualization (Med), sandbox +thread-parking (Low). +**Files:** `CentralUI` — `Health.razor`, `AlarmSummary.razor`, `DebugView.razor`, +`Deployments.razor`, a new process-level KPI snapshot service, sandbox host (+ tests where the +project has them). +**Changes:** +- **Process-level memoized KPI snapshot service** (singleton, 5–10s TTL, single-flight so N + circuits share one SQL round per interval) consumed by Health + Notification Outbox + Site + Calls pages. This plus WP2.2's query shapes closes finding #8 from both ends. +- AlarmSummary derives not-reporting from the alarm aggregator's own reconcile state instead of + re-running the per-instance snapshot fan-out per circuit per 15s. +- DebugView: coalesce renders at ~250ms; rebuild the trees behind a version stamp instead of per + event; marshal off the gRPC thread once per coalesce window. +- Deployments: apply the pushed delta (or debounce reloads ~500ms) instead of reloading all + records + instances per push; server-side paging. +- Health tick: `Task.WhenAll` with short per-tile timeouts so one hung singleton degrades one + tile, not the tick. `Virtualize` on the flat alarm table. +- Sandbox Test Runs: bound with a semaphore instead of parking thread-pool threads on sync accessors. +- Preserve the existing visual design exactly — these are behavioral changes only, no restyling, + no new component frameworks (repo rule). +**Tests:** KPI cache single-flight test; render-coalescing unit test if the code shape allows. +**Commit:** `perf(ui): shared KPI cache, live-cache-backed alarm summary, coalesced debug renders` + +### WP2.5 — Deployment & authoring pipeline `[opus]` +**Findings:** #9 (High, both halves), verdict-cache Clear() leak + unpaged management queries (Med). +**Files:** `TemplateEngine/.../FlatteningPipeline.cs`, `TemplateEngineRepository.cs`, +`TemplateService.cs`, `DeploymentManager/.../DeploymentService.cs`, +`ManagementService/ManagementActor.cs`, `ScriptAnalysis/.../ScriptCompileVerdictCache.cs`, +CLI (+ tests). +**Changes:** +- **Flatten-session cache:** within one deploy/validation session, memoize template chain and + composition loads keyed by template id + version; hoist the three global queries (shared + scripts, schemas, connections) once per session — `ArtifactDeploymentService` is the in-repo + reference shape. +- **`DeploySiteAsync`:** bulk orchestration over a site's instances with bounded parallel fan-out + and per-instance timeouts, reusing the per-instance operation lock; surfaced via ManagementActor + command + CLI (`instance deploy-site ` or consistent naming — follow the CLI's existing + verb conventions); UI wiring optional, defer if it inflates scope. +- **Staleness watermark:** per-template monotonic version bumped on any template-graph mutation; + staleness detection compares watermarks before paying a full flatten. +- **Authoring:** slim projections for acyclicity/collision checks (no script bodies); + `AsNoTracking` on read-only walks; `ReconcileDescendantsAsync` receives the already-loaded + graph instead of re-calling `GetAllTemplatesAsync`. +- **Management queries:** `QueryDeployments` pages DB-side with a summary projection; `ListTemplates` + pages DB-side without full script bodies; batch override apply becomes one read + one commit. +- **Verdict cache:** LRU/segmented eviction instead of wholesale `Clear()` (the Clear re-opens the + non-collectible `InteractiveAssemblyLoader` leak the cache exists to bound); add a retention + policy for terminal deployment records (config, default generous). +**Tests:** flatten count test (N instances, same template ⇒ chain loaded once); DeploySiteAsync +fan-out with an injected slow site (bounded, others complete); verdict-cache eviction keeps hot +entries. +**Commit:** `perf(deploy): flatten-session caching, bulk DeploySiteAsync, paged management queries` + +### WP2.6 — Cross-cutting misc `[sonnet]` +**Findings:** site external-system resolution (Med), Inbound API per-request SQL (Low), unbounded +S&F observer queue (Low), CLI timeout (Low), failback-probe heartbeat pollution (Low), site stream +alarm-vs-attribute shared buffer (Med). +**Files:** `SiteRuntime/Repositories/SiteExternalSystemRepository.cs`, InboundAPI method-lookup +path, `StoreAndForward` observer queue, `SiteRuntime/Streaming/SiteStreamManager.cs`, CLI +`HttpClient` setup, failback probe (+ tests). +**Changes:** +- External-system resolution: name-keyed lookup (the ID reverse-map scan goes); cache the parsed + method list per system, invalidated on redeploy — honoring the documented "indexed query" contract. +- Inbound API: short-TTL ApiMethod cache invalidated by the existing `ScriptArtifactChangeSubscriber` + bus (the invalidation plumbing already exists — lean on it). +- S&F observer queue: bound it DropOldest like its siblings (it is the one unbounded channel in + the system) + a dropped counter. +- `SiteStreamManager`: separate alarm publish source (or priority path) so attribute storms can't + evict alarm transitions; drop counter on the site health report; skip publish at zero subscribers. +- CLI `HttpClient`: explicit timeout (30s, config-overridable). +- Failback probe: mark the probe heartbeat synthetic (flag or reserved SiteId) so it stops + polluting liveness data — check what consumes heartbeats before choosing the mechanism. +**Commit:** `perf(misc): cached hot-path lookups, bounded observer queue, alarm-priority stream path` + +**Phase 2 gate:** merge order WP2.2 → WP2.3 → WP2.1 → WP2.4 → WP2.5 → WP2.6 (Communication-heavy +first — likeliest residual conflicts surface early). Full build + test; rig rebuild; Phase 2 live +probes (§6). + +--- + +## 5. Phase 3 — Structural (design-first; 3 tracks) + +Design stages run in parallel (fable); implementations start as each design lands. Defaults are +pre-decided below so automation never blocks — the design memos may refine but not stall. + +### WP3.1 — Script execution pool split `[design: fable → implement: opus]` +**Findings:** #4 (High) + actor-per-execution overhead / compile-cache Clear (Med). +**Stage a — design doc** (`docs/plans/2026-08-XX-script-pool-split-design.md`), deciding: +- Trigger-expression evaluation leaves the blocking pool — default decision: evals are + non-blocking by construction, so run them as plain async on the default dispatcher with a + concurrency gate, no second dedicated pool. +- Blocking-script pool sizing scales with deployed instance count (default: `max(8, instances/8)` + capped at a config ceiling); stuck-script watchdog **replaces** lost threads. +- Timeout budget includes queue wait (deadline captured at enqueue). +- Per-script in-flight cap — default policy: bounded queue per script (cap 4), overflow **sheds + newest with a site event + counter** (an alarm-triggered run that can't start within its + deadline is stale anyway); design memo must justify or amend. +- `ScriptExecutionActor` elimination: run the guarded task directly under `ScriptActor`, + preserving supervision semantics (Stop-on-failure equivalence), telemetry, and the audit + `ExecutionId`/`ParentExecutionId` threading — this is the riskiest part; the design memo maps + every behavior the actor currently provides to its replacement. +- `SiteScriptCompileCache`: LRU eviction; cache-miss compiles move off-dispatcher (the + DeploymentManager shared-script path is the in-repo reference). +**Stage b — implement + tests:** starvation regression test (8 blocked scripts ⇒ alarm eval still +completes <2s); queue-time-inclusive timeout test; watchdog replacement test; supervision-parity +tests for the actor removal. +**Commit:** `perf(runtime): split trigger evals from the blocking pool; bounded, deadline-aware execution` + +### WP3.2 — site_events volume policy `[design: fable → implement: sonnet]` +**Findings:** #High site_events (policy half; mechanics done in WP1.7). +**Stage a — design memo** deciding: per-run Started/Completed Info events become **sampled/opt-in** +(default: off per-run, aggregate counters per interval instead; per-script opt-in flag for +debugging); whether `site_events` should replicate at all (default recommendation: keep replicated +— central event-log queries hit either node — but the memo must check what actually reads +`site_events` on the peer before confirming; if nothing does, deregister it and use clock-based +local purges that never enter the oplog, coordinating with WP3.3's dereg API). +**Stage b:** implement per memo; update `Component-SiteEventLogging` doc (Phase 4 collects). +**Commit:** `perf(sitelog): sampled per-run events; site_events replication policy` + +### WP3.3 — LocalDb library (CROSS-REPO: `~/Desktop/scadaproj`) `[opus]` +**Findings:** #5 (library half), replication row-count batching (Med), library-level costs (Low). +**Files:** `~/Desktop/scadaproj/ZB.MOM.WW.LocalDb/` on its own branch; then the consuming update here. +**Changes (library):** +- `DeregisterReplicated(table)` / cleanup API: idempotently drops capture triggers (and optionally + prunes that table's oplog rows) — consumed by `SiteLocalDbSetup` when replication is + unconfigured, closing WP1.3's stale-trigger residual. +- Backlog depth becomes O(1) (`MAX(seq) − last_acked_seq` or a maintained counter) — it currently + runs an O(backlog) `COUNT(*)` per minute and per metrics scrape. +- Replication batching by **summed bytes** (budget under the 4 MB gRPC cap, row-count as a + secondary cap) + ack coalescing; the rig's `MaxBatchSize=16` pin can then be retired. +- `LwwApplier` prepare-once-rebind per table; drop the unused `__localdb_oplog_hlc` index (handle + existing files in the schema-version upgrade). +**Changes (this repo, after the library lands):** bump the package/project reference; call the +cleanup API from `SiteLocalDbSetup` when unregistered; relax the rig `MaxBatchSize` pin to the +byte budget; update the CLAUDE.md LocalDb bullet and the **umbrella `../scadaproj/CLAUDE.md` +ScadaBridge entry in the same change** (repo rule). +**Tests:** library suite for dereg idempotency, O(1) depth, byte-budget batching; ScadaBridge +integration: site-b boots with zero triggers and zero oplog growth. +**Commits:** library commit in scadaproj + `chore(deps): LocalDb — dereg API, byte-budget replication` here. + +**Phase 3 gate:** full build + test in BOTH repos; rig rebuild; live probes. + +--- + +## 6. Phase 4 — Verification, review, docs (mixed parallel) + +### Live probes (orchestrator + rig, accumulate per phase gate) +- **Stream lifetime:** set max stream lifetime to ~2 min on the rig; verify the alarm stream + reconnects within one reconcile tick of OK completion and `IsLive` reflects the gap (WP1.1). +- **CDC:** on site-b/site-c nodes, `sqlite_master` shows no capture triggers; oplog table empty + and not growing (WP1.3/WP3.3). Site-a pair still converges (replication regression). +- **Audit path:** site `auditlog.db` present under `/app/data`; survives `docker compose + rm`+recreate of one site container (WP1.2). +- **Query plans:** `EXPLAIN QUERY PLAN` on the S&F due-sweep (index-terminated); SQL Server + showplan on the notification KPI and audit KPI queries (seeks, no full scans) (WP1.4/WP2.2). +- **Deploy timing:** bulk-redeploy a site on the rig before/after; record wall clock (WP2.5). +- **Failover drill:** `docker/failover-drill.sh` still passes (~25s takeover) — protects against + regressions from the actor/timeout changes. +- **Load validation:** the deferred target-scale load test (deferred-work register #25) is the + final proof the ceilings moved — schedule as its own follow-on; this plan's exit criterion is + the probes above, not #25. + +### Adversarial review `[6 parallel code-reviewer agents, opus; fable synthesis]` +One reviewer per area over the full remediation diff (site runtime, DCL, comms, site persistence, +central SQL, UI/deploy), each instructed to try to **refute** the fixes (regressions, changed +semantics, missed call sites, test gaps). Confirmed findings are fixed by targeted follow-up +agents before the docs commit. + +### Documentation propagation `[sonnet]` +- Component docs touched: `Component-DataConnectionLayer`, `Component-SiteRuntime`, + `Component-CentralSiteCommunication`, `Component-AuditLog`, `Component-SiteEventLogging`, + `Component-StoreAndForward`, `Component-NotificationOutbox`, `Component-SiteCallAudit`, + `Component-TemplateEngine`, `Component-DeploymentManager`, `Component-CentralUI`, + `Component-ClusterInfrastructure` (only where behavior/config changed — no cosmetic edits). +- CLAUDE.md Key Design Decisions: stream-completion reconnect, conditional CDC, required site + audit DB path, DCL batch seam, `DeploySiteAsync`, script-pool split, LocalDb byte-budget + replication (retire the MaxBatchSize=16 note). +- Umbrella `../scadaproj/CLAUDE.md`: LocalDb entry (done with WP3.3). +- Close/annotate any related `docs/known-issues/` entries; README component table needs no change + (no components added/removed). +- Final `git diff` review by the orchestrator, then the docs commit. **Push only on user request.** + +--- + +## 7. Parallelism & conflict matrix + +| Phase | Package | Projects touched | Conflicts within phase | +|---|---|---|---| +| 1 | WP1.1 | Communication (client+actors) | WP1.6 same project, different files — merge-safe | +| 1 | WP1.2 | AuditLog(site), docker configs | none | +| 1 | WP1.3 | Host | none | +| 1 | WP1.4 | ConfigurationDatabase, StoreAndForward | none | +| 1 | WP1.5 | SiteRuntime, ESG, ManagementService, CentralUI | single-line edits, none | +| 1 | WP1.6 | Communication (transport) | see WP1.1 | +| 1 | WP1.7 | SiteEventLogging | none | +| 2 | WP2.1 | DataConnectionLayer | none | +| 2 | WP2.2 | AuditLog(central), SiteCallAudit, ConfigurationDatabase, NotificationOutbox | none (WP2.3 owns Communication) | +| 2 | WP2.3 | Communication, site audit queue | none | +| 2 | WP2.4 | CentralUI | none | +| 2 | WP2.5 | TemplateEngine, DeploymentManager, ManagementService, ScriptAnalysis, CLI | none | +| 2 | WP2.6 | SiteRuntime(streaming+repos), InboundAPI, S&F, CLI(http) | CLI touched by WP2.5 (commands) and WP2.6 (HttpClient) — different files; merge WP2.5 first | +| 3 | WP3.1 | SiteRuntime(scripts) | none | +| 3 | WP3.2 | SiteEventLogging, SiteRuntime(one call site) | coordinate with WP3.1 merge order (3.1 first) | +| 3 | WP3.3 | scadaproj (separate repo) + Host/config here | in-repo half lands after 3.1/3.2 | + +## 8. Risks & mitigations + +- **Vendored proto drift** (WP2.3): additive fields only, never reuse numbers; follow the + documented manual regeneration toggle. Mitigation: the executor verifies whether the change is + DTO-only before touching proto at all. +- **Partition-aligned uniqueness** (WP2.2) changes the enforcement scope of EventId uniqueness + from global to per-partition. GUID collision across partitions is not a real risk, but the + ingest probe's semantics must be re-tested for duplicate delivery across a month boundary. +- **Pooled DbContext** (WP2.2): pooling breaks contexts with per-scope injected state — verify + before switching; skipping with a documented reason is an acceptable outcome. +- **ScriptExecutionActor removal** (WP3.1) is the highest-semantic-risk change: supervision, + telemetry, and audit-correlation parity are explicit design-memo obligations, and it ships in + its own commit so it can be reverted independently. +- **Rolling-upgrade caveat** (WP3.3): site pairs already require stop/start together (repo rule); + the LocalDb schema-version bump rides that existing constraint — call it out in the topology guide. +- **EF migration gotcha:** always build before `dotnet ef migrations add`; delete empty scaffolds. +- **Merge conflicts:** worktree isolation + the matrix above; residual conflicts are resolved by + the orchestrator, never by re-running an executor blindly. + +## 9. Exit criteria + +1. Full solution build + tests green in both repos; baseline test count not reduced. +2. All §6 live probes pass on the rig; failover drill unchanged (~25s). +3. Adversarial review pass complete with all CONFIRMED findings resolved. +4. Docs propagated (component docs, both CLAUDE.mds, known-issues) with no stale cross-references. +5. One reviewed commit per package on `arch-review-remediation`; branch ready for the user to + merge/push; target-scale load test (#25) scheduled as the follow-on validation. From c5e66ed4e4788152f5c95be2846f61a7df23a5ca Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Fri, 14 Aug 2026 19:49:00 -0400 Subject: [PATCH 02/31] fix(comms): fail known-dead sends immediately instead of burning Ask timeouts --- .../Actors/ISiteCommandTransport.cs | 6 +- .../Grpc/GrpcSiteTransport.cs | 23 +++-- .../Grpc/SitePairChannelProvider.cs | 8 +- .../GrpcSiteTransportFailFastTests.cs | 95 +++++++++++++++++++ 4 files changed, 120 insertions(+), 12 deletions(-) create mode 100644 tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/GrpcSiteTransportFailFastTests.cs diff --git a/src/ZB.MOM.WW.ScadaBridge.Communication/Actors/ISiteCommandTransport.cs b/src/ZB.MOM.WW.ScadaBridge.Communication/Actors/ISiteCommandTransport.cs index 89545aa6..1b3efc0f 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Communication/Actors/ISiteCommandTransport.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Communication/Actors/ISiteCommandTransport.cs @@ -24,8 +24,10 @@ public interface ISiteCommandTransport /// Routes 's message to its site. Any reply the site produces is /// delivered to — for an Ask that is the temporary ask actor /// (completing the caller's task); for a Tell-with-sender (the debug bridge) that is the - /// originating actor. A message with no route (an unknown site) is warned and dropped so the - /// caller's Ask times out — central never buffers for an unreachable site. + /// originating actor. A message with no route (an unknown/unconfigured site) fails fast: a + /// is delivered to immediately instead of + /// letting the caller's Ask burn its full timeout — central still never buffers for an + /// unreachable site. /// /// The site-addressed command envelope. /// Where a reply (or a ) is delivered. diff --git a/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/GrpcSiteTransport.cs b/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/GrpcSiteTransport.cs index 33322fe3..7d3bffad 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/GrpcSiteTransport.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/GrpcSiteTransport.cs @@ -22,9 +22,10 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc; /// /// /// Per-call deadlines match today's Ask timeouts exactly — see . -/// Behaviour is otherwise unchanged from the Akka path: an unknown/unconfigured site is warned and -/// dropped (the caller's Ask times out), and a transport fault surfaces to the caller as a -/// , which the S&F/audit layers already treat as transient. +/// An unknown/unconfigured site now fails fast: it surfaces to the caller as a +/// immediately (WP1.6) instead of being warned-and-dropped to let the +/// caller's Ask burn its full timeout. A transport fault takes the same +/// path, which the S&F/audit layers already treat as transient. /// /// /// Cross-node retry is the channel provider's job and happens only on @@ -79,13 +80,21 @@ public sealed class GrpcSiteTransport : ISiteCommandTransport replyTo.Tell(reply, ActorRefs.NoSender); } } - catch (SiteChannelUnavailableException) + catch (SiteChannelUnavailableException ex) { - // Parity with the removed Akka "no ClusterClient for site" path: warn and drop, so the caller's - // Ask times out. Central never buffers. + // WP1.6: a known-dead send (no configured channel for this site) used to warn-and-drop, + // leaving the caller's Ask to burn its full timeout for a failure we already know about. + // Fail fast instead — same Status.Failure completion path a transport fault takes below — + // so S&F/audit treat it as transient immediately rather than tens of seconds later. + // Central still never buffers for an unreachable site; only the completion timing changed. _logger.LogWarning( - "No gRPC channel for site {SiteId}; dropping {Message} (caller's Ask will time out)", + "No gRPC channel for site {SiteId}; failing {Message} immediately instead of dropping it", envelope.SiteId, envelope.Message.GetType().Name); + + if (!fireAndForget && !replyTo.IsNobody()) + { + replyTo.Tell(new Status.Failure(ex), ActorRefs.NoSender); + } } catch (Exception ex) { diff --git a/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SitePairChannelProvider.cs b/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SitePairChannelProvider.cs index f2cf359e..b8128f9f 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SitePairChannelProvider.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SitePairChannelProvider.cs @@ -8,9 +8,11 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc; /// /// Raised when a site has no usable gRPC channel — an unknown site, or a site with neither -/// GrpcNodeAAddress nor GrpcNodeBAddress configured. The gRPC transport treats this -/// the way the removed Akka path treated "no ClusterClient for site": warn and drop, so the caller's Ask -/// times out (central never buffers). +/// GrpcNodeAAddress nor GrpcNodeBAddress configured. +/// treats this as a known-dead send: it fails the caller's Ask fast with an +/// Akka.Actor.Status.Failure wrapping this exception (WP1.6) instead of warning and dropping +/// the message, so the caller does not burn its full Ask timeout on a failure already known at send +/// time (central still never buffers). /// public sealed class SiteChannelUnavailableException(string siteId) : Exception($"No gRPC channel is configured for site '{siteId}'.") diff --git a/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/GrpcSiteTransportFailFastTests.cs b/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/GrpcSiteTransportFailFastTests.cs new file mode 100644 index 00000000..fb934e0c --- /dev/null +++ b/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/GrpcSiteTransportFailFastTests.cs @@ -0,0 +1,95 @@ +using System.Diagnostics; +using Akka.Actor; +using Akka.TestKit.Xunit2; +using Microsoft.Extensions.Logging.Abstractions; +using Microsoft.Extensions.Options; +using ZB.MOM.WW.ScadaBridge.Commons.Messages.Lifecycle; +using ZB.MOM.WW.ScadaBridge.Communication.Grpc; + +namespace ZB.MOM.WW.ScadaBridge.Communication.Tests; + +/// +/// WP1.6: a send to a site with no configured gRPC channel must fail the caller's Ask immediately +/// ( wrapping ) instead of +/// warning-and-dropping the message and leaving the Ask to expire at its full timeout. +/// +public class GrpcSiteTransportFailFastTests : TestKit +{ + private static readonly CommunicationOptions Opts = new() + { + DeploymentTimeout = TimeSpan.FromSeconds(120), + LifecycleTimeout = TimeSpan.FromSeconds(30), + ArtifactDeploymentTimeout = TimeSpan.FromSeconds(60), + QueryTimeout = TimeSpan.FromSeconds(30), + IntegrationTimeout = TimeSpan.FromSeconds(30), + DebugViewTimeout = TimeSpan.FromSeconds(10) + }; + + private static GrpcSiteTransport BuildTransport() => new( + new SitePairChannelProvider( + new NoKeyProvider(), Options.Create(Opts), NullLogger.Instance), + Opts, + NullLogger.Instance); + + [Fact] + public void Send_ToUnconfiguredSite_FaultsTheAskFastInsteadOfTimingOut() + { + // No ReconcileSites/UpdateSite call was ever made for this site — exactly the "unknown + // site" shape SitePairChannelProvider throws SiteChannelUnavailableException for. + var transport = BuildTransport(); + var probe = CreateTestProbe(); + var envelope = new SiteEnvelope( + "unconfigured-site", + new EnableInstanceCommand("cmd-1", "Site1.Pump1", DateTimeOffset.UtcNow)); + + var stopwatch = Stopwatch.StartNew(); + transport.Send(envelope, probe.Ref); + + // The old warn-and-drop behavior would leave the caller silent until its own Ask timeout + // (30s in production). Well under 1s here proves the fail-fast path, not a lucky race. + var failure = probe.ExpectMsg(TimeSpan.FromSeconds(1)); + stopwatch.Stop(); + + Assert.True(stopwatch.Elapsed < TimeSpan.FromSeconds(1), + $"expected the Ask to fault well under 1s; took {stopwatch.Elapsed}"); + var cause = Assert.IsType(failure.Cause); + Assert.Equal("unconfigured-site", cause.SiteId); + Assert.Contains("unconfigured-site", cause.Message); + } + + [Fact] + public async Task Ask_AgainstUnconfiguredSite_CompletesFaultedFastViaTellStatusFailure() + { + // Mirrors how CommunicationService actually calls this seam: Ask a temp actor, transport + // Tells it back. Proves the fail-fast reply satisfies a real Akka Ask, not just a TestProbe. + var transport = BuildTransport(); + var askActor = Sys.ActorOf(Props.Create(() => new EchoingAskTarget(transport))); + + var stopwatch = Stopwatch.StartNew(); + var ex = await Assert.ThrowsAsync(async () => + await askActor.Ask( + new SiteEnvelope("unconfigured-site", new EnableInstanceCommand( + "cmd-1", "Site1.Pump1", DateTimeOffset.UtcNow)), + TimeSpan.FromSeconds(30))); + stopwatch.Stop(); + + Assert.Equal("unconfigured-site", ex.SiteId); + Assert.True(stopwatch.Elapsed < TimeSpan.FromSeconds(1), + $"expected the Ask to fault well under the 30s timeout; took {stopwatch.Elapsed}"); + } + + /// Relays an Ask into the transport, exactly as CentralCommunicationActor does. + private sealed class EchoingAskTarget : ReceiveActor + { + public EchoingAskTarget(GrpcSiteTransport transport) + { + Receive(env => transport.Send(env, Sender)); + } + } + + private sealed class NoKeyProvider : ISitePskProvider + { + public ValueTask GetAsync(string siteId, CancellationToken ct) => new("k"); + public void Invalidate(string siteId) { } + } +} From 2cfcd890527975e9df1e675f5ceffcc9aedabbaa Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Fri, 14 Aug 2026 19:54:58 -0400 Subject: [PATCH 03/31] perf(sitelog): batched event-log commits and sliced retention purge --- .../EventLogPurgeService.cs | 44 ++++- .../SiteEventLogger.cs | 163 ++++++++++++++---- .../EventLogPurgeServiceTests.cs | 34 ++++ .../SiteEventLoggerBatchingTests.cs | 158 +++++++++++++++++ 4 files changed, 358 insertions(+), 41 deletions(-) create mode 100644 tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/SiteEventLoggerBatchingTests.cs diff --git a/src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/EventLogPurgeService.cs b/src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/EventLogPurgeService.cs index 5668d75f..13bf874d 100644 --- a/src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/EventLogPurgeService.cs +++ b/src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/EventLogPurgeService.cs @@ -30,6 +30,9 @@ public class EventLogPurgeService : BackgroundService /// Number of events deleted per cap-purge batch. private const int CapPurgeBatchSize = 1000; + /// Number of events deleted per retention-purge batch — same size as . + private const int RetentionPurgeBatchSize = 1000; + private readonly SiteEventLogger _eventLogger; private readonly SiteEventLogOptions _options; private readonly ILogger _logger; @@ -118,21 +121,46 @@ public class EventLogPurgeService : BackgroundService } } + /// + /// Deletes events older than the retention window, in bounded + /// -row slices per DELETE statement rather than + /// one unbounded DELETE for the whole expired set — mirroring the batching already + /// used a few lines below in (). + /// An expired backlog in the tens/hundreds of thousands of rows would otherwise hold a + /// single SQLite write transaction (and the shared _writeLock) open for the + /// whole delete, blocking every concurrent + /// writer-loop flush for that entire duration. + /// private void PurgeByRetention() { var cutoff = DateTimeOffset.UtcNow.AddDays(-_options.RetentionDays).ToString("o"); + var totalDeleted = 0; - var deleted = _eventLogger.WithConnection(connection => + while (true) { - using var cmd = connection.CreateCommand(); - cmd.CommandText = "DELETE FROM site_events WHERE timestamp < $cutoff"; - cmd.Parameters.AddWithValue("$cutoff", cutoff); - return cmd.ExecuteNonQuery(); - }); + var deleted = _eventLogger.WithConnection(connection => + { + using var cmd = connection.CreateCommand(); + cmd.CommandText = $""" + DELETE FROM site_events WHERE id IN ( + SELECT id FROM site_events + WHERE timestamp < $cutoff + LIMIT {RetentionPurgeBatchSize} + ) + """; + cmd.Parameters.AddWithValue("$cutoff", cutoff); + return cmd.ExecuteNonQuery(); + }); - if (deleted > 0) + totalDeleted += deleted; + + if (deleted < RetentionPurgeBatchSize) + break; + } + + if (totalDeleted > 0) { - _logger.LogInformation("Purged {Count} events older than {Days} days", deleted, _options.RetentionDays); + _logger.LogInformation("Purged {Count} events older than {Days} days", totalDeleted, _options.RetentionDays); } } diff --git a/src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogger.cs b/src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogger.cs index 95f2361c..169477c0 100644 --- a/src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogger.cs +++ b/src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogger.cs @@ -34,12 +34,23 @@ namespace ZB.MOM.WW.ScadaBridge.SiteEventLogging; /// public class SiteEventLogger : ISiteEventLogger, IDisposable { + /// + /// Maximum number of queued events committed together in a single transaction + /// per writer-loop drain. Mirrors SqliteAuditWriter's batch size + /// (ZB.MOM.WW.ScadaBridge.AuditLog/Site/SqliteAuditWriter.cs, + /// ProcessWriteQueueAsync/FlushBatch) — see the comment on + /// for why one transaction per drain + /// replaces the previous one-transaction-per-event shape. + /// + internal const int WriteBatchSize = 256; + private readonly SqliteConnection _connection; private readonly ILogger _logger; private readonly object _writeLock = new(); private readonly Channel _writeQueue; private readonly Task _writerLoop; private long _failedWriteCount; + private long _committedBatchCount; private bool _disposed; /// @@ -102,6 +113,14 @@ public class SiteEventLogger : ISiteEventLogger, IDisposable /// public long FailedWriteCount => Interlocked.Read(ref _failedWriteCount); + /// + /// Number of write-batch transactions successfully committed since startup. Test-only + /// observability hook (not part of ) proving the + /// "drain up to queued events into one transaction" shape + /// described on — production code does not read it. + /// + internal long CommittedBatchCount => Interlocked.Read(ref _committedBatchCount); + /// /// Runs against the shared connection while holding the /// write lock, so purge / query / record callers on different threads never use @@ -212,54 +231,132 @@ public class SiteEventLogger : ISiteEventLogger, IDisposable return pending.Completion.Task; } + /// + /// Drains up to queued events per iteration and commits + /// them in a single transaction, instead of one transaction per event. Mirrors the + /// reference shape in SqliteAuditWriter.ProcessWriteQueueAsync/FlushBatch + /// (ZB.MOM.WW.ScadaBridge.AuditLog/Site/SqliteAuditWriter.cs): pull the first + /// item with the blocking ReadAllAsync enumerator, then greedily TryRead + /// whatever else is already queued (non-blocking) up to the batch cap, so a burst of + /// concurrent callers amortises SQLite's per-transaction + /// fsync/journal overhead across the whole burst instead of paying it per row. + /// private async Task ProcessWriteQueueAsync() { - await foreach (var pending in _writeQueue.Reader.ReadAllAsync().ConfigureAwait(false)) + var batch = new List(WriteBatchSize); + + await foreach (var first in _writeQueue.Reader.ReadAllAsync().ConfigureAwait(false)) { - try + batch.Clear(); + batch.Add(first); + + while (batch.Count < WriteBatchSize && _writeQueue.Reader.TryRead(out var next)) { - var written = WithConnection(connection => + batch.Add(next); + } + + FlushBatch(batch); + } + } + + /// + /// Commits a drained batch of pending events in one transaction. All-or-nothing per + /// batch: site_events ids are freshly minted GUIDs per + /// call (no cross-call replay/de-dup concern the way SqliteAuditWriter has), so + /// unlike that reference there is no per-row duplicate-key swallow here — any failure + /// rolls back the whole batch and faults every event in it. site_events is a + /// CDC-replicated table (LocalDb Phase 1); its capture triggers fire per row on INSERT + /// same as before, so batching the commit changes only how many rows land per + /// transaction, not per-row trigger/replication behavior. + /// + private void FlushBatch(IReadOnlyList batch) + { + bool written; + Exception? failure = null; + + try + { + written = WithConnection(connection => + { + using var transaction = connection.BeginTransaction(); + try { using var cmd = connection.CreateCommand(); + cmd.Transaction = transaction; cmd.CommandText = """ INSERT INTO site_events (id, timestamp, event_type, severity, instance_id, source, message, details) VALUES ($id, $timestamp, $event_type, $severity, $instance_id, $source, $message, $details) """; - cmd.Parameters.AddWithValue("$id", pending.Id); - cmd.Parameters.AddWithValue("$timestamp", pending.Timestamp); - cmd.Parameters.AddWithValue("$event_type", pending.EventType); - cmd.Parameters.AddWithValue("$severity", pending.Severity); - cmd.Parameters.AddWithValue("$instance_id", (object?)pending.InstanceId ?? DBNull.Value); - cmd.Parameters.AddWithValue("$source", pending.Source); - cmd.Parameters.AddWithValue("$message", pending.Message); - cmd.Parameters.AddWithValue("$details", (object?)pending.Details ?? DBNull.Value); - cmd.ExecuteNonQuery(); - }); + var pId = cmd.Parameters.Add("$id", SqliteType.Text); + var pTimestamp = cmd.Parameters.Add("$timestamp", SqliteType.Text); + var pEventType = cmd.Parameters.Add("$event_type", SqliteType.Text); + var pSeverity = cmd.Parameters.Add("$severity", SqliteType.Text); + var pInstanceId = cmd.Parameters.Add("$instance_id", SqliteType.Text); + var pSource = cmd.Parameters.Add("$source", SqliteType.Text); + var pMessage = cmd.Parameters.Add("$message", SqliteType.Text); + var pDetails = cmd.Parameters.Add("$details", SqliteType.Text); - if (written) - { - pending.Completion.TrySetResult(); + foreach (var pending in batch) + { + pId.Value = pending.Id; + pTimestamp.Value = pending.Timestamp; + pEventType.Value = pending.EventType; + pSeverity.Value = pending.Severity; + pInstanceId.Value = (object?)pending.InstanceId ?? DBNull.Value; + pSource.Value = pending.Source; + pMessage.Value = pending.Message; + pDetails.Value = (object?)pending.Details ?? DBNull.Value; + cmd.ExecuteNonQuery(); + } + + transaction.Commit(); + Interlocked.Increment(ref _committedBatchCount); } - else + catch { - // WithConnection returns false only when the logger has been - // disposed mid-drain; the event was not persisted. Fault the - // Task instead of reporting false - // success for a dropped audit event. - pending.Completion.TrySetException( - new ObjectDisposedException(nameof(SiteEventLogger), - "Event could not be recorded: the event logger was disposed before the write completed.")); + transaction.Rollback(); + throw; } - } - catch (Exception ex) + }); + } + catch (Exception ex) + { + failure = ex; + written = false; + } + + if (failure is not null) + { + // A write failure must be observable. Count every event in the failed + // batch (Health Monitoring reads FailedWriteCount) and fault each + // caller's Task instead of silently discarding the exception. + Interlocked.Add(ref _failedWriteCount, batch.Count); + _logger.LogError(failure, "Failed to record {Count} event(s) in batch (sqlite {SqliteError})", + batch.Count, DescribeSqliteError(failure)); + foreach (var pending in batch) { - // A write failure must be observable. Count it - // (Health Monitoring reads FailedWriteCount) and fault the caller's - // Task instead of silently discarding the exception. - Interlocked.Increment(ref _failedWriteCount); - _logger.LogError(ex, "Failed to record event: {EventType} from {Source} (sqlite {SqliteError})", - pending.EventType, pending.Source, DescribeSqliteError(ex)); - pending.Completion.TrySetException(ex); + pending.Completion.TrySetException(failure); + } + return; + } + + if (written) + { + foreach (var pending in batch) + { + pending.Completion.TrySetResult(); + } + } + else + { + // WithConnection returns false only when the logger has been + // disposed mid-drain; none of the batch was persisted. Fault every + // Task instead of reporting false success for a dropped event. + var disposedEx = new ObjectDisposedException(nameof(SiteEventLogger), + "Event could not be recorded: the event logger was disposed before the write completed."); + foreach (var pending in batch) + { + pending.Completion.TrySetException(disposedEx); } } } diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/EventLogPurgeServiceTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/EventLogPurgeServiceTests.cs index 22240faa..494385e6 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/EventLogPurgeServiceTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/EventLogPurgeServiceTests.cs @@ -106,6 +106,40 @@ public class EventLogPurgeServiceTests : IDisposable Assert.Equal(3, GetEventCount()); } + [Fact] + public void PurgeByRetention_OverBatchSize_DeletesAllExpiredRowsAcrossSlices() + { + // WP1.7: the retention purge must slice a large expired backlog into + // bounded DELETE batches (1000 rows/iteration, mirroring the storage-cap + // purge a few lines below in the same file) rather than one unbounded + // DELETE. Seed more than one slice's worth of expired rows plus a + // handful of recent rows, and assert the whole expired set is gone — + // not just the first 1000 rows — while the recent rows survive. + const int expiredCount = 2500; // > one 1000-row slice, not an exact multiple + var expiredBase = DateTimeOffset.UtcNow.AddDays(-31); + _eventLogger.WithConnection(connection => + { + for (var i = 0; i < expiredCount; i++) + { + using var cmd = connection.CreateCommand(); + cmd.CommandText = """ + INSERT INTO site_events (id, timestamp, event_type, severity, source, message) + VALUES ($id, $ts, 'script', 'Info', 'Test', 'Test message') + """; + cmd.Parameters.AddWithValue("$id", Guid.NewGuid().ToString("N")); + cmd.Parameters.AddWithValue("$ts", expiredBase.AddSeconds(i).ToString("o")); + cmd.ExecuteNonQuery(); + } + }); + InsertEventWithTimestamp(DateTimeOffset.UtcNow.AddDays(-1)); + InsertEventWithTimestamp(DateTimeOffset.UtcNow); + + var purge = CreatePurgeService(); + purge.RunPurge(); + + Assert.Equal(2, GetEventCount()); + } + [Fact] public void PurgeByStorageCap_DeletesOldestWhenOverCap() { diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/SiteEventLoggerBatchingTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/SiteEventLoggerBatchingTests.cs new file mode 100644 index 00000000..c274f871 --- /dev/null +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/SiteEventLoggerBatchingTests.cs @@ -0,0 +1,158 @@ +using Microsoft.Extensions.Logging.Abstractions; +using Microsoft.Extensions.Options; + +namespace ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests; + +/// +/// WP1.7 (arch-review remediation, 2026-08-14): 's +/// background writer loop must drain its bounded channel into batched transactions +/// (up to events per commit) instead of +/// committing one transaction per event — mirroring the reference shape in +/// ZB.MOM.WW.ScadaBridge.AuditLog/Site/SqliteAuditWriter.cs +/// (ProcessWriteQueueAsync/FlushBatch). +/// +public class SiteEventLoggerBatchingTests : IDisposable +{ + private readonly SiteEventLogger _logger; + private readonly string _dbPath; + private readonly TestLocalDb _localDb; + + public SiteEventLoggerBatchingTests() + { + _dbPath = Path.Combine(Path.GetTempPath(), $"test_batch_{Guid.NewGuid()}.db"); + var options = Options.Create(new SiteEventLogOptions { DatabasePath = _dbPath }); + _localDb = TestLocalDb.Create(_dbPath); + _logger = new SiteEventLogger(options, NullLogger.Instance, _localDb.Db); + } + + public void Dispose() + { + _logger.Dispose(); + _localDb.Dispose(); + TestLocalDb.DeleteFiles(_dbPath); + } + + [Fact] + public async Task LogEventAsync_SingleEvent_CommitsOneBatch() + { + await _logger.LogEventAsync("script", "Info", null, "Source", "one event"); + + Assert.Equal(1, _logger.CommittedBatchCount); + } + + [Fact] + public async Task ProcessWriteQueue_BurstOfQueuedEvents_CommitsFarFewerBatchesThanEvents() + { + // Deterministically force a large burst of events to be sitting in the + // channel BEFORE the writer loop is free to drain them, so the drain is + // not at the mercy of scheduler timing (the same "hold the connection + // busy" technique SiteEventLoggerAsyncTests uses to prove the caller + // never blocks). While the write lock is held, the writer loop's first + // FlushBatch call blocks trying to acquire it — every event enqueued + // during that window queues up untouched. Once released, the writer + // loop's next iterations greedily TryRead everything already queued + // into batches capped at WriteBatchSize (256), so a burst well above + // that cap must still land in only a handful of transactions. + var busyStarted = new ManualResetEventSlim(false); + var releaseBusy = new ManualResetEventSlim(false); + + var busyThread = new Thread(() => + { + _logger.WithConnection(_ => + { + busyStarted.Set(); + releaseBusy.Wait(TimeSpan.FromSeconds(10)); + }); + }); + busyThread.Start(); + Assert.True(busyStarted.Wait(TimeSpan.FromSeconds(5)), "Busy thread did not start."); + + // > WriteBatchSize (256) so a correctly batching writer must still take + // at least ceil(burstCount / WriteBatchSize) = 2 post-release batches, + // plus at most one partial batch already blocked on the lock — nowhere + // near burstCount (300) transactions. + const int burstCount = 300; + var tasks = new List(burstCount); + for (var i = 0; i < burstCount; i++) + { + tasks.Add(_logger.LogEventAsync("script", "Info", null, "Burst", $"event {i}")); + } + + releaseBusy.Set(); + busyThread.Join(TimeSpan.FromSeconds(10)); + + await Task.WhenAll(tasks).WaitAsync(TimeSpan.FromSeconds(30)); + + var batchesUsed = _logger.CommittedBatchCount; + var expectedMax = (long)Math.Ceiling(burstCount / (double)SiteEventLogger.WriteBatchSize) + 1; + + Assert.True(batchesUsed <= expectedMax, + $"Expected at most {expectedMax} batch transactions for {burstCount} queued events " + + $"(WriteBatchSize={SiteEventLogger.WriteBatchSize}), but {batchesUsed} were committed — " + + "the writer loop is not batching the drain into few transactions."); + Assert.True(batchesUsed < burstCount, + $"{batchesUsed} batches were committed for {burstCount} events — that is one-transaction-per-event, not batched."); + + var count = _logger.WithConnection(connection => + { + using var cmd = connection.CreateCommand(); + cmd.CommandText = "SELECT COUNT(*) FROM site_events"; + return (long)cmd.ExecuteScalar()!; + }); + Assert.Equal(burstCount, count); + } + + [Fact] + public async Task ProcessWriteQueue_BatchFailure_FaultsEveryQueuedEvent_NoneSilentlyDropped() + { + // Batching introduces a risk absent from the old one-transaction-per-event + // shape: a failure could silently drop or under-report events that were + // swept into the same failed batch as the one that actually errored, + // instead of faulting every affected Task and counting every affected + // event in FailedWriteCount. Force a burst to queue up while the writer + // is blocked on the shared lock, then make every subsequent write fail + // (drop the table) before releasing — regardless of how the burst splits + // across batch(es) once released, every queued event's Task must still + // fault and FailedWriteCount must still account for all of them. + var busyStarted = new ManualResetEventSlim(false); + var releaseBusy = new ManualResetEventSlim(false); + + var busyThread = new Thread(() => + { + _logger.WithConnection(connection => + { + busyStarted.Set(); + releaseBusy.Wait(TimeSpan.FromSeconds(10)); + + // Drop the table while still holding the lock, immediately before + // releasing it, so the burst below (already fully queued during + // the busy window) hits a missing table as ONE batch. + using var cmd = connection.CreateCommand(); + cmd.CommandText = "DROP TABLE site_events"; + cmd.ExecuteNonQuery(); + }); + }); + busyThread.Start(); + Assert.True(busyStarted.Wait(TimeSpan.FromSeconds(5)), "Busy thread did not start."); + + const int burstCount = 10; + var tasks = new List(burstCount); + for (var i = 0; i < burstCount; i++) + { + tasks.Add(_logger.LogEventAsync("script", "Info", null, "Burst", $"event {i}")); + } + + releaseBusy.Set(); + busyThread.Join(TimeSpan.FromSeconds(10)); + + // Every event in the batch must fault — none silently "succeeds" while + // its siblings fail, and none is silently dropped. + foreach (var task in tasks) + { + await Assert.ThrowsAnyAsync(() => task).WaitAsync(TimeSpan.FromSeconds(10)); + } + + Assert.True(_logger.FailedWriteCount >= burstCount, + $"Expected FailedWriteCount to account for all {burstCount} events in the rolled-back batch, got {_logger.FailedWriteCount}."); + } +} From 600659d579a050dea6aa18cd5a3dca4254a12a4c Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Fri, 14 Aug 2026 19:55:48 -0400 Subject: [PATCH 04/31] perf(sql): sweep/KPI covering indexes + sliced notification terminal purge --- ...AuditLogAndNotificationCoveringIndexes.sql | 35 + .../AuditLogEntityTypeConfiguration.cs | 7 +- .../NotificationOutboxConfiguration.cs | 13 + ...AndNotificationCoveringIndexes.Designer.cs | 2100 +++++++++++++++++ ...dAuditLogAndNotificationCoveringIndexes.cs | 65 + .../ScadaBridgeDbContextModelSnapshot.cs | 6 + .../NotificationOutboxRepository.cs | 25 +- .../StoreAndForwardSchema.cs | 17 + 8 files changed, 2265 insertions(+), 3 deletions(-) create mode 100644 docs/plans/sql/AddAuditLogAndNotificationCoveringIndexes.sql create mode 100644 src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/20260814235335_AddAuditLogAndNotificationCoveringIndexes.Designer.cs create mode 100644 src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/20260814235335_AddAuditLogAndNotificationCoveringIndexes.cs diff --git a/docs/plans/sql/AddAuditLogAndNotificationCoveringIndexes.sql b/docs/plans/sql/AddAuditLogAndNotificationCoveringIndexes.sql new file mode 100644 index 00000000..c051c106 --- /dev/null +++ b/docs/plans/sql/AddAuditLogAndNotificationCoveringIndexes.sql @@ -0,0 +1,35 @@ +BEGIN TRANSACTION; +IF NOT EXISTS ( + SELECT * FROM [__EFMigrationsHistory] + WHERE [MigrationId] = N'20260814235335_AddAuditLogAndNotificationCoveringIndexes' +) +BEGIN + EXEC(N'CREATE INDEX [IX_Notifications_Delivered] ON [Notifications] ([DeliveredAt]) WHERE [Status] = ''Delivered'''); +END; + +IF NOT EXISTS ( + SELECT * FROM [__EFMigrationsHistory] + WHERE [MigrationId] = N'20260814235335_AddAuditLogAndNotificationCoveringIndexes' +) +BEGIN + + DROP INDEX IX_AuditLog_OccurredAtUtc ON dbo.AuditLog; + + CREATE NONCLUSTERED INDEX IX_AuditLog_OccurredAtUtc + ON dbo.AuditLog (OccurredAtUtc DESC) + INCLUDE (Status) + ON ps_AuditLog_Month(OccurredAtUtc); +END; + +IF NOT EXISTS ( + SELECT * FROM [__EFMigrationsHistory] + WHERE [MigrationId] = N'20260814235335_AddAuditLogAndNotificationCoveringIndexes' +) +BEGIN + INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) + VALUES (N'20260814235335_AddAuditLogAndNotificationCoveringIndexes', N'10.0.7'); +END; + +COMMIT; +GO + diff --git a/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Configurations/AuditLogEntityTypeConfiguration.cs b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Configurations/AuditLogEntityTypeConfiguration.cs index 9b86006e..269854d9 100644 --- a/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Configurations/AuditLogEntityTypeConfiguration.cs +++ b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Configurations/AuditLogEntityTypeConfiguration.cs @@ -185,9 +185,14 @@ public class AuditLogEntityTypeConfiguration : IEntityTypeConfiguration e.OccurredAtUtc) .IsDescending(true) - .HasDatabaseName("IX_AuditLog_OccurredAtUtc"); + .HasDatabaseName("IX_AuditLog_OccurredAtUtc") + .IncludeProperties(nameof(AuditLogRow.Status)); builder.HasIndex(e => new { e.SourceSiteId, e.OccurredAtUtc }) .IsDescending(false, true) diff --git a/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Configurations/NotificationOutboxConfiguration.cs b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Configurations/NotificationOutboxConfiguration.cs index f1f9490e..b346fbed 100644 --- a/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Configurations/NotificationOutboxConfiguration.cs +++ b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Configurations/NotificationOutboxConfiguration.cs @@ -70,5 +70,18 @@ public class NotificationOutboxConfiguration : IEntityTypeConfiguration new { n.Status, n.NextAttemptAt }); builder.HasIndex(n => new { n.SourceSiteId, n.CreatedAt }); + + // Covers ComputeKpisAsync's "DeliveredLastInterval" count + // (Status == Delivered && DeliveredAt >= deliveredSince): filtered to just the + // Delivered rows so the index stays small relative to the full table and the KPI + // query can seek it directly instead of scanning every status (arch-review WP1.4). + // Status is stored as its enum name (HasConversion() above), so the filter + // predicate matches on the literal 'Delivered', not the underlying int value. + // No N'' prefix: the value is ASCII-only (fine on SQL Server's nvarchar column) + // and the repository tests' SQLite provider rejects the T-SQL N'' national-string + // literal syntax outright when EnsureCreated() runs this filter as DDL. + builder.HasIndex(n => n.DeliveredAt) + .HasDatabaseName("IX_Notifications_Delivered") + .HasFilter("[Status] = 'Delivered'"); } } diff --git a/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/20260814235335_AddAuditLogAndNotificationCoveringIndexes.Designer.cs b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/20260814235335_AddAuditLogAndNotificationCoveringIndexes.Designer.cs new file mode 100644 index 00000000..5f9c7757 --- /dev/null +++ b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/20260814235335_AddAuditLogAndNotificationCoveringIndexes.Designer.cs @@ -0,0 +1,2100 @@ +// +using System; +using Microsoft.EntityFrameworkCore; +using Microsoft.EntityFrameworkCore.Infrastructure; +using Microsoft.EntityFrameworkCore.Metadata; +using Microsoft.EntityFrameworkCore.Migrations; +using Microsoft.EntityFrameworkCore.Storage.ValueConversion; +using ZB.MOM.WW.ScadaBridge.ConfigurationDatabase; + +#nullable disable + +namespace ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Migrations +{ + [DbContext(typeof(ScadaBridgeDbContext))] + [Migration("20260814235335_AddAuditLogAndNotificationCoveringIndexes")] + partial class AddAuditLogAndNotificationCoveringIndexes + { + /// + protected override void BuildTargetModel(ModelBuilder modelBuilder) + { +#pragma warning disable 612, 618 + modelBuilder + .HasAnnotation("ProductVersion", "10.0.7") + .HasAnnotation("Relational:MaxIdentifierLength", 128); + + SqlServerModelBuilderExtensions.UseIdentityColumns(modelBuilder); + + modelBuilder.Entity("Microsoft.AspNetCore.DataProtection.EntityFrameworkCore.DataProtectionKey", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("FriendlyName") + .HasColumnType("nvarchar(max)"); + + b.Property("Xml") + .HasColumnType("nvarchar(max)"); + + b.HasKey("Id"); + + b.ToTable("DataProtectionKeys"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Audit.AuditLogEntry", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("Action") + .IsRequired() + .HasMaxLength(100) + .HasColumnType("nvarchar(100)"); + + b.Property("AfterStateJson") + .HasColumnType("nvarchar(max)"); + + b.Property("BundleImportId") + .HasColumnType("uniqueidentifier"); + + b.Property("EntityId") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("EntityName") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("EntityType") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("Timestamp") + .HasColumnType("datetimeoffset"); + + b.Property("User") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.HasKey("Id"); + + b.HasIndex("Action"); + + b.HasIndex("BundleImportId") + .HasDatabaseName("IX_AuditLogEntries_BundleImportId"); + + b.HasIndex("EntityId"); + + b.HasIndex("EntityType"); + + b.HasIndex("Timestamp"); + + b.HasIndex("User"); + + b.ToTable("AuditLogEntries"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Audit.SiteCall", b => + { + b.Property("TrackedOperationId") + .HasMaxLength(36) + .IsUnicode(false) + .HasColumnType("varchar(36)"); + + b.Property("Channel") + .IsRequired() + .HasMaxLength(32) + .IsUnicode(false) + .HasColumnType("varchar(32)"); + + b.Property("CreatedAtUtc") + .HasColumnType("datetime2"); + + b.Property("HttpStatus") + .HasColumnType("int"); + + b.Property("IngestedAtUtc") + .HasColumnType("datetime2"); + + b.Property("LastError") + .HasMaxLength(1024) + .HasColumnType("nvarchar(1024)"); + + b.Property("RetryCount") + .HasColumnType("int"); + + b.Property("SourceNode") + .HasMaxLength(64) + .IsUnicode(false) + .HasColumnType("varchar(64)"); + + b.Property("SourceSite") + .IsRequired() + .HasMaxLength(64) + .IsUnicode(false) + .HasColumnType("varchar(64)"); + + b.Property("Status") + .IsRequired() + .HasMaxLength(32) + .IsUnicode(false) + .HasColumnType("varchar(32)"); + + b.Property("Target") + .IsRequired() + .HasMaxLength(256) + .IsUnicode(false) + .HasColumnType("varchar(256)"); + + b.Property("TerminalAtUtc") + .HasColumnType("datetime2"); + + b.Property("UpdatedAtUtc") + .HasColumnType("datetime2"); + + b.HasKey("TrackedOperationId"); + + b.HasIndex("CreatedAtUtc") + .HasDatabaseName("IX_SiteCalls_NonTerminal") + .HasFilter("[TerminalAtUtc] IS NULL"); + + SqlServerIndexBuilderExtensions.IncludeProperties(b.HasIndex("CreatedAtUtc"), new[] { "SourceSite", "SourceNode", "Status" }); + + b.HasIndex("TerminalAtUtc") + .HasDatabaseName("IX_SiteCalls_Terminal") + .HasFilter("[TerminalAtUtc] IS NOT NULL"); + + b.HasIndex("SourceSite", "CreatedAtUtc") + .IsDescending(false, true) + .HasDatabaseName("IX_SiteCalls_Source_Created"); + + b.HasIndex("Status", "UpdatedAtUtc") + .IsDescending(false, true) + .HasDatabaseName("IX_SiteCalls_Status_Updated"); + + b.ToTable("SiteCalls", (string)null); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Deployment.DeployedConfigSnapshot", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("ConfigurationJson") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("DeployedAt") + .HasColumnType("datetimeoffset"); + + b.Property("DeploymentId") + .IsRequired() + .HasMaxLength(100) + .HasColumnType("nvarchar(100)"); + + b.Property("InstanceId") + .HasColumnType("int"); + + b.Property("RevisionHash") + .IsRequired() + .HasMaxLength(100) + .HasColumnType("nvarchar(100)"); + + b.HasKey("Id"); + + b.HasIndex("DeploymentId"); + + b.HasIndex("InstanceId") + .IsUnique(); + + b.ToTable("DeployedConfigSnapshots"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Deployment.DeploymentRecord", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("CompletedAt") + .HasColumnType("datetimeoffset"); + + b.Property("DeployedAt") + .HasColumnType("datetimeoffset"); + + b.Property("DeployedBy") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("DeploymentId") + .IsRequired() + .HasMaxLength(100) + .HasColumnType("nvarchar(100)"); + + b.Property("ErrorMessage") + .HasColumnType("nvarchar(max)"); + + b.Property("InstanceId") + .HasColumnType("int"); + + b.Property("RevisionHash") + .HasMaxLength(100) + .HasColumnType("nvarchar(100)"); + + b.Property("RowVersion") + .IsConcurrencyToken() + .IsRequired() + .ValueGeneratedOnAddOrUpdate() + .HasColumnType("rowversion"); + + b.Property("Status") + .IsRequired() + .HasMaxLength(50) + .HasColumnType("nvarchar(50)"); + + b.HasKey("Id"); + + b.HasIndex("DeployedAt"); + + b.HasIndex("DeploymentId") + .IsUnique(); + + b.HasIndex("InstanceId"); + + b.ToTable("DeploymentRecords"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Deployment.PendingDeployment", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("ConfigurationJson") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("CreatedAtUtc") + .HasColumnType("datetimeoffset"); + + b.Property("DeploymentId") + .IsRequired() + .HasMaxLength(100) + .HasColumnType("nvarchar(100)"); + + b.Property("ExpiresAtUtc") + .HasColumnType("datetimeoffset"); + + b.Property("InstanceId") + .HasColumnType("int"); + + b.Property("RevisionHash") + .IsRequired() + .HasMaxLength(100) + .HasColumnType("nvarchar(100)"); + + b.Property("Token") + .IsRequired() + .HasMaxLength(128) + .HasColumnType("nvarchar(128)"); + + b.HasKey("Id"); + + b.HasIndex("DeploymentId") + .IsUnique(); + + b.HasIndex("ExpiresAtUtc"); + + b.HasIndex("InstanceId"); + + b.ToTable("PendingDeployments", (string)null); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Deployment.SystemArtifactDeploymentRecord", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("ArtifactType") + .IsRequired() + .HasMaxLength(100) + .HasColumnType("nvarchar(100)"); + + b.Property("DeployedAt") + .HasColumnType("datetimeoffset"); + + b.Property("DeployedBy") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("PerSiteStatus") + .HasMaxLength(4000) + .HasColumnType("nvarchar(4000)"); + + b.HasKey("Id"); + + b.HasIndex("DeployedAt"); + + b.ToTable("SystemArtifactDeploymentRecords"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.ExternalSystems.DatabaseConnectionDefinition", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("ConnectionString") + .IsRequired() + .HasMaxLength(8000) + .HasColumnType("nvarchar(max)"); + + b.Property("MaxRetries") + .HasColumnType("int"); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("RetryDelay") + .HasColumnType("time"); + + b.HasKey("Id"); + + b.HasIndex("Name") + .IsUnique(); + + b.ToTable("DatabaseConnectionDefinitions"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.ExternalSystems.ExternalSystemDefinition", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("AuthConfiguration") + .HasMaxLength(8000) + .HasColumnType("nvarchar(max)"); + + b.Property("AuthType") + .IsRequired() + .HasMaxLength(50) + .HasColumnType("nvarchar(50)"); + + b.Property("EndpointUrl") + .IsRequired() + .HasMaxLength(2000) + .HasColumnType("nvarchar(2000)"); + + b.Property("MaxRetries") + .HasColumnType("int"); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("RetryDelay") + .HasColumnType("time"); + + b.Property("TimeoutSeconds") + .ValueGeneratedOnAdd() + .HasColumnType("int") + .HasDefaultValue(0); + + b.HasKey("Id"); + + b.HasIndex("Name") + .IsUnique(); + + b.ToTable("ExternalSystemDefinitions"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.ExternalSystems.ExternalSystemMethod", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("ExternalSystemDefinitionId") + .HasColumnType("int"); + + b.Property("HttpMethod") + .IsRequired() + .HasMaxLength(10) + .HasColumnType("nvarchar(10)"); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("ParameterDefinitions") + .HasMaxLength(4000) + .HasColumnType("nvarchar(4000)"); + + b.Property("Path") + .IsRequired() + .HasMaxLength(2000) + .HasColumnType("nvarchar(2000)"); + + b.Property("ReturnDefinition") + .HasMaxLength(4000) + .HasColumnType("nvarchar(4000)"); + + b.HasKey("Id"); + + b.HasIndex("ExternalSystemDefinitionId", "Name") + .IsUnique(); + + b.ToTable("ExternalSystemMethods"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.InboundApi.ApiMethod", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("ParameterDefinitions") + .HasMaxLength(4000) + .HasColumnType("nvarchar(4000)"); + + b.Property("ReturnDefinition") + .HasMaxLength(4000) + .HasColumnType("nvarchar(4000)"); + + b.Property("Script") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("TimeoutSeconds") + .HasColumnType("int"); + + b.HasKey("Id"); + + b.HasIndex("Name") + .IsUnique(); + + b.ToTable("ApiMethods"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.Area", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("ParentAreaId") + .HasColumnType("int"); + + b.Property("SiteId") + .HasColumnType("int"); + + b.HasKey("Id"); + + b.HasIndex("ParentAreaId"); + + b.HasIndex("SiteId", "ParentAreaId", "Name") + .IsUnique() + .HasFilter("[ParentAreaId] IS NOT NULL"); + + b.ToTable("Areas"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.Instance", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("AreaId") + .HasColumnType("int"); + + b.Property("SiteId") + .HasColumnType("int"); + + b.Property("State") + .IsRequired() + .HasMaxLength(50) + .HasColumnType("nvarchar(50)"); + + b.Property("TemplateId") + .HasColumnType("int"); + + b.Property("UniqueName") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.HasKey("Id"); + + b.HasIndex("AreaId"); + + b.HasIndex("TemplateId"); + + b.HasIndex("SiteId", "UniqueName") + .IsUnique(); + + b.ToTable("Instances"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.InstanceAlarmOverride", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("AlarmCanonicalName") + .IsRequired() + .HasMaxLength(400) + .HasColumnType("nvarchar(400)"); + + b.Property("InstanceId") + .HasColumnType("int"); + + b.Property("PriorityLevelOverride") + .HasColumnType("int"); + + b.Property("TriggerConfigurationOverride") + .HasMaxLength(4000) + .HasColumnType("nvarchar(4000)"); + + b.HasKey("Id"); + + b.HasIndex("InstanceId", "AlarmCanonicalName") + .IsUnique(); + + b.ToTable("InstanceAlarmOverrides"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.InstanceAttributeOverride", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("AttributeName") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("ElementDataType") + .HasMaxLength(50) + .HasColumnType("nvarchar(50)"); + + b.Property("InstanceId") + .HasColumnType("int"); + + b.Property("OverrideValue") + .HasColumnType("nvarchar(max)"); + + b.HasKey("Id"); + + b.HasIndex("InstanceId", "AttributeName") + .IsUnique(); + + b.ToTable("InstanceAttributeOverrides"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.InstanceConnectionBinding", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("AttributeName") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("DataConnectionId") + .HasColumnType("int"); + + b.Property("DataSourceReferenceOverride") + .HasMaxLength(512) + .HasColumnType("nvarchar(512)"); + + b.Property("InstanceId") + .HasColumnType("int"); + + b.HasKey("Id"); + + b.HasIndex("DataConnectionId"); + + b.HasIndex("InstanceId", "AttributeName") + .IsUnique(); + + b.ToTable("InstanceConnectionBindings"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.InstanceNativeAlarmSourceOverride", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("ConditionFilterOverride") + .HasMaxLength(1000) + .HasColumnType("nvarchar(1000)"); + + b.Property("ConnectionNameOverride") + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("InstanceId") + .HasColumnType("int"); + + b.Property("SourceCanonicalName") + .IsRequired() + .HasMaxLength(400) + .HasColumnType("nvarchar(400)"); + + b.Property("SourceReferenceOverride") + .HasMaxLength(1000) + .HasColumnType("nvarchar(1000)"); + + b.HasKey("Id"); + + b.HasIndex("InstanceId", "SourceCanonicalName") + .IsUnique(); + + b.ToTable("InstanceNativeAlarmSourceOverrides"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Kpi.KpiRollupHourly", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("bigint"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("HourStartUtc") + .HasColumnType("datetime2"); + + b.Property("MaxValue") + .HasColumnType("float"); + + b.Property("Metric") + .IsRequired() + .HasMaxLength(64) + .IsUnicode(false) + .HasColumnType("varchar(64)"); + + b.Property("MinValue") + .HasColumnType("float"); + + b.Property("SampleCount") + .HasColumnType("int"); + + b.Property("Scope") + .IsRequired() + .HasMaxLength(16) + .IsUnicode(false) + .HasColumnType("varchar(16)"); + + b.Property("ScopeKey") + .HasMaxLength(64) + .IsUnicode(false) + .HasColumnType("varchar(64)"); + + b.Property("Source") + .IsRequired() + .HasMaxLength(64) + .IsUnicode(false) + .HasColumnType("varchar(64)"); + + b.Property("Value") + .HasColumnType("float"); + + b.HasKey("Id"); + + b.HasIndex("HourStartUtc") + .HasDatabaseName("IX_KpiRollupHourly_Hour"); + + b.HasIndex("Source", "Metric", "Scope", "ScopeKey", "HourStartUtc") + .IsUnique() + .HasDatabaseName("IX_KpiRollupHourly_Series"); + + b.ToTable("KpiRollupHourly", (string)null); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Kpi.KpiSample", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("bigint"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("CapturedAtUtc") + .HasColumnType("datetime2"); + + b.Property("Metric") + .IsRequired() + .HasMaxLength(64) + .IsUnicode(false) + .HasColumnType("varchar(64)"); + + b.Property("Scope") + .IsRequired() + .HasMaxLength(16) + .IsUnicode(false) + .HasColumnType("varchar(16)"); + + b.Property("ScopeKey") + .HasMaxLength(64) + .IsUnicode(false) + .HasColumnType("varchar(64)"); + + b.Property("Source") + .IsRequired() + .HasMaxLength(64) + .IsUnicode(false) + .HasColumnType("varchar(64)"); + + b.Property("Value") + .HasColumnType("float"); + + b.HasKey("Id"); + + b.HasIndex("CapturedAtUtc") + .HasDatabaseName("IX_KpiSample_Captured"); + + b.HasIndex("Source", "Metric", "Scope", "ScopeKey", "CapturedAtUtc") + .HasDatabaseName("IX_KpiSample_Series"); + + b.ToTable("KpiSample", (string)null); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Notifications.Notification", b => + { + b.Property("NotificationId") + .HasMaxLength(64) + .HasColumnType("nvarchar(64)"); + + b.Property("Body") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("CreatedAt") + .HasColumnType("datetimeoffset"); + + b.Property("DeliveredAt") + .HasColumnType("datetimeoffset"); + + b.Property("LastAttemptAt") + .HasColumnType("datetimeoffset"); + + b.Property("LastError") + .HasMaxLength(4000) + .HasColumnType("nvarchar(4000)"); + + b.Property("ListName") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("NextAttemptAt") + .HasColumnType("datetimeoffset"); + + b.Property("OriginExecutionId") + .HasColumnType("uniqueidentifier"); + + b.Property("OriginParentExecutionId") + .HasColumnType("uniqueidentifier"); + + b.Property("ResolvedTargets") + .HasColumnType("nvarchar(max)"); + + b.Property("RetryCount") + .HasColumnType("int"); + + b.Property("SiteEnqueuedAt") + .HasColumnType("datetimeoffset"); + + b.Property("SourceInstanceId") + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("SourceNode") + .HasMaxLength(64) + .IsUnicode(false) + .HasColumnType("varchar(64)"); + + b.Property("SourceScript") + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("SourceSiteId") + .IsRequired() + .HasMaxLength(100) + .HasColumnType("nvarchar(100)"); + + b.Property("Status") + .IsRequired() + .HasMaxLength(32) + .HasColumnType("nvarchar(32)"); + + b.Property("Subject") + .IsRequired() + .HasMaxLength(1000) + .HasColumnType("nvarchar(1000)"); + + b.Property("Type") + .IsRequired() + .HasMaxLength(32) + .HasColumnType("nvarchar(32)"); + + b.Property("TypeData") + .HasColumnType("nvarchar(max)"); + + b.HasKey("NotificationId"); + + b.HasIndex("DeliveredAt") + .HasDatabaseName("IX_Notifications_Delivered") + .HasFilter("[Status] = 'Delivered'"); + + b.HasIndex("SourceSiteId", "CreatedAt"); + + b.HasIndex("Status", "NextAttemptAt"); + + b.ToTable("Notifications"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Notifications.NotificationList", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("Type") + .IsRequired() + .HasMaxLength(32) + .HasColumnType("nvarchar(32)"); + + b.HasKey("Id"); + + b.HasIndex("Name") + .IsUnique(); + + b.ToTable("NotificationLists"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Notifications.NotificationRecipient", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("EmailAddress") + .HasMaxLength(500) + .HasColumnType("nvarchar(500)"); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("NotificationListId") + .HasColumnType("int"); + + b.Property("PhoneNumber") + .HasMaxLength(32) + .HasColumnType("nvarchar(32)"); + + b.HasKey("Id"); + + b.HasIndex("NotificationListId"); + + b.ToTable("NotificationRecipients"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Notifications.SmsConfiguration", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("AccountSid") + .IsRequired() + .HasMaxLength(100) + .HasColumnType("nvarchar(100)"); + + b.Property("ApiBaseUrl") + .HasMaxLength(500) + .HasColumnType("nvarchar(500)"); + + b.Property("AuthToken") + .HasMaxLength(8000) + .HasColumnType("nvarchar(max)"); + + b.Property("ConnectionTimeoutSeconds") + .HasColumnType("int"); + + b.Property("FromNumber") + .HasMaxLength(32) + .HasColumnType("nvarchar(32)"); + + b.Property("MaxRetries") + .HasColumnType("int"); + + b.Property("MessagingServiceSid") + .HasMaxLength(100) + .HasColumnType("nvarchar(100)"); + + b.Property("RetryDelay") + .HasColumnType("time"); + + b.HasKey("Id"); + + b.ToTable("SmsConfigurations"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Notifications.SmtpConfiguration", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("AuthType") + .IsRequired() + .HasMaxLength(50) + .HasColumnType("nvarchar(50)"); + + b.Property("ConnectionTimeoutSeconds") + .HasColumnType("int"); + + b.Property("Credentials") + .HasMaxLength(8000) + .HasColumnType("nvarchar(max)"); + + b.Property("FromAddress") + .IsRequired() + .HasMaxLength(500) + .HasColumnType("nvarchar(500)"); + + b.Property("Host") + .IsRequired() + .HasMaxLength(500) + .HasColumnType("nvarchar(500)"); + + b.Property("MaxConcurrentConnections") + .HasColumnType("int"); + + b.Property("MaxRetries") + .HasColumnType("int"); + + b.Property("OAuth2Authority") + .HasMaxLength(500) + .HasColumnType("nvarchar(500)"); + + b.Property("OAuth2Scope") + .HasMaxLength(500) + .HasColumnType("nvarchar(500)"); + + b.Property("Port") + .HasColumnType("int"); + + b.Property("RetryDelay") + .HasColumnType("time"); + + b.Property("TlsMode") + .HasMaxLength(50) + .HasColumnType("nvarchar(50)"); + + b.Property("Transport") + .HasMaxLength(50) + .HasColumnType("nvarchar(50)"); + + b.HasKey("Id"); + + b.ToTable("SmtpConfigurations"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Schemas.SharedSchema", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("SchemaJson") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("Scope") + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.HasKey("Id"); + + b.HasIndex("Name") + .IsUnique(); + + b.ToTable("SharedSchemas"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Scripts.SharedScript", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("Code") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("ParameterDefinitions") + .HasMaxLength(4000) + .HasColumnType("nvarchar(4000)"); + + b.Property("ReturnDefinition") + .HasMaxLength(4000) + .HasColumnType("nvarchar(4000)"); + + b.HasKey("Id"); + + b.HasIndex("Name") + .IsUnique(); + + b.ToTable("SharedScripts"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.SecuredWrites.PendingSecuredWrite", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("bigint"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("ConnectionName") + .IsRequired() + .HasMaxLength(128) + .IsUnicode(false) + .HasColumnType("varchar(128)"); + + b.Property("DecidedAtUtc") + .HasColumnType("datetime2"); + + b.Property("ExecutedAtUtc") + .HasColumnType("datetime2"); + + b.Property("ExecutionError") + .HasMaxLength(2048) + .IsUnicode(false) + .HasColumnType("varchar(2048)"); + + b.Property("OperatorComment") + .HasMaxLength(1024) + .IsUnicode(false) + .HasColumnType("varchar(1024)"); + + b.Property("OperatorUser") + .IsRequired() + .HasMaxLength(256) + .IsUnicode(false) + .HasColumnType("varchar(256)"); + + b.Property("SiteId") + .IsRequired() + .HasMaxLength(128) + .IsUnicode(false) + .HasColumnType("varchar(128)"); + + b.Property("Status") + .IsRequired() + .HasMaxLength(32) + .IsUnicode(false) + .HasColumnType("varchar(32)"); + + b.Property("SubmittedAtUtc") + .HasColumnType("datetime2"); + + b.Property("TagPath") + .IsRequired() + .HasMaxLength(512) + .IsUnicode(false) + .HasColumnType("varchar(512)"); + + b.Property("ValueJson") + .IsRequired() + .HasMaxLength(4000) + .IsUnicode(false) + .HasColumnType("varchar(4000)"); + + b.Property("ValueType") + .IsRequired() + .HasMaxLength(128) + .IsUnicode(false) + .HasColumnType("varchar(128)"); + + b.Property("VerifierComment") + .HasMaxLength(1024) + .IsUnicode(false) + .HasColumnType("varchar(1024)"); + + b.Property("VerifierUser") + .HasMaxLength(256) + .IsUnicode(false) + .HasColumnType("varchar(256)"); + + b.HasKey("Id"); + + b.HasIndex("SiteId") + .HasDatabaseName("IX_PendingSecuredWrites_Site"); + + b.HasIndex("Status", "SubmittedAtUtc") + .HasDatabaseName("IX_PendingSecuredWrites_Status_Submitted"); + + b.ToTable("PendingSecuredWrites", (string)null); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Security.LdapGroupMapping", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("LdapGroupName") + .IsRequired() + .HasMaxLength(500) + .HasColumnType("nvarchar(500)"); + + b.Property("Role") + .IsRequired() + .HasMaxLength(100) + .HasColumnType("nvarchar(100)"); + + b.HasKey("Id"); + + b.HasIndex("LdapGroupName") + .IsUnique(); + + b.ToTable("LdapGroupMappings"); + + b.HasData( + new + { + Id = 1, + LdapGroupName = "SCADA-Admins", + Role = "Administrator" + }, + new + { + Id = 2, + LdapGroupName = "SCADA-Designers", + Role = "Designer" + }, + new + { + Id = 3, + LdapGroupName = "SCADA-Deploy-All", + Role = "Deployer" + }, + new + { + Id = 4, + LdapGroupName = "SCADA-Deploy-SiteA", + Role = "Deployer" + }, + new + { + Id = 5, + LdapGroupName = "SCADA-Viewers", + Role = "Viewer" + }); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Security.SiteScopeRule", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("LdapGroupMappingId") + .HasColumnType("int"); + + b.Property("SiteId") + .HasColumnType("int"); + + b.HasKey("Id"); + + b.HasIndex("SiteId"); + + b.HasIndex("LdapGroupMappingId", "SiteId") + .IsUnique(); + + b.ToTable("SiteScopeRules"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites.DataConnection", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("BackupConfiguration") + .HasMaxLength(4000) + .HasColumnType("nvarchar(4000)"); + + b.Property("FailoverRetryCount") + .ValueGeneratedOnAdd() + .HasColumnType("int") + .HasDefaultValue(3); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("PrimaryConfiguration") + .HasMaxLength(4000) + .HasColumnType("nvarchar(4000)"); + + b.Property("Protocol") + .IsRequired() + .HasMaxLength(50) + .HasColumnType("nvarchar(50)"); + + b.Property("SiteId") + .HasColumnType("int"); + + b.HasKey("Id"); + + b.HasIndex("SiteId", "Name") + .IsUnique(); + + b.ToTable("DataConnections"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites.Site", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("Description") + .HasMaxLength(2000) + .HasColumnType("nvarchar(2000)"); + + b.Property("GrpcNodeAAddress") + .HasMaxLength(500) + .HasColumnType("nvarchar(500)"); + + b.Property("GrpcNodeBAddress") + .HasMaxLength(500) + .HasColumnType("nvarchar(500)"); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("NodeAAddress") + .HasMaxLength(500) + .HasColumnType("nvarchar(500)"); + + b.Property("NodeBAddress") + .HasMaxLength(500) + .HasColumnType("nvarchar(500)"); + + b.Property("SiteIdentifier") + .IsRequired() + .HasMaxLength(100) + .HasColumnType("nvarchar(100)"); + + b.HasKey("Id"); + + b.HasIndex("Name") + .IsUnique(); + + b.HasIndex("SiteIdentifier") + .IsUnique(); + + b.ToTable("Sites"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.Template", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("Description") + .HasMaxLength(2000) + .HasColumnType("nvarchar(2000)"); + + b.Property("FolderId") + .HasColumnType("int"); + + b.Property("IsDerived") + .HasColumnType("bit"); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("OwnerCompositionId") + .HasColumnType("int"); + + b.Property("ParentTemplateId") + .HasColumnType("int"); + + b.HasKey("Id"); + + b.HasIndex("FolderId"); + + b.HasIndex("Name") + .IsUnique() + .HasFilter("[IsDerived] = 0"); + + b.HasIndex("ParentTemplateId"); + + b.ToTable("Templates"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.TemplateAlarm", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("Description") + .HasMaxLength(2000) + .HasColumnType("nvarchar(2000)"); + + b.Property("IsInherited") + .HasColumnType("bit"); + + b.Property("IsLocked") + .HasColumnType("bit"); + + b.Property("LockedInDerived") + .HasColumnType("bit"); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("OnTriggerScriptId") + .HasColumnType("int"); + + b.Property("PriorityLevel") + .HasColumnType("int"); + + b.Property("TemplateId") + .HasColumnType("int"); + + b.Property("TriggerConfiguration") + .HasMaxLength(4000) + .HasColumnType("nvarchar(4000)"); + + b.Property("TriggerType") + .IsRequired() + .HasMaxLength(50) + .HasColumnType("nvarchar(50)"); + + b.HasKey("Id"); + + b.HasIndex("TemplateId", "Name") + .IsUnique(); + + b.ToTable("TemplateAlarms"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.TemplateAttribute", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("DataSourceReference") + .HasMaxLength(500) + .HasColumnType("nvarchar(500)"); + + b.Property("DataType") + .IsRequired() + .HasMaxLength(50) + .HasColumnType("nvarchar(50)"); + + b.Property("Description") + .HasMaxLength(2000) + .HasColumnType("nvarchar(2000)"); + + b.Property("ElementDataType") + .HasMaxLength(50) + .HasColumnType("nvarchar(50)"); + + b.Property("IsInherited") + .HasColumnType("bit"); + + b.Property("IsLocked") + .HasColumnType("bit"); + + b.Property("LockedInDerived") + .HasColumnType("bit"); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("TemplateId") + .HasColumnType("int"); + + b.Property("Value") + .HasColumnType("nvarchar(max)"); + + b.HasKey("Id"); + + b.HasIndex("TemplateId", "Name") + .IsUnique(); + + b.ToTable("TemplateAttributes"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.TemplateComposition", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("ComposedTemplateId") + .HasColumnType("int"); + + b.Property("InstanceName") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("TemplateId") + .HasColumnType("int"); + + b.HasKey("Id"); + + b.HasIndex("ComposedTemplateId"); + + b.HasIndex("TemplateId", "InstanceName") + .IsUnique(); + + b.ToTable("TemplateCompositions"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.TemplateFolder", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("ParentFolderId") + .HasColumnType("int"); + + b.Property("SortOrder") + .HasColumnType("int"); + + b.HasKey("Id"); + + b.HasIndex("ParentFolderId", "Name") + .IsUnique() + .HasFilter("[ParentFolderId] IS NOT NULL"); + + b.ToTable("TemplateFolders"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.TemplateNativeAlarmSource", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("ConditionFilter") + .HasMaxLength(1000) + .HasColumnType("nvarchar(1000)"); + + b.Property("ConnectionName") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("Description") + .HasMaxLength(2000) + .HasColumnType("nvarchar(2000)"); + + b.Property("IsInherited") + .HasColumnType("bit"); + + b.Property("IsLocked") + .HasColumnType("bit"); + + b.Property("LockedInDerived") + .HasColumnType("bit"); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("SourceReference") + .IsRequired() + .HasMaxLength(1000) + .HasColumnType("nvarchar(1000)"); + + b.Property("TemplateId") + .HasColumnType("int"); + + b.HasKey("Id"); + + b.HasIndex("TemplateId", "Name") + .IsUnique(); + + b.ToTable("TemplateNativeAlarmSources"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.TemplateScript", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("Code") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("ExecutionTimeoutSeconds") + .HasColumnType("int"); + + b.Property("IsInherited") + .HasColumnType("bit"); + + b.Property("IsLocked") + .HasColumnType("bit"); + + b.Property("LockedInDerived") + .HasColumnType("bit"); + + b.Property("MinTimeBetweenRuns") + .HasColumnType("time"); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("ParameterDefinitions") + .HasMaxLength(4000) + .HasColumnType("nvarchar(4000)"); + + b.Property("ReturnDefinition") + .HasMaxLength(4000) + .HasColumnType("nvarchar(4000)"); + + b.Property("TemplateId") + .HasColumnType("int"); + + b.Property("TriggerConfiguration") + .HasMaxLength(4000) + .HasColumnType("nvarchar(4000)"); + + b.Property("TriggerType") + .HasMaxLength(50) + .HasColumnType("nvarchar(50)"); + + b.HasKey("Id"); + + b.HasIndex("TemplateId", "Name") + .IsUnique(); + + b.ToTable("TemplateScripts"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Entities.AuditLogRow", b => + { + b.Property("EventId") + .HasColumnType("uniqueidentifier"); + + b.Property("OccurredAtUtc") + .HasColumnType("datetime2"); + + b.Property("Action") + .IsRequired() + .HasMaxLength(64) + .IsUnicode(false) + .HasColumnType("varchar(64)"); + + b.Property("Actor") + .HasMaxLength(256) + .HasColumnType("nvarchar(256)"); + + b.Property("Channel") + .IsRequired() + .HasMaxLength(32) + .IsUnicode(false) + .HasColumnType("varchar(32)") + .HasColumnName("Category"); + + b.Property("CorrelationId") + .HasColumnType("uniqueidentifier"); + + b.Property("DetailsJson") + .HasColumnType("nvarchar(max)"); + + b.Property("ExecutionId") + .ValueGeneratedOnAddOrUpdate() + .HasColumnType("uniqueidentifier") + .HasComputedColumnSql("CAST(JSON_VALUE(DetailsJson,'$.executionId') AS uniqueidentifier)", true); + + b.Property("IngestedAtUtc") + .ValueGeneratedOnAddOrUpdate() + .HasColumnType("datetime2(7)") + .HasComputedColumnSql("CAST(SWITCHOFFSET(CAST(JSON_VALUE(DetailsJson,'$.ingestedAtUtc') AS datetimeoffset), 0) AS datetime2(7))", false); + + b.Property("Kind") + .IsRequired() + .ValueGeneratedOnAddOrUpdate() + .HasMaxLength(32) + .IsUnicode(false) + .HasColumnType("varchar(32)") + .HasComputedColumnSql("JSON_VALUE(DetailsJson,'$.kind')", true); + + b.Property("Outcome") + .IsRequired() + .HasMaxLength(16) + .IsUnicode(false) + .HasColumnType("varchar(16)"); + + b.Property("ParentExecutionId") + .ValueGeneratedOnAddOrUpdate() + .HasColumnType("uniqueidentifier") + .HasComputedColumnSql("CAST(JSON_VALUE(DetailsJson,'$.parentExecutionId') AS uniqueidentifier)", true); + + b.Property("SourceNode") + .HasMaxLength(64) + .IsUnicode(false) + .HasColumnType("varchar(64)"); + + b.Property("SourceSiteId") + .ValueGeneratedOnAddOrUpdate() + .HasMaxLength(64) + .IsUnicode(false) + .HasColumnType("varchar(64)") + .HasComputedColumnSql("JSON_VALUE(DetailsJson,'$.sourceSiteId')", true); + + b.Property("Status") + .IsRequired() + .ValueGeneratedOnAddOrUpdate() + .HasMaxLength(32) + .IsUnicode(false) + .HasColumnType("varchar(32)") + .HasComputedColumnSql("JSON_VALUE(DetailsJson,'$.status')", true); + + b.Property("Target") + .HasMaxLength(256) + .HasColumnType("nvarchar(256)"); + + b.HasKey("EventId", "OccurredAtUtc"); + + b.HasIndex("CorrelationId") + .HasDatabaseName("IX_AuditLog_CorrelationId") + .HasFilter("[CorrelationId] IS NOT NULL"); + + b.HasIndex("EventId") + .IsUnique() + .HasDatabaseName("UX_AuditLog_EventId"); + + b.HasIndex("ExecutionId") + .HasDatabaseName("IX_AuditLog_Execution"); + + b.HasIndex("OccurredAtUtc") + .IsDescending() + .HasDatabaseName("IX_AuditLog_OccurredAtUtc"); + + SqlServerIndexBuilderExtensions.IncludeProperties(b.HasIndex("OccurredAtUtc"), new[] { "Status" }); + + b.HasIndex("ParentExecutionId") + .HasDatabaseName("IX_AuditLog_ParentExecution"); + + b.HasIndex("SourceNode", "OccurredAtUtc") + .HasDatabaseName("IX_AuditLog_Node_Occurred"); + + b.HasIndex("SourceSiteId", "OccurredAtUtc") + .IsDescending(false, true) + .HasDatabaseName("IX_AuditLog_Site_Occurred"); + + b.HasIndex("Target", "OccurredAtUtc") + .IsDescending(false, true) + .HasDatabaseName("IX_AuditLog_Target_Occurred") + .HasFilter("[Target] IS NOT NULL"); + + b.HasIndex("Channel", "Status", "OccurredAtUtc") + .IsDescending(false, false, true) + .HasDatabaseName("IX_AuditLog_Channel_Status_Occurred"); + + b.ToTable("AuditLog", (string)null); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Deployment.DeployedConfigSnapshot", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.Instance", null) + .WithMany() + .HasForeignKey("InstanceId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Deployment.DeploymentRecord", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.Instance", null) + .WithMany() + .HasForeignKey("InstanceId") + .OnDelete(DeleteBehavior.Restrict) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Deployment.PendingDeployment", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.Instance", null) + .WithMany() + .HasForeignKey("InstanceId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.ExternalSystems.ExternalSystemMethod", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.ExternalSystems.ExternalSystemDefinition", null) + .WithMany() + .HasForeignKey("ExternalSystemDefinitionId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.Area", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.Area", null) + .WithMany("Children") + .HasForeignKey("ParentAreaId") + .OnDelete(DeleteBehavior.Restrict); + + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites.Site", null) + .WithMany() + .HasForeignKey("SiteId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.Instance", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.Area", null) + .WithMany() + .HasForeignKey("AreaId") + .OnDelete(DeleteBehavior.SetNull); + + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites.Site", null) + .WithMany() + .HasForeignKey("SiteId") + .OnDelete(DeleteBehavior.Restrict) + .IsRequired(); + + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.Template", null) + .WithMany() + .HasForeignKey("TemplateId") + .OnDelete(DeleteBehavior.Restrict) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.InstanceAlarmOverride", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.Instance", null) + .WithMany("AlarmOverrides") + .HasForeignKey("InstanceId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.InstanceAttributeOverride", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.Instance", null) + .WithMany("AttributeOverrides") + .HasForeignKey("InstanceId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.InstanceConnectionBinding", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites.DataConnection", null) + .WithMany() + .HasForeignKey("DataConnectionId") + .OnDelete(DeleteBehavior.Restrict) + .IsRequired(); + + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.Instance", null) + .WithMany("ConnectionBindings") + .HasForeignKey("InstanceId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.InstanceNativeAlarmSourceOverride", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.Instance", null) + .WithMany("NativeAlarmSourceOverrides") + .HasForeignKey("InstanceId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Notifications.NotificationRecipient", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Notifications.NotificationList", null) + .WithMany("Recipients") + .HasForeignKey("NotificationListId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Security.SiteScopeRule", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Security.LdapGroupMapping", null) + .WithMany() + .HasForeignKey("LdapGroupMappingId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites.Site", null) + .WithMany() + .HasForeignKey("SiteId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites.DataConnection", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites.Site", null) + .WithMany() + .HasForeignKey("SiteId") + .OnDelete(DeleteBehavior.Restrict) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.Template", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.TemplateFolder", null) + .WithMany() + .HasForeignKey("FolderId") + .OnDelete(DeleteBehavior.Restrict); + + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.Template", null) + .WithMany() + .HasForeignKey("ParentTemplateId") + .OnDelete(DeleteBehavior.Restrict); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.TemplateAlarm", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.Template", null) + .WithMany("Alarms") + .HasForeignKey("TemplateId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.TemplateAttribute", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.Template", null) + .WithMany("Attributes") + .HasForeignKey("TemplateId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.TemplateComposition", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.Template", null) + .WithMany() + .HasForeignKey("ComposedTemplateId") + .OnDelete(DeleteBehavior.Restrict) + .IsRequired(); + + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.Template", null) + .WithMany("Compositions") + .HasForeignKey("TemplateId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.TemplateFolder", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.TemplateFolder", null) + .WithMany() + .HasForeignKey("ParentFolderId") + .OnDelete(DeleteBehavior.Restrict); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.TemplateNativeAlarmSource", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.Template", null) + .WithMany("NativeAlarmSources") + .HasForeignKey("TemplateId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.TemplateScript", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.Template", null) + .WithMany("Scripts") + .HasForeignKey("TemplateId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.Area", b => + { + b.Navigation("Children"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.Instance", b => + { + b.Navigation("AlarmOverrides"); + + b.Navigation("AttributeOverrides"); + + b.Navigation("ConnectionBindings"); + + b.Navigation("NativeAlarmSourceOverrides"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Notifications.NotificationList", b => + { + b.Navigation("Recipients"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.Template", b => + { + b.Navigation("Alarms"); + + b.Navigation("Attributes"); + + b.Navigation("Compositions"); + + b.Navigation("NativeAlarmSources"); + + b.Navigation("Scripts"); + }); +#pragma warning restore 612, 618 + } + } +} diff --git a/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/20260814235335_AddAuditLogAndNotificationCoveringIndexes.cs b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/20260814235335_AddAuditLogAndNotificationCoveringIndexes.cs new file mode 100644 index 00000000..b09088d6 --- /dev/null +++ b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/20260814235335_AddAuditLogAndNotificationCoveringIndexes.cs @@ -0,0 +1,65 @@ +using Microsoft.EntityFrameworkCore.Migrations; + +#nullable disable + +namespace ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Migrations +{ + /// + /// Two covering indexes (arch-review WP1.4): IX_AuditLog_OccurredAtUtc gains + /// INCLUDE (Status) so the audit KPI window query (count/aggregate by Status + /// over a recent OccurredAtUtc range) can seek without a lookup back into the + /// clustered index per row; IX_Notifications_Delivered is a new filtered index + /// backing the notification outbox KPI's delivered-last-interval count. + /// + /// + /// IX_AuditLog_OccurredAtUtc is partition-aligned + /// (ON ps_AuditLog_Month(OccurredAtUtc), see CollapseAuditLogToCanonical) + /// so the monthly partition-switch purge keeps touching a single partition. EF's + /// generic DropIndex/CreateIndex migration builders have no concept of + /// partition schemes and would silently rebuild the index off [PRIMARY] instead + /// of the scheme — this migration uses raw SQL for that index (mirroring + /// CollapseAuditLogToCanonical's DDL) to preserve the alignment. The scaffolded + /// EF model itself carries no partition annotation either way (pre-existing gap, same + /// as every other AuditLog index) — only the physical DDL below matters. + /// Notifications is not partitioned, so its index uses the ordinary EF builder. + /// The filter predicate is a plain 'Delivered' literal (no N'' prefix): + /// the value is ASCII-only, and the ConfigurationDatabase repository tests run this + /// same filter as DDL against the SQLite provider via EnsureCreated(), which + /// rejects the T-SQL national-string-literal syntax outright. + /// + public partial class AddAuditLogAndNotificationCoveringIndexes : Migration + { + /// + protected override void Up(MigrationBuilder migrationBuilder) + { + migrationBuilder.CreateIndex( + name: "IX_Notifications_Delivered", + table: "Notifications", + column: "DeliveredAt", + filter: "[Status] = 'Delivered'"); + + migrationBuilder.Sql(@" +DROP INDEX IX_AuditLog_OccurredAtUtc ON dbo.AuditLog; + +CREATE NONCLUSTERED INDEX IX_AuditLog_OccurredAtUtc +ON dbo.AuditLog (OccurredAtUtc DESC) +INCLUDE (Status) +ON ps_AuditLog_Month(OccurredAtUtc);"); + } + + /// + protected override void Down(MigrationBuilder migrationBuilder) + { + migrationBuilder.DropIndex( + name: "IX_Notifications_Delivered", + table: "Notifications"); + + migrationBuilder.Sql(@" +DROP INDEX IX_AuditLog_OccurredAtUtc ON dbo.AuditLog; + +CREATE NONCLUSTERED INDEX IX_AuditLog_OccurredAtUtc +ON dbo.AuditLog (OccurredAtUtc DESC) +ON ps_AuditLog_Month(OccurredAtUtc);"); + } + } +} diff --git a/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/ScadaBridgeDbContextModelSnapshot.cs b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/ScadaBridgeDbContextModelSnapshot.cs index dccfa59e..a4ba82be 100644 --- a/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/ScadaBridgeDbContextModelSnapshot.cs +++ b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/ScadaBridgeDbContextModelSnapshot.cs @@ -903,6 +903,10 @@ namespace ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Migrations b.HasKey("NotificationId"); + b.HasIndex("DeliveredAt") + .HasDatabaseName("IX_Notifications_Delivered") + .HasFilter("[Status] = 'Delivered'"); + b.HasIndex("SourceSiteId", "CreatedAt"); b.HasIndex("Status", "NextAttemptAt"); @@ -1813,6 +1817,8 @@ namespace ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Migrations .IsDescending() .HasDatabaseName("IX_AuditLog_OccurredAtUtc"); + SqlServerIndexBuilderExtensions.IncludeProperties(b.HasIndex("OccurredAtUtc"), new[] { "Status" }); + b.HasIndex("ParentExecutionId") .HasDatabaseName("IX_AuditLog_ParentExecution"); diff --git a/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/NotificationOutboxRepository.cs b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/NotificationOutboxRepository.cs index bc3690a2..6754bfa2 100644 --- a/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/NotificationOutboxRepository.cs +++ b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/NotificationOutboxRepository.cs @@ -230,9 +230,30 @@ VALUES /// public async Task DeleteTerminalOlderThanAsync(DateTimeOffset cutoff, CancellationToken cancellationToken = default) { - return await _context.Notifications + // Time-sliced batches: each DELETE covers at most one hour of terminal rows, + // capping the lock/log footprint per statement — mirrors + // KpiHistoryRepository.PurgeOlderThanAsync (arch-review WP1.4; an unbounded + // single-statement ExecuteDeleteAsync would otherwise turn a long-idle catch-up + // of the default 365-day window into one giant transaction against + // dbo.Notifications). Pure EF LINQ (no raw SQL) so this stays portable across + // the SQL Server production provider and the SQLite provider the repository + // tests run against. + var total = 0; + var floor = await _context.Notifications .Where(n => TerminalStatuses.Contains(n.Status) && n.CreatedAt < cutoff) - .ExecuteDeleteAsync(cancellationToken); + .MinAsync(n => (DateTimeOffset?)n.CreatedAt, cancellationToken); + while (floor is not null && floor < cutoff) + { + var ceiling = floor.Value.AddHours(1) < cutoff ? floor.Value.AddHours(1) : cutoff; + total += await _context.Notifications + .Where(n => TerminalStatuses.Contains(n.Status) && n.CreatedAt < ceiling) + .ExecuteDeleteAsync(cancellationToken); + floor = await _context.Notifications + .Where(n => TerminalStatuses.Contains(n.Status) && n.CreatedAt < cutoff) + .MinAsync(n => (DateTimeOffset?)n.CreatedAt, cancellationToken); + } + + return total; } /// diff --git a/src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardSchema.cs b/src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardSchema.cs index 70a8c270..cc533206 100644 --- a/src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardSchema.cs +++ b/src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardSchema.cs @@ -94,6 +94,23 @@ public static class StoreAndForwardSchema "CREATE INDEX IF NOT EXISTS idx_sf_messages_status_due ON sf_messages(status, last_attempt_at_ms)"; dueIndex.ExecuteNonQuery(); } + + // Covering index for GetMessagesForRetryAsync's ORDER BY created_at ASC. + // idx_sf_messages_status_due (above) matches the status filter and the + // last_attempt_at_ms half of the due predicate, but its column order does + // NOT match the query's "ORDER BY created_at ASC", so SQLite still needs a + // sort/scan step to satisfy that ordering on a large due-sweep. This index's + // leading (status, created_at) column order lets the planner walk matching + // rows already in created_at order and stop at LIMIT without touching + // non-pending rows or re-sorting (arch-review WP1.4). idx_sf_messages_status_due + // is left in place — it still backs status+due-time lookups that don't order by + // created_at. + using (var orderIndex = connection.CreateCommand()) + { + orderIndex.CommandText = + "CREATE INDEX IF NOT EXISTS idx_sf_messages_status_created ON sf_messages(status, created_at)"; + orderIndex.ExecuteNonQuery(); + } } /// From 34a3f4bb6930863fbc51c8493de997aed81cc2cd Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Fri, 14 Aug 2026 19:57:08 -0400 Subject: [PATCH 05/31] =?UTF-8?q?fix(comms):=20reconnect=20on=20graceful?= =?UTF-8?q?=20stream=20completion=20=E2=80=94=20kills=20the=204h=20silent?= =?UTF-8?q?=20stream=20death?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/requirements/Component-Communication.md | 1 + .../Actors/DebugStreamBridgeActor.cs | 86 ++++++++++++- .../Actors/SiteAlarmAggregatorActor.cs | 81 +++++++++++- .../Grpc/SiteStreamGrpcClient.cs | 113 +++++++++++++---- .../Grpc/DebugStreamBridgeActorTests.cs | 66 +++++++++- .../Grpc/SiteAlarmAggregatorActorTests.cs | 84 ++++++++++++- .../Grpc/SiteStreamGrpcClientTests.cs | 116 +++++++++++++++++- .../SiteAlarmLiveCacheServiceTests.cs | 2 +- .../Grpc/SiteAlarmStreamEndToEndTests.cs | 3 +- 9 files changed, 507 insertions(+), 45 deletions(-) diff --git a/docs/requirements/Component-Communication.md b/docs/requirements/Component-Communication.md index ed8dd036..cee3f326 100644 --- a/docs/requirements/Component-Communication.md +++ b/docs/requirements/Component-Communication.md @@ -343,6 +343,7 @@ Disconnect is detected at the **transport layer**, never via an application-leve - **In-flight command/control messages (gRPC call + deadline)**: When a connection drops while a request is in flight (e.g., a deployment sent but no response received), the gRPC call fails or hits its deadline and the caller receives a failure. There is **no automatic retry or buffering at central** — the engineer sees the failure in the UI and re-initiates the action. This is consistent with the design principle that central does not buffer messages. An in-progress deployment whose round-trip exceeds the timeout (default 120 s at `CommunicationService.DeployInstanceAsync`) surfaces as `DeploymentStatus.Failed` to the caller. - **Debug streams (gRPC)**: Any gRPC stream interruption is detected by the HTTP/2 keepalive PING (~25 s) and triggers reconnection logic in the `DebugStreamBridgeActor`. The bridge actor attempts to reconnect to the other site node endpoint (NodeB if NodeA failed, or vice versa), with up to 3 retries and 5-second backoff. If all retries fail, the consumer is notified via `OnStreamTerminated` and the bridge actor is stopped. Events during the reconnection gap are lost (acceptable for real-time debug view). On successful reconnection, the consumer can request a fresh snapshot to re-sync state. +- **Graceful end of stream is NOT a fault, but it IS a reconnect trigger.** When the site's `GrpcMaxStreamLifetime` (4 h) elapses — and on a graceful site shutdown — the server ends the RPC with status **OK**, so the client's read loop simply finishes with no exception. `SiteStreamGrpcClient` reports that through a dedicated `onCompleted` callback, distinct from `onError`, and both consuming actors treat it as a reconnect trigger that **does not spend (or refund) the error-retry budget** and does **not** flip the node — the peer that closed cleanly is healthy. `SiteAlarmAggregatorActor` marks the stream down and lets the periodic reconcile tick reopen it (which also re-seeds); `DebugStreamBridgeActor` reopens on the same endpoint after `ReconnectDelay`. Both stamp the stream generation on the callback so a completion racing out of an already-replaced stream is ignored, and both observe the subscription `Task` itself so a fault escaping the subscribe call can never go unobserved. Without the `onCompleted` leg an OK completion was invisible to the reconnect logic and the stream stayed silently dead until the central process restarted. ## Failover Behavior diff --git a/src/ZB.MOM.WW.ScadaBridge.Communication/Actors/DebugStreamBridgeActor.cs b/src/ZB.MOM.WW.ScadaBridge.Communication/Actors/DebugStreamBridgeActor.cs index 7496b053..e0b26082 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Communication/Actors/DebugStreamBridgeActor.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Communication/Actors/DebugStreamBridgeActor.cs @@ -63,6 +63,14 @@ public class DebugStreamBridgeActor : ReceiveActor, IWithTimers private bool _stopped; private CancellationTokenSource? _grpcCts; + /// + /// Monotonic stream generation stamped on each opened gRPC stream and echoed back on its + /// error/completion callbacks: a late callback raced out of a previous (cancelled) stream + /// carries a stale generation and is ignored, so it can neither burn retry budget nor + /// open a duplicate stream. Mirrors SiteAlarmAggregatorActor. Actor-thread only. + /// + private int _streamGeneration; + /// /// Phase flag. until the initial /// has been delivered and the pre-snapshot buffer @@ -198,10 +206,33 @@ public class DebugStreamBridgeActor : ReceiveActor, IWithTimers // gRPC stream error — attempt reconnection Receive(msg => { + // Ignore a late error raced out of a previous (cancelled) stream: it must not + // burn retry budget or flip the node a second time. + if (msg.Generation != _streamGeneration) + { + _log.Debug("Ignoring stale gRPC error from stream generation {0} (current {1})", + msg.Generation, _streamGeneration); + return; + } _log.Warning("gRPC stream error for {0}: {1}", _instanceUniqueName, msg.Exception.Message); HandleGrpcError(); }); + // gRPC stream ended GRACEFULLY (server status OK) — the site's 4h max stream + // lifetime elapsing or a graceful site shutdown. Not a fault: reopen on the SAME + // node without spending retry budget. Without this the session went silently deaf. + Receive(msg => + { + if (_stopped) return; + if (msg.Generation != _streamGeneration) + { + _log.Debug("Ignoring stale gRPC completion from stream generation {0} (current {1})", + msg.Generation, _streamGeneration); + return; + } + HandleGrpcCompleted(); + }); + // Scheduled reconnection Receive(_ => OpenGrpcStream()); @@ -424,20 +455,57 @@ public class DebugStreamBridgeActor : ReceiveActor, IWithTimers // retry budget is recovered. Cancelled by HandleGrpcError. Timers.StartSingleTimer(StabilityTimerKey, new GrpcStreamStable(), StabilityWindow); + var generation = ++_streamGeneration; var client = _grpcFactory.GetOrCreate(_siteIdentifier, endpoint); var self = Self; var ct = _grpcCts.Token; - // Launch as background task — onEvent and onError marshal back to actor via Tell + // Launch as background task — the callbacks marshal back to the actor via Tell. + // The task itself is observed below: a fault escaping SubscribeAsync would otherwise + // leave the session waiting on a stream that does not exist, exception unobserved. Task.Run(async () => { await client.SubscribeAsync( _correlationId, _instanceUniqueName, evt => self.Tell(evt), - ex => self.Tell(new GrpcStreamError(ex)), + ex => self.Tell(new GrpcStreamError(ex, generation)), + () => self.Tell(new GrpcStreamCompleted(generation)), ct); - }, ct); + }, ct).ContinueWith(t => + { + if (t.IsFaulted) + self.Tell(new GrpcStreamError(t.Exception!.GetBaseException(), generation)); + else if (t.IsCanceled && !ct.IsCancellationRequested) + self.Tell(new GrpcStreamCompleted(generation)); + // RanToCompletion: SubscribeAsync already reported its own outcome. + }, TaskContinuationOptions.ExecuteSynchronously); + } + + /// + /// Handles a graceful end of stream (server status OK). The stream simply expired or the + /// site shut down cleanly, so the retry budget is left untouched and the endpoint is not + /// flipped; the reopen is scheduled through the existing reconnect timer, which also + /// rate-limits a pathological site that keeps closing streams immediately. + /// + private void HandleGrpcCompleted() + { + // The stream is gone, so its armed stability timer must not later "recover" a + // budget that its successor has since spent. + Timers.Cancel(StabilityTimerKey); + + _log.Info("gRPC stream for {0} completed gracefully (server end of stream); reopening", + _instanceUniqueName); + + // Release the site-side relay for the finished stream before reopening, so the site + // is not left with a zombie relay actor for this correlation id. + _grpcCts?.Cancel(); + _grpcCts?.Dispose(); + _grpcCts = null; + _grpcFactory.TryGet(_siteIdentifier, _useNodeA ? _grpcNodeAAddress : _grpcNodeBAddress) + ?.Unsubscribe(_correlationId); + + Timers.StartSingleTimer(ReconnectTimerKey, new ReconnectGrpcStream(), ReconnectDelay); } private void HandleGrpcError() @@ -512,9 +580,17 @@ public class DebugStreamBridgeActor : ReceiveActor, IWithTimers public record StopDebugStream; /// -/// Internal message indicating a gRPC stream error occurred. +/// Internal message indicating a gRPC stream error occurred, stamped with the stream +/// generation it came from so a late error out of a cancelled stream can be ignored. /// -internal record GrpcStreamError(Exception Exception); +internal record GrpcStreamError(Exception Exception, int Generation); + +/// +/// Internal message indicating the gRPC stream ended gracefully (server status OK — the +/// site's max stream lifetime elapsed, or the site shut down cleanly), stamped with its +/// stream generation. +/// +internal record GrpcStreamCompleted(int Generation); /// /// Internal message to trigger gRPC stream reconnection. diff --git a/src/ZB.MOM.WW.ScadaBridge.Communication/Actors/SiteAlarmAggregatorActor.cs b/src/ZB.MOM.WW.ScadaBridge.Communication/Actors/SiteAlarmAggregatorActor.cs index a320479a..94fecd5e 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Communication/Actors/SiteAlarmAggregatorActor.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Communication/Actors/SiteAlarmAggregatorActor.cs @@ -78,12 +78,23 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers private bool _stopped; /// - /// True once the live gRPC stream has been given up (retry budget exhausted). Reconcile - /// snapshots keep serving in the meantime; the next reconcile tick self-heals the stream - /// by resetting the retry budget and reopening it, so a sustained site outage does not - /// permanently drop the live feed. Actor-thread only. + /// True while there is no live gRPC stream — either it was given up (retry budget + /// exhausted) or it ended gracefully (server status OK at the site's max stream + /// lifetime). Reconcile snapshots keep serving in the meantime; the next reconcile tick + /// self-heals the stream by resetting the retry budget and reopening it, so neither a + /// sustained site outage nor a routine 4h stream expiry permanently drops the live feed. + /// Actor-thread only. /// private bool _streamDown; + + /// + /// Why the stream is down: true = the retry budget was exhausted, so the + /// self-healing reopen must also reset it; false = it ended gracefully, and the + /// budget — which a completion neither spends nor refunds — is carried across the reopen + /// untouched (a stream flapping between faults and clean closes must still trip + /// MaxRetries). Actor-thread only. + /// + private bool _retryBudgetExhausted; private CancellationTokenSource? _grpcCts; private CancellationTokenSource? _lifetimeCts; @@ -218,6 +229,22 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers HandleGrpcError(); }); + // gRPC stream ended GRACEFULLY (server status OK) — the site's 4h max stream + // lifetime elapsing or a graceful site shutdown. Not a fault: the stream is marked + // down so the reconcile tick reopens it (which also re-seeds), but the retry budget + // is untouched and the node is not flipped. Same generation fence as the error path. + Receive(msg => + { + if (_stopped) return; + if (msg.Generation != _streamGeneration) + { + _log.Debug("Ignoring stale gRPC completion from stream generation {0} (current {1})", + msg.Generation, _streamGeneration); + return; + } + HandleGrpcCompleted(); + }); + Receive(_ => OpenGrpcStream()); // Owning service asks us to stop (last viewer left + linger elapsed). @@ -279,7 +306,11 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers if (_streamDown) { _log.Info("Site-alarm gRPC stream for {0} was down; reopening on reconcile tick", _siteIdentifier); - _retryCount = 0; + if (_retryBudgetExhausted) + { + _retryBudgetExhausted = false; + _retryCount = 0; + } // Telemetry: a reconcile-driven reopen after the stream was given up is a reconnect. ScadaBridgeTelemetry.RecordLiveAlarmStreamReconnect(); OpenGrpcStream(); @@ -514,14 +545,26 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers var self = Self; var ct = _grpcCts.Token; + // The subscription task itself is observed (below): a fault escaping + // SubscribeSiteAsync — or a Task.Run that never started because ct was already + // cancelled — would otherwise leave the actor waiting on a stream that does not + // exist, with the exception silently unobserved. Task.Run(async () => { await client.SubscribeSiteAsync( _correlationId, alarm => self.Tell(alarm), ex => self.Tell(new GrpcAlarmStreamError(ex, generation)), + () => self.Tell(new GrpcAlarmStreamCompleted(generation)), ct); - }, ct); + }, ct).ContinueWith(t => + { + if (t.IsFaulted) + self.Tell(new GrpcAlarmStreamError(t.Exception!.GetBaseException(), generation)); + else if (t.IsCanceled && !ct.IsCancellationRequested) + self.Tell(new GrpcAlarmStreamCompleted(generation)); + // RanToCompletion: SubscribeSiteAsync already reported its own outcome. + }, TaskContinuationOptions.ExecuteSynchronously); } private void HandleGrpcError() @@ -543,6 +586,7 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers "reconcile snapshots continue and the next reconcile tick will retry the stream", _siteIdentifier, MaxRetries); _streamDown = true; + _retryBudgetExhausted = true; CleanupGrpc(); return; } @@ -569,6 +613,26 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers Timers.StartSingleTimer(ReconnectTimerKey, new ReconnectAlarmStream(), _reconnectDelay); } + /// + /// Handles a graceful end of stream. The stream is torn down and left down for the + /// reconcile tick to reopen — deliberately NOT reopened inline, so a site that keeps + /// completing streams immediately can never spin this actor into a hot reconnect loop. + /// The retry budget is neither spent nor reset here (completion is not a fault), and the + /// endpoint is not flipped: the node that just closed a stream cleanly is healthy. + /// + private void HandleGrpcCompleted() + { + // The stream is gone, so its armed stability timer must not later "recover" a + // budget that its successor has since spent. + Timers.Cancel(StabilityTimerKey); + + _log.Info("Site-alarm gRPC stream for {0} completed gracefully (server end of stream); " + + "reopening on the next reconcile tick", _siteIdentifier); + + _streamDown = true; + CleanupGrpc(); + } + private void CleanupGrpc() { _grpcCts?.Cancel(); @@ -619,6 +683,11 @@ internal sealed record PublishCoalesced; /// generation it came from so a late error from a cancelled stream can be ignored (N7.2). internal sealed record GrpcAlarmStreamError(Exception Exception, int Generation); +/// Internal: the site-alarm gRPC stream ended gracefully (server status OK — max +/// stream lifetime or site shutdown), stamped with its stream generation so a late completion +/// from a cancelled stream can be ignored. +internal sealed record GrpcAlarmStreamCompleted(int Generation); + /// Internal: reconnect the site-alarm gRPC stream (flip node). internal sealed record ReconnectAlarmStream; diff --git a/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcClient.cs b/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcClient.cs index 52584b40..5763c6c4 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcClient.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcClient.cs @@ -167,13 +167,19 @@ public class SiteStreamGrpcClient : IAsyncDisposable, IDisposable /// /// Opens a server-streaming subscription for a specific instance. /// This is a long-running async method; the caller launches it as a background task. - /// The callback delivers domain events, and - /// lets the caller handle reconnection. + /// The callback delivers domain events, + /// lets the caller handle reconnection after a fault, and + /// reports a graceful end of stream. /// /// Unique identifier for this subscription. /// Unique name of the instance to subscribe to. /// Callback invoked for each domain event received from the stream. /// Callback invoked when the subscription encounters an error. + /// + /// Callback invoked when the server ended the RPC with OK — the site's max stream + /// lifetime elapsing or a graceful site shutdown. Mutually exclusive with + /// ; see . + /// /// Cancellation token to stop the subscription. /// A task that represents the asynchronous operation. public virtual async Task SubscribeAsync( @@ -181,6 +187,7 @@ public class SiteStreamGrpcClient : IAsyncDisposable, IDisposable string instanceUniqueName, Action onEvent, Action onError, + Action onCompleted, CancellationToken ct) { if (_client is null) @@ -195,30 +202,18 @@ public class SiteStreamGrpcClient : IAsyncDisposable, IDisposable InstanceUniqueName = instanceUniqueName }; - try - { - using var call = _client.SubscribeInstance(request, cancellationToken: cts.Token); - - await foreach (var evt in call.ResponseStream.ReadAllAsync(cts.Token)) + await ConsumeStreamAsync( + correlationId, + cts, + () => _client.SubscribeInstance(request, cancellationToken: cts.Token), + evt => { var domainEvent = ConvertToDomainEvent(evt); if (domainEvent != null) onEvent(domainEvent); - } - } - catch (RpcException ex) when (ex.StatusCode == StatusCode.Cancelled) - { - // Normal cancellation — not an error - } - catch (Exception ex) - { - onError(ex); - } - finally - { - // Remove only our own entry -- a racing reconnect may already own the slot. - RemoveSubscription(correlationId, cts); - } + }, + onError, + onCompleted); } /// @@ -244,12 +239,18 @@ public class SiteStreamGrpcClient : IAsyncDisposable, IDisposable /// Unique identifier for this subscription. /// Callback invoked for each alarm delta received from the site-wide stream. /// Callback invoked when the subscription encounters an error. + /// + /// Callback invoked when the server ended the RPC with OK — the site's max stream + /// lifetime elapsing or a graceful site shutdown. Mutually exclusive with + /// ; see . + /// /// Cancellation token to stop the subscription. /// A task that represents the asynchronous operation. public virtual async Task SubscribeSiteAsync( string correlationId, Action onAlarmEvent, Action onError, + Action onCompleted, CancellationToken ct) { if (_client is null) @@ -263,21 +264,73 @@ public class SiteStreamGrpcClient : IAsyncDisposable, IDisposable CorrelationId = correlationId }; - try - { - using var call = _client.SubscribeSite(request, cancellationToken: cts.Token); - - await foreach (var evt in call.ResponseStream.ReadAllAsync(cts.Token)) + await ConsumeStreamAsync( + correlationId, + cts, + () => _client.SubscribeSite(request, cancellationToken: cts.Token), + evt => { // Site-wide stream is alarm-only by contract; defensively ignore anything else. if (ConvertToAlarmEvent(evt) is { } alarm) onAlarmEvent(alarm); + }, + onError, + onCompleted); + } + + /// + /// Drains a server-streaming call to its end, classifying the outcome into exactly one + /// of three terminations: a fault (), a graceful server-side + /// end of stream (), or our own cancellation (neither). + /// + /// The graceful case is the one that matters operationally: the site server caps every + /// stream at CommunicationOptions.GrpcMaxStreamLifetime (4 h) and, when that + /// elapses, ends the RPC with status OK. The client-side await foreach then simply + /// finishes, so before onCompleted existed the consuming actor observed nothing at + /// all and the stream stayed silently dead until the process restarted. + /// + /// Internal so the completion/fault classification can be unit-tested with a fake + /// and no live channel. + /// + /// Unique identifier for this subscription. + /// The subscription's linked token source (owned by the caller). + /// + /// Opens the server-streaming call. Invoked inside the guarded region so a throw while + /// opening is reported through like any other stream fault. + /// + /// Invoked per wire event. + /// Invoked once if the stream faulted. + /// Invoked once if the server ended the stream with OK. + /// A task that completes when the stream has ended and its outcome been reported. + internal async Task ConsumeStreamAsync( + string correlationId, + CancellationTokenSource cts, + Func> openCall, + Action onEvent, + Action onError, + Action onCompleted) + { + var completedGracefully = false; + try + { + using (var call = openCall()) + { + await foreach (var evt in call.ResponseStream.ReadAllAsync(cts.Token)) + { + onEvent(evt); + } } + + completedGracefully = true; } catch (RpcException ex) when (ex.StatusCode == StatusCode.Cancelled) { // Normal cancellation — not an error } + catch (OperationCanceledException) when (cts.IsCancellationRequested) + { + // Our own Unsubscribe/reconnect cancelled the read — not an error either. + } catch (Exception ex) { onError(ex); @@ -287,6 +340,12 @@ public class SiteStreamGrpcClient : IAsyncDisposable, IDisposable // Remove only our own entry -- a racing reconnect may already own the slot. RemoveSubscription(correlationId, cts); } + + // Raised outside the try so a throwing callback is not misreported as a stream + // fault. Suppressed when our own token was cancelled as the stream ended: that is + // a teardown, and the caller has already moved on to a newer stream. + if (completedGracefully && !cts.IsCancellationRequested) + onCompleted(); } /// diff --git a/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/DebugStreamBridgeActorTests.cs b/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/DebugStreamBridgeActorTests.cs index 7f79e3b7..c0832f16 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/DebugStreamBridgeActorTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/DebugStreamBridgeActorTests.cs @@ -400,6 +400,67 @@ public class DebugStreamBridgeActorTests : TestKit Assert.Equal("corr-1", factory.ClientFor(GrpcNodeB).SubscribeCalls[0].CorrelationId); } + // ── WP1.1: graceful (status OK) stream completion is a reconnect trigger ── + + [Fact] + public void On_GracefulStreamCompletion_Reopens_On_The_Same_Node() + { + // The site caps every stream at GrpcMaxStreamLifetime and then ends the RPC with + // OK. Pre-fix the client's read loop just finished, the actor was told nothing, and + // the debug session went silently deaf for the rest of its life. + var (_, factory) = CreateBridgeWithTrackingFactory(); + AwaitCondition(() => factory.ClientFor(GrpcNodeA).SubscribeCalls.Count == 1, + TimeSpan.FromSeconds(3)); + + factory.ClientFor(GrpcNodeA).SubscribeCalls[0].OnCompleted(); + + // Reopened on the SAME node — a clean close is not a fault, so there is nothing to + // fail over from — and the finished stream is released, not left zombie. + AwaitCondition(() => factory.ClientFor(GrpcNodeA).SubscribeCalls.Count == 2, + TimeSpan.FromSeconds(5)); + Assert.Empty(factory.ClientFor(GrpcNodeB).SubscribeCalls); + Assert.Contains("corr-1", factory.ClientFor(GrpcNodeA).UnsubscribedCorrelationIds); + } + + [Fact] + public void RepeatedGracefulCompletions_DoNotConsume_TheErrorRetryBudget() + { + // Five completions — two more than MaxRetries. Had completion been routed through + // the error path the session would have terminated on the fourth. + var ctx = CreateBridgeActor(); + ctx.CommProbe.ExpectMsg(); + AwaitCondition(() => ctx.MockGrpcClient.SubscribeCalls.Count == 1, TimeSpan.FromSeconds(3)); + + for (var i = 1; i <= 5; i++) + { + ctx.MockGrpcClient.SubscribeCalls[i - 1].OnCompleted(); + var expected = i + 1; + AwaitCondition(() => ctx.MockGrpcClient.SubscribeCalls.Count == expected, + TimeSpan.FromSeconds(5)); + } + + Assert.False(ctx.TerminatedFlag[0]); + } + + [Fact] + public void LateCompletionFromAPreviousStreamGeneration_IsIgnored() + { + var (_, factory) = CreateBridgeWithTrackingFactory(); + AwaitCondition(() => factory.ClientFor(GrpcNodeA).SubscribeCalls.Count == 1, + TimeSpan.FromSeconds(3)); + + var firstSub = factory.ClientFor(GrpcNodeA).SubscribeCalls[0]; + firstSub.OnError(new Exception("NodeA down")); // gen 1 dies → gen 2 opens on NodeB + AwaitCondition(() => factory.ClientFor(GrpcNodeB).SubscribeCalls.Count == 1, + TimeSpan.FromSeconds(5)); + + // A completion racing out of the dead gen-1 stream must not tear down the live one. + firstSub.OnCompleted(); + Thread.Sleep(300); + Assert.DoesNotContain("corr-1", factory.ClientFor(GrpcNodeB).UnsubscribedCorrelationIds); + Assert.Single(factory.ClientFor(GrpcNodeB).SubscribeCalls); + } + // ── Task 6 (arch review 02, High): teardown/failover unsubscribe is endpoint-safe ── // Both paths use TryGet, never GetOrCreate, so cleanup can never open a fresh // channel or (with (site,endpoint) keying) touch another session's channel. @@ -894,9 +955,11 @@ internal class MockSiteStreamGrpcClient : SiteStreamGrpcClient string instanceUniqueName, Action onEvent, Action onError, + Action onCompleted, CancellationToken ct) { - var subscription = new MockSubscription(correlationId, instanceUniqueName, onEvent, onError, ct); + var subscription = new MockSubscription( + correlationId, instanceUniqueName, onEvent, onError, onCompleted, ct); lock (_lock) { _subscribeCalls.Add(subscription); } // Return a task that completes when cancelled (simulates long-running stream) @@ -916,6 +979,7 @@ internal record MockSubscription( string InstanceUniqueName, Action OnEvent, Action OnError, + Action OnCompleted, CancellationToken CancellationToken); /// diff --git a/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/SiteAlarmAggregatorActorTests.cs b/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/SiteAlarmAggregatorActorTests.cs index 35c09662..98aa76a5 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/SiteAlarmAggregatorActorTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/SiteAlarmAggregatorActorTests.cs @@ -93,7 +93,8 @@ public class SiteAlarmAggregatorActorTests : TestKit } private sealed record SiteSub( - string CorrelationId, Action OnAlarm, Action OnError, CancellationToken Ct); + string CorrelationId, Action OnAlarm, Action OnError, + Action OnCompleted, CancellationToken Ct); private sealed class MockSiteAlarmStreamClient : SiteStreamGrpcClient { @@ -107,9 +108,10 @@ public class SiteAlarmAggregatorActorTests : TestKit public MockSiteAlarmStreamClient() : base() { } public override Task SubscribeSiteAsync( - string correlationId, Action onAlarmEvent, Action onError, CancellationToken ct) + string correlationId, Action onAlarmEvent, Action onError, + Action onCompleted, CancellationToken ct) { - lock (_lock) { _subs.Add(new SiteSub(correlationId, onAlarmEvent, onError, ct)); } + lock (_lock) { _subs.Add(new SiteSub(correlationId, onAlarmEvent, onError, onCompleted, ct)); } var tcs = new TaskCompletionSource(); ct.Register(() => tcs.TrySetResult()); return tcs.Task; // never completes until cancelled (simulates a live stream) @@ -445,6 +447,82 @@ public class SiteAlarmAggregatorActorTests : TestKit Assert.Equal(2, TotalSubs()); } + // ── WP1.1: graceful (status OK) stream completion is a reconnect trigger ── + + [Fact] + public void GracefulStreamCompletion_ReopensOnReconcileTick_OnTheSameNode() + { + // The site ends every stream with OK at its 4h max lifetime. Pre-fix the client's + // read loop just finished, nothing was told to the actor, and the site's alarm feed + // stayed silently dead until the central node restarted. + var (_, seed, _, factory) = CreateActor(reconcileInterval: TimeSpan.FromMilliseconds(300)); + AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3)); + AwaitCondition(() => factory.ClientFor(GrpcNodeA).Subs.Count == 1, TimeSpan.FromSeconds(3)); + seed.CompleteNext(); + + factory.ClientFor(GrpcNodeA).Subs[0].OnCompleted(); + + // Reopened by the reconcile tick, on the SAME node — a clean close is not a fault, + // so there is nothing to fail over from. + AwaitCondition(() => factory.ClientFor(GrpcNodeA).Subs.Count == 2, TimeSpan.FromSeconds(3)); + Assert.Empty(factory.ClientFor(GrpcNodeB).Subs); + // The finished stream was released, so the site keeps no zombie relay actor. + Assert.Contains("corr-1", factory.ClientFor(GrpcNodeA).Unsubscribed); + } + + [Fact] + public void GracefulStreamCompletion_NeitherSpendsNorRefunds_TheErrorRetryBudget() + { + // Reconcile is far away; reopens are driven explicitly so the budget can be observed. + var (actor, seed, _, factory) = CreateActor(reconcileInterval: TimeSpan.FromMinutes(10)); + AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3)); + AwaitCondition(() => factory.ClientFor(GrpcNodeA).Subs.Count == 1, TimeSpan.FromSeconds(3)); + seed.CompleteNext(); + + // Spend the whole error budget (MaxRetries = 3), each error flipping the node. + factory.ClientFor(GrpcNodeA).Subs[0].OnError(new Exception("1")); + AwaitCondition(() => factory.ClientFor(GrpcNodeB).Subs.Count == 1, TimeSpan.FromSeconds(5)); + factory.ClientFor(GrpcNodeB).Subs[0].OnError(new Exception("2")); + AwaitCondition(() => factory.ClientFor(GrpcNodeA).Subs.Count == 2, TimeSpan.FromSeconds(5)); + factory.ClientFor(GrpcNodeA).Subs[1].OnError(new Exception("3")); + AwaitCondition(() => factory.ClientFor(GrpcNodeB).Subs.Count == 2, TimeSpan.FromSeconds(5)); + + // A graceful completion reopens without flipping the node (budget not spent) … + factory.ClientFor(GrpcNodeB).Subs[1].OnCompleted(); + actor.Tell(new RunReconcile()); + AwaitCondition(() => factory.ClientFor(GrpcNodeB).Subs.Count == 3, TimeSpan.FromSeconds(5)); + Assert.Equal(2, factory.ClientFor(GrpcNodeA).Subs.Count); + + int TotalSubs() => factory.ClientFor(GrpcNodeA).Subs.Count + factory.ClientFor(GrpcNodeB).Subs.Count; + var before = TotalSubs(); + + // … and without refunding it either: the next error is the 4th, so it exceeds + // MaxRetries and the stream is given up rather than reconnected. + factory.ClientFor(GrpcNodeB).Subs[2].OnError(new Exception("4")); + Thread.Sleep(400); + Assert.Equal(before, TotalSubs()); + } + + [Fact] + public void LateCompletionFromAPreviousStreamGeneration_IsIgnored() + { + var (_, seed, _, factory) = CreateActor(reconcileInterval: TimeSpan.FromMinutes(10)); + AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3)); + AwaitCondition(() => factory.ClientFor(GrpcNodeA).Subs.Count == 1, TimeSpan.FromSeconds(3)); + seed.CompleteNext(); + + var firstSub = factory.ClientFor(GrpcNodeA).Subs.Single(); + firstSub.OnError(new Exception("real fault")); // gen 1 dies → gen 2 opens on NodeB + AwaitCondition(() => factory.ClientFor(GrpcNodeB).Subs.Count == 1, TimeSpan.FromSeconds(5)); + + // A completion racing out of the dead gen-1 stream must not tear down the live + // gen-2 stream (which would go deltaless until the next reconcile tick). + firstSub.OnCompleted(); + Thread.Sleep(300); + Assert.DoesNotContain("corr-1", factory.ClientFor(GrpcNodeB).Unsubscribed); + Assert.Single(factory.ClientFor(GrpcNodeB).Subs); + } + // ── R2 T10: live-delta publish coalescing (N6) ── [Fact] diff --git a/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/SiteStreamGrpcClientTests.cs b/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/SiteStreamGrpcClientTests.cs index 34b8563b..cb21d41b 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/SiteStreamGrpcClientTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/SiteStreamGrpcClientTests.cs @@ -1,4 +1,5 @@ using Google.Protobuf.WellKnownTypes; +using Grpc.Core; using ZB.MOM.WW.ScadaBridge.Communication.Grpc; using ZB.MOM.WW.ScadaBridge.Commons.Messages.Streaming; using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; @@ -277,7 +278,120 @@ public class SiteStreamGrpcClientTests var client = SiteStreamGrpcClient.CreateForTesting(); await Assert.ThrowsAsync(() => - client.SubscribeSiteAsync("corr", _ => { }, _ => { }, CancellationToken.None)); + client.SubscribeSiteAsync("corr", _ => { }, _ => { }, () => { }, CancellationToken.None)); + } + + // --- WP1.1: graceful (status OK) stream completion is reported, not swallowed --- + + [Fact] + public async Task ConsumeStream_ServerEndsStreamWithOk_InvokesOnCompleted_NotOnError() + { + // The site server caps every stream at GrpcMaxStreamLifetime (4h) and then ends the + // RPC with OK. That surfaces here as a read loop that simply runs out of events. + var client = SiteStreamGrpcClient.CreateForTesting(); + var cts = new CancellationTokenSource(); + var events = new List(); + Exception? error = null; + var completed = 0; + + await client.ConsumeStreamAsync( + "corr-ok", + cts, + () => FakeCall(new StubStreamReader(new SiteStreamEvent { CorrelationId = "corr-ok" })), + events.Add, + ex => error = ex, + () => completed++); + + Assert.Single(events); + Assert.Null(error); + Assert.Equal(1, completed); + } + + [Fact] + public async Task ConsumeStream_StreamFaults_InvokesOnError_NotOnCompleted() + { + var client = SiteStreamGrpcClient.CreateForTesting(); + var cts = new CancellationTokenSource(); + Exception? error = null; + var completed = 0; + + await client.ConsumeStreamAsync( + "corr-fault", + cts, + () => FakeCall(new StubStreamReader( + new RpcException(new Status(StatusCode.Unavailable, "site gone")))), + _ => { }, + ex => error = ex, + () => completed++); + + Assert.IsType(error); + Assert.Equal(0, completed); + } + + [Fact] + public async Task ConsumeStream_OwnCancellation_InvokesNeitherCallback() + { + // Our own Unsubscribe/reconnect is a teardown, not a fault and not a graceful end: + // the caller has already moved on to a newer stream. + var client = SiteStreamGrpcClient.CreateForTesting(); + var cts = new CancellationTokenSource(); + await cts.CancelAsync(); + Exception? error = null; + var completed = 0; + + await client.ConsumeStreamAsync( + "corr-cancel", + cts, + () => FakeCall(new StubStreamReader()), + _ => { }, + ex => error = ex, + () => completed++); + + Assert.Null(error); + Assert.Equal(0, completed); + } + + private static AsyncServerStreamingCall FakeCall(StubStreamReader reader) => + new(reader, + Task.FromResult(new Metadata()), + () => Status.DefaultSuccess, + () => new Metadata(), + () => { }); + + /// + /// Server stream stand-in: yields the queued events, then either ends the stream (the + /// status-OK completion the site's max-lifetime cap produces) or throws. + /// + private sealed class StubStreamReader : IAsyncStreamReader + { + private readonly Queue _events; + private readonly Exception? _fault; + + public StubStreamReader(params SiteStreamEvent[] events) + { + _events = new Queue(events); + } + + public StubStreamReader(Exception fault) + : this() + { + _fault = fault; + } + + public SiteStreamEvent Current { get; private set; } = null!; + + public Task MoveNext(CancellationToken cancellationToken) + { + cancellationToken.ThrowIfCancellationRequested(); + + if (_events.Count > 0) + { + Current = _events.Dequeue(); + return Task.FromResult(true); + } + + return _fault is null ? Task.FromResult(false) : Task.FromException(_fault); + } } // --- Communication-003 regression tests --- diff --git a/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/SiteAlarmLiveCacheServiceTests.cs b/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/SiteAlarmLiveCacheServiceTests.cs index 533e203c..e386f760 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/SiteAlarmLiveCacheServiceTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/SiteAlarmLiveCacheServiceTests.cs @@ -29,7 +29,7 @@ public class SiteAlarmLiveCacheServiceTests : TestKit public HangingClient() : base() { } public override Task SubscribeSiteAsync( string correlationId, Action onAlarmEvent, - Action onError, CancellationToken ct) + Action onError, Action onCompleted, CancellationToken ct) { var tcs = new TaskCompletionSource(); ct.Register(() => tcs.TrySetResult()); diff --git a/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Grpc/SiteAlarmStreamEndToEndTests.cs b/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Grpc/SiteAlarmStreamEndToEndTests.cs index 4d51bba3..53f47c57 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Grpc/SiteAlarmStreamEndToEndTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Grpc/SiteAlarmStreamEndToEndTests.cs @@ -279,7 +279,8 @@ public class SiteAlarmStreamEndToEndTests : TestKit private sealed class NoopSiteStreamClient : SiteStreamGrpcClient { public override Task SubscribeSiteAsync( - string correlationId, Action onAlarmEvent, Action onError, CancellationToken ct) + string correlationId, Action onAlarmEvent, Action onError, + Action onCompleted, CancellationToken ct) { var tcs = new TaskCompletionSource(); ct.Register(() => tcs.TrySetResult()); From 125055d998fda0d158d5f17b063d71e3daf22461 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Fri, 14 Aug 2026 19:57:26 -0400 Subject: [PATCH 06/31] perf(runtime): O(1) attribute resolution, precomputed types, coalesced static writes, shared JSON options --- .../ScriptAnalysis/ScriptAnalysisService.cs | 11 +- .../ExternalSystemClient.cs | 11 +- .../ManagementEndpoints.cs | 11 +- .../Actors/InstanceActor.cs | 271 +++++++++++++----- .../Persistence/SiteStorageService.cs | 47 +++ .../SiteExternalSystemRepository.cs | 11 +- .../Actors/InstanceActorTests.cs | 71 +++++ 7 files changed, 350 insertions(+), 83 deletions(-) diff --git a/src/ZB.MOM.WW.ScadaBridge.CentralUI/ScriptAnalysis/ScriptAnalysisService.cs b/src/ZB.MOM.WW.ScadaBridge.CentralUI/ScriptAnalysis/ScriptAnalysisService.cs index 5157ed09..b36f106a 100644 --- a/src/ZB.MOM.WW.ScadaBridge.CentralUI/ScriptAnalysis/ScriptAnalysisService.cs +++ b/src/ZB.MOM.WW.ScadaBridge.CentralUI/ScriptAnalysis/ScriptAnalysisService.cs @@ -177,6 +177,15 @@ public class ScriptAnalysisService private const int SandboxMaxConsoleChars = 32_000; private const int SandboxMaxReturnJsonChars = 32_000; + /// + /// Shared options for the per-Test-Run return-value JSON serialize in + /// (perf remediation, arch-review WP1.5) — avoids + /// allocating a new per run. Settings preserved + /// exactly. + /// + private static readonly JsonSerializerOptions SandboxReturnJsonOptions = + new() { WriteIndented = true }; + private const int SandboxMaxCallSharedDepth = 16; /// @@ -520,7 +529,7 @@ public class ScriptAnalysisService var typeName = value.GetType().Name; try { - var json = JsonSerializer.Serialize(value, new JsonSerializerOptions { WriteIndented = true }); + var json = JsonSerializer.Serialize(value, SandboxReturnJsonOptions); if (json.Length > SandboxMaxReturnJsonChars) json = json[..SandboxMaxReturnJsonChars] + "\n… (truncated)"; return (json, typeName); diff --git a/src/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway/ExternalSystemClient.cs b/src/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway/ExternalSystemClient.cs index 850ed1f9..ac91678f 100644 --- a/src/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway/ExternalSystemClient.cs +++ b/src/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway/ExternalSystemClient.cs @@ -27,6 +27,15 @@ public class ExternalSystemClient : IExternalSystemClient private readonly ILogger _logger; private readonly ExternalSystemGatewayOptions _options; + /// + /// Shared options for the per-call AuthConfiguration JSON parse in + /// (perf remediation, arch-review WP1.5) — avoids + /// allocating a new (and its internal cached + /// metadata) on every outbound call. Settings preserved exactly. + /// + private static readonly JsonSerializerOptions AuthJsonOptions = + new() { PropertyNameCaseInsensitive = true }; + /// /// Initializes a new instance of the ExternalSystemClient. /// @@ -755,7 +764,7 @@ public class ExternalSystemClient : IExternalSystemClient { fields = JsonSerializer.Deserialize>( config, - new JsonSerializerOptions { PropertyNameCaseInsensitive = true }) + AuthJsonOptions) ?? new Dictionary(StringComparer.OrdinalIgnoreCase); // Normalize to case-insensitive so "Header"/"header" both resolve. diff --git a/src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementEndpoints.cs b/src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementEndpoints.cs index 9c44f042..033a0504 100644 --- a/src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementEndpoints.cs +++ b/src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementEndpoints.cs @@ -17,6 +17,15 @@ public static class ManagementEndpoints { private static readonly TimeSpan DefaultAskTimeout = TimeSpan.FromSeconds(30); + /// + /// Shared options for the per-request command payload JSON parse in + /// (perf remediation, arch-review WP1.5) — avoids + /// allocating a new per inbound management + /// request. Settings preserved exactly. + /// + private static readonly JsonSerializerOptions CommandPayloadJsonOptions = + new() { PropertyNameCaseInsensitive = true }; + /// /// Resolves the ManagementActor Ask timeout from configuration. /// Falls back to @@ -197,7 +206,7 @@ public static class ManagementEndpoints ? p.GetRawText() : "{}"; var command = JsonSerializer.Deserialize(payloadJson, commandType, - new JsonSerializerOptions { PropertyNameCaseInsensitive = true })!; + CommandPayloadJsonOptions)!; return CommandParseResult.Ok(command); } catch (Exception ex) diff --git a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/InstanceActor.cs b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/InstanceActor.cs index aeed7315..8285bd41 100644 --- a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/InstanceActor.cs +++ b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/InstanceActor.cs @@ -7,6 +7,7 @@ using ZB.MOM.WW.ScadaBridge.Commons.Messages.DebugView; using ZB.MOM.WW.ScadaBridge.Commons.Messages.Instance; using ZB.MOM.WW.ScadaBridge.Commons.Messages.ScriptExecution; using ZB.MOM.WW.ScadaBridge.Commons.Messages.Streaming; +using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Protocol; using ZB.MOM.WW.ScadaBridge.Commons.Types; using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; using ZB.MOM.WW.ScadaBridge.Commons.Types.Flattening; @@ -92,8 +93,51 @@ public class InstanceActor : ReceiveActor // attribute lookup must be O(1) rather than a linear scan of // _configuration.Attributes. Built once in the constructor from the // deserialized configuration (last-wins on duplicate canonical names, - // mirroring the rest of the actor's by-name dictionaries). - private readonly Dictionary _resolvedAttributeByName = new(); + // mirroring the rest of the actor's by-name dictionaries). Each entry + // additionally carries its DataType/ElementDataType parsed ONCE at + // config-apply time (perf remediation, arch-review WP1.5) so the hot + // TagValueUpdate path never re-Enum.TryParse's a string per update. + private readonly Dictionary _resolvedAttributeByName = new(); + + // Static-override write coalescing (perf remediation, arch-review WP1.5): + // mirrors NativeAlarmActor's MarkDirtyUpsert/FlushDirtyUpserts (P4) shape. + // A burst of rapid SetAttribute calls on the same or different attributes + // collapses into one batched SQLite transaction on a short single-shot + // timer instead of a write per call. Keyed by attribute name — only the + // LATEST value per attribute survives a coalescing window, which matches + // the "in-memory state is authoritative" fire-and-forget persistence model + // already documented on the single-write path this replaces. + private readonly Dictionary _dirtyStaticOverrides = new(); + private ICancelable? _staticOverrideFlushTimer; + private readonly TimeSpan _staticOverrideFlushInterval; + + /// + /// Cached string forms of (perf remediation, arch-review + /// WP1.5) — the TagValueUpdate ingest path calls this once or twice per update, so a + /// static lookup replaces repeated reflection. + /// + private static readonly Dictionary QualityStrings = new() + { + [QualityCode.Good] = "Good", + [QualityCode.Bad] = "Bad", + [QualityCode.Uncertain] = "Uncertain", + }; + + private static string QualityToString(QualityCode quality) => + QualityStrings.TryGetValue(quality, out var s) ? s : quality.ToString(); + + /// + /// O(1) attribute-resolution index entry (perf remediation, arch-review WP1.5): wraps a + /// together with its / + /// element / IsList classification, parsed ONCE when the entry + /// is built (constructor) instead of re-parsed via + /// on every access. + /// + private sealed record AttributeIndexEntry( + ResolvedAttribute Attribute, + DataType DataType, + DataType? ElementDataType, + bool IsList); // WaitForAttribute (spec §4.2): one-shot waiter registry keyed by the // request CorrelationId. Each entry holds the watched attribute name, the @@ -149,7 +193,8 @@ public class InstanceActor : ReceiveActor ILogger logger, IActorRef? dclManager = null, ISiteHealthCollector? healthCollector = null, - IServiceProvider? serviceProvider = null) + IServiceProvider? serviceProvider = null, + TimeSpan? staticOverrideFlushInterval = null) { _instanceUniqueName = instanceUniqueName; _storage = storage; @@ -161,6 +206,8 @@ public class InstanceActor : ReceiveActor _dclManager = dclManager; _healthCollector = healthCollector; _serviceProvider = serviceProvider; + // Defaults to 100 ms, mirroring NativeAlarmActor's persistFlushInterval (P4). + _staticOverrideFlushInterval = staticOverrideFlushInterval ?? TimeSpan.FromMilliseconds(100); // Deserialize the flattened configuration _configuration = JsonSerializer.Deserialize(configJson); @@ -172,18 +219,23 @@ public class InstanceActor : ReceiveActor { foreach (var attr in _configuration.Attributes) { - // Index resolved attributes for O(1) lookup on the hot - // TagValueUpdate ingest path (last-wins on duplicate names). - _resolvedAttributeByName[attr.CanonicalName] = attr; + // Parse DataType/ElementDataType ONCE here (config-apply time) and index + // the wrapping entry for O(1) lookup on the hot TagValueUpdate ingest path + // (last-wins on duplicate names) — perf remediation, arch-review WP1.5. + var dataType = ParseDataType(attr.DataType); + var elementDataType = ParseElementDataType(attr.ElementDataType); + var isList = dataType == DataType.List; + _resolvedAttributeByName[attr.CanonicalName] = + new AttributeIndexEntry(attr, dataType, elementDataType, isList); // A STATIC List attribute's default is the canonical JSON // array string. Decode it to a typed List for in-memory reads // so scripts see a real collection. Scalars store their raw // string unchanged. A malformed List default decodes to null and // is marked Bad quality rather than crashing the actor. - if (IsListAttribute(attr)) + if (isList) { - var decoded = DecodeAttributeValue(attr, attr.Value); + var decoded = DecodeAttributeValue(dataType, elementDataType, attr.Value, attr.CanonicalName); _attributes[attr.CanonicalName] = decoded; _attributeQualities[attr.CanonicalName] = decoded is null && !string.IsNullOrEmpty(attr.Value) ? "Bad" @@ -257,6 +309,10 @@ public class InstanceActor : ReceiveActor // Handle internal messages Receive(HandleOverridesLoaded); + + // Coalesced flush of buffered static-override writes (P4 pattern, see + // NativeAlarmActor.FlushDirtyUpserts). + Receive(_ => FlushDirtyStaticOverrides()); } /// @@ -300,6 +356,13 @@ public class InstanceActor : ReceiveActor timer.Cancel(); _tagSubscribeRetryTimers.Clear(); + // Best-effort final flush of any coalesced static-override writes so a graceful + // stop does not lose the last <= flush-interval of writes (P4 pattern, mirrors + // NativeAlarmActor.PostStop). Fire-and-forget — the in-memory attribute value + // (already applied) remains authoritative even if this final persist is lost. + _staticOverrideFlushTimer?.Cancel(); + FlushDirtyStaticOverrides(); + // Operational `instance_lifecycle` event — instance stopped. An // instance stops on disable, delete, redeployment, and graceful shutdown; // this single point covers them all. @@ -369,9 +432,9 @@ public class InstanceActor : ReceiveActor /// private void HandleSetStaticAttribute(SetStaticAttributeCommand command) { - // Resolve the target attribute's data binding from the flattened config. - var resolved = _configuration?.Attributes - .FirstOrDefault(a => a.CanonicalName == command.AttributeName); + // O(1) lookup via the existing hot-path index instead of a linear scan over + // _configuration.Attributes (perf remediation, arch-review WP1.5). + _resolvedAttributeByName.TryGetValue(command.AttributeName, out var resolved); // Reject writes targeting an attribute that does not exist // on the deployed instance. Without this check, an inbound API @@ -398,8 +461,8 @@ public class InstanceActor : ReceiveActor } var isDataSourced = - !string.IsNullOrEmpty(resolved.DataSourceReference) - && !string.IsNullOrEmpty(resolved.BoundDataConnectionName); + !string.IsNullOrEmpty(resolved.Attribute.DataSourceReference) + && !string.IsNullOrEmpty(resolved.Attribute.BoundDataConnectionName); if (isDataSourced) { @@ -423,7 +486,7 @@ public class InstanceActor : ReceiveActor // store the string verbatim. (HandleSetStaticAttribute already rejected // unknown attributes, so resolved is non-null here, but guard defensively.) if (_resolvedAttributeByName.TryGetValue(command.AttributeName, out var resolved) - && IsListAttribute(resolved)) + && resolved.IsList) { // The script path pre-encodes valid canonical JSON via ScopeAccessors, // but the Inbound API / direct-command path can submit an arbitrary @@ -435,7 +498,8 @@ public class InstanceActor : ReceiveActor // (valid — clearing) AND a malformed non-empty input (invalid). Only the // latter is rejected, hence the explicit IsNullOrWhiteSpace guard. An empty // list "[]" decodes to a non-null empty List, so it passes through. - var decoded = DecodeAttributeValue(resolved, command.Value); + var decoded = DecodeAttributeValue( + resolved.DataType, resolved.ElementDataType, command.Value, command.AttributeName); if (!string.IsNullOrWhiteSpace(command.Value) && decoded == null) { _logger.LogWarning( @@ -474,32 +538,67 @@ public class InstanceActor : ReceiveActor PublishAndNotifyChildren(changed); - // Persist asynchronously -- fire and forget since the actor is the source of truth. - var instanceName = _instanceUniqueName; - var attributeName = command.AttributeName; - var logger = _logger; - _storage.SetStaticOverrideAsync(_instanceUniqueName, command.AttributeName, command.Value) - .ContinueWith(t => - { - logger.LogWarning( - t.Exception?.GetBaseException(), - "Failed to persist static override for {Instance}.{Attribute}; in-memory state is authoritative", - instanceName, - attributeName); - }, TaskContinuationOptions.OnlyOnFaulted); + // Coalesce the persist: record the latest value and arm the short batched-flush + // timer (P4 pattern) instead of firing a SQLite write per call — the actor's + // in-memory state (already updated above) remains the source of truth regardless + // of when the buffered write lands. + MarkDirtyStaticOverride(command.AttributeName, command.Value); Sender.Tell(new SetStaticAttributeResponse( command.CorrelationId, _instanceUniqueName, command.AttributeName, true, null, DateTimeOffset.UtcNow)); } + /// + /// Records the latest value per attribute name and arms the coalesced flush timer + /// (perf remediation, arch-review WP1.5) — mirrors NativeAlarmActor's + /// MarkDirtyUpsert/FlushDirtyUpserts (P4) shape. A burst of rapid static writes + /// collapses into one batched SQLite transaction instead of a write per call. + /// + private void MarkDirtyStaticOverride(string attributeName, string value) + { + _dirtyStaticOverrides[attributeName] = value; + if (_staticOverrideFlushTimer == null) + { + _staticOverrideFlushTimer = Context.System.Scheduler.ScheduleTellOnceCancelable( + _staticOverrideFlushInterval, Self, FlushStaticOverrides.Instance, Self); + } + } + + /// + /// Persists all coalesced static-override writes in one batched transaction (P4), + /// then disarms the flush timer. Fire-and-forget with OnlyOnFaulted logging — a + /// failed write never blocks the actor; the in-memory attribute value (already + /// applied) remains authoritative. + /// + private void FlushDirtyStaticOverrides() + { + _staticOverrideFlushTimer = null; + if (_dirtyStaticOverrides.Count == 0) + return; + + var batch = _dirtyStaticOverrides + .Select(kv => (AttributeName: kv.Key, Value: kv.Value)) + .ToList(); + _dirtyStaticOverrides.Clear(); + + var instanceName = _instanceUniqueName; + var logger = _logger; + _storage.SetStaticOverridesAsync(instanceName, batch) + .ContinueWith(t => logger.LogWarning( + t.Exception?.GetBaseException(), + "Failed to persist {Count} static overrides for {Instance}; in-memory state is authoritative", + batch.Count, instanceName), + TaskContinuationOptions.OnlyOnFaulted); + } + /// /// Data-sourced attribute write: forwards a write request to the DCL and pipes /// the device write result back to the caller. The in-memory value is left /// untouched (it is refreshed by the subscription when the device confirms); /// no static override is persisted for a data-sourced attribute. /// - private void HandleSetDataAttribute(SetStaticAttributeCommand command, ResolvedAttribute resolved) + private void HandleSetDataAttribute(SetStaticAttributeCommand command, AttributeIndexEntry resolved) { var caller = Sender; var correlationId = command.CorrelationId; @@ -526,9 +625,10 @@ public class InstanceActor : ReceiveActor // write rather than forward garbage to the device (mirrors the static-path // rejection in HandleSetStaticAttributeCore). Scalars are unchanged. object? writeValue = command.Value; - if (IsListAttribute(resolved) && !string.IsNullOrWhiteSpace(command.Value)) + if (resolved.IsList && !string.IsNullOrWhiteSpace(command.Value)) { - var decoded = DecodeAttributeValue(resolved, command.Value); + var decoded = DecodeAttributeValue( + resolved.DataType, resolved.ElementDataType, command.Value, attributeName); if (decoded == null) { _logger.LogWarning( @@ -545,8 +645,8 @@ public class InstanceActor : ReceiveActor var writeRequest = new WriteTagRequest( correlationId, - resolved.BoundDataConnectionName!, - resolved.DataSourceReference!, + resolved.Attribute.BoundDataConnectionName!, + resolved.Attribute.DataSourceReference!, writeValue, DateTimeOffset.UtcNow); @@ -598,8 +698,8 @@ public class InstanceActor : ReceiveActor foreach (var kv in request.AttributeEncodedValues) { if (!_resolvedAttributeByName.TryGetValue(kv.Key, out var resolved) - || string.IsNullOrEmpty(resolved.DataSourceReference) - || string.IsNullOrEmpty(resolved.BoundDataConnectionName)) + || string.IsNullOrEmpty(resolved.Attribute.DataSourceReference) + || string.IsNullOrEmpty(resolved.Attribute.BoundDataConnectionName)) { caller.Tell(new WriteAttributeBatchResponse( cid, false, $"Attribute '{kv.Key}' is not a data-sourced attribute.")); @@ -607,8 +707,8 @@ public class InstanceActor : ReceiveActor } if (connName == null) - connName = resolved.BoundDataConnectionName; - else if (!string.Equals(connName, resolved.BoundDataConnectionName, StringComparison.Ordinal)) + connName = resolved.Attribute.BoundDataConnectionName; + else if (!string.Equals(connName, resolved.Attribute.BoundDataConnectionName, StringComparison.Ordinal)) { caller.Tell(new WriteAttributeBatchResponse( cid, false, "Batch write spans multiple data connections; not supported.")); @@ -619,9 +719,10 @@ public class InstanceActor : ReceiveActor // MV: a data-sourced List attribute's encoded value is the canonical JSON // array string — decode it to a typed List so the DCL/Variant write // produces a real array (same poison-rejection rule as HandleSetDataAttribute). - if (IsListAttribute(resolved) && !string.IsNullOrWhiteSpace(kv.Value)) + if (resolved.IsList && !string.IsNullOrWhiteSpace(kv.Value)) { - var decoded = DecodeAttributeValue(resolved, kv.Value); + var decoded = DecodeAttributeValue( + resolved.DataType, resolved.ElementDataType, kv.Value, kv.Key); if (decoded == null) { caller.Tell(new WriteAttributeBatchResponse( @@ -631,7 +732,7 @@ public class InstanceActor : ReceiveActor wv = decoded; } - values[resolved.DataSourceReference!] = wv; + values[resolved.Attribute.DataSourceReference!] = wv; } string? triggerPath = null; @@ -639,23 +740,23 @@ public class InstanceActor : ReceiveActor if (!string.IsNullOrEmpty(request.TriggerAttribute)) { if (!_resolvedAttributeByName.TryGetValue(request.TriggerAttribute, out var tr) - || string.IsNullOrEmpty(tr.DataSourceReference) - || string.IsNullOrEmpty(tr.BoundDataConnectionName)) + || string.IsNullOrEmpty(tr.Attribute.DataSourceReference) + || string.IsNullOrEmpty(tr.Attribute.BoundDataConnectionName)) { caller.Tell(new WriteAttributeBatchResponse( cid, false, $"Trigger attribute '{request.TriggerAttribute}' is not data-sourced.")); return; } - if (connName != null && !string.Equals(connName, tr.BoundDataConnectionName, StringComparison.Ordinal)) + if (connName != null && !string.Equals(connName, tr.Attribute.BoundDataConnectionName, StringComparison.Ordinal)) { caller.Tell(new WriteAttributeBatchResponse( cid, false, "Trigger attribute is on a different data connection.")); return; } - connName ??= tr.BoundDataConnectionName; - triggerPath = tr.DataSourceReference; + connName ??= tr.Attribute.BoundDataConnectionName; + triggerPath = tr.Attribute.DataSourceReference; triggerVal = request.TriggerEncodedValue; } @@ -846,19 +947,19 @@ public class InstanceActor : ReceiveActor // (a CLR array/IEnumerable from the SDK) into a typed List. On an // element-type mismatch we set the attribute's quality to Bad, log a // warning, and skip storing a value rather than crashing the actor. - if (resolved != null && IsListAttribute(resolved)) + if (resolved != null && resolved.IsList) { - if (TryCoerceListValue(resolved, update.Value, out var typedList)) + if (TryCoerceListValue(resolved.ElementDataType, update.Value, out var typedList)) { HandleAttributeValueChanged(new AttributeValueChanged( _instanceUniqueName, update.TagPath, attrName, - typedList, update.Quality.ToString(), update.Timestamp)); + typedList, QualityToString(update.Quality), update.Timestamp)); } else { _logger.LogWarning( "List attribute {Instance}.{Attribute} received a value that could not be coerced to List<{Element}>; marking quality Bad", - _instanceUniqueName, attrName, resolved.ElementDataType); + _instanceUniqueName, attrName, resolved.Attribute.ElementDataType); _attributeQualities[attrName] = "Bad"; _attributeTimestamps[attrName] = update.Timestamp; var currentValue = _attributes.GetValueOrDefault(attrName); @@ -883,14 +984,21 @@ public class InstanceActor : ReceiveActor HandleAttributeValueChanged(new AttributeValueChanged( _instanceUniqueName, update.TagPath, attrName, - value, update.Quality.ToString(), update.Timestamp)); + value, QualityToString(update.Quality), update.Timestamp)); } } - /// True if the resolved attribute is declared as a . - private static bool IsListAttribute(ResolvedAttribute attr) => - Enum.TryParse(attr.DataType, ignoreCase: true, out var dt) - && dt == DataType.List; + /// Parses a string, defaulting to + /// on an unrecognized value (never throws). + private static DataType ParseDataType(string dataType) => + Enum.TryParse(dataType, ignoreCase: true, out var dt) ? dt : DataType.String; + + /// Parses a string, returning + /// when absent or unrecognized (never throws). + private static DataType? ParseElementDataType(string? elementDataType) => + string.IsNullOrEmpty(elementDataType) + ? null + : (Enum.TryParse(elementDataType, ignoreCase: true, out var et) ? et : (DataType?)null); /// /// Decodes a STATIC (authored / overridden) attribute's canonical value @@ -903,16 +1011,15 @@ public class InstanceActor : ReceiveActor /// bad element, missing element type) degrades to + a /// warning — the caller marks the attribute Bad quality. NEVER throws into the /// actor. + /// + /// Takes the already-parsed / + /// (perf remediation, arch-review WP1.5) rather than re-parsing from the attribute's + /// raw DataType strings — every call site now resolves these once via + /// at config-apply time. /// - private object? DecodeAttributeValue(ResolvedAttribute attr, string? raw) + private object? DecodeAttributeValue( + DataType dataType, DataType? elementType, string? raw, string attributeCanonicalName) { - DataType dataType = Enum.TryParse(attr.DataType, ignoreCase: true, out var dt) - ? dt - : DataType.String; - DataType? elementType = string.IsNullOrEmpty(attr.ElementDataType) - ? null - : (Enum.TryParse(attr.ElementDataType, ignoreCase: true, out var et) ? et : null); - try { return AttributeValueCodec.Decode(raw, dataType, elementType); @@ -921,30 +1028,28 @@ public class InstanceActor : ReceiveActor { _logger.LogWarning(ex, "Attribute '{Attr}' on '{Instance}' has an undecodable List value; marking Bad quality", - attr.CanonicalName, _instanceUniqueName); + attributeCanonicalName, _instanceUniqueName); return null; // caller marks quality Bad } } /// /// Coerces an incoming data-sourced value (an OPC UA array / IEnumerable) - /// into a typed List<elementClrType> matching the attribute's - /// . Each element is converted - /// with invariant culture (round-trip parse for DateTime). Returns - /// on a missing/invalid element type, a non-enumerable - /// value, or any element that cannot be coerced — the caller then marks the - /// attribute quality Bad. Never throws. + /// into a typed List<elementClrType> matching the attribute's element + /// type. Each element is converted with invariant culture (round-trip parse for + /// DateTime). Returns on a missing/invalid element type, a + /// non-enumerable value, or any element that cannot be coerced — the caller then + /// marks the attribute quality Bad. Never throws. + /// + /// Takes the already-parsed (perf remediation, + /// arch-review WP1.5) — see . /// - private bool TryCoerceListValue(ResolvedAttribute attr, object? incoming, out object? typedList) + private bool TryCoerceListValue(DataType? elementType, object? incoming, out object? typedList) { typedList = null; - if (string.IsNullOrEmpty(attr.ElementDataType) - || !Enum.TryParse(attr.ElementDataType, ignoreCase: true, out var elementType) - || !AttributeValueCodec.IsValidElementType(elementType)) - { + if (elementType is not { } et || !AttributeValueCodec.IsValidElementType(et)) return false; - } if (incoming is not System.Collections.IEnumerable enumerable || incoming is string) return false; @@ -956,7 +1061,7 @@ public class InstanceActor : ReceiveActor // inside the guarded block means any future change that introduces a // throw is caught and turned into a Bad-quality result rather than // escaping into the actor and tripping supervision. - typedList = AttributeValueCodec.CoerceEnumerable(enumerable, elementType); + typedList = AttributeValueCodec.CoerceEnumerable(enumerable, et); return true; } catch (Exception ex) @@ -964,7 +1069,7 @@ public class InstanceActor : ReceiveActor // Any coercion / construction failure → Bad quality, never a crash. _logger.LogWarning(ex, "Failed to coerce value to List<{Element}> for instance {Instance}; marking quality Bad", - attr.ElementDataType, _instanceUniqueName); + et, _instanceUniqueName); return false; } } @@ -1417,7 +1522,7 @@ public class InstanceActor : ReceiveActor // overrides to a typed list (matching the config-default load), set // Bad quality on a malformed stored value, and never crash the actor. if (_resolvedAttributeByName.TryGetValue(kvp.Key, out var resolved) - && IsListAttribute(resolved)) + && resolved.IsList) { // Decode the stored List override (both old array-of-strings // and native-typed forms decode) and re-persist the native form if @@ -1425,7 +1530,8 @@ public class InstanceActor : ReceiveActor // list and comparing to the stored string detects old-form values // (native → native is byte-identical, so a native value is a no-op). // The re-persist is fire-and-forget and never throws into the actor. - var decoded = DecodeAttributeValue(resolved, kvp.Value); + var decoded = DecodeAttributeValue( + resolved.DataType, resolved.ElementDataType, kvp.Value, kvp.Key); _attributes[kvp.Key] = decoded; if (decoded is null && !string.IsNullOrEmpty(kvp.Value)) { @@ -1742,4 +1848,11 @@ public class InstanceActor : ReceiveActor /// connection whose previous attempt failed or whose response was lost. /// private sealed record RetryTagSubscribe(string ConnectionName); + + /// Self-tell that fires the coalesced static-override persist flush (P4, arch-review WP1.5). + private sealed class FlushStaticOverrides + { + public static readonly FlushStaticOverrides Instance = new(); + private FlushStaticOverrides() { } + } } diff --git a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs index 1f97e8c4..63f2c697 100644 --- a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs +++ b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs @@ -345,6 +345,53 @@ public class SiteStorageService await command.ExecuteNonQueryAsync(); } + /// + /// Batch counterpart of (perf remediation, + /// arch-review WP1.5): upserts a set of static attribute overrides for one instance + /// in a SINGLE connection + transaction with one prepared command re-bound per row. + /// Used by the InstanceActor's coalesced flush — mirrors NativeAlarmActor's + /// MarkDirtyUpsert/FlushDirtyUpserts (P4) pattern — so a burst of rapid SetAttribute + /// calls on an instance collapses into one batched write instead of a write per call. + /// + /// The unique name of the instance. + /// The attribute name/value pairs to upsert (latest value per name). + /// A task that completes when all rows have been committed. + public async Task SetStaticOverridesAsync( + string instanceName, IReadOnlyList<(string AttributeName, string Value)> overrides) + { + if (overrides.Count == 0) + return; + + await using var connection = OpenConnection(); + await using var transaction = (SqliteTransaction)await connection.BeginTransactionAsync(); + + await using var command = connection.CreateCommand(); + command.Transaction = transaction; + command.CommandText = @" + INSERT INTO static_attribute_overrides (instance_unique_name, attribute_name, override_value, updated_at) + VALUES (@name, @attr, @val, @updatedAt) + ON CONFLICT(instance_unique_name, attribute_name) DO UPDATE SET + override_value = excluded.override_value, + updated_at = excluded.updated_at"; + + var pName = command.Parameters.Add("@name", SqliteType.Text); + var pAttr = command.Parameters.Add("@attr", SqliteType.Text); + var pVal = command.Parameters.Add("@val", SqliteType.Text); + var pUpdatedAt = command.Parameters.Add("@updatedAt", SqliteType.Text); + pName.Value = instanceName; + + var updatedAt = DateTimeOffset.UtcNow.ToString("O"); + foreach (var (attributeName, value) in overrides) + { + pAttr.Value = attributeName; + pVal.Value = value; + pUpdatedAt.Value = updatedAt; + await command.ExecuteNonQueryAsync(); + } + + await transaction.CommitAsync(); + } + /// /// Clears all static attribute overrides for an instance. /// Called on redeployment to reset overrides. diff --git a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Repositories/SiteExternalSystemRepository.cs b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Repositories/SiteExternalSystemRepository.cs index 1fe373f4..7daba84e 100644 --- a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Repositories/SiteExternalSystemRepository.cs +++ b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Repositories/SiteExternalSystemRepository.cs @@ -16,6 +16,15 @@ public class SiteExternalSystemRepository : IExternalSystemRepository { private readonly SiteStorageService _storage; + /// + /// Shared options for the per-row MethodDefinitionDto JSON parse in + /// (perf remediation, arch-review WP1.5) — + /// avoids allocating a new per call. Settings + /// preserved exactly; the query shape itself is untouched (WP2.6). + /// + private static readonly JsonSerializerOptions MethodDefinitionJsonOptions = + new() { PropertyNameCaseInsensitive = true }; + /// /// Initializes a new site-side external system repository. /// @@ -279,7 +288,7 @@ public class SiteExternalSystemRepository : IExternalSystemRepository try { var methods = JsonSerializer.Deserialize>(json, - new JsonSerializerOptions { PropertyNameCaseInsensitive = true }); + MethodDefinitionJsonOptions); if (methods is null) return Array.Empty(); diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorTests.cs index 5657c1a8..ee13aea0 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorTests.cs @@ -242,6 +242,77 @@ public class InstanceActorTests : TestKit, IDisposable Assert.Equal("100.0", overrides["Temperature"]); } + /// + /// Perf remediation (arch-review WP1.5): a burst of rapid static-attribute writes must + /// coalesce into ONE batched SQLite transaction on a short single-shot timer — mirroring + /// NativeAlarmActor's MarkDirtyUpsert/FlushDirtyUpserts (P4) shape — rather than a write + /// per . Pinned two ways: (1) nothing is persisted + /// before the flush interval elapses, even though every write already replied + /// synchronously and updated in-memory state; (2) only the LATEST value per attribute + /// survives, across writes to two different attributes coalesced into the same flush. + /// + [Fact] + public async Task InstanceActor_RapidStaticWrites_CoalesceIntoSingleFlush() + { + var config = new FlattenedConfiguration + { + InstanceUniqueName = "PumpCoalesce1", + Attributes = + [ + new ResolvedAttribute { CanonicalName = "Counter", Value = "0", DataType = "Int32" }, + new ResolvedAttribute { CanonicalName = "Label", Value = "Idle", DataType = "String" } + ] + }; + + var flushInterval = TimeSpan.FromSeconds(1); + var actor = ActorOf(Props.Create(() => new InstanceActor( + "PumpCoalesce1", + JsonSerializer.Serialize(config), + _storage, + _compilationService, + _sharedScriptLibrary, + null, + _options, + NullLogger.Instance, + null, // dclManager + null, // healthCollector + null, // serviceProvider + flushInterval))); + + // Five rapid writes to "Counter" plus one to "Label", all sent back-to-back with no + // delay between them (well inside the 1s coalescing window). + for (var i = 1; i <= 5; i++) + { + actor.Tell(new SetStaticAttributeCommand( + $"corr-counter-{i}", "PumpCoalesce1", "Counter", i.ToString(), DateTimeOffset.UtcNow)); + } + actor.Tell(new SetStaticAttributeCommand( + "corr-label", "PumpCoalesce1", "Label", "Running", DateTimeOffset.UtcNow)); + + // Every write replies synchronously (in-memory state is authoritative regardless of + // when the buffered persist lands) — six writes, six responses. + for (var i = 0; i < 6; i++) + { + var response = ExpectMsg(TimeSpan.FromSeconds(5)); + Assert.True(response.Success); + } + + // Well before the flush interval elapses, NOTHING has reached SQLite yet — proves the + // writes did not each fire their own persist (the old per-write fire-and-forget shape + // would already show a row here). + await Task.Delay(200); + var beforeFlush = await _storage.GetStaticOverridesAsync("PumpCoalesce1"); + Assert.Empty(beforeFlush); + + // After the flush interval elapses, exactly one coalesced write has landed: only the + // LATEST value per attribute survives. + await Task.Delay(1500); + var afterFlush = await _storage.GetStaticOverridesAsync("PumpCoalesce1"); + Assert.Equal(2, afterFlush.Count); + Assert.Equal("5", afterFlush["Counter"]); + Assert.Equal("Running", afterFlush["Label"]); + } + [Fact] public async Task InstanceActor_LoadsStaticOverridesFromSQLite() { From 7ebdcd370a36582da29718462f7cef6144b07c85 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Fri, 14 Aug 2026 19:59:53 -0400 Subject: [PATCH 07/31] perf(host): install CDC capture only when replication is configured MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit SiteLocalDbSetup.OnReady registered all ten replicated tables unconditionally, so a deliberately unreplicated site node (site-b and site-c on the rig) carried the full 30-trigger CDC set forever. Every write to those tables paid two extra INSERTs plus a json_object serialization of the whole row, inside the caller's own transaction, and appended to an oplog nothing ever drains. Arch-review finding #5 (High), repo half; the library half — trigger cleanup API and O(1) backlog — is WP3.3. The ten RegisterReplicated calls are now behind a guard on whether the node has LocalDb:Replication:PeerAddress OR LocalDb:Replication:ApiKey. Either key counts, and the OR is load-bearing rather than defensive: replication is one bidirectional stream that exactly one side dials, so only the initiator sets PeerAddress. Verified against the rig — site-a node-a has PeerAddress + ApiKey, site-a node-b (passive) has ApiKey alone, site-b/site-c have no Replication section at all. Keying on PeerAddress alone would have stripped capture from every passive node and silently made each pair converge in one direction only. The load-bearing ordering documented in the file is preserved: DDL still precedes registration, and the legacy migrator still runs unconditionally after it — an unreplicated node must still absorb its pre-Phase-1 files, and it has no peer for those rows to be invisible to. Known residual, documented in-file and in the topology guide: a database file first created by an older build keeps its stale __localdb_* triggers. The guard decides whether triggers are installed, not whether existing ones are removed, and the library has no removal API until WP3.3. Moot on the docker rig, where a schema-change redeploy recreates the volumes. The inverse is also now documented: enabling replication on a site that has run without it does not baseline existing rows, since CDC never recorded them in __localdb_row_version and the snapshot resync streams from that ledger. Tests: new SiteLocalDbCdcRegistrationTests asserts trigger presence and absence via sqlite_master across all four config shapes (none, ApiKey only, PeerAddress + ApiKey, and the notification-table exclusion), plus DDL-still-runs and migrator-still-runs on the unreplicated branch. SiteLocalDbWiringTests and the integration site-pair harness now configure an ApiKey — mirroring the rig's passive node — so their registration and convergence assertions still describe a replicating node. 483/483 Host.Tests pass; the 20 offline LocalDb convergence tests still pass. --- docs/deployment/topology-guide.md | 20 ++ .../SiteLocalDbSetup.cs | 127 +++++++--- .../SiteLocalDbCdcRegistrationTests.cs | 227 ++++++++++++++++++ .../SiteLocalDbWiringTests.cs | 11 +- .../LocalDbSitePairHarness.cs | 6 + 5 files changed, 358 insertions(+), 33 deletions(-) create mode 100644 tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbCdcRegistrationTests.cs diff --git a/docs/deployment/topology-guide.md b/docs/deployment/topology-guide.md index 84343034..8e2a49dd 100644 --- a/docs/deployment/topology-guide.md +++ b/docs/deployment/topology-guide.md @@ -157,6 +157,26 @@ Each site has its own two-node cluster: - SQLite persistence: each node owns its own consolidated LocalDb database, kept in step by asynchronous CDC replication over a gRPC sync stream (LocalDb Phase 1 + 2). The nodes do NOT share a SQLite file. +- CDC capture triggers are installed **only on a node that has replication configured** — + `LocalDb:Replication:PeerAddress` *or* `LocalDb:Replication:ApiKey`. Either key counts, because + only the initiating half of a pair sets `PeerAddress` (one bidirectional stream, dialled by one + side); the passive half carries the key alone. A deliberately unreplicated node — site-b and + site-c on the rig — runs with no triggers at all and stops paying the per-write capture cost. + +#### Turning replication ON for a site that has been running without it + +Set the keys on **both** nodes and restart both. Two things to know before you do: + +- **Existing rows are not baselined.** Capture is change-data-capture: rows written while the node + had no triggers were never recorded in `__localdb_row_version`, and LocalDb's snapshot resync + streams from that ledger, so it will not ship them. The pair converges on everything written + *after* the restart and stays silently divergent on everything before it. Start from a copy of one + node's database on both sides, or accept that only new writes converge. +- **A node upgraded in place keeps stale triggers.** The guard decides whether triggers are + *installed*, not whether existing ones are removed, and the library has no removal API yet. A + database file first created by a build that always registered keeps capturing until that lands. + Recreating the node's data volume clears it — which is what a schema-change redeploy does on the + docker rig, so the rig is unaffected. ### Site Pair Upgrades — stop and start BOTH nodes together diff --git a/src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbSetup.cs b/src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbSetup.cs index beb7f311..74698fa7 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbSetup.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbSetup.cs @@ -21,6 +21,14 @@ namespace ZB.MOM.WW.ScadaBridge.Host; /// with no error anywhere. /// /// +/// Registration is conditional on the node actually having a replication peer — +/// see . An unreplicated node used to install the +/// full CDC trigger set anyway and then pay for it on every single write, forever, for an +/// oplog nothing ever drains. The ordering rule above is unaffected: when registration is +/// skipped there is no capture to be too late for, and the legacy migrator still runs +/// (its rows simply do not replicate, which is what "unreplicated node" means). +/// +/// /// Runs inside the AddZbLocalDb onReady callback, once, before any caller /// receives the singleton. onReady is a synchronous /// Action<ILocalDb>, hence the direct CreateConnection() rather than @@ -32,11 +40,14 @@ namespace ZB.MOM.WW.ScadaBridge.Host; public static class SiteLocalDbSetup { /// - /// Creates the site node's replicated tables, opts them into change capture, and - /// migrates any pre-Phase-1 databases in. + /// Creates the site node's tables, opts them into change capture when this node + /// replicates, and migrates any pre-Phase-1 databases in. /// /// The LocalDb instance being initialized. - /// Configuration, for the legacy migrator's old path keys and node name. + /// + /// Configuration, for the replication predicate, the legacy migrator's old path keys, + /// and the node name. + /// public static void OnReady(ILocalDb db, IConfiguration config) { ArgumentNullException.ThrowIfNull(db); @@ -53,39 +64,93 @@ public static class SiteLocalDbSetup StoreAndForwardSchema.Apply(connection); } - // Both tables qualify: each has an explicit primary key (RegisterReplicated - // rejects tables without one) and no BLOB columns (which json_object cannot - // capture). Registration is idempotent and installs the capture triggers. - db.RegisterReplicated("OperationTracking"); - db.RegisterReplicated("site_events"); + if (ReplicationIsConfigured(config)) + { + // Both tables qualify: each has an explicit primary key (RegisterReplicated + // rejects tables without one) and no BLOB columns (which json_object cannot + // capture). Registration is idempotent and installs the capture triggers. + db.RegisterReplicated("OperationTracking"); + db.RegisterReplicated("site_events"); - // Phase 2: the store-and-forward buffer and the seven site configuration tables. - // These replaced the bespoke SiteReplicationActor and StoreAndForward - // ReplicationService, which shipped hand-written Add/Remove/Park/Requeue operations - // over Akka; both were deleted in the same commit that added these lines, so the - // two mechanisms never ran at once. - // - // Both composite-PK tables are fine: RegisterReplicated orders multi-column PKs by - // ordinal. No Phase 2 table has a BLOB column, which it would reject. - db.RegisterReplicated("sf_messages"); - db.RegisterReplicated("deployed_configurations"); - db.RegisterReplicated("static_attribute_overrides"); - db.RegisterReplicated("shared_scripts"); - db.RegisterReplicated("external_systems"); - db.RegisterReplicated("database_connections"); - db.RegisterReplicated("data_connection_definitions"); - db.RegisterReplicated("native_alarm_state"); + // Phase 2: the store-and-forward buffer and the seven site configuration tables. + // These replaced the bespoke SiteReplicationActor and StoreAndForward + // ReplicationService, which shipped hand-written Add/Remove/Park/Requeue operations + // over Akka; both were deleted in the same commit that added these lines, so the + // two mechanisms never ran at once. + // + // Both composite-PK tables are fine: RegisterReplicated orders multi-column PKs by + // ordinal. No Phase 2 table has a BLOB column, which it would reject. + db.RegisterReplicated("sf_messages"); + db.RegisterReplicated("deployed_configurations"); + db.RegisterReplicated("static_attribute_overrides"); + db.RegisterReplicated("shared_scripts"); + db.RegisterReplicated("external_systems"); + db.RegisterReplicated("database_connections"); + db.RegisterReplicated("data_connection_definitions"); + db.RegisterReplicated("native_alarm_state"); - // notification_lists and smtp_configurations are created but deliberately NOT - // registered. They are permanently empty by design — the site-side write paths were - // removed on 2026-07-10, the legacy migrator skips them, and the active node's - // artifact apply purges them on every deploy. Registering them would open a standing - // replication channel whose only historical payload was plaintext SMTP passwords, in - // exchange for replicating nothing. Anyone adding them here should first establish - // that a site has a legitimate reason to hold SMTP credentials at all. + // notification_lists and smtp_configurations are created but deliberately NOT + // registered. They are permanently empty by design — the site-side write paths were + // removed on 2026-07-10, the legacy migrator skips them, and the active node's + // artifact apply purges them on every deploy. Registering them would open a standing + // replication channel whose only historical payload was plaintext SMTP passwords, in + // exchange for replicating nothing. Anyone adding them here should first establish + // that a site has a legitimate reason to hold SMTP credentials at all. + } // AFTER registration, so migrated rows enter the oplog and reach the peer like // any other write. Before it, they would be invisible to replication forever. + // Deliberately OUTSIDE the guard: an unreplicated node still has to absorb its + // pre-Phase-1 files, and there is no peer for its rows to be invisible to. SiteLocalDbLegacyMigrator.Migrate(db, config); } + + /// + /// Whether this node participates in LocalDb replication, and therefore needs the CDC + /// capture triggers at all. + /// + /// + /// + /// Either key counts, and that is not redundancy — it is the only predicate that is + /// true on BOTH halves of a replicated pair. Replication is one bidirectional + /// stream that exactly one side dials, so only the initiator sets + /// LocalDb:Replication:PeerAddress; the passive node has an ApiKey and + /// nothing else. (The rig is the reference: site-a node-a carries PeerAddress + ApiKey, + /// site-a node-b carries ApiKey alone, and site-b/site-c carry no Replication + /// section at all.) Keying on PeerAddress alone would strip capture from every passive + /// node — its local writes would stop reaching the initiator, and the pair would + /// silently converge in one direction only. + /// + /// + /// The keys are read straight from configuration rather than through + /// IOptions<ReplicationOptions> because this runs inside the + /// AddZbLocalDb factory, where resolving another option would nest a service + /// resolution inside a singleton construction. The section name matches what + /// AddZbLocalDbReplication binds, so the two cannot disagree about which node + /// replicates. + /// + /// + /// Known residual: a database file that was registered by an OLDER build keeps + /// its stale __localdb_* triggers — this only decides whether new ones are + /// installed, and the library has no removal API yet (it arrives with the WP3.3 library + /// work, which will also drop them on an unconfigured node). On the docker rig this is + /// moot: a schema change recreates the volumes. On a long-lived unreplicated node + /// upgraded in place, capture continues until that lands. + /// + /// + /// The other direction has a consequence too: turning replication ON for a site + /// that has been running without it does NOT baseline the rows already in the file. + /// Capture never recorded them in __localdb_row_version, and the snapshot resync + /// streams from that ledger, so the pair converges only on writes made after the + /// restart. Seed both nodes from one node's database if the existing rows matter. See + /// docs/deployment/topology-guide.md. + /// + /// + private static bool ReplicationIsConfigured(IConfiguration config) + { + var section = config.GetSection("LocalDb:Replication"); + + return !string.IsNullOrWhiteSpace(section["PeerAddress"]) + || !string.IsNullOrWhiteSpace(section["ApiKey"]); + } } diff --git a/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbCdcRegistrationTests.cs b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbCdcRegistrationTests.cs new file mode 100644 index 00000000..52586a7a --- /dev/null +++ b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbCdcRegistrationTests.cs @@ -0,0 +1,227 @@ +using Microsoft.Extensions.Configuration; +using Microsoft.Extensions.DependencyInjection; +using ZB.MOM.WW.LocalDb; + +namespace ZB.MOM.WW.ScadaBridge.Host.Tests; + +/// +/// WP1.3 — CDC capture is installed only on a node that actually replicates. +/// +/// +/// +/// These assert on the TRIGGERS in sqlite_master rather than on +/// ILocalDb.ReplicatedTables, because the trigger set is what costs anything: the +/// registry entry is a dictionary lookup, while each trigger runs two extra INSERTs plus a +/// json_object serialization of the full row inside every write transaction on the +/// table. An unreplicated node paid that on every store-and-forward enqueue, every static +/// override, every event log row, forever, for an oplog with no reader. +/// +/// +/// The naming __localdb_{table}_{ai|au|ad} is the library's +/// (TriggerSqlGenerator.TriggerName), matched by prefix here so a fourth trigger kind +/// would be caught rather than quietly ignored. +/// +/// +public class SiteLocalDbCdcRegistrationTests : IDisposable +{ + private readonly string _root; + private readonly List _providers = []; + + public SiteLocalDbCdcRegistrationTests() + { + _root = Path.Combine(Path.GetTempPath(), $"localdb-cdc-{Guid.NewGuid():N}"); + Directory.CreateDirectory(_root); + } + + public void Dispose() + { + foreach (var provider in _providers) + { + try { provider.Dispose(); } catch { /* best effort */ } + } + + Microsoft.Data.Sqlite.SqliteConnection.ClearAllPools(); + try { Directory.Delete(_root, recursive: true); } catch { /* best effort */ } + GC.SuppressFinalize(this); + } + + [Fact] + public void UnreplicatedNode_InstallsNoCaptureTriggers() + { + // No LocalDb:Replication section at all — site-b and site-c on the rig. + var db = BuildDatabase(Config()); + + Assert.Empty(CaptureTriggers(db)); + } + + [Fact] + public void UnreplicatedNode_StillCreatesEveryTable() + { + // Skipping registration must not skip the DDL: the node is fully functional + // standalone, it just has nobody to ship its changes to. A guard placed around the + // schema block instead of the registration block would fail here and nowhere else. + var db = BuildDatabase(Config()); + + var tables = TableNames(db); + + foreach (var table in AllSiteTables) + Assert.Contains(table, tables); + } + + [Fact] + public void InitiatorNode_InstallsCaptureTriggers() + { + // PeerAddress + ApiKey — site-a node-a, the half that dials. + var db = BuildDatabase(Config( + peerAddress: "http://peer:8083", apiKey: "cdc-test-key")); + + AssertCaptureTriggersCoverTheReplicatedTables(db); + } + + [Fact] + public void PassiveNode_WithApiKeyButNoPeerAddress_StillInstallsCaptureTriggers() + { + // The reason the predicate is an OR. Site-a node-b sets ApiKey and nothing else: + // one bidirectional stream, dialled by one side. Keying on PeerAddress alone would + // leave this node capturing nothing, so its own writes would never reach the + // initiator and the pair would converge in one direction only — silently. + var db = BuildDatabase(Config(apiKey: "cdc-test-key")); + + AssertCaptureTriggersCoverTheReplicatedTables(db); + } + + [Fact] + public void ReplicatedNode_InstallsNoCaptureTriggersOnTheCentralOnlyNotificationTables() + { + // The security property, restated at the trigger level: notification_lists and + // smtp_configurations exist but must never be captured, because the only payload + // they ever historically held was plaintext SMTP passwords. + var db = BuildDatabase(Config(apiKey: "cdc-test-key")); + + var triggers = CaptureTriggers(db); + + Assert.DoesNotContain(triggers, t => t.StartsWith("__localdb_notification_lists_", StringComparison.Ordinal)); + Assert.DoesNotContain(triggers, t => t.StartsWith("__localdb_smtp_configurations_", StringComparison.Ordinal)); + } + + [Fact] + public void UnreplicatedNode_StillRunsTheLegacyMigrator() + { + // The migrator is deliberately outside the guard. It renames the legacy file to + // ".migrated" on success, which is the observable proof it ran without needing a + // capture trigger to look at. + var legacyPath = Path.Combine(_root, "legacy-store-and-forward.db"); + SeedLegacyStoreAndForward(legacyPath); + + _ = BuildDatabase(Config(storeAndForwardPath: legacyPath)); + + Assert.False(File.Exists(legacyPath)); + Assert.True(File.Exists(legacyPath + ".migrated")); + } + + // ---- helpers ---------------------------------------------------------------------- + + /// The ten replicated tables plus the two deliberately-unregistered ones. + private static readonly string[] AllSiteTables = + [ + "OperationTracking", "site_events", "sf_messages", "deployed_configurations", + "static_attribute_overrides", "shared_scripts", "external_systems", + "database_connections", "data_connection_definitions", "native_alarm_state", + "notification_lists", "smtp_configurations", + ]; + + private static readonly string[] ReplicatedTables = + [ + "OperationTracking", "site_events", "sf_messages", "deployed_configurations", + "static_attribute_overrides", "shared_scripts", "external_systems", + "database_connections", "data_connection_definitions", "native_alarm_state", + ]; + + private static void AssertCaptureTriggersCoverTheReplicatedTables(ILocalDb db) + { + var triggers = CaptureTriggers(db); + + foreach (var table in ReplicatedTables) + { + // All three kinds, so a partial install is a failure rather than a pass. + foreach (var suffix in new[] { "ai", "au", "ad" }) + Assert.Contains($"__localdb_{table}_{suffix}", triggers); + } + } + + private ILocalDb BuildDatabase(IConfiguration config) + { + var provider = new ServiceCollection() + .AddZbLocalDb(config, db => SiteLocalDbSetup.OnReady(db, config)) + .BuildServiceProvider(); + _providers.Add(provider); + + return provider.GetRequiredService(); + } + + private IConfiguration Config( + string? peerAddress = null, string? apiKey = null, string? storeAndForwardPath = null) + { + var values = new Dictionary + { + ["LocalDb:Path"] = Path.Combine(_root, "consolidated.db"), + ["ScadaBridge:Node:NodeName"] = "node-a", + + // Every legacy path is pinned inside this test's own directory. Left unset they + // resolve CWD-relative (./data/…), and the migrator RENAMES whatever it finds — + // so an unlucky run could eat a real file from the test binary's output folder. + ["ScadaBridge:StoreAndForward:SqliteDbPath"] = + storeAndForwardPath ?? Path.Combine(_root, "absent-store-and-forward.db"), + ["ScadaBridge:Database:SiteDbPath"] = Path.Combine(_root, "absent-scadabridge.db"), + ["ScadaBridge:SiteEventLog:DatabasePath"] = Path.Combine(_root, "absent-events.db"), + ["ScadaBridge:OperationTracking:ConnectionString"] = + $"Data Source={Path.Combine(_root, "absent-tracking.db")}", + }; + + if (peerAddress is not null) values["LocalDb:Replication:PeerAddress"] = peerAddress; + if (apiKey is not null) values["LocalDb:Replication:ApiKey"] = apiKey; + + return new ConfigurationBuilder().AddInMemoryCollection(values).Build(); + } + + private static void SeedLegacyStoreAndForward(string path) + { + using var connection = new Microsoft.Data.Sqlite.SqliteConnection($"Data Source={path}"); + connection.Open(); + ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardSchema.Apply(connection); + } + + /// + /// Every LocalDb capture trigger in the file. Matched on the library's + /// __localdb_ prefix in C# rather than with SQL LIKE, where the underscores + /// are single-character wildcards and the pattern would need escaping to mean itself. + /// + private static HashSet CaptureTriggers(ILocalDb db) + { + using var connection = db.CreateConnection(); + using var cmd = connection.CreateCommand(); + cmd.CommandText = "SELECT name FROM sqlite_master WHERE type = 'trigger'"; + using var reader = cmd.ExecuteReader(); + + var names = new HashSet(StringComparer.Ordinal); + while (reader.Read()) + { + var name = reader.GetString(0); + if (name.StartsWith("__localdb_", StringComparison.Ordinal)) names.Add(name); + } + + return names; + } + + private static HashSet TableNames(ILocalDb db) + { + using var connection = db.CreateConnection(); + using var cmd = connection.CreateCommand(); + cmd.CommandText = "SELECT name FROM sqlite_master WHERE type = 'table'"; + using var reader = cmd.ExecuteReader(); + + var names = new HashSet(StringComparer.Ordinal); + while (reader.Read()) names.Add(reader.GetString(0)); + return names; + } +} diff --git a/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbWiringTests.cs b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbWiringTests.cs index 4340a148..fc4a296c 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbWiringTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbWiringTests.cs @@ -56,9 +56,16 @@ public class SiteLocalDbWiringTests : IDisposable ["ScadaBridge:Cluster:SeedNodes:0"] = "akka.tcp://scadabridge@localhost:2551", ["ScadaBridge:Cluster:SeedNodes:1"] = "akka.tcp://scadabridge@localhost:2552", - // The consolidated site database. No Replication section at all — this - // fixture is also the default-OFF pin: storage must work standalone. + // The consolidated site database, configured exactly like the PASSIVE half of + // the rig's replicated pair (site-a node-b): an ApiKey and no PeerAddress. + // + // The key is what makes this a replicating node, and therefore what makes the + // CDC registration assertions below apply at all — capture is installed only + // when replication is configured. The absent PeerAddress keeps this fixture the + // default-OFF pin at the same time: nothing dials, so the engine must still + // resolve and idle rather than throw or report a connection. ["LocalDb:Path"] = _tempDbPath, + ["LocalDb:Replication:ApiKey"] = "wiring-test-localdb-sync-key", }); builder.Services.AddGrpc(); diff --git a/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbSitePairHarness.cs b/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbSitePairHarness.cs index 0bc73e80..20bf2e48 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbSitePairHarness.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbSitePairHarness.cs @@ -113,6 +113,12 @@ public abstract class LocalDbSitePairHarness : IAsyncLifetime { ["LocalDb:Path"] = path, ["ScadaBridge:Node:NodeName"] = nodeName, + // OnReady installs the CDC capture triggers only on a node that has + // replication configured, so the key has to be here and not only in + // ReplicationConfig below — without it these nodes would run the sync + // engine over an oplog nothing ever writes to, and every convergence + // scenario would time out with two intact but unrelated databases. + ["LocalDb:Replication:ApiKey"] = SharedApiKey, // Point the legacy migrators at paths that do not exist, so they no-op rather // than picking up stray files from the test working directory. The two Phase 2 // defaults matter most: unlike the Phase 1 pair they resolve inside ./data/, From 2e4e41a8f73da995d6b390dd33035175da27d4cc Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Fri, 14 Aug 2026 20:13:31 -0400 Subject: [PATCH 08/31] fix(auditlog): site audit DB onto the data volume; required path + soft flush MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes WP1.2 of the arch-review remediation plan (finding #2, High): SqliteAuditWriterOptions.DatabasePath defaulted to CWD-relative "auditlog.db", which on the docker rig resolves onto the container's ephemeral overlayfs (not the mounted /app/data volume), silently discarding the pending audit forward-state backlog on every recreate; nothing in docker/ or docker-env2/ overrode it; FlushIntervalMs was validated but never read by the writer loop (one commit per event even at trickle rate); and no PRAGMA synchronous was set (SQLite's FULL default fsyncs every commit). - DatabasePath now has no default (mirrors ZB.MOM.WW.LocalDb's LocalDbOptions.Path) and is required pre-host for Site nodes only, via a new StartupValidator raw-config check (top-level "AuditLog:SiteWriter:DatabasePath", NOT nested under ScadaBridge: AddAuditLog binds that section off the configuration root). SqliteAuditWriterOptionsValidator deliberately does NOT check DatabasePath itself, because AddAuditLog runs its ValidateOnStart on both Central and Site composition roots but only Site nodes ever resolve the writer — checking it there would fail Central's boot too. - All 8 site-node appsettings under docker/ and docker-env2/ now set AuditLog:SiteWriter:DatabasePath to /app/data/auditlog.db (mounted volume, survives container recreate, same convention as LocalDb:Path); the local-dev base appsettings.Site.json sets ./data/auditlog.db to match. - The writer loop now honors FlushIntervalMs: after draining the immediately available burst, it keeps the transaction open (bounded by FlushIntervalMs from the first event) waiting for more trickle-rate events before committing, instead of flushing (and fsyncing) per event. - PRAGMA synchronous = NORMAL on the write connection — audit is best-effort by design (CLAUDE.md: "Audit-write failure NEVER aborts the user-facing action"), so NORMAL's narrower power-loss window is an acceptable trade for far fewer fsyncs; WAL mode still guarantees no corruption. - Tests: StartupValidator site-required/blank/central-exempt cases; writer trickle-load single-transaction coalescing + beyond-interval separate-transaction regression (new FlushCountForTests seam); options-validator doc updates reflecting the moved responsibility. Full suite runs green: AuditLog.Tests 368/368, Host.Tests 480/480. One-time migration note: the existing container-local auditlog.db (wherever it landed under CWD) is abandoned by this change, not migrated — already-forwarded rows are safe centrally (AuditLog is the durable copy), and any still-Pending rows on the abandoned path are lost once. This is the exact bug being fixed, not a new loss: those rows were already living outside the mounted volume and would not have survived the next container recreate regardless. Cross-reference docs/known-issues/2026-07-20-cached-telemetry-drain-hot-loop.md, which this placement bug caused. --- .../site-x-node-a/appsettings.Site.json | 11 +++ .../site-x-node-b/appsettings.Site.json | 11 +++ docker/site-a-node-a/appsettings.Site.json | 11 +++ docker/site-a-node-b/appsettings.Site.json | 11 +++ docker/site-b-node-a/appsettings.Site.json | 11 +++ docker/site-b-node-b/appsettings.Site.json | 11 +++ docker/site-c-node-a/appsettings.Site.json | 11 +++ docker/site-c-node-b/appsettings.Site.json | 11 +++ .../Site/SqliteAuditWriter.cs | 74 ++++++++++++++++++- .../Site/SqliteAuditWriterOptions.cs | 19 ++++- .../Site/SqliteAuditWriterOptionsValidator.cs | 16 ++-- .../StartupValidator.cs | 22 ++++++ .../appsettings.Site.json | 9 +++ .../SqliteAuditWriterOptionsValidatorTests.cs | 24 +++--- .../Site/SqliteAuditWriterWriteTests.cs | 70 +++++++++++++++++- .../SiteHealthEndpointTests.cs | 8 ++ .../StartupValidatorTests.cs | 41 ++++++++++ 17 files changed, 347 insertions(+), 24 deletions(-) diff --git a/docker-env2/site-x-node-a/appsettings.Site.json b/docker-env2/site-x-node-a/appsettings.Site.json index 25262f90..9caef4df 100644 --- a/docker-env2/site-x-node-a/appsettings.Site.json +++ b/docker-env2/site-x-node-a/appsettings.Site.json @@ -78,5 +78,16 @@ // Replication is opt-in and configured separately; absent = local-only. "LocalDb": { "Path": "/app/data/site-localdb.db" + }, + // arch-review remediation WP1.2: the site hot-path audit writer's SQLite file has no + // default path (mirrors LocalDb:Path above) and StartupValidator now requires it for + // Site nodes — an unset value used to fall back to a bare "auditlog.db" resolved + // relative to CWD, i.e. the container's ephemeral overlayfs, so the pending audit + // backlog was silently discarded on every recreate. On the mounted /app/data volume, + // same as LocalDb:Path, so it survives container recreate. + "AuditLog": { + "SiteWriter": { + "DatabasePath": "/app/data/auditlog.db" + } } } diff --git a/docker-env2/site-x-node-b/appsettings.Site.json b/docker-env2/site-x-node-b/appsettings.Site.json index 586d1f5a..a45cb574 100644 --- a/docker-env2/site-x-node-b/appsettings.Site.json +++ b/docker-env2/site-x-node-b/appsettings.Site.json @@ -78,5 +78,16 @@ // Replication is opt-in and configured separately; absent = local-only. "LocalDb": { "Path": "/app/data/site-localdb.db" + }, + // arch-review remediation WP1.2: the site hot-path audit writer's SQLite file has no + // default path (mirrors LocalDb:Path above) and StartupValidator now requires it for + // Site nodes — an unset value used to fall back to a bare "auditlog.db" resolved + // relative to CWD, i.e. the container's ephemeral overlayfs, so the pending audit + // backlog was silently discarded on every recreate. On the mounted /app/data volume, + // same as LocalDb:Path, so it survives container recreate. + "AuditLog": { + "SiteWriter": { + "DatabasePath": "/app/data/auditlog.db" + } } } diff --git a/docker/site-a-node-a/appsettings.Site.json b/docker/site-a-node-a/appsettings.Site.json index 47877e02..3fb91088 100644 --- a/docker/site-a-node-a/appsettings.Site.json +++ b/docker/site-a-node-a/appsettings.Site.json @@ -120,5 +120,16 @@ "MaxOplogRows": 250000, "MaxOplogAge": "2.00:00:00" } + }, + // arch-review remediation WP1.2: the site hot-path audit writer's SQLite file has no + // default path (mirrors LocalDb:Path above) and StartupValidator now requires it for + // Site nodes — an unset value used to fall back to a bare "auditlog.db" resolved + // relative to CWD, i.e. the container's ephemeral overlayfs, so the pending audit + // backlog was silently discarded on every recreate. On the mounted /app/data volume, + // same as LocalDb:Path, so it survives container recreate. + "AuditLog": { + "SiteWriter": { + "DatabasePath": "/app/data/auditlog.db" + } } } diff --git a/docker/site-a-node-b/appsettings.Site.json b/docker/site-a-node-b/appsettings.Site.json index 676696a6..bfce17f5 100644 --- a/docker/site-a-node-b/appsettings.Site.json +++ b/docker/site-a-node-b/appsettings.Site.json @@ -113,5 +113,16 @@ "MaxOplogRows": 250000, "MaxOplogAge": "2.00:00:00" } + }, + // arch-review remediation WP1.2: the site hot-path audit writer's SQLite file has no + // default path (mirrors LocalDb:Path above) and StartupValidator now requires it for + // Site nodes — an unset value used to fall back to a bare "auditlog.db" resolved + // relative to CWD, i.e. the container's ephemeral overlayfs, so the pending audit + // backlog was silently discarded on every recreate. On the mounted /app/data volume, + // same as LocalDb:Path, so it survives container recreate. + "AuditLog": { + "SiteWriter": { + "DatabasePath": "/app/data/auditlog.db" + } } } diff --git a/docker/site-b-node-a/appsettings.Site.json b/docker/site-b-node-a/appsettings.Site.json index 84b77600..e5701e9e 100644 --- a/docker/site-b-node-a/appsettings.Site.json +++ b/docker/site-b-node-a/appsettings.Site.json @@ -86,5 +86,16 @@ // Replication is opt-in and configured separately; absent = local-only. "LocalDb": { "Path": "/app/data/site-localdb.db" + }, + // arch-review remediation WP1.2: the site hot-path audit writer's SQLite file has no + // default path (mirrors LocalDb:Path above) and StartupValidator now requires it for + // Site nodes — an unset value used to fall back to a bare "auditlog.db" resolved + // relative to CWD, i.e. the container's ephemeral overlayfs, so the pending audit + // backlog was silently discarded on every recreate. On the mounted /app/data volume, + // same as LocalDb:Path, so it survives container recreate. + "AuditLog": { + "SiteWriter": { + "DatabasePath": "/app/data/auditlog.db" + } } } diff --git a/docker/site-b-node-b/appsettings.Site.json b/docker/site-b-node-b/appsettings.Site.json index 3ca78cd1..785f9642 100644 --- a/docker/site-b-node-b/appsettings.Site.json +++ b/docker/site-b-node-b/appsettings.Site.json @@ -86,5 +86,16 @@ // Replication is opt-in and configured separately; absent = local-only. "LocalDb": { "Path": "/app/data/site-localdb.db" + }, + // arch-review remediation WP1.2: the site hot-path audit writer's SQLite file has no + // default path (mirrors LocalDb:Path above) and StartupValidator now requires it for + // Site nodes — an unset value used to fall back to a bare "auditlog.db" resolved + // relative to CWD, i.e. the container's ephemeral overlayfs, so the pending audit + // backlog was silently discarded on every recreate. On the mounted /app/data volume, + // same as LocalDb:Path, so it survives container recreate. + "AuditLog": { + "SiteWriter": { + "DatabasePath": "/app/data/auditlog.db" + } } } diff --git a/docker/site-c-node-a/appsettings.Site.json b/docker/site-c-node-a/appsettings.Site.json index d58e13b5..02d482ee 100644 --- a/docker/site-c-node-a/appsettings.Site.json +++ b/docker/site-c-node-a/appsettings.Site.json @@ -86,5 +86,16 @@ // Replication is opt-in and configured separately; absent = local-only. "LocalDb": { "Path": "/app/data/site-localdb.db" + }, + // arch-review remediation WP1.2: the site hot-path audit writer's SQLite file has no + // default path (mirrors LocalDb:Path above) and StartupValidator now requires it for + // Site nodes — an unset value used to fall back to a bare "auditlog.db" resolved + // relative to CWD, i.e. the container's ephemeral overlayfs, so the pending audit + // backlog was silently discarded on every recreate. On the mounted /app/data volume, + // same as LocalDb:Path, so it survives container recreate. + "AuditLog": { + "SiteWriter": { + "DatabasePath": "/app/data/auditlog.db" + } } } diff --git a/docker/site-c-node-b/appsettings.Site.json b/docker/site-c-node-b/appsettings.Site.json index cc155b04..78e987cb 100644 --- a/docker/site-c-node-b/appsettings.Site.json +++ b/docker/site-c-node-b/appsettings.Site.json @@ -86,5 +86,16 @@ // Replication is opt-in and configured separately; absent = local-only. "LocalDb": { "Path": "/app/data/site-localdb.db" + }, + // arch-review remediation WP1.2: the site hot-path audit writer's SQLite file has no + // default path (mirrors LocalDb:Path above) and StartupValidator now requires it for + // Site nodes — an unset value used to fall back to a bare "auditlog.db" resolved + // relative to CWD, i.e. the container's ephemeral overlayfs, so the pending audit + // backlog was silently discarded on every recreate. On the mounted /app/data volume, + // same as LocalDb:Path, so it survives container recreate. + "AuditLog": { + "SiteWriter": { + "DatabasePath": "/app/data/auditlog.db" + } } } diff --git a/src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/SqliteAuditWriter.cs b/src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/SqliteAuditWriter.cs index 62108a76..4247e592 100644 --- a/src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/SqliteAuditWriter.cs +++ b/src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/SqliteAuditWriter.cs @@ -174,6 +174,24 @@ public class SqliteAuditWriter : IAuditWriter, ISiteAuditQueue, IAsyncDisposable pragmaCmd.ExecuteNonQuery(); } + // synchronous=NORMAL (arch-review remediation WP1.2): SQLite's FULL default + // fsyncs on every transaction commit, which is the dominant cost of the + // per-event write path this same change coalesces via FlushIntervalMs — FULL + // would still fsync once per FLUSH even after that fix. NORMAL fsyncs at + // fewer, WAL-checkpoint-aligned points instead of every commit; the window + // this opens is a handful of the most recent commits lost on OS crash / power + // loss (WAL mode itself still guarantees no *corruption*, only a possible + // short rollback on unclean restart). That trade is acceptable here because + // audit writes are explicitly best-effort by design (CLAUDE.md: "Audit-write + // failure NEVER aborts the user-facing action — audit is best-effort, the + // action's own success/failure path is authoritative") — this is the same + // durability class as a dropped audit row, not a new risk class. + using (var pragmaCmd = _connection.CreateCommand()) + { + pragmaCmd.CommandText = "PRAGMA synchronous = NORMAL"; + pragmaCmd.ExecuteNonQuery(); + } + // Enable FK enforcement on the WRITE connection. PRAGMA foreign_keys is // a per-connection, per-session setting in SQLite — it is NOT persisted // in the database file, so every new connection that may INSERT into @@ -306,20 +324,67 @@ public class SqliteAuditWriter : IAuditWriter, ISiteAuditQueue, IAsyncDisposable { batch.Clear(); batch.Add(first); + var deadline = DateTime.UtcNow.AddMilliseconds(_options.FlushIntervalMs); // Pull additional ready events up to BatchSize. TryRead is non- // blocking and lets us amortise the transaction overhead across a - // burst of concurrent enqueues. - while (batch.Count < _options.BatchSize && - _writeQueue.Reader.TryRead(out var next)) + // burst of concurrent enqueues. Once the immediately-available + // burst is drained, keep the transaction open for up to + // FlushIntervalMs waiting for MORE events to trickle in — this is + // the soft-flush coalescing arch-review remediation WP1.2 adds: + // without it, a low-rate trickle of writes (one script call every + // few ms, well under BatchSize) fsyncs once PER EVENT, because the + // writer loop reaches an empty channel and flushes immediately. + // FlushIntervalMs bounds the worst-case added latency any single + // event can see from this coalescing. + while (batch.Count < _options.BatchSize) { - batch.Add(next); + if (_writeQueue.Reader.TryRead(out var next)) + { + batch.Add(next); + continue; + } + + var remaining = deadline - DateTime.UtcNow; + if (remaining <= TimeSpan.Zero) + { + break; + } + + var waitToRead = _writeQueue.Reader.WaitToReadAsync().AsTask(); + var completed = await Task.WhenAny( + waitToRead, Task.Delay(remaining)).ConfigureAwait(false); + + if (completed != waitToRead) + { + // Deadline elapsed with nothing new — flush what we have. + break; + } + + // WaitToReadAsync resolving false means the channel completed + // (Dispose) with nothing left to read; resolving true means at + // least one item is available, so the top of the loop's + // TryRead will pick it up. + if (!await waitToRead.ConfigureAwait(false)) + { + break; + } } FlushBatch(batch); } } + /// + /// Total number of transactions committed since + /// construction. Test-only observability seam (arch-review remediation + /// WP1.2) for asserting that FlushIntervalMs coalesces a trickle of + /// near-simultaneous writes into a single transaction rather than one + /// commit per event; production code never reads this. + /// + internal int FlushCountForTests => _flushCountForTests; + private int _flushCountForTests; + private void FlushBatch(IReadOnlyList batch) { lock (_writeLock) @@ -436,6 +501,7 @@ public class SqliteAuditWriter : IAuditWriter, ISiteAuditQueue, IAsyncDisposable } transaction.Commit(); + _flushCountForTests++; } catch (Exception ex) { diff --git a/src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/SqliteAuditWriterOptions.cs b/src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/SqliteAuditWriterOptions.cs index 3f558a54..ab8c1f8f 100644 --- a/src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/SqliteAuditWriterOptions.cs +++ b/src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/SqliteAuditWriterOptions.cs @@ -9,8 +9,23 @@ namespace ZB.MOM.WW.ScadaBridge.AuditLog.Site; /// public sealed class SqliteAuditWriterOptions { - /// SQLite database path (or in-memory URI for tests). - public string DatabasePath { get; set; } = "auditlog.db"; + /// + /// SQLite database path (or in-memory URI for tests). No default — + /// mirrors LocalDb:Path (ZB.MOM.WW.LocalDb.LocalDbOptions.Path): a + /// bare filename here would resolve relative to the process CWD, which on the + /// docker rig is the container's ephemeral overlayfs, not the mounted + /// /app/data volume — the file (and its pending forward-state backlog) + /// would be silently discarded on every container recreate. Site nodes MUST set + /// this explicitly (e.g. /app/data/auditlog.db); StartupValidator + /// in ZB.MOM.WW.ScadaBridge.Host enforces the requirement pre-host for Site nodes only, + /// the same way it enforces Communication:GrpcPsk. Central composition roots also + /// bind this options type (AddAuditLog is shared between roles) but never resolve + /// , so the empty default is inert there and deliberately + /// left unvalidated by (which only checks + /// the role-agnostic channel/batch/flush knobs) — enforcing it there too would fail + /// central's boot via ValidateOnStart, since AddAuditLog runs on both roles. + /// + public string DatabasePath { get; set; } = ""; /// /// Capacity of the bounded write queue. Set high enough that ordinary diff --git a/src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/SqliteAuditWriterOptionsValidator.cs b/src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/SqliteAuditWriterOptionsValidator.cs index 68c586ce..cc79e76b 100644 --- a/src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/SqliteAuditWriterOptionsValidator.cs +++ b/src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/SqliteAuditWriterOptionsValidator.cs @@ -6,19 +6,25 @@ namespace ZB.MOM.WW.ScadaBridge.AuditLog.Site; /// Validates at host startup /// (arch-review 08 round 2 NF4). The channel/batch/flush knobs feed the /// background writer task; a zero anywhere stalls it (nothing drains, nothing -/// flushes), and an empty DatabasePath leaves the SQLite writer with no -/// backing store. +/// flushes). /// is intentionally NOT validated — a non-positive value has a documented /// fall-back-to-30s contract in SiteAuditBacklogReporter. /// +/// +/// is deliberately NOT +/// validated here (arch-review remediation WP1.2). AddAuditLog runs on +/// BOTH Central and Site composition roots and binds this options type with +/// ValidateOnStart either way, but only Site nodes ever resolve +/// — requiring a non-empty path in this +/// role-agnostic validator would fail Central's boot too. The requirement is +/// instead enforced pre-host, Site-role-only, by StartupValidator in +/// ZB.MOM.WW.ScadaBridge.Host (mirrors how it gates Communication:GrpcPsk). +/// public sealed class SqliteAuditWriterOptionsValidator : OptionsValidatorBase { /// protected override void Validate(ValidationBuilder builder, SqliteAuditWriterOptions options) { - builder.RequireThat(!string.IsNullOrWhiteSpace(options.DatabasePath), - $"AuditLog:SiteWriter:{nameof(SqliteAuditWriterOptions.DatabasePath)} must be a non-empty path."); - builder.RequireThat(options.ChannelCapacity > 0, $"AuditLog:SiteWriter:{nameof(SqliteAuditWriterOptions.ChannelCapacity)} " + $"({options.ChannelCapacity}) must be > 0."); diff --git a/src/ZB.MOM.WW.ScadaBridge.Host/StartupValidator.cs b/src/ZB.MOM.WW.ScadaBridge.Host/StartupValidator.cs index e5e5f1e7..1985ff29 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Host/StartupValidator.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Host/StartupValidator.cs @@ -159,6 +159,28 @@ public static class StartupValidator + "endpoint under ScadaBridge:Communication:CentralGrpcEndpoints " + "(e.g. http://scadabridge-central-a:8083). Central nodes leave it empty."); + // The site hot-path audit writer's SQLite file (arch-review remediation + // WP1.2). SqliteAuditWriterOptions.DatabasePath has no default (mirrors + // LocalDb:Path) — an unset value would previously fall back to a bare + // "auditlog.db" resolved relative to the process CWD, which on the docker + // rig is the container's ephemeral overlayfs, not the mounted /app/data + // volume: the file, and every pending (not-yet-forwarded) audit row in it, + // was silently discarded on every container recreate. AddAuditLog binds + // this options type on BOTH roles (SqliteAuditWriterOptionsValidator + // deliberately does not check DatabasePath there, so Central's boot is + // unaffected — see that validator's remarks), so the Site-only requirement + // lives here, the same way GrpcPsk is gated just above. NOTE: unlike every + // other key in this method, AuditLog:SiteWriter is a TOP-LEVEL config + // section (AddAuditLog binds "AuditLog:SiteWriter" off the configuration + // root, not "ScadaBridge:AuditLog:SiteWriter") — no ScadaBridge: prefix. + p.Require("AuditLog:SiteWriter:DatabasePath", + value => !string.IsNullOrWhiteSpace(value), + "is required for Site nodes: the SQLite hot-path audit writer has no " + + "default path (mirrors LocalDb:Path) — an unset value would silently " + + "resolve to a CWD-relative file on the container's ephemeral overlayfs " + + "and lose the pending audit backlog on every redeploy. Point it at the " + + "mounted data volume, e.g. /app/data/auditlog.db."); + // ScadaBridge:Database:SiteDbPath was required here until LocalDb // Phase 2. The site's tables now live in the consolidated LocalDb // database (LocalDb:Path, which SiteServiceRegistration requires), diff --git a/src/ZB.MOM.WW.ScadaBridge.Host/appsettings.Site.json b/src/ZB.MOM.WW.ScadaBridge.Host/appsettings.Site.json index 397041c2..8e065162 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Host/appsettings.Site.json +++ b/src/ZB.MOM.WW.ScadaBridge.Host/appsettings.Site.json @@ -80,5 +80,14 @@ // fails to boot, so every site config must set it. "LocalDb": { "Path": "./data/site-localdb.db" + }, + // arch-review remediation WP1.2: the site hot-path audit writer's SQLite file has no + // default path (mirrors LocalDb:Path above) - AuditLog:SiteWriter:DatabasePath is + // REQUIRED and validated pre-host on Site nodes (StartupValidator), the same way + // LocalDb:Path is, so every site config must set it. + "AuditLog": { + "SiteWriter": { + "DatabasePath": "./data/auditlog.db" + } } } diff --git a/tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Site/SqliteAuditWriterOptionsValidatorTests.cs b/tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Site/SqliteAuditWriterOptionsValidatorTests.cs index 05ba0570..ca0345ef 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Site/SqliteAuditWriterOptionsValidatorTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Site/SqliteAuditWriterOptionsValidatorTests.cs @@ -5,30 +5,30 @@ namespace ZB.MOM.WW.ScadaBridge.AuditLog.Tests.Site; /// /// Eager startup validation for /// (arch-review 08 round 2 NF4). The site hot-path writer's channel/batch/flush -/// knobs must be positive or the background writer task cannot make progress; -/// an empty DatabasePath would leave the SQLite writer with nowhere to go. +/// knobs must be positive or the background writer task cannot make progress. /// BacklogPollIntervalSeconds is deliberately NOT validated — a /// non-positive value has a documented fall-back-to-30s contract. /// +/// +/// DatabasePath is intentionally NOT covered here (arch-review remediation +/// WP1.2): it defaults to "" (no CWD-relative default — mirrors +/// LocalDb:Path), but this validator runs on both Central and Site +/// composition roots via ValidateOnStart, and only Site nodes ever resolve +/// the writer. The Site-only requirement is enforced pre-host by +/// StartupValidator in ZB.MOM.WW.ScadaBridge.Host — see +/// StartupValidatorTests.SiteWithoutAuditLogDatabasePath_FailsValidation. +/// public class SqliteAuditWriterOptionsValidatorTests { [Fact] public void DefaultOptions_AreValid() { + // Includes the empty DatabasePath default — this validator does not check + // it (see class remarks); only StartupValidator does, and only for Site nodes. var validator = new SqliteAuditWriterOptionsValidator(); Assert.True(validator.Validate(null, new SqliteAuditWriterOptions()).Succeeded); } - [Fact] - public void EmptyDatabasePath_Fails() - { - var validator = new SqliteAuditWriterOptionsValidator(); - var result = validator.Validate(null, new SqliteAuditWriterOptions { DatabasePath = "" }); - Assert.False(result.Succeeded); - Assert.Contains(result.Failures!, - f => f.Contains(nameof(SqliteAuditWriterOptions.DatabasePath), StringComparison.Ordinal)); - } - [Fact] public void ZeroChannelCapacity_Fails() { diff --git a/tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Site/SqliteAuditWriterWriteTests.cs b/tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Site/SqliteAuditWriterWriteTests.cs index 9ee16e6c..db18c2a6 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Site/SqliteAuditWriterWriteTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Site/SqliteAuditWriterWriteTests.cs @@ -24,7 +24,8 @@ public class SqliteAuditWriterWriteTests private static (SqliteAuditWriter writer, string dataSource) CreateWriter( string testName, int? channelCapacity = null, - INodeIdentityProvider? nodeIdentity = null) + INodeIdentityProvider? nodeIdentity = null, + int? flushIntervalMs = null) { var dataSource = $"file:{testName}-{Guid.NewGuid():N}?mode=memory&cache=shared"; var opts = new SqliteAuditWriterOptions { DatabasePath = dataSource }; @@ -32,6 +33,10 @@ public class SqliteAuditWriterWriteTests { opts.ChannelCapacity = cap; } + if (flushIntervalMs is int flush) + { + opts.FlushIntervalMs = flush; + } // Default identity provider returns null — existing tests pre-date // SourceNode stamping and have no expectation about it. New stamping @@ -181,6 +186,69 @@ public class SqliteAuditWriterWriteTests Assert.Equal(1000, Convert.ToInt64(sidecarCmd.ExecuteScalar())); } + // ----- FlushIntervalMs soft-flush coalescing (arch-review remediation WP1.2) ----- // + + /// + /// A trickle of events — well under , + /// spaced closer together than — + /// must coalesce into ONE transaction rather than fsyncing once per event. This is the + /// bug WP1.2 fixes: FlushIntervalMs was validated but never read by the writer + /// loop, so every trickle-rate write incurred its own commit (and, under SQLite's + /// default synchronous=FULL, its own fsync). + /// + [Fact] + public async Task WriteAsync_TrickleLoad_CoalescesIntoOneTransaction() + { + var (writer, dataSource) = CreateWriter( + nameof(WriteAsync_TrickleLoad_CoalescesIntoOneTransaction), + flushIntervalMs: 500); + await using var _ = writer; + + const int count = 5; + var writeTasks = new List(count); + for (int i = 0; i < count; i++) + { + writeTasks.Add(writer.WriteAsync(NewEvent())); + await Task.Delay(15); // spaced well under the 500ms flush window + } + await Task.WhenAll(writeTasks); + + using var connection = OpenVerifierConnection(dataSource); + using var cmd = connection.CreateCommand(); + cmd.CommandText = "SELECT COUNT(*) FROM audit_event;"; + Assert.Equal(count, Convert.ToInt64(cmd.ExecuteScalar())); + + // The defining assertion: all 5 trickled-in writes landed in a single + // FlushBatch transaction, not up to 5 separate commits. + Assert.Equal(1, writer.FlushCountForTests); + } + + /// + /// The flip side: two events spaced FURTHER apart than + /// must NOT be held open + /// waiting for a partner — the deadline bounds worst-case added latency, it does + /// not turn into an unbounded debounce. + /// + [Fact] + public async Task WriteAsync_EventsSpacedBeyondFlushInterval_FlushSeparately() + { + var (writer, dataSource) = CreateWriter( + nameof(WriteAsync_EventsSpacedBeyondFlushInterval_FlushSeparately), + flushIntervalMs: 50); + await using var _ = writer; + + await writer.WriteAsync(NewEvent()); + await Task.Delay(300); // well beyond the 50ms flush window + await writer.WriteAsync(NewEvent()); + + using var connection = OpenVerifierConnection(dataSource); + using var cmd = connection.CreateCommand(); + cmd.CommandText = "SELECT COUNT(*) FROM audit_event;"; + Assert.Equal(2, Convert.ToInt64(cmd.ExecuteScalar())); + + Assert.Equal(2, writer.FlushCountForTests); + } + [Fact] public async Task WriteAsync_DuplicateEventId_FirstWriteWins_NoException() { diff --git a/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteHealthEndpointTests.cs b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteHealthEndpointTests.cs index 5be47b9e..6f0718d7 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteHealthEndpointTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteHealthEndpointTests.cs @@ -27,10 +27,12 @@ public class SiteHealthEndpointTests : IDisposable private readonly List _disposables = new(); private readonly Dictionary _previousEnv = new(StringComparer.Ordinal); private readonly string _tempDbPath; + private readonly string _tempAuditDbPath; public SiteHealthEndpointTests() { _tempDbPath = Path.Combine(Path.GetTempPath(), $"scadabridge_health_ep_{Guid.NewGuid()}.db"); + _tempAuditDbPath = Path.Combine(Path.GetTempPath(), $"scadabridge_health_ep_audit_{Guid.NewGuid()}.db"); // Whole-key env overrides, the sanctioned path: supplying GrpcPsk concretely makes the // pre-host secrets expander skip appsettings.Site.json's ${secret:SB-GRPC-PSK-site-1} @@ -50,6 +52,11 @@ public class SiteHealthEndpointTests : IDisposable SetEnv("ScadaBridge__Cluster__SeedNodes__1", "akka.tcp://scadabridge@localhost:18085"); SetEnv("ScadaBridge__Communication__GrpcPsk", "test-psk-0123456789"); SetEnv("LocalDb__Path", _tempDbPath); + // arch-review remediation WP1.2: SqliteAuditWriterOptions.DatabasePath has no default + // and StartupValidator now requires it for Site nodes — same reason LocalDb__Path is + // overridden above (appsettings.Site.json's own default is CWD-relative and would + // otherwise litter the test working directory when SiteAuditBacklogReporter probes it). + SetEnv("AuditLog__SiteWriter__DatabasePath", _tempAuditDbPath); } private void SetEnv(string key, string? value) @@ -71,6 +78,7 @@ public class SiteHealthEndpointTests : IDisposable } try { File.Delete(_tempDbPath); } catch { /* best effort */ } + try { File.Delete(_tempAuditDbPath); } catch { /* best effort */ } GC.SuppressFinalize(this); } diff --git a/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/StartupValidatorTests.cs b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/StartupValidatorTests.cs index f00544b0..c489e9e9 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/StartupValidatorTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/StartupValidatorTests.cs @@ -44,6 +44,10 @@ public class StartupValidatorTests // Phase 4: gRPC (CentralControlService) is the only site→central transport, so a Site // node must list at least one central gRPC endpoint to dial (no Akka fallback remains). ["ScadaBridge:Communication:CentralGrpcEndpoints:0"] = "http://central-a:8083", + // arch-review remediation WP1.2: the site hot-path audit writer's SQLite file has no + // default path (mirrors LocalDb:Path) — required so it lands on the mounted data volume, + // not the container's ephemeral overlayfs. + ["AuditLog:SiteWriter:DatabasePath"] = "/app/data/auditlog.db", }; [Fact] @@ -124,6 +128,43 @@ public class StartupValidatorTests Assert.Null(Record.Exception(() => StartupValidator.Validate(config))); } + [Fact] + public void SiteWithoutAuditLogDatabasePath_FailsValidation() + { + // arch-review remediation WP1.2: SqliteAuditWriterOptions.DatabasePath has no default + // (mirrors LocalDb:Path). An unset value used to fall back to a bare "auditlog.db" + // resolved relative to CWD — on the docker rig, the container's ephemeral overlayfs — + // so the node booted, looked healthy, and lost its pending audit backlog on every + // recreate. Fail fast at boot instead. + var values = ValidSiteConfig(); + values.Remove("AuditLog:SiteWriter:DatabasePath"); + var config = BuildConfig(values); + + var ex = Assert.Throws(() => StartupValidator.Validate(config)); + Assert.Contains("DatabasePath", ex.Message); + } + + [Fact] + public void SiteWithBlankAuditLogDatabasePath_FailsValidation() + { + var values = ValidSiteConfig(); + values["AuditLog:SiteWriter:DatabasePath"] = " "; + var config = BuildConfig(values); + + var ex = Assert.Throws(() => StartupValidator.Validate(config)); + Assert.Contains("DatabasePath", ex.Message); + } + + [Fact] + public void CentralWithoutAuditLogDatabasePath_PassesValidation() + { + // AddAuditLog binds SqliteAuditWriterOptions on both roles, but only Site nodes ever + // resolve the writer — the requirement must not fire for Central. + var config = BuildConfig(ValidCentralConfig()); + + Assert.Null(Record.Exception(() => StartupValidator.Validate(config))); + } + [Fact] public void MissingRole_FailsValidation() { From 1040dc0fcc4a4a0fea9673c7db218e1b16a26b76 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Fri, 14 Aug 2026 20:30:35 -0400 Subject: [PATCH 09/31] =?UTF-8?q?docs(plans):=20DCL=20batch=20seam=20desig?= =?UTF-8?q?n=20=E2=80=94=20WP2.1a?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../plans/2026-08-14-dcl-batch-seam-design.md | 103 ++++++++++++++++++ 1 file changed, 103 insertions(+) create mode 100644 docs/plans/2026-08-14-dcl-batch-seam-design.md diff --git a/docs/plans/2026-08-14-dcl-batch-seam-design.md b/docs/plans/2026-08-14-dcl-batch-seam-design.md new file mode 100644 index 00000000..ca1ff105 --- /dev/null +++ b/docs/plans/2026-08-14-dcl-batch-seam-design.md @@ -0,0 +1,103 @@ +# DCL Batch Seam Design — WP2.1a + +**Date:** 2026-08-14 · **Scope:** design contract for WP2.1b (implementation) · **Findings:** arch-review #3 (High) + MxGateway 2-RPC subscribe / no-backoff retry (Med) + quality-counter and alarm-fanout costs (Med) · **Sized for:** 37,500 tags/site (500 instances × 75 tags, `docs/deployment/topology-guide.md:255`). + +## 1. Batch API on `IDataConnection` + +```csharp +public record TagSubscribeResult(string TagPath, bool Success, string? SubscriptionId, string? ErrorMessage); + +Task> SubscribeBatchAsync( + IReadOnlyList tagPaths, SubscriptionCallback callback, CancellationToken ct = default); +Task UnsubscribeBatchAsync(IReadOnlyList subscriptionIds, CancellationToken ct = default); +``` + +- **One shared callback, plain tag list** — deliberately simpler than the plan's sketched `IReadOnlyList` input record. The only caller passes an identical closure for every tag (`DataConnectionActor.HandleSubscribe`, `DataConnectionActor.cs:745-748`: `(path, value) => self.Tell(new TagValueReceived(path, value, generation))`), and `SubscriptionCallback` already delivers the tag path. A per-tag record would carry nothing. +- **Partial-failure contract:** per-tag faults (bad node id, resolution failure) come back as result rows with `Success:false` and never throw — mirroring the gateway's `SubscribeResult`/`BulkSubscribeReply` (mxaccess_gateway.proto:632) and OPC UA per-item create status. A **thrown** exception means the whole batch failed at connection level; the actor classifies it with the existing `IsConnectionLevelFailure` (`DataConnectionActor.cs:1056`) and drives Reconnecting, exactly as the per-tag path does today (`SubscribeTagResult.ConnectionLevelFailure`, line 758-761). +- **`ReadBatchAsync`/`WriteBatchAsync` keep their existing signatures** (`IDataConnection.cs:47,58` — dictionary-of-results, already right) but the contract is strengthened from "may loop over singles" to **true bulk**: `OpcUaDataConnection` currently loops per tag (`OpcUaDataConnection.cs:250-274, 289-315`); it becomes one `ReadValueIdCollection`/`WriteValueCollection` service call (the MxGateway adapter is the in-repo reference — `MxGatewayDataConnection.cs:267-290` already delegates to `ReadBulkAsync`/`WriteBulkAsync`). Adapters chunk internally at 1,000 items per service call to stay under typical OPC UA `MaxNodesPerRead/Write` operation limits. Preserved semantics: per-tag failure rows, `OperationCanceledException` aborts the whole batch (`OpcUaDataConnection.cs:263-267`). +- **Single-tag methods are KEPT**, implemented as batch-of-one delegations. They have real callers outside the hot path: the OPC UA heartbeat monitor (`OpcUaDataConnection.cs:145`), `WriteBatchAndWaitAsync`'s poll loop (line 355), `HandleWrite`/`HandleReadTagValues`. Removing them churns every test fake for zero gain; the actor's hot paths simply stop using them. + +## 2. OPC UA: sharding + apply serialization + +- **Budget option:** `OpcUaEndpointConfig.MaxMonitoredItemsPerSubscription`, default **5000**, flowing into `OpcUaConnectionOptions` like the other subscription knobs (`OpcUaDataConnection.cs:74-96`). Per-endpoint, not global — item-count ceilings are a property of the target server. 37,500 tags → 8 shards. +- `RealOpcUaClient` replaces the single `_subscription` (`RealOpcUaClient.cs:22,150-162`) with a shard list. Placement: first shard with free capacity, else create a new shard from the same options; a shard is deleted when it empties. A `handle → shard` map serves removal and keeps `RemoveSubscriptionsAsync` grouping one `ApplyChangesAsync` per touched shard. +- **Alarm (A&C) monitored items get a dedicated event shard.** `TriggerConditionRefreshAsync` passes a subscription id (`RealOpcUaClient.cs:784-786`); pinning event items to one shard keeps that bookkeeping trivial and keeps `QueueSize:1000` event items from occupying data-shard capacity. +- **Batch subscribe = N×`AddItem` + one `ApplyChangesAsync` per touched shard per chunk** (today it is one Apply per tag — `RealOpcUaClient.cs:494-495`, the core of finding #3). New `IOpcUaClient` members: `CreateSubscriptionsAsync`, `RemoveSubscriptionsAsync`, `ReadValuesAsync`, `WriteValuesAsync`, all with per-item results. +- **All `ApplyChangesAsync`/`CreateAsync`/`DeleteAsync` calls serialize behind ONE `SemaphoreSlim(1,1)` per client.** The SDK Subscription is not safe under concurrent structural mutation, and today nothing prevents `HandleSubscribe`'s background task, the resolution-retry tick, and an alarm subscribe from applying concurrently. Batching makes contention rare; the single lock (rather than per-shard) also serializes shard creation and is the simplest thing that is correct. + +## 3. Reconnect orchestration & re-seed + +Two distinct paths, deliberately staggered differently: + +- **Instance-driven subscribes** (site failover): DeploymentManager already staggers instance creation (`SiteRuntimeOptions.StartupBatchSize` 20 / `StartupBatchDelayMs` 100, `SiteRuntimeOptions.cs:13-19`), so `SubscribeTagsRequest`s arrive pre-spaced at ~75 tags each. Each request becomes ONE `SubscribeBatchAsync` (a single chunk). **The DCL adds no extra delay on this path** — double-staggering would only slow failover. +- **Adapter-level reconnect** (`ReSubscribeAll`, `DataConnectionActor.cs:1600-1684`): today it fires 37,500 independent `SubscribeAsync` tasks in a tight loop (lines 1642-1653). It becomes sequential chunks of **`SubscribeBatchSize` = 500** with **`SubscribeBatchDelay` = 50 ms** between chunks, all inside one background task carrying the captured adapter + generation; each chunk pipes one completion message (per-tag results inside) back to the actor. 75 chunks ≈ 15-25 s at realistic RTTs — versus today's unbounded task storm. Sequential (not parallel) chunks: subscription creation on one session is serialized by the apply lock anyway, and sequencing bounds device load. +- **`SeedTagsAsync`** (`DataConnectionActor.cs:800-851`) currently reads **per tag, serially**, each with a 30 s timeout — the 30 s-per-tag worst case the plan calls out. It becomes chunked `ReadBatchAsync` calls: **`SeedReadBatchSize` = 250**, **`SeedReadMaxParallelism` = 4** concurrent chunks, `SeedReadTimeout` (existing, 30 s) now applying **per chunk**, under an **overall deadline `SeedOverallTimeout` = 120 s** enforced by one linked CTS across all chunks and retry rounds. The existing retry semantics survive unchanged in shape: still-empty tags re-read up to `SeedReadMaxAttempts` (3) with `SeedReadRetryDelay` (250 ms) between rounds; on deadline expiry the remaining tags are logged and stay Uncertain until a change notification — the exact behavior documented at `DataConnectionOptions.cs:17-25`. Chunk size stays moderate on purpose: the "some gateways time out on a large batch" caveat that motivated per-tag reads (`DataConnectionActor.cs:792-793`) is answered by chunking + per-chunk timeout, not by giving up bulk. Sizing: 150 chunks / 4-way ≈ 38 waves; ~20-40 s typical, so 120 s has honest headroom. +- All new knobs live on `DataConnectionOptions` (validated in `DataConnectionOptionsValidator`): `SubscribeBatchSize`, `SubscribeBatchDelay`, `SeedReadBatchSize`, `SeedReadMaxParallelism`, `SeedOverallTimeout`. + +## 4. Tag-resolution retry backoff + +Today: fixed 10 s periodic timer, one `SubscribeAsync` task per unresolved tag per tick (`DataConnectionActor.cs:1536-1574`). A dead device with thousands of unresolved tags probes forever at full width. + +- **Exponential backoff:** interval starts at `TagResolutionRetryInterval` (existing, 10 s), doubles per fully-failed round, capped at **`TagResolutionRetryMaxInterval` = 5 min** (new option). Reset to the floor when any tag resolves or on reconnect (`ReSubscribeAll` already clears `_unresolvedTags`, line 1617). +- **Batched probes:** each tick issues `SubscribeBatchAsync` over the not-in-flight unresolved set, chunked at `SubscribeBatchSize` — not N single subscribes. `_resolutionInFlight` dedup survives at chunk granularity. +- **Single-shot timer, rescheduled on probe completion** — a periodic timer cannot back off. This inherently preserves the anti-starvation property the current code defends with its `IsTimerActive` gate (`DataConnectionActor.cs:1015-1028`): the next tick is scheduled only when the previous round's completion message arrives, so a fan-out of failures can never reset the clock. + +## 5. MxGateway strategy — no cross-repo change needed for the main path + +Checked `~/Desktop/MxAccessGateway`: the proto **already has** the bulk family — `SubscribeBulkCommand` (AddItem + Advise per tag in one worker STA pass, one gRPC round trip; `mxaccess_gateway.proto:325`, worker `MxAccessSession.SubscribeBulk`), `UnsubscribeBulkCommand`, `AddItemBulkCommand`, `ReadBulkCommand`, `WriteBulkCommand`, all with per-item results. The **shipped `ZB.MOM.WW.MxGateway.Client` 0.2.0** — the exact version this repo pins (`Directory.Packages.props:98`) — exposes `SubscribeBulkAsync`/`AddItemBulkAsync`/`UnsubscribeBulkAsync` (verified in the packaged DLL, `dist/ZB.MOM.WW.MxGateway.Client.0.2.0.nupkg`). So: + +- **Plain-advise mode** (a configured write-user): `SubscribeBatchAsync` = chunked `SubscribeBulkAsync` — the 2-RPC-per-tag subscribe (`RealMxGatewayClient.cs:93-103`: `AddItemAsync` + `AdviseAsync`) collapses to one RPC per 500 tags. Unsubscribe = `UnsubscribeBulkAsync`. +- **Supervisory mode** (`WriteUserId == 0`, `RealMxGatewayClient.cs:52`): the worker has **no bulk supervisory advise** (`SubscribeBulk` issues plain `Advise`; only single `AdviseSupervisory` exists). Fallback per plan: one `AddItemBulkAsync` for the chunk, then client-side **pipelined** per-item `AdviseSupervisory` commands with bounded parallelism **`MxSupervisoryAdviseParallelism` = 16** (new option) — 37,500 tags ≈ 2,344 in-flight windows instead of 75,000 serial round trips. The existing `Lazy` advise-once map (`RealMxGatewayClient.cs:47`) is pre-populated from the bulk path so a concurrent write still awaits the same advise. +- **Cross-repo follow-up (note, not a blocker):** add an additive `bool supervisory` field to `SubscribeBulkCommand` (or an `AdviseSupervisoryBulkCommand`) in mxaccessgw so supervisory sites also get one-RPC subscribe. File against mxaccessgw; ScadaBridge picks it up on the next client bump with a capability/`Unimplemented` fallback to the pipelined path. +- Read/Write are already true bulk in this adapter — no change. + +## 6. Quality-counter flush + +`HandleTagValueReceived` calls `_healthCollector.UpdateTagQuality` on **every** value received, even when quality is unchanged (`DataConnectionActor.cs:1811`). New policy: in-actor bucket math stays per-message (cheap), but the collector push is **transition-gated + coalesced** — skip entirely when `prevQuality == newQuality`; on a genuine transition set a dirty flag and flush on a **`QualityFlushInterval` = 1 s** single-shot timer. Health reports poll at 30 s, so 1 s coalescing loses nothing. **Immediate synchronous flushes are retained** where correctness depends on them: `PushBadQualityForAllTags` on disconnect (line 1595), unsubscribe (line 1135), and the `ReSubscribeAll` reset (line 1635). + +## 7. Alarm subscriber prefix index + gateway union filter + +- **Index:** `HandleAlarmTransitionReceived` scans every subscribed source and does two `StartsWith` per source per transition (`DataConnectionActor.cs:1974-1979`). Replace with buckets keyed by the source reference's **first path segment** (split on `.`): lookup extracts the first segment of `SourceObjectReference`/`SourceReference` and tests only that bucket. Sources whose reference contains no separator (a prefix shorter than one full segment) go to a small linear residue list — exact semantics preserved, including the per-source condition-type gate and the `SnapshotComplete` broadcast-to-all (lines 1965-1972), which bypasses the index entirely. +- **Gateway union filter:** the MxGateway alarm stream is opened **gateway-wide** today (`alarmFilterPrefix: null`, `MxGatewayDataConnection.cs:231-236`). `StreamAlarmsRequest` carries a single `alarm_filter_prefix` (`mxaccess_gateway.proto:986`), so the pushable union is the **longest common prefix** of the active source references. The adapter recomputes the LCP on subscribe; the stream restarts (cancel + reopen — the source replays a fresh snapshot, which `NativeAlarmActor` already handles) **only when a new source falls outside the current prefix**. Unsubscribes never restart — a too-broad prefix is only bandwidth. The prefix stays a bandwidth optimisation, never correctness: the actor's per-source + condition-type gate remains authoritative, mirroring the OPC UA WhereClause stance (`RealOpcUaClient.cs:659-662`). + +## 8. Behaviors that MUST be preserved (stage-b checklist) + +1. **Become/Stash lifecycle** — Connecting/Connected/Reconnecting states, stash rules per state (`DataConnectionActor.cs:249-550`) unchanged. +2. **Generation fencing** — batch callbacks capture `_adapterGeneration` exactly as the per-tag closures do (`:697,1637`); `TagValueReceived`/`AlarmTransitionReceived` still drop stale generations (`:1771,1949`). Chunked reconnect runs against the adapter captured at chunk-dispatch time (S7 discipline, `:698-704`). +3. **Immediate bad quality on disconnect** — `PushBadQualityForAllTags` stays synchronous in `BecomeReconnecting` (`:473`), never deferred to the quality-flush timer. +4. **Transparent re-subscribe** — derived from `_subscriptionsByInstance` (durable truth, `:1602-1607`), `_instancesByTag` NOT cleared (`:1612-1615`), reconnect re-seed included (`:1655-1683`). +5. **Write failures synchronous to the calling script** — `HandleWrite`/`HandleWriteBatch` `PipeTo` with `WriteTimeout` translation unchanged (`:1141-1242`). +6. **Seed-after-registration** — seeds still ride `SubscribeCompleted` and are delivered only after `_subscriptionsByInstance` registration (the static-tag fix, `:765-782,999-1012`); only resolved tags are seeded. +7. **In-flight orphan guards** — unsubscribe-during-subscribe releases adapter handles (`:860-904`), duplicate alarm feeds released not overwritten (`:1908-1932`); batch completions apply per-tag through the same guard logic. +8. **Unresolved-tag semantics** — Bad-quality `TagValueUpdate` signal (`:991-995`), `_subscribesInFlight`/`AlreadySubscribed` dedup (`:706-729`), counter promotion rules (`:938-996`). +9. **Alarm filter last-writer-wins warning** (`:1841-1848`) and the client-side gate as sole authority. + +## 9. Stage-b test plan (fake-client) + +Existing DCL suite (`tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/` — actor lifecycle, generation fencing, alarm routing) must stay green. New pins: + +- **One apply per chunk:** fake `IOpcUaClient` counts `ApplyChangesAsync` calls — N tags through `SubscribeBatchAsync` → `ceil(N/SubscribeBatchSize)` applies, not N. +- **Sharding:** 12,001 tags with budget 5,000 → 3 shards; removal empties → shard deleted; alarm item lands on the event shard and `ConditionRefresh` targets it. +- **Apply serialization:** fake apply with an injected delay + concurrency flag — concurrent batch subscribe / unsubscribe / alarm subscribe never overlap an apply. +- **Re-seed deadline:** fake `ReadBatchAsync` that hangs → seed returns at `SeedOverallTimeout`, remaining tags pending/Uncertain, `SubscribeCompleted` still delivered and acked. +- **Backoff caps:** all-fail probe rounds produce intervals 10 s, 20 s, 40 s … capped at `TagResolutionRetryMaxInterval` (compressed values in-test); any success resets to the floor; probes are batched (one client call per chunk, not per tag). +- **MxGateway modes:** plain mode → exactly one `SubscribeBulkAsync` per chunk; supervisory mode → one `AddItemBulkAsync` + per-item advises with in-flight count ≤ `MxSupervisoryAdviseParallelism`. +- **Partial failure surface:** a per-tag failure row → `_unresolvedTags` + Bad-quality push, response `Success:true`; a thrown batch → all-tags connection-level → `SubscribeTagsResponse(Success:false)` + Reconnecting. +- **Quality flush:** M updates at unchanged quality → zero collector calls; one transition → one flush within the interval; disconnect flush still immediate. +- **Alarm index equivalence:** routing decisions identical to the linear scan over a sample set including shared-prefix sources, sub-segment prefixes (residue list), and the empty-source `SnapshotComplete` broadcast. LCP stream restarts only when a new source escapes the current prefix. + +## 10. Defaults summary + +| Option | Home | Default | +|---|---|---| +| `MaxMonitoredItemsPerSubscription` | `OpcUaEndpointConfig` | 5000 | +| `SubscribeBatchSize` | `DataConnectionOptions` | 500 | +| `SubscribeBatchDelay` | `DataConnectionOptions` | 50 ms | +| `SeedReadBatchSize` | `DataConnectionOptions` | 250 | +| `SeedReadMaxParallelism` | `DataConnectionOptions` | 4 | +| `SeedOverallTimeout` | `DataConnectionOptions` | 120 s | +| `TagResolutionRetryMaxInterval` | `DataConnectionOptions` | 5 min | +| `QualityFlushInterval` | `DataConnectionOptions` | 1 s | +| `MxSupervisoryAdviseParallelism` | `DataConnectionOptions` | 16 | + +Existing options keep their meanings; `SeedReadTimeout` (30 s) becomes per-chunk rather than per-tag. From 8c0b36b2fa5d07273cc8fc1f6ea2b91bd8d8fabb Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Fri, 14 Aug 2026 20:53:38 -0400 Subject: [PATCH 10/31] perf(ui): shared KPI cache, live-cache-backed alarm summary, coalesced debug renders --- .../Pages/Deployment/DebugView.razor | 162 ++++++++++++++--- .../Pages/Deployment/Deployments.razor | 58 ++++++ .../Pages/Monitoring/AlarmSummary.razor | 43 ++++- .../Components/Pages/Monitoring/Health.razor | 73 ++++++-- .../Notifications/NotificationKpis.razor | 32 ++-- .../ScriptAnalysis/ScriptAnalysisService.cs | 36 ++++ .../ServiceCollectionExtensions.cs | 12 +- .../Services/AlarmSummaryService.cs | 25 ++- .../Services/IKpiSnapshotCache.cs | 70 ++++++++ .../Services/KpiSnapshotCache.cs | 122 +++++++++++++ .../Services/SharedAlarmSummaryService.cs | 115 ++++++++++++ .../Services/SingleFlightMemo.cs | 136 ++++++++++++++ .../DebugViewRenderCoalescingTests.cs | 144 +++++++++++++++ .../DeploymentsReloadDebounceTests.cs | 127 +++++++++++++ .../Monitoring/AlarmSummaryVirtualizeTests.cs | 113 ++++++++++++ .../Pages/HealthPageTests.cs | 8 + .../Pages/NotificationKpisPageTests.cs | 7 + .../SharedAlarmSummaryServiceTests.cs | 169 ++++++++++++++++++ .../Services/SingleFlightMemoTests.cs | 166 +++++++++++++++++ 19 files changed, 1554 insertions(+), 64 deletions(-) create mode 100644 src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/IKpiSnapshotCache.cs create mode 100644 src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/KpiSnapshotCache.cs create mode 100644 src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/SharedAlarmSummaryService.cs create mode 100644 src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/SingleFlightMemo.cs create mode 100644 tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Deployment/DebugViewRenderCoalescingTests.cs create mode 100644 tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Deployment/DeploymentsReloadDebounceTests.cs create mode 100644 tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Monitoring/AlarmSummaryVirtualizeTests.cs create mode 100644 tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Services/SharedAlarmSummaryServiceTests.cs create mode 100644 tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Services/SingleFlightMemoTests.cs diff --git a/src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Deployment/DebugView.razor b/src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Deployment/DebugView.razor index 7238786c..0051a151 100644 --- a/src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Deployment/DebugView.razor +++ b/src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Deployment/DebugView.razor @@ -291,13 +291,56 @@ .OrderBy(a => a.AttributeName) .ToList(); - /// Attribute composition forest, rebuilt from the live latest-per-name dictionary. - private IReadOnlyList AttributeForest => - DebugTreeBuilder.BuildAttributeTree(_attributeValues.Values, _attrFilter); + // ── Tree memoization (arch-review WP2.4) ────────────────────────────────── + // Both forests used to be rebuilt from scratch on EVERY render — and a render + // fired per streamed event. A busy instance therefore rebuilt two whole + // composition trees hundreds of times a second, on the circuit's single + // dispatcher thread. The rebuild now sits behind a version stamp bumped + // whenever the underlying dictionaries change (a coalesced stream flush, + // Connect, Disconnect), so an unrelated re-render (tab switch, toast, status + // strip tick) reuses the forest instead of rebuilding it. Reusing the same + // instance also lets Blazor skip re-rendering the TreeView subtree entirely. + // + // The cached key includes the dictionary Count as well as the version: the + // trees must never go stale if the dictionaries are populated by some path + // that forgets to bump the stamp. + private int _dataVersion; - /// Alarm composition forest, rebuilt from the live latest-per-name dictionary. - private IReadOnlyList AlarmForest => - DebugTreeBuilder.BuildAlarmTree(_alarmStates.Values, _alarmFilter); + private IReadOnlyList? _attrForestCache; + private (int Version, int Count, string Filter) _attrForestKey = (-1, -1, string.Empty); + + private IReadOnlyList? _alarmForestCache; + private (int Version, int Count, string Filter) _alarmForestKey = (-1, -1, string.Empty); + + /// Attribute composition forest, rebuilt only when the underlying data or filter changed. + private IReadOnlyList AttributeForest + { + get + { + var key = (_dataVersion, _attributeValues.Count, _attrFilter); + if (_attrForestCache is null || _attrForestKey != key) + { + _attrForestCache = DebugTreeBuilder.BuildAttributeTree(_attributeValues.Values, _attrFilter); + _attrForestKey = key; + } + return _attrForestCache; + } + } + + /// Alarm composition forest, rebuilt only when the underlying data or filter changed. + private IReadOnlyList AlarmForest + { + get + { + var key = (_dataVersion, _alarmStates.Count, _alarmFilter); + if (_alarmForestCache is null || _alarmForestKey != key) + { + _alarmForestCache = DebugTreeBuilder.BuildAlarmTree(_alarmStates.Values, _alarmFilter); + _alarmForestKey = key; + } + return _alarmForestCache; + } + } private DebugStreamSession? _session; private ToastNotification _toast = default!; @@ -458,6 +501,9 @@ foreach (var al in session.InitialSnapshot.AlarmStates) _alarmStates[al.AlarmName] = al; + // Snapshot seeding is a data change like any other — bump the stamp so + // the memoized forests rebuild from it. + _dataVersion++; _snapshot = session.InitialSnapshot; _connected = true; @@ -492,6 +538,11 @@ _snapshot = null; _attributeValues.Clear(); _alarmStates.Clear(); + // Drop anything the stream buffered for the session we just tore down, so a + // late flush can't repopulate a disconnected view. + _pendingAttributes.Clear(); + _pendingAlarms.Clear(); + _dataVersion++; } /// @@ -508,37 +559,108 @@ _toast.ShowInfo("Cleared previous session — select a site and instance to begin.", autoDismissMs: 5000); } + // ── Render coalescing (arch-review WP2.4) ───────────────────────────────── + // Every streamed event used to marshal onto the dispatcher and call + // StateHasChanged individually, so a chatty instance drove one full render + // (and two full tree rebuilds) per value change. Events now land in a + // thread-safe pending map keyed by name — repeated updates to the same tag + // inside one window collapse to the latest — and exactly ONE dispatcher + // marshal per window drains them, bumps the version stamp and renders once. + private const int RenderCoalesceMs = 250; + + private readonly System.Collections.Concurrent.ConcurrentDictionary _pendingAttributes = new(); + private readonly System.Collections.Concurrent.ConcurrentDictionary _pendingAlarms = new(); + + /// 0/1 flag: whether a coalesce window is already armed. Interlocked-guarded. + private int _flushArmed; + /// /// Handles one debug-stream event. The callback is invoked on an Akka/gRPC /// thread, but / are /// instances also enumerated by the /// render thread (the tree forests + are /// built from them). Dictionary is not thread-safe (CentralUI-021): a - /// write racing an enumeration can throw or corrupt the buckets. The mutation - /// is therefore marshalled onto the renderer's dispatcher via - /// so every access to the dictionaries — read - /// and write — happens on the render thread. + /// write racing an enumeration can throw or corrupt the buckets. The event is + /// therefore parked in a concurrent pending map here and applied on the + /// renderer's dispatcher by , so every access to + /// the render dictionaries — read and write — still happens on one thread. /// private void HandleStreamEvent(object evt) { // CentralUI-009: the component may have been disposed while this event // was in flight on the Akka/gRPC thread. if (_disposed) return; - _ = SafeInvokeAsync(() => + + switch (evt) { + case AttributeValueChanged av: + _pendingAttributes[av.AttributeName] = av; + break; + case AlarmStateChanged al: + _pendingAlarms[al.AlarmName] = al; + break; + default: + // Unknown event type — nothing to apply, no render needed. + return; + } + + // Arm the window if it isn't already. Losers of the CAS have nothing to do: + // their event is in the pending map and the armed flush will pick it up. + if (Interlocked.CompareExchange(ref _flushArmed, 1, 0) == 0) + { + _ = FlushPendingAsync(); + } + } + + /// + /// Waits out the coalesce window, then applies everything buffered in one + /// dispatcher pass and renders once. + /// + private async Task FlushPendingAsync() + { + try + { + await Task.Delay(RenderCoalesceMs); + } + catch + { + // Delay never faults in practice; if it ever did, fall through and drain. + } + + if (_disposed) + { + Volatile.Write(ref _flushArmed, 0); + return; + } + + await SafeInvokeAsync(() => + { + // Disarm BEFORE draining: an event arriving mid-drain then arms a fresh + // window rather than being stranded until the next unrelated event. + Volatile.Write(ref _flushArmed, 0); if (_disposed) return; - switch (evt) + + var applied = false; + foreach (var key in _pendingAttributes.Keys) { - case AttributeValueChanged av: + if (_pendingAttributes.TryRemove(key, out var av)) + { _attributeValues[av.AttributeName] = av; - break; - case AlarmStateChanged al: - _alarmStates[al.AlarmName] = al; - break; - default: - // Unknown event type — no re-render needed. - return; + applied = true; + } } + foreach (var key in _pendingAlarms.Keys) + { + if (_pendingAlarms.TryRemove(key, out var al)) + { + _alarmStates[al.AlarmName] = al; + applied = true; + } + } + + if (!applied) return; + + _dataVersion++; StateHasChanged(); }); } diff --git a/src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Deployment/Deployments.razor b/src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Deployment/Deployments.razor index 33b8ccd8..69bedd7a 100644 --- a/src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Deployment/Deployments.razor +++ b/src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Deployment/Deployments.razor @@ -236,13 +236,66 @@ DeploymentStatusNotifier.StatusChanged += OnDeploymentStatusChanged; } + // ── Push coalescing (arch-review WP2.4) ─────────────────────────────────── + // One reload = every deployment record + every instance, re-read and re-filtered. + // The notifier fires per status WRITE, so a site deploy of N instances produced + // 2N+ of those full reloads back to back, per circuit. The reload is now + // leading-edge debounced: the first push after an idle gap still reloads + // immediately (a single deployment stays as responsive as before), and every + // push inside the window is absorbed into ONE trailing reload. + private const int ReloadDebounceMs = 500; + + private DateTimeOffset _lastReloadAt = DateTimeOffset.MinValue; + + /// Coalescing timer for pushes arriving inside the debounce window. Disposed with the component. + private Timer? _coalesceTimer; + private void OnDeploymentStatusChanged(ZB.MOM.WW.ScadaBridge.DeploymentManager.DeploymentStatusChange change) { // CentralUI-022: a callback racing disposal must not touch the component. if (_disposed || !_autoRefresh) return; + + lock (_coalesceLock) + { + if (_disposed) return; + + var sinceLast = DateTimeOffset.UtcNow - _lastReloadAt; + if (sinceLast >= TimeSpan.FromMilliseconds(ReloadDebounceMs)) + { + // Idle — reload straight away (leading edge). + _lastReloadAt = DateTimeOffset.UtcNow; + } + else + { + // Inside the window: arm one trailing reload for the remainder, and + // let any further push in this window ride it. + if (_coalesceTimer is not null) return; + + var delay = TimeSpan.FromMilliseconds(ReloadDebounceMs) - sinceLast; + _coalesceTimer = new Timer(_ => OnCoalesceElapsed(), null, delay, Timeout.InfiniteTimeSpan); + return; + } + } + _ = DispatchReloadAsync(); } + /// Trailing edge of the debounce: disarm, stamp, and run the one coalesced reload. + private void OnCoalesceElapsed() + { + lock (_coalesceLock) + { + _coalesceTimer?.Dispose(); + _coalesceTimer = null; + if (_disposed) return; + _lastReloadAt = DateTimeOffset.UtcNow; + } + + _ = DispatchReloadAsync(); + } + + private readonly object _coalesceLock = new(); + /// /// Reloads the deployment table on the renderer's dispatcher, guarded /// against the component being disposed mid-flight (CentralUI-022): @@ -361,5 +414,10 @@ // status change reaches this disposed component. _disposed = true; DeploymentStatusNotifier.StatusChanged -= OnDeploymentStatusChanged; + lock (_coalesceLock) + { + _coalesceTimer?.Dispose(); + _coalesceTimer = null; + } } } diff --git a/src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Monitoring/AlarmSummary.razor b/src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Monitoring/AlarmSummary.razor index c84aeb59..6571d165 100644 --- a/src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Monitoring/AlarmSummary.razor +++ b/src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Monitoring/AlarmSummary.razor @@ -1,5 +1,6 @@ @page "/monitoring/alarms" @attribute [Authorize(Policy = ZB.MOM.WW.ScadaBridge.Security.AuthorizationPolicies.RequireDeployment)] +@using Microsoft.AspNetCore.Components.Web.Virtualization @using ZB.MOM.WW.ScadaBridge.CentralUI.Components.Shared @using ZB.MOM.WW.ScadaBridge.CentralUI.Services @using ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites @@ -196,15 +197,25 @@ @onkeydown='e => OnHeaderKeyDown(e, "severity")' @onkeydown:preventDefault="_preventHeaderDefault">Severity @SortGlyph("severity") + @* Row markup is identical on both paths — the same , the same cells, + the same data-test hook. Only WHO enumerates it differs: a plain + foreach for the ordinary case, and Virtualize once the flat table + gets long enough that rendering every row per delta dominates the + circuit (arch-review WP2.4). Virtualize needs spacers or the + browser hoists its
s out of the table and the layout collapses. *@ - @foreach (var row in _visibleRows) + @if (_visibleRows.Count <= VirtualizeThreshold) { - - @row.InstanceUniqueName - @row.Alarm.AlarmName - - @row.Alarm.Condition.Severity - + @foreach (var row in _visibleRows) + { + @AlarmRow(row) + } + } + else + { + + @AlarmRow(row) + } @@ -222,7 +233,23 @@ // P4 (arch-review): the filtered+sorted view is memoized rather than recomputed // on every render. RecomputeVisibleRows() refreshes it whenever the inputs change // (a fresh snapshot in RefreshAsync, a filter change via @bind:after, or a sort). - private IReadOnlyList _visibleRows = Array.Empty(); + // Concrete List because Virtualize binds ICollection, not IReadOnlyList. + private List _visibleRows = new(); + + // Above this many visible rows the flat table switches to Virtualize. Below it the + // plain foreach is cheaper than the virtualization machinery (and needs no JS), and + // an operator filtered down to a handful of alarms should never pay for either. + private const int VirtualizeThreshold = 150; + + // One row template, shared by the plain and virtualized paths so the two can never + // drift visually. + private RenderFragment AlarmRow => row => + @ + @row.InstanceUniqueName + @row.Alarm.AlarmName + + @row.Alarm.Condition.Severity + ; private IReadOnlyList _notReporting = Array.Empty(); private AlarmRollup _rollup = new(0, 0, 0, new Dictionary()); diff --git a/src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Monitoring/Health.razor b/src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Monitoring/Health.razor index beb3743e..bbcd16a6 100644 --- a/src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Monitoring/Health.razor +++ b/src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Monitoring/Health.razor @@ -15,8 +15,7 @@ @implements IDisposable @inject ICentralHealthAggregator HealthAggregator @inject ISiteRepository SiteRepository -@inject CommunicationService CommunicationService -@inject IAuditLogQueryService AuditLogQueryService +@inject IKpiSnapshotCache KpiSnapshots @inject IKpiHistoryQueryService KpiHistory @inject Microsoft.Extensions.Options.IOptions HealthOptions @@ -640,7 +639,9 @@ // Non-fatal — fall back to showing siteId only } - await RefreshNow(); + // First load rides whatever the shared KPI cache already has fresh — a second + // operator opening the dashboard inside the window costs no extra query. + await RefreshAllAsync(forceRefresh: false); // Site Health Trends (M6) load on their own path — never from the // timer tick below — so a trend-query fault can't disturb the live tile @@ -655,7 +656,7 @@ { try { - await RefreshNow(); + await RefreshAllAsync(forceRefresh: false); StateHasChanged(); } finally @@ -769,20 +770,38 @@ ? "Central Cluster" : $"{GetSiteName(siteKey)} ({siteKey})"; - private async Task RefreshNow() + // Per-tile budget for one refresh tick. The underlying KPI Asks carry the + // 30s CommunicationOptions.QueryTimeout, so without this a single hung + // singleton stalled the WHOLE tick (and, with a 10s timer, stacked ticks) + // rather than degrading its own tile. Cancelling here abandons only this + // circuit's wait — the shared KPI flight continues and its result lands in + // the process-level cache for the next tick. + private static readonly TimeSpan TileTimeout = TimeSpan.FromSeconds(5); + + // The operator's explicit "Refresh Now" bypasses the shared cache's freshness + // window; the timer tick and the first load ride whatever is already fresh + // (which is the whole point — N dashboards share one query per window). + private Task RefreshNow() => RefreshAllAsync(forceRefresh: true); + + // Every tile loads concurrently, each with its own short budget, so one slow + // KPI degrades one tile instead of the tick. Every loader swallows its own + // faults, so Task.WhenAll here can never observe an exception. + private async Task RefreshAllAsync(bool forceRefresh) { _siteStates = HealthAggregator.GetAllSiteStates(); - await LoadOutboxKpis(); - await Task.WhenAll(LoadSiteCallKpis(), LoadSiteCallNodeKpis()); - await LoadAuditKpis(); + await Task.WhenAll( + LoadOutboxKpis(forceRefresh), + LoadSiteCallKpis(forceRefresh), + LoadSiteCallNodeKpis(forceRefresh), + LoadAuditKpis(forceRefresh)); } - private async Task LoadOutboxKpis() + private async Task LoadOutboxKpis(bool forceRefresh) { try { - var response = await CommunicationService.GetNotificationKpisAsync( - new NotificationKpiRequest(Guid.NewGuid().ToString("N"))); + using var timeout = new CancellationTokenSource(TileTimeout); + var response = await KpiSnapshots.GetNotificationKpisAsync(forceRefresh, timeout.Token); if (response.Success) { _outboxKpi = response; @@ -795,6 +814,11 @@ _outboxKpiError = response.ErrorMessage ?? "KPI query failed."; } } + catch (OperationCanceledException) + { + _outboxKpiAvailable = false; + _outboxKpiError = "KPI query timed out."; + } catch (Exception ex) { _outboxKpiAvailable = false; @@ -807,12 +831,12 @@ // killing the dashboard. Mirrors LoadOutboxKpis's error handling shape — a // response with Success == false (repository fault) and an Ask that threw // (transport fault) both collapse to "unavailable". - private async Task LoadSiteCallKpis() + private async Task LoadSiteCallKpis(bool forceRefresh) { try { - var response = await CommunicationService.GetSiteCallKpisAsync( - new SiteCallKpiRequest(Guid.NewGuid().ToString("N"))); + using var timeout = new CancellationTokenSource(TileTimeout); + var response = await KpiSnapshots.GetSiteCallKpisAsync(forceRefresh, timeout.Token); if (response.Success) { _siteCallKpi = response; @@ -825,6 +849,11 @@ _siteCallKpiError = response.ErrorMessage ?? "KPI query failed."; } } + catch (OperationCanceledException) + { + _siteCallKpiAvailable = false; + _siteCallKpiError = "KPI query timed out."; + } catch (Exception ex) { _siteCallKpiAvailable = false; @@ -834,12 +863,12 @@ // Per-node site-call KPI loader (M5.2). Best-effort; a fault silently // suppresses the per-node sub-table rather than degrading the dashboard. - private async Task LoadSiteCallNodeKpis() + private async Task LoadSiteCallNodeKpis(bool forceRefresh) { try { - var response = await CommunicationService.GetPerNodeSiteCallKpisAsync( - new PerNodeSiteCallKpiRequest(Guid.NewGuid().ToString("N"))); + using var timeout = new CancellationTokenSource(TileTimeout); + var response = await KpiSnapshots.GetPerNodeSiteCallKpisAsync(forceRefresh, timeout.Token); if (response.Success) { _siteCallNodeKpis = response.Nodes; @@ -864,14 +893,20 @@ // Audit KPI loader: wraps the service call so a transient DB outage degrades // the three tiles to em dashes with an inline error rather than killing the // dashboard. Mirrors LoadOutboxKpis's error handling shape. - private async Task LoadAuditKpis() + private async Task LoadAuditKpis(bool forceRefresh) { try { - _auditKpi = await AuditLogQueryService.GetKpiSnapshotAsync(); + using var timeout = new CancellationTokenSource(TileTimeout); + _auditKpi = await KpiSnapshots.GetAuditKpiSnapshotAsync(forceRefresh, timeout.Token); _auditKpiAvailable = true; _auditKpiError = null; } + catch (OperationCanceledException) + { + _auditKpiAvailable = false; + _auditKpiError = "KPI query timed out."; + } catch (Exception ex) { _auditKpiAvailable = false; diff --git a/src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Notifications/NotificationKpis.razor b/src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Notifications/NotificationKpis.razor index 93df1e40..a8793aab 100644 --- a/src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Notifications/NotificationKpis.razor +++ b/src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Notifications/NotificationKpis.razor @@ -7,7 +7,7 @@ @using ZB.MOM.WW.ScadaBridge.Commons.Types.Kpi @using ZB.MOM.WW.ScadaBridge.Communication @using ZB.MOM.WW.ScadaBridge.CentralUI.Services -@inject CommunicationService CommunicationService +@inject IKpiSnapshotCache KpiSnapshots @inject ISiteRepository SiteRepository @inject IKpiHistoryQueryService KpiHistory @inject ILogger Logger @@ -250,15 +250,24 @@ Logger.LogWarning(ex, "Failed to load sites for the KPI per-site breakdown."); } - await RefreshAll(); + // First load rides whatever the shared KPI cache already holds fresh; the + // Refresh button forces a new query (see RefreshAll). + await LoadAllAsync(forceRefresh: false); } - private async Task RefreshAll() + // The operator's explicit Refresh bypasses the shared cache's freshness window. + private Task RefreshAll() => LoadAllAsync(forceRefresh: true); + + private async Task LoadAllAsync(bool forceRefresh) { _loading = true; // Race-free despite all tasks mutating component fields: Blazor Server runs // every continuation on the circuit's single-threaded synchronization context. - await Task.WhenAll(LoadGlobalKpis(), LoadPerSiteKpis(), LoadPerNodeKpis(), LoadTrends()); + await Task.WhenAll( + LoadGlobalKpis(forceRefresh), + LoadPerSiteKpis(forceRefresh), + LoadPerNodeKpis(forceRefresh), + LoadTrends()); _loading = false; } @@ -320,12 +329,11 @@ } } - private async Task LoadGlobalKpis() + private async Task LoadGlobalKpis(bool forceRefresh) { try { - var response = await CommunicationService.GetNotificationKpisAsync( - new NotificationKpiRequest(Guid.NewGuid().ToString("N"))); + var response = await KpiSnapshots.GetNotificationKpisAsync(forceRefresh); if (response.Success) { _kpi = response; @@ -342,12 +350,11 @@ } } - private async Task LoadPerSiteKpis() + private async Task LoadPerSiteKpis(bool forceRefresh) { try { - var response = await CommunicationService.GetPerSiteNotificationKpisAsync( - new PerSiteNotificationKpiRequest(Guid.NewGuid().ToString("N"))); + var response = await KpiSnapshots.GetPerSiteNotificationKpisAsync(forceRefresh); if (response.Success) { _perSite = response.Sites; @@ -364,12 +371,11 @@ } } - private async Task LoadPerNodeKpis() + private async Task LoadPerNodeKpis(bool forceRefresh) { try { - var response = await CommunicationService.GetPerNodeNotificationKpisAsync( - new PerNodeNotificationKpiRequest(Guid.NewGuid().ToString("N"))); + var response = await KpiSnapshots.GetPerNodeNotificationKpisAsync(forceRefresh); if (response.Success) { _perNode = response.Nodes; diff --git a/src/ZB.MOM.WW.ScadaBridge.CentralUI/ScriptAnalysis/ScriptAnalysisService.cs b/src/ZB.MOM.WW.ScadaBridge.CentralUI/ScriptAnalysis/ScriptAnalysisService.cs index 5157ed09..6bf0c178 100644 --- a/src/ZB.MOM.WW.ScadaBridge.CentralUI/ScriptAnalysis/ScriptAnalysisService.cs +++ b/src/ZB.MOM.WW.ScadaBridge.CentralUI/ScriptAnalysis/ScriptAnalysisService.cs @@ -179,6 +179,31 @@ public class ScriptAnalysisService private const int SandboxMaxCallSharedDepth = 16; + /// + /// Process-wide cap on concurrently executing Test Runs (arch-review WP2.4). + /// + /// A sandbox run occupies a thread-pool thread for its whole duration — the + /// SandboxScriptHost attribute accessors are synchronous by contract and block on + /// cross-site I/O (GetAwaiter().GetResult()), so the thread is parked, not + /// merely busy, for up to the 10s sandbox timeout. Unbounded, a handful of Designers + /// each hitting Run on a slow-bound script could park enough pool threads to starve the + /// Blazor circuits sharing this process. The gate bounds that: excess runs wait for a + /// slot instead of taking a thread, and the wait is charged to the caller's own timeout. + /// + /// + /// Static because ScriptAnalysisService is registered scoped — a per-instance gate + /// would be per-request and bound nothing. + /// + /// + private static readonly SemaphoreSlim SandboxRunGate = + new(SandboxMaxConcurrentRuns, SandboxMaxConcurrentRuns); + + /// + /// Slot count for . Deliberately small: Test Run is an + /// authoring convenience on a node whose real job is serving circuits. + /// + private const int SandboxMaxConcurrentRuns = 4; + /// /// Compiles and runs a script in the central process. The globals surface /// depends on : template and shared @@ -428,8 +453,15 @@ public class ScriptAnalysisService using var errorScope = captureError.BeginCapture(captured); var stopwatch = Stopwatch.StartNew(); + var gateAcquired = false; try { + // Bound concurrent Test Runs (see SandboxRunGate). The wait is charged to the + // caller's own linked token, so queue time counts against the sandbox timeout + // and a queued run reports Timeout rather than hanging. + await SandboxRunGate.WaitAsync(linkedCts.Token).ConfigureAwait(false); + gateAcquired = true; + // Run on a thread-pool thread with no SynchronizationContext: a // bound script's Instance.SetAttribute / Attributes[...] block // synchronously on cross-site I/O (the API surface is sync by @@ -480,6 +512,10 @@ public class ScriptAnalysisService $"{inner.GetType().Name}: {inner.Message}", SandboxErrorKind.RuntimeError, stopwatch.ElapsedMilliseconds, null); } + finally + { + if (gateAcquired) SandboxRunGate.Release(); + } // outScope / errorScope are disposed by their `using` declarations when the // method returns, restoring the previous capture scope on this call-tree // without touching process-global Console state. diff --git a/src/ZB.MOM.WW.ScadaBridge.CentralUI/ServiceCollectionExtensions.cs b/src/ZB.MOM.WW.ScadaBridge.CentralUI/ServiceCollectionExtensions.cs index 7ba4f291..2509313a 100644 --- a/src/ZB.MOM.WW.ScadaBridge.CentralUI/ServiceCollectionExtensions.cs +++ b/src/ZB.MOM.WW.ScadaBridge.CentralUI/ServiceCollectionExtensions.cs @@ -99,7 +99,17 @@ public static class ServiceCollectionExtensions // facade over CommunicationService.RequestDebugSnapshotAsync (the same // single-shot Ask the Debug View uses) — and flattens the alarm states. services.AddScoped(); - services.AddScoped(); + // The fan-out itself stays scoped (it holds scoped repositories + the snapshot + // client), but the page resolves it through SharedAlarmSummaryService — a process + // singleton that memoizes the fan-out per site with single-flight, so N circuits + // watching one site cost ONE fan-out per window instead of N (arch-review WP2.4). + services.AddScoped(); + services.AddSingleton(); + + // Process-level memoized point-in-time KPI snapshots (arch-review WP2.4, finding + // #8's per-circuit half). Singleton on purpose: the memo has to outlive any one + // circuit for N dashboards to share a single SQL round per refresh window. + services.AddSingleton(); // Secured Writes: dispatches the two-person secured-write commands // (submit / approve / reject / list) to the central ManagementActor through the diff --git a/src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/AlarmSummaryService.cs b/src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/AlarmSummaryService.cs index 0e2bcc9d..6d22d117 100644 --- a/src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/AlarmSummaryService.cs +++ b/src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/AlarmSummaryService.cs @@ -128,7 +128,20 @@ public sealed class AlarmSummaryService : IAlarmSummaryService } /// - public AlarmSummaryResult BuildFromLiveAlarms(IReadOnlyList alarms) + public AlarmSummaryResult BuildFromLiveAlarms(IReadOnlyList alarms) => + BuildFromLiveAlarmsCore(alarms); + + /// + public AlarmRollup ComputeRollup(IReadOnlyList rows) => ComputeRollupCore(rows); + + /// + /// Pure flattening of a live-cache snapshot into summary rows. Shared with + /// , which memoizes the fan-out but must produce + /// byte-identical rows on the live path — one implementation, no drift. + /// + /// The live-cache alarm snapshot. + /// The flattened, deterministically ordered rows (never any not-reporting names). + internal static AlarmSummaryResult BuildFromLiveAlarmsCore(IReadOnlyList alarms) { ArgumentNullException.ThrowIfNull(alarms); @@ -147,8 +160,14 @@ public sealed class AlarmSummaryService : IAlarmSummaryService return new AlarmSummaryResult(orderedRows, Array.Empty()); } - /// - public AlarmRollup ComputeRollup(IReadOnlyList rows) + /// + /// Pure roll-up over already-flattened rows. Shared with + /// for the same one-implementation reason as + /// . + /// + /// The rows to roll up. + /// The active/worst-severity/unacked/by-kind roll-up. + internal static AlarmRollup ComputeRollupCore(IReadOnlyList rows) { ArgumentNullException.ThrowIfNull(rows); diff --git a/src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/IKpiSnapshotCache.cs b/src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/IKpiSnapshotCache.cs new file mode 100644 index 00000000..4785c4a1 --- /dev/null +++ b/src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/IKpiSnapshotCache.cs @@ -0,0 +1,70 @@ +using ZB.MOM.WW.ScadaBridge.Commons.Messages.Audit; +using ZB.MOM.WW.ScadaBridge.Commons.Messages.Notification; +using ZB.MOM.WW.ScadaBridge.Commons.Types; + +namespace ZB.MOM.WW.ScadaBridge.CentralUI.Services; + +/// +/// Process-level memoized point-in-time KPI snapshots shared by every Blazor circuit on +/// this node (arch-review WP2.4, finding #8 — the per-circuit polling half). +/// +/// Every KPI here is a global, point-in-time aggregate: it does not vary by user, +/// site scope, or role, so N circuits asking for it inside one refresh window are asking +/// the identical question. Without memoization, ten Health dashboards meant ten Asks → +/// ten aggregate SQL round trips every ten seconds. Each accessor is independently +/// memoized with a short TTL and single-flight production, so the fleet costs ONE round +/// trip per KPI per window regardless of viewer count. +/// +/// +/// Faults are never memoized: a failed query is re-attempted by the next caller, and the +/// caller still sees the exception exactly as it would from a direct query — the pages' +/// existing per-tile "unavailable" degradation is unchanged. +/// +/// +public interface IKpiSnapshotCache +{ + /// Global Notification Outbox KPIs (queue depth, stuck, parked, delivered-last-interval). + /// Bypass a still-fresh value (operator "Refresh"); an in-flight round is still joined. + /// Cancels this caller's wait, never the shared flight. + /// The memoized notification KPI response. + Task GetNotificationKpisAsync( + bool forceRefresh = false, CancellationToken cancellationToken = default); + + /// Per-source-site Notification Outbox KPI breakdown. + /// Bypass a still-fresh value; an in-flight round is still joined. + /// Cancels this caller's wait, never the shared flight. + /// The memoized per-site notification KPI response. + Task GetPerSiteNotificationKpisAsync( + bool forceRefresh = false, CancellationToken cancellationToken = default); + + /// Per-node Notification Outbox KPI breakdown (grouped by SourceNode). + /// Bypass a still-fresh value; an in-flight round is still joined. + /// Cancels this caller's wait, never the shared flight. + /// The memoized per-node notification KPI response. + Task GetPerNodeNotificationKpisAsync( + bool forceRefresh = false, CancellationToken cancellationToken = default); + + /// Global Site Call Audit KPIs (buffered, stuck, parked). + /// Bypass a still-fresh value; an in-flight round is still joined. + /// Cancels this caller's wait, never the shared flight. + /// The memoized site call KPI response. + Task GetSiteCallKpisAsync( + bool forceRefresh = false, CancellationToken cancellationToken = default); + + /// Per-node Site Call Audit KPI breakdown (grouped by SourceNode). + /// Bypass a still-fresh value; an in-flight round is still joined. + /// Cancels this caller's wait, never the shared flight. + /// The memoized per-node site call KPI response. + Task GetPerNodeSiteCallKpisAsync( + bool forceRefresh = false, CancellationToken cancellationToken = default); + + /// + /// Audit Log KPI snapshot (1h volume + error rate over the central AuditLog table, + /// plus the summed per-site backlog). + /// + /// Bypass a still-fresh value; an in-flight round is still joined. + /// Cancels this caller's wait, never the shared flight. + /// The memoized audit KPI snapshot. + Task GetAuditKpiSnapshotAsync( + bool forceRefresh = false, CancellationToken cancellationToken = default); +} diff --git a/src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/KpiSnapshotCache.cs b/src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/KpiSnapshotCache.cs new file mode 100644 index 00000000..f18b56d7 --- /dev/null +++ b/src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/KpiSnapshotCache.cs @@ -0,0 +1,122 @@ +using Microsoft.Extensions.DependencyInjection; +using ZB.MOM.WW.ScadaBridge.Commons.Messages.Audit; +using ZB.MOM.WW.ScadaBridge.Commons.Messages.Notification; +using ZB.MOM.WW.ScadaBridge.Commons.Types; +using ZB.MOM.WW.ScadaBridge.Communication; + +namespace ZB.MOM.WW.ScadaBridge.CentralUI.Services; + +/// +/// Default — a DI singleton holding one +/// per KPI query. +/// +/// Registered as a singleton on purpose: the whole point is that the memo outlives any one +/// circuit. It depends only on (itself a singleton) and +/// — the audit KPI resolves its scoped query service from +/// a fresh scope per flight, the same shape AuditLogQueryService uses internally so a +/// KPI round never shares a circuit-scoped DbContext. +/// +/// +public sealed class KpiSnapshotCache : IKpiSnapshotCache +{ + /// + /// Freshness window. Sized just under the Health dashboard's 10s tick so each tick still + /// yields ONE fresh query for the whole node (rather than one per circuit), while a + /// circuit whose tick lands mid-window is served the memoized value instead of duplicating it. + /// + internal static readonly TimeSpan Ttl = TimeSpan.FromSeconds(8); + + private readonly CommunicationService _communication; + private readonly IServiceScopeFactory _scopeFactory; + + private readonly SingleFlightMemo _notificationKpis; + private readonly SingleFlightMemo _perSiteNotificationKpis; + private readonly SingleFlightMemo _perNodeNotificationKpis; + private readonly SingleFlightMemo _siteCallKpis; + private readonly SingleFlightMemo _perNodeSiteCallKpis; + private readonly SingleFlightMemo _auditKpis; + + /// + /// Initializes the shared KPI snapshot cache. + /// + /// Central-side comms used to Ask the outbox / site-call singletons. + /// Opens a fresh DI scope per audit-KPI flight. + public KpiSnapshotCache(CommunicationService communication, IServiceScopeFactory scopeFactory) + : this(communication, scopeFactory, Ttl, clock: null) + { + } + + /// + /// Test seam: same cache with an explicit TTL and clock so freshness can be asserted + /// without real delays. + /// + /// Central-side comms used to Ask the outbox / site-call singletons. + /// Opens a fresh DI scope per audit-KPI flight. + /// Freshness window. + /// Clock used for freshness. + internal KpiSnapshotCache( + CommunicationService communication, + IServiceScopeFactory scopeFactory, + TimeSpan ttl, + Func? clock) + { + _communication = communication ?? throw new ArgumentNullException(nameof(communication)); + _scopeFactory = scopeFactory ?? throw new ArgumentNullException(nameof(scopeFactory)); + + _notificationKpis = new SingleFlightMemo(ttl, clock); + _perSiteNotificationKpis = new SingleFlightMemo(ttl, clock); + _perNodeNotificationKpis = new SingleFlightMemo(ttl, clock); + _siteCallKpis = new SingleFlightMemo(ttl, clock); + _perNodeSiteCallKpis = new SingleFlightMemo(ttl, clock); + _auditKpis = new SingleFlightMemo(ttl, clock); + } + + /// + public Task GetNotificationKpisAsync( + bool forceRefresh = false, CancellationToken cancellationToken = default) => + _notificationKpis.GetAsync( + () => _communication.GetNotificationKpisAsync(new NotificationKpiRequest(NewCorrelationId())), + forceRefresh, cancellationToken); + + /// + public Task GetPerSiteNotificationKpisAsync( + bool forceRefresh = false, CancellationToken cancellationToken = default) => + _perSiteNotificationKpis.GetAsync( + () => _communication.GetPerSiteNotificationKpisAsync(new PerSiteNotificationKpiRequest(NewCorrelationId())), + forceRefresh, cancellationToken); + + /// + public Task GetPerNodeNotificationKpisAsync( + bool forceRefresh = false, CancellationToken cancellationToken = default) => + _perNodeNotificationKpis.GetAsync( + () => _communication.GetPerNodeNotificationKpisAsync(new PerNodeNotificationKpiRequest(NewCorrelationId())), + forceRefresh, cancellationToken); + + /// + public Task GetSiteCallKpisAsync( + bool forceRefresh = false, CancellationToken cancellationToken = default) => + _siteCallKpis.GetAsync( + () => _communication.GetSiteCallKpisAsync(new SiteCallKpiRequest(NewCorrelationId())), + forceRefresh, cancellationToken); + + /// + public Task GetPerNodeSiteCallKpisAsync( + bool forceRefresh = false, CancellationToken cancellationToken = default) => + _perNodeSiteCallKpis.GetAsync( + () => _communication.GetPerNodeSiteCallKpisAsync(new PerNodeSiteCallKpiRequest(NewCorrelationId())), + forceRefresh, cancellationToken); + + /// + public Task GetAuditKpiSnapshotAsync( + bool forceRefresh = false, CancellationToken cancellationToken = default) => + _auditKpis.GetAsync(LoadAuditKpisAsync, forceRefresh, cancellationToken); + + private async Task LoadAuditKpisAsync() + { + await using var scope = _scopeFactory.CreateAsyncScope(); + var queryService = scope.ServiceProvider.GetRequiredService(); + return await queryService.GetKpiSnapshotAsync(); + } + + private static string NewCorrelationId() => Guid.NewGuid().ToString("N"); +} diff --git a/src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/SharedAlarmSummaryService.cs b/src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/SharedAlarmSummaryService.cs new file mode 100644 index 00000000..29a6359e --- /dev/null +++ b/src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/SharedAlarmSummaryService.cs @@ -0,0 +1,115 @@ +using System.Collections.Concurrent; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Options; +using ZB.MOM.WW.ScadaBridge.Commons.Messages.Streaming; +using ZB.MOM.WW.ScadaBridge.Communication; + +namespace ZB.MOM.WW.ScadaBridge.CentralUI.Services; + +/// +/// Process-level memoizing façade over (arch-review WP2.4). +/// +/// The Alarm Summary page polls every 15s per circuit, and each poll fans one debug +/// snapshot Ask out to every Enabled instance on the site. Ten operators watching one site +/// meant ten independent fan-outs per 15s across the same instances — on top of the shared +/// live-alarm aggregator already seeding and reconciling from the identical fan-out. The +/// question is not user-specific, so this service collapses it: one memo slot per site, +/// single-flight, so the node costs ONE fan-out per site per window no matter how many +/// viewers are watching. +/// +/// +/// The freshness window follows the live cache. While +/// is true the page deliberately ignores the poll's +/// alarm rows (the live deltas own them — arch-review R2 N5), so the only thing the poll +/// still supplies is the not-reporting list, and the window widens to the aggregator's own +/// reconcile interval. While the cache is cold the poll is the page's full-rebuild safety +/// net, so the window stays just under the page's 15s tick and every tick gets fresh data. +/// +/// +public sealed class SharedAlarmSummaryService : IAlarmSummaryService +{ + /// + /// Freshness window while the live alarm cache is NOT serving this site: the poll is the + /// page's authoritative rebuild path, so it must stay under the 15s page tick. + /// + internal static readonly TimeSpan ColdCacheTtl = TimeSpan.FromSeconds(12); + + private readonly IServiceScopeFactory _scopeFactory; + private readonly ISiteAlarmLiveCache _liveCache; + private readonly TimeSpan _liveCacheTtl; + private readonly Func? _clock; + + private readonly ConcurrentDictionary> _bySite = new(); + + /// + /// Initializes the shared alarm summary façade. + /// + /// Opens a fresh DI scope per fan-out (fresh repositories, off any circuit scope). + /// The shared live alarm cache, consulted only for its per-site liveness. + /// Communication options; supplies the aggregator reconcile interval. + public SharedAlarmSummaryService( + IServiceScopeFactory scopeFactory, + ISiteAlarmLiveCache liveCache, + IOptions options) + : this(scopeFactory, liveCache, (options ?? throw new ArgumentNullException(nameof(options))) + .Value.LiveAlarmCacheReconcileInterval, clock: null) + { + } + + /// + /// Test seam: same façade with an explicit live-cache window and clock. + /// + /// Opens a fresh DI scope per fan-out. + /// The shared live alarm cache. + /// Freshness window used while the live cache is serving the site. + /// Clock used for freshness. + internal SharedAlarmSummaryService( + IServiceScopeFactory scopeFactory, + ISiteAlarmLiveCache liveCache, + TimeSpan liveCacheTtl, + Func? clock) + { + _scopeFactory = scopeFactory ?? throw new ArgumentNullException(nameof(scopeFactory)); + _liveCache = liveCache ?? throw new ArgumentNullException(nameof(liveCache)); + // Never shorter than the cold window — a misconfigured reconcile interval must not + // silently make the memo useless. + _liveCacheTtl = liveCacheTtl > ColdCacheTtl ? liveCacheTtl : ColdCacheTtl; + _clock = clock; + } + + /// + public Task GetSiteAlarmsAsync( + int siteId, CancellationToken cancellationToken = default) + { + var memo = _bySite.GetOrAdd( + siteId, + _ => new SingleFlightMemo(ColdCacheTtl, _clock)); + + var ttl = _liveCache.IsLive(siteId) ? _liveCacheTtl : ColdCacheTtl; + + return memo.GetAsync( + () => FanOutAsync(siteId), + forceRefresh: false, + cancellationToken, + ttlOverride: ttl); + } + + /// + public AlarmSummaryResult BuildFromLiveAlarms(IReadOnlyList alarms) => + AlarmSummaryService.BuildFromLiveAlarmsCore(alarms); + + /// + public AlarmRollup ComputeRollup(IReadOnlyList rows) => + AlarmSummaryService.ComputeRollupCore(rows); + + /// + /// One shared fan-out. Deliberately runs with no caller cancellation token: the flight is + /// shared, so one circuit navigating away must not cancel the round the others are awaiting. + /// + private async Task FanOutAsync(int siteId) + { + await using var scope = _scopeFactory.CreateAsyncScope(); + var inner = scope.ServiceProvider.GetRequiredService(); + return await inner.GetSiteAlarmsAsync(siteId, CancellationToken.None); + } +} diff --git a/src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/SingleFlightMemo.cs b/src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/SingleFlightMemo.cs new file mode 100644 index 00000000..a95731e8 --- /dev/null +++ b/src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/SingleFlightMemo.cs @@ -0,0 +1,136 @@ +namespace ZB.MOM.WW.ScadaBridge.CentralUI.Services; + +/// +/// A process-level memo slot: one value, a time-to-live, and single-flight production. +/// +/// Blazor Server runs one circuit per connected browser, so a page that polls on a timer +/// runs its query per circuit. Ten operators on the Health dashboard meant ten +/// identical KPI round trips every ten seconds. This slot collapses them: the first caller +/// in a TTL window starts the flight, every concurrent caller awaits the same +/// , and callers arriving after it completes are served the +/// memoized value until the TTL expires. +/// +/// +/// Failures are never cached — a faulted flight is discarded so the next caller retries. +/// The caller's own exception handling is therefore unchanged: it still sees the fault +/// (all callers of that one flight see it), and the next window starts clean. +/// +/// +/// The memoized value type. +internal sealed class SingleFlightMemo +{ + private readonly object _lock = new(); + private readonly TimeSpan _ttl; + private readonly Func _clock; + + /// The current flight, completed or not. Null until the first call. + private Task? _current; + + /// Wall-clock instant after which is stale. Only meaningful when it completed successfully. + private DateTimeOffset _freshUntil; + + /// + /// Initializes a memo slot. + /// + /// How long a successfully produced value stays fresh. + /// Clock used for freshness, injectable so tests need no real delay. + public SingleFlightMemo(TimeSpan ttl, Func? clock = null) + { + _ttl = ttl; + _clock = clock ?? (() => DateTimeOffset.UtcNow); + } + + /// + /// Number of times the factory has actually been invoked. Diagnostics and tests only. + /// + public int FlightCount { get; private set; } + + /// + /// Returns the memoized value, starting a flight if none is fresh. + /// + /// Produces a fresh value. Invoked outside the slot's lock. + /// + /// When , a completed-but-still-fresh value is discarded and a new + /// flight starts. An in-flight round is still joined rather than duplicated, so a + /// burst of operator "Refresh" clicks remains one round trip. + /// + /// + /// Cancels this caller's wait only — never the shared flight, which other callers may still + /// be awaiting. + /// + /// + /// Freshness window for this flight, overriding the slot's default. Used where the + /// acceptable staleness depends on state the caller knows (the Alarm Summary poll needs a + /// far shorter window while the live alarm cache is cold than while it is serving deltas). + /// + /// The memoized (or freshly produced) value. + public Task GetAsync( + Func> factory, + bool forceRefresh = false, + CancellationToken cancellationToken = default, + TimeSpan? ttlOverride = null) + { + ArgumentNullException.ThrowIfNull(factory); + + TaskCompletionSource? started = null; + Task result; + + lock (_lock) + { + var current = _current; + if (current is { IsCompleted: false }) + { + // A round is already in progress — join it, force or not. + result = current; + } + else if (!forceRefresh + && current is { IsCompletedSuccessfully: true } + && _clock() < _freshUntil) + { + result = current; + } + else + { + // RunContinuationsAsynchronously: a caller's continuation must never run + // inline on the thread that completes the flight (that thread is another + // circuit's dispatcher). + started = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + _current = started.Task; + FlightCount++; + result = started.Task; + } + } + + // The factory runs OUTSIDE the lock: it does I/O (an Ask, a SQL round trip) and must + // never be able to re-enter this slot on the calling thread while the lock is held. + if (started is not null) + { + _ = RunAsync(factory, started, ttlOverride ?? _ttl); + } + + return cancellationToken.CanBeCanceled ? result.WaitAsync(cancellationToken) : result; + } + + private async Task RunAsync(Func> factory, TaskCompletionSource completion, TimeSpan ttl) + { + try + { + var value = await factory().ConfigureAwait(false); + lock (_lock) + { + _freshUntil = _clock() + ttl; + } + completion.SetResult(value); + } + catch (Exception ex) + { + // A failure is not memoized: leave _freshUntil in the past so the next caller + // starts a new flight instead of re-serving the fault for the whole TTL. + lock (_lock) + { + _freshUntil = default; + } + completion.SetException(ex); + } + } +} diff --git a/tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Deployment/DebugViewRenderCoalescingTests.cs b/tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Deployment/DebugViewRenderCoalescingTests.cs new file mode 100644 index 00000000..86678da3 --- /dev/null +++ b/tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Deployment/DebugViewRenderCoalescingTests.cs @@ -0,0 +1,144 @@ +using System.Collections; +using System.Reflection; +using System.Security.Claims; +using Bunit; +using Microsoft.AspNetCore.Components.Authorization; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Logging.Abstractions; +using Microsoft.Extensions.Options; +using NSubstitute; +using ZB.MOM.WW.ScadaBridge.CentralUI.Auth; +using ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites; +using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories; +using ZB.MOM.WW.ScadaBridge.Commons.Messages.Streaming; +using ZB.MOM.WW.ScadaBridge.Communication; +using ZB.MOM.WW.ScadaBridge.Communication.Grpc; +using DebugViewPage = ZB.MOM.WW.ScadaBridge.CentralUI.Components.Pages.Deployment.DebugView; + +namespace ZB.MOM.WW.ScadaBridge.CentralUI.Tests.Deployment; + +/// +/// Regression tests for the Debug View render coalescing (arch-review WP2.4). Every streamed +/// event used to marshal onto the circuit dispatcher and call StateHasChanged on its +/// own, so a chatty instance drove one full render — and two full composition-tree rebuilds — +/// per value change. Events are now buffered and applied once per ~250 ms window: a burst +/// costs a handful of renders instead of one per event, and no event is dropped. +/// +public class DebugViewRenderCoalescingTests : BunitContext +{ + private IRenderedComponent RenderPage() + { + JSInterop.Mode = JSRuntimeMode.Loose; + + var repo = Substitute.For(); + var siteRepo = Substitute.For(); + siteRepo.GetAllSitesAsync().Returns(new List()); + Services.AddSingleton(repo); + Services.AddSingleton(siteRepo); + + var comms = new CommunicationService( + Options.Create(new CommunicationOptions()), + NullLogger.Instance); + Services.AddSingleton(comms); + + var grpcFactory = new SiteStreamGrpcClientFactory(NullLoggerFactory.Instance); + var debugStream = new DebugStreamService( + comms, new ServiceCollection().BuildServiceProvider(), grpcFactory, + NullLogger.Instance); + Services.AddSingleton(debugStream); + + var identity = new ClaimsIdentity( + new[] { new Claim(ClaimTypes.Name, "deployer") }, "TestCookie"); + var stubAuth = new StubAuthStateProvider( + new AuthenticationState(new ClaimsPrincipal(identity))); + Services.AddSingleton(stubAuth); + Services.AddScoped(_ => new SiteScopeService(stubAuth)); + + return Render(); + } + + private sealed class StubAuthStateProvider : AuthenticationStateProvider + { + private readonly AuthenticationState _state; + public StubAuthStateProvider(AuthenticationState state) => _state = state; + public override Task GetAuthenticationStateAsync() + => Task.FromResult(_state); + } + + private static MethodInfo HandleStreamEvent => typeof(DebugViewPage).GetMethod( + "HandleStreamEvent", BindingFlags.Instance | BindingFlags.NonPublic)!; + + private static IDictionary AttributeValues(DebugViewPage c) => (IDictionary) + typeof(DebugViewPage).GetField("_attributeValues", + BindingFlags.Instance | BindingFlags.NonPublic)!.GetValue(c)!; + + private static void Send(DebugViewPage page, string name, object value) => + HandleStreamEvent.Invoke(page, new object[] + { + new AttributeValueChanged("Inst-1", name, name, value, "Good", DateTimeOffset.UtcNow), + }); + + [Fact] + public void BurstOfEvents_RendersOnce_ButAppliesEveryEvent() + { + var cut = RenderPage(); + var dict = AttributeValues(cut.Instance); + var rendersBefore = cut.RenderCount; + + const int burst = 300; + for (var i = 0; i < burst; i++) + { + Send(cut.Instance, $"Tag.{i}", i); + } + + cut.WaitForState(() => dict.Count == burst, TimeSpan.FromSeconds(5)); + + // Pre-fix this was one render per event. The window may close mid-burst, so allow a + // few flushes — the point is that it is a small constant, not O(events). + var renders = cut.RenderCount - rendersBefore; + Assert.InRange(renders, 1, 5); + } + + [Fact] + public void RepeatedUpdatesToOneTag_CollapseToTheLatestValue() + { + var cut = RenderPage(); + var dict = AttributeValues(cut.Instance); + + for (var i = 0; i < 50; i++) + { + Send(cut.Instance, "Pump.Speed", i); + } + + cut.WaitForState(() => dict.Count == 1, TimeSpan.FromSeconds(5)); + var applied = (AttributeValueChanged)dict["Pump.Speed"]!; + Assert.Equal(49, applied.Value); + } + + [Fact] + public void EventArrivingAfterAFlush_StillArmsANewWindow() + { + var cut = RenderPage(); + var dict = AttributeValues(cut.Instance); + + Send(cut.Instance, "First", 1); + cut.WaitForState(() => dict.Count == 1, TimeSpan.FromSeconds(5)); + + // The window disarms on drain; a later event must arm a fresh one rather than + // sitting in the pending map until some unrelated event happens to arrive. + Send(cut.Instance, "Second", 2); + cut.WaitForState(() => dict.Count == 2, TimeSpan.FromSeconds(5)); + } + + [Fact] + public void EventsAfterDispose_AreDroppedWithoutThrowing() + { + var cut = RenderPage(); + cut.Instance.Dispose(); + + var ex = Record.Exception(() => Send(cut.Instance, "Late", 1)); + + Assert.Null(ex); + Assert.Empty(AttributeValues(cut.Instance)); + } +} diff --git a/tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Deployment/DeploymentsReloadDebounceTests.cs b/tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Deployment/DeploymentsReloadDebounceTests.cs new file mode 100644 index 00000000..83e578b5 --- /dev/null +++ b/tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Deployment/DeploymentsReloadDebounceTests.cs @@ -0,0 +1,127 @@ +using System.Security.Claims; +using Bunit; +using Microsoft.AspNetCore.Components.Authorization; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Logging.Abstractions; +using NSubstitute; +using ZB.MOM.WW.ScadaBridge.CentralUI.Auth; +using ZB.MOM.WW.ScadaBridge.Commons.Entities.Deployment; +using ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances; +using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories; +using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; +using ZB.MOM.WW.ScadaBridge.DeploymentManager; +using DeploymentsPage = ZB.MOM.WW.ScadaBridge.CentralUI.Components.Pages.Deployment.Deployments; + +namespace ZB.MOM.WW.ScadaBridge.CentralUI.Tests.Deployment; + +/// +/// Regression tests for the Deployment Status push coalescing (arch-review WP2.4). One +/// notifier callback reloads EVERY deployment record plus EVERY instance, and the notifier +/// fires per status write — so a multi-instance deploy produced a stampede of full reloads +/// per circuit. The reload is now leading-edge debounced: the first push after an idle gap +/// is still immediate, and a burst behind it collapses into one trailing reload. +/// +public class DeploymentsReloadDebounceTests : BunitContext +{ + private IDeploymentManagerRepository _deployRepo = null!; + private ITemplateEngineRepository _templateRepo = null!; + private DeploymentStatusNotifier _notifier = null!; + + private void RegisterServices() + { + _deployRepo = Substitute.For(); + _templateRepo = Substitute.For(); + _notifier = new DeploymentStatusNotifier(NullLogger.Instance); + + _templateRepo.GetAllInstancesAsync(Arg.Any()) + .Returns(new List { new("Inst-1") { Id = 1, SiteId = 1 } }); + _deployRepo.GetAllDeploymentRecordsAsync(Arg.Any()) + .Returns(new List()); + + Services.AddSingleton(_deployRepo); + Services.AddSingleton(_templateRepo); + Services.AddSingleton(_notifier); + + var identity = new ClaimsIdentity( + new[] { new Claim(ClaimTypes.Name, "deployer") }, "TestCookie"); + var stubAuth = new StubAuthStateProvider( + new AuthenticationState(new ClaimsPrincipal(identity))); + Services.AddSingleton(stubAuth); + Services.AddScoped(_ => new SiteScopeService(stubAuth)); + } + + private sealed class StubAuthStateProvider : AuthenticationStateProvider + { + private readonly AuthenticationState _state; + public StubAuthStateProvider(AuthenticationState state) => _state = state; + public override Task GetAuthenticationStateAsync() + => Task.FromResult(_state); + } + + [Fact] + public void BurstOfStatusWrites_CollapsesIntoFarFewerReloads() + { + RegisterServices(); + var cut = Render(); + cut.WaitForAssertion(() => + _deployRepo.Received().GetAllDeploymentRecordsAsync(Arg.Any())); + _deployRepo.ClearReceivedCalls(); + + // A 40-instance site deploy: every status write raises the notifier. + for (var i = 0; i < 40; i++) + { + _notifier.NotifyStatusChanged( + new DeploymentStatusChange($"dep-{i}", 1, DeploymentStatus.InProgress)); + } + + // Leading edge fires at once; the rest ride one trailing reload. + cut.WaitForAssertion(() => + _deployRepo.Received().GetAllDeploymentRecordsAsync(Arg.Any())); + + // Let the trailing window close before counting. + Thread.Sleep(900); + + var reloads = _deployRepo.ReceivedCalls() + .Count(c => c.GetMethodInfo().Name == nameof(IDeploymentManagerRepository.GetAllDeploymentRecordsAsync)); + Assert.InRange(reloads, 1, 4); + } + + [Fact] + public void FirstPushAfterIdle_ReloadsImmediately() + { + RegisterServices(); + var cut = Render(); + cut.WaitForAssertion(() => + _deployRepo.Received().GetAllDeploymentRecordsAsync(Arg.Any())); + _deployRepo.ClearReceivedCalls(); + + // Idle since the initial load — the leading edge must not wait out the window. + Thread.Sleep(600); + _notifier.NotifyStatusChanged( + new DeploymentStatusChange("dep-1", 1, DeploymentStatus.Success)); + + cut.WaitForAssertion( + () => _deployRepo.Received().GetAllDeploymentRecordsAsync(Arg.Any()), + TimeSpan.FromMilliseconds(400)); + } + + [Fact] + public void DisposeDuringACoalesceWindow_DoesNotReload() + { + RegisterServices(); + var cut = Render(); + cut.WaitForAssertion(() => + _deployRepo.Received().GetAllDeploymentRecordsAsync(Arg.Any())); + + // Two pushes: the first takes the leading edge, the second arms the trailing timer. + _notifier.NotifyStatusChanged(new DeploymentStatusChange("dep-1", 1, DeploymentStatus.InProgress)); + _notifier.NotifyStatusChanged(new DeploymentStatusChange("dep-2", 1, DeploymentStatus.InProgress)); + + cut.Instance.Dispose(); + _deployRepo.ClearReceivedCalls(); + + // The armed timer must be disposed with the component, not fire against it. + Thread.Sleep(900); + _deployRepo.DidNotReceive().GetAllDeploymentRecordsAsync(Arg.Any()); + } +} diff --git a/tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Monitoring/AlarmSummaryVirtualizeTests.cs b/tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Monitoring/AlarmSummaryVirtualizeTests.cs new file mode 100644 index 00000000..a51d9c9b --- /dev/null +++ b/tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Monitoring/AlarmSummaryVirtualizeTests.cs @@ -0,0 +1,113 @@ +using System.Security.Claims; +using Bunit; +using Microsoft.AspNetCore.Components.Authorization; +using Microsoft.Extensions.DependencyInjection; +using NSubstitute; +using ZB.MOM.WW.ScadaBridge.CentralUI.Services; +using ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites; +using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories; +using ZB.MOM.WW.ScadaBridge.Commons.Messages.Streaming; +using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; +using ZB.MOM.WW.ScadaBridge.Communication; +using ZB.MOM.WW.ScadaBridge.Security; +using AlarmSummaryPage = ZB.MOM.WW.ScadaBridge.CentralUI.Components.Pages.Monitoring.AlarmSummary; + +namespace ZB.MOM.WW.ScadaBridge.CentralUI.Tests.Monitoring; + +/// +/// Covers the Alarm Summary flat table's two rendering paths (arch-review WP2.4). A short +/// list renders every row with a plain foreach; past the threshold the same row template is +/// handed to Virtualize so a site with thousands of alarms stops materialising the +/// whole table into the circuit's render tree on every delta. Row markup — including the +/// data-test hook the operator tests key off — is identical either way. +/// +public class AlarmSummaryVirtualizeTests : BunitContext +{ + private readonly IAlarmSummaryService _summary = Substitute.For(); + private readonly ISiteRepository _siteRepo = Substitute.For(); + + private void Arrange(int rowCount) + { + JSInterop.Mode = JSRuntimeMode.Loose; + + var rows = Enumerable.Range(0, rowCount) + .Select(i => new AlarmSummaryRow( + $"inst-{i:D4}", + new AlarmStateChanged($"inst-{i:D4}", $"alarm-{i:D4}", AlarmState.Active, i, DateTimeOffset.UtcNow))) + .ToList(); + + _summary.GetSiteAlarmsAsync(Arg.Any(), Arg.Any()) + .Returns(Task.FromResult(new AlarmSummaryResult(rows, Array.Empty()))); + _summary.ComputeRollup(Arg.Any>()) + .Returns(new AlarmRollup(rowCount, rowCount, 0, new Dictionary())); + Services.AddSingleton(_summary); + + _siteRepo.GetAllSitesAsync(Arg.Any()) + .Returns(Task.FromResult>(new List + { + new("Site 1", "site1") { Id = 1 }, + })); + Services.AddSingleton(_siteRepo); + Services.AddSingleton(new InertLiveCache()); + + var claims = new[] + { + new Claim(JwtTokenService.UsernameClaimType, "tester"), + new Claim(JwtTokenService.RoleClaimType, "Administrator"), + }; + Services.AddSingleton( + new TestAuthStateProvider(new ClaimsPrincipal(new ClaimsIdentity(claims, "TestAuth")))); + Services.AddAuthorizationCore(); + } + + private IRenderedComponent RenderWithSiteSelected() + { + var cut = Render(); + cut.Find("[data-test='alarm-summary-site']").Change("1"); + return cut; + } + + [Fact] + public void ShortList_RendersEveryRowInline() + { + Arrange(rowCount: 20); + + var cut = RenderWithSiteSelected(); + + cut.WaitForAssertion(() => + Assert.Equal(20, cut.FindAll("tr[data-test='alarm-summary-row']").Count)); + // The row-count line is unchanged by either path. + Assert.Contains("Showing 20 of 20", cut.Markup); + } + + [Fact] + public void LongList_VirtualizesWithoutRenderingEveryRow() + { + Arrange(rowCount: 2000); + + var cut = RenderWithSiteSelected(); + + cut.WaitForAssertion(() => Assert.Contains("Showing 2000 of 2000", cut.Markup)); + + var rendered = cut.FindAll("tr[data-test='alarm-summary-row']").Count; + Assert.InRange(rendered, 1, 1999); + // Rows still carry the same shape — instance name, alarm name, severity. + Assert.Contains("alarm-", cut.Markup); + } + + /// Never goes live, so the page keeps its poll snapshot for these tests. + private sealed class InertLiveCache : ISiteAlarmLiveCache + { + public IDisposable Subscribe(int siteId, Action onChanged) => new NoOp(); + + public IReadOnlyList GetCurrentAlarms(int siteId) => + Array.Empty(); + + public bool IsLive(int siteId) => false; + + private sealed class NoOp : IDisposable + { + public void Dispose() { } + } + } +} diff --git a/tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Pages/HealthPageTests.cs b/tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Pages/HealthPageTests.cs index 04cf08e7..c7905911 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Pages/HealthPageTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Pages/HealthPageTests.cs @@ -64,6 +64,14 @@ public class HealthPageTests : BunitContext _comms.SetSiteCallAudit(siteCallAudit); Services.AddSingleton(_comms); + // arch-review WP2.4 — the page no longer queries CommunicationService / + // IAuditLogQueryService per circuit; it reads the process-level memoized + // KPI cache. The real cache is registered here (not a substitute) so the + // scripted-actor seam above is still what actually answers. + Services.AddSingleton(sp => new KpiSnapshotCache( + sp.GetRequiredService(), + sp.GetRequiredService())); + var aggregator = Substitute.For(); aggregator.GetAllSiteStates() .Returns(new Dictionary()); diff --git a/tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Pages/NotificationKpisPageTests.cs b/tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Pages/NotificationKpisPageTests.cs index 8ee1f85c..4b393783 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Pages/NotificationKpisPageTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Pages/NotificationKpisPageTests.cs @@ -71,6 +71,13 @@ public class NotificationKpisPageTests : BunitContext Services.AddSingleton(_comms); + // arch-review WP2.4 — the page reads the process-level memoized KPI cache + // rather than querying CommunicationService per circuit. The real cache is + // registered so the scripted-actor seam above still answers the queries. + Services.AddSingleton(sp => new KpiSnapshotCache( + sp.GetRequiredService(), + sp.GetRequiredService())); + var siteRepo = Substitute.For(); siteRepo.GetAllSitesAsync(Arg.Any()) .Returns(Task.FromResult>(new List diff --git a/tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Services/SharedAlarmSummaryServiceTests.cs b/tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Services/SharedAlarmSummaryServiceTests.cs new file mode 100644 index 00000000..83c4873a --- /dev/null +++ b/tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Services/SharedAlarmSummaryServiceTests.cs @@ -0,0 +1,169 @@ +using Microsoft.Extensions.DependencyInjection; +using NSubstitute; +using ZB.MOM.WW.ScadaBridge.CentralUI.Services; +using ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances; +using ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites; +using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories; +using ZB.MOM.WW.ScadaBridge.Commons.Messages.DebugView; +using ZB.MOM.WW.ScadaBridge.Commons.Messages.Streaming; +using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; +using ZB.MOM.WW.ScadaBridge.Communication; + +namespace ZB.MOM.WW.ScadaBridge.CentralUI.Tests.Services; + +/// +/// Unit tests for (arch-review WP2.4). The Alarm +/// Summary page polls per circuit; the shared façade must collapse those into ONE per-site +/// fan-out per freshness window, and must widen that window while the live alarm cache is +/// serving the site (where the poll only still supplies the not-reporting list). +/// +public class SharedAlarmSummaryServiceTests : IDisposable +{ + private const int SiteId = 7; + private const string SiteIdentifier = "plant-a"; + + private static readonly DateTimeOffset T0 = new(2026, 8, 14, 9, 0, 0, TimeSpan.Zero); + + private readonly ITemplateEngineRepository _instanceRepo = Substitute.For(); + private readonly ISiteRepository _siteRepo = Substitute.For(); + private readonly IInstanceSnapshotClient _snapshotClient = Substitute.For(); + private readonly FakeLiveCache _liveCache = new(); + private readonly ServiceProvider _provider; + + private DateTimeOffset _now = T0; + + public SharedAlarmSummaryServiceTests() + { + _siteRepo.GetSiteByIdAsync(SiteId, Arg.Any()) + .Returns(new Site("Plant A", SiteIdentifier) { Id = SiteId }); + _instanceRepo.GetInstancesBySiteIdAsync(SiteId, Arg.Any()) + .Returns(new List + { + new("inst-a") { Id = 1, SiteId = SiteId, State = InstanceState.Enabled }, + }); + _snapshotClient.GetSnapshotAsync(SiteIdentifier, "inst-a", Arg.Any()) + .Returns(new DebugViewSnapshot( + "inst-a", + Array.Empty(), + new[] { new AlarmStateChanged("inst-a", "A-alarm", AlarmState.Active, 500, T0) }, + T0)); + + var services = new ServiceCollection(); + services.AddSingleton(_instanceRepo); + services.AddSingleton(_siteRepo); + services.AddSingleton(_snapshotClient); + services.AddScoped(); + _provider = services.BuildServiceProvider(); + } + + private SharedAlarmSummaryService CreateSut(TimeSpan liveCacheTtl) => + new(_provider.GetRequiredService(), _liveCache, liveCacheTtl, () => _now); + + [Fact] + public async Task ConcurrentCircuits_ShareOneFanOut() + { + var sut = CreateSut(TimeSpan.FromSeconds(60)); + + var results = await Task.WhenAll(Enumerable.Range(0, 8).Select(_ => sut.GetSiteAlarmsAsync(SiteId))); + + await _instanceRepo.Received(1).GetInstancesBySiteIdAsync(SiteId, Arg.Any()); + Assert.All(results, r => Assert.Single(r.Alarms)); + } + + [Fact] + public async Task ColdLiveCache_RefreshesWithinThePageTick() + { + var sut = CreateSut(TimeSpan.FromSeconds(60)); + _liveCache.Live = false; + + await sut.GetSiteAlarmsAsync(SiteId); + // The page polls every 15s and, while the cache is cold, the poll is its + // authoritative rebuild — so the memo must have expired by then. + _now = T0.AddSeconds(15); + await sut.GetSiteAlarmsAsync(SiteId); + + await _instanceRepo.Received(2).GetInstancesBySiteIdAsync(SiteId, Arg.Any()); + } + + [Fact] + public async Task LiveCacheServing_WidensTheWindowToTheReconcileInterval() + { + var sut = CreateSut(TimeSpan.FromSeconds(60)); + _liveCache.Live = true; + + await sut.GetSiteAlarmsAsync(SiteId); + _now = T0.AddSeconds(30); + await sut.GetSiteAlarmsAsync(SiteId); + + // Live deltas own the rows; only the not-reporting list still comes from the + // fan-out, so a 30s-old answer is fine and costs no second fan-out. + await _instanceRepo.Received(1).GetInstancesBySiteIdAsync(SiteId, Arg.Any()); + + _now = T0.AddSeconds(61); + await sut.GetSiteAlarmsAsync(SiteId); + await _instanceRepo.Received(2).GetInstancesBySiteIdAsync(SiteId, Arg.Any()); + } + + [Fact] + public async Task DifferentSites_DoNotShareAMemoSlot() + { + const int otherSite = 8; + _siteRepo.GetSiteByIdAsync(otherSite, Arg.Any()) + .Returns(new Site("Plant B", "plant-b") { Id = otherSite }); + _instanceRepo.GetInstancesBySiteIdAsync(otherSite, Arg.Any()) + .Returns(new List()); + + var sut = CreateSut(TimeSpan.FromSeconds(60)); + + await sut.GetSiteAlarmsAsync(SiteId); + await sut.GetSiteAlarmsAsync(otherSite); + + await _instanceRepo.Received(1).GetInstancesBySiteIdAsync(SiteId, Arg.Any()); + await _instanceRepo.Received(1).GetInstancesBySiteIdAsync(otherSite, Arg.Any()); + } + + [Fact] + public void PureMethods_MatchTheDirectImplementation() + { + var sut = CreateSut(TimeSpan.FromSeconds(60)); + var direct = new AlarmSummaryService(_instanceRepo, _siteRepo, _snapshotClient); + var alarms = new List + { + new("inst-b", "B-alarm", AlarmState.Active, 900, T0), + new("inst-a", "A-alarm", AlarmState.Normal, 100, T0), + }; + + var shared = sut.BuildFromLiveAlarms(alarms); + var expected = direct.BuildFromLiveAlarms(alarms); + + Assert.Equal( + expected.Alarms.Select(r => r.Alarm.AlarmName), + shared.Alarms.Select(r => r.Alarm.AlarmName)); + var expectedRollup = direct.ComputeRollup(expected.Alarms); + var sharedRollup = sut.ComputeRollup(shared.Alarms); + Assert.Equal(expectedRollup.TotalActive, sharedRollup.TotalActive); + Assert.Equal(expectedRollup.WorstSeverity, sharedRollup.WorstSeverity); + Assert.Equal(expectedRollup.UnackedCount, sharedRollup.UnackedCount); + Assert.Equal(expectedRollup.CountsByKind, sharedRollup.CountsByKind); + } + + public void Dispose() => _provider.Dispose(); + + /// Liveness-only stub — the façade consults nothing else on the live cache. + private sealed class FakeLiveCache : ISiteAlarmLiveCache + { + public bool Live { get; set; } + + public IDisposable Subscribe(int siteId, Action onChanged) => new NoOp(); + + public IReadOnlyList GetCurrentAlarms(int siteId) => + Array.Empty(); + + public bool IsLive(int siteId) => Live; + + private sealed class NoOp : IDisposable + { + public void Dispose() { } + } + } +} diff --git a/tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Services/SingleFlightMemoTests.cs b/tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Services/SingleFlightMemoTests.cs new file mode 100644 index 00000000..0dd5ad5a --- /dev/null +++ b/tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Services/SingleFlightMemoTests.cs @@ -0,0 +1,166 @@ +using ZB.MOM.WW.ScadaBridge.CentralUI.Services; + +namespace ZB.MOM.WW.ScadaBridge.CentralUI.Tests.Services; + +/// +/// Unit tests for — the primitive behind the process-level +/// KPI snapshot cache and the shared Alarm Summary fan-out (arch-review WP2.4). The two +/// properties the whole optimisation rests on are tested here: single-flight (N +/// concurrent circuits asking the same question produce ONE query) and TTL (a query +/// answered inside the freshness window is served from memory, and one outside it is not). +/// +public class SingleFlightMemoTests +{ + private static readonly DateTimeOffset T0 = new(2026, 8, 14, 9, 0, 0, TimeSpan.Zero); + + [Fact] + public async Task ConcurrentCallers_ShareOneFlight() + { + var released = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var invocations = 0; + var memo = new SingleFlightMemo(TimeSpan.FromSeconds(10), () => T0); + + // Ten circuits ask while the first flight is still in progress. + var waiters = Enumerable.Range(0, 10) + .Select(_ => memo.GetAsync(() => + { + Interlocked.Increment(ref invocations); + return released.Task; + })) + .ToArray(); + + released.SetResult(42); + var results = await Task.WhenAll(waiters); + + Assert.Equal(1, invocations); + Assert.Equal(1, memo.FlightCount); + Assert.All(results, r => Assert.Equal(42, r)); + } + + [Fact] + public async Task WithinTtl_ServesTheMemoizedValue_WithoutRequerying() + { + var now = T0; + var invocations = 0; + var memo = new SingleFlightMemo(TimeSpan.FromSeconds(8), () => now); + + var first = await memo.GetAsync(() => Task.FromResult(++invocations)); + + now = T0.AddSeconds(7); + var second = await memo.GetAsync(() => Task.FromResult(++invocations)); + + Assert.Equal(1, first); + Assert.Equal(1, second); + Assert.Equal(1, memo.FlightCount); + } + + [Fact] + public async Task AfterTtl_RequeriesOnce() + { + var now = T0; + var invocations = 0; + var memo = new SingleFlightMemo(TimeSpan.FromSeconds(8), () => now); + + await memo.GetAsync(() => Task.FromResult(++invocations)); + + now = T0.AddSeconds(9); + var refreshed = await memo.GetAsync(() => Task.FromResult(++invocations)); + + Assert.Equal(2, refreshed); + Assert.Equal(2, memo.FlightCount); + } + + [Fact] + public async Task ForceRefresh_BypassesAFreshValue() + { + var now = T0; + var invocations = 0; + var memo = new SingleFlightMemo(TimeSpan.FromSeconds(60), () => now); + + await memo.GetAsync(() => Task.FromResult(++invocations)); + var forced = await memo.GetAsync(() => Task.FromResult(++invocations), forceRefresh: true); + + Assert.Equal(2, forced); + Assert.Equal(2, memo.FlightCount); + } + + [Fact] + public async Task ForceRefresh_StillJoinsAnInFlightRound() + { + var released = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var invocations = 0; + var memo = new SingleFlightMemo(TimeSpan.FromSeconds(60), () => T0); + + var first = memo.GetAsync(() => + { + Interlocked.Increment(ref invocations); + return released.Task; + }); + // A burst of operator "Refresh" clicks must not multiply the query. + var forcedA = memo.GetAsync(() => Task.FromResult(-1), forceRefresh: true); + var forcedB = memo.GetAsync(() => Task.FromResult(-1), forceRefresh: true); + + released.SetResult(7); + Assert.Equal(7, await first); + Assert.Equal(7, await forcedA); + Assert.Equal(7, await forcedB); + Assert.Equal(1, invocations); + } + + [Fact] + public async Task Failure_IsNotMemoized_AndTheNextCallerRetries() + { + var now = T0; + var invocations = 0; + var memo = new SingleFlightMemo(TimeSpan.FromSeconds(60), () => now); + + await Assert.ThrowsAsync(() => + memo.GetAsync(() => + { + invocations++; + return Task.FromException(new InvalidOperationException("KPI query failed")); + })); + + // Same instant, well inside the TTL — a cached fault would starve the tiles for a + // whole window, so the next caller must produce a new flight. + var recovered = await memo.GetAsync(() => Task.FromResult(++invocations)); + + Assert.Equal(2, recovered); + Assert.Equal(2, memo.FlightCount); + } + + [Fact] + public async Task TtlOverride_AppliesToThatFlight() + { + var now = T0; + var invocations = 0; + var memo = new SingleFlightMemo(TimeSpan.FromSeconds(5), () => now); + + // Produced under a 60s override — still fresh 30s later despite the 5s default. + await memo.GetAsync(() => Task.FromResult(++invocations), ttlOverride: TimeSpan.FromSeconds(60)); + + now = T0.AddSeconds(30); + var second = await memo.GetAsync(() => Task.FromResult(++invocations)); + + Assert.Equal(1, second); + Assert.Equal(1, memo.FlightCount); + } + + [Fact] + public async Task CallerCancellation_DoesNotCancelTheSharedFlight() + { + var released = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var memo = new SingleFlightMemo(TimeSpan.FromSeconds(60), () => T0); + + using var giveUp = new CancellationTokenSource(); + var impatient = memo.GetAsync(() => released.Task, cancellationToken: giveUp.Token); + var patient = memo.GetAsync(() => Task.FromResult(-1)); + + giveUp.Cancel(); + await Assert.ThrowsAnyAsync(() => impatient); + + // The abandoned wait must not have taken the round down with it. + released.SetResult(99); + Assert.Equal(99, await patient); + } +} From a212283104217a34e722d63b8186c4969dd831c3 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Fri, 14 Aug 2026 20:59:43 -0400 Subject: [PATCH 11/31] perf(misc): cached hot-path lookups, bounded observer queue, alarm-priority stream path MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit WP2.6 (arch-review remediation, cross-cutting misc): - SiteExternalSystemRepository: name/ID-indexed ExternalSystemDefinitionCache replaces the fetch-all + reverse-map scan on every by-ID/method lookup; loaded once per redeploy, invalidated by DeploymentManagerActor after HandleDeployArtifacts applies external-system changes. Static JsonSerializerOptions for method-list parsing. - Inbound API: short-TTL ApiMethodCache fronts the per-request ApiMethod repository fetch; invalidated by name via the existing ScriptArtifactChangeSubscriber/ IScriptArtifactChangeBus pipeline, self-healing via TTL for changes the bus doesn't cover (e.g. Management API edits). - StoreAndForward: the cached-call audit-observer queue — the one unbounded channel left in the system — is now bounded (ObserverQueueCapacity, default 10,000) with DropOldest overflow and a dropped-notification counter. - SiteStreamManager: alarm state changes now travel a dedicated publish source/broadcast hub, isolated from the (far higher-volume) attribute path, so an attribute storm can no longer evict a pending alarm transition; the alarm hand-off queue is bounded with a drop counter surfaced on the site health report (SiteStreamAlarmDropCount via the new SiteStreamAlarmDropReporter), and publishing is skipped entirely at zero subscribers on either path. - CLI ManagementHttpClient: explicit 30s HttpClient.Timeout on the shared construction (was the 100s framework default), overridable via SCADABRIDGE_HTTP_TIMEOUT_SECONDS. Deviation: the failback-probe heartbeat item is NOT included — its only viable surface (CentralChannelProvider.cs / heartbeat consumers) lives entirely in the Communication project, explicitly off-limits to this work package this phase. Tests: SiteRuntime.Tests (550), InboundAPI.Tests (278), StoreAndForward.Tests (133), CLI.Tests (390), HealthMonitoring.Tests (97) — all green after full solution build. --- .../ManagementHttpClient.cs | 47 +++- .../Messages/Health/SiteHealthReport.cs | 11 + .../ISiteHealthCollector.cs | 16 ++ .../SiteHealthCollector.cs | 13 +- .../Actors/AkkaHostedService.cs | 8 +- .../ApiMethodCache.cs | 85 ++++++++ .../EndpointExtensions.cs | 11 +- .../InboundApiOptions.cs | 14 ++ .../InboundApiOptionsValidator.cs | 4 + .../ScriptArtifactChangeSubscriber.cs | 8 + .../ServiceCollectionExtensions.cs | 8 + .../Actors/DeploymentManagerActor.cs | 24 +- .../ExternalSystemDefinitionCache.cs | 89 ++++++++ .../SiteExternalSystemRepository.cs | 167 ++++++++------ .../ServiceCollectionExtensions.cs | 19 +- .../SiteRuntimeOptions.cs | 13 ++ .../SiteRuntimeOptionsValidator.cs | 4 + .../Streaming/SiteStreamAlarmDropReporter.cs | 78 +++++++ .../Streaming/SiteStreamManager.cs | 206 +++++++++++++++--- .../StoreAndForwardOptions.cs | 14 ++ .../StoreAndForwardOptionsValidator.cs | 4 + .../StoreAndForwardService.cs | 60 ++++- .../ManagementHttpClientTests.cs | 75 +++++++ .../ApiMethodCacheTests.cs | 110 ++++++++++ .../InboundApiOptionsValidatorTests.cs | 18 ++ .../ScriptArtifactChangeSubscriberTests.cs | 31 ++- .../Repositories/SiteRepositoryTests.cs | 95 ++++++++ .../SiteRuntimeOptionsValidatorTests.cs | 9 + .../SiteStreamAlarmDropReporterTests.cs | 57 +++++ .../Streaming/SiteStreamManagerTests.cs | 82 +++++++ .../StoreAndForwardOptionsValidatorTests.cs | 12 + .../StoreAndForwardServiceTests.cs | 80 +++++++ 32 files changed, 1361 insertions(+), 111 deletions(-) create mode 100644 src/ZB.MOM.WW.ScadaBridge.InboundAPI/ApiMethodCache.cs create mode 100644 src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Repositories/ExternalSystemDefinitionCache.cs create mode 100644 src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Streaming/SiteStreamAlarmDropReporter.cs create mode 100644 tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests/ApiMethodCacheTests.cs create mode 100644 tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Streaming/SiteStreamAlarmDropReporterTests.cs diff --git a/src/ZB.MOM.WW.ScadaBridge.CLI/ManagementHttpClient.cs b/src/ZB.MOM.WW.ScadaBridge.CLI/ManagementHttpClient.cs index e1b62e62..18304a41 100644 --- a/src/ZB.MOM.WW.ScadaBridge.CLI/ManagementHttpClient.cs +++ b/src/ZB.MOM.WW.ScadaBridge.CLI/ManagementHttpClient.cs @@ -9,13 +9,53 @@ public class ManagementHttpClient : IDisposable private readonly HttpClient _httpClient; /// - /// Initializes a new instance of the class. + /// WP2.6e (arch-review misc — CLI HttpClient timeout): default overall + /// for the shared client construction (30 s). This + /// bounds a hung/black-holed connection — before this, the public constructor left + /// at its framework default (100 s), silently longer + /// than most CLI callers' own per-request TimeSpan timeout argument + /// (// + /// already bound each call via their own , but a + /// connection attempt that never completes at all — no response headers, ever — is + /// bounded by instead, since that governs the whole + /// request/response including connect). Config-overridable via the + /// SCADABRIDGE_HTTP_TIMEOUT_SECONDS environment variable, consistent with how + /// every other CLI setting is overridden (see ) — kept + /// self-contained here (no /command-file plumbing) since CLI + /// commands are owned by a separate work package this phase. + /// + public static readonly TimeSpan DefaultTimeout = TimeSpan.FromSeconds(30); + + /// Test seam (WP2.6e) — the effective this instance was constructed with. + internal TimeSpan EffectiveTimeout { get; } + + /// + /// Resolves the effective default timeout: the SCADABRIDGE_HTTP_TIMEOUT_SECONDS + /// environment variable when set to a positive integer, otherwise . + /// + private static TimeSpan ResolveDefaultTimeout() + { + var env = Environment.GetEnvironmentVariable("SCADABRIDGE_HTTP_TIMEOUT_SECONDS"); + if (!string.IsNullOrWhiteSpace(env) + && int.TryParse(env, out var seconds) + && seconds > 0) + { + return TimeSpan.FromSeconds(seconds); + } + + return DefaultTimeout; + } + + /// + /// Initializes a new instance of the class, with + /// set to + /// (30 s, or the SCADABRIDGE_HTTP_TIMEOUT_SECONDS override). /// /// The base URL for the management API. /// The username for HTTP Basic authentication. /// The password for HTTP Basic authentication. public ManagementHttpClient(string baseUrl, string username, string password) - : this(new HttpClient(), baseUrl, username, password) + : this(new HttpClient { Timeout = ResolveDefaultTimeout() }, baseUrl, username, password) { } @@ -31,6 +71,9 @@ public class ManagementHttpClient : IDisposable internal ManagementHttpClient(HttpClient httpClient, string baseUrl, string username, string password) { _httpClient = httpClient; + // Test seam (WP2.6e): exposes the constructed HttpClient's effective Timeout + // without requiring reflection. + EffectiveTimeout = httpClient.Timeout; _httpClient.BaseAddress = new Uri(baseUrl.TrimEnd('/') + "/"); var credentials = Convert.ToBase64String(Encoding.UTF8.GetBytes($"{username}:{password}")); _httpClient.DefaultRequestHeaders.Authorization = diff --git a/src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Health/SiteHealthReport.cs b/src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Health/SiteHealthReport.cs index 8306ffdc..78d77ea0 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Health/SiteHealthReport.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Health/SiteHealthReport.cs @@ -104,6 +104,17 @@ public record SiteHealthReport( /// pair as perfectly healthy. Collapsing null to 0 anywhere on this path is a bug. /// public long? LocalDbOplogBacklog { get; init; } + + /// + /// WP2.6d (arch-review misc — site stream alarm-vs-attribute shared buffer): + /// cumulative count of alarm state changes dropped from + /// SiteStreamManager's dedicated alarm publish queue because it was at + /// capacity — i.e. the site-wide alarm hand-off itself fell behind, not a + /// per-subscriber buffer. Point-in-time (not reset on collect), refreshed by + /// the site-side SiteStreamAlarmDropReporter hosted service. Defaults to + /// 0 so existing producers/tests that don't populate it stay valid. + /// + public long SiteStreamAlarmDropCount { get; init; } } /// diff --git a/src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/ISiteHealthCollector.cs b/src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/ISiteHealthCollector.cs index 8781553a..a1a19a4d 100644 --- a/src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/ISiteHealthCollector.cs +++ b/src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/ISiteHealthCollector.cs @@ -178,6 +178,22 @@ public interface ISiteHealthCollector // Default no-op so test fakes do not need to be updated. } + /// + /// WP2.6d (arch-review misc — site stream alarm-vs-attribute shared buffer): + /// replace the latest cumulative count of alarm state changes dropped from + /// SiteStreamManager's dedicated alarm publish queue, used by the next + /// call. Refreshed periodically by the site-side + /// SiteStreamAlarmDropReporter hosted service. Point-in-time: NOT reset on + /// . Default interface implementation is a no-op so + /// existing test fakes continue to compile without per-fake updates. + /// + /// The cumulative dropped-alarm count from SiteStreamManager.AlarmPublishDroppedCount. + void SetSiteStreamAlarmDropCount(long count) + { + // Default no-op so test fakes do not need to be updated. The real + // SiteHealthCollector overrides this with the Interlocked.Exchange store. + } + /// /// Sets the hostname of this node. /// diff --git a/src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/SiteHealthCollector.cs b/src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/SiteHealthCollector.cs index be008182..505a4bd1 100644 --- a/src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/SiteHealthCollector.cs +++ b/src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/SiteHealthCollector.cs @@ -39,6 +39,10 @@ public class SiteHealthCollector : ISiteHealthCollector private int _scriptQueueDepth; private int _scriptBusyThreads; private long _scriptOldestBusyAgeBits = BitConverter.DoubleToInt64Bits(double.NaN); + // WP2.6d: cumulative alarm-publish-queue drop count, refreshed by + // SiteStreamAlarmDropReporter. Point-in-time (not reset on collect), like the + // scheduler gauges above. + private long _siteStreamAlarmDropCount; private volatile string _nodeHostname = ""; private volatile IReadOnlyList? _clusterNodes; private volatile bool _isActiveNode; @@ -173,6 +177,12 @@ public class SiteHealthCollector : ISiteHealthCollector return double.IsNaN(value) ? null : value; } + /// + public void SetSiteStreamAlarmDropCount(long count) + { + Interlocked.Exchange(ref _siteStreamAlarmDropCount, count); + } + /// public void SetNodeHostname(string hostname) => _nodeHostname = hostname; @@ -277,7 +287,8 @@ public class SiteHealthCollector : ISiteHealthCollector // not run) leaves both report fields null — "no data", not "disconnected // with an empty backlog". LocalDbReplicationConnected = localDbReplication?.Item1, - LocalDbOplogBacklog = localDbReplication?.Item2 + LocalDbOplogBacklog = localDbReplication?.Item2, + SiteStreamAlarmDropCount = Interlocked.Read(ref _siteStreamAlarmDropCount) }; } } diff --git a/src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs b/src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs index bc9fdc2e..66d6996d 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs @@ -836,12 +836,18 @@ akka {{ // writes (static overrides, native_alarm_state) could be cut off on // graceful failover. Names unchanged: deployment-manager-singleton / // deployment-manager-proxy. + // WP2.6a: the shared cache backing SiteExternalSystemRepository reads + // (registered singleton by AddSiteRuntime); DeploymentManagerActor invalidates + // it after an artifact deploy applies external-system changes. + var externalSystemCache = _serviceProvider + .GetService(); var dm = SingletonRegistrar.Start( _actorSystem!, "deployment-manager", Props.Create(() => new DeploymentManagerActor( storage, compilationService, sharedScriptLibrary, streamManager, siteRuntimeOptionsValue, dmLogger, dclManager, - siteHealthCollector, _serviceProvider, null, deploymentConfigFetcher)), + siteHealthCollector, _serviceProvider, null, deploymentConfigFetcher, + null, null, externalSystemCache)), _logger, role: siteRole); var dmProxy = dm.Proxy; diff --git a/src/ZB.MOM.WW.ScadaBridge.InboundAPI/ApiMethodCache.cs b/src/ZB.MOM.WW.ScadaBridge.InboundAPI/ApiMethodCache.cs new file mode 100644 index 00000000..6322b3d0 --- /dev/null +++ b/src/ZB.MOM.WW.ScadaBridge.InboundAPI/ApiMethodCache.cs @@ -0,0 +1,85 @@ +using System.Collections.Concurrent; +using ZB.MOM.WW.ScadaBridge.Commons.Entities.InboundApi; + +namespace ZB.MOM.WW.ScadaBridge.InboundAPI; + +/// +/// WP2.6b (arch-review misc — Inbound API per-request SQL): a short-TTL cache of +/// resolved ApiMethod rows, sitting in front of IInboundApiRepository.GetMethodByNameAsync +/// so a repeated POST /api/{methodName} burst for the same method does not each pay a +/// database round-trip. +/// +/// Two invalidation paths, by design: +/// +/// +/// Fast path. calls +/// for every name in an ApiMethod +/// ScriptArtifactsChanged notification (bundle-import changes today). +/// Fallback. Every entry — including a negative (not-found) result — +/// expires after regardless of whether an +/// invalidation notification ever arrives. This is what keeps an operator edit via the +/// Management API/UI (which does not publish to the change bus) correct within a bounded +/// window, and is also the correctness backstop for the bus's at-least-once/in-process-only +/// contract — a missed or duplicate notification is harmless. +/// +/// +/// A negative (method not found) result is cached too — bounded by the same short TTL — so a +/// caller with a provisioned-but-nonexistent method scope (a real misconfiguration case, not an +/// enumeration probe: EndpointExtensions only reaches this cache after the in-memory +/// scope check already passed) does not repeatedly pay a miss query. +/// +/// +public sealed class ApiMethodCache +{ + private sealed record Entry(ApiMethod? Method, DateTimeOffset ExpiresAtUtc); + + private readonly ConcurrentDictionary _entries = new(StringComparer.Ordinal); + private readonly TimeSpan _ttl; + private readonly TimeProvider _timeProvider; + + /// Initializes the cache. + /// How long a resolved (or not-found) entry stays valid. + /// Optional time source; defaults to . + public ApiMethodCache(TimeSpan ttl, TimeProvider? timeProvider = null) + { + if (ttl <= TimeSpan.Zero) + throw new ArgumentOutOfRangeException(nameof(ttl), ttl, "ApiMethodCache TTL must be positive."); + + _ttl = ttl; + _timeProvider = timeProvider ?? TimeProvider.System; + } + + /// + /// Returns the cached entry for if it has not expired; + /// otherwise invokes , caches the result (found or not-found), and + /// returns it. Concurrent callers on a cache miss for the SAME name may each invoke + /// once (last write wins) — an acceptable, rare race in exchange + /// for not serializing every miss behind a lock; the point of the cache is to absorb + /// repeated hits, not to fully dedupe a first-request thundering herd. + /// + /// The method name to resolve. + /// Loads the current row from the repository on a cache miss. + /// Cancellation token for the fetch. + public async Task GetOrFetchAsync( + string methodName, + Func> fetch, + CancellationToken cancellationToken) + { + var now = _timeProvider.GetUtcNow(); + if (_entries.TryGetValue(methodName, out var cached) && cached.ExpiresAtUtc > now) + { + return cached.Method; + } + + var method = await fetch(cancellationToken).ConfigureAwait(false); + _entries[methodName] = new Entry(method, now + _ttl); + return method; + } + + /// Drops the cached entry for one method name. Safe to call for an unknown name. + /// The method name to invalidate. + public void Invalidate(string methodName) => _entries.TryRemove(methodName, out _); + + /// Diagnostic helper — the number of currently-cached entries (including expired-but-not-evicted ones). + internal int Count => _entries.Count; +} diff --git a/src/ZB.MOM.WW.ScadaBridge.InboundAPI/EndpointExtensions.cs b/src/ZB.MOM.WW.ScadaBridge.InboundAPI/EndpointExtensions.cs index 0f753ef5..abb6e0cd 100644 --- a/src/ZB.MOM.WW.ScadaBridge.InboundAPI/EndpointExtensions.cs +++ b/src/ZB.MOM.WW.ScadaBridge.InboundAPI/EndpointExtensions.cs @@ -91,6 +91,7 @@ public static class EndpointExtensions var logger = httpContext.RequestServices.GetRequiredService>(); var verifier = httpContext.RequestServices.GetRequiredService(); var repository = httpContext.RequestServices.GetRequiredService(); + var methodCache = httpContext.RequestServices.GetRequiredService(); var executor = httpContext.RequestServices.GetRequiredService(); var routeHelper = httpContext.RequestServices.GetRequiredService(); var options = httpContext.RequestServices.GetRequiredService>().Value; @@ -150,9 +151,17 @@ public static class EndpointExtensions // "Echo" does not grant "echo". This is the intended invariant: method // names are case-sensitive identifiers, and a key's granted scopes must be // provisioned with the exact casing of the methods they authorize. + // WP2.6b: the per-request repository round-trip is fronted by a short-TTL + // ApiMethodCache (invalidated by name on an ApiMethod ScriptArtifactsChanged + // notification, and self-healing via TTL expiry otherwise — see ApiMethodCache + // and ScriptArtifactChangeSubscriber). Only reached after the in-scope check + // passes, preserving the existing enumeration-safety ordering below. var inScope = identity.Scopes.Contains(methodName); var method = inScope - ? await repository.GetMethodByNameAsync(methodName, httpContext.RequestAborted) + ? await methodCache.GetOrFetchAsync( + methodName, + ct => repository.GetMethodByNameAsync(methodName, ct), + httpContext.RequestAborted) : null; if (method == null || !inScope) diff --git a/src/ZB.MOM.WW.ScadaBridge.InboundAPI/InboundApiOptions.cs b/src/ZB.MOM.WW.ScadaBridge.InboundAPI/InboundApiOptions.cs index d9ce3ddf..cd172a61 100644 --- a/src/ZB.MOM.WW.ScadaBridge.InboundAPI/InboundApiOptions.cs +++ b/src/ZB.MOM.WW.ScadaBridge.InboundAPI/InboundApiOptions.cs @@ -34,4 +34,18 @@ public class InboundApiOptions /// /// public string ApiKeyPepper { get; set; } = string.Empty; + + /// + /// WP2.6b (arch-review misc — Inbound API per-request SQL): how long a resolved + /// ApiMethod row (found OR not-found) is served from + /// before the next request re-fetches it from the repository. The + /// ScriptArtifactChangeSubscriber/IScriptArtifactChangeBus pipeline + /// invalidates a specific method's entry immediately on a bundle-import change (the fast + /// path); this TTL is the self-healing fallback for changes the bus does not cover (e.g. + /// an operator edit via the Management API/UI, which does not publish a bus notification), + /// bounding staleness without requiring every mutation path to remember to invalidate. + /// Short by design — the whole point is to absorb repeated-request-storm SQL, not to + /// serve minutes-old method definitions. + /// + public TimeSpan ApiMethodCacheTtl { get; set; } = TimeSpan.FromSeconds(5); } diff --git a/src/ZB.MOM.WW.ScadaBridge.InboundAPI/InboundApiOptionsValidator.cs b/src/ZB.MOM.WW.ScadaBridge.InboundAPI/InboundApiOptionsValidator.cs index 23bc7b0b..574dcd37 100644 --- a/src/ZB.MOM.WW.ScadaBridge.InboundAPI/InboundApiOptionsValidator.cs +++ b/src/ZB.MOM.WW.ScadaBridge.InboundAPI/InboundApiOptionsValidator.cs @@ -27,5 +27,9 @@ public sealed class InboundApiOptionsValidator : OptionsValidatorBase 0, $"ScadaBridge:InboundApi:MaxRequestBodyBytes must be greater than zero " + $"(was {options.MaxRequestBodyBytes}); it caps the accepted request body before buffering."); + + builder.RequireThat(options.ApiMethodCacheTtl > TimeSpan.Zero, + $"ScadaBridge:InboundApi:ApiMethodCacheTtl must be a positive duration " + + $"(was {options.ApiMethodCacheTtl}); it bounds how long a resolved ApiMethod row is cached."); } } diff --git a/src/ZB.MOM.WW.ScadaBridge.InboundAPI/ScriptArtifactChangeSubscriber.cs b/src/ZB.MOM.WW.ScadaBridge.InboundAPI/ScriptArtifactChangeSubscriber.cs index dff2f3cf..8d8bd5c1 100644 --- a/src/ZB.MOM.WW.ScadaBridge.InboundAPI/ScriptArtifactChangeSubscriber.cs +++ b/src/ZB.MOM.WW.ScadaBridge.InboundAPI/ScriptArtifactChangeSubscriber.cs @@ -27,20 +27,27 @@ namespace ZB.MOM.WW.ScadaBridge.InboundAPI; public sealed class ScriptArtifactChangeSubscriber : IHostedService { private readonly InboundScriptExecutor _executor; + private readonly ApiMethodCache _methodCache; private readonly ILogger _logger; private readonly IScriptArtifactChangeBus? _bus; private IDisposable? _subscription; /// Initializes the subscriber. /// The compiled-handler cache to invalidate. + /// + /// WP2.6b: the short-TTL resolved-ApiMethod-row cache to invalidate alongside + /// the compiled handler. + /// /// Logger instance. /// The change bus, or null when the host registers none (site roles, tests). public ScriptArtifactChangeSubscriber( InboundScriptExecutor executor, + ApiMethodCache methodCache, ILogger logger, IScriptArtifactChangeBus? bus = null) { _executor = executor; + _methodCache = methodCache; _logger = logger; _bus = bus; } @@ -78,6 +85,7 @@ public sealed class ScriptArtifactChangeSubscriber : IHostedService foreach (var name in notification.Names) { _executor.InvalidateMethod(name); + _methodCache.Invalidate(name); } _logger.LogInformation( diff --git a/src/ZB.MOM.WW.ScadaBridge.InboundAPI/ServiceCollectionExtensions.cs b/src/ZB.MOM.WW.ScadaBridge.InboundAPI/ServiceCollectionExtensions.cs index bf694c50..57669485 100644 --- a/src/ZB.MOM.WW.ScadaBridge.InboundAPI/ServiceCollectionExtensions.cs +++ b/src/ZB.MOM.WW.ScadaBridge.InboundAPI/ServiceCollectionExtensions.cs @@ -1,4 +1,5 @@ using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Options; namespace ZB.MOM.WW.ScadaBridge.InboundAPI; @@ -19,6 +20,13 @@ public static class ServiceCollectionExtensions services.AddSingleton(); services.AddScoped(); + // WP2.6b: short-TTL cache of resolved ApiMethod rows, sitting in front of the + // per-request repository fetch in EndpointExtensions. Invalidated by name via + // ScriptArtifactChangeSubscriber (below) and self-heals via TTL expiry for + // changes the bus does not cover. + services.AddSingleton(sp => new ApiMethodCache( + sp.GetRequiredService>().Value.ApiMethodCacheTtl)); + // Routed calls go through the IInstanceRouter seam; the // production implementation delegates to CommunicationService. services.AddScoped(); diff --git a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs index 338818c6..938de1fa 100644 --- a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs +++ b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs @@ -16,6 +16,7 @@ using ZB.MOM.WW.ScadaBridge.SiteEventLogging; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Deployment; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Messages; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; +using ZB.MOM.WW.ScadaBridge.SiteRuntime.Repositories; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Streaming; @@ -40,6 +41,14 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers // DeployInstanceCommand is validated before any actor/persistence work. private readonly DeployCompileValidator _deployCompileValidator; private readonly SharedScriptLibrary _sharedScriptLibrary; + /// + /// WP2.6a: the shared cache backing SiteExternalSystemRepository reads. + /// Invalidated wholesale after applies + /// external-system changes so a script's next call re-reads fresh definitions/method + /// lists instead of serving a stale cached snapshot. Null in tests that do not wire + /// external-system caching — the invalidation call is then simply skipped. + /// + private readonly ExternalSystemDefinitionCache? _externalSystemCache; private readonly SiteStreamManager? _streamManager; private readonly SiteRuntimeOptions _options; private readonly ILogger _logger; @@ -157,6 +166,12 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers /// deployed configuration at startup; defaults to 5 seconds when null. /// Optional override for loading all deployed configurations /// at startup; defaults to reading from . Primarily for tests. + /// + /// WP2.6a: the shared backing + /// SiteExternalSystemRepository reads, invalidated after this actor applies + /// external-system changes. Optional/null in tests that do not exercise external-system + /// caching. + /// public DeploymentManagerActor( SiteStorageService storage, ScriptCompilationService compilationService, @@ -170,12 +185,14 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers ILoggerFactory? loggerFactory = null, IDeploymentConfigFetcher? configFetcher = null, TimeSpan? startupLoadRetryInterval = null, - Func>>? configLoader = null) + Func>>? configLoader = null, + ExternalSystemDefinitionCache? externalSystemCache = null) { _storage = storage; _compilationService = compilationService; _deployCompileValidator = new DeployCompileValidator(compilationService); _sharedScriptLibrary = sharedScriptLibrary; + _externalSystemCache = externalSystemCache; _streamManager = streamManager; _options = options; _dclManager = dclManager; @@ -1914,6 +1931,11 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers await _storage.DeleteExternalSystemsExceptAsync( command.ExternalSystems.Select(es => es.Name).ToList()); + + // WP2.6a: drop the shared read cache so the next script call sees + // the freshly-applied definitions/method lists instead of a stale + // snapshot loaded before this deploy. + _externalSystemCache?.InvalidateAll(); } // Store database connection definitions diff --git a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Repositories/ExternalSystemDefinitionCache.cs b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Repositories/ExternalSystemDefinitionCache.cs new file mode 100644 index 00000000..b4f6b512 --- /dev/null +++ b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Repositories/ExternalSystemDefinitionCache.cs @@ -0,0 +1,89 @@ +using ZB.MOM.WW.ScadaBridge.Commons.Entities.ExternalSystems; + +namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Repositories; + +/// +/// WP2.6a (arch-review misc — site external-system resolution): the shared, +/// redeploy-invalidated snapshot backing 's +/// reads. +/// +/// Registered as a DI singleton in production (AddSiteRuntime) so every +/// scoped instance — one per script +/// execution's DI scope — shares ONE snapshot instead of re-querying and +/// re-parsing the external_systems table (including its JSON +/// method_definitions column) on every call. The repository's default +/// constructor also accepts a private, unshared instance of this class so tests that +/// construct the repository directly keep seeing freshly-written rows without any +/// caching surprise. +/// +/// +/// Loaded lazily and wholesale on first read (one query covering every system's +/// definition AND its parsed method list, indexed by name and by +/// for O(1) by-ID lookups) and dropped wholesale by +/// — called by DeploymentManagerActor after an +/// artifact deploy applies new/changed/removed external systems, so the cache never +/// serves a stale method list or a deleted system past the next read. +/// +/// +public sealed class ExternalSystemDefinitionCache +{ + /// + /// One fully-loaded, immutable view of the site's external-system catalogue. + /// + /// System definitions keyed by name (ordinal). + /// Synthetic system ID → name, for O(1) by-ID resolution. + /// Parsed method list per system name. + /// Synthetic method ID → (system name, method name), for O(1) by-ID resolution. + internal sealed record Snapshot( + IReadOnlyDictionary ByName, + IReadOnlyDictionary IdToName, + IReadOnlyDictionary> MethodsByName, + IReadOnlyDictionary MethodIdIndex); + + // Guards concurrent cache-miss loads so a burst of first-callers after a redeploy + // collapses onto a single SQLite read rather than each racing its own full scan. + private readonly SemaphoreSlim _loadGate = new(1, 1); + + // Reference reads/writes of a class type are already atomic in .NET; volatile only + // adds the memory-visibility guarantee across threads, which is all the + // check-then-load double-check pattern below needs. + private volatile Snapshot? _snapshot; + + /// + /// Drops the cached snapshot. The next read reloads a fresh one from storage. + /// Safe to call at any time, including with no snapshot yet loaded. + /// + public void InvalidateAll() => _snapshot = null; + + /// + /// Returns the cached snapshot, loading it via on a + /// cache miss (first read, or the first read after ). + /// Concurrent misses collapse onto a single load. + /// + /// Loads a fresh snapshot from storage; invoked at most once per miss. + /// Cancellation token for the load. + internal async Task GetOrLoadAsync( + Func> loader, CancellationToken cancellationToken) + { + var snapshot = _snapshot; + if (snapshot != null) + return snapshot; + + await _loadGate.WaitAsync(cancellationToken).ConfigureAwait(false); + try + { + // Re-check: another caller may have already loaded while we waited. + snapshot = _snapshot; + if (snapshot != null) + return snapshot; + + snapshot = await loader(cancellationToken).ConfigureAwait(false); + _snapshot = snapshot; + return snapshot; + } + finally + { + _loadGate.Release(); + } + } +} diff --git a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Repositories/SiteExternalSystemRepository.cs b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Repositories/SiteExternalSystemRepository.cs index 1fe373f4..e03801fb 100644 --- a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Repositories/SiteExternalSystemRepository.cs +++ b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Repositories/SiteExternalSystemRepository.cs @@ -11,18 +11,45 @@ namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Repositories; /// backed by the local SQLite database via . /// Write operations throw because site-local /// artifacts are managed exclusively through deployment from Central. +/// +/// WP2.6a (arch-review misc): every ExternalSystemDefinition/ExternalSystemMethod +/// read is served from an snapshot instead of +/// a fresh SQLite scan per call — the by-ID lookups in particular used to re-fetch every +/// row (and, for methods, re-parse every system's JSON method list) just to find one +/// match. See for the sharing/invalidation +/// contract; DeploymentManagerActor invalidates it after an artifact deploy +/// applies external-system changes. +/// /// public class SiteExternalSystemRepository : IExternalSystemRepository { private readonly SiteStorageService _storage; + private readonly ExternalSystemDefinitionCache _cache; /// - /// Initializes a new site-side external system repository. + /// Initializes a new site-side external system repository with a private, + /// unshared cache — used by tests and any caller that wants each instance to see + /// its own fresh view with no cross-instance sharing. /// /// Storage service providing database access. public SiteExternalSystemRepository(SiteStorageService storage) + : this(storage, new ExternalSystemDefinitionCache()) + { + } + + /// + /// Initializes a new site-side external system repository backed by a shared + /// — the production DI registration + /// (AddSiteRuntime) injects one singleton cache shared by every scoped + /// repository instance, so repeated resolutions (e.g. one per script execution's + /// DI scope) do not each pay a fresh SQLite scan. + /// + /// Storage service providing database access. + /// The shared (or private) definition cache backing reads. + public SiteExternalSystemRepository(SiteStorageService storage, ExternalSystemDefinitionCache cache) { _storage = storage ?? throw new ArgumentNullException(nameof(storage)); + _cache = cache ?? throw new ArgumentNullException(nameof(cache)); } // ── ExternalSystemDefinition (read) ── @@ -31,55 +58,27 @@ public class SiteExternalSystemRepository : IExternalSystemRepository public async Task> GetAllExternalSystemsAsync( CancellationToken cancellationToken = default) { - // Already open — SiteStorageService.CreateConnection now hands out a - // LocalDb-managed connection. Opening it again would throw. - await using var connection = CreateConnection(); - - await using var command = connection.CreateCommand(); - command.CommandText = @" - SELECT name, endpoint_url, auth_type, auth_configuration, timeout_seconds - FROM external_systems"; - - var results = new List(); - await using var reader = await command.ExecuteReaderAsync(cancellationToken); - while (await reader.ReadAsync(cancellationToken)) - { - results.Add(MapExternalSystem(reader)); - } - - return results; + var snapshot = await _cache.GetOrLoadAsync(LoadSnapshotAsync, cancellationToken); + return snapshot.ByName.Values.ToList(); } /// public async Task GetExternalSystemByIdAsync( int id, CancellationToken cancellationToken = default) { - // The SQLite table is keyed by name, not by integer ID. - // Scan all rows and match on the synthetic ID derived from the name. - var all = await GetAllExternalSystemsAsync(cancellationToken); - return all.FirstOrDefault(e => e.Id == id); + var snapshot = await _cache.GetOrLoadAsync(LoadSnapshotAsync, cancellationToken); + return snapshot.IdToName.TryGetValue(id, out var name) + && snapshot.ByName.TryGetValue(name, out var definition) + ? definition + : null; } /// public async Task GetExternalSystemByNameAsync( string name, CancellationToken cancellationToken = default) { - // Already open — SiteStorageService.CreateConnection now hands out a - // LocalDb-managed connection. Opening it again would throw. - await using var connection = CreateConnection(); - - await using var command = connection.CreateCommand(); - command.CommandText = @" - SELECT name, endpoint_url, auth_type, auth_configuration, timeout_seconds - FROM external_systems - WHERE name = @name"; - command.Parameters.AddWithValue("@name", name); - - await using var reader = await command.ExecuteReaderAsync(cancellationToken); - if (!await reader.ReadAsync(cancellationToken)) - return null; - - return MapExternalSystem(reader); + var snapshot = await _cache.GetOrLoadAsync(LoadSnapshotAsync, cancellationToken); + return snapshot.ByName.TryGetValue(name, out var definition) ? definition : null; } // ── ExternalSystemMethod (read) ── @@ -88,27 +87,13 @@ public class SiteExternalSystemRepository : IExternalSystemRepository public async Task> GetMethodsByExternalSystemIdAsync( int externalSystemId, CancellationToken cancellationToken = default) { - // Find the parent system to get its name, then parse its method_definitions JSON. - var system = await GetExternalSystemByIdAsync(externalSystemId, cancellationToken); - if (system is null) + var snapshot = await _cache.GetOrLoadAsync(LoadSnapshotAsync, cancellationToken); + if (!snapshot.IdToName.TryGetValue(externalSystemId, out var name)) return Array.Empty(); - // Already open — SiteStorageService.CreateConnection now hands out a - // LocalDb-managed connection. Opening it again would throw. - await using var connection = CreateConnection(); - - await using var command = connection.CreateCommand(); - command.CommandText = @" - SELECT method_definitions - FROM external_systems - WHERE name = @name"; - command.Parameters.AddWithValue("@name", system.Name); - - var json = (string?)await command.ExecuteScalarAsync(cancellationToken); - if (string.IsNullOrWhiteSpace(json)) - return Array.Empty(); - - return ParseMethodDefinitions(json, externalSystemId); + return snapshot.MethodsByName.TryGetValue(name, out var methods) + ? methods + : Array.Empty(); } /// @@ -124,17 +109,13 @@ public class SiteExternalSystemRepository : IExternalSystemRepository public async Task GetExternalSystemMethodByIdAsync( int id, CancellationToken cancellationToken = default) { - // Scan all systems and their methods to find the matching synthetic ID. - var systems = await GetAllExternalSystemsAsync(cancellationToken); - foreach (var system in systems) - { - var methods = await GetMethodsByExternalSystemIdAsync(system.Id, cancellationToken); - var match = methods.FirstOrDefault(m => m.Id == id); - if (match is not null) - return match; - } + var snapshot = await _cache.GetOrLoadAsync(LoadSnapshotAsync, cancellationToken); + if (!snapshot.MethodIdIndex.TryGetValue(id, out var key)) + return null; + if (!snapshot.MethodsByName.TryGetValue(key.SystemName, out var methods)) + return null; - return null; + return methods.FirstOrDefault(m => m.Id == id); } // ── DatabaseConnectionDefinition (read) ── @@ -273,13 +254,63 @@ public class SiteExternalSystemRepository : IExternalSystemRepository }; } + /// + /// Loads a full, fresh snapshot of every external system and its parsed method + /// list in ONE query (name, endpoint config, and the method_definitions JSON + /// column together) — the cache-miss path behind every + /// read. Building both the by-name and + /// by-ID indexes here, in a single pass, is what turns the by-ID lookups into O(1) + /// dictionary gets instead of the prior full-table-then-LINQ-scan. + /// + private async Task LoadSnapshotAsync( + CancellationToken cancellationToken) + { + // Already open — SiteStorageService.CreateConnection now hands out a + // LocalDb-managed connection. Opening it again would throw. + await using var connection = CreateConnection(); + + await using var command = connection.CreateCommand(); + command.CommandText = @" + SELECT name, endpoint_url, auth_type, auth_configuration, timeout_seconds, method_definitions + FROM external_systems"; + + var byName = new Dictionary(StringComparer.Ordinal); + var idToName = new Dictionary(); + var methodsByName = new Dictionary>(StringComparer.Ordinal); + var methodIdIndex = new Dictionary(); + + await using var reader = await command.ExecuteReaderAsync(cancellationToken); + while (await reader.ReadAsync(cancellationToken)) + { + var definition = MapExternalSystem(reader); + byName[definition.Name] = definition; + idToName[definition.Id] = definition.Name; + + var methodsJson = reader.IsDBNull(5) ? null : reader.GetString(5); + var methods = string.IsNullOrWhiteSpace(methodsJson) + ? Array.Empty() + : ParseMethodDefinitions(methodsJson, definition.Id); + methodsByName[definition.Name] = methods; + foreach (var method in methods) + methodIdIndex[method.Id] = (definition.Name, method.Name); + } + + return new ExternalSystemDefinitionCache.Snapshot(byName, idToName, methodsByName, methodIdIndex); + } + + // Built ONCE per assembly (Phase 1, arch-review misc): JsonSerializerOptions + // construction is not free and every method-list parse used to mint a new + // instance — now shared across every LoadSnapshotAsync call. + private static readonly JsonSerializerOptions MethodDefinitionsJsonOptions = + new() { PropertyNameCaseInsensitive = true }; + private static IReadOnlyList ParseMethodDefinitions( string json, int externalSystemId) { try { var methods = JsonSerializer.Deserialize>(json, - new JsonSerializerOptions { PropertyNameCaseInsensitive = true }); + MethodDefinitionsJsonOptions); if (methods is null) return Array.Empty(); diff --git a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/ServiceCollectionExtensions.cs b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/ServiceCollectionExtensions.cs index 6b018167..796a5e41 100644 --- a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/ServiceCollectionExtensions.cs +++ b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/ServiceCollectionExtensions.cs @@ -65,8 +65,16 @@ public static class ServiceCollectionExtensions // construction, which is what runs SiteLocalDbSetup.OnReady. services.AddSingleton(); - // Site-local repository implementations backed by SQLite - services.AddScoped(); + // Site-local repository implementations backed by SQLite. + // WP2.6a: ExternalSystemDefinitionCache is a singleton shared by every scoped + // SiteExternalSystemRepository instance (one per script-execution DI scope), so + // resolving the repository repeatedly does not each pay a fresh SQLite + // fetch-all — DeploymentManagerActor invalidates the shared cache after an + // artifact deploy applies external-system changes. + services.AddSingleton(); + services.AddScoped(sp => new SiteExternalSystemRepository( + sp.GetRequiredService(), + sp.GetRequiredService())); // Notify-and-fetch: typed HttpClient for fetching deployment configs from central. services.AddHttpClient() @@ -83,6 +91,13 @@ public static class ServiceCollectionExtensions sp.GetRequiredService>().Value, sp.GetRequiredService>())); + // WP2.6d: periodically lift SiteStreamManager's alarm-publish-queue drop count + // onto the site health report. + services.AddHostedService(sp => new Streaming.SiteStreamAlarmDropReporter( + sp.GetRequiredService(), + sp.GetRequiredService(), + sp.GetRequiredService>())); + return services; } diff --git a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptions.cs b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptions.cs index 42503ca5..ba2506f3 100644 --- a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptions.cs +++ b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptions.cs @@ -77,4 +77,17 @@ public class SiteRuntimeOptions /// bounded script-execution threads is gone. Default: 30000ms. ///
public int StuckScriptGraceMs { get; set; } = 30000; + + /// + /// WP2.6d (arch-review misc — site stream alarm-vs-attribute shared buffer): + /// capacity of the bounded hand-off queue feeding 's + /// dedicated alarm publish path. Alarm state changes are staged through this + /// queue into their OWN broadcast source — separate from the (far higher-volume) + /// attribute publish source — so an attribute storm cannot evict a pending alarm + /// transition. DropOldest overflow; drops are counted (surfaced on the site health + /// report) and logged. Default: 2000 (alarm volume is inherently far lower than + /// attribute volume, so this rarely needs to be anywhere near + /// ). + /// + public int AlarmPublishQueueCapacity { get; set; } = 2000; } diff --git a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptionsValidator.cs b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptionsValidator.cs index f3ed4789..6515c648 100644 --- a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptionsValidator.cs +++ b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptionsValidator.cs @@ -61,5 +61,9 @@ public sealed class SiteRuntimeOptionsValidator : OptionsValidatorBase= 0, $"ScadaBridge:SiteRuntime:StuckScriptGraceMs must be >= 0 " + $"(was {options.StuckScriptGraceMs}); a negative grace throws inside the stuck-script watchdog's delay."); + + builder.RequireThat(options.AlarmPublishQueueCapacity > 0, + $"ScadaBridge:SiteRuntime:AlarmPublishQueueCapacity must be greater than 0 " + + $"(was {options.AlarmPublishQueueCapacity}); it bounds SiteStreamManager's dedicated alarm publish queue."); } } diff --git a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Streaming/SiteStreamAlarmDropReporter.cs b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Streaming/SiteStreamAlarmDropReporter.cs new file mode 100644 index 00000000..f3784c50 --- /dev/null +++ b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Streaming/SiteStreamAlarmDropReporter.cs @@ -0,0 +1,78 @@ +using Microsoft.Extensions.Hosting; +using Microsoft.Extensions.Logging; +using ZB.MOM.WW.ScadaBridge.HealthMonitoring; + +namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Streaming; + +/// +/// WP2.6d (arch-review misc — site stream alarm-vs-attribute shared buffer): site-side +/// hosted service that periodically reads +/// and pushes it into so +/// the next carries a fresh snapshot on +/// the site health report. Mirrors ScriptSchedulerStatsReporter: immediate first +/// probe, fixed cadence, exceptions logged and swallowed so the loop survives every probe +/// failure. +/// +public sealed class SiteStreamAlarmDropReporter : BackgroundService +{ + /// Default poll cadence (10 s) — coarse enough to amortise across health reports. + internal static readonly TimeSpan DefaultPollInterval = TimeSpan.FromSeconds(10); + + private readonly ISiteHealthCollector _collector; + private readonly SiteStreamManager _streamManager; + private readonly ILogger _logger; + private readonly TimeSpan _pollInterval; + + /// Initializes a new instance of . + /// The site health collector that receives the drop count. + /// The site stream manager whose alarm-queue drop count is sampled. + /// Logger instance. + /// Poll interval override; defaults to (10 s). + public SiteStreamAlarmDropReporter( + ISiteHealthCollector collector, + SiteStreamManager streamManager, + ILogger logger, + TimeSpan? pollInterval = null) + { + _collector = collector ?? throw new ArgumentNullException(nameof(collector)); + _streamManager = streamManager ?? throw new ArgumentNullException(nameof(streamManager)); + _logger = logger ?? throw new ArgumentNullException(nameof(logger)); + _pollInterval = pollInterval ?? DefaultPollInterval; + } + + /// + protected override async Task ExecuteAsync(CancellationToken stoppingToken) + { + // Immediate first probe so the first health report after start carries a + // real snapshot instead of a zero. + Probe(); + + while (!stoppingToken.IsCancellationRequested) + { + try + { + await Task.Delay(_pollInterval, stoppingToken).ConfigureAwait(false); + } + catch (OperationCanceledException) + { + break; + } + + Probe(); + } + } + + private void Probe() + { + try + { + _collector.SetSiteStreamAlarmDropCount(_streamManager.AlarmPublishDroppedCount); + } + catch (Exception ex) + { + // Catch-all is deliberate: the hosted service must survive every class + // of probe failure so the next tick gets a chance. + _logger.LogWarning(ex, "SiteStreamAlarmDropReporter probe failed; next tick will retry."); + } + } +} diff --git a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Streaming/SiteStreamManager.cs b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Streaming/SiteStreamManager.cs index a49fedb4..5a91247b 100644 --- a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Streaming/SiteStreamManager.cs +++ b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Streaming/SiteStreamManager.cs @@ -1,3 +1,4 @@ +using System.Threading.Channels; using Akka; using Akka.Actor; using Akka.Streams; @@ -18,6 +19,21 @@ namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Streaming; /// Implements ISiteStreamSubscriber so the gRPC server can subscribe actors /// to instance events without referencing SiteRuntime directly. /// +/// +/// WP2.6d (arch-review misc — site stream alarm-vs-attribute shared buffer): attribute +/// and alarm events used to share ONE upstream Source.ActorRef + BroadcastHub +/// pair, so a burst of attribute changes could fill that shared buffer and evict a +/// pending alarm transition before any subscriber ever saw it. Attribute and alarm +/// events now travel through entirely SEPARATE publish sources/hubs +/// (/ vs. +/// /), so an attribute storm +/// can only ever evict other attribute events. The alarm path additionally stages +/// through a bounded () so drops +/// are counted () — Source.ActorRef's +/// built-in DropHead overflow has no drop callback to instrument. Publishing is also +/// skipped entirely when nobody is subscribed to that source (see +/// /). +/// public class SiteStreamManager : ISiteStreamSubscriber { /// Sentinel instance name recorded for site-wide (non-instance-scoped) subscriptions. @@ -26,11 +42,44 @@ public class SiteStreamManager : ISiteStreamSubscriber private ActorSystem? _system; private IMaterializer? _materializer; private readonly int _bufferSize; + private readonly int _alarmPublishQueueCapacity; private readonly ILogger _logger; private readonly object _lock = new(); - private IActorRef? _sourceActor; - private Source? _hubSource; + // Attribute-only publish source/hub. By far the higher-volume of the two — + // deliberately kept on Source.ActorRef's built-in bounded-mailbox + DropHead + // behavior (unchanged from before WP2.6d); losing an occasional attribute update + // under a storm is the accepted, pre-existing trade-off. + private IActorRef? _attributeSourceActor; + private Source? _attributeHubSource; + + // Alarm-only publish source/hub — isolated from the attribute path above so an + // attribute storm can never evict a pending alarm transition. + private IActorRef? _alarmSourceActor; + private Source? _alarmHubSource; + + // Bounded hand-off queue feeding the alarm source actor. Staging alarm publishes + // through this (rather than Tell-ing _alarmSourceActor directly) is what makes + // AlarmPublishDroppedCount possible — Source.ActorRef's own DropHead overflow has + // no drop callback to observe. + private Channel? _alarmPublishQueue; + private Task? _alarmPump; + private long _alarmPublishDroppedCount; + + /// + /// Cumulative count of alarm state changes dropped because + /// was at capacity (WP2.6d) — i.e. the site-wide + /// alarm hand-off queue itself fell behind, not the downstream per-subscriber + /// buffers. Surfaced on the site health report. + /// + public long AlarmPublishDroppedCount => Interlocked.Read(ref _alarmPublishDroppedCount); + + // Per-source subscriber counts (WP2.6d "skip publish at zero subscribers"). + // A per-instance Subscribe touches BOTH sources (it needs both attribute and alarm + // events for that instance); SubscribeSiteAlarms touches only the alarm source. + private int _attributeSubscriberCount; + private int _alarmSubscriberCount; + private readonly Dictionary _subscriptions = new(); /// Initializes the stream manager with configuration and logger; the Akka stream is not started until is called. @@ -41,66 +90,138 @@ public class SiteStreamManager : ISiteStreamSubscriber ILogger logger) { _bufferSize = options.StreamBufferSize; + _alarmPublishQueueCapacity = options.AlarmPublishQueueCapacity; _logger = logger; } /// - /// Initializes the broadcast stream. Must be called after ActorSystem is ready. - /// The ActorSystem is passed here rather than via the constructor so that - /// SiteStreamManager can be created by DI before the actor system exists. + /// Initializes the broadcast streams (one for attribute events, one for alarm + /// events — see the type-level remarks) and starts the alarm hand-off pump. Must be + /// called after ActorSystem is ready. The ActorSystem is passed here rather than via + /// the constructor so that SiteStreamManager can be created by DI before the actor + /// system exists. /// - /// The running Akka used to materialize the broadcast stream. + /// The running Akka used to materialize the broadcast streams. public void Initialize(ActorSystem system) { _system = system; _materializer = _system.Materializer(); - var (sourceActor, hubSource) = Source.ActorRef( + var (attributeSourceActor, attributeHubSource) = Source.ActorRef( _bufferSize, OverflowStrategy.DropHead) .ToMaterialized( BroadcastHub.Sink(bufferSize: 256), Keep.Both) .Run(_materializer); + _attributeSourceActor = attributeSourceActor; + _attributeHubSource = attributeHubSource; - _sourceActor = sourceActor; - _hubSource = hubSource; + var (alarmSourceActor, alarmHubSource) = Source.ActorRef( + _bufferSize, + OverflowStrategy.DropHead) + .ToMaterialized( + BroadcastHub.Sink(bufferSize: 256), + Keep.Both) + .Run(_materializer); + _alarmSourceActor = alarmSourceActor; + _alarmHubSource = alarmHubSource; + + _alarmPublishQueue = CreateAlarmPublishQueue(_alarmPublishQueueCapacity, () => + { + var total = Interlocked.Increment(ref _alarmPublishDroppedCount); + _logger.LogWarning( + "Alarm publish queue exceeded its bounded capacity ({Capacity}); " + + "oldest pending alarm transition dropped (total dropped: {Dropped})", + _alarmPublishQueueCapacity, total); + }); + + var alarmQueue = _alarmPublishQueue; + var alarmActor = _alarmSourceActor; + _alarmPump = Task.Run(async () => + { + await foreach (var changed in alarmQueue.Reader.ReadAllAsync()) + { + alarmActor.Tell(changed); + } + }); _logger.LogInformation( - "SiteStreamManager initialized with publish buffer size {BufferSize}", _bufferSize); + "SiteStreamManager initialized with publish buffer size {BufferSize} " + + "and alarm publish queue capacity {AlarmQueueCapacity}", + _bufferSize, _alarmPublishQueueCapacity); } /// - /// Publishes an attribute value change to the broadcast hub. - /// Fire-and-forget — never blocks the calling actor. + /// Builds the bounded, single-reader alarm hand-off queue with DropOldest overflow, + /// invoking on every eviction. Extracted as a pure, + /// testable helper (internal — see InternalsVisibleTo) so the drop-counting + /// wiring can be verified deterministically, without racing the async pump/actor + /// system against a burst of publishes. + /// + internal static Channel CreateAlarmPublishQueue(int capacity, Action onDropped) => + Channel.CreateBounded( + new BoundedChannelOptions(Math.Max(1, capacity)) + { + SingleReader = true, + SingleWriter = false, + FullMode = BoundedChannelFullMode.DropOldest, + }, + itemDropped: _ => onDropped()); + + /// + /// Publishes an attribute value change to the attribute broadcast hub. + /// Fire-and-forget — never blocks the calling actor. Skipped entirely when no + /// subscriber is currently interested in attribute events (WP2.6d). /// /// The attribute value change event to publish. public void PublishAttributeValueChanged(AttributeValueChanged changed) { - _sourceActor?.Tell(changed); + if (Volatile.Read(ref _attributeSubscriberCount) == 0) + return; + + _attributeSourceActor?.Tell(changed); } /// - /// Publishes an alarm state change to the broadcast hub. - /// Fire-and-forget — never blocks the calling actor. + /// Publishes an alarm state change to the DEDICATED alarm broadcast hub — isolated + /// from the (far higher-volume) attribute path, so an attribute storm can never + /// evict a pending alarm transition. Fire-and-forget — never blocks the calling + /// actor. Skipped entirely when no subscriber is currently interested in alarm + /// events (WP2.6d). /// /// The alarm state change event to publish. public void PublishAlarmStateChanged(AlarmStateChanged changed) { - _sourceActor?.Tell(changed); + if (Volatile.Read(ref _alarmSubscriberCount) == 0) + return; + + _alarmPublishQueue?.Writer.TryWrite(changed); } /// public string Subscribe(string instanceName, IActorRef subscriber) { - if (_hubSource is null || _materializer is null) + if (_attributeHubSource is null || _alarmHubSource is null || _materializer is null) throw new InvalidOperationException("SiteStreamManager.Initialize must be called before Subscribe"); var subscriptionId = Guid.NewGuid().ToString(); var capturedInstance = instanceName; var capturedSubscriber = subscriber; - var killSwitch = _hubSource + // Two independent graphs — one per source — both forwarding to the same + // subscriber actor. The actor's own mailbox serializes delivery; this instance + // subscription (used by Debug View) does not need cross-source ordering + // guarantees, only that alarm events for it are never lost to an attribute + // storm on a DIFFERENT instance sharing the same (now attribute-only) hub. + var attributeKillSwitch = _attributeHubSource + .Where(ev => ev.InstanceUniqueName == capturedInstance) + .Buffer(_bufferSize, OverflowStrategy.DropHead) + .ViaMaterialized(KillSwitches.Single(), Keep.Right) + .To(Sink.ForEach(ev => capturedSubscriber.Tell(ev))) + .Run(_materializer); + + var alarmKillSwitch = _alarmHubSource .Where(ev => ev.InstanceUniqueName == capturedInstance) .Buffer(_bufferSize, OverflowStrategy.DropHead) .ViaMaterialized(KillSwitches.Single(), Keep.Right) @@ -110,9 +231,15 @@ public class SiteStreamManager : ISiteStreamSubscriber lock (_lock) { _subscriptions[subscriptionId] = new SubscriptionInfo( - instanceName, subscriber, killSwitch, DateTimeOffset.UtcNow); + instanceName, subscriber, + new[] { attributeKillSwitch, alarmKillSwitch }, + TouchesAttributes: true, TouchesAlarms: true, + DateTimeOffset.UtcNow); } + Interlocked.Increment(ref _attributeSubscriberCount); + Interlocked.Increment(ref _alarmSubscriberCount); + _logger.LogDebug( "Subscriber {SubscriptionId} registered for instance {Instance}", subscriptionId, instanceName); @@ -122,10 +249,10 @@ public class SiteStreamManager : ISiteStreamSubscriber /// /// Subscribe to ALARM events for ALL instances on the site (no per-instance - /// filter). Only events are forwarded; - /// events are dropped (attributes are far - /// higher-volume and the aggregated Alarm Summary never shows them). Same - /// broadcast-hub wiring as , and the returned + /// filter). The dedicated alarm hub carries only + /// events (the Where below is a defensive no-op, not a load-bearing filter — + /// see the type-level remarks); the aggregated Alarm Summary never sees attribute + /// events. Same broadcast-hub wiring as , and the returned /// subscription id is torn down via exactly like the /// per-instance variant. /// @@ -133,13 +260,13 @@ public class SiteStreamManager : ISiteStreamSubscriber /// A subscription id to pass to . public string SubscribeSiteAlarms(IActorRef subscriber) { - if (_hubSource is null || _materializer is null) + if (_alarmHubSource is null || _materializer is null) throw new InvalidOperationException("SiteStreamManager.Initialize must be called before SubscribeSiteAlarms"); var subscriptionId = Guid.NewGuid().ToString(); var capturedSubscriber = subscriber; - var killSwitch = _hubSource + var killSwitch = _alarmHubSource .Where(ev => ev is AlarmStateChanged) .Buffer(_bufferSize, OverflowStrategy.DropHead) .ViaMaterialized(KillSwitches.Single(), Keep.Right) @@ -149,9 +276,14 @@ public class SiteStreamManager : ISiteStreamSubscriber lock (_lock) { _subscriptions[subscriptionId] = new SubscriptionInfo( - SiteWideInstanceName, subscriber, killSwitch, DateTimeOffset.UtcNow); + SiteWideInstanceName, subscriber, + new[] { killSwitch }, + TouchesAttributes: false, TouchesAlarms: true, + DateTimeOffset.UtcNow); } + Interlocked.Increment(ref _alarmSubscriberCount); + _logger.LogDebug( "Subscriber {SubscriptionId} registered for site-wide alarm events", subscriptionId); @@ -160,7 +292,7 @@ public class SiteStreamManager : ISiteStreamSubscriber /// /// Unsubscribe from instance events. Shuts down the per-subscriber - /// stream graph via its KillSwitch. + /// stream graph(s) via their KillSwitch(es). /// /// The subscription ID returned by . /// true if the subscription was found and removed; false if it was already gone. @@ -173,7 +305,7 @@ public class SiteStreamManager : ISiteStreamSubscriber return false; } - info.KillSwitch.Shutdown(); + TearDown(info); _logger.LogDebug("Subscriber {SubscriptionId} removed", subscriptionId); return true; } @@ -193,7 +325,7 @@ public class SiteStreamManager : ISiteStreamSubscriber } foreach (var info in toShutdown) - info.KillSwitch.Shutdown(); + TearDown(info); if (toShutdown.Count > 0) { @@ -202,6 +334,18 @@ public class SiteStreamManager : ISiteStreamSubscriber } } + /// Shuts down every kill switch for a subscription and decrements the matching per-source counters. + private void TearDown(SubscriptionInfo info) + { + foreach (var killSwitch in info.KillSwitches) + killSwitch.Shutdown(); + + if (info.TouchesAttributes) + Interlocked.Decrement(ref _attributeSubscriberCount); + if (info.TouchesAlarms) + Interlocked.Decrement(ref _alarmSubscriberCount); + } + /// /// Returns the count of active subscriptions (for diagnostics/testing). /// @@ -213,6 +357,8 @@ public class SiteStreamManager : ISiteStreamSubscriber private record SubscriptionInfo( string InstanceName, IActorRef Subscriber, - IKillSwitch KillSwitch, + IReadOnlyList KillSwitches, + bool TouchesAttributes, + bool TouchesAlarms, DateTimeOffset SubscribedAt); } diff --git a/src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardOptions.cs b/src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardOptions.cs index 5425c10b..f91c8335 100644 --- a/src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardOptions.cs +++ b/src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardOptions.cs @@ -38,4 +38,18 @@ public class StoreAndForwardOptions /// 1 = legacy serial. Within a lane delivery stays sequential (per-target FIFO). /// public int SweepTargetParallelism { get; set; } = 4; + + /// + /// WP2.6c (arch-review misc — unbounded S&F observer queue): capacity of the + /// cached-call audit-observer pump's queue. It was the one unbounded + /// Channel<T> left in the system — a slow/stuck + /// ICachedCallLifecycleObserver (a SQLite audit write) could not stretch + /// the retry sweep, but nothing stopped it growing without bound while the sweep kept + /// posting. Bounded with + /// — the same overflow policy as its sibling bounded channels + /// (SiteEventLogger's write queue, the debug/alarm stream hubs) — so a stuck + /// observer sheds the oldest unprocessed notifications instead of leaking memory; drops + /// are counted (see StoreAndForwardService.ObserverQueueDroppedCount). + /// + public int ObserverQueueCapacity { get; set; } = 10_000; } diff --git a/src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardOptionsValidator.cs b/src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardOptionsValidator.cs index 41bef58b..a2da666f 100644 --- a/src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardOptionsValidator.cs +++ b/src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardOptionsValidator.cs @@ -41,5 +41,9 @@ public sealed class StoreAndForwardOptionsValidator : OptionsValidatorBase= 1, $"ScadaBridge:StoreAndForward:SweepTargetParallelism must be >= 1 " + $"(was {options.SweepTargetParallelism}); it caps concurrent (category,target) sweep lanes — 1 means serial."); + + builder.RequireThat(options.ObserverQueueCapacity >= 1, + $"ScadaBridge:StoreAndForward:ObserverQueueCapacity must be >= 1 " + + $"(was {options.ObserverQueueCapacity}); it bounds the cached-call audit-observer pump's queue."); } } diff --git a/src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardService.cs b/src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardService.cs index 53a6ad2f..e0143d4c 100644 --- a/src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardService.cs +++ b/src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardService.cs @@ -122,9 +122,52 @@ public class StoreAndForwardService /// completes it (a restarted instance needs a fresh /// channel). Before starts the pump, posts fall back /// to inline processing (see ). + /// + /// WP2.6c: bounded (, + /// default 10,000) with — this was + /// the one unbounded channel left in the system; a pump that falls behind (a stuck + /// observer) now sheds the oldest unprocessed notification instead of growing + /// without bound. Drops are counted via + /// and exposed by . + /// /// - private Channel> _observerQueue = - Channel.CreateUnbounded>(new UnboundedChannelOptions { SingleReader = true }); + private Channel> _observerQueue = CreateObserverQueue( + FieldInitializerObserverQueueCapacity, onDropped: null); + + /// + /// Capacity used only for the field initializer, before + /// re-creates the channel sized from + /// (unavailable at field-init + /// time — is assigned in the constructor body). Irrelevant in + /// practice: a post before falls back to inline processing + /// (see ), so nothing is ever queued at this size. + /// + private const int FieldInitializerObserverQueueCapacity = 1; + + /// + /// Cumulative count of cached-call audit-observer notifications dropped because + /// was at capacity (WP2.6c). Not reset across + /// / cycles — a diagnostic total for + /// the lifetime of this service instance. + /// + private long _observerQueueDroppedCount; + + /// Diagnostic counter — see . + public long ObserverQueueDroppedCount => Interlocked.Read(ref _observerQueueDroppedCount); + + /// + /// Builds a bounded, single-reader observer queue with DropOldest overflow, invoking + /// (if supplied) on every eviction. + /// + private static Channel> CreateObserverQueue(int capacity, Action? onDropped) => + Channel.CreateBounded>( + new BoundedChannelOptions(Math.Max(1, capacity)) + { + SingleReader = true, + SingleWriter = false, + FullMode = BoundedChannelFullMode.DropOldest, + }, + itemDropped: _ => onDropped?.Invoke()); /// /// The single-reader pump draining , or @@ -371,8 +414,17 @@ public class StoreAndForwardService // StopAsync completes the channel, so a restarted instance needs a fresh // one. The pump is best-effort: an observer that throws is logged and // swallowed so a failing audit pipeline never corrupts retry bookkeeping. - _observerQueue = Channel.CreateUnbounded>( - new UnboundedChannelOptions { SingleReader = true }); + // WP2.6c: bounded + DropOldest, sized from options; a drop increments + // _observerQueueDroppedCount (surfaced via ObserverQueueDroppedCount) and is + // logged at Warning so a stuck observer is visible, not just silently lossy. + _observerQueue = CreateObserverQueue(_options.ObserverQueueCapacity, onDropped: () => + { + Interlocked.Increment(ref _observerQueueDroppedCount); + _logger.LogWarning( + "Cached-call audit-observer queue exceeded its bounded capacity ({Capacity}); " + + "oldest pending notification dropped (total dropped: {Dropped})", + _options.ObserverQueueCapacity, Interlocked.Read(ref _observerQueueDroppedCount)); + }); _observerPump = Task.Run(async () => { await foreach (var work in _observerQueue.Reader.ReadAllAsync()) diff --git a/tests/ZB.MOM.WW.ScadaBridge.CLI.Tests/ManagementHttpClientTests.cs b/tests/ZB.MOM.WW.ScadaBridge.CLI.Tests/ManagementHttpClientTests.cs index 60013abb..e35da643 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.CLI.Tests/ManagementHttpClientTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.CLI.Tests/ManagementHttpClientTests.cs @@ -104,3 +104,78 @@ public class ManagementHttpClientTests Assert.Equal("TIMEOUT", response.ErrorCode); } } + +/// +/// WP2.6e (arch-review misc — CLI HttpClient timeout): the public +/// constructor must bound its underlying +/// explicitly (30 s default) rather than leaving the +/// framework's 100 s default in place, and must honor the +/// SCADABRIDGE_HTTP_TIMEOUT_SECONDS override — consistent with how every other +/// CLI setting is environment-overridable (). Runs in the shared +/// "Environment" collection (see ) so it never races another +/// test mutating process-wide environment variables. +/// +[Collection("Environment")] +public class ManagementHttpClientTimeoutTests +{ + private const string EnvVar = "SCADABRIDGE_HTTP_TIMEOUT_SECONDS"; + + [Fact] + public void DefaultConstructor_SetsThirtySecondTimeout_WhenEnvVarUnset() + { + var original = Environment.GetEnvironmentVariable(EnvVar); + try + { + Environment.SetEnvironmentVariable(EnvVar, null); + + using var client = new ManagementHttpClient("http://localhost:9001", "user", "pass"); + + Assert.Equal(TimeSpan.FromSeconds(30), ManagementHttpClient.DefaultTimeout); + Assert.Equal(TimeSpan.FromSeconds(30), client.EffectiveTimeout); + } + finally + { + Environment.SetEnvironmentVariable(EnvVar, original); + } + } + + [Theory] + [InlineData("0")] + [InlineData("-5")] + [InlineData("not-a-number")] + [InlineData("")] + public void InvalidOrNonPositiveEnvValue_FallsBackToDefault(string value) + { + var original = Environment.GetEnvironmentVariable(EnvVar); + try + { + Environment.SetEnvironmentVariable(EnvVar, value); + + using var client = new ManagementHttpClient("http://localhost:9001", "user", "pass"); + + Assert.Equal(TimeSpan.FromSeconds(30), client.EffectiveTimeout); + } + finally + { + Environment.SetEnvironmentVariable(EnvVar, original); + } + } + + [Fact] + public void PositiveEnvValue_OverridesDefaultTimeout() + { + var original = Environment.GetEnvironmentVariable(EnvVar); + try + { + Environment.SetEnvironmentVariable(EnvVar, "5"); + + using var client = new ManagementHttpClient("http://localhost:9001", "user", "pass"); + + Assert.Equal(TimeSpan.FromSeconds(5), client.EffectiveTimeout); + } + finally + { + Environment.SetEnvironmentVariable(EnvVar, original); + } + } +} diff --git a/tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests/ApiMethodCacheTests.cs b/tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests/ApiMethodCacheTests.cs new file mode 100644 index 00000000..99a4e262 --- /dev/null +++ b/tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests/ApiMethodCacheTests.cs @@ -0,0 +1,110 @@ +using ZB.MOM.WW.ScadaBridge.Commons.Entities.InboundApi; + +namespace ZB.MOM.WW.ScadaBridge.InboundAPI.Tests; + +/// +/// WP2.6b (arch-review misc — Inbound API per-request SQL): +/// hit/miss/expiry/invalidation behavior, independent of the endpoint and subscriber wiring +/// (covered separately by ). +/// +public class ApiMethodCacheTests +{ + private static ApiMethod Method(string name) => new(name, "return 1;") { Id = 1 }; + + [Fact] + public async Task GetOrFetchAsync_SecondCallWithinTtl_ServesFromCache_NoRefetch() + { + var cache = new ApiMethodCache(TimeSpan.FromMinutes(5)); + var fetchCount = 0; + Task Fetch(CancellationToken _) + { + fetchCount++; + return Task.FromResult(Method("m")); + } + + var first = await cache.GetOrFetchAsync("m", Fetch, CancellationToken.None); + var second = await cache.GetOrFetchAsync("m", Fetch, CancellationToken.None); + + Assert.Equal(1, fetchCount); + Assert.Same(first, second); + } + + [Fact] + public async Task GetOrFetchAsync_AfterTtlExpiry_Refetches() + { + var time = new FakeTimeProvider(DateTimeOffset.UtcNow); + var cache = new ApiMethodCache(TimeSpan.FromSeconds(1), time); + var fetchCount = 0; + Task Fetch(CancellationToken _) + { + fetchCount++; + return Task.FromResult(Method("m")); + } + + await cache.GetOrFetchAsync("m", Fetch, CancellationToken.None); + time.Advance(TimeSpan.FromSeconds(2)); + await cache.GetOrFetchAsync("m", Fetch, CancellationToken.None); + + Assert.Equal(2, fetchCount); + } + + [Fact] + public async Task GetOrFetchAsync_NegativeResult_IsCachedUntilTtlExpires() + { + var cache = new ApiMethodCache(TimeSpan.FromMinutes(5)); + var fetchCount = 0; + Task Fetch(CancellationToken _) + { + fetchCount++; + return Task.FromResult(null); + } + + var first = await cache.GetOrFetchAsync("missing", Fetch, CancellationToken.None); + var second = await cache.GetOrFetchAsync("missing", Fetch, CancellationToken.None); + + Assert.Null(first); + Assert.Null(second); + Assert.Equal(1, fetchCount); + } + + [Fact] + public async Task Invalidate_DropsEntry_NextCallRefetches() + { + var cache = new ApiMethodCache(TimeSpan.FromMinutes(5)); + var fetchCount = 0; + Task Fetch(CancellationToken _) + { + fetchCount++; + return Task.FromResult(Method("m")); + } + + await cache.GetOrFetchAsync("m", Fetch, CancellationToken.None); + cache.Invalidate("m"); + await cache.GetOrFetchAsync("m", Fetch, CancellationToken.None); + + Assert.Equal(2, fetchCount); + } + + [Fact] + public void Invalidate_UnknownName_DoesNotThrow() + { + var cache = new ApiMethodCache(TimeSpan.FromMinutes(5)); + cache.Invalidate("never-cached"); + } + + [Fact] + public void Constructor_NonPositiveTtl_Throws() + { + Assert.Throws(() => new ApiMethodCache(TimeSpan.Zero)); + Assert.Throws(() => new ApiMethodCache(TimeSpan.FromSeconds(-1))); + } + + /// Minimal controllable for TTL-expiry tests. + private sealed class FakeTimeProvider : TimeProvider + { + private DateTimeOffset _now; + public FakeTimeProvider(DateTimeOffset start) => _now = start; + public void Advance(TimeSpan by) => _now += by; + public override DateTimeOffset GetUtcNow() => _now; + } +} diff --git a/tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests/InboundApiOptionsValidatorTests.cs b/tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests/InboundApiOptionsValidatorTests.cs index 65af18bc..a5c9f381 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests/InboundApiOptionsValidatorTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests/InboundApiOptionsValidatorTests.cs @@ -65,4 +65,22 @@ public class InboundApiOptionsValidatorTests Assert.True(result.Failed); Assert.Contains("MaxRequestBodyBytes", result.FailureMessage); } + + [Fact] + public void ZeroApiMethodCacheTtl_IsRejected() + { + var result = Validate(new InboundApiOptions { ApiMethodCacheTtl = TimeSpan.Zero }); + + Assert.True(result.Failed); + Assert.Contains("ApiMethodCacheTtl", result.FailureMessage); + } + + [Fact] + public void NegativeApiMethodCacheTtl_IsRejected() + { + var result = Validate(new InboundApiOptions { ApiMethodCacheTtl = TimeSpan.FromSeconds(-1) }); + + Assert.True(result.Failed); + Assert.Contains("ApiMethodCacheTtl", result.FailureMessage); + } } diff --git a/tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests/ScriptArtifactChangeSubscriberTests.cs b/tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests/ScriptArtifactChangeSubscriberTests.cs index 37b3076f..d03d7f52 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests/ScriptArtifactChangeSubscriberTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests/ScriptArtifactChangeSubscriberTests.cs @@ -41,12 +41,13 @@ public class ScriptArtifactChangeSubscriberTests private readonly InboundScriptExecutor _executor = new( NullLogger.Instance, Substitute.For()); + private readonly ApiMethodCache _methodCache = new(TimeSpan.FromMinutes(5)); private readonly RecordingBus _bus = new(); private readonly RouteHelper _route = new( Substitute.For(), Substitute.For()); private ScriptArtifactChangeSubscriber CreateSubscriber(IScriptArtifactChangeBus? bus) => - new(_executor, NullLogger.Instance, bus); + new(_executor, _methodCache, NullLogger.Instance, bus); private Task Run(ApiMethod m) => _executor.ExecuteAsync( m, new Dictionary(), _route, TimeSpan.FromSeconds(10)); @@ -121,6 +122,34 @@ public class ScriptArtifactChangeSubscriberTests Assert.Equal(0, _bus.SubscriberCount); } + /// + /// WP2.6b: an ApiMethod change notification must also drop the resolved-row cache + /// entry (not just the compiled handler) — proving EndpointExtensions' next lookup + /// re-fetches instead of serving a stale cached row for the rest of the TTL window. + /// + [Fact] + public async Task ApiMethodPublish_InvalidatesResolvedMethodCache() + { + var subscriber = CreateSubscriber(_bus); + await subscriber.StartAsync(CancellationToken.None); + + var fetchCount = 0; + Task Fetch(CancellationToken _) + { + fetchCount++; + return Task.FromResult(new ApiMethod("m", "return 1;") { Id = 1 }); + } + + await _methodCache.GetOrFetchAsync("m", Fetch, CancellationToken.None); + await _methodCache.GetOrFetchAsync("m", Fetch, CancellationToken.None); + Assert.Equal(1, fetchCount); // second call served from cache + + _bus.Publish(ApiMethodChanged("m")); + + await _methodCache.GetOrFetchAsync("m", Fetch, CancellationToken.None); + Assert.Equal(2, fetchCount); // cache entry was dropped — re-fetched + } + [Fact] public async Task NullBus_NoOps() { diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Repositories/SiteRepositoryTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Repositories/SiteRepositoryTests.cs index 366d5eb8..201bf176 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Repositories/SiteRepositoryTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Repositories/SiteRepositoryTests.cs @@ -215,4 +215,99 @@ public class SiteRepositoryTests : IDisposable // (arch-review 08 §1.3/#23) along with the vestigial SiteNotificationRepository — // notification config is central-only and never lives on a site. The synthetic-ID // stability guarantee is still exercised by ExternalSystemRepository_SyntheticId_IsStableAcrossRestart. + + // ── WP2.6a: ExternalSystemDefinitionCache (arch-review misc — site external-system resolution) ── + + /// + /// WP2.6a: two repository instances sharing one + /// (the production DI shape — one singleton cache, many scoped repository instances) must + /// see the SAME cached snapshot: a row written after the cache was already populated by + /// the first repository is invisible to the second until the cache is invalidated. + /// + [Fact] + public async Task ExternalSystemRepository_SharedCache_HitServesStaleSnapshotUntilInvalidated() + { + var storage = NewStorage(); + await storage.InitializeAsync(); + await storage.StoreExternalSystemAsync("Alpha", "https://alpha.test", "None", null, null); + + var cache = new ExternalSystemDefinitionCache(); + var repo1 = new SiteExternalSystemRepository(storage, cache); + var repo2 = new SiteExternalSystemRepository(storage, cache); + + // repo1 populates the shared cache. + var initial = await repo1.GetAllExternalSystemsAsync(); + Assert.Single(initial); + + // A new system is written directly to storage — repo2, sharing the same + // cache, must NOT see it yet (cache hit serves the stale snapshot). + await storage.StoreExternalSystemAsync("Beta", "https://beta.test", "None", null, null); + var stillStale = await repo2.GetAllExternalSystemsAsync(); + Assert.Single(stillStale); + + // After invalidation, the next read reloads from storage and sees both rows. + cache.InvalidateAll(); + var fresh = await repo2.GetAllExternalSystemsAsync(); + Assert.Equal(2, fresh.Count); + } + + /// + /// WP2.6a: the by-ID lookups (system and method) must resolve correctly out of the + /// cached snapshot — proving the O(1) id-index build in LoadSnapshotAsync is + /// wired correctly for both entity kinds, not just the by-name path already covered + /// by ExternalSystemGateway-011. + /// + [Fact] + public async Task ExternalSystemRepository_SharedCache_ByIdLookups_ResolveSystemAndMethod() + { + var storage = NewStorage(); + await storage.InitializeAsync(); + var methodDefs = "[{\"Name\":\"getData\",\"HttpMethod\":\"GET\",\"Path\":\"/data\"}]"; + await storage.StoreExternalSystemAsync( + "WeatherApi", "https://api.example.com", "ApiKey", null, methodDefs); + + var cache = new ExternalSystemDefinitionCache(); + var repo = new SiteExternalSystemRepository(storage, cache); + + var system = await repo.GetExternalSystemByNameAsync("WeatherApi"); + Assert.NotNull(system); + + var byId = await repo.GetExternalSystemByIdAsync(system!.Id); + Assert.NotNull(byId); + Assert.Equal("WeatherApi", byId!.Name); + + var methods = await repo.GetMethodsByExternalSystemIdAsync(system.Id); + Assert.Single(methods); + + var methodById = await repo.GetExternalSystemMethodByIdAsync(methods[0].Id); + Assert.NotNull(methodById); + Assert.Equal("getData", methodById!.Name); + + Assert.Null(await repo.GetExternalSystemByIdAsync(-1)); + Assert.Null(await repo.GetExternalSystemMethodByIdAsync(-1)); + } + + /// + /// WP2.6a: a repository constructed via the single-arg (no shared cache) constructor + /// gets its own private cache — a second such repository over the same storage must + /// see rows written after the first repository's cache was already populated, + /// preserving the pre-cache "always fresh" behavior for callers that opt out of + /// sharing. + /// + [Fact] + public async Task ExternalSystemRepository_PrivateCache_DoesNotShareAcrossInstances() + { + var storage = NewStorage(); + await storage.InitializeAsync(); + await storage.StoreExternalSystemAsync("Alpha", "https://alpha.test", "None", null, null); + + var repo1 = new SiteExternalSystemRepository(storage); + Assert.Single(await repo1.GetAllExternalSystemsAsync()); + + await storage.StoreExternalSystemAsync("Beta", "https://beta.test", "None", null, null); + + // A brand-new repository instance (its own private cache, unpopulated) sees both rows. + var repo2 = new SiteExternalSystemRepository(storage); + Assert.Equal(2, (await repo2.GetAllExternalSystemsAsync()).Count); + } } diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/SiteRuntimeOptionsValidatorTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/SiteRuntimeOptionsValidatorTests.cs index 1d627316..13a6b419 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/SiteRuntimeOptionsValidatorTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/SiteRuntimeOptionsValidatorTests.cs @@ -75,4 +75,13 @@ public class SiteRuntimeOptionsValidatorTests Assert.True(result.Failed); Assert.Contains("StartupBatchDelayMs", result.FailureMessage); } + + [Fact] + public void ZeroAlarmPublishQueueCapacity_IsRejected() + { + var result = Validate(new SiteRuntimeOptions { AlarmPublishQueueCapacity = 0 }); + + Assert.True(result.Failed); + Assert.Contains("AlarmPublishQueueCapacity", result.FailureMessage); + } } diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Streaming/SiteStreamAlarmDropReporterTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Streaming/SiteStreamAlarmDropReporterTests.cs new file mode 100644 index 00000000..acdad930 --- /dev/null +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Streaming/SiteStreamAlarmDropReporterTests.cs @@ -0,0 +1,57 @@ +using Akka.TestKit.Xunit2; +using Microsoft.Extensions.Logging.Abstractions; +using ZB.MOM.WW.ScadaBridge.HealthMonitoring; +using ZB.MOM.WW.ScadaBridge.SiteRuntime.Streaming; + +namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Streaming; + +/// +/// WP2.6d: the hosted reporter must lift +/// onto the site health report via . Uses the real +/// collector (mirrors ScriptSchedulerStatsReporterTests — NSubstitute is not +/// referenced by this test project). The queue's own drop-counting mechanism is verified +/// deterministically and separately by +/// ; +/// this test proves the reporter's poll-and-push wiring runs correctly. +/// +public class SiteStreamAlarmDropReporterTests : TestKit, IDisposable +{ + void IDisposable.Dispose() => Shutdown(); + + [Fact] + public async Task Reporter_PushesAlarmDropCountToCollector() + { + var options = new SiteRuntimeOptions { StreamBufferSize = 100 }; + var collector = new SiteHealthCollector(); + var streamManager = new SiteStreamManager(options, NullLogger.Instance); + streamManager.Initialize(Sys); + + using var reporter = new SiteStreamAlarmDropReporter( + collector, streamManager, NullLogger.Instance, + pollInterval: TimeSpan.FromMilliseconds(50)); + + await reporter.StartAsync(CancellationToken.None); + try + { + // The immediate first probe (no drops yet) must reach the report as 0 — + // proving the reporter actually ran and pushed a value, not that the field + // simply defaulted. + await WaitUntilAsync(() => + collector.CollectReport("site-1").SiteStreamAlarmDropCount == streamManager.AlarmPublishDroppedCount); + + var report = collector.CollectReport("site-1"); + Assert.Equal(streamManager.AlarmPublishDroppedCount, report.SiteStreamAlarmDropCount); + } + finally + { + await reporter.StopAsync(CancellationToken.None); + } + } + + private static async Task WaitUntilAsync(Func condition) + { + for (var i = 0; i < 100 && !condition(); i++) + await Task.Delay(50); + Assert.True(condition(), "condition not met within timeout"); + } +} diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Streaming/SiteStreamManagerTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Streaming/SiteStreamManagerTests.cs index e4dbfbe7..f187e6a0 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Streaming/SiteStreamManagerTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Streaming/SiteStreamManagerTests.cs @@ -159,4 +159,86 @@ public class SiteStreamManagerTests : TestKit, IDisposable _streamManager.RemoveSubscriber(probe.Ref); Assert.Equal(0, _streamManager.SubscriptionCount); } + + // ── WP2.6d: separate alarm publish path, drop counter, skip-at-zero-subscribers ── + + /// + /// WP2.6d: alarm state changes now travel a dedicated publish source, isolated from + /// the (far higher-volume) attribute path — a burst of attribute events for OTHER + /// instances, interleaved with an alarm-only subscriber's events, must not cause any + /// alarm to be lost. + /// + [Fact] + public void PublishAlarmStateChanged_SurvivesConcurrentAttributeStorm_ForUnrelatedInstances() + { + var alarmProbe = CreateTestProbe(); + _streamManager.SubscribeSiteAlarms(alarmProbe.Ref); + + // A storm of attribute events for a DIFFERENT instance — none of which the + // alarm subscriber is even listening to — interleaved with alarm events. Before + // WP2.6d these shared one upstream buffer; now they are fully separate sources. + for (var i = 0; i < 500; i++) + { + _streamManager.PublishAttributeValueChanged(new AttributeValueChanged( + "NoisyPump", "Temperature", "Temperature", i.ToString(), "Good", DateTimeOffset.UtcNow)); + } + + _streamManager.PublishAlarmStateChanged(new AlarmStateChanged( + "Pump1", "HighTemp", AlarmState.Active, 1, DateTimeOffset.UtcNow)); + + var received = alarmProbe.ExpectMsg(TimeSpan.FromSeconds(3)); + Assert.Equal("Pump1", received.InstanceUniqueName); + } + + /// + /// WP2.6d: with zero subscribers of any kind, PublishAlarmStateChanged/ + /// PublishAttributeValueChanged must be no-ops — no exception, and (for alarms) the + /// event never reaches the bounded hand-off queue, so + /// stays at zero rather than + /// counting events nobody could ever have received anyway. + /// + [Fact] + public void Publish_WithNoSubscribers_IsNoOp_AndDoesNotCountAsDropped() + { + _streamManager.PublishAlarmStateChanged(new AlarmStateChanged( + "Pump1", "HighTemp", AlarmState.Active, 1, DateTimeOffset.UtcNow)); + _streamManager.PublishAttributeValueChanged(new AttributeValueChanged( + "Pump1", "Temperature", "Temperature", "1", "Good", DateTimeOffset.UtcNow)); + + Assert.Equal(0, _streamManager.AlarmPublishDroppedCount); + + // Publishing resumes working normally once a subscriber exists. + var probe = CreateTestProbe(); + _streamManager.SubscribeSiteAlarms(probe.Ref); + _streamManager.PublishAlarmStateChanged(new AlarmStateChanged( + "Pump1", "HighTemp", AlarmState.Active, 2, DateTimeOffset.UtcNow)); + + probe.ExpectMsg(TimeSpan.FromSeconds(3)); + } + + /// + /// WP2.6d: once the bounded alarm hand-off queue (the exact factory + /// wires into the drop-count callback) is + /// full, a further write evicts the oldest queued item and fires the drop callback. + /// Exercised directly against the queue factory — deterministic, no actor + /// system/async pump race involved (see the factory's own doc comment). + /// + [Fact] + public void CreateAlarmPublishQueue_OverCapacity_DropsOldestAndInvokesCallback() + { + var dropped = 0; + var queue = SiteStreamManager.CreateAlarmPublishQueue(capacity: 2, () => Interlocked.Increment(ref dropped)); + + // Nothing reads from this queue, so all five writes stay purely upstream — + // capacity 2 means the 3rd/4th/5th writes must each evict the oldest. + for (var i = 0; i < 5; i++) + { + var wrote = queue.Writer.TryWrite(new AlarmStateChanged( + "Pump1", "HighTemp", AlarmState.Active, i, DateTimeOffset.UtcNow)); + Assert.True(wrote); // DropOldest TryWrite always succeeds once capacity > 0 + } + + Assert.Equal(3, dropped); + Assert.Equal(2, queue.Reader.Count); + } } diff --git a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardOptionsValidatorTests.cs b/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardOptionsValidatorTests.cs index 2a7da0e5..bec17fcb 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardOptionsValidatorTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardOptionsValidatorTests.cs @@ -83,4 +83,16 @@ public class StoreAndForwardOptionsValidatorTests Assert.True(result.Failed); Assert.Contains("SweepTargetParallelism", result.FailureMessage); } + + // ── WP2.6c: bounded observer queue capacity ── + + [Theory] + [InlineData(0)] + [InlineData(-1)] + public void Validate_NonPositiveObserverQueueCapacity_Fails(int capacity) + { + var result = Validate(new StoreAndForwardOptions { ObserverQueueCapacity = capacity }); + Assert.True(result.Failed); + Assert.Contains("ObserverQueueCapacity", result.FailureMessage); + } } diff --git a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardServiceTests.cs b/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardServiceTests.cs index 434b3d60..82f6c724 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardServiceTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardServiceTests.cs @@ -1,4 +1,6 @@ using Microsoft.Extensions.Logging.Abstractions; +using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services; +using ZB.MOM.WW.ScadaBridge.Commons.Types; using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; using ZB.MOM.WW.ScadaBridge.TestSupport; @@ -885,4 +887,82 @@ public class StoreAndForwardServiceTests : IAsyncLifetime, IDisposable release.TrySetResult(); await stop.WaitAsync(TimeSpan.FromSeconds(5)); // drains the real sweep promptly once released } + + // ── WP2.6c: bounded, DropOldest observer queue + drop counter ── + + private sealed class BlockingObserver : ICachedCallLifecycleObserver + { + private readonly TaskCompletionSource _gate; + public BlockingObserver(TaskCompletionSource gate) => _gate = gate; + public async Task OnAttemptCompletedAsync(CachedCallAttemptContext context, CancellationToken ct = default) + => await _gate.Task; + } + + /// + /// WP2.6c: the cached-call audit-observer queue is bounded — once the single-reader + /// pump is stuck awaiting a slow/stuck observer, further posted notifications must + /// evict the oldest queued one (DropOldest) instead of growing without bound, and + /// every eviction must increment . + /// + [Fact] + public async Task ObserverQueue_BoundedCapacity_DropsOldestAndCountsDrops() + { + var gate = new TaskCompletionSource(); + var observer = new BlockingObserver(gate); + + var localDb = TestLocalDb.CreateTemp("ObsQueueBound"); + var storage = new StoreAndForwardStorage(localDb.Db, NullLogger.Instance); + await storage.InitializeAsync(); + var service = new StoreAndForwardService( + storage, + new StoreAndForwardOptions + { + DefaultRetryInterval = TimeSpan.Zero, + DefaultMaxRetries = 5, + RetryTimerInterval = TimeSpan.FromHours(1), // timer never fires in-test + ObserverQueueCapacity = 2, + }, + NullLogger.Instance, + cachedCallObserver: observer, + siteId: "site-77"); + + await service.StartAsync(); + try + { + service.RegisterDeliveryHandler(StoreAndForwardCategory.ExternalSystem, + _ => throw new HttpRequestException("transient")); + + // Enqueue more messages than the bounded capacity (2) — the pump dequeues + // the first notification and blocks on the observer gate, so every + // subsequent notification posted during this sweep queues (and, past + // capacity, evicts the oldest still-queued one) rather than being + // processed. + for (var i = 0; i < 6; i++) + { + await service.EnqueueAsync( + StoreAndForwardCategory.ExternalSystem, $"t{i}", "{}", + attemptImmediateDelivery: false, retryInterval: TimeSpan.Zero, + messageId: TrackedOperationId.New().ToString()); + } + + await service.RetryPendingMessagesAsync(); + + // Give the bounded channel a moment to have absorbed/evicted every post + // (the pump itself stays blocked on the gate throughout). + var deadline = DateTime.UtcNow + TimeSpan.FromSeconds(5); + while (DateTime.UtcNow < deadline && service.ObserverQueueDroppedCount == 0) + await Task.Delay(10); + + Assert.True(service.ObserverQueueDroppedCount > 0, + "expected at least one notification to be dropped once the bounded queue filled"); + } + finally + { + gate.TrySetResult(); + await service.StopAsync(); + var path = localDb.Path; + localDb.Dispose(); + TestLocalDb.DeleteFiles(path); + } + } } From 5db2a810c095067038e048366c964030d043e901 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Fri, 14 Aug 2026 21:07:12 -0400 Subject: [PATCH 12/31] perf(central): set-based ingest, aligned partition purge, KPI query shapes, EF hygiene --- docs/components/ConfigurationDatabase.md | 16 +- .../sql/AlignAuditLogEventIdUniqueness.sql | 24 + docs/requirements/Component-AuditLog.md | 36 +- .../Component-ConfigurationDatabase.md | 6 +- docs/requirements/Component-SiteCallAudit.md | 22 + .../Central/AuditLogIngestActor.cs | 230 +- .../Central/AuditLogPurgeActor.cs | 14 +- .../Central/AuditLogPurgeOptions.cs | 19 +- .../Repositories/IAuditLogRepository.cs | 96 +- .../INotificationOutboxRepository.cs | 21 +- .../AuditLogEntityTypeConfiguration.cs | 17 +- ...AlignAuditLogEventIdUniqueness.Designer.cs | 2090 +++++++++++++++++ ...15004957_AlignAuditLogEventIdUniqueness.cs | 93 + .../ScadaBridgeDbContextModelSnapshot.cs | 4 - .../Repositories/AuditLogRepository.cs | 269 ++- .../Repositories/KpiHistoryRepository.cs | 48 +- .../NotificationOutboxRepository.cs | 81 +- .../Repositories/SiteCallAuditRepository.cs | 113 +- .../ServiceCollectionExtensions.cs | 16 + .../SiteCallAuditActor.cs | 118 +- .../Central/AuditLogIngestActorTests.cs | 99 + .../Integration/PartitionPurgeTests.cs | 87 +- .../AuditLogEntityTypeConfigurationTests.cs | 34 +- .../AddAuditLogTableMigrationTests.cs | 30 +- ...ationOutboxRepositoryKpiQueryShapeTests.cs | 10 + .../Repositories/AuditLogRepositoryTests.cs | 216 +- .../Repositories/KpiHistoryRepositoryTests.cs | 87 + .../SiteCallAuditRepositoryTests.cs | 41 + .../SiteCallAuditReconciliationTests.cs | 119 + 29 files changed, 3790 insertions(+), 266 deletions(-) create mode 100644 docs/plans/sql/AlignAuditLogEventIdUniqueness.sql create mode 100644 src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/20260815004957_AlignAuditLogEventIdUniqueness.Designer.cs create mode 100644 src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/20260815004957_AlignAuditLogEventIdUniqueness.cs diff --git a/docs/components/ConfigurationDatabase.md b/docs/components/ConfigurationDatabase.md index f98a9661..5ca6a6c7 100644 --- a/docs/components/ConfigurationDatabase.md +++ b/docs/components/ConfigurationDatabase.md @@ -78,13 +78,15 @@ builder.Property(e => e.ExecutionId) "CAST(JSON_VALUE(DetailsJson,'$.executionId') AS uniqueidentifier)", stored: true) .ValueGeneratedOnAddOrUpdate(); -// Composite PK includes OccurredAtUtc for partition alignment +// Composite PK includes OccurredAtUtc for partition alignment — and is the ONLY +// EventId uniqueness enforcement (WP2.2 / AlignAuditLogEventIdUniqueness). EventId +// is a GUID minted once at the site alongside OccurredAtUtc, so a given EventId +// always lands in one partition and pair-uniqueness is EventId-uniqueness. builder.HasKey(e => new { e.EventId, e.OccurredAtUtc }); - -builder.HasIndex(e => e.EventId).IsUnique() - .HasDatabaseName("UX_AuditLog_EventId"); ``` +The predecessor `UX_AuditLog_EventId` — a single-column unique index on `[PRIMARY]`, deliberately NOT partition-aligned — was dropped: a non-aligned index blocks `ALTER TABLE … SWITCH PARTITION`, so the retention purge had to drop it, switch, and rebuild it offline inside the switch transaction on every run. + **`TemplateConfiguration`** (representative of the domain-area configs) sets up the self-referencing parent FK, folder FK, cascade-delete relationships to attributes/alarms/scripts/compositions/native alarm sources, and the filtered unique index that enforces name uniqueness only on non-derived (base) templates. **`SiteCallEntityTypeConfiguration`** maps `SiteCall` to `dbo.SiteCalls` with a `TrackedOperationId` PK stored as `varchar(36)` (GUID in `"D"` format) so the column shape matches the wire format and the site SQLite store — one consistent format for operational debugging. @@ -108,11 +110,13 @@ VALUES `FormattableString` interpolation parameterises every value so there is no injection surface. SQL error numbers `2601` and `2627` (unique-index violation) are swallowed as no-ops because the IF NOT EXISTS check has a race window; both the check-loser and the retrying telemetry path are semantically correct duplicates. +`InsertManyIfNotExistsAsync` is the set-based form the ingest hot path uses: one `INSERT … SELECT … FROM (VALUES …) WHERE NOT EXISTS` per chunk of 100 rows (ten bound parameters per row against SQL Server's 2,100-parameter ceiling), so a telemetry packet costs one round trip instead of one per event. Because the anti-semi-join only sees committed rows, the packet is de-duplicated by `EventId` in C# first (first-write-wins, matching the single-row contract); a duplicate-key fault from a concurrent writer falls back to the per-row path so the batch is never a correctness dependency. It uses raw ADO.NET with explicitly typed `SqlParameter`s — the `VALUES` constructor derives its column types from the first row's parameters, so an untyped null would give the derived column the wrong type — and enlists in the DbContext's ambient transaction when one is open (the cached-telemetry dual-write). + `QueryAsync` builds LINQ predicates over `AuditLogRow` using `AsNoTracking()`, translating filter dimensions (`Channels`, `Kinds`, `Statuses`, `SourceSiteIds`, `SourceNodes`, `ExecutionId`, `ParentExecutionId`, time range) to server-side SQL IN/equality predicates and using keyset pagination on `(OccurredAtUtc DESC, EventId DESC)`. `GetExecutionTreeAsync` walks the `ParentExecutionId` graph in two phases: a loop climbs to the root (bounded at 32 levels), then a recursive CTE descends the full tree and LEFT JOINs back to `AuditLog` so stub nodes (purged or row-less executions) still appear with `RowCount = 0`. -`SwitchOutPartitionAsync` executes a drop-and-rebuild dance — dropping `UX_AuditLog_EventId`, creating a byte-identical staging table (including the computed-column definitions), switching the target partition to staging, dropping staging, and rebuilding the unique index — all inside a single `BEGIN TRY / BEGIN CATCH` block that guarantees the index is present whether the switch succeeds or rolls back. +`SwitchOutPartitionAsync` creates a byte-identical staging table (including the computed-column definitions), switches the target partition to staging, and drops staging, inside a single `BEGIN TRY / BEGIN CATCH` block whose CATCH cleans up the staging table on any failure. No index is dropped or rebuilt: uniqueness rides the partition-aligned clustered PK, so the switch is metadata-only. A guarded defensive `DROP INDEX UX_AuditLog_EventId` remains at the head of the batch purely so a database restored from a pre-alignment backup still purges. ### IAuditService — config-change audit @@ -241,7 +245,7 @@ The host is running in production mode and `GetPendingMigrationsAsync` found una ### AuditLog partition switch fails mid-operation -`SwitchOutPartitionAsync` wraps the drop-and-rebuild dance in `BEGIN TRY / BEGIN CATCH`. On failure the CATCH block drops the staging table if it exists and rebuilds `UX_AuditLog_EventId` if it was dropped before the failure. The original exception is re-thrown so the Audit Log purge actor logs it and retries on the next daily tick. Verify that the `scadabridge_audit_purger` role still holds `ALTER ON SCHEMA::dbo` if the operation fails with a permissions error. +`SwitchOutPartitionAsync` wraps the staging/switch batch in `BEGIN TRY / BEGIN CATCH`. On failure the CATCH block drops the staging table if it exists, so no orphaned `AuditLog_Staging_*` object is left behind. There is no index to repair — uniqueness lives on the clustered PK, which the switch never touches. The original exception is re-thrown so the Audit Log purge actor logs it and retries on the next daily tick. Verify that the `scadabridge_audit_purger` role still holds `ALTER ON SCHEMA::dbo` if the operation fails with a permissions error. ### Design-time `dotnet ef` tooling cannot find a connection string diff --git a/docs/plans/sql/AlignAuditLogEventIdUniqueness.sql b/docs/plans/sql/AlignAuditLogEventIdUniqueness.sql new file mode 100644 index 00000000..90b3003f --- /dev/null +++ b/docs/plans/sql/AlignAuditLogEventIdUniqueness.sql @@ -0,0 +1,24 @@ +BEGIN TRANSACTION; +IF NOT EXISTS ( + SELECT * FROM [__EFMigrationsHistory] + WHERE [MigrationId] = N'20260815004957_AlignAuditLogEventIdUniqueness' +) +BEGIN + + IF EXISTS (SELECT 1 FROM sys.indexes + WHERE name = 'UX_AuditLog_EventId' AND object_id = OBJECT_ID('dbo.AuditLog')) + DROP INDEX UX_AuditLog_EventId ON dbo.AuditLog; +END; + +IF NOT EXISTS ( + SELECT * FROM [__EFMigrationsHistory] + WHERE [MigrationId] = N'20260815004957_AlignAuditLogEventIdUniqueness' +) +BEGIN + INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) + VALUES (N'20260815004957_AlignAuditLogEventIdUniqueness', N'10.0.7'); +END; + +COMMIT; +GO + diff --git a/docs/requirements/Component-AuditLog.md b/docs/requirements/Component-AuditLog.md index c36ef201..67429493 100644 --- a/docs/requirements/Component-AuditLog.md +++ b/docs/requirements/Component-AuditLog.md @@ -291,6 +291,27 @@ stay `Pending` for the next sweep. Cadence is short (default 5 s) when non-empty, longer (default 30 s) when idle; telemetry runs on a dedicated dispatcher. +**Central-side ingest is set-based.** `AuditLogIngestActor` writes a whole +packet with ONE `InsertManyIfNotExistsAsync` statement rather than one +`IF NOT EXISTS … INSERT` round trip per event; the cached-telemetry dual-write +similarly runs the whole packet in ONE transaction (set-based audit insert plus +one single-statement `SiteCalls` upsert per entry) instead of a transaction per +entry. Idempotency is unchanged: duplicates that repeat *within* a packet +collapse first-write-wins before the statement is built, duplicates *across* +packets are eliminated by the anti-semi-join, and any failure falls back to the +per-row / per-entry path — so the documented invariant that one bad row cannot +sink the rest of the batch still holds, it is simply no longer paid for on the +healthy path. + +**Timeout ladder.** The ingest budget is deliberately the smallest on the path: +the site's Ask and the central gRPC handler's Ask are both 30 s, the actor's +own database budget is 20 s and the per-statement SQL timeout is 15 s. Before +this the three were identical, so they expired at the same instant and the +caller learned nothing but "it took 30 s" — no partial ack, no way to tell a +slow database from a wedged singleton. With the inner budgets strictly smaller, +a slow batch is abandoned by the actor first and the accepted-so-far ids are +still replied while the outer Asks are still waiting. + ### Reconciliation pull (self-healing for missed telemetry) A central `SiteAuditReconciliationActor` periodically (default 5 min per site) @@ -490,12 +511,19 @@ MS SQL for direct-write events). Unredacted secrets never persist. silently failing retention job would otherwise be invisible until the table grew unbounded. Per-boundary/per-channel error isolation is unchanged — a single failure still never abandons the rest of the tick. -- **Maintenance command timeout:** the switch-out drop-and-rebuild dance and each +- **Maintenance command timeout:** the switch-out staging batch and each per-channel `DELETE TOP` batch run with an explicit command timeout (`AuditLog:Purge:MaintenanceCommandTimeoutMinutes`, default 30, floor 1 min) rather - than the ~30 s ADO.NET default, which could abort the metadata-only SWITCH mid-dance - on a large or contended partition and leave the live table without - `UX_AuditLog_EventId` until a later tick's CATCH branch rebuilt it. + than the ~30 s ADO.NET default, which could abort the metadata-only SWITCH mid-batch + on a large or contended partition and leave an orphaned staging table for the next + tick's CATCH branch to clean up. +- **Partition-aligned uniqueness:** the switch no longer drops and rebuilds an index. + `EventId` uniqueness rides the clustered `PK_AuditLog (EventId, OccurredAtUtc)`, + which is aligned on `ps_AuditLog_Month(OccurredAtUtc)`, so `SWITCH PARTITION` has no + non-aligned unique index to object to. The predecessor `UX_AuditLog_EventId` forced + an offline whole-table index build inside the switch transaction — blocking every + audit writer for its duration — and left a window in which the idempotency-supporting + index did not exist at all. See migration `AlignAuditLogEventIdUniqueness`. - **Per-channel retention overrides (M5.5 T3):** `AuditLog:PerChannelRetentionDays` is a dictionary keyed by canonical channel name (`ApiOutbound`, `DbOutbound`, `Notification`, `ApiInbound`, `SecuredWrite` — all five `AuditChannel` values are diff --git a/docs/requirements/Component-ConfigurationDatabase.md b/docs/requirements/Component-ConfigurationDatabase.md index e5bd42b0..4fd3e8d8 100644 --- a/docs/requirements/Component-ConfigurationDatabase.md +++ b/docs/requirements/Component-ConfigurationDatabase.md @@ -95,9 +95,11 @@ The configuration database stores all central system data, organized by domain a - `ParentExecutionId` — `CAST(JSON_VALUE(DetailsJson,'$.parentExecutionId') AS uniqueidentifier)` PERSISTED — spawner's `ExecutionId`; null for top-level runs. - `IngestedAtUtc` — `CAST(SWITCHOFFSET(CAST(JSON_VALUE(DetailsJson,'$.ingestedAtUtc') AS datetimeoffset), 0) AS datetime2(7))` — central ingest timestamp; **not** persisted (SQL Server rejects PERSISTED on the non-deterministic `SWITCHOFFSET` expression). - *Clustered primary key:* `(EventId, OccurredAtUtc)` — composite so the key is partition-aligned. `UX_AuditLog_EventId` (unique, non-aligned on `[PRIMARY]`) enforces global `EventId` uniqueness for `InsertIfNotExistsAsync` idempotency. + *Clustered primary key:* `(EventId, OccurredAtUtc)` — composite so the key is partition-aligned, and the **sole** `EventId` uniqueness enforcement backing `InsertIfNotExistsAsync` idempotency. `EventId` is a GUID minted once at the emitting site in the same operation that stamps `OccurredAtUtc`, and neither field is ever re-stamped downstream, so a given `EventId` always arrives with the same `OccurredAtUtc` and can only map to one partition — pair-uniqueness is `EventId`-uniqueness for every row the system produces. The idempotency probe (`WHERE EventId = @id`) still seeks the clustered key's leading column, at the cost of one seek per partition rather than one seek overall. - *Indexes* (all non-clustered, partition-aligned on `ps_AuditLog_Month(OccurredAtUtc)` except `UX_AuditLog_EventId`): + A non-aligned single-column `UX_AuditLog_EventId` on `[PRIMARY]` used to carry that uniqueness; it was dropped by the `AlignAuditLogEventIdUniqueness` migration because a non-aligned index blocks `ALTER TABLE … SWITCH PARTITION`, forcing the retention purge to drop it, switch, and rebuild it **offline inside the switch transaction** on every run. + + *Indexes* (all non-clustered, partition-aligned on `ps_AuditLog_Month(OccurredAtUtc)`): - `IX_AuditLog_OccurredAtUtc` (primary time-range index for global scans) - `IX_AuditLog_Site_Occurred (SourceSiteId, OccurredAtUtc)` (per-site filters) - `IX_AuditLog_CorrelationId (CorrelationId) WHERE CorrelationId IS NOT NULL` (drilldown from a single operation) diff --git a/docs/requirements/Component-SiteCallAudit.md b/docs/requirements/Component-SiteCallAudit.md index 0b2c4f09..58b2388f 100644 --- a/docs/requirements/Component-SiteCallAudit.md +++ b/docs/requirements/Component-SiteCallAudit.md @@ -135,6 +135,28 @@ Pinned)` on the EventStream (transition-only, mirroring health-observable condition rather than a silent log line, and the latch clears with `Pinned=false` once a later tick makes progress. +**The drain runs off the mailbox.** A reconciliation pass is unbounded work — +every site, up to a page ceiling of network pulls each, one upsert per row — so +it runs as a background task with a `PipeTo`-delivered completion message and a +single-flight guard, not as an actor message handler. A handler occupies the +actor for its whole duration, so a post-outage catch-up used to park telemetry +ingest, UI queries and KPI Asks behind it; those callers time out rather than +queue, which made a slow site look like a dead central. The single-flight guard +is raised and lowered ON the actor thread, so the per-site cursor and pinned-latch +dictionaries the pass mutates are still only ever touched by one task at a time, +and the mailbox supplies the memory barrier between consecutive passes. The +daily terminal-row purge uses the same shape. (`NotificationOutboxActor`'s +dispatch sweep is the in-repo reference.) + +**The central upsert is one statement.** `SiteCallAuditRepository.UpsertAsync` +issues a single batch that runs the monotonic UPDATE first and INSERTs only when +nothing matched *and* the row genuinely does not exist — instead of an +unconditional insert-if-absent followed by the update, which cost two round trips +on every packet and wasted the insert half for every packet after the first. The +existence re-check is load-bearing: a zero row count also means "the monotonic +guard rejected this packet", and inserting there would fork the mirror with a +second row for an id that already exists. + ## Retry / Discard Relay Parked cached calls live in the owning site's S&F buffer. Operator Retry/Discard diff --git a/src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/AuditLogIngestActor.cs b/src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/AuditLogIngestActor.cs index 8cece24d..e7b3f9a0 100644 --- a/src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/AuditLogIngestActor.cs +++ b/src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/AuditLogIngestActor.cs @@ -18,8 +18,9 @@ namespace ZB.MOM.WW.ScadaBridge.AuditLog.Central; /// ingest timestamp into DetailsJson (there is no promoted IngestedAtUtc /// column — the value is a DetailsJson field set via /// ) and inserted idempotently -/// via — duplicates are -/// silently swallowed (first-write-wins). +/// via — duplicates +/// are silently swallowed (first-write-wins), whether they repeat inside one +/// packet or across packets. /// /// /// @@ -28,9 +29,11 @@ namespace ZB.MOM.WW.ScadaBridge.AuditLog.Central; /// consistent and the site is free to flip its local row to Forwarded. /// /// -/// Audit-write failures must NEVER abort the user-facing action. The actor -/// wraps each repository call in its own try/catch so a single bad row cannot -/// cause the rest of the batch to be lost, and it guards scope/repository +/// Audit-write failures must NEVER abort the user-facing action. Each message +/// is written as ONE set-based statement (or one transaction, for the cached +/// dual-write); if that fails the actor retries the same work row-by-row inside +/// per-row try/catch, so a single bad row still cannot cause the rest of the +/// batch to be lost. It also guards scope/repository /// resolution so a transient DI fault cannot restart the singleton — those /// catches are what keep this actor alive across handler throws, not the /// supervisor strategy. The override returns @@ -50,6 +53,31 @@ namespace ZB.MOM.WW.ScadaBridge.AuditLog.Central; /// public class AuditLogIngestActor : ReceiveActor { + /// + /// Overall budget for one ingest message's database work, deliberately + /// SHORTER than the gRPC Ask that wraps it. + /// + /// + /// The path used to stack three identical 30 s budgets — the site's Ask + /// (CommunicationOptions.NotificationForwardTimeout), the central gRPC + /// handler's Ask (SiteStreamGrpcServer.AuditIngestAskTimeout) and the + /// ADO.NET command default — so they all expired at the same instant. The + /// caller therefore learned nothing except "it took 30 s": no partial ack, no + /// distinction between a slow database and a wedged singleton. Making the + /// innermost budget strictly smallest means a slow batch is abandoned by the + /// actor FIRST, with the accepted-so-far ids still replied, while the outer + /// Asks are still waiting. + /// + internal static readonly TimeSpan IngestBudget = TimeSpan.FromSeconds(20); + + /// + /// Per-statement SQL timeout for the ingest write, strictly inside + /// so a single wedged statement surfaces as a SQL + /// timeout the per-batch catch can log, rather than consuming the whole + /// actor-level budget and starving the rest of the batch. + /// + internal static readonly TimeSpan IngestSqlCommandTimeout = TimeSpan.FromSeconds(15); + private readonly IServiceProvider? _serviceProvider; private readonly IAuditLogRepository? _injectedRepository; private readonly ILogger _logger; @@ -185,39 +213,75 @@ public class AuditLogIngestActor : ReceiveActor DateTime nowUtc, List accepted) { + // Stamp the ingest timestamp here, not at the site. Redact BEFORE the + // IngestedAtUtc stamp so the redacted copy carries the central-side + // ingest timestamp. The redactor is contract-bound to never throw; a null + // redactor (test composition root, no IAuditRedactor registered) falls + // back to the SafeDefault rather than pass-through, so HTTP header + // redaction always runs. IngestedAtUtc is a DetailsJson field on the + // canonical record, so stamp it via the projection helper. + var safeRedactor = redactor ?? SafeDefaultAuditRedactor.Instance; + var projected = new List(cmd.Events.Count); foreach (var evt in cmd.Events) { - try + var filtered = safeRedactor.Apply(evt); + projected.Add(AuditRowProjection.WithIngestedAtUtc(filtered, nowUtc)); + } + + // ONE set-based statement for the whole packet (WP2.2) instead of one + // IF NOT EXISTS … INSERT round trip per event. A telemetry packet of 200 + // rows used to cost 200 sequential round trips on the central singleton's + // dispatcher; it now costs two. Idempotency is unchanged — duplicates + // within the packet collapse first-write-wins, duplicates against + // committed rows are eliminated by the anti-semi-join, and a concurrent + // writer degrades to the repository's per-row fallback. + using var budget = new CancellationTokenSource(IngestBudget); + try + { + await repository + .InsertManyIfNotExistsAsync(projected, IngestSqlCommandTimeout, budget.Token) + .ConfigureAwait(false); + + // A batch that returned without throwing means every row is now + // present — inserted here or already committed by an earlier delivery. + // Both count as accepted: the storage state is consistent and the site + // is free to flip its local rows to Forwarded. + foreach (var evt in cmd.Events) { - // Stamp the ingest timestamp here, not at the site. The - // repository's duplicate-key hardening already swallows - // duplicate-key races, so the same id arriving twice (site - // retry, reconciliation) is a silent no-op. - // Redact BEFORE the IngestedAtUtc stamp so the redacted - // copy carries the central-side ingest timestamp. The redactor - // is contract-bound to never throw. A null - // redactor (test composition root, no IAuditRedactor - // registered) now falls back to the SafeDefault rather than - // pass-through, so HTTP header redaction always runs. - // IngestedAtUtc is a DetailsJson field on - // the canonical record, so stamp it via the projection helper. - var safeRedactor = redactor ?? SafeDefaultAuditRedactor.Instance; - var filtered = safeRedactor.Apply(evt); - var ingested = AuditRowProjection.WithIngestedAtUtc(filtered, nowUtc); - await repository.InsertIfNotExistsAsync(ingested).ConfigureAwait(false); accepted.Add(evt.EventId); } - catch (Exception ex) + } + catch (Exception ex) + { + // The batch is a throughput optimisation, NOT a change to the + // failure grain. The documented invariant — "a single bad row cannot + // cause the rest of the batch to be lost" — is preserved by falling + // back to the per-row path here, so a poison row still costs only + // itself. Re-running rows the failed batch may already have committed + // is safe: every insert is idempotent on EventId. + _logger.LogWarning(ex, + "Set-based ingest of {Count} audit event(s) failed; falling back to per-row inserts so one bad row does not sink the batch.", + cmd.Events.Count); + + for (var i = 0; i < projected.Count; i++) { - // Per-row catch — one bad row never sinks the whole batch. - // The row stays Pending at the site; the next drain retries. - // Bump the central health counter so a - // sustained insert-throw failure surfaces on the dashboard. - try { failureCounter?.Increment(); } - catch { /* counter must never throw — defence in depth */ } - _logger.LogError(ex, - "Failed to persist audit event {EventId} during batch ingest; row will be retried by the site.", - evt.EventId); + try + { + await repository.InsertIfNotExistsAsync(projected[i], budget.Token).ConfigureAwait(false); + accepted.Add(cmd.Events[i].EventId); + } + catch (Exception rowEx) + { + // Per-row catch — one bad row never sinks the whole batch. + // The row stays Pending at the site; the next drain retries. + // Bump the central health counter so a sustained insert-throw + // failure surfaces on the dashboard. + try { failureCounter?.Increment(); } + catch { /* counter must never throw — defence in depth */ } + _logger.LogError(rowEx, + "Failed to persist audit event {EventId} during batch ingest; row will be retried by the site.", + cmd.Events[i].EventId); + } } } } @@ -262,6 +326,24 @@ public class AuditLogIngestActor : ReceiveActor var strategy = dbContext.Database.CreateExecutionStrategy(); + // Fast path (WP2.2): the WHOLE packet in ONE transaction — one + // set-based audit insert plus one single-statement upsert per entry. + // The predecessor opened a transaction PER entry and issued three + // statements inside it (IF NOT EXISTS insert, then the two-statement + // SiteCalls upsert), so a 50-entry packet cost ~200 round trips and 50 + // commits. If anything faults, the per-entry loop below re-runs the + // packet with its original per-entry isolation, so the documented + // "one entry's failure does not abort the others" invariant survives — + // it is simply no longer paid for on the healthy path. + using var budget = new CancellationTokenSource(IngestBudget); + if (await TryIngestCachedBatchAsync( + strategy, dbContext, auditRepo, siteCallRepo, redactor, cmd, accepted, budget.Token) + .ConfigureAwait(false)) + { + replyTo.Tell(new IngestCachedTelemetryReply(accepted)); + return; + } + foreach (var entry in cmd.Entries) { try @@ -330,6 +412,90 @@ public class AuditLogIngestActor : ReceiveActor replyTo.Tell(new IngestCachedTelemetryReply(accepted)); } + /// + /// Attempts the whole cached-telemetry packet as ONE transaction: a single + /// set-based audit insert followed by one monotonic SiteCalls upsert + /// per entry. Returns when it committed (and only then + /// appends to ), when the + /// caller should fall back to the per-entry transaction loop. + /// + /// + /// The batch is all-or-nothing by construction — it is a single transaction — + /// which is why a failure MUST fall back rather than report partial success: + /// the per-entry loop is what turns one poison entry into one lost entry + /// instead of a lost packet. Nothing is appended to + /// until the commit returns, so a failed attempt + /// leaves the caller's list untouched and the retry starts from a clean slate. + /// + private async Task TryIngestCachedBatchAsync( + Microsoft.EntityFrameworkCore.Storage.IExecutionStrategy strategy, + ScadaBridgeDbContext dbContext, + IAuditLogRepository auditRepo, + ISiteCallAuditRepository siteCallRepo, + IAuditRedactor? redactor, + IngestCachedTelemetryCommand cmd, + List accepted, + CancellationToken ct) + { + try + { + await strategy.ExecuteAsync(async () => + { + await using var tx = await dbContext.Database + .BeginTransactionAsync(ct) + .ConfigureAwait(false); + + // One central-side instant for the whole packet so a join across + // the two tables sees matching timestamps (debugging convenience, + // not a correctness invariant). + var ingestedAt = DateTime.UtcNow; + var safeRedactor = redactor ?? SafeDefaultAuditRedactor.Instance; + + var auditRows = new List(cmd.Entries.Count); + foreach (var entry in cmd.Entries) + { + // Only the AuditLog row's payload columns are redactable; + // SiteCalls carries operational state only (status, retry + // count) and is left untouched. + var filteredAudit = safeRedactor.Apply(entry.Audit); + auditRows.Add(AuditRowProjection.WithIngestedAtUtc(filteredAudit, ingestedAt)); + } + + await auditRepo + .InsertManyIfNotExistsAsync(auditRows, IngestSqlCommandTimeout, ct) + .ConfigureAwait(false); + + foreach (var entry in cmd.Entries) + { + await siteCallRepo + .UpsertAsync(entry.SiteCall with { IngestedAtUtc = ingestedAt }, ct) + .ConfigureAwait(false); + } + + await tx.CommitAsync(ct).ConfigureAwait(false); + }).ConfigureAwait(false); + } + catch (Exception ex) + { + // No health-counter bump here — the per-entry retry is the authority + // on whether this packet genuinely failed, and double-counting a + // batch that the fallback then writes successfully would make the + // dashboard read as a sustained fault during normal contention. + _logger.LogWarning( + ex, + "Batched cached-telemetry dual-write of {Count} entr(ies) failed; falling back to per-entry transactions.", + cmd.Entries.Count); + return false; + } + + foreach (var entry in cmd.Entries) + { + accepted.Add(entry.Audit.EventId); + } + + return true; + } + /// /// Fallback handler installed on the single-repository test ctor — that /// ctor has no DbContext and no , so diff --git a/src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/AuditLogPurgeActor.cs b/src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/AuditLogPurgeActor.cs index dc95d731..d824c32d 100644 --- a/src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/AuditLogPurgeActor.cs +++ b/src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/AuditLogPurgeActor.cs @@ -17,8 +17,9 @@ namespace ZB.MOM.WW.ScadaBridge.AuditLog.Central; /// for monthly boundaries whose latest OccurredAtUtc is older /// than DateTime.UtcNow - RetentionDays. /// For each eligible boundary, calls -/// which runs -/// the drop-and-rebuild dance around UX_AuditLog_EventId. +/// , a +/// metadata-only staging-table switch (WP2.2 removed the index +/// drop/rebuild that used to bracket it). /// Publishes on the actor-system /// EventStream so the central health collector + ops surfaces /// can subscribe without coupling to this actor. @@ -26,11 +27,10 @@ namespace ZB.MOM.WW.ScadaBridge.AuditLog.Central; /// /// /// -/// Daily cadence. Partition switch is metadata-only but the -/// drop-and-rebuild dance briefly removes UX_AuditLog_EventId; running -/// more often than necessary trades unique-index rebuild outages for -/// negligible freshness wins. The default 24-hour interval matches -/// alog.md §10's retention policy. +/// Daily cadence. The partition switch is metadata-only, but it still +/// takes schema-modification locks on a table the ingest path is writing to; +/// running more often than necessary trades contention for negligible freshness +/// wins. The default 24-hour interval matches alog.md §10's retention policy. /// /// /// Continue-on-error. A single boundary that throws (transient SQL diff --git a/src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/AuditLogPurgeOptions.cs b/src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/AuditLogPurgeOptions.cs index 5621cd56..0d97571c 100644 --- a/src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/AuditLogPurgeOptions.cs +++ b/src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/AuditLogPurgeOptions.cs @@ -10,12 +10,12 @@ namespace ZB.MOM.WW.ScadaBridge.AuditLog.Central; /// /// /// -/// The purge actor is a daily-cadence singleton, not a hot-loop, because -/// partition-switch I/O is metadata-only but the drop-and-rebuild dance -/// briefly removes the UX_AuditLog_EventId unique index — running -/// more often than necessary trades index-rebuild outages for marginal -/// freshness gains. Lower this only when an operator can prove they need -/// sub-daily purge granularity. +/// The purge actor is a daily-cadence singleton, not a hot-loop. The +/// partition switch itself is metadata-only (since WP2.2's +/// AlignAuditLogEventIdUniqueness it no longer drops and rebuilds a +/// non-aligned unique index around it), but it still takes schema-modification +/// locks and competes with ingest for the same table. Lower this only when an +/// operator can prove they need sub-daily purge granularity. /// /// /// exists for tests to drop the cadence to @@ -58,15 +58,14 @@ public sealed class AuditLogPurgeOptions /// /// Per-command timeout (in minutes) for the maintenance SQL the purge tick issues — - /// both the partition switch-out drop-and-rebuild dance + /// both the partition switch-out staging batch /// () /// and each per-channel DELETE TOP batch. Default 30 minutes. /// /// /// The ADO.NET default command timeout is ~30 seconds. On a large or contended partition the - /// SWITCH dance (which briefly drops UX_AuditLog_EventId) can exceed that and abort - /// mid-flight — leaving the live table without its idempotency-supporting unique index until a - /// later tick's CATCH branch rebuilds it (arch-review 04, S2). A generous maintenance timeout + /// SWITCH batch can exceed that and abort mid-flight, leaving an orphaned staging table for the + /// next tick's CATCH branch to clean up (arch-review 04, S2). A generous maintenance timeout /// lets the metadata-only switch complete rather than self-locking. Resolved via /// , clamped to a 1-minute floor. /// diff --git a/src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/IAuditLogRepository.cs b/src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/IAuditLogRepository.cs index 6aa0e707..72e31a66 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/IAuditLogRepository.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/IAuditLogRepository.cs @@ -34,6 +34,62 @@ public interface IAuditLogRepository /// A task that represents the asynchronous operation. Task InsertIfNotExistsAsync(AuditEvent evt, CancellationToken ct = default); + /// + /// Set-based form of : inserts every + /// event in that does not already exist, in as few + /// round trips as the provider allows, and returns the number of rows + /// actually written. First-write-wins idempotency is unchanged — an EventId + /// already present is silently skipped. + /// + /// + /// + /// Duplicate EventIds are tolerated both within and across packets. + /// Duplicates inside one call collapse to the first occurrence before the + /// statement is built (an append-only row is immutable, so later copies of an + /// EventId carry the same content); duplicates against already-committed rows + /// are eliminated by the anti-semi-join. A concurrent writer that commits a + /// row mid-statement is handled by falling back to the per-row path, so the + /// batch is a throughput optimisation and never a correctness dependency. + /// + /// + /// Default implementation. The interface supplies a per-row loop so a + /// test double or an alternative store keeps working unchanged; the EF Core + /// implementation overrides it with a genuine set-based statement. Callers on + /// the ingest hot path should always prefer this method — the per-row form + /// cost one IF NOT EXISTS … INSERT round trip per audit event. + /// + /// + /// Audit events to insert; may contain duplicates. + /// + /// Optional per-statement timeout. The ingest actor passes a budget strictly + /// SHORTER than the gRPC Ask that wraps it, so the reply is produced before + /// the caller gives up rather than at the same instant. + /// + /// Cancellation token. + /// A task that resolves to the number of rows inserted (duplicates excluded). + async Task InsertManyIfNotExistsAsync( + IReadOnlyList events, + TimeSpan? commandTimeout = null, + CancellationToken ct = default) + { + ArgumentNullException.ThrowIfNull(events); + + var inserted = 0; + var seen = new HashSet(events.Count); + foreach (var evt in events) + { + if (evt is null || !seen.Add(evt.EventId)) + { + continue; + } + + await InsertIfNotExistsAsync(evt, ct).ConfigureAwait(false); + inserted++; + } + + return inserted; + } + /// /// Returns up to rows matching /// , ordered by (OccurredAtUtc DESC, EventId DESC). @@ -60,36 +116,32 @@ public interface IAuditLogRepository /// /// /// - /// Drop-and-rebuild dance. UX_AuditLog_EventId is intentionally - /// non-partition-aligned (it lives on [PRIMARY] so single-column - /// EventId uniqueness — required by — - /// can be enforced cheaply). SQL Server rejects - /// ALTER TABLE … SWITCH PARTITION while a non-aligned unique index - /// is present, so the implementation drops the index, creates a staging - /// table with byte-identical schema, switches the partition's data into - /// staging, drops staging (discarding the rows), and rebuilds the unique - /// index. The CATCH branch guarantees the index is rebuilt even on partial - /// failure so the table never returns to live traffic without its - /// idempotency-supporting index. + /// Partition-aligned uniqueness — no index drop. EventId uniqueness is + /// enforced by the clustered PK_AuditLog (EventId, OccurredAtUtc), + /// which is aligned on ps_AuditLog_Month(OccurredAtUtc), so + /// ALTER TABLE … SWITCH PARTITION has no non-aligned unique index to + /// object to. The implementation creates a staging table with byte-identical + /// schema, switches the partition's data into staging, and drops staging + /// (discarding the rows). Nothing is dropped from the live table, so there is + /// no window in which the idempotency enforcement is absent and no offline + /// index rebuild inside the switch transaction. /// /// - /// Outage window. The dance briefly removes the unique index, so - /// concurrent calls during the switch - /// could in principle race past the IF NOT EXISTS check without the index - /// catching the duplicate. This is acceptable for the daily purge cadence - /// — the inserts that the IF NOT EXISTS check guards are themselves rare - /// enough that a sub-second collision window is operationally negligible, - /// and the composite PK still rejects same-(EventId, OccurredAtUtc) rows. + /// The predecessor design carried a non-aligned + /// UX_AuditLog_EventId on [PRIMARY] that had to be dropped and + /// rebuilt around every switch. A defensive guarded DROP INDEX remains + /// in the batch so a database restored from a pre-alignment backup still + /// purges; it is a one-way cleanup, never rebuilt. /// /// /// Lower-bound datetime of the monthly partition to switch out. /// /// Optional per-command timeout for the maintenance SQL (the row-count sample plus the - /// drop-and-rebuild dance). When null the provider default applies. The purge actor passes + /// staging/switch batch). When null the provider default applies. The purge actor passes /// - /// (default 30 min) because the ~30s ADO.NET default can abort the switch mid-dance on a large - /// or contended partition, leaving the table without UX_AuditLog_EventId until the next - /// tick recovers (arch-review 04, S2). + /// (default 30 min) because the ~30s ADO.NET default can abort the switch mid-batch on a large + /// or contended partition, leaving an orphaned staging table for the next tick's CATCH to clean + /// up (arch-review 04, S2). /// /// Cancellation token. /// A task that resolves to the approximate number of rows discarded by the partition switch. diff --git a/src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/INotificationOutboxRepository.cs b/src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/INotificationOutboxRepository.cs index fa8299de..9165e576 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/INotificationOutboxRepository.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/INotificationOutboxRepository.cs @@ -46,10 +46,25 @@ public interface INotificationOutboxRepository Task GetByIdAsync(string notificationId, CancellationToken cancellationToken = default); /// - /// Marks modified and persists it (status transitions). - /// Commits internally — this call is its own transaction. + /// Persists 's delivery-state columns — + /// Status, RetryCount, LastError, ResolvedTargets, + /// LastAttemptAt, NextAttemptAt, DeliveredAt. Commits + /// internally — this call is its own transaction. /// - /// The notification to update. + /// + /// Scope is deliberately narrow. Those seven columns are the ONLY + /// mutable state a notification has: everything else (identity, type, list, + /// subject, body, type data, source/origin attribution, enqueue and creation + /// timestamps) is written once at ingest and is immutable by contract. Every + /// caller — the dispatcher's per-attempt write and the operator retry/discard + /// one-shots — touches only this set. Implementations are therefore free to + /// issue a targeted UPDATE of these columns rather than rewriting the whole + /// row, which matters because the row carries nvarchar(max) body and + /// payload columns and the dispatcher writes on EVERY attempt. A future + /// caller that needs to change an immutable column must add its own method + /// rather than widening this one. + /// + /// The notification whose delivery state should be persisted. /// Cancellation token. /// A task that completes when the notification has been persisted. Task UpdateAsync(Notification n, CancellationToken cancellationToken = default); diff --git a/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Configurations/AuditLogEntityTypeConfiguration.cs b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Configurations/AuditLogEntityTypeConfiguration.cs index 9b86006e..325ab596 100644 --- a/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Configurations/AuditLogEntityTypeConfiguration.cs +++ b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Configurations/AuditLogEntityTypeConfiguration.cs @@ -172,15 +172,18 @@ public class AuditLogEntityTypeConfiguration : IEntityTypeConfiguration new { e.EventId, e.OccurredAtUtc }); - builder.HasIndex(e => e.EventId) - .IsUnique() - .HasDatabaseName("UX_AuditLog_EventId"); - // Index names are locked for reconciliation/migration discoverability. The // column SETS migrate to the canonical/computed shape (alog.md §4 semantics // preserved): Channel→Category, Site/Node/Execution/ParentExecution now read diff --git a/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/20260815004957_AlignAuditLogEventIdUniqueness.Designer.cs b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/20260815004957_AlignAuditLogEventIdUniqueness.Designer.cs new file mode 100644 index 00000000..fc39c29b --- /dev/null +++ b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/20260815004957_AlignAuditLogEventIdUniqueness.Designer.cs @@ -0,0 +1,2090 @@ +// +using System; +using Microsoft.EntityFrameworkCore; +using Microsoft.EntityFrameworkCore.Infrastructure; +using Microsoft.EntityFrameworkCore.Metadata; +using Microsoft.EntityFrameworkCore.Migrations; +using Microsoft.EntityFrameworkCore.Storage.ValueConversion; +using ZB.MOM.WW.ScadaBridge.ConfigurationDatabase; + +#nullable disable + +namespace ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Migrations +{ + [DbContext(typeof(ScadaBridgeDbContext))] + [Migration("20260815004957_AlignAuditLogEventIdUniqueness")] + partial class AlignAuditLogEventIdUniqueness + { + /// + protected override void BuildTargetModel(ModelBuilder modelBuilder) + { +#pragma warning disable 612, 618 + modelBuilder + .HasAnnotation("ProductVersion", "10.0.7") + .HasAnnotation("Relational:MaxIdentifierLength", 128); + + SqlServerModelBuilderExtensions.UseIdentityColumns(modelBuilder); + + modelBuilder.Entity("Microsoft.AspNetCore.DataProtection.EntityFrameworkCore.DataProtectionKey", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("FriendlyName") + .HasColumnType("nvarchar(max)"); + + b.Property("Xml") + .HasColumnType("nvarchar(max)"); + + b.HasKey("Id"); + + b.ToTable("DataProtectionKeys"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Audit.AuditLogEntry", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("Action") + .IsRequired() + .HasMaxLength(100) + .HasColumnType("nvarchar(100)"); + + b.Property("AfterStateJson") + .HasColumnType("nvarchar(max)"); + + b.Property("BundleImportId") + .HasColumnType("uniqueidentifier"); + + b.Property("EntityId") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("EntityName") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("EntityType") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("Timestamp") + .HasColumnType("datetimeoffset"); + + b.Property("User") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.HasKey("Id"); + + b.HasIndex("Action"); + + b.HasIndex("BundleImportId") + .HasDatabaseName("IX_AuditLogEntries_BundleImportId"); + + b.HasIndex("EntityId"); + + b.HasIndex("EntityType"); + + b.HasIndex("Timestamp"); + + b.HasIndex("User"); + + b.ToTable("AuditLogEntries"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Audit.SiteCall", b => + { + b.Property("TrackedOperationId") + .HasMaxLength(36) + .IsUnicode(false) + .HasColumnType("varchar(36)"); + + b.Property("Channel") + .IsRequired() + .HasMaxLength(32) + .IsUnicode(false) + .HasColumnType("varchar(32)"); + + b.Property("CreatedAtUtc") + .HasColumnType("datetime2"); + + b.Property("HttpStatus") + .HasColumnType("int"); + + b.Property("IngestedAtUtc") + .HasColumnType("datetime2"); + + b.Property("LastError") + .HasMaxLength(1024) + .HasColumnType("nvarchar(1024)"); + + b.Property("RetryCount") + .HasColumnType("int"); + + b.Property("SourceNode") + .HasMaxLength(64) + .IsUnicode(false) + .HasColumnType("varchar(64)"); + + b.Property("SourceSite") + .IsRequired() + .HasMaxLength(64) + .IsUnicode(false) + .HasColumnType("varchar(64)"); + + b.Property("Status") + .IsRequired() + .HasMaxLength(32) + .IsUnicode(false) + .HasColumnType("varchar(32)"); + + b.Property("Target") + .IsRequired() + .HasMaxLength(256) + .IsUnicode(false) + .HasColumnType("varchar(256)"); + + b.Property("TerminalAtUtc") + .HasColumnType("datetime2"); + + b.Property("UpdatedAtUtc") + .HasColumnType("datetime2"); + + b.HasKey("TrackedOperationId"); + + b.HasIndex("CreatedAtUtc") + .HasDatabaseName("IX_SiteCalls_NonTerminal") + .HasFilter("[TerminalAtUtc] IS NULL"); + + SqlServerIndexBuilderExtensions.IncludeProperties(b.HasIndex("CreatedAtUtc"), new[] { "SourceSite", "SourceNode", "Status" }); + + b.HasIndex("TerminalAtUtc") + .HasDatabaseName("IX_SiteCalls_Terminal") + .HasFilter("[TerminalAtUtc] IS NOT NULL"); + + b.HasIndex("SourceSite", "CreatedAtUtc") + .IsDescending(false, true) + .HasDatabaseName("IX_SiteCalls_Source_Created"); + + b.HasIndex("Status", "UpdatedAtUtc") + .IsDescending(false, true) + .HasDatabaseName("IX_SiteCalls_Status_Updated"); + + b.ToTable("SiteCalls", (string)null); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Deployment.DeployedConfigSnapshot", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("ConfigurationJson") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("DeployedAt") + .HasColumnType("datetimeoffset"); + + b.Property("DeploymentId") + .IsRequired() + .HasMaxLength(100) + .HasColumnType("nvarchar(100)"); + + b.Property("InstanceId") + .HasColumnType("int"); + + b.Property("RevisionHash") + .IsRequired() + .HasMaxLength(100) + .HasColumnType("nvarchar(100)"); + + b.HasKey("Id"); + + b.HasIndex("DeploymentId"); + + b.HasIndex("InstanceId") + .IsUnique(); + + b.ToTable("DeployedConfigSnapshots"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Deployment.DeploymentRecord", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("CompletedAt") + .HasColumnType("datetimeoffset"); + + b.Property("DeployedAt") + .HasColumnType("datetimeoffset"); + + b.Property("DeployedBy") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("DeploymentId") + .IsRequired() + .HasMaxLength(100) + .HasColumnType("nvarchar(100)"); + + b.Property("ErrorMessage") + .HasColumnType("nvarchar(max)"); + + b.Property("InstanceId") + .HasColumnType("int"); + + b.Property("RevisionHash") + .HasMaxLength(100) + .HasColumnType("nvarchar(100)"); + + b.Property("RowVersion") + .IsConcurrencyToken() + .IsRequired() + .ValueGeneratedOnAddOrUpdate() + .HasColumnType("rowversion"); + + b.Property("Status") + .IsRequired() + .HasMaxLength(50) + .HasColumnType("nvarchar(50)"); + + b.HasKey("Id"); + + b.HasIndex("DeployedAt"); + + b.HasIndex("DeploymentId") + .IsUnique(); + + b.HasIndex("InstanceId"); + + b.ToTable("DeploymentRecords"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Deployment.PendingDeployment", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("ConfigurationJson") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("CreatedAtUtc") + .HasColumnType("datetimeoffset"); + + b.Property("DeploymentId") + .IsRequired() + .HasMaxLength(100) + .HasColumnType("nvarchar(100)"); + + b.Property("ExpiresAtUtc") + .HasColumnType("datetimeoffset"); + + b.Property("InstanceId") + .HasColumnType("int"); + + b.Property("RevisionHash") + .IsRequired() + .HasMaxLength(100) + .HasColumnType("nvarchar(100)"); + + b.Property("Token") + .IsRequired() + .HasMaxLength(128) + .HasColumnType("nvarchar(128)"); + + b.HasKey("Id"); + + b.HasIndex("DeploymentId") + .IsUnique(); + + b.HasIndex("ExpiresAtUtc"); + + b.HasIndex("InstanceId"); + + b.ToTable("PendingDeployments", (string)null); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Deployment.SystemArtifactDeploymentRecord", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("ArtifactType") + .IsRequired() + .HasMaxLength(100) + .HasColumnType("nvarchar(100)"); + + b.Property("DeployedAt") + .HasColumnType("datetimeoffset"); + + b.Property("DeployedBy") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("PerSiteStatus") + .HasMaxLength(4000) + .HasColumnType("nvarchar(4000)"); + + b.HasKey("Id"); + + b.HasIndex("DeployedAt"); + + b.ToTable("SystemArtifactDeploymentRecords"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.ExternalSystems.DatabaseConnectionDefinition", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("ConnectionString") + .IsRequired() + .HasMaxLength(8000) + .HasColumnType("nvarchar(max)"); + + b.Property("MaxRetries") + .HasColumnType("int"); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("RetryDelay") + .HasColumnType("time"); + + b.HasKey("Id"); + + b.HasIndex("Name") + .IsUnique(); + + b.ToTable("DatabaseConnectionDefinitions"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.ExternalSystems.ExternalSystemDefinition", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("AuthConfiguration") + .HasMaxLength(8000) + .HasColumnType("nvarchar(max)"); + + b.Property("AuthType") + .IsRequired() + .HasMaxLength(50) + .HasColumnType("nvarchar(50)"); + + b.Property("EndpointUrl") + .IsRequired() + .HasMaxLength(2000) + .HasColumnType("nvarchar(2000)"); + + b.Property("MaxRetries") + .HasColumnType("int"); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("RetryDelay") + .HasColumnType("time"); + + b.Property("TimeoutSeconds") + .ValueGeneratedOnAdd() + .HasColumnType("int") + .HasDefaultValue(0); + + b.HasKey("Id"); + + b.HasIndex("Name") + .IsUnique(); + + b.ToTable("ExternalSystemDefinitions"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.ExternalSystems.ExternalSystemMethod", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("ExternalSystemDefinitionId") + .HasColumnType("int"); + + b.Property("HttpMethod") + .IsRequired() + .HasMaxLength(10) + .HasColumnType("nvarchar(10)"); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("ParameterDefinitions") + .HasMaxLength(4000) + .HasColumnType("nvarchar(4000)"); + + b.Property("Path") + .IsRequired() + .HasMaxLength(2000) + .HasColumnType("nvarchar(2000)"); + + b.Property("ReturnDefinition") + .HasMaxLength(4000) + .HasColumnType("nvarchar(4000)"); + + b.HasKey("Id"); + + b.HasIndex("ExternalSystemDefinitionId", "Name") + .IsUnique(); + + b.ToTable("ExternalSystemMethods"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.InboundApi.ApiMethod", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("ParameterDefinitions") + .HasMaxLength(4000) + .HasColumnType("nvarchar(4000)"); + + b.Property("ReturnDefinition") + .HasMaxLength(4000) + .HasColumnType("nvarchar(4000)"); + + b.Property("Script") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("TimeoutSeconds") + .HasColumnType("int"); + + b.HasKey("Id"); + + b.HasIndex("Name") + .IsUnique(); + + b.ToTable("ApiMethods"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.Area", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("ParentAreaId") + .HasColumnType("int"); + + b.Property("SiteId") + .HasColumnType("int"); + + b.HasKey("Id"); + + b.HasIndex("ParentAreaId"); + + b.HasIndex("SiteId", "ParentAreaId", "Name") + .IsUnique() + .HasFilter("[ParentAreaId] IS NOT NULL"); + + b.ToTable("Areas"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.Instance", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("AreaId") + .HasColumnType("int"); + + b.Property("SiteId") + .HasColumnType("int"); + + b.Property("State") + .IsRequired() + .HasMaxLength(50) + .HasColumnType("nvarchar(50)"); + + b.Property("TemplateId") + .HasColumnType("int"); + + b.Property("UniqueName") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.HasKey("Id"); + + b.HasIndex("AreaId"); + + b.HasIndex("TemplateId"); + + b.HasIndex("SiteId", "UniqueName") + .IsUnique(); + + b.ToTable("Instances"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.InstanceAlarmOverride", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("AlarmCanonicalName") + .IsRequired() + .HasMaxLength(400) + .HasColumnType("nvarchar(400)"); + + b.Property("InstanceId") + .HasColumnType("int"); + + b.Property("PriorityLevelOverride") + .HasColumnType("int"); + + b.Property("TriggerConfigurationOverride") + .HasMaxLength(4000) + .HasColumnType("nvarchar(4000)"); + + b.HasKey("Id"); + + b.HasIndex("InstanceId", "AlarmCanonicalName") + .IsUnique(); + + b.ToTable("InstanceAlarmOverrides"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.InstanceAttributeOverride", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("AttributeName") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("ElementDataType") + .HasMaxLength(50) + .HasColumnType("nvarchar(50)"); + + b.Property("InstanceId") + .HasColumnType("int"); + + b.Property("OverrideValue") + .HasColumnType("nvarchar(max)"); + + b.HasKey("Id"); + + b.HasIndex("InstanceId", "AttributeName") + .IsUnique(); + + b.ToTable("InstanceAttributeOverrides"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.InstanceConnectionBinding", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("AttributeName") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("DataConnectionId") + .HasColumnType("int"); + + b.Property("DataSourceReferenceOverride") + .HasMaxLength(512) + .HasColumnType("nvarchar(512)"); + + b.Property("InstanceId") + .HasColumnType("int"); + + b.HasKey("Id"); + + b.HasIndex("DataConnectionId"); + + b.HasIndex("InstanceId", "AttributeName") + .IsUnique(); + + b.ToTable("InstanceConnectionBindings"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.InstanceNativeAlarmSourceOverride", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("ConditionFilterOverride") + .HasMaxLength(1000) + .HasColumnType("nvarchar(1000)"); + + b.Property("ConnectionNameOverride") + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("InstanceId") + .HasColumnType("int"); + + b.Property("SourceCanonicalName") + .IsRequired() + .HasMaxLength(400) + .HasColumnType("nvarchar(400)"); + + b.Property("SourceReferenceOverride") + .HasMaxLength(1000) + .HasColumnType("nvarchar(1000)"); + + b.HasKey("Id"); + + b.HasIndex("InstanceId", "SourceCanonicalName") + .IsUnique(); + + b.ToTable("InstanceNativeAlarmSourceOverrides"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Kpi.KpiRollupHourly", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("bigint"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("HourStartUtc") + .HasColumnType("datetime2"); + + b.Property("MaxValue") + .HasColumnType("float"); + + b.Property("Metric") + .IsRequired() + .HasMaxLength(64) + .IsUnicode(false) + .HasColumnType("varchar(64)"); + + b.Property("MinValue") + .HasColumnType("float"); + + b.Property("SampleCount") + .HasColumnType("int"); + + b.Property("Scope") + .IsRequired() + .HasMaxLength(16) + .IsUnicode(false) + .HasColumnType("varchar(16)"); + + b.Property("ScopeKey") + .HasMaxLength(64) + .IsUnicode(false) + .HasColumnType("varchar(64)"); + + b.Property("Source") + .IsRequired() + .HasMaxLength(64) + .IsUnicode(false) + .HasColumnType("varchar(64)"); + + b.Property("Value") + .HasColumnType("float"); + + b.HasKey("Id"); + + b.HasIndex("HourStartUtc") + .HasDatabaseName("IX_KpiRollupHourly_Hour"); + + b.HasIndex("Source", "Metric", "Scope", "ScopeKey", "HourStartUtc") + .IsUnique() + .HasDatabaseName("IX_KpiRollupHourly_Series"); + + b.ToTable("KpiRollupHourly", (string)null); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Kpi.KpiSample", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("bigint"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("CapturedAtUtc") + .HasColumnType("datetime2"); + + b.Property("Metric") + .IsRequired() + .HasMaxLength(64) + .IsUnicode(false) + .HasColumnType("varchar(64)"); + + b.Property("Scope") + .IsRequired() + .HasMaxLength(16) + .IsUnicode(false) + .HasColumnType("varchar(16)"); + + b.Property("ScopeKey") + .HasMaxLength(64) + .IsUnicode(false) + .HasColumnType("varchar(64)"); + + b.Property("Source") + .IsRequired() + .HasMaxLength(64) + .IsUnicode(false) + .HasColumnType("varchar(64)"); + + b.Property("Value") + .HasColumnType("float"); + + b.HasKey("Id"); + + b.HasIndex("CapturedAtUtc") + .HasDatabaseName("IX_KpiSample_Captured"); + + b.HasIndex("Source", "Metric", "Scope", "ScopeKey", "CapturedAtUtc") + .HasDatabaseName("IX_KpiSample_Series"); + + b.ToTable("KpiSample", (string)null); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Notifications.Notification", b => + { + b.Property("NotificationId") + .HasMaxLength(64) + .HasColumnType("nvarchar(64)"); + + b.Property("Body") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("CreatedAt") + .HasColumnType("datetimeoffset"); + + b.Property("DeliveredAt") + .HasColumnType("datetimeoffset"); + + b.Property("LastAttemptAt") + .HasColumnType("datetimeoffset"); + + b.Property("LastError") + .HasMaxLength(4000) + .HasColumnType("nvarchar(4000)"); + + b.Property("ListName") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("NextAttemptAt") + .HasColumnType("datetimeoffset"); + + b.Property("OriginExecutionId") + .HasColumnType("uniqueidentifier"); + + b.Property("OriginParentExecutionId") + .HasColumnType("uniqueidentifier"); + + b.Property("ResolvedTargets") + .HasColumnType("nvarchar(max)"); + + b.Property("RetryCount") + .HasColumnType("int"); + + b.Property("SiteEnqueuedAt") + .HasColumnType("datetimeoffset"); + + b.Property("SourceInstanceId") + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("SourceNode") + .HasMaxLength(64) + .IsUnicode(false) + .HasColumnType("varchar(64)"); + + b.Property("SourceScript") + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("SourceSiteId") + .IsRequired() + .HasMaxLength(100) + .HasColumnType("nvarchar(100)"); + + b.Property("Status") + .IsRequired() + .HasMaxLength(32) + .HasColumnType("nvarchar(32)"); + + b.Property("Subject") + .IsRequired() + .HasMaxLength(1000) + .HasColumnType("nvarchar(1000)"); + + b.Property("Type") + .IsRequired() + .HasMaxLength(32) + .HasColumnType("nvarchar(32)"); + + b.Property("TypeData") + .HasColumnType("nvarchar(max)"); + + b.HasKey("NotificationId"); + + b.HasIndex("SourceSiteId", "CreatedAt"); + + b.HasIndex("Status", "NextAttemptAt"); + + b.ToTable("Notifications"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Notifications.NotificationList", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("Type") + .IsRequired() + .HasMaxLength(32) + .HasColumnType("nvarchar(32)"); + + b.HasKey("Id"); + + b.HasIndex("Name") + .IsUnique(); + + b.ToTable("NotificationLists"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Notifications.NotificationRecipient", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("EmailAddress") + .HasMaxLength(500) + .HasColumnType("nvarchar(500)"); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("NotificationListId") + .HasColumnType("int"); + + b.Property("PhoneNumber") + .HasMaxLength(32) + .HasColumnType("nvarchar(32)"); + + b.HasKey("Id"); + + b.HasIndex("NotificationListId"); + + b.ToTable("NotificationRecipients"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Notifications.SmsConfiguration", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("AccountSid") + .IsRequired() + .HasMaxLength(100) + .HasColumnType("nvarchar(100)"); + + b.Property("ApiBaseUrl") + .HasMaxLength(500) + .HasColumnType("nvarchar(500)"); + + b.Property("AuthToken") + .HasMaxLength(8000) + .HasColumnType("nvarchar(max)"); + + b.Property("ConnectionTimeoutSeconds") + .HasColumnType("int"); + + b.Property("FromNumber") + .HasMaxLength(32) + .HasColumnType("nvarchar(32)"); + + b.Property("MaxRetries") + .HasColumnType("int"); + + b.Property("MessagingServiceSid") + .HasMaxLength(100) + .HasColumnType("nvarchar(100)"); + + b.Property("RetryDelay") + .HasColumnType("time"); + + b.HasKey("Id"); + + b.ToTable("SmsConfigurations"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Notifications.SmtpConfiguration", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("AuthType") + .IsRequired() + .HasMaxLength(50) + .HasColumnType("nvarchar(50)"); + + b.Property("ConnectionTimeoutSeconds") + .HasColumnType("int"); + + b.Property("Credentials") + .HasMaxLength(8000) + .HasColumnType("nvarchar(max)"); + + b.Property("FromAddress") + .IsRequired() + .HasMaxLength(500) + .HasColumnType("nvarchar(500)"); + + b.Property("Host") + .IsRequired() + .HasMaxLength(500) + .HasColumnType("nvarchar(500)"); + + b.Property("MaxConcurrentConnections") + .HasColumnType("int"); + + b.Property("MaxRetries") + .HasColumnType("int"); + + b.Property("OAuth2Authority") + .HasMaxLength(500) + .HasColumnType("nvarchar(500)"); + + b.Property("OAuth2Scope") + .HasMaxLength(500) + .HasColumnType("nvarchar(500)"); + + b.Property("Port") + .HasColumnType("int"); + + b.Property("RetryDelay") + .HasColumnType("time"); + + b.Property("TlsMode") + .HasMaxLength(50) + .HasColumnType("nvarchar(50)"); + + b.Property("Transport") + .HasMaxLength(50) + .HasColumnType("nvarchar(50)"); + + b.HasKey("Id"); + + b.ToTable("SmtpConfigurations"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Schemas.SharedSchema", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("SchemaJson") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("Scope") + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.HasKey("Id"); + + b.HasIndex("Name") + .IsUnique(); + + b.ToTable("SharedSchemas"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Scripts.SharedScript", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("Code") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("ParameterDefinitions") + .HasMaxLength(4000) + .HasColumnType("nvarchar(4000)"); + + b.Property("ReturnDefinition") + .HasMaxLength(4000) + .HasColumnType("nvarchar(4000)"); + + b.HasKey("Id"); + + b.HasIndex("Name") + .IsUnique(); + + b.ToTable("SharedScripts"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.SecuredWrites.PendingSecuredWrite", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("bigint"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("ConnectionName") + .IsRequired() + .HasMaxLength(128) + .IsUnicode(false) + .HasColumnType("varchar(128)"); + + b.Property("DecidedAtUtc") + .HasColumnType("datetime2"); + + b.Property("ExecutedAtUtc") + .HasColumnType("datetime2"); + + b.Property("ExecutionError") + .HasMaxLength(2048) + .IsUnicode(false) + .HasColumnType("varchar(2048)"); + + b.Property("OperatorComment") + .HasMaxLength(1024) + .IsUnicode(false) + .HasColumnType("varchar(1024)"); + + b.Property("OperatorUser") + .IsRequired() + .HasMaxLength(256) + .IsUnicode(false) + .HasColumnType("varchar(256)"); + + b.Property("SiteId") + .IsRequired() + .HasMaxLength(128) + .IsUnicode(false) + .HasColumnType("varchar(128)"); + + b.Property("Status") + .IsRequired() + .HasMaxLength(32) + .IsUnicode(false) + .HasColumnType("varchar(32)"); + + b.Property("SubmittedAtUtc") + .HasColumnType("datetime2"); + + b.Property("TagPath") + .IsRequired() + .HasMaxLength(512) + .IsUnicode(false) + .HasColumnType("varchar(512)"); + + b.Property("ValueJson") + .IsRequired() + .HasMaxLength(4000) + .IsUnicode(false) + .HasColumnType("varchar(4000)"); + + b.Property("ValueType") + .IsRequired() + .HasMaxLength(128) + .IsUnicode(false) + .HasColumnType("varchar(128)"); + + b.Property("VerifierComment") + .HasMaxLength(1024) + .IsUnicode(false) + .HasColumnType("varchar(1024)"); + + b.Property("VerifierUser") + .HasMaxLength(256) + .IsUnicode(false) + .HasColumnType("varchar(256)"); + + b.HasKey("Id"); + + b.HasIndex("SiteId") + .HasDatabaseName("IX_PendingSecuredWrites_Site"); + + b.HasIndex("Status", "SubmittedAtUtc") + .HasDatabaseName("IX_PendingSecuredWrites_Status_Submitted"); + + b.ToTable("PendingSecuredWrites", (string)null); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Security.LdapGroupMapping", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("LdapGroupName") + .IsRequired() + .HasMaxLength(500) + .HasColumnType("nvarchar(500)"); + + b.Property("Role") + .IsRequired() + .HasMaxLength(100) + .HasColumnType("nvarchar(100)"); + + b.HasKey("Id"); + + b.HasIndex("LdapGroupName") + .IsUnique(); + + b.ToTable("LdapGroupMappings"); + + b.HasData( + new + { + Id = 1, + LdapGroupName = "SCADA-Admins", + Role = "Administrator" + }, + new + { + Id = 2, + LdapGroupName = "SCADA-Designers", + Role = "Designer" + }, + new + { + Id = 3, + LdapGroupName = "SCADA-Deploy-All", + Role = "Deployer" + }, + new + { + Id = 4, + LdapGroupName = "SCADA-Deploy-SiteA", + Role = "Deployer" + }, + new + { + Id = 5, + LdapGroupName = "SCADA-Viewers", + Role = "Viewer" + }); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Security.SiteScopeRule", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("LdapGroupMappingId") + .HasColumnType("int"); + + b.Property("SiteId") + .HasColumnType("int"); + + b.HasKey("Id"); + + b.HasIndex("SiteId"); + + b.HasIndex("LdapGroupMappingId", "SiteId") + .IsUnique(); + + b.ToTable("SiteScopeRules"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites.DataConnection", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("BackupConfiguration") + .HasMaxLength(4000) + .HasColumnType("nvarchar(4000)"); + + b.Property("FailoverRetryCount") + .ValueGeneratedOnAdd() + .HasColumnType("int") + .HasDefaultValue(3); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("PrimaryConfiguration") + .HasMaxLength(4000) + .HasColumnType("nvarchar(4000)"); + + b.Property("Protocol") + .IsRequired() + .HasMaxLength(50) + .HasColumnType("nvarchar(50)"); + + b.Property("SiteId") + .HasColumnType("int"); + + b.HasKey("Id"); + + b.HasIndex("SiteId", "Name") + .IsUnique(); + + b.ToTable("DataConnections"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites.Site", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("Description") + .HasMaxLength(2000) + .HasColumnType("nvarchar(2000)"); + + b.Property("GrpcNodeAAddress") + .HasMaxLength(500) + .HasColumnType("nvarchar(500)"); + + b.Property("GrpcNodeBAddress") + .HasMaxLength(500) + .HasColumnType("nvarchar(500)"); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("NodeAAddress") + .HasMaxLength(500) + .HasColumnType("nvarchar(500)"); + + b.Property("NodeBAddress") + .HasMaxLength(500) + .HasColumnType("nvarchar(500)"); + + b.Property("SiteIdentifier") + .IsRequired() + .HasMaxLength(100) + .HasColumnType("nvarchar(100)"); + + b.HasKey("Id"); + + b.HasIndex("Name") + .IsUnique(); + + b.HasIndex("SiteIdentifier") + .IsUnique(); + + b.ToTable("Sites"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.Template", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("Description") + .HasMaxLength(2000) + .HasColumnType("nvarchar(2000)"); + + b.Property("FolderId") + .HasColumnType("int"); + + b.Property("IsDerived") + .HasColumnType("bit"); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("OwnerCompositionId") + .HasColumnType("int"); + + b.Property("ParentTemplateId") + .HasColumnType("int"); + + b.HasKey("Id"); + + b.HasIndex("FolderId"); + + b.HasIndex("Name") + .IsUnique() + .HasFilter("[IsDerived] = 0"); + + b.HasIndex("ParentTemplateId"); + + b.ToTable("Templates"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.TemplateAlarm", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("Description") + .HasMaxLength(2000) + .HasColumnType("nvarchar(2000)"); + + b.Property("IsInherited") + .HasColumnType("bit"); + + b.Property("IsLocked") + .HasColumnType("bit"); + + b.Property("LockedInDerived") + .HasColumnType("bit"); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("OnTriggerScriptId") + .HasColumnType("int"); + + b.Property("PriorityLevel") + .HasColumnType("int"); + + b.Property("TemplateId") + .HasColumnType("int"); + + b.Property("TriggerConfiguration") + .HasMaxLength(4000) + .HasColumnType("nvarchar(4000)"); + + b.Property("TriggerType") + .IsRequired() + .HasMaxLength(50) + .HasColumnType("nvarchar(50)"); + + b.HasKey("Id"); + + b.HasIndex("TemplateId", "Name") + .IsUnique(); + + b.ToTable("TemplateAlarms"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.TemplateAttribute", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("DataSourceReference") + .HasMaxLength(500) + .HasColumnType("nvarchar(500)"); + + b.Property("DataType") + .IsRequired() + .HasMaxLength(50) + .HasColumnType("nvarchar(50)"); + + b.Property("Description") + .HasMaxLength(2000) + .HasColumnType("nvarchar(2000)"); + + b.Property("ElementDataType") + .HasMaxLength(50) + .HasColumnType("nvarchar(50)"); + + b.Property("IsInherited") + .HasColumnType("bit"); + + b.Property("IsLocked") + .HasColumnType("bit"); + + b.Property("LockedInDerived") + .HasColumnType("bit"); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("TemplateId") + .HasColumnType("int"); + + b.Property("Value") + .HasColumnType("nvarchar(max)"); + + b.HasKey("Id"); + + b.HasIndex("TemplateId", "Name") + .IsUnique(); + + b.ToTable("TemplateAttributes"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.TemplateComposition", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("ComposedTemplateId") + .HasColumnType("int"); + + b.Property("InstanceName") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("TemplateId") + .HasColumnType("int"); + + b.HasKey("Id"); + + b.HasIndex("ComposedTemplateId"); + + b.HasIndex("TemplateId", "InstanceName") + .IsUnique(); + + b.ToTable("TemplateCompositions"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.TemplateFolder", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("ParentFolderId") + .HasColumnType("int"); + + b.Property("SortOrder") + .HasColumnType("int"); + + b.HasKey("Id"); + + b.HasIndex("ParentFolderId", "Name") + .IsUnique() + .HasFilter("[ParentFolderId] IS NOT NULL"); + + b.ToTable("TemplateFolders"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.TemplateNativeAlarmSource", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("ConditionFilter") + .HasMaxLength(1000) + .HasColumnType("nvarchar(1000)"); + + b.Property("ConnectionName") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("Description") + .HasMaxLength(2000) + .HasColumnType("nvarchar(2000)"); + + b.Property("IsInherited") + .HasColumnType("bit"); + + b.Property("IsLocked") + .HasColumnType("bit"); + + b.Property("LockedInDerived") + .HasColumnType("bit"); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("SourceReference") + .IsRequired() + .HasMaxLength(1000) + .HasColumnType("nvarchar(1000)"); + + b.Property("TemplateId") + .HasColumnType("int"); + + b.HasKey("Id"); + + b.HasIndex("TemplateId", "Name") + .IsUnique(); + + b.ToTable("TemplateNativeAlarmSources"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.TemplateScript", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("Code") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("ExecutionTimeoutSeconds") + .HasColumnType("int"); + + b.Property("IsInherited") + .HasColumnType("bit"); + + b.Property("IsLocked") + .HasColumnType("bit"); + + b.Property("LockedInDerived") + .HasColumnType("bit"); + + b.Property("MinTimeBetweenRuns") + .HasColumnType("time"); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("ParameterDefinitions") + .HasMaxLength(4000) + .HasColumnType("nvarchar(4000)"); + + b.Property("ReturnDefinition") + .HasMaxLength(4000) + .HasColumnType("nvarchar(4000)"); + + b.Property("TemplateId") + .HasColumnType("int"); + + b.Property("TriggerConfiguration") + .HasMaxLength(4000) + .HasColumnType("nvarchar(4000)"); + + b.Property("TriggerType") + .HasMaxLength(50) + .HasColumnType("nvarchar(50)"); + + b.HasKey("Id"); + + b.HasIndex("TemplateId", "Name") + .IsUnique(); + + b.ToTable("TemplateScripts"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Entities.AuditLogRow", b => + { + b.Property("EventId") + .HasColumnType("uniqueidentifier"); + + b.Property("OccurredAtUtc") + .HasColumnType("datetime2"); + + b.Property("Action") + .IsRequired() + .HasMaxLength(64) + .IsUnicode(false) + .HasColumnType("varchar(64)"); + + b.Property("Actor") + .HasMaxLength(256) + .HasColumnType("nvarchar(256)"); + + b.Property("Channel") + .IsRequired() + .HasMaxLength(32) + .IsUnicode(false) + .HasColumnType("varchar(32)") + .HasColumnName("Category"); + + b.Property("CorrelationId") + .HasColumnType("uniqueidentifier"); + + b.Property("DetailsJson") + .HasColumnType("nvarchar(max)"); + + b.Property("ExecutionId") + .ValueGeneratedOnAddOrUpdate() + .HasColumnType("uniqueidentifier") + .HasComputedColumnSql("CAST(JSON_VALUE(DetailsJson,'$.executionId') AS uniqueidentifier)", true); + + b.Property("IngestedAtUtc") + .ValueGeneratedOnAddOrUpdate() + .HasColumnType("datetime2(7)") + .HasComputedColumnSql("CAST(SWITCHOFFSET(CAST(JSON_VALUE(DetailsJson,'$.ingestedAtUtc') AS datetimeoffset), 0) AS datetime2(7))", false); + + b.Property("Kind") + .IsRequired() + .ValueGeneratedOnAddOrUpdate() + .HasMaxLength(32) + .IsUnicode(false) + .HasColumnType("varchar(32)") + .HasComputedColumnSql("JSON_VALUE(DetailsJson,'$.kind')", true); + + b.Property("Outcome") + .IsRequired() + .HasMaxLength(16) + .IsUnicode(false) + .HasColumnType("varchar(16)"); + + b.Property("ParentExecutionId") + .ValueGeneratedOnAddOrUpdate() + .HasColumnType("uniqueidentifier") + .HasComputedColumnSql("CAST(JSON_VALUE(DetailsJson,'$.parentExecutionId') AS uniqueidentifier)", true); + + b.Property("SourceNode") + .HasMaxLength(64) + .IsUnicode(false) + .HasColumnType("varchar(64)"); + + b.Property("SourceSiteId") + .ValueGeneratedOnAddOrUpdate() + .HasMaxLength(64) + .IsUnicode(false) + .HasColumnType("varchar(64)") + .HasComputedColumnSql("JSON_VALUE(DetailsJson,'$.sourceSiteId')", true); + + b.Property("Status") + .IsRequired() + .ValueGeneratedOnAddOrUpdate() + .HasMaxLength(32) + .IsUnicode(false) + .HasColumnType("varchar(32)") + .HasComputedColumnSql("JSON_VALUE(DetailsJson,'$.status')", true); + + b.Property("Target") + .HasMaxLength(256) + .HasColumnType("nvarchar(256)"); + + b.HasKey("EventId", "OccurredAtUtc"); + + b.HasIndex("CorrelationId") + .HasDatabaseName("IX_AuditLog_CorrelationId") + .HasFilter("[CorrelationId] IS NOT NULL"); + + b.HasIndex("ExecutionId") + .HasDatabaseName("IX_AuditLog_Execution"); + + b.HasIndex("OccurredAtUtc") + .IsDescending() + .HasDatabaseName("IX_AuditLog_OccurredAtUtc"); + + b.HasIndex("ParentExecutionId") + .HasDatabaseName("IX_AuditLog_ParentExecution"); + + b.HasIndex("SourceNode", "OccurredAtUtc") + .HasDatabaseName("IX_AuditLog_Node_Occurred"); + + b.HasIndex("SourceSiteId", "OccurredAtUtc") + .IsDescending(false, true) + .HasDatabaseName("IX_AuditLog_Site_Occurred"); + + b.HasIndex("Target", "OccurredAtUtc") + .IsDescending(false, true) + .HasDatabaseName("IX_AuditLog_Target_Occurred") + .HasFilter("[Target] IS NOT NULL"); + + b.HasIndex("Channel", "Status", "OccurredAtUtc") + .IsDescending(false, false, true) + .HasDatabaseName("IX_AuditLog_Channel_Status_Occurred"); + + b.ToTable("AuditLog", (string)null); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Deployment.DeployedConfigSnapshot", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.Instance", null) + .WithMany() + .HasForeignKey("InstanceId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Deployment.DeploymentRecord", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.Instance", null) + .WithMany() + .HasForeignKey("InstanceId") + .OnDelete(DeleteBehavior.Restrict) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Deployment.PendingDeployment", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.Instance", null) + .WithMany() + .HasForeignKey("InstanceId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.ExternalSystems.ExternalSystemMethod", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.ExternalSystems.ExternalSystemDefinition", null) + .WithMany() + .HasForeignKey("ExternalSystemDefinitionId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.Area", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.Area", null) + .WithMany("Children") + .HasForeignKey("ParentAreaId") + .OnDelete(DeleteBehavior.Restrict); + + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites.Site", null) + .WithMany() + .HasForeignKey("SiteId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.Instance", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.Area", null) + .WithMany() + .HasForeignKey("AreaId") + .OnDelete(DeleteBehavior.SetNull); + + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites.Site", null) + .WithMany() + .HasForeignKey("SiteId") + .OnDelete(DeleteBehavior.Restrict) + .IsRequired(); + + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.Template", null) + .WithMany() + .HasForeignKey("TemplateId") + .OnDelete(DeleteBehavior.Restrict) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.InstanceAlarmOverride", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.Instance", null) + .WithMany("AlarmOverrides") + .HasForeignKey("InstanceId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.InstanceAttributeOverride", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.Instance", null) + .WithMany("AttributeOverrides") + .HasForeignKey("InstanceId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.InstanceConnectionBinding", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites.DataConnection", null) + .WithMany() + .HasForeignKey("DataConnectionId") + .OnDelete(DeleteBehavior.Restrict) + .IsRequired(); + + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.Instance", null) + .WithMany("ConnectionBindings") + .HasForeignKey("InstanceId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.InstanceNativeAlarmSourceOverride", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.Instance", null) + .WithMany("NativeAlarmSourceOverrides") + .HasForeignKey("InstanceId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Notifications.NotificationRecipient", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Notifications.NotificationList", null) + .WithMany("Recipients") + .HasForeignKey("NotificationListId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Security.SiteScopeRule", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Security.LdapGroupMapping", null) + .WithMany() + .HasForeignKey("LdapGroupMappingId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites.Site", null) + .WithMany() + .HasForeignKey("SiteId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites.DataConnection", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites.Site", null) + .WithMany() + .HasForeignKey("SiteId") + .OnDelete(DeleteBehavior.Restrict) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.Template", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.TemplateFolder", null) + .WithMany() + .HasForeignKey("FolderId") + .OnDelete(DeleteBehavior.Restrict); + + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.Template", null) + .WithMany() + .HasForeignKey("ParentTemplateId") + .OnDelete(DeleteBehavior.Restrict); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.TemplateAlarm", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.Template", null) + .WithMany("Alarms") + .HasForeignKey("TemplateId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.TemplateAttribute", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.Template", null) + .WithMany("Attributes") + .HasForeignKey("TemplateId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.TemplateComposition", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.Template", null) + .WithMany() + .HasForeignKey("ComposedTemplateId") + .OnDelete(DeleteBehavior.Restrict) + .IsRequired(); + + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.Template", null) + .WithMany("Compositions") + .HasForeignKey("TemplateId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.TemplateFolder", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.TemplateFolder", null) + .WithMany() + .HasForeignKey("ParentFolderId") + .OnDelete(DeleteBehavior.Restrict); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.TemplateNativeAlarmSource", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.Template", null) + .WithMany("NativeAlarmSources") + .HasForeignKey("TemplateId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.TemplateScript", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.Template", null) + .WithMany("Scripts") + .HasForeignKey("TemplateId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.Area", b => + { + b.Navigation("Children"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.Instance", b => + { + b.Navigation("AlarmOverrides"); + + b.Navigation("AttributeOverrides"); + + b.Navigation("ConnectionBindings"); + + b.Navigation("NativeAlarmSourceOverrides"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Notifications.NotificationList", b => + { + b.Navigation("Recipients"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.Template", b => + { + b.Navigation("Alarms"); + + b.Navigation("Attributes"); + + b.Navigation("Compositions"); + + b.Navigation("NativeAlarmSources"); + + b.Navigation("Scripts"); + }); +#pragma warning restore 612, 618 + } + } +} diff --git a/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/20260815004957_AlignAuditLogEventIdUniqueness.cs b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/20260815004957_AlignAuditLogEventIdUniqueness.cs new file mode 100644 index 00000000..44495515 --- /dev/null +++ b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/20260815004957_AlignAuditLogEventIdUniqueness.cs @@ -0,0 +1,93 @@ +using Microsoft.EntityFrameworkCore.Migrations; + +#nullable disable + +namespace ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Migrations +{ + /// + /// Makes dbo.AuditLog's EventId uniqueness partition-aligned by + /// dropping the non-aligned UX_AuditLog_EventId and leaving the clustered + /// PK_AuditLog (EventId, OccurredAtUtc) — already aligned on + /// ps_AuditLog_Month(OccurredAtUtc) — as the sole enforcement. + /// + /// + /// + /// Why. ALTER TABLE … SWITCH PARTITION refuses to run while a + /// non-aligned index exists on the table, so the monthly retention purge + /// (AuditLogRepository.SwitchOutPartitionAsync) had to DROP + /// UX_AuditLog_EventId, switch, and then CREATE it again — an OFFLINE + /// whole-table unique-index build, inside the switch transaction, blocking every + /// audit writer for its duration. It also opened a window in which the index that + /// backs ingest idempotency did not exist at all, and a mid-dance failure could + /// leave the live table without it until a later tick's CATCH branch repaired it. + /// With alignment there is nothing to drop, so the switch is metadata-only and the + /// purge stops competing with ingest. + /// + /// + /// Why dropping it is safe — EventId is globally unique by construction. + /// The composite key enforces uniqueness of the PAIR, not of EventId alone, so in + /// principle the same EventId could now be stored twice under two different + /// OccurredAtUtc values (in two different partitions). That cannot happen + /// here: EventId is a GUID minted ONCE at the emitting site, in the same + /// operation that stamps OccurredAtUtc, and both travel together verbatim + /// through telemetry and reconciliation — nothing downstream re-stamps either + /// field. A given EventId therefore always arrives with the same OccurredAtUtc and + /// can only ever map to one partition, which makes pair-uniqueness equivalent to + /// EventId-uniqueness for every row this system produces. GUID collision across + /// partitions is not a real risk. + /// + /// + /// The idempotency probe still seeks. Both ingest forms test + /// WHERE EventId = @id, which is the LEADING column of the clustered PK, so + /// the probe remains an index seek. The cost changes shape rather than order: it + /// becomes one seek per partition (the partition column is not in the predicate, so + /// SQL Server cannot eliminate partitions) instead of a single seek on a + /// non-partitioned index. Against a monthly scheme that is a couple of dozen + /// shallow B-tree seeks — cheap, and paid on a path that now issues one statement + /// per telemetry packet rather than one per row. + /// + /// + /// Edition note. The alternative remedy — keeping the non-aligned index and + /// rebuilding it with ONLINE = ON outside the switch transaction — requires + /// Enterprise (or Azure SQL / Developer) edition; online index rebuild is not + /// available on Standard, which this deployment does not guarantee. Alignment + /// needs no edition-specific feature and removes the rebuild entirely, so it is + /// preferred regardless of edition. + /// + /// + /// Down is a faithful reverse and recreates the index on [PRIMARY] + /// exactly as CollapseAuditLogToCanonical created it. Reverting also + /// reinstates the SWITCH incompatibility, so the purge's guarded defensive + /// DROP INDEX (retained in SwitchOutPartitionAsync for databases + /// restored from pre-alignment backups) would remove it again on the next purge. + /// The partition function/scheme (pf_AuditLog_Month / + /// ps_AuditLog_Month) and every aligned index are untouched by both + /// directions. + /// + /// + public partial class AlignAuditLogEventIdUniqueness : Migration + { + /// + protected override void Up(MigrationBuilder migrationBuilder) + { + // Raw, existence-guarded SQL rather than the scaffolded DropIndex: the + // AuditLog table is raw-SQL managed (partition scheme, persisted computed + // columns, append-only role grants), so its migrations stay explicit and + // re-runnable. The guard also lets this apply cleanly to a database whose + // index was already removed by the purge path's defensive cleanup. + migrationBuilder.Sql(@" +IF EXISTS (SELECT 1 FROM sys.indexes + WHERE name = 'UX_AuditLog_EventId' AND object_id = OBJECT_ID('dbo.AuditLog')) + DROP INDEX UX_AuditLog_EventId ON dbo.AuditLog;"); + } + + /// + protected override void Down(MigrationBuilder migrationBuilder) + { + migrationBuilder.Sql(@" +IF NOT EXISTS (SELECT 1 FROM sys.indexes + WHERE name = 'UX_AuditLog_EventId' AND object_id = OBJECT_ID('dbo.AuditLog')) + CREATE UNIQUE NONCLUSTERED INDEX UX_AuditLog_EventId ON dbo.AuditLog (EventId) ON [PRIMARY];"); + } + } +} diff --git a/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/ScadaBridgeDbContextModelSnapshot.cs b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/ScadaBridgeDbContextModelSnapshot.cs index dccfa59e..95d418ab 100644 --- a/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/ScadaBridgeDbContextModelSnapshot.cs +++ b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/ScadaBridgeDbContextModelSnapshot.cs @@ -1802,10 +1802,6 @@ namespace ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Migrations .HasDatabaseName("IX_AuditLog_CorrelationId") .HasFilter("[CorrelationId] IS NOT NULL"); - b.HasIndex("EventId") - .IsUnique() - .HasDatabaseName("UX_AuditLog_EventId"); - b.HasIndex("ExecutionId") .HasDatabaseName("IX_AuditLog_Execution"); diff --git a/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/AuditLogRepository.cs b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/AuditLogRepository.cs index a165319c..a38a6b9c 100644 --- a/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/AuditLogRepository.cs +++ b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/AuditLogRepository.cs @@ -1,5 +1,8 @@ +using System.Data; +using System.Text; using Microsoft.Data.SqlClient; using Microsoft.EntityFrameworkCore; +using Microsoft.EntityFrameworkCore.Storage; using Microsoft.Extensions.Logging; using Microsoft.Extensions.Logging.Abstractions; using ZB.MOM.WW.Audit; @@ -17,15 +20,32 @@ namespace ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Repositories; /// public class AuditLogRepository : IAuditLogRepository { - // SQL Server error numbers for duplicate-key violations on - // UX_AuditLog_EventId. 2601 is a unique-index violation; 2627 is a - // primary-key/unique-constraint violation. The IF NOT EXISTS … INSERT - // pattern has a check-then-act race window — two sessions can both pass - // the EXISTS check and then both attempt the INSERT — and the loser - // surfaces as one of these errors. Idempotency demands we swallow them. + // SQL Server error numbers for duplicate-key violations on the + // partition-aligned clustered PK_AuditLog (EventId, OccurredAtUtc). + // 2601 is a unique-index violation; 2627 is a primary-key/unique-constraint + // violation. The IF NOT EXISTS … INSERT pattern has a check-then-act race + // window — two sessions can both pass the EXISTS check and then both attempt + // the INSERT — and the loser surfaces as one of these errors. Idempotency + // demands we swallow them. private const int SqlErrorUniqueIndexViolation = 2601; private const int SqlErrorPrimaryKeyViolation = 2627; + // Rows per set-based ingest statement. Ten bound parameters per row against + // SQL Server's 2,100-parameter ceiling leaves ample headroom at 100 rows + // (1,000 parameters) while still collapsing a typical telemetry packet into + // a single round trip. Larger chunks buy little — the win is round-trip + // elimination, not statement size — and would push plan-cache churn up + // (one cached plan per distinct row count). + private const int IngestChunkRows = 100; + + // Ordinal-stable column list shared by the single-row and set-based inserts. + // The five persisted computed columns (Kind/Status/SourceSiteId/ExecutionId/ + // ParentExecutionId) plus the non-persisted IngestedAtUtc are derived + // server-side from DetailsJson and must NEVER appear here — writing a + // computed column is an error. + private const string CanonicalColumnList = + "EventId, OccurredAtUtc, Actor, Action, Outcome, Category, Target, SourceNode, CorrelationId, DetailsJson"; + private readonly ScadaBridgeDbContext _context; private readonly ILogger _logger; @@ -83,7 +103,7 @@ VALUES { // Two concurrent sessions both passed the IF NOT EXISTS check and // both attempted the INSERT — the loser raises 2601/2627 against - // UX_AuditLog_EventId. First-write-wins idempotency is already the + // the clustered PK. First-write-wins idempotency is already the // documented contract for this method, so the race outcome is // semantically a no-op. Swallow at Debug; other SqlExceptions // bubble. @@ -95,6 +115,196 @@ VALUES } } + /// + public async Task InsertManyIfNotExistsAsync( + IReadOnlyList events, + TimeSpan? commandTimeout = null, + CancellationToken ct = default) + { + ArgumentNullException.ThrowIfNull(events); + + if (events.Count == 0) + { + return 0; + } + + // De-duplicate WITHIN the packet before the statement is built. The + // set-based INSERT … SELECT … WHERE NOT EXISTS only tests rows that are + // ALREADY committed, so two copies of one EventId inside a single VALUES + // constructor would both pass the anti-semi-join and collide on the + // clustered PK — losing the whole statement to a duplicate-key fault. + // First-write-wins matches the single-row contract exactly (a later copy + // of the same EventId is by definition the same immutable append-only + // row), so keeping the first occurrence is not merely convenient, it is + // the documented semantics. + var seen = new HashSet(events.Count); + var distinct = new List(events.Count); + foreach (var evt in events) + { + if (evt is not null && seen.Add(evt.EventId)) + { + distinct.Add(evt); + } + } + + var inserted = 0; + for (var offset = 0; offset < distinct.Count; offset += IngestChunkRows) + { + var length = Math.Min(IngestChunkRows, distinct.Count - offset); + var chunk = distinct.GetRange(offset, length); + + try + { + inserted += await InsertChunkAsync(chunk, commandTimeout, ct).ConfigureAwait(false); + } + catch (SqlException ex) when ( + ex.Number == SqlErrorUniqueIndexViolation + || ex.Number == SqlErrorPrimaryKeyViolation) + { + // A CONCURRENT writer committed one of this chunk's EventIds + // between the anti-semi-join and the insert (the same + // check-then-act window the single-row path documents), and the + // whole set-based statement rolled back with it. Fall back to + // the per-row path so the rows that are genuinely new still land + // — the batch is a throughput optimisation, never a correctness + // dependency. Every row is retried, including the one that + // collided, because InsertIfNotExistsAsync swallows its own + // duplicate-key fault as a no-op. + _logger.LogDebug( + ex, + "Set-based audit ingest chunk of {Count} row(s) hit a duplicate-key violation (error {SqlErrorNumber}); falling back to per-row inserts.", + chunk.Count, + ex.Number); + + foreach (var evt in chunk) + { + await InsertIfNotExistsAsync(evt, ct).ConfigureAwait(false); + } + } + } + + return inserted; + } + + /// + /// Executes one set-based idempotent insert: a VALUES table constructor + /// anti-semi-joined against the committed rows, so a whole telemetry packet + /// costs ONE round trip instead of one IF NOT EXISTS … INSERT per row. + /// + /// + /// Raw ADO.NET (rather than ExecuteSqlInterpolated) because the + /// statement's parameter count varies with the chunk size and every parameter + /// needs an explicit : the VALUES constructor's column + /// types are inferred from the first row's parameters, so leaving a null + /// Target/SourceNode untyped would give the derived column the + /// wrong type and defeat the seek on the anti-semi-join. The command enlists + /// in the DbContext's ambient transaction when one is open — the cached + /// telemetry dual-write runs the audit insert and the SiteCalls upsert inside + /// a single transaction and both must commit or roll back together. + /// + private async Task InsertChunkAsync( + IReadOnlyList chunk, TimeSpan? commandTimeout, CancellationToken ct) + { + var sql = new StringBuilder(256 + (chunk.Count * 64)); + sql.Append("INSERT INTO dbo.AuditLog (").Append(CanonicalColumnList).Append(")\n"); + sql.Append("SELECT v.EventId, v.OccurredAtUtc, v.Actor, v.Action, v.Outcome, v.Category, "); + sql.Append("v.Target, v.SourceNode, v.CorrelationId, v.DetailsJson\n"); + sql.Append("FROM (VALUES\n"); + for (var i = 0; i < chunk.Count; i++) + { + if (i > 0) + { + sql.Append(",\n"); + } + + sql.Append(" (@e").Append(i) + .Append(",@t").Append(i) + .Append(",@a").Append(i) + .Append(",@n").Append(i) + .Append(",@o").Append(i) + .Append(",@c").Append(i) + .Append(",@g").Append(i) + .Append(",@s").Append(i) + .Append(",@r").Append(i) + .Append(",@d").Append(i) + .Append(')'); + } + + sql.Append("\n) AS v (").Append(CanonicalColumnList).Append(")\n"); + sql.Append("WHERE NOT EXISTS (SELECT 1 FROM dbo.AuditLog x WHERE x.EventId = v.EventId);"); + + var conn = _context.Database.GetDbConnection(); + var openedHere = false; + if (conn.State != ConnectionState.Open) + { + await conn.OpenAsync(ct).ConfigureAwait(false); + openedHere = true; + } + + try + { + await using var cmd = conn.CreateCommand(); + cmd.CommandText = sql.ToString(); + cmd.Transaction = _context.Database.CurrentTransaction?.GetDbTransaction(); + if (commandTimeout is { } timeout) + { + cmd.CommandTimeout = (int)timeout.TotalSeconds; + } + + for (var i = 0; i < chunk.Count; i++) + { + var evt = chunk[i]; + + // Same canonical projection as the single-row path: UTC-kind + // OccurredAtUtc, empty Actor collapses to NULL, Outcome/Category + // bound as their varchar storage form. + var occurred = DateTime.SpecifyKind(evt.OccurredAtUtc.UtcDateTime, DateTimeKind.Utc); + object actor = string.IsNullOrEmpty(evt.Actor) ? DBNull.Value : evt.Actor; + + AddParameter(cmd, "@e" + i, SqlDbType.UniqueIdentifier, size: 0, evt.EventId); + AddParameter(cmd, "@t" + i, SqlDbType.DateTime2, size: 0, occurred); + AddParameter(cmd, "@a" + i, SqlDbType.NVarChar, size: 256, actor); + AddParameter(cmd, "@n" + i, SqlDbType.VarChar, size: 64, evt.Action); + AddParameter(cmd, "@o" + i, SqlDbType.VarChar, size: 16, evt.Outcome.ToString()); + AddParameter(cmd, "@c" + i, SqlDbType.VarChar, size: 32, evt.Category); + AddParameter(cmd, "@g" + i, SqlDbType.NVarChar, size: 256, evt.Target); + AddParameter(cmd, "@s" + i, SqlDbType.VarChar, size: 64, evt.SourceNode); + AddParameter(cmd, "@r" + i, SqlDbType.UniqueIdentifier, size: 0, evt.CorrelationId); + AddParameter(cmd, "@d" + i, SqlDbType.NVarChar, size: -1, evt.DetailsJson); + } + + return await cmd.ExecuteNonQueryAsync(ct).ConfigureAwait(false); + } + finally + { + if (openedHere) + { + await conn.CloseAsync().ConfigureAwait(false); + } + } + } + + /// + /// Binds one explicitly-typed parameter. A null CLR value binds as + /// while KEEPING its declared , + /// which is what makes the VALUES constructor's derived column types stable + /// regardless of which rows happen to carry nulls. + /// + private static void AddParameter( + System.Data.Common.DbCommand cmd, string name, SqlDbType type, int size, object? value) + { + var p = (SqlParameter)cmd.CreateParameter(); + p.ParameterName = name; + p.SqlDbType = type; + if (size != 0) + { + p.Size = size; + } + + p.Value = value ?? DBNull.Value; + cmd.Parameters.Add(p); + } + /// public async Task> QueryAsync( AuditLogQueryFilter filter, AuditLogPaging paging, CancellationToken ct = default) @@ -229,13 +439,23 @@ VALUES /// public async Task SwitchOutPartitionAsync(DateTime monthBoundary, TimeSpan? commandTimeout = null, CancellationToken ct = default) { - // The drop-and-rebuild batch below runs via + // The switch batch below runs via // ExecuteSqlRaw with NO EF user-transaction — it carries its own server-side // BEGIN TRANSACTION / TRY-CATCH / ROLLBACK — so the DbContext's retrying // execution strategy (EnableRetryOnFailure) MAY auto-replay the whole batch on // a transient fault. That replay is safe: every step is IF-EXISTS / IF-NOT-EXISTS // guarded and the staging table is GUID-suffixed, so a re-run is idempotent. // + // ALIGNED UNIQUENESS (WP2.2): there is no longer an index drop/rebuild around + // the SWITCH. EventId uniqueness is now enforced solely by the clustered + // PK_AuditLog (EventId, OccurredAtUtc), which is partition-aligned on + // ps_AuditLog_Month(OccurredAtUtc) — so ALTER TABLE … SWITCH PARTITION has no + // non-aligned index to object to. The former dance dropped UX_AuditLog_EventId, + // switched, then rebuilt it OFFLINE inside the same transaction: a whole-table + // unique-index build blocking every writer for the duration of the purge, and a + // window in which the idempotency-supporting index did not exist at all. Both + // are gone. See migration AlignAuditLogEventIdUniqueness for the reasoning. + // // Maintenance timeout in whole seconds (ADO.NET CommandTimeout unit). Null leaves the // provider default in place. See AuditLogPurgeOptions.MaintenanceCommandTimeoutMinutes / // arch-review 04 S2 for why the ~30s default is unsafe for the switch-out dance. @@ -270,8 +490,13 @@ VALUES BEGIN TRY BEGIN TRANSACTION; - -- 1. Drop the non-aligned unique index. ALTER TABLE SWITCH refuses - -- to run while it exists. + -- 1. Defensive cleanup for databases created before + -- AlignAuditLogEventIdUniqueness: the migration drops the + -- non-aligned UX_AuditLog_EventId, but a database restored from an + -- older backup could still carry it and SWITCH refuses to run while + -- a non-aligned unique index exists. Dropping it here is idempotent + -- and permanent — the aligned clustered PK is the only uniqueness + -- enforcement the ingest path needs, so there is nothing to rebuild. IF EXISTS (SELECT 1 FROM sys.indexes WHERE name = 'UX_AuditLog_EventId' AND object_id = OBJECT_ID('dbo.AuditLog')) DROP INDEX UX_AuditLog_EventId ON dbo.AuditLog; @@ -313,31 +538,19 @@ VALUES -- 4. Drop staging — the rows are discarded here. This is the purge. DROP TABLE dbo.[{stagingTableName}]; - -- 5. Rebuild the non-aligned unique index. Live traffic that hit the - -- table during steps 1-4 saw composite-PK uniqueness only; from - -- here on, single-column EventId uniqueness is restored. - CREATE UNIQUE NONCLUSTERED INDEX UX_AuditLog_EventId ON dbo.AuditLog (EventId) ON [PRIMARY]; - COMMIT TRANSACTION; END TRY BEGIN CATCH IF @@TRANCOUNT > 0 ROLLBACK TRANSACTION; - -- Best-effort staging cleanup. The DROP INDEX in step 1 is now - -- rolled back (so the index is back), but the staging table from - -- step 2 may or may not survive the rollback depending on the - -- failure point. Guard the DROP so a missing staging table doesn't - -- mask the original error. + -- Best-effort staging cleanup. The staging table from step 2 may or + -- may not survive the rollback depending on the failure point. Guard + -- the DROP so a missing staging table doesn't mask the original error. + -- Nothing else needs repairing: uniqueness lives on the clustered PK, + -- which the switch never touches, so a failed purge can no longer + -- leave the live table without its idempotency enforcement. IF OBJECT_ID('dbo.[{stagingTableName}]', 'U') IS NOT NULL DROP TABLE dbo.[{stagingTableName}]; - -- Idempotent index rebuild — covers the niche case where ROLLBACK - -- failed to restore UX_AuditLog_EventId (or the failure happened - -- AFTER the COMMIT, which shouldn't be possible inside this TRY - -- but is cheap insurance). Without this, a failed switch could - -- leave the live table without its idempotency-supporting index. - IF NOT EXISTS (SELECT 1 FROM sys.indexes WHERE name = 'UX_AuditLog_EventId' AND object_id = OBJECT_ID('dbo.AuditLog')) - CREATE UNIQUE NONCLUSTERED INDEX UX_AuditLog_EventId ON dbo.AuditLog (EventId) ON [PRIMARY]; - -- Surface the original error to the caller — the purge actor logs -- and continues with the next boundary. THROW; diff --git a/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/KpiHistoryRepository.cs b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/KpiHistoryRepository.cs index b1298cfb..4f16d50c 100644 --- a/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/KpiHistoryRepository.cs +++ b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/KpiHistoryRepository.cs @@ -128,6 +128,37 @@ public sealed class KpiHistoryRepository : IKpiHistoryRepository var groups = samples.GroupBy(s => new SeriesHourKey( s.Source, s.Metric, s.Scope, s.ScopeKey, TruncateToHour(s.CapturedAtUtc))); + // Preload every rollup row already covering this window in ONE query and + // index it by series+hour (WP2.2). The predecessor issued a + // FirstOrDefaultAsync existence probe PER (series, hour) group — an N+1 + // that scaled with the metric catalogue times the lookback: a 3 h re-fold + // over ~40 series cost ~120 sequential round trips before a single row was + // written. The window is bounded by the caller's small trailing lookback + // and the rollup table holds exactly one row per series-hour, so the + // preload is a narrow range seek on IX_KpiRollupHourly_Series. + // + // Deliberately TRACKED (not a projection): the re-fold path mutates the + // existing entity in place and relies on the change tracker to emit the + // UPDATE. The dictionary's ScopeKey comparison is ordinal where the + // previous SQL predicate used the database collation; both sides are + // written from the same KpiSample.ScopeKey values, so they are + // byte-identical in practice and a residual mismatch degrades to the + // already-handled upsert-race path rather than a wrong aggregate. + var existingRollups = await _context.KpiRollupHourly + .Where(r => r.HourStartUtc >= from && r.HourStartUtc < to) + .ToListAsync(cancellationToken); + + var existingByKey = new Dictionary(existingRollups.Count); + foreach (var row in existingRollups) + { + existingByKey[new SeriesHourKey( + row.Source, + row.Metric, + row.Scope, + row.ScopeKey, + DateTime.SpecifyKind(row.HourStartUtc, DateTimeKind.Utc))] = row; + } + foreach (var group in groups) { var key = group.Key; @@ -143,18 +174,11 @@ public sealed class KpiHistoryRepository : IKpiHistoryRepository var maxValue = group.Max(s => s.Value); var sampleCount = group.Count(); - // Idempotent upsert on the unique series+hour key. The ScopeKey == key.ScopeKey - // comparison matches null against the Global-scope rows (IS NULL) exactly as the - // UNIQUE IX_KpiRollupHourly_Series index treats a null key as participating. - var existing = await _context.KpiRollupHourly.FirstOrDefaultAsync( - r => r.Source == key.Source - && r.Metric == key.Metric - && r.Scope == key.Scope - && r.ScopeKey == key.ScopeKey - && r.HourStartUtc == key.HourStartUtc, - cancellationToken); - - if (existing is null) + // Idempotent upsert on the unique series+hour key, resolved against the + // preloaded dictionary. A null ScopeKey keys the Global-scope rows exactly + // as the UNIQUE IX_KpiRollupHourly_Series index treats a null key as + // participating. + if (!existingByKey.TryGetValue(key, out var existing)) { _context.KpiRollupHourly.Add(new KpiRollupHourly { diff --git a/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/NotificationOutboxRepository.cs b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/NotificationOutboxRepository.cs index bc3690a2..90c874e1 100644 --- a/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/NotificationOutboxRepository.cs +++ b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/NotificationOutboxRepository.cs @@ -140,7 +140,14 @@ VALUES public async Task> GetDueAsync( DateTimeOffset now, int batchSize, CancellationToken cancellationToken = default) { + // AsNoTracking (WP2.2): the dispatcher mutates each row's delivery state in + // memory and persists it through UpdateAsync, which is now a targeted + // server-side ExecuteUpdate and needs no change tracker. Tracking a batch + // of notifications — each carrying an nvarchar(max) Body and TypeData — + // paid for a full snapshot copy per row plus a DetectChanges scan of the + // whole batch on every save. return await _context.Notifications + .AsNoTracking() .Where(n => n.Status == NotificationStatus.Pending || (n.Status == NotificationStatus.Retrying && n.NextAttemptAt != null @@ -150,22 +157,59 @@ VALUES .ToListAsync(cancellationToken); } - /// - public async Task GetByIdAsync(string notificationId, CancellationToken cancellationToken = default) - => await _context.Notifications.FindAsync(new object[] { notificationId }, cancellationToken); - /// public async Task UpdateAsync(Notification n, CancellationToken cancellationToken = default) { - _context.Notifications.Update(n); - await _context.SaveChangesAsync(cancellationToken); + ArgumentNullException.ThrowIfNull(n); + + // Targeted server-side UPDATE of the seven mutable delivery-state columns + // (WP2.2). The predecessor called DbSet.Update(n) + SaveChanges, which + // marks EVERY property modified and rewrites all 21 columns — including + // the immutable nvarchar(max) Body/TypeData payloads — on every single + // delivery attempt. ExecuteUpdate also bypasses the change tracker + // entirely, so it composes with the untracked GetDueAsync read. + // + // Immutable-by-contract columns (NotificationId, Type, ListName, Subject, + // Body, TypeData, Source*, Origin*, SiteEnqueuedAt, CreatedAt) are + // deliberately absent — see the interface contract: nothing in the + // notification lifecycle ever changes them, and omitting them is what + // makes the write narrow. + var status = n.Status; + var retryCount = n.RetryCount; + var lastError = n.LastError; + var resolvedTargets = n.ResolvedTargets; + var lastAttemptAt = n.LastAttemptAt; + var nextAttemptAt = n.NextAttemptAt; + var deliveredAt = n.DeliveredAt; + + var notificationId = n.NotificationId; + + await _context.Notifications + .Where(row => row.NotificationId == notificationId) + .ExecuteUpdateAsync( + setters => setters + .SetProperty(row => row.Status, status) + .SetProperty(row => row.RetryCount, retryCount) + .SetProperty(row => row.LastError, lastError) + .SetProperty(row => row.ResolvedTargets, resolvedTargets) + .SetProperty(row => row.LastAttemptAt, lastAttemptAt) + .SetProperty(row => row.NextAttemptAt, nextAttemptAt) + .SetProperty(row => row.DeliveredAt, deliveredAt), + cancellationToken); } + /// + public async Task GetByIdAsync(string notificationId, CancellationToken cancellationToken = default) + => await _context.Notifications.FindAsync(new object[] { notificationId }, cancellationToken); + /// public async Task<(IReadOnlyList Rows, int TotalCount)> QueryAsync( NotificationOutboxFilter filter, int pageNumber, int pageSize, CancellationToken cancellationToken = default) { - var query = _context.Notifications.AsQueryable(); + // AsNoTracking (WP2.2): this is the Central UI's read-only list page. The + // rows are projected to the wire and never saved, so tracking them cost a + // snapshot copy of every nvarchar(max) Body in the page for nothing. + var query = _context.Notifications.AsNoTracking().AsQueryable(); if (filter.Status is { } status) { @@ -218,8 +262,14 @@ VALUES var totalCount = await query.CountAsync(cancellationToken); + // NotificationId breaks CreatedAt ties so the OFFSET window is deterministic — + // without it two rows sharing a CreatedAt could appear on both pages or on + // neither. (This page keeps OFFSET paging rather than the sibling repos' + // keyset cursor because its contract surfaces a page number and a total + // count, neither of which a keyset cursor can express.) var rows = await query .OrderByDescending(n => n.CreatedAt) + .ThenByDescending(n => n.NotificationId) .Skip((pageNumber - 1) * pageSize) .Take(pageSize) .ToListAsync(cancellationToken); @@ -244,7 +294,24 @@ VALUES // One conditional-aggregation pass replaces four sequential COUNT round trips: // each metric is a COUNT(CASE WHEN THEN 1 END) over the same scan // (arch-review 04). GroupBy(_ => 1) yields a single group (no rows → no group). + // + // WP2.2 — the aggregation is now PREDICATE-RESTRICTED instead of scanning the + // whole table. Every KPI here is about the live queue (Pending/Retrying), the + // parked backlog, or the last delivery interval; the overwhelming bulk of the + // Notifications table is historical Delivered and Discarded rows that + // contribute to NONE of them. The pre-filter below is the union of the + // metric-contributing predicates, which lets the optimizer seek the status + // index for the live/parked legs and the filtered + // (DeliveredAt) WHERE Status='Delivered' index (WP1.4) for the interval leg, + // instead of paying a full scan whose cost grows with retained history. + // Mirrors the pre-filter ComputePerSiteKpisAsync/ComputePerNodeKpisAsync + // already use — the global snapshot was the odd one out. var counts = await _context.Notifications + .Where(n => n.Status == NotificationStatus.Pending + || n.Status == NotificationStatus.Retrying + || n.Status == NotificationStatus.Parked + || (n.Status == NotificationStatus.Delivered + && n.DeliveredAt != null && n.DeliveredAt >= deliveredSince)) .GroupBy(_ => 1) .Select(g => new { diff --git a/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/SiteCallAuditRepository.cs b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/SiteCallAuditRepository.cs index 29137a77..ac31c9e0 100644 --- a/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/SiteCallAuditRepository.cs +++ b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/SiteCallAuditRepository.cs @@ -72,49 +72,36 @@ public class SiteCallAuditRepository : ISiteCallAuditRepository var idText = siteCall.TrackedOperationId.Value.ToString("D"); var incomingRank = GetRankOrThrow(siteCall.Status); - // Step 1: insert-if-not-exists. Like AuditLogRepository.InsertIfNotExistsAsync - // this is check-then-act so a duplicate-key violation may surface under - // concurrent inserts on the same id — caught + logged at Debug. + // ONE round trip, UPDATE-first (WP2.2). The predecessor issued an + // unconditional IF NOT EXISTS … INSERT and THEN a monotonic UPDATE — two + // statements, two round trips, on every single packet, of which the insert + // half was wasted work for every packet after the first (the steady state: + // a cached call emits Submitted → Forwarded → Attempted → terminal, so + // three of four packets hit an existing row). // - // SourceNode-stamping: the column is included in the INSERT - // column list / VALUES so a fresh row carries the originating node - // name (node-a/node-b for site rows). A null SourceNode (legacy hosts - // / unstamped reconciled rows) writes NULL straight through. - try - { - await _context.Database.ExecuteSqlInterpolatedAsync( - $@"IF NOT EXISTS (SELECT 1 FROM dbo.SiteCalls WHERE TrackedOperationId = {idText}) -INSERT INTO dbo.SiteCalls - (TrackedOperationId, Channel, Target, SourceSite, SourceNode, Status, RetryCount, - LastError, HttpStatus, CreatedAtUtc, UpdatedAtUtc, TerminalAtUtc, IngestedAtUtc) -VALUES - ({idText}, {siteCall.Channel}, {siteCall.Target}, {siteCall.SourceSite}, {siteCall.SourceNode}, {siteCall.Status}, {siteCall.RetryCount}, - {siteCall.LastError}, {siteCall.HttpStatus}, {siteCall.CreatedAtUtc}, {siteCall.UpdatedAtUtc}, {siteCall.TerminalAtUtc}, {siteCall.IngestedAtUtc});", - ct); - } - catch (SqlException ex) when ( - ex.Number == SqlErrorUniqueIndexViolation - || ex.Number == SqlErrorPrimaryKeyViolation) - { - _logger.LogDebug( - ex, - "SiteCallAuditRepository.UpsertAsync swallowed duplicate-key violation (error {SqlErrorNumber}) for TrackedOperationId {TrackedOperationId}; falling through to monotonic update.", - ex.Number, - idText); - } - - // Step 2: monotonic update with a same-rank freshness tiebreaker. The - // CASE expression maps the stored Status string to the same rank table - // the caller uses. We mutate when EITHER the incoming rank is strictly - // greater, OR the incoming rank equals the stored rank AND that rank is - // non-terminal (< TerminalRank) AND the incoming UpdatedAtUtc is strictly - // newer than the stored one — so a retrying call's Attempted-phase - // RetryCount/LastError/HttpStatus stay live instead of freezing at the - // first Attempted packet. Terminal ranks are excluded from the - // tiebreaker, so a later terminal NEVER overwrites an earlier one; equal - // stamps are inert (idempotent replay) and a lower rank is always a no-op. + // The combined batch below runs UPDATE first and inserts only when the + // UPDATE matched nothing AND the row genuinely does not exist. The + // NOT EXISTS re-check is load-bearing: @@ROWCOUNT = 0 is ALSO what a + // monotonic REJECTION looks like (a stale or regressive packet against an + // existing row), and inserting there would resurrect a row the guard just + // refused. Both statements ship in one command text, so this is one + // round trip, not two. // - // SourceNode-stamping: SourceNode is updated via + // Monotonic update semantics are unchanged: mutate when EITHER the + // incoming rank is strictly greater, OR the incoming rank equals the + // stored rank AND that rank is non-terminal (< TerminalRank) AND the + // incoming UpdatedAtUtc is strictly newer than the stored one — so a + // retrying call's Attempted-phase RetryCount/LastError/HttpStatus stay + // live instead of freezing at the first Attempted packet. Terminal ranks + // are excluded from the tiebreaker, so a later terminal NEVER overwrites + // an earlier one; equal stamps are inert (idempotent replay) and a lower + // rank is always a no-op. + // + // SourceNode-stamping: the column is included in the INSERT column list / + // VALUES so a fresh row carries the originating node name (node-a/node-b + // for site rows). A null SourceNode (legacy hosts / unstamped reconciled + // rows) writes NULL straight through. On the UPDATE leg SourceNode is + // written via // COALESCE(@SourceNode, SourceNode). The operator returns @SourceNode // when it is non-null, otherwise the stored value — so the column // behaves protectively: a later packet that carries a null @@ -128,8 +115,12 @@ VALUES // lifecycle every packet should carry the same SourceNode value (one // execution, one node) so the "overwrite" path is in practice // idempotent. - await _context.Database.ExecuteSqlInterpolatedAsync( - $@"UPDATE dbo.SiteCalls + try + { + await _context.Database.ExecuteSqlInterpolatedAsync( + $@"DECLARE @updated int; + +UPDATE dbo.SiteCalls SET Status = {siteCall.Status}, RetryCount = {siteCall.RetryCount}, LastError = {siteCall.LastError}, @@ -162,8 +153,40 @@ WHERE TrackedOperationId = {idText} ELSE -1 END) AND {incomingRank} < {TerminalRank} - AND UpdatedAtUtc < {siteCall.UpdatedAtUtc} ) );", - ct); + AND UpdatedAtUtc < {siteCall.UpdatedAtUtc} ) ); + +-- Captured IMMEDIATELY after the UPDATE: @@ROWCOUNT is reset by the next +-- statement, and reading it inline inside a compound IF condition alongside a +-- subquery is not safe (the subquery's own execution can clobber it). +SET @updated = @@ROWCOUNT; + +IF @updated = 0 AND NOT EXISTS (SELECT 1 FROM dbo.SiteCalls WHERE TrackedOperationId = {idText}) +INSERT INTO dbo.SiteCalls + (TrackedOperationId, Channel, Target, SourceSite, SourceNode, Status, RetryCount, + LastError, HttpStatus, CreatedAtUtc, UpdatedAtUtc, TerminalAtUtc, IngestedAtUtc) +VALUES + ({idText}, {siteCall.Channel}, {siteCall.Target}, {siteCall.SourceSite}, {siteCall.SourceNode}, {siteCall.Status}, {siteCall.RetryCount}, + {siteCall.LastError}, {siteCall.HttpStatus}, {siteCall.CreatedAtUtc}, {siteCall.UpdatedAtUtc}, {siteCall.TerminalAtUtc}, {siteCall.IngestedAtUtc});", + ct); + } + catch (SqlException ex) when ( + ex.Number == SqlErrorUniqueIndexViolation + || ex.Number == SqlErrorPrimaryKeyViolation) + { + // Two concurrent sessions both found the row absent and both raced to + // INSERT; the loser raises 2601/2627 against the TrackedOperationId + // primary key. The winner's row IS the first-write, and this packet's + // content is by construction the same lifecycle state, so the race + // outcome is semantically a no-op. Swallow at Debug — the same + // check-then-act contract the sibling AuditLog/Notification repos + // document. Note the loser's UPDATE leg already ran (against no row), + // so nothing is left half-applied. + _logger.LogDebug( + ex, + "SiteCallAuditRepository.UpsertAsync swallowed duplicate-key violation (error {SqlErrorNumber}) for TrackedOperationId {TrackedOperationId}; treating as no-op.", + ex.Number, + idText); + } } /// diff --git a/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/ServiceCollectionExtensions.cs b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/ServiceCollectionExtensions.cs index 1752e17e..de74823e 100644 --- a/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/ServiceCollectionExtensions.cs +++ b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/ServiceCollectionExtensions.cs @@ -26,6 +26,22 @@ public static class ServiceCollectionExtensions // registers IDataProtectionProvider as a singleton; resolving it here does not recurse // because key-ring loading is lazy (first Protect/Unprotect), not triggered by // CreateProtector during model building. + // + // POOLING IS DELIBERATELY NOT USED (WP2.2 — verified, not overlooked). + // AddDbContextPool requires a context with a SINGLE public constructor taking + // only DbContextOptions; EF Core constructs pooled instances through + // its own activator and cannot supply anything else. ScadaBridgeDbContext has + // two public constructors and the runtime one takes IDataProtectionProvider, + // because the encrypting value converter for secret-bearing columns is built + // during OnModelCreating from that provider. Worse, the model itself DIFFERS + // between the two constructors (no provider ⇒ no encrypting converter), so a + // pooled activator would silently produce a context that reads secret columns + // as ciphertext. Making this poolable means moving the protector out of the + // constructor and into a DbContextOptions extension — a change to the + // secrets-at-rest path, which is not a performance refactor. The registration + // below (a scoped factory overriding AddDbContext's activator) is what makes + // the provider reach the context at all, and it also bypasses pooling by + // construction. Revisit only alongside a deliberate secrets-plumbing change. services.AddDbContext((serviceProvider, options) => { options.UseSqlServer( diff --git a/src/ZB.MOM.WW.ScadaBridge.SiteCallAudit/SiteCallAuditActor.cs b/src/ZB.MOM.WW.ScadaBridge.SiteCallAudit/SiteCallAuditActor.cs index 398c712f..eff92c94 100644 --- a/src/ZB.MOM.WW.ScadaBridge.SiteCallAudit/SiteCallAuditActor.cs +++ b/src/ZB.MOM.WW.ScadaBridge.SiteCallAudit/SiteCallAuditActor.cs @@ -158,6 +158,26 @@ public class SiteCallAuditActor : ReceiveActor /// private readonly Dictionary _reconciliationPinned = new(); + /// + /// Actor-system EventStream captured on the actor thread at handler-registration + /// time. The reconciliation pass runs off-mailbox and publishes the pinned-state + /// transition from there, so it must not reach through Context. + /// + private Akka.Event.EventStream _eventStream = null!; + + /// + /// Single-flight guard for the off-mailbox reconciliation pass. Raised on the + /// actor thread when a tick launches a pass and lowered on the actor thread + /// when the piped arrives, so the + /// per-site cursor/pinned dictionaries the pass mutates are only ever touched + /// by one task at a time and the mailbox supplies the memory barrier between + /// consecutive passes. + /// + private bool _reconciling; + + /// Single-flight guard for the off-mailbox terminal-row purge pass. + private bool _purging; + private ICancelable? _reconciliationTimer; private ICancelable? _purgeTimer; @@ -338,8 +358,79 @@ public class SiteCallAuditActor : ReceiveActor // the daily terminal-row purge. Handlers stay alive across faults via // their own per-site / per-tick try/catch (mirroring the ingest path); // the timers are only started when their collaborators are available. - ReceiveAsync(_ => OnReconciliationTickAsync()); - ReceiveAsync(_ => OnPurgeTickAsync()); + // + // OFF-MAILBOX (WP2.2). Both passes run as PipeTo-completed background + // tasks behind a single-flight guard rather than as ReceiveAsync bodies. + // A ReceiveAsync handler occupies the actor for its whole duration, and a + // reconciliation pass is unbounded work — every site, up to + // MaxReconciliationPagesPerTick network pulls each, one upsert per row. + // Post-outage catch-up therefore blocked telemetry ingest, UI queries and + // KPI Asks behind it until the drain finished, and those callers timed out + // rather than queued. NotificationOutboxActor's dispatch sweep is the + // in-repo reference for this shape. + Receive(_ => HandleReconciliationTick()); + Receive(_ => _reconciling = false); + Receive(_ => HandlePurgeTick()); + Receive(_ => _purging = false); + + // Captured on the actor thread so the background passes never touch + // Context off-thread. EventStream itself is thread-safe. + _eventStream = Context.System.EventStream; + } + + /// + /// Launches a reconciliation pass unless one is already in flight, dropping + /// the tick if so. Overlapping passes are not merely wasteful — they would + /// race on the per-site cursor and pinned-latch dictionaries, which the + /// single-flight guard keeps confined to one task at a time (the guard itself + /// is only ever mutated on the actor thread: raised here, lowered by the + /// piped completion message). + /// + private void HandleReconciliationTick() + { + if (_reconciling) + { + return; + } + + _reconciling = true; + + // OnReconciliationTickAsync swallows its own per-site errors, but the + // failure projection is kept as a belt-and-braces guard so even a faulted + // task still lowers the guard — otherwise reconciliation would wedge + // permanently after a single unexpected throw. + OnReconciliationTickAsync().PipeTo( + Self, + success: () => ReconciliationComplete.Instance, + failure: ex => + { + _logger.LogError(ex, "SiteCallAudit reconciliation pass faulted unexpectedly."); + return ReconciliationComplete.Instance; + }); + } + + /// + /// Launches a purge pass unless one is already in flight. Same single-flight + /// discipline as : a purge that outlives + /// its interval (a large catch-up after an outage) must not stack. + /// + private void HandlePurgeTick() + { + if (_purging) + { + return; + } + + _purging = true; + + OnPurgeTickAsync().PipeTo( + Self, + success: () => PurgeComplete.Instance, + failure: ex => + { + _logger.LogError(ex, "SiteCallAudit purge pass faulted unexpectedly."); + return PurgeComplete.Instance; + }); } /// @@ -743,7 +834,10 @@ public class SiteCallAuditActor : ReceiveActor } _reconciliationPinned[siteId] = pinned; - Context.System.EventStream.Publish(new SiteCallReconciliationPinnedChanged(siteId, pinned)); + + // _eventStream, not Context.System.EventStream: this runs on the + // off-mailbox reconciliation pass, where Context must not be touched. + _eventStream.Publish(new SiteCallReconciliationPinnedChanged(siteId, pinned)); } // ── Piece B: daily terminal-row purge scheduler ── @@ -1452,6 +1546,24 @@ public class SiteCallAuditActor : ReceiveActor public static readonly PurgeTick Instance = new(); private PurgeTick() { } } + + /// + /// Piped back to Self when an off-mailbox reconciliation pass ends + /// (successfully or not) so the single-flight guard is lowered on the actor + /// thread rather than from the background task. + /// + internal sealed class ReconciliationComplete + { + public static readonly ReconciliationComplete Instance = new(); + private ReconciliationComplete() { } + } + + /// Purge counterpart of . + internal sealed class PurgeComplete + { + public static readonly PurgeComplete Instance = new(); + private PurgeComplete() { } + } } /// diff --git a/tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Central/AuditLogIngestActorTests.cs b/tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Central/AuditLogIngestActorTests.cs index 356d9619..e2d2c927 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Central/AuditLogIngestActorTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Central/AuditLogIngestActorTests.cs @@ -121,6 +121,105 @@ public class AuditLogIngestActorTests : TestKit, IClassFixture { repeated, other, repeated, repeated }; + + await using var context = CreateContext(); + var repo = new AuditLogRepository(context); + var actor = CreateActor(repo); + + actor.Tell(new IngestAuditEventsCommand(batch), TestActor); + + var reply = ExpectMsg(TimeSpan.FromSeconds(10)); + Assert.Equal(4, reply.AcceptedEventIds.Count); + Assert.True( + new[] { repeated.EventId, other.EventId }.ToHashSet() + .SetEquals(reply.AcceptedEventIds.ToHashSet())); + + await using var readContext = CreateContext(); + var rows = await readContext.Set() + .Where(e => e.SourceSiteId == siteId) + .ToListAsync(); + Assert.Equal(2, rows.Count); + } + + [SkippableFact] + public async Task Receive_SamePacketTwice_IsIdempotent_AcrossPackets() + { + Skip.IfNot(_fixture.Available, _fixture.SkipReason); + + // A site whose ack was lost re-delivers the identical packet on the next + // drain, and the reconciliation pull can re-deliver it a third time. Each + // replay must ack fully (so the site can finally flip its rows to + // Forwarded) while writing nothing new. + var siteId = NewSiteId(); + var events = Enumerable.Range(0, 6).Select(_ => NewEvent(siteId)).ToList(); + + await using var context = CreateContext(); + var repo = new AuditLogRepository(context); + var actor = CreateActor(repo); + + for (var attempt = 0; attempt < 3; attempt++) + { + actor.Tell(new IngestAuditEventsCommand(events), TestActor); + var reply = ExpectMsg(TimeSpan.FromSeconds(10)); + Assert.Equal(6, reply.AcceptedEventIds.Count); + Assert.True( + events.Select(e => e.EventId).ToHashSet() + .SetEquals(reply.AcceptedEventIds.ToHashSet())); + } + + await using var readContext = CreateContext(); + var rows = await readContext.Set() + .Where(e => e.SourceSiteId == siteId) + .ToListAsync(); + Assert.Equal(6, rows.Count); + } + + [SkippableFact] + public async Task Receive_OverlappingPackets_InsertOnlyTheNewRows() + { + Skip.IfNot(_fixture.Available, _fixture.SkipReason); + + // Partially-overlapping packets are the normal reconciliation shape: the + // pull cursor re-serves a tail the push already delivered. The overlap + // must be a silent no-op and the new rows must land. + var siteId = NewSiteId(); + var first = Enumerable.Range(0, 4).Select(_ => NewEvent(siteId)).ToList(); + var second = first.Skip(2).Concat( + Enumerable.Range(0, 3).Select(_ => NewEvent(siteId))).ToList(); + + await using var context = CreateContext(); + var repo = new AuditLogRepository(context); + var actor = CreateActor(repo); + + actor.Tell(new IngestAuditEventsCommand(first), TestActor); + ExpectMsg(TimeSpan.FromSeconds(10)); + + actor.Tell(new IngestAuditEventsCommand(second), TestActor); + var reply = ExpectMsg(TimeSpan.FromSeconds(10)); + Assert.Equal(5, reply.AcceptedEventIds.Count); + + await using var readContext = CreateContext(); + var rows = await readContext.Set() + .Where(e => e.SourceSiteId == siteId) + .ToListAsync(); + Assert.Equal(7, rows.Count); + } + [SkippableFact] public async Task Receive_Sets_IngestedAtUtc_Before_Insert() { diff --git a/tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Integration/PartitionPurgeTests.cs b/tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Integration/PartitionPurgeTests.cs index 2e3a3a91..59c702e4 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Integration/PartitionPurgeTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Integration/PartitionPurgeTests.cs @@ -28,20 +28,19 @@ namespace ZB.MOM.WW.ScadaBridge.AuditLog.Tests.Integration; /// /// The oldest partition (Jan) is removed. /// Newer partitions (Feb + Mar) are untouched. -/// The UX_AuditLog_EventId unique index survives the -/// drop-and-rebuild dance. +/// The switch leaves the aligned clustered PK_AuditLog intact and +/// does NOT (re)create the non-aligned UX_AuditLog_EventId. /// remains -/// idempotent against the rebuilt index after the purge. +/// idempotent against the aligned key after the purge. /// /// /// /// The brief calls out that direct INSERTs bypass the writer role's INSERT-only /// grant; the fixture connects as sa (see /// 's default admin connection string), so -/// the seed step does not need the writer role at all. The drop-and-rebuild -/// dance itself runs under the same admin connection because the test owns -/// the database — the role granularity is exercised in the repository tests, -/// not here. +/// the seed step does not need the writer role at all. The switch batch itself +/// runs under the same admin connection because the test owns the database — +/// the role granularity is exercised in the repository tests, not here. /// public class PartitionPurgeTests : TestKit, IClassFixture { @@ -110,22 +109,39 @@ VALUES } /// - /// Asserts that UX_AuditLog_EventId exists in - /// sys.indexes. The drop-and-rebuild dance briefly removes the - /// index inside its transaction; this check is meant to fire AFTER the - /// actor's purge tick has committed so the rebuilt index is observable. + /// Asserts the post-purge index state: the partition-ALIGNED clustered + /// PK_AuditLog is intact and the non-aligned UX_AuditLog_EventId + /// is absent (WP2.2 — AlignAuditLogEventIdUniqueness). /// - private static async Task AssertUxIndexExistsAsync(SqlConnection conn) + /// + /// The purge used to bracket its SWITCH with a DROP/CREATE of + /// UX_AuditLog_EventId, because a non-aligned unique index blocks + /// ALTER TABLE … SWITCH PARTITION — an offline whole-table index build + /// inside the switch transaction, plus a window with no idempotency index at + /// all. Uniqueness now rides the aligned clustered PK, so the switch is + /// metadata-only. Asserting the index's ABSENCE is what keeps that property: + /// anything that recreates it silently reinstates the rebuild. + /// + private static async Task AssertAlignedUniquenessAsync(SqlConnection conn) { await using var cmd = conn.CreateCommand(); cmd.CommandText = @" -SELECT COUNT(*) -FROM sys.indexes -WHERE name = 'UX_AuditLog_EventId' - AND object_id = OBJECT_ID('dbo.AuditLog');"; - var raw = await cmd.ExecuteScalarAsync(); - var count = Convert.ToInt32(raw); - Assert.True(count == 1, $"UX_AuditLog_EventId should be present post-purge; sys.indexes count was {count}."); +SELECT + (SELECT COUNT(*) FROM sys.indexes + WHERE name = 'UX_AuditLog_EventId' AND object_id = OBJECT_ID('dbo.AuditLog')) AS NonAligned, + (SELECT COUNT(*) FROM sys.indexes + WHERE object_id = OBJECT_ID('dbo.AuditLog') AND is_primary_key = 1) AS ClusteredPk;"; + await using var reader = await cmd.ExecuteReaderAsync(); + Assert.True(await reader.ReadAsync()); + var nonAligned = reader.GetInt32(0); + var clusteredPk = reader.GetInt32(1); + + Assert.True( + nonAligned == 0, + $"UX_AuditLog_EventId must NOT exist post-purge (it blocks SWITCH PARTITION); sys.indexes count was {nonAligned}."); + Assert.True( + clusteredPk == 1, + $"The aligned clustered PK_AuditLog must survive the purge; sys.indexes primary-key count was {clusteredPk}."); } private IActorRef CreateActor( @@ -255,20 +271,19 @@ WHERE name = 'UX_AuditLog_EventId' } // --------------------------------------------------------------------- - // 2. EndToEnd_UxIndexRebuilt_AfterPurge + // 2. EndToEnd_AlignedUniqueness_AfterPurge // --------------------------------------------------------------------- [SkippableFact] - public async Task EndToEnd_UxIndexRebuilt_AfterPurge() + public async Task EndToEnd_AlignedUniqueness_AfterPurge() { Skip.IfNot(_fixture.Available, _fixture.SkipReason); // Same shape as test 1 — purge the Jan-2026 partition and then assert the - // UX_AuditLog_EventId index is still present. RetentionDays is computed - // dynamically so the threshold always lands near 2026-01-20 (see SeedOccurredAt()). - // The drop-and-rebuild dance briefly removes the index inside its transaction - // (the SWITCH PARTITION step requires the non-aligned unique index to be absent), - // but step 5 rebuilds it before committing. + // index state. RetentionDays is computed dynamically so the threshold always + // lands near 2026-01-20 (see SeedOccurredAt()). Since WP2.2 the switch touches + // no index at all: uniqueness rides the aligned clustered PK, so there is + // nothing to drop before the SWITCH and nothing to rebuild after it. var siteId = "purge-uxidx-" + Guid.NewGuid().ToString("N").Substring(0, 8); var oldEventId = Guid.NewGuid(); var (oldOccurred, _, _, retentionDays) = SeedOccurredAt(); @@ -305,7 +320,7 @@ WHERE name = 'UX_AuditLog_EventId' // Open a fresh connection (the actor's pool is owned by EF) and // assert the index is present post-purge. await using var check = _fixture.OpenConnection(); - await AssertUxIndexExistsAsync(check); + await AssertAlignedUniquenessAsync(check); } // --------------------------------------------------------------------- @@ -320,7 +335,7 @@ WHERE name = 'UX_AuditLog_EventId' // Seed + purge the Jan-2026 row, THEN exercise InsertIfNotExistsAsync twice for // a fresh recent EventId. The second call must be a no-op (duplicate-key collision // swallowed by the repository, per M2 Bundle A's race-fix) — which means the - // rebuilt UX_AuditLog_EventId unique index is functioning as intended. + // aligned clustered PK is still enforcing uniqueness as intended. // RetentionDays is computed dynamically so the threshold always lands near // 2026-01-20 (see SeedOccurredAt()). var siteId = "purge-idem-" + Guid.NewGuid().ToString("N").Substring(0, 8); @@ -357,11 +372,11 @@ WHERE name = 'UX_AuditLog_EventId' max: TimeSpan.FromSeconds(30)); // Settle then exercise InsertIfNotExistsAsync twice for the same - // EventId. The repository's idempotency relies on - // UX_AuditLog_EventId being present so the IF NOT EXISTS … INSERT - // race window resolves to a duplicate-key violation the repo - // swallows. If the index were missing here, two rows would land - // and the second InsertIfNotExistsAsync would silently double-insert. + // EventId. The repository's idempotency relies on the aligned clustered + // PK (EventId, OccurredAtUtc) being intact so the IF NOT EXISTS … INSERT + // race window resolves to a duplicate-key violation the repo swallows. + // If the switch had disturbed that key, two rows would land and the second + // InsertIfNotExistsAsync would silently double-insert. await Task.Delay(TimeSpan.FromMilliseconds(500)); var freshEventId = Guid.NewGuid(); @@ -482,7 +497,7 @@ WHERE name = 'UX_AuditLog_EventId' /// /// Task 4 (arch-review 04, S2): proves the explicit- maintenance-timeout /// overload of runs the real - /// drop-and-rebuild dance to completion against SQL Server — the old row is purged, the kept row + /// staging/switch batch to completion against SQL Server — the old row is purged, the kept row /// survives, and the returned sampled row-count reflects the switched partition. Exercising the /// timeout path end-to-end guards against a regression that only sets the timeout on the sample /// command and forgets the DDL batch (or vice versa). @@ -520,8 +535,8 @@ WHERE name = 'UX_AuditLog_EventId' Assert.DoesNotContain(rows, r => r.EventId == oldEventId); Assert.Contains(rows, r => r.EventId == keptEventId); - // The dance must leave the idempotency-supporting unique index rebuilt. + // The switch must leave uniqueness enforcement exactly as it found it. await using var check = _fixture.OpenConnection(); - await AssertUxIndexExistsAsync(check); + await AssertAlignedUniquenessAsync(check); } } diff --git a/tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/Configurations/AuditLogEntityTypeConfigurationTests.cs b/tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/Configurations/AuditLogEntityTypeConfigurationTests.cs index d5446b79..28ce7cf0 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/Configurations/AuditLogEntityTypeConfigurationTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/Configurations/AuditLogEntityTypeConfigurationTests.cs @@ -52,22 +52,31 @@ public class AuditLogEntityTypeConfigurationTests : IDisposable } [Fact] - public void Configure_DeclaresUniqueIndex_OnEventIdAlone_ForIdempotencyLookups() + public void Configure_DeclaresNoNonAlignedEventIdIndex_UniquenessRidesTheAlignedClusteredKey() { - // EventId remains globally unique (the idempotency key for - // InsertIfNotExistsAsync) via a dedicated unique index independent of the - // composite PK. + // WP2.2 (AlignAuditLogEventIdUniqueness): the standalone non-aligned + // UX_AuditLog_EventId is gone. EventId uniqueness now rides the clustered + // PK (EventId, OccurredAtUtc), which is partition-aligned on + // ps_AuditLog_Month — so ALTER TABLE ... SWITCH PARTITION no longer needs + // an offline index drop/rebuild around every retention purge. EventId is a + // GUID minted once at the site alongside OccurredAtUtc, so pair-uniqueness + // is EventId-uniqueness in practice, and the idempotency probe + // (WHERE EventId = @id) still seeks the clustered key's leading column. + // + // Re-declaring a single-column unique index here would silently reinstate + // the SWITCH incompatibility, so this test pins its absence. var entity = _context.Model.FindEntityType(typeof(AuditLogRow)); Assert.NotNull(entity); - var eventIdIndex = entity!.GetIndexes() - .SingleOrDefault(i => i.GetDatabaseName() == "UX_AuditLog_EventId"); + Assert.DoesNotContain( + entity!.GetIndexes(), + i => i.GetDatabaseName() == "UX_AuditLog_EventId"); - Assert.NotNull(eventIdIndex); - Assert.True(eventIdIndex!.IsUnique); - - var indexedProperty = Assert.Single(eventIdIndex.Properties); - Assert.Equal(nameof(AuditLogRow.EventId), indexedProperty.Name); + var pk = entity.FindPrimaryKey(); + Assert.NotNull(pk); + Assert.Equal( + new[] { nameof(AuditLogRow.EventId), nameof(AuditLogRow.OccurredAtUtc) }, + pk!.Properties.Select(p => p.Name).ToArray()); } [Fact] @@ -164,7 +173,8 @@ public class AuditLogEntityTypeConfigurationTests : IDisposable "IX_AuditLog_ParentExecution", "IX_AuditLog_Site_Occurred", "IX_AuditLog_Target_Occurred", - "UX_AuditLog_EventId", + // UX_AuditLog_EventId is intentionally absent — dropped by + // AlignAuditLogEventIdUniqueness; the aligned clustered PK carries it. }; Assert.Equal(expected, indexNames); diff --git a/tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/Migrations/AddAuditLogTableMigrationTests.cs b/tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/Migrations/AddAuditLogTableMigrationTests.cs index a3d7ddcc..02816828 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/Migrations/AddAuditLogTableMigrationTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/Migrations/AddAuditLogTableMigrationTests.cs @@ -85,22 +85,27 @@ public class AddAuditLogTableMigrationTests : IClassFixture - /// Verifies all nine named non-clustered indexes exist on the final - /// dbo.AuditLog table after all migrations have been applied. + /// Verifies all eight named non-clustered indexes exist on the final + /// dbo.AuditLog table after all migrations have been applied, and that + /// the non-aligned UX_AuditLog_EventId does NOT. /// The original five indexes were created by AddAuditLogTable; /// the CollapseAuditLogToCanonical (C5, Task 2.5) migration rebuilt /// the table and added IX_AuditLog_Execution, - /// IX_AuditLog_ParentExecution, IX_AuditLog_Node_Occurred, - /// and UX_AuditLog_EventId — nine in total. + /// IX_AuditLog_ParentExecution and IX_AuditLog_Node_Occurred. + /// UX_AuditLog_EventId was created alongside them but dropped again by + /// AlignAuditLogEventIdUniqueness (WP2.2) — its non-alignment forced an + /// offline drop/rebuild around every partition-switch purge, and the clustered + /// PK_AuditLog (EventId, OccurredAtUtc) already enforces the same + /// uniqueness partition-aligned. /// [SkippableFact] - public async Task AppliesMigration_CreatesNineNamedIndexes() + public async Task AppliesMigration_CreatesEightNamedIndexes_AndNoNonAlignedUniqueIndex() { Skip.IfNot(_fixture.Available, _fixture.SkipReason); - // All nine named non-clustered indexes present on dbo.AuditLog after + // All eight named non-clustered indexes present on dbo.AuditLog after // the full migration history is applied (AddAuditLogTable through - // CollapseAuditLogToCanonical). + // AlignAuditLogEventIdUniqueness). var expected = new[] { // Original five (AddAuditLogTable / AddAuditLogSourceNode): @@ -113,7 +118,6 @@ public class AddAuditLogTableMigrationTests : IClassFixture( + "SELECT COUNT(*) FROM sys.indexes i " + + "INNER JOIN sys.objects o ON i.object_id = o.object_id " + + "WHERE o.name = 'AuditLog' AND i.name = 'UX_AuditLog_EventId';"); + Assert.True( + nonAligned == 0, + "UX_AuditLog_EventId must NOT exist after AlignAuditLogEventIdUniqueness — " + + "a non-aligned index blocks ALTER TABLE ... SWITCH PARTITION and reinstates " + + $"the offline drop/rebuild the purge path was freed from; found {nonAligned}."); } [SkippableFact] diff --git a/tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/NotificationOutboxRepositoryKpiQueryShapeTests.cs b/tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/NotificationOutboxRepositoryKpiQueryShapeTests.cs index 5ca60fe6..f6fe9500 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/NotificationOutboxRepositoryKpiQueryShapeTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/NotificationOutboxRepositoryKpiQueryShapeTests.cs @@ -50,6 +50,16 @@ public class NotificationOutboxRepositoryKpiQueryShapeTests $"ComputeKpisAsync issued {counter.Count} queries against Notifications; expected <= 2"); // The oldest lookup must be bounded (LIMIT/TOP), never a full non-terminal SELECT. Assert.Contains(counter.Commands, sql => sql.Contains("LIMIT", StringComparison.OrdinalIgnoreCase)); + // WP2.2: the aggregation must be PREDICATE-RESTRICTED, not an unrestricted + // scan. Every KPI here concerns the live queue, the parked backlog or the + // last delivery interval; historical Delivered/Discarded rows — which are + // the overwhelming bulk of a retained table — contribute to none of them. + // Without the WHERE the query's cost grows with retention rather than with + // the working set. (An index-restricted aggregation was already the shape + // of the per-site/per-node snapshots; the global one was the odd one out.) + Assert.All( + counter.Commands.Where(sql => sql.Contains("COUNT", StringComparison.OrdinalIgnoreCase)), + sql => Assert.Contains("WHERE", sql, StringComparison.OrdinalIgnoreCase)); Assert.Equal(3, global.QueueDepth); Assert.NotNull(global.OldestPendingAge); diff --git a/tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/Repositories/AuditLogRepositoryTests.cs b/tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/Repositories/AuditLogRepositoryTests.cs index adda88ba..9f880125 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/Repositories/AuditLogRepositoryTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/Repositories/AuditLogRepositoryTests.cs @@ -560,6 +560,164 @@ public class AuditLogRepositoryTests : IClassFixture Assert.Equal(1, count); } + // ------------------------------------------------------------------------ + // WP2.2: set-based InsertManyIfNotExistsAsync + // ------------------------------------------------------------------------ + + [SkippableFact] + public async Task InsertManyIfNotExistsAsync_WritesEveryDistinctEvent() + { + Skip.IfNot(_fixture.Available, _fixture.SkipReason); + + var siteId = NewSiteId(); + await using var context = CreateContext(); + var repo = new AuditLogRepository(context); + + var baseTime = new DateTime(2026, 5, 21, 8, 0, 0, DateTimeKind.Utc); + var events = Enumerable.Range(0, 12) + .Select(i => NewEvent(siteId, occurredAtUtc: baseTime.AddSeconds(i))) + .ToList(); + + var inserted = await repo.InsertManyIfNotExistsAsync(events); + + Assert.Equal(12, inserted); + + await using var readContext = CreateContext(); + var rows = await readContext.Set() + .Where(e => e.SourceSiteId == siteId) + .ToListAsync(); + + Assert.Equal(12, rows.Count); + Assert.Equal( + events.Select(e => e.EventId).OrderBy(g => g).ToArray(), + rows.Select(r => r.EventId).OrderBy(g => g).ToArray()); + } + + [SkippableFact] + public async Task InsertManyIfNotExistsAsync_DuplicateEventIdsWithinOnePacket_CollapseToOneRow() + { + Skip.IfNot(_fixture.Available, _fixture.SkipReason); + + var siteId = NewSiteId(); + await using var context = CreateContext(); + var repo = new AuditLogRepository(context); + + // A set-based INSERT … SELECT … WHERE NOT EXISTS only tests rows that are + // already COMMITTED, so two copies of one EventId inside a single VALUES + // constructor would both pass the anti-semi-join and collide on the + // clustered PK — taking the whole statement down with them. The repository + // de-duplicates the packet first (first-write-wins, matching the + // single-row contract), which this pins. + var occurred = new DateTime(2026, 5, 21, 9, 0, 0, DateTimeKind.Utc); + var first = NewEvent(siteId, occurredAtUtc: occurred); + var duplicate = ScadaBridgeAuditEventFactory.Create( + channel: AuditChannel.ApiOutbound, + kind: AuditKind.ApiCall, + status: AuditStatus.Delivered, + eventId: first.EventId, + occurredAtUtc: occurred, + sourceSiteId: siteId, + errorMessage: "duplicate-within-packet-should-be-ignored"); + var other = NewEvent(siteId, occurredAtUtc: occurred.AddSeconds(1)); + + var inserted = await repo.InsertManyIfNotExistsAsync( + new[] { first, duplicate, other, duplicate }); + + Assert.Equal(2, inserted); + + await using var readContext = CreateContext(); + var rows = await readContext.Set() + .Where(e => e.SourceSiteId == siteId) + .ToListAsync(); + + Assert.Equal(2, rows.Count); + + // First-write-wins: the surviving row is the FIRST occurrence, so the + // duplicate's ErrorMessage never lands. + var stored = Assert.Single(rows, r => r.EventId == first.EventId); + Assert.Null(AuditDetailsCodec.Deserialize(stored.DetailsJson).ErrorMessage); + } + + [SkippableFact] + public async Task InsertManyIfNotExistsAsync_DuplicateEventIdsAcrossPackets_AreIdempotent() + { + Skip.IfNot(_fixture.Available, _fixture.SkipReason); + + var siteId = NewSiteId(); + await using var context = CreateContext(); + var repo = new AuditLogRepository(context); + + var baseTime = new DateTime(2026, 5, 21, 10, 0, 0, DateTimeKind.Utc); + var packet = Enumerable.Range(0, 5) + .Select(i => NewEvent(siteId, occurredAtUtc: baseTime.AddSeconds(i))) + .ToList(); + + // A site retry / reconciliation pull re-delivers a packet that overlaps + // an already-ingested one. The second call must insert only the genuinely + // new rows and silently skip the rest (first-write-wins across packets). + var firstInserted = await repo.InsertManyIfNotExistsAsync(packet); + var extra = NewEvent(siteId, occurredAtUtc: baseTime.AddSeconds(99)); + var secondInserted = await repo.InsertManyIfNotExistsAsync( + packet.Concat(new[] { extra }).ToList()); + + // A third, wholly-redundant replay writes nothing at all. + var thirdInserted = await repo.InsertManyIfNotExistsAsync(packet); + + Assert.Equal(5, firstInserted); + Assert.Equal(1, secondInserted); + Assert.Equal(0, thirdInserted); + + await using var readContext = CreateContext(); + var rows = await readContext.Set() + .Where(e => e.SourceSiteId == siteId) + .ToListAsync(); + + Assert.Equal(6, rows.Count); + } + + [SkippableFact] + public async Task InsertManyIfNotExistsAsync_ChunksBeyondTheParameterCeiling() + { + Skip.IfNot(_fixture.Available, _fixture.SkipReason); + + var siteId = NewSiteId(); + await using var context = CreateContext(); + var repo = new AuditLogRepository(context); + + // 250 rows × 10 bound parameters = 2,500 — past SQL Server's 2,100 + // parameter ceiling for a single statement, so this only succeeds if the + // repository chunks. Nulls in the optional columns (Target, SourceNode, + // CorrelationId, Actor) are left null on purpose: the VALUES constructor + // derives its column types from the parameters, so an untyped null would + // surface here as a conversion failure rather than silently. + var baseTime = new DateTime(2026, 5, 21, 11, 0, 0, DateTimeKind.Utc); + var events = Enumerable.Range(0, 250) + .Select(i => NewEvent(siteId, occurredAtUtc: baseTime.AddSeconds(i))) + .ToList(); + + var inserted = await repo.InsertManyIfNotExistsAsync(events); + + Assert.Equal(250, inserted); + + await using var readContext = CreateContext(); + var count = await readContext.Set() + .Where(e => e.SourceSiteId == siteId) + .CountAsync(); + + Assert.Equal(250, count); + } + + [SkippableFact] + public async Task InsertManyIfNotExistsAsync_EmptyBatch_IsANoOp() + { + Skip.IfNot(_fixture.Available, _fixture.SkipReason); + + await using var context = CreateContext(); + var repo = new AuditLogRepository(context); + + Assert.Equal(0, await repo.InsertManyIfNotExistsAsync(Array.Empty())); + } + [SkippableFact] public async Task QueryAsync_Keyset_SameOccurredAtUtc_TiebreaksOnEventId() { @@ -622,16 +780,17 @@ public class AuditLogRepositoryTests : IClassFixture // ------------------------------------------------------------------------ // // The partition-switch path replaces M1's NotSupportedException stub with - // the production drop-DROP-INDEX → CREATE-staging → SWITCH PARTITION → - // DROP-staging → CREATE-INDEX dance documented in alog.md §4. These tests + // the production CREATE-staging → SWITCH PARTITION → DROP-staging batch + // documented in alog.md §4. WP2.2 removed the index drop/rebuild that used to + // bracket it: uniqueness rides the partition-ALIGNED clustered PK + // (EventId, OccurredAtUtc), so SWITCH has nothing to object to. These tests // verify the side effects an outsider can observe: // * rows in the targeted month are removed // * rows in OTHER months are NOT touched - // * UX_AuditLog_EventId still exists after a successful switch + // * no non-aligned UX_AuditLog_EventId is (re)created by a switch // * InsertIfNotExistsAsync's first-write-wins idempotency still holds - // after a switch (the rebuilt index is real) - // * a thrown SqlException leaves UX_AuditLog_EventId rebuilt (the CATCH - // branch's recovery path runs) + // after a switch (the aligned key really does enforce it) + // * a thrown SqlException leaves the table intact with no orphaned staging [SkippableFact] public async Task SwitchOutPartitionAsync_OldPartition_RemovesRows_NewPartitionsKept() @@ -669,7 +828,7 @@ public class AuditLogRepositoryTests : IClassFixture } [SkippableFact] - public async Task SwitchOutPartitionAsync_RebuildsUxIndex_AfterSwitch() + public async Task SwitchOutPartitionAsync_LeavesNoNonAlignedUniqueIndex_AfterSwitch() { Skip.IfNot(_fixture.Available, _fixture.SkipReason); @@ -680,12 +839,23 @@ public class AuditLogRepositoryTests : IClassFixture // the fixture's MSSQL database) don't tread on each other. await repo.SwitchOutPartitionAsync(new DateTime(2026, 4, 1, 0, 0, 0, DateTimeKind.Utc)); + // WP2.2: the switch must NOT recreate UX_AuditLog_EventId. Re-adding it + // would reinstate the offline drop/rebuild the purge was freed from and + // block the next SWITCH until something dropped it again. await using var verifyContext = CreateContext(); var indexExists = await ScalarAsync( verifyContext, "SELECT COUNT(*) FROM sys.indexes " + "WHERE name = 'UX_AuditLog_EventId' AND object_id = OBJECT_ID('dbo.AuditLog');"); - Assert.Equal(1, indexExists); + Assert.Equal(0, indexExists); + + // The aligned clustered PK is what enforces uniqueness now, and the + // switch must leave it untouched. + var pkExists = await ScalarAsync( + verifyContext, + "SELECT COUNT(*) FROM sys.indexes " + + "WHERE name = 'PK_AuditLog' AND object_id = OBJECT_ID('dbo.AuditLog') AND is_primary_key = 1;"); + Assert.Equal(1, pkExists); } [SkippableFact] @@ -705,10 +875,11 @@ public class AuditLogRepositoryTests : IClassFixture // Switch out the June 2026 partition (different month, empty). await repo.SwitchOutPartitionAsync(new DateTime(2026, 6, 1, 0, 0, 0, DateTimeKind.Utc)); - // Re-attempting the same EventId after the switch must STILL be a no-op - // (UX_AuditLog_EventId is the index that enables idempotency; if the - // rebuild left it broken, this insert would silently produce a duplicate - // row and the count assertion below would catch it). + // Re-attempting the same EventId after the switch must STILL be a no-op. + // The idempotency probe now seeks the aligned clustered PK's leading + // column instead of a dedicated unique index; if the switch had disturbed + // that key, this insert would silently produce a duplicate row and the + // count assertion below would catch it. // C3 (Task 2.5): rebuild a sibling row with the same EventId via the factory // (ErrorMessage rides in DetailsJson, so a top-level `with` no longer applies). var dup = ScadaBridgeAuditEventFactory.Create( @@ -734,7 +905,7 @@ public class AuditLogRepositoryTests : IClassFixture } [SkippableFact] - public async Task SwitchOutPartitionAsync_PartialFailure_RebuildsUxIndex_RaisesException() + public async Task SwitchOutPartitionAsync_PartialFailure_RaisesException_LeavesNoOrphanedStaging() { Skip.IfNot(_fixture.Available, _fixture.SkipReason); @@ -745,8 +916,8 @@ public class AuditLogRepositoryTests : IClassFixture // ALTER TABLE … SWITCH refuses to move rows out of a partition that's // referenced by an FK from another table, raising msg 4928 // ("ALTER TABLE SWITCH statement failed because target table … has a - // foreign key …"). The CATCH branch then rolls back and rebuilds the - // unique index — which the assertion below verifies. + // foreign key …"). The CATCH branch then rolls back and drops the staging + // table — which the assertion below verifies. // // The probe table is uniquely named with a guid suffix so reruns of // this test inside the same fixture DB never collide. We clean it up @@ -786,14 +957,23 @@ public class AuditLogRepositoryTests : IClassFixture await cmd.ExecuteNonQueryAsync(); } - // The CATCH block in the production SQL guarantees UX_AuditLog_EventId - // is rebuilt regardless of which step failed inside the TRY. + // The CATCH block drops the GUID-suffixed staging table regardless of + // which step failed inside the TRY, so a failed purge leaves no orphaned + // AuditLog_Staging_* object behind for the next tick to trip over. await using var verifyContext = CreateContext(); + var orphanedStaging = await ScalarAsync( + verifyContext, + "SELECT COUNT(*) FROM sys.tables WHERE name LIKE 'AuditLog\\_Staging\\_%' ESCAPE '\\';"); + Assert.Equal(0, orphanedStaging); + + // And it must NOT have (re)created the non-aligned unique index: WP2.2 + // deleted that rebuild entirely, so a failed switch cannot reintroduce + // the object that blocks the NEXT switch. var indexExists = await ScalarAsync( verifyContext, "SELECT COUNT(*) FROM sys.indexes " + "WHERE name = 'UX_AuditLog_EventId' AND object_id = OBJECT_ID('dbo.AuditLog');"); - Assert.Equal(1, indexExists); + Assert.Equal(0, indexExists); } // ------------------------------------------------------------------------ diff --git a/tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/Repositories/KpiHistoryRepositoryTests.cs b/tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/Repositories/KpiHistoryRepositoryTests.cs index 195b36b5..fddf5028 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/Repositories/KpiHistoryRepositoryTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/Repositories/KpiHistoryRepositoryTests.cs @@ -325,6 +325,57 @@ public class KpiHistoryRepositoryTests Assert.Equal(2, row.SampleCount); } + [Fact] + public async Task FoldHourlyRollupsAsync_PreloadsExistingRollups_WithoutAPerSeriesHourProbe() + { + // WP2.2: the fold used to issue one FirstOrDefaultAsync existence probe per + // (series, hour) group before writing anything — an N+1 that scaled with + // the metric catalogue times the lookback window. It now preloads the whole + // window in a single query and resolves each group from a dictionary. + // + // Six distinct series across two hours = twelve groups, so the old shape + // issued twelve SELECTs against KpiRollupHourly. The new shape issues one + // (plus the KpiSample read). This asserts the count stays small and + // constant rather than tracking the group count. + var counter = new RollupSelectCountingInterceptor(); + await using var ctx = SqliteTestHelper.CreateInMemoryContext(counter); + var repo = new KpiHistoryRepository(ctx); + + var samples = new List(); + for (var series = 0; series < 6; series++) + { + foreach (var hourOffset in new[] { 0, 1 }) + { + samples.Add(Sample( + "NotificationOutbox", + "metric" + series, + "Global", + null, + value: series + hourOffset, + capturedAtUtc: Base.AddHours(hourOffset).AddMinutes(10))); + } + } + + await repo.RecordSamplesAsync(samples); + + // Seed the rollups so the SECOND fold takes the re-fold (update) path — + // the branch that most needed the per-group probe. + await repo.FoldHourlyRollupsAsync(Base, Base.AddHours(2)); + + counter.Reset(); + await repo.FoldHourlyRollupsAsync(Base, Base.AddHours(2)); + + Assert.True( + counter.RollupSelectCount <= 1, + $"expected the fold to preload existing rollups in a single query; it issued {counter.RollupSelectCount} SELECTs against KpiRollupHourly"); + + // And the fold is still correct: twelve series-hours, values unchanged by + // the re-fold. + var rollups = await ctx.KpiRollupHourly.AsNoTracking().ToListAsync(); + Assert.Equal(12, rollups.Count); + Assert.All(rollups, r => Assert.Equal(1, r.SampleCount)); + } + [Fact] public async Task GetHourlySeriesAsync_ReturnsAscending_AndHonorsNullVsSiteScopeKey() { @@ -540,6 +591,42 @@ public class KpiHistoryRepositoryTests /// async non-query entry points (ExecuteDeleteAsync routes through the /// async path). /// + /// + /// Counts reader commands that SELECT from KpiRollupHourly, so a test + /// can prove the hourly fold resolves existing rows from a single preload + /// rather than one probe per (series, hour) group. + /// + private sealed class RollupSelectCountingInterceptor : DbCommandInterceptor + { + public int RollupSelectCount { get; private set; } + + public void Reset() => RollupSelectCount = 0; + + private void CountIfRollupSelect(DbCommand command) + { + if (command.CommandText.Contains("KpiRollupHourly", StringComparison.OrdinalIgnoreCase) + && command.CommandText.Contains("SELECT", StringComparison.OrdinalIgnoreCase)) + { + RollupSelectCount++; + } + } + + public override InterceptionResult ReaderExecuting( + DbCommand command, CommandEventData eventData, InterceptionResult result) + { + CountIfRollupSelect(command); + return base.ReaderExecuting(command, eventData, result); + } + + public override ValueTask> ReaderExecutingAsync( + DbCommand command, CommandEventData eventData, InterceptionResult result, + CancellationToken cancellationToken = default) + { + CountIfRollupSelect(command); + return base.ReaderExecutingAsync(command, eventData, result, cancellationToken); + } + } + private sealed class DeleteCountingInterceptor : DbCommandInterceptor { public int DeleteCount { get; private set; } diff --git a/tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/Repositories/SiteCallAuditRepositoryTests.cs b/tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/Repositories/SiteCallAuditRepositoryTests.cs index 79569052..9244e8ae 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/Repositories/SiteCallAuditRepositoryTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/Repositories/SiteCallAuditRepositoryTests.cs @@ -96,6 +96,47 @@ public class SiteCallAuditRepositoryTests : IClassFixture Assert.Equal(attemptedSnapshot!.UpdatedAtUtc, afterStale.UpdatedAtUtc); } + [SkippableFact] + public async Task UpsertAsync_RejectedMonotonicUpdate_DoesNotFallThroughToInsert() + { + Skip.IfNot(_fixture.Available, _fixture.SkipReason); + + // WP2.2 collapsed the two-round-trip "insert-if-absent then monotonic + // update" into ONE batch that updates first and inserts only when nothing + // matched. That makes @@ROWCOUNT = 0 ambiguous: it means "no such row" + // AND "the monotonic guard rejected this packet". Guarding the insert on + // @@ROWCOUNT alone would therefore let every stale/regressive packet + // append a SECOND row for an id that already exists — silently forking + // the mirror. The re-check for the row's existence is what prevents that, + // and this pins it: after a rejected regressive upsert there is still + // exactly ONE row, still carrying the advanced state. + var id = TrackedOperationId.New(); + await using var context = CreateContext(); + var repo = new SiteCallAuditRepository(context); + + await repo.UpsertAsync(NewRow(id, status: "Delivered", retryCount: 3, lastError: null)); + + // Every flavour of rejection: lower rank, equal terminal rank, and an + // equal non-terminal rank with a stale timestamp. + await repo.UpsertAsync(NewRow(id, status: "Submitted", retryCount: 0)); + await repo.UpsertAsync(NewRow(id, status: "Parked", retryCount: 9, lastError: "should-not-apply")); + await repo.UpsertAsync(NewRow( + id, + status: "Delivered", + retryCount: 99, + updatedAtUtc: new DateTime(2020, 1, 1, 0, 0, 0, DateTimeKind.Utc))); + + await using var readContext = CreateContext(); + var loaded = await readContext.Set() + .Where(s => s.TrackedOperationId == id) + .ToListAsync(); + + Assert.Single(loaded); + Assert.Equal("Delivered", loaded[0].Status); + Assert.Equal(3, loaded[0].RetryCount); + Assert.Null(loaded[0].LastError); + } + [SkippableFact] public async Task UpsertAsync_SameStatus_EqualUpdatedAt_IsNoOp() { diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteCallAudit.Tests/SiteCallAuditReconciliationTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteCallAudit.Tests/SiteCallAuditReconciliationTests.cs index 661e885b..d64d3c8c 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteCallAudit.Tests/SiteCallAuditReconciliationTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteCallAudit.Tests/SiteCallAuditReconciliationTests.cs @@ -4,6 +4,7 @@ using Microsoft.Extensions.Logging.Abstractions; using ZB.MOM.WW.ScadaBridge.AuditLog.Central; using ZB.MOM.WW.ScadaBridge.Commons.Entities.Audit; using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories; +using ZB.MOM.WW.ScadaBridge.Commons.Messages.Audit; using ZB.MOM.WW.ScadaBridge.Commons.Messages.Integration; using ZB.MOM.WW.ScadaBridge.Commons.Types; using ZB.MOM.WW.ScadaBridge.Commons.Types.Audit; @@ -582,4 +583,122 @@ public class SiteCallAuditReconciliationTests : TestKit Assert.Equal(siteId, evt.SiteId); Assert.True(evt.Pinned, "a legacy site that ignores after_id must publish Pinned=true"); } + + // --------------------------------------------------------------------- + // 9. WP2.2: the reconciliation drain runs OFF the mailbox, so ingest, + // query and KPI messages are answered while a long post-outage + // catch-up is still pulling. + // --------------------------------------------------------------------- + + /// + /// Pull client whose first call blocks until released, simulating a + /// post-outage catch-up that takes far longer than the caller's Ask timeout. + /// + private sealed class BlockingPullClient : IPullSiteCallsClient + { + private readonly TaskCompletionSource _release = + new(TaskCreationOptions.RunContinuationsAsynchronously); + private readonly TaskCompletionSource _entered = + new(TaskCreationOptions.RunContinuationsAsynchronously); + + /// Completes once the drain has actually started pulling. + public Task Entered => _entered.Task; + + public void Release() => _release.TrySetResult(); + + public async Task PullAsync( + string siteId, DateTime sinceUtc, string? afterId, int batchSize, CancellationToken ct) + { + _entered.TrySetResult(); + await _release.Task.ConfigureAwait(false); + return new PullSiteCallsResponse(Array.Empty(), MoreAvailable: false); + } + } + + [Fact] + public async Task ReconciliationDrain_InFlight_DoesNotBlockIngestUpsert() + { + // The drain used to run inside ReceiveAsync, which occupies the actor for + // its whole duration. A post-outage catch-up (every site, many paged + // network pulls, one upsert per row) therefore parked telemetry ingest, + // UI queries and KPI Asks behind it — and those callers timed out rather + // than queued, so a slow site could make central look dead. This pins the + // fix: with the drain off-mailbox behind a single-flight guard, an ingest + // Ask completes promptly while the pull is still blocked. + var siteId = "siteSlow"; + var sites = new StaticEnumerator(new SiteEntry(siteId, "http://siteSlow:8083")); + var client = new BlockingPullClient(); + var repo = new RecordingRepo(); + + var actor = CreateActor(sites, client, repo, FastTickOptions()); + + // Wait until the drain is genuinely in flight and blocked inside PullAsync. + await client.Entered.WaitAsync(TimeSpan.FromSeconds(5)); + + // The mailbox must still be serving. A generous-but-finite budget: this + // fails at the pre-fix behaviour (the reply only arrives once the pull + // unblocks, which never happens until Release below). + var id = TrackedOperationId.New(); + var reply = await actor.Ask( + new UpsertSiteCallCommand(NewRow(id, sourceSite: siteId)), + TimeSpan.FromSeconds(3)); + + Assert.True(reply.Accepted); + Assert.Equal(id, reply.TrackedOperationId); + + // Let the drain finish so the actor shuts down cleanly. + client.Release(); + } + + [Fact] + public async Task ReconciliationTicks_DoNotOverlap_WhileADrainIsInFlight() + { + // Single-flight guard: with a 100 ms tick and a drain blocked for far + // longer, every subsequent tick must be dropped rather than starting a + // second concurrent pass — overlapping passes would race on the per-site + // cursor and pinned-latch dictionaries the drain mutates. + var siteId = "siteSlow"; + var sites = new StaticEnumerator(new SiteEntry(siteId, "http://siteSlow:8083")); + var client = new CountingBlockingPullClient(); + var repo = new RecordingRepo(); + + CreateActor(sites, client, repo, FastTickOptions()); + + await client.Entered.WaitAsync(TimeSpan.FromSeconds(5)); + + // Several tick intervals elapse while the first pass is still blocked. + await Task.Delay(TimeSpan.FromMilliseconds(600)); + + Assert.Equal(1, client.CallCount); + + client.Release(); + } + + /// + /// that also counts invocations, so a test + /// can prove no second pass started while the first was blocked. + /// + private sealed class CountingBlockingPullClient : IPullSiteCallsClient + { + private readonly TaskCompletionSource _release = + new(TaskCreationOptions.RunContinuationsAsynchronously); + private readonly TaskCompletionSource _entered = + new(TaskCreationOptions.RunContinuationsAsynchronously); + private int _callCount; + + public Task Entered => _entered.Task; + + public int CallCount => Volatile.Read(ref _callCount); + + public void Release() => _release.TrySetResult(); + + public async Task PullAsync( + string siteId, DateTime sinceUtc, string? afterId, int batchSize, CancellationToken ct) + { + Interlocked.Increment(ref _callCount); + _entered.TrySetResult(); + await _release.Task.ConfigureAwait(false); + return new PullSiteCallsResponse(Array.Empty(), MoreAvailable: false); + } + } } From 2ce0ad7ed1987af66154cde5b4db394b7b7f06a3 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Fri, 14 Aug 2026 21:10:19 -0400 Subject: [PATCH 13/31] perf(comms): alarms-only seed, capped buffers, at-least-once audit pull --- docs/requirements/Component-AuditLog.md | 38 +- docs/requirements/Component-CentralUI.md | 2 +- docs/requirements/Component-Communication.md | 23 +- .../Component-ConfigurationDatabase.md | 2 +- docs/requirements/Component-SiteRuntime.md | 2 +- .../Site/SqliteAuditWriter.cs | 70 ++- .../Interfaces/Services/ISiteAuditQueue.cs | 39 +- .../DebugView/DebugSnapshotRequest.cs | 16 +- .../Observability/ScadaBridgeTelemetry.cs | 42 ++ .../Actors/DebugStreamBridgeActor.cs | 132 ++++- .../Actors/SiteAlarmAggregatorActor.cs | 357 ++++++++++-- .../CommunicationOptions.cs | 23 + .../Grpc/SiteCommandDtoMapper.cs | 7 +- .../Grpc/SiteStreamGrpcClient.cs | 61 ++- .../Grpc/SiteStreamGrpcServer.cs | 147 +++-- .../Protos/site_command.proto | 8 + .../Protos/sitestream.proto | 17 +- .../SiteAlarmLiveCacheService.cs | 30 +- .../SiteCommandGrpc/SiteCommand.cs | 517 ++++++++++-------- .../SiteStreamGrpc/Sitestream.cs | 115 ++-- .../Actors/InstanceActor.cs | 22 +- .../Integration/OutageReconciliationTests.cs | 49 +- .../Site/SiteAuditRetentionServiceTests.cs | 5 +- .../Site/SqliteAuditWriterWriteTests.cs | 117 ++++ .../Grpc/DebugStreamBridgeActorTests.cs | 143 +++++ .../Grpc/SiteAlarmAggregatorActorTests.cs | 182 +++++- .../Grpc/SiteStreamGrpcServerTests.cs | 50 ++ .../SiteAlarmLiveCacheServiceTests.cs | 67 ++- .../SiteStreamPullAuditEventsTests.cs | 135 ++++- .../Grpc/SiteAlarmStreamEndToEndTests.cs | 5 +- 30 files changed, 1941 insertions(+), 482 deletions(-) diff --git a/docs/requirements/Component-AuditLog.md b/docs/requirements/Component-AuditLog.md index c36ef201..8796d00b 100644 --- a/docs/requirements/Component-AuditLog.md +++ b/docs/requirements/Component-AuditLog.md @@ -296,9 +296,19 @@ dispatcher. A central `SiteAuditReconciliationActor` periodically (default 5 min per site) asks each site for its oldest `Pending` row and pending count; if backlog is non-draining (e.g., telemetry actor wedged), central issues a -`PullAuditEvents(sinceUtc, batchSize)` and inserts-if-not-exists. Accepted rows -are flipped to `ForwardState = 'Reconciled'` site-side. Same self-healing -pattern as Site Call Audit's reconciliation of `SiteCalls`. +`PullAuditEvents(sinceUtc, batchSize[, afterId])` and inserts-if-not-exists. +Same self-healing pattern as Site Call Audit's reconciliation of `SiteCalls`. + +**The pull is at-least-once (WP2.3).** Serving a row is not proof that central +received it, so rows are **not** flipped when they are served. The site flips +`ForwardState = 'Reconciled'` for everything at or before the cursor carried by +the **next** pull — the only evidence central actually consumed them +(`ISiteAuditQueue.MarkReconciledUpToAsync`, run before the read so retired rows +do not consume the batch budget). A fault between the response leaving the site +and central committing it therefore re-serves the batch instead of losing it; +central dedups on `EventId`, so a re-ship is a no-op. Previously the site flipped +each served batch immediately, and a central-side fault in that window dropped +those rows permanently — `ReadPendingSinceAsync` would never return them again. **Endpoint resolution & NodeB failover.** Each pull dials the site's `NodeA` gRPC address first; if `NodeA` is blank the site's `NodeB` address becomes the @@ -312,13 +322,21 @@ mapping/unexpected fault collapses to empty without a second dial (the other node would hit the same fault). The same resolution + failover applies to Site Call Audit's `PullSiteCalls`. -> **Cursor keyset (tracked follow-up).** The `PullAuditEvents` cursor is still a -> single `sinceUtc` timestamp. Site Call Audit's pull now uses a composite -> `(UpdatedAtUtc, TrackedOperationId)` keyset to avoid a single-timestamp pin -> (see Component-SiteCallAudit.md → Reconciliation). The same keyset should be -> applied here, but it is lower urgency because the audit cursor already re-pulls -> idempotently on `EventId` — a saturated single-timestamp window re-inserts -> harmless no-ops rather than losing rows. +**Cursor keyset.** The site side of the composite `(OccurredAtUtc, EventId)` +keyset now exists (WP2.3): `PullAuditEventsRequest.after_id` (additive field 3) +mirrors `PullSiteCallsRequest.after_id`, and `ISiteAuditQueue.ReadPendingSinceAsync` +switches from the inclusive `OccurredAtUtc >= since` to a strict composite +comparison when it is set, so a burst sharing one instant drains via the id +tiebreak instead of pinning the cursor. + +> **Central still sends a bare timestamp (tracked follow-up).** +> `IPullAuditEventsClient`/`SiteAuditReconciliationActor` do not yet populate +> `after_id`, so today's cursor remains a single `sinceUtc` under the legacy +> inclusive read. Two consequences, both benign: a window whose rows all share one +> instant re-serves rather than advancing (idempotent on `EventId`, as before), and +> the rows **at** the cursor instant cannot be proven received, so they stay +> servable until a newer row moves the cursor. Wiring `after_id` at central makes +> the retirement exact; the site accepts it already. ### Central direct-write (central-originated events) diff --git a/docs/requirements/Component-CentralUI.md b/docs/requirements/Component-CentralUI.md index 747b1917..07caf5c1 100644 --- a/docs/requirements/Component-CentralUI.md +++ b/docs/requirements/Component-CentralUI.md @@ -193,7 +193,7 @@ Per-leaf alarm rendering (leaf nodes are individual conditions for native alarms - **Live updates** — the page is driven by a **transient, per-site central live alarm cache** (`ISiteAlarmLiveCache`, owned by the Communication component; see [Component-Communication](Component-Communication.md)). On site select the page subscribes to the cache; the cache runs one shared, reference-counted per-site aggregator that **seeds** from the snapshot fan-out and then stays warm on a single **site-wide, alarm-only** `SubscribeSite` gRPC stream (seed-then-stream, dedup by `(InstanceUniqueName, AlarmName, SourceReference)`). Applied deltas raise an in-process change event (mirroring `IDeploymentStatusNotifier`) that the Blazor circuit pushes to the browser via `StateHasChanged()` — no new SignalR hub. `AlarmSummaryService.BuildFromLiveAlarms` rebuilds the roll-up + rows from the cache's current alarm set. The cache is **purely in-memory on the active central node** — there is still **no persisted central alarm store**; on a NodeA↔NodeB failover the new active node re-seeds from scratch. - **View** — roll-up tiles (total active, worst severity, unacked count, per-`AlarmKind` counts) plus a flat, sortable, filterable table. Filters cover instance, `AlarmKind` (Computed / NativeOpcUa / NativeMxAccess), state, acked/unacked, severity threshold, and name search. - **Read-only** — there are no ack / shelve / suppress controls (native alarms remain read-only by design). -- **Refresh** — manual refresh button plus the 15s poll timer (mirroring the Health dashboard), now retained as a **fallback + `NotReporting` authority** behind the live cache: when the cache reports `IsLive`, the page renders live-cache state; when a stream is unhealthy, **the aggregator has died (deathwatch resets `IsLive`)**, or a site has not yet seeded, the poll keeps the page fresh so a stream failure never blanks it. When live, the poll updates only the `NotReporting` list and leaves the row set to the delta path, so a slow fan-out can never momentarily revert a fresher live delta (R2 N5). (Aggregated live stream **delivered 2026-07-10** — see `docs/plans/2026-07-10-aggregated-live-alarm-stream-plan.md`.) +- **Refresh** — manual refresh button plus the 15s poll timer (mirroring the Health dashboard), now retained as a **fallback + `NotReporting` authority** behind the live cache: when the cache reports `IsLive`, the page renders live-cache state; when a stream is unhealthy, **the aggregator has died (deathwatch resets `IsLive`)**, or a site has not yet seeded, the poll keeps the page fresh so a stream failure never blanks it. Since WP2.3 `IsLive` also tracks the *stream* itself: a site-wide stream that faults or ends gracefully drops `IsLive` on the spot, so the page falls back to polling for the reopen window instead of rendering a snapshot that has quietly stopped updating. When live, the poll updates only the `NotReporting` list and leaves the row set to the delta path, so a slow fan-out can never momentarily revert a fresher live delta (R2 N5). (Aggregated live stream **delivered 2026-07-10** — see `docs/plans/2026-07-10-aggregated-live-alarm-stream-plan.md`.) - **Reuse** — the alarm badge/formatter markup is factored out of Debug View into a shared `AlarmStateBadges` component consumed by both Debug View and this page. ### Parked Message Management (Deployment Role) diff --git a/docs/requirements/Component-Communication.md b/docs/requirements/Component-Communication.md index cee3f326..b232c97e 100644 --- a/docs/requirements/Component-Communication.md +++ b/docs/requirements/Component-Communication.md @@ -66,18 +66,23 @@ Both central and site clusters. Each side has communication actors that handle m - Central sends an unsubscribe request over `SiteCommandService` (gRPC command/control) when the debug session ends. The gRPC stream is cancelled. The site's `StreamRelayActor` is stopped and the SiteStreamManager subscription is removed. - The stream is session-based and temporary. - **Accepted limitation (central-side session locality):** debug sessions are process-local state on the central node hosting the `DebugStreamBridgeActor`. A central **restart or failover drops all active sessions** with no `OnStreamTerminated` signal to the operator — the engineer simply re-establishes the debug session from the UI. Debug streaming is an interactive, transient diagnostic aid, so this is an accepted trade-off rather than a durability gap (arch review 02, U7). -- **Backpressure is lossy-by-design and now observable:** the per-session `Channel` (bounded 1000, `DropOldest`) silently evicts the oldest event when a slow consumer falls behind. The eviction is counted via the channel's `itemDropped` callback and logged at Warning (first eviction, then every 500th) so real event loss on a debug stream is visible instead of silent. +- **Backpressure is lossy-by-design and now observable:** the per-session `Channel` (bounded `GrpcInstanceStreamChannelCapacity`, default 1000, `DropOldest`) silently evicts the oldest event when a slow consumer falls behind. The eviction is counted via the channel's `itemDropped` callback, logged at Warning (first eviction, then every 500th) and exported as `scadabridge.site.stream.events_dropped` (tagged `stream=instance`), so real event loss on a debug stream is visible instead of silent. The **site-wide alarm feed has its own, larger channel** — see §6.1. +- **Bridge-session hardening (WP2.3):** + - The pre-snapshot buffer is **bounded (20 000 events, drop-oldest)** and its evictions counted (`scadabridge.central.debug_view.presnapshot_dropped`). It was previously unbounded, so a session whose snapshot never arrived grew without limit on the CENTRAL node. Dropping the oldest is correct here: the snapshot that ends the buffering phase is authoritative for anything that old. + - A **hard snapshot deadline** (`DebugStreamBridgeActor.SnapshotTimeout`, 60 s) fails the session if no `DebugViewSnapshot` arrives. Nothing else ended a session wedged in the buffering phase — a lost site reply raises no gRPC error. + - **Stream events no longer influence the orphan receive timeout.** The gRPC callback wraps each event in an envelope marked `INotInfluenceReceiveTimeout`, so a busy site can no longer keep an abandoned session alive forever by feeding it events. The 5-minute timeout now measures session/consumer liveness, which is what it was for. ### 6.1 Aggregated Live Alarm Stream (Site → Central) Delivered 2026-07-10 (`docs/plans/2026-07-10-aggregated-live-alarm-stream-plan.md`) for the operator Alarm Summary page (Central UI #9). Where §6 streams **one instance** for the interactive Debug View, this feeds a **transient, per-site central live alarm cache** so the Alarm Summary reflects alarm transitions in near-real-time instead of re-polling the whole site every 15s. - **Site-wide, alarm-only stream**: a single `SubscribeSite` gRPC stream per site carries `AlarmStateChanged` events for **all** instances (attribute events are dropped — the summary never shows them, and attributes are far higher-volume). On the site this reuses the existing `SiteStreamManager` broadcast hub via `SiteStreamManager.SubscribeSiteAlarms(...)` (same wiring as the per-instance `Subscribe`, minus the `InstanceUniqueName` filter and filtered to alarms). `StreamRelayActor` is reused unchanged, so placeholder rows (`is_configured_placeholder`) are still dropped at the relay. -- **Central live cache** (`ISiteAlarmLiveCache`, singleton `SiteAlarmLiveCacheService`): a DI singleton on the active central node. For each site with ≥1 active viewer it runs ONE shared, **reference-counted** per-site aggregator (`SiteAlarmAggregatorActor`); the first `Subscribe(siteId, onChanged)` starts it, the last subscriber leaving stops it after a short **linger** to avoid re-seed thrash. `GetCurrentAlarms(siteId)` returns the current immutable snapshot; `IsLive(siteId)` reports whether the aggregator has seeded and published at least once. +- **Central live cache** (`ISiteAlarmLiveCache`, singleton `SiteAlarmLiveCacheService`): a DI singleton on the active central node. For each site with ≥1 active viewer it runs ONE shared, **reference-counted** per-site aggregator (`SiteAlarmAggregatorActor`); the first `Subscribe(siteId, onChanged)` starts it, the last subscriber leaving stops it after a short **linger** to avoid re-seed thrash. `GetCurrentAlarms(siteId)` returns the current immutable snapshot; `IsLive(siteId)` reports whether the aggregator has seeded **and its site-wide stream is currently up**. Liveness rides every publish from the aggregator, so a stream that faults or ends gracefully drops `IsLive` immediately rather than leaving the page grafting a freezing snapshot over fresh poll data until the next reconcile (WP2.3). - **Seed-then-stream** (copied from `DebugStreamBridgeActor` ordering): open the `SubscribeSite` stream first (buffer live deltas), run the snapshot fan-out once via the existing `DebugViewSnapshot` path (bounded by `LiveAlarmCacheSeedConcurrency`), flush the buffer with **dedup by `(InstanceUniqueName, AlarmName, SourceReference)`**, then live pass-through. Placeholders are seeded from the snapshot and never expected on the live stream. -- **Failover & drift**: NodeA↔NodeB reconnect (client via `SiteStreamGrpcClient.SubscribeSiteAsync`, factory endpoint failover) re-seeds rather than serving stale; a periodic **reconcile snapshot** (default 60s) corrects any missed enable/disable so the cache can't drift indefinitely. `[PERM]` (`docs/plans/2026-05-29-native-alarms-design.md`): the cache is **purely in-memory** — no EF entity/table/migration, no persisted central alarm store — so a new active node simply re-seeds from scratch. -- **Options** (`Communication` section, `CommunicationOptions`; eagerly validated by `CommunicationOptionsValidator` / `ValidateOnStart`): `LiveAlarmCacheLinger` (default 30s), `LiveAlarmCacheReconcileInterval` (default 60s), `LiveAlarmCacheSeedConcurrency` (default 8), `LiveAlarmCacheMaxSubscribersPerSite` (default 200), `LiveAlarmCachePublishCoalesce` (default 250ms; `0` = publish per delta — legacy — batches an alarm storm into one snapshot copy + one viewer fan-out per window; arch review 02 round 2, N6). -- **Telemetry** (`ScadaBridgeTelemetry` meter): observable gauge `scadabridge.site.alarm_cache.aggregators.active` (running per-site aggregators) and counter `scadabridge.site.alarm_cache.reconnects` (site-wide stream reconnects — a NodeA↔NodeB flip or reconcile-driven reopen; a sustained climb signals a flapping site link). +- **Alarms-only seed (WP2.3)**: the seed/reconcile fan-out sets `DebugSnapshotRequest.AlarmsOnly` (wire: `DebugSnapshotRequestDto.alarms_only`, field 3, additive), so the site builds and ships only the alarm half of the snapshot. The fan-out discarded every attribute row anyway, and an instance's attribute surface dwarfs its alarm set. A pre-WP2.3 site ignores the flag and returns the full snapshot, which reads identically. +- **Failover & drift**: a re-seed runs **once per successful (re)connect**, not once per reconnect ATTEMPT — the connect signal is `SiteStreamGrpcClient.SubscribeSiteAsync`'s `onConnected` callback, raised when the site's response headers arrive (the site flushes them as soon as its relay is attached, so no event can be missed after it). Fanning a whole-site snapshot out per retry meant N snapshots against a site that was, by definition of the retry, unreachable. A periodic **reconcile snapshot** (default 60s, **jittered** by `LiveAlarmCacheReconcileJitterFraction` so aggregators started together do not stampede one boundary) remains the drift backstop, but it is **skipped when a fan-out already ran in that window** and **publishes only when the snapshot actually changed the cache** (a diff, not an unconditional viewer fan-out). Staleness stays bounded at two intervals: the skip consumes its flag, so the next tick always fans out. A fan-out that fails as a whole now retries on its own **backoff** timer (`reconnectDelay` doubling, capped at 8× the reconcile interval) instead of waiting a full interval. `[PERM]` (`docs/plans/2026-05-29-native-alarms-design.md`): the cache is **purely in-memory** — no EF entity/table/migration, no persisted central alarm store — so a new active node simply re-seeds from scratch. +- **Options** (`Communication` section, `CommunicationOptions`; eagerly validated by `CommunicationOptionsValidator` / `ValidateOnStart`): `LiveAlarmCacheLinger` (default 30s), `LiveAlarmCacheReconcileInterval` (default 60s), `LiveAlarmCacheSeedConcurrency` (default 8), `LiveAlarmCacheMaxSubscribersPerSite` (default 200), `LiveAlarmCachePublishCoalesce` (default 250ms; `0` = publish per delta — legacy — batches an alarm storm into one snapshot copy + one viewer fan-out per window; arch review 02 round 2, N6), `LiveAlarmCacheReconcileJitterFraction` (default 0.2 = up to +20% per tick; `0` disables). Stream channel sizing lives alongside the other gRPC limits: `GrpcInstanceStreamChannelCapacity` (default 1000) and `GrpcSiteAlarmStreamChannelCapacity` (default 20 000). +- **Telemetry** (`ScadaBridgeTelemetry` meter): observable gauge `scadabridge.site.alarm_cache.aggregators.active` (running per-site aggregators) and counter `scadabridge.site.alarm_cache.reconnects` (site-wide stream reconnects — a NodeA↔NodeB flip or reconcile-driven reopen; a sustained climb signals a flapping site link), and counter `scadabridge.site.alarm_cache.buffer_dropped` (deltas evicted from the aggregator's **bounded** 20 000-entry pre-seed buffer, drop-oldest — non-zero means a fan-out ran long enough for the delta storm behind it to exceed the cap; the fan-out's snapshot is authoritative for the evicted rows). - **Accepted limitations (arch review 02 round 2, N8):** - **Standby-node aggregators**: the live cache is per-node and `SetActorSystem` is wired on every central node, so browsing the standby node directly (diagnostic ports, e.g. 9002) starts a second, fully-functional read-only aggregator + `SubscribeSite` stream there. Accepted — not gated: the aggregator is read-only, bounded (one stream/site, viewer-capped), and torn down by the viewer linger; gating `SetActorSystem` behind the active check would break the diagnostic-browse path and buy nothing. The `[PERM]` "lives only on the active central node" claim is corrected to "per-node; in routine operation only the active node hosts viewers". - **Site deleted while an Alarm Summary viewer is open**: the viewer's aggregator reconciles to an empty snapshot (the deleted site's instances vanish from the fan-out) until its viewers leave, then the linger stop reaps it; the site's cached gRPC channels are disposed at deletion via `SiteStreamGrpcClientFactory.RemoveSiteAsync` (see `ManagementActor.HandleDeleteSite`, arch review 02 round 2, N8). @@ -101,12 +106,12 @@ The streaming protocol is defined in `sitestream.proto` (`src/ZB.MOM.WW.ScadaBri - **Service**: `SiteStreamService` — hosted on each site node by `SiteStreamGrpcServer` — exposes six RPCs. Two are real-time **server-streaming** subscriptions (`SubscribeInstance`, `SubscribeSite`); the other four are **unary request/response** calls added by the Audit Log (#23) and Site Call Audit (#22) components. These are distinct from the command/control gRPC services (`CentralControlService` on central, `SiteCommandService` on the site) — `SiteStreamService` is the streaming + audit-pull surface, not the command/control channel: - `SubscribeInstance(InstanceStreamRequest) returns (stream SiteStreamEvent)` — the per-instance real-time debug stream (§6). - - `SubscribeSite(SiteStreamRequest) returns (stream SiteStreamEvent)` — the **site-wide, alarm-only** aggregated stream (§6.1) added for the operator Alarm Summary live cache. Additive (new RPC + new `SiteStreamRequest { correlation_id }` message; no field renumbering). The server handler mirrors `SubscribeInstance` — same bounded `Channel(1000, DropOldest)`, `GrpcMaxConcurrentStreams`/`GrpcMaxStreamLifetime` limits, `SiteConnectionOpened/Closed` telemetry, and `StreamRelayActor` mapping — but subscribes via `SiteStreamManager.SubscribeSiteAlarms` (all instances, `AlarmStateChanged` only; attribute events dropped, no per-instance filter). + - `SubscribeSite(SiteStreamRequest) returns (stream SiteStreamEvent)` — the **site-wide, alarm-only** aggregated stream (§6.1) added for the operator Alarm Summary live cache. Additive (new RPC + new `SiteStreamRequest { correlation_id }` message; no field renumbering). The server handler mirrors `SubscribeInstance` — but with **its own, larger send channel** (`GrpcSiteAlarmStreamChannelCapacity`, default 20 000, `DropOldest`; WP2.3): sharing the Debug View's 1000-slot channel meant an alarm burst during a WAN stall silently evicted operator-visible transitions to make room for diagnostics traffic. Drops are tagged `stream=site-alarms` on `scadabridge.site.stream.events_dropped`. Otherwise identical — same `GrpcMaxConcurrentStreams`/`GrpcMaxStreamLifetime` limits, `SiteConnectionOpened/Closed` telemetry, and `StreamRelayActor` mapping — but subscribes via `SiteStreamManager.SubscribeSiteAlarms` (all instances, `AlarmStateChanged` only; attribute events dropped, no per-instance filter). - `IngestAuditEvents(AuditEventBatch) returns (IngestAck)` — a legacy central-side ingest surface on the *site*-hosted service; it is **dead in the shipped topology** because no site ever dials a site for ingest. The production audit-telemetry *push* path is site→central gRPC to `CentralControlService`, which routes the batch to the central `AuditLogIngestActor` proxy and returns the accepted `EventId`s. - `IngestCachedTelemetry(CachedTelemetryBatch) returns (IngestAck)` — ingest receiving surface for the combined cached-call telemetry packet (audit row + `SiteCalls` operational upsert written in one transaction). - - `PullAuditEvents(PullAuditEventsRequest) returns (PullAuditEventsResponse)` — central→site **reconciliation pull** for the Audit Log self-heal feed; the site serves `Pending`/`Forwarded` rows from its `ISiteAuditQueue`. + - `PullAuditEvents(PullAuditEventsRequest) returns (PullAuditEventsResponse)` — central→site **reconciliation pull** for the Audit Log self-heal feed; the site serves `Pending`/`Forwarded` rows from its `ISiteAuditQueue`. **At-least-once (WP2.3):** rows are NOT retired when served. The site retires (`Reconciled`) everything at or before the cursor of the **next** pull, because that cursor is central's only proof of receipt — a fault between the response leaving the site and central committing it re-serves the rows instead of losing them (central dedups on `EventId`). The request carries an additive composite-keyset `after_id` (field 3) mirroring `PullSiteCallsRequest`; with it the read is a strict `(OccurredAtUtc, EventId)` keyset and the retirement is exact, without it the legacy inclusive `>=` read applies and only rows strictly older than the cursor instant are provably received. - `PullSiteCalls(PullSiteCallsRequest) returns (PullSiteCallsResponse)` — central→site reconciliation pull for the Site Call Audit (#22) self-heal feed; the site serves operation-tracking rows changed since a cursor from its `IOperationTrackingStore`. A separate RPC from `PullAuditEvents` because the tracking store is the operational source of truth, distinct from the site audit queue. -- **Messages**: `InstanceStreamRequest` (correlation_id, instance_unique_name), `SiteStreamRequest` (correlation_id only — no instance name; drives the site-wide `SubscribeSite` stream), `SiteStreamEvent` (correlation_id, oneof event: `AttributeValueUpdate`, `AlarmStateUpdate`); `AuditEventDto`/`AuditEventBatch`/`IngestAck` for ingest; `CachedTelemetryPacket`/`CachedTelemetryBatch` (each packet pairing an `AuditEventDto` with a `SiteCallOperationalDto`); `PullAuditEventsRequest`/`PullAuditEventsResponse` and `PullSiteCallsRequest`/`PullSiteCallsResponse` (each request carries `since_utc` + `batch_size`; each response carries `more_available` to signal a saturated batch). +- **Messages**: `InstanceStreamRequest` (correlation_id, instance_unique_name), `SiteStreamRequest` (correlation_id only — no instance name; drives the site-wide `SubscribeSite` stream), `SiteStreamEvent` (correlation_id, oneof event: `AttributeValueUpdate`, `AlarmStateUpdate`); `AuditEventDto`/`AuditEventBatch`/`IngestAck` for ingest; `CachedTelemetryPacket`/`CachedTelemetryBatch` (each packet pairing an `AuditEventDto` with a `SiteCallOperationalDto`); `PullAuditEventsRequest`/`PullAuditEventsResponse` and `PullSiteCallsRequest`/`PullSiteCallsResponse` (each request carries `since_utc` + `batch_size` + an optional composite-keyset `after_id`; each response carries `more_available` to signal a saturated batch). - The `oneof event` pattern is extensible — future event types (health metrics, connection state changes) are added as new fields without breaking existing consumers. - Proto field numbers are never reused; new RPCs and message fields are appended additively. Old clients ignore unknown `oneof` variants. @@ -162,7 +167,7 @@ read the traffic". TLS is follow-on hardening and does not change this design. - **Placeholder rows are dropped at the relay**: `is_configured_placeholder` (field 23) is a **Debug View snapshot-only** concept emitted by `InstanceActor.BuildAlarmStatesSnapshot` for quiet bindings — it is never a real alarm transition (its timestamp may be `DateTimeOffset.MinValue`, the Protobuf `Timestamp` lower boundary). `StreamRelayActor.HandleAlarmStateChanged` therefore returns early — **never relaying a placeholder row to the live gRPC stream** — so field 23 is always `false` on the live stream and only ever carries `true` in the snapshot path. - **Client-side mapping (`SiteStreamGrpcClient.ConvertToDomainEvent`)**: reconstructs the domain `AlarmStateChanged` from the proto — `Kind` is parsed via `ParseAlarmKind`, the `Condition` is rebuilt with `severity` taken from the existing wire `priority`, and native metadata is repopulated from fields 8–23 (`native_source_canonical_name` → `NativeSourceCanonicalName`, `is_configured_placeholder` → `IsConfiguredPlaceholder`) — so central-side consumers receive the same domain event the site emitted. -> **Regeneration is manual (macOS-only).** `sitestream.proto` is **not** auto-compiled: the `` include is commented out in the `.csproj`, and the generated C# is **vendored** under `SiteStreamGrpc/`. To regenerate after editing the proto: toggle the `` include on, build so `Grpc.Tools` regenerates the C#, copy the generated files into `SiteStreamGrpc/`, then re-comment the include. Adding `AlarmStateUpdate` fields 8–23 and the four unary RPCs (`IngestAuditEvents`, `IngestCachedTelemetry`, `PullAuditEvents`, `PullSiteCalls`) plus their message types followed this process. +> **Regeneration is manual (macOS-only).** `sitestream.proto` is **not** auto-compiled: the `` include is commented out in the `.csproj`, and the generated C# is **vendored** under `SiteStreamGrpc/`. To regenerate after editing the proto: toggle the `` include on, build so `Grpc.Tools` regenerates the C#, copy the generated files into `SiteStreamGrpc/`, then re-comment the include. Adding `AlarmStateUpdate` fields 8–23 and the four unary RPCs (`IngestAuditEvents`, `IngestCachedTelemetry`, `PullAuditEvents`, `PullSiteCalls`) plus their message types followed this process, as did WP2.3's two additive fields — `PullAuditEventsRequest.after_id` (field 3, `sitestream.proto`) and `DebugSnapshotRequestDto.alarms_only` (field 3, `site_command.proto`). The same applies to `site_command.proto`/`SiteCommandGrpc/` and `central_control.proto`/`CentralControlGrpc/`; `docker/regen-proto.sh [sitestream|centralcontrol|sitecommand|all]` automates the toggle-build-copy-untoggle and always restores the csproj. #### gRPC Connection Keepalive diff --git a/docs/requirements/Component-ConfigurationDatabase.md b/docs/requirements/Component-ConfigurationDatabase.md index e5bd42b0..605230af 100644 --- a/docs/requirements/Component-ConfigurationDatabase.md +++ b/docs/requirements/Component-ConfigurationDatabase.md @@ -68,7 +68,7 @@ The configuration database stores all central system data, organized by domain a - **New tables use UTC `DateTime` + the UTC value converter**, not `DateTimeOffset`. The `Notification` entity's `DateTimeOffset` timestamps are the **documented legacy exception** — on the SQLite test harness the `DateTimeOffset`→ISO-8601-string converter forced in-memory reductions in a few KPI paths (now pushed server-side where translatable). Prefer `datetime2` UTC columns for any new table. - **New repositories write MSSQL-only SQL with MSSQL-backed test fixtures** (`SkippableFact` gated on the live SQL Server container). The `NotificationOutboxRepository` dual SQLite/T-SQL dialect is the **documented legacy exception** (finding S8) — it predates the MSSQL-fixture convention; do not copy that pattern. - **AuditLog clustered key.** The `dbo.AuditLog` primary/clustered key is `(EventId, OccurredAtUtc)` — `OccurredAtUtc` is required by the monthly partition scheme (`ps_AuditLog_Month`), and `EventId` carries a separate unique index for `InsertIfNotExists` idempotency. **Tracked follow-up (P3):** benchmark whether flipping the clustered key to `(OccurredAtUtc, EventId)` improves range-scan locality once real volume exists. -- **Tracked follow-ups (P4/P6):** the per-node KPI sampling cadence and the AuditLog reconciliation-pull keyset paging (the SiteCalls reconciliation pull already uses a composite keyset cursor; `PullAuditEvents` remains inclusive-`>=`, idempotent on `EventId`) are deferred, lower-urgency hardening items. +- **Tracked follow-ups (P4/P6):** the per-node KPI sampling cadence and the AuditLog reconciliation-pull keyset paging (the SiteCalls reconciliation pull already uses a composite keyset cursor; `PullAuditEvents` gained the site-side `(OccurredAtUtc, EventId)` keyset + `after_id` field in WP2.3, but central still sends a bare `sinceUtc`, so the effective read stays inclusive-`>=`, idempotent on `EventId`) are deferred, lower-urgency hardening items. ### Audit Log - **AuditLog**: The central, append-only audit table owned by the Audit Log component — one row per script-trust-boundary lifecycle event across all channels (outbound API calls, outbound DB writes/reads, notifications, and inbound API requests). Sibling of the `Notifications` and `SiteCalls` tables but distinct: `AuditLog` is the immutable history that observes the other subsystems, not an operational state store. diff --git a/docs/requirements/Component-SiteRuntime.md b/docs/requirements/Component-SiteRuntime.md index 3083a9e6..208ec16f 100644 --- a/docs/requirements/Component-SiteRuntime.md +++ b/docs/requirements/Component-SiteRuntime.md @@ -171,7 +171,7 @@ flowchart TD ### Debug View Support - On request from central (via Communication Layer), the Instance Actor provides a **snapshot** of all current attribute values and alarm states. - Subsequent changes are delivered via the **SiteStreamManager** → **SiteStreamGrpcServer** → gRPC stream to central. The Instance Actor publishes attribute value and alarm state changes to the SiteStreamManager; it does not forward events directly to the Communication Layer. -- The Instance Actor also handles one-shot `DebugSnapshotRequest` messages: it builds the same snapshot (attribute values and alarm states) and replies directly to the sender. Unlike `SubscribeDebugViewRequest`, no subscriber is registered and no stream is established. +- The Instance Actor also handles one-shot `DebugSnapshotRequest` messages: it builds the same snapshot (attribute values and alarm states) and replies directly to the sender. Unlike `SubscribeDebugViewRequest`, no subscriber is registered and no stream is established. When the request sets `AlarmsOnly` (WP2.3 — the central live alarm cache's seed/reconcile fan-out, which discards attribute rows), only the alarm half is built and `AttributeValues` comes back empty; the flag defaults to `false`, so the Debug View path is unchanged. ### Supervision Strategy diff --git a/src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/SqliteAuditWriter.cs b/src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/SqliteAuditWriter.cs index 4247e592..bc0cf237 100644 --- a/src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/SqliteAuditWriter.cs +++ b/src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/SqliteAuditWriter.cs @@ -758,10 +758,17 @@ public class SqliteAuditWriter : IAuditWriter, ISiteAuditQueue, IAsyncDisposable /// /// Lower bound timestamp (UTC) for event occurrence. /// Maximum number of rows to return. + /// + /// Composite-keyset tiebreak: the EventId of the last row already consumed at + /// . Non-null switches the predicate from the inclusive + /// OccurredAtUtc >= $since to the strict composite + /// (OccurredAtUtc, EventId) > ($since, $afterId), matching the query's own + /// ORDER BY so a batch cannot stall on rows sharing one instant. + /// /// Cancellation token. /// A task that resolves to a read-only list of audit events since the given timestamp. public Task> ReadPendingSinceAsync( - DateTime sinceUtc, int batchSize, CancellationToken ct = default) + DateTime sinceUtc, int batchSize, string? afterId = null, CancellationToken ct = default) { if (batchSize <= 0) { @@ -779,13 +786,19 @@ public class SqliteAuditWriter : IAuditWriter, ISiteAuditQueue, IAsyncDisposable ObjectDisposedException.ThrowIf(_disposed, this); using var cmd = _readConnection.CreateCommand(); - cmd.CommandText = """ + var cursorPredicate = afterId is null + ? "fs.OccurredAtUtc >= $since" + // Composite keyset, lexicographic on (OccurredAtUtc, EventId) — the same + // ordering the query applies, so it is a strict "everything after this row". + : "(fs.OccurredAtUtc > $since OR (fs.OccurredAtUtc = $since AND ae.EventId > $afterId))"; + + cmd.CommandText = $""" SELECT ae.EventId, ae.OccurredAtUtc, ae.Actor, ae.Action, ae.Outcome, ae.Category, ae.Target, ae.SourceNode, ae.CorrelationId, ae.DetailsJson FROM audit_event ae JOIN audit_forward_state fs ON fs.EventId = ae.EventId WHERE fs.ForwardState IN ($pending, $forwarded) - AND fs.OccurredAtUtc >= $since + AND {cursorPredicate} ORDER BY fs.OccurredAtUtc ASC, ae.EventId ASC LIMIT $limit; """; @@ -796,12 +809,63 @@ public class SqliteAuditWriter : IAuditWriter, ISiteAuditQueue, IAsyncDisposable // that encoding so we can index-scan against it. cmd.Parameters.AddWithValue("$since", EnsureUtc(sinceUtc).ToString( "o", System.Globalization.CultureInfo.InvariantCulture)); + if (afterId is not null) + { + // EventIds are stored as Guid.ToString() ("D"), so compare in that form. + cmd.Parameters.AddWithValue("$afterId", NormalizeEventId(afterId)); + } cmd.Parameters.AddWithValue("$limit", batchSize); return Task.FromResult(ReadRows(cmd, batchSize)); } } + /// + /// Normalises a wire-supplied event id to the exact textual form stored in + /// audit_event.EventId so the keyset comparison is apples-to-apples. An + /// unparseable value is passed through verbatim rather than throwing — a malformed + /// cursor must degrade to "serves a bit too much", never to a failed pull. + /// + private static string NormalizeEventId(string afterId) => + Guid.TryParse(afterId, out var g) ? g.ToString() : afterId; + + /// + public Task MarkReconciledUpToAsync( + DateTime sinceUtc, string? afterId, CancellationToken ct = default) + { + lock (_writeLock) + { + ObjectDisposedException.ThrowIf(_disposed, this); + + // Everything at or before central's cursor is proven received. With no + // afterId the read contract is inclusive (>= $since), so only rows STRICTLY + // older than the cursor instant are proven — the boundary instant may be + // half-consumed and must stay servable. + var predicate = afterId is null + ? "fs.OccurredAtUtc < $since" + : "(fs.OccurredAtUtc < $since OR (fs.OccurredAtUtc = $since AND fs.EventId <= $afterId))"; + + using var cmd = _connection.CreateCommand(); + cmd.CommandText = $""" + UPDATE audit_forward_state AS fs + SET ForwardState = $reconciled + WHERE fs.ForwardState IN ($pending, $forwarded) + AND {predicate}; + """; + cmd.Parameters.AddWithValue("$reconciled", AuditForwardState.Reconciled.ToString()); + cmd.Parameters.AddWithValue("$pending", AuditForwardState.Pending.ToString()); + cmd.Parameters.AddWithValue("$forwarded", AuditForwardState.Forwarded.ToString()); + cmd.Parameters.AddWithValue("$since", EnsureUtc(sinceUtc).ToString( + "o", System.Globalization.CultureInfo.InvariantCulture)); + if (afterId is not null) + { + cmd.Parameters.AddWithValue("$afterId", NormalizeEventId(afterId)); + } + + return Task.FromResult(cmd.ExecuteNonQuery()); + } + } + /// public Task MarkReconciledAsync(IReadOnlyList eventIds, CancellationToken ct = default) { diff --git a/src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Services/ISiteAuditQueue.cs b/src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Services/ISiteAuditQueue.cs index 17058c91..da545ff5 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Services/ISiteAuditQueue.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Services/ISiteAuditQueue.cs @@ -109,10 +109,47 @@ public interface ISiteAuditQueue /// /// Lower bound timestamp (UTC). /// Maximum number of rows to return. + /// + /// Composite-keyset tiebreak cursor (WP2.3), mirroring + /// : when non-null it is the + /// ("D" GUID form) of the last row already consumed at + /// , and only rows strictly after the composite + /// (OccurredAtUtc, EventId) pair are returned — so a burst sharing one exact + /// instant drains via the id tiebreak instead of pinning the inclusive-timestamp cursor. + /// Null (the first pull, or a central that never sets it) keeps the inclusive + /// >= contract. + /// /// Cancellation token. /// A task that resolves to audit events at or after in pending or forwarded state, up to . Task> ReadPendingSinceAsync( - DateTime sinceUtc, int batchSize, CancellationToken ct = default); + DateTime sinceUtc, int batchSize, string? afterId = null, CancellationToken ct = default); + + /// + /// Reconciliation-pull commit surface, cursor form (WP2.3): flips every row at or before + /// the composite cursor (, ) + /// from / + /// to + /// . + /// + /// + /// The cursor is central's proof of receipt: it only advances past rows central has + /// actually ingested. Flipping on THAT — rather than on having merely served the rows — + /// is what makes the pull at-least-once: a fault between the response leaving the site + /// and central committing it leaves the rows unflipped, so the next pull re-serves them + /// (central dedups on ). + /// + /// With null the cursor is a bare timestamp under the legacy + /// inclusive >= read contract, so only rows STRICTLY older than + /// are proven received; rows at the boundary instant are left + /// alone. Idempotent; already-Reconciled rows are untouched. + /// + /// + /// The cursor timestamp central has consumed up to (UTC). + /// The last consumed at that instant, or null. + /// Cancellation token. + /// A task that resolves to the number of rows flipped. + Task MarkReconciledUpToAsync( + DateTime sinceUtc, string? afterId, CancellationToken ct = default); /// /// Reconciliation-pull commit surface: flips the supplied EventIds to diff --git a/src/ZB.MOM.WW.ScadaBridge.Commons/Messages/DebugView/DebugSnapshotRequest.cs b/src/ZB.MOM.WW.ScadaBridge.Commons/Messages/DebugView/DebugSnapshotRequest.cs index d6d5668a..306eb9fb 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Commons/Messages/DebugView/DebugSnapshotRequest.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Commons/Messages/DebugView/DebugSnapshotRequest.cs @@ -1,5 +1,19 @@ namespace ZB.MOM.WW.ScadaBridge.Commons.Messages.DebugView; +/// +/// Asks one instance for a point-in-time . +/// +/// Unique name of the instance to snapshot. +/// Correlation id echoed on the reply. +/// +/// When true the site builds ONLY the alarm half of the snapshot and returns an +/// empty list (wire efficiency, WP2.3). +/// Set by the central per-site live alarm cache, whose seed/reconcile fan-out discards +/// every attribute row anyway. Defaults to false so the Debug View and every other +/// caller keep the full snapshot; additive on the wire +/// (DebugSnapshotRequestDto.alarms_only, field 3). +/// public record DebugSnapshotRequest( string InstanceUniqueName, - string CorrelationId); + string CorrelationId, + bool AlarmsOnly = false); diff --git a/src/ZB.MOM.WW.ScadaBridge.Commons/Observability/ScadaBridgeTelemetry.cs b/src/ZB.MOM.WW.ScadaBridge.Commons/Observability/ScadaBridgeTelemetry.cs index 0d1474c2..aa5a60ae 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Commons/Observability/ScadaBridgeTelemetry.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Commons/Observability/ScadaBridgeTelemetry.cs @@ -64,6 +64,35 @@ public static class ScadaBridgeTelemetry Meter.CreateCounter("scadabridge.site.alarm_cache.reconnects", unit: "1", description: "Live-alarm aggregator site-wide gRPC stream reconnects (NodeA↔NodeB flip or reconcile-driven reopen)."); + /// + /// Incremented for each live delta evicted from a per-site live-alarm aggregator's + /// bounded pre-seed buffer (drop-oldest, WP2.3). Non-zero means a seed/reconcile + /// fan-out ran long enough for the delta storm behind it to exceed the cap — the + /// dropped transitions are recovered by the fan-out's authoritative snapshot, but a + /// sustained climb points at a slow site. + /// + private static readonly Counter _liveAlarmBufferDrops = + Meter.CreateCounter("scadabridge.site.alarm_cache.buffer_dropped", unit: "1", + description: "Live deltas evicted from a live-alarm aggregator's bounded pre-seed buffer (drop-oldest)."); + + /// + /// Incremented for each debug event evicted from a central debug session's bounded + /// pre-snapshot buffer (drop-oldest, WP2.3). The Debug View is lossy-under-backpressure + /// by design; this makes the loss measurable instead of unbounded memory growth. + /// + private static readonly Counter _debugPreSnapshotDrops = + Meter.CreateCounter("scadabridge.central.debug_view.presnapshot_dropped", unit: "1", + description: "Debug events evicted from a central debug session's bounded pre-snapshot buffer (drop-oldest)."); + + /// + /// Incremented for each event evicted from a site-hosted gRPC stream's bounded send + /// channel, tagged by stream kind (instance = Debug View, site-alarms = + /// the site-wide alarm feed behind the operator Alarm Summary). + /// + private static readonly Counter _siteStreamEventDrops = + Meter.CreateCounter("scadabridge.site.stream.events_dropped", unit: "1", + description: "Events evicted from a site gRPC stream's bounded send channel, tagged by stream kind."); + // ---------------- Observable gauges ---------------- /// Current count of open site connections, mutated via . @@ -136,6 +165,19 @@ public static class ScadaBridgeTelemetry /// Records that a per-site live-alarm aggregator re-established its site-wide gRPC stream. public static void RecordLiveAlarmStreamReconnect() => _liveAlarmStreamReconnects.Add(1); + /// Records live deltas evicted from a live-alarm aggregator's bounded pre-seed buffer. + /// Number of deltas evicted. + public static void RecordLiveAlarmBufferDrop(long count = 1) => _liveAlarmBufferDrops.Add(count); + + /// Records debug events evicted from a debug session's bounded pre-snapshot buffer. + /// Number of events evicted. + public static void RecordDebugPreSnapshotDrop(long count = 1) => _debugPreSnapshotDrops.Add(count); + + /// Records an event evicted from a site gRPC stream's bounded send channel. + /// Stream kind tag (instance or site-alarms). + public static void RecordSiteStreamEventDropped(string streamKind) => + _siteStreamEventDrops.Add(1, new KeyValuePair("stream", streamKind)); + /// /// Registers the provider the StoreAndForward queue-depth gauge reads on each observation. /// A later task supplies a provider that reads the real StoreAndForward depth. A null diff --git a/src/ZB.MOM.WW.ScadaBridge.Communication/Actors/DebugStreamBridgeActor.cs b/src/ZB.MOM.WW.ScadaBridge.Communication/Actors/DebugStreamBridgeActor.cs index e0b26082..dc83fb91 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Communication/Actors/DebugStreamBridgeActor.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Communication/Actors/DebugStreamBridgeActor.cs @@ -2,6 +2,7 @@ using Akka.Actor; using Akka.Event; using ZB.MOM.WW.ScadaBridge.Commons.Messages.DebugView; using ZB.MOM.WW.ScadaBridge.Commons.Messages.Streaming; +using ZB.MOM.WW.ScadaBridge.Commons.Observability; using ZB.MOM.WW.ScadaBridge.Communication.Grpc; namespace ZB.MOM.WW.ScadaBridge.Communication.Actors; @@ -45,9 +46,19 @@ public class DebugStreamBridgeActor : ReceiveActor, IWithTimers private const int MaxRetries = 3; private const string ReconnectTimerKey = "grpc-reconnect"; private const string StabilityTimerKey = "grpc-stability"; + private const string SnapshotTimerKey = "debug-snapshot-deadline"; /// Delay between gRPC reconnection attempts. internal static TimeSpan ReconnectDelay { get; set; } = TimeSpan.FromSeconds(5); + /// + /// Hard deadline on the initial (WP2.3). The site builds + /// it in milliseconds; if none arrives inside this window the site never answered (the + /// Ask was lost, the singleton moved mid-request, the instance actor is wedged) and the + /// session must FAIL rather than sit in the buffering phase accumulating live events + /// behind a snapshot that is never coming. Settable for tests. + /// + internal static TimeSpan SnapshotTimeout { get; set; } = TimeSpan.FromSeconds(60); + /// /// How long a freshly-opened gRPC stream must stay up before its retry budget /// is considered "recovered" and is reset to 0. @@ -85,17 +96,35 @@ public class DebugStreamBridgeActor : ReceiveActor, IWithTimers /// Ordered buffer of live gRPC events (/ /// ) that arrived before the snapshot was delivered. /// Flushed (with per-entity dedup against the snapshot) when the snapshot arrives, - /// then never used again. Mutated only on the actor thread. + /// then never used again. Bounded by with drop-oldest + /// eviction (WP2.3): a snapshot that never arrives used to buffer without limit on the + /// central node. Mutated only on the actor thread. /// - private readonly List _preSnapshotBuffer = new(); + private readonly Queue _preSnapshotBuffer = new(); /// - /// Defensive log threshold: if the pre-snapshot buffer grows past this many events - /// during a slow snapshot we log once (events are NOT dropped — the window is short). + /// Defensive log threshold: the first warning fires when the pre-snapshot buffer grows + /// past this many events during a slow snapshot, before the hard cap starts evicting. /// private const int BufferWarnThreshold = 10_000; private bool _bufferWarned; + /// + /// Hard cap on the pre-snapshot buffer. Beyond it the OLDEST event is evicted — the + /// snapshot that ends the buffering phase is authoritative for anything that old, so + /// keeping the newest events is what preserves the post-snapshot delta chain. + /// + private const int MaxPreSnapshotBuffer = 20_000; + + /// Events evicted from the pre-snapshot buffer in this session. Actor-thread only. + private long _preSnapshotDropped; + + /// + /// Total pre-snapshot events dropped across all debug sessions on this node — the raw + /// counter behind scadabridge.central.debug_view.presnapshot_dropped. + /// + internal static long TotalPreSnapshotDropped; + /// Timer scheduler for reconnect and stability window timers. public ITimerScheduler Timers { get; set; } = null!; @@ -176,6 +205,9 @@ public class DebugStreamBridgeActor : ReceiveActor, IWithTimers _instanceUniqueName, snapshot.AttributeValues.Count, snapshot.AlarmStates.Count, _preSnapshotBuffer.Count); + // The snapshot arrived — stand the hard deadline down. + Timers.Cancel(SnapshotTimerKey); + // Deliver the snapshot, then flush the gap-window buffer (deduped), then // switch to pass-through. Order matters: snapshot first, buffered events next. _onEvent(snapshot); @@ -183,14 +215,40 @@ public class DebugStreamBridgeActor : ReceiveActor, IWithTimers _snapshotDelivered = true; }); - // Domain events arriving via Self.Tell from gRPC callback. - // Receiving an event must NOT reset _retryCount — a - // flapping stream that delivers a single event between failures would - // otherwise never trip MaxRetries. The retry budget is recovered only by - // GrpcStreamStable (a stream that has stayed up for StabilityWindow). - // Before the snapshot has been delivered, BUFFER (in arrival order) - // rather than deliver — these may be gap-window events. After the snapshot has - // been flushed, pass through directly (same handler, phase-dependent behavior). + // Hard snapshot deadline (WP2.3). Nothing else ends a session stuck in the + // buffering phase: the site's reply was lost, so no gRPC error fires, the stream + // keeps delivering events, and (with the wrapper above) they no longer even reset + // the orphan timeout. Fail the session so the consumer is told and can reopen. + Receive(_ => + { + if (_stopped || _snapshotDelivered) return; + _log.Error( + "No debug snapshot for {0} within {1}s ({2} event(s) buffered, {3} dropped); failing the session", + _instanceUniqueName, SnapshotTimeout.TotalSeconds, + _preSnapshotBuffer.Count, _preSnapshotDropped); + CleanupGrpc(); + SendUnsubscribe(); + _stopped = true; + _preSnapshotBuffer.Clear(); + _onTerminated(); + Context.Stop(Self); + }); + + // Domain events arriving via Self.Tell from the gRPC callback, wrapped so they do + // NOT influence the receive timeout (WP2.3): the orphan safety net exists to end a + // session whose CONSUMER is gone, and a busy site's event flood used to keep that + // net permanently reset — an abandoned session on a chatty instance never timed out. + // Receiving an event must not reset _retryCount either: a flapping stream that + // delivers a single event between failures would otherwise never trip MaxRetries. + // The retry budget is recovered only by GrpcStreamStable (a stream that has stayed + // up for StabilityWindow). Before the snapshot has been delivered, BUFFER (in arrival + // order) rather than deliver — these may be gap-window events; after the snapshot has + // been flushed, pass through directly (phase-dependent behavior). + Receive(wrapped => HandleStreamEvent(wrapped.Event)); + + // Unwrapped forms are still accepted (a direct Tell from a test or a future + // in-process producer); those DO influence the receive timeout, which is correct — + // they are not the high-volume stream path. Receive(changed => HandleStreamEvent(changed)); Receive(changed => HandleStreamEvent(changed)); @@ -286,15 +344,30 @@ public class DebugStreamBridgeActor : ReceiveActor, IWithTimers return; } - _preSnapshotBuffer.Add(evt); - if (!_bufferWarned && _preSnapshotBuffer.Count > BufferWarnThreshold) + if (!_bufferWarned && _preSnapshotBuffer.Count + 1 > BufferWarnThreshold) { _bufferWarned = true; _log.Warning( - "Pre-snapshot debug-event buffer for {0} exceeded {1} events while awaiting the snapshot; " + - "events are still retained (not dropped).", - _instanceUniqueName, BufferWarnThreshold); + "Pre-snapshot debug-event buffer for {0} exceeded {1} events while awaiting the snapshot " + + "(hard cap {2}, drop-oldest beyond it).", + _instanceUniqueName, BufferWarnThreshold, MaxPreSnapshotBuffer); } + + while (_preSnapshotBuffer.Count >= MaxPreSnapshotBuffer) + { + _preSnapshotBuffer.Dequeue(); + _preSnapshotDropped++; + Interlocked.Increment(ref TotalPreSnapshotDropped); + ScadaBridgeTelemetry.RecordDebugPreSnapshotDrop(); + if (_preSnapshotDropped == 1 || _preSnapshotDropped % 500 == 0) + { + _log.Warning( + "Pre-snapshot debug-event buffer for {0} is at its {1}-event cap; {2} event(s) evicted so far", + _instanceUniqueName, MaxPreSnapshotBuffer, _preSnapshotDropped); + } + } + + _preSnapshotBuffer.Enqueue(evt); } /// @@ -352,8 +425,9 @@ public class DebugStreamBridgeActor : ReceiveActor, IWithTimers if (dropped > 0 || flushed > 0) { - _log.Debug("Flushed {0} buffered debug event(s) for {1}, dropped {2} as already-in-snapshot", - flushed, _instanceUniqueName, dropped); + _log.Debug("Flushed {0} buffered debug event(s) for {1}, dropped {2} as already-in-snapshot" + + " ({3} previously evicted at the buffer cap)", + flushed, _instanceUniqueName, dropped, _preSnapshotDropped); } _preSnapshotBuffer.Clear(); @@ -429,6 +503,10 @@ public class DebugStreamBridgeActor : ReceiveActor, IWithTimers var request = new SubscribeDebugViewRequest(_instanceUniqueName, _correlationId); var envelope = new SiteEnvelope(_siteIdentifier, request); _centralCommunicationActor.Tell(envelope, Self); + + // Arm the hard snapshot deadline alongside the request. + if (SnapshotTimeout > TimeSpan.Zero) + Timers.StartSingleTimer(SnapshotTimerKey, new DebugSnapshotDeadline(), SnapshotTimeout); } /// @@ -468,7 +546,8 @@ public class DebugStreamBridgeActor : ReceiveActor, IWithTimers await client.SubscribeAsync( _correlationId, _instanceUniqueName, - evt => self.Tell(evt), + // Wrapped: stream traffic must not reset the orphan receive timeout. + evt => self.Tell(new LiveDebugStreamEvent(evt)), ex => self.Tell(new GrpcStreamError(ex, generation)), () => self.Tell(new GrpcStreamCompleted(generation)), ct); @@ -579,6 +658,19 @@ public class DebugStreamBridgeActor : ReceiveActor, IWithTimers /// public record StopDebugStream; +/// +/// Envelope for a live gRPC stream event (AttributeValueChanged/ +/// AlarmStateChanged). Implements so a busy +/// site's event flood cannot keep resetting the orphan-session receive timeout — the timeout +/// measures consumer/session liveness, not site chatter (WP2.3). +/// +internal record LiveDebugStreamEvent(object Event) : INotInfluenceReceiveTimeout; + +/// +/// Internal message: the hard deadline for the initial DebugViewSnapshot expired. +/// +internal record DebugSnapshotDeadline; + /// /// Internal message indicating a gRPC stream error occurred, stamped with the stream /// generation it came from so a late error out of a cancelled stream can be ignored. diff --git a/src/ZB.MOM.WW.ScadaBridge.Communication/Actors/SiteAlarmAggregatorActor.cs b/src/ZB.MOM.WW.ScadaBridge.Communication/Actors/SiteAlarmAggregatorActor.cs index 94fecd5e..fc092ba2 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Communication/Actors/SiteAlarmAggregatorActor.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Communication/Actors/SiteAlarmAggregatorActor.cs @@ -31,10 +31,14 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Actors; /// /// /// Failover + drift: a gRPC error flips NodeA↔NodeB with the same retry budget + -/// stability window as , and each reconnect triggers -/// a RE-SEED (never silently serve stale). A periodic reconcile snapshot -/// (, default 60s) corrects instance-set drift and any -/// missed delta. +/// stability window as . A re-seed runs once per +/// successful (re)connect — driven by the stream's connected callback, not by each +/// reconnect ATTEMPT (WP2.3: a site outage used to fan a full snapshot out per retry, all +/// of them against the very site that is unreachable). A periodic reconcile snapshot +/// (, default 60s, jittered per site so N aggregators do +/// not fan out in lockstep) remains the drift/backstop: it corrects instance-set drift and +/// any missed delta, but it is skipped when a fan-out already completed inside the +/// window, and it publishes to viewers only when the snapshot actually changed the cache. /// /// All state is mutated only on the actor thread: gRPC callbacks and fan-out results are /// marshalled back via Self.Tell, so the cache needs no internal lock. The @@ -47,18 +51,20 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers private readonly string _siteIdentifier; private readonly string _correlationId; private readonly Func>> _seedFn; - private readonly Action> _publish; + private readonly Action, bool> _publish; private readonly SiteStreamGrpcClientFactory _grpcFactory; private readonly string _grpcNodeAAddress; private readonly string _grpcNodeBAddress; private readonly TimeSpan _reconcileInterval; private readonly TimeSpan _publishCoalesce; + private readonly double _reconcileJitterFraction; private const int MaxRetries = 3; private const string ReconnectTimerKey = "alarm-grpc-reconnect"; private const string StabilityTimerKey = "alarm-grpc-stability"; private const string ReconcileTimerKey = "alarm-reconcile"; private const string PublishTimerKey = "alarm-publish-coalesce"; + private const string SeedRetryTimerKey = "alarm-seed-retry"; /// True while a coalesced publish is armed (dirty deltas awaiting one tick). Actor-thread only. private bool _publishPending; @@ -83,9 +89,10 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers /// lifetime). Reconcile snapshots keep serving in the meantime; the next reconcile tick /// self-heals the stream by resetting the retry budget and reopening it, so neither a /// sustained site outage nor a routine 4h stream expiry permanently drops the live feed. - /// Actor-thread only. + /// Starts true: until the site accepts the first subscription there is no live + /// feed, and the owning cache must not advertise one. Actor-thread only. /// - private bool _streamDown; + private bool _streamDown = true; /// /// Why the stream is down: true = the retry budget was exhausted, so the @@ -107,8 +114,57 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers /// True while a seed/reconcile snapshot fan-out is in flight (deltas buffer). Actor-thread only. private bool _fanoutInFlight; - /// Ordered buffer of live deltas that arrived while a fan-out was in flight. Actor-thread only. - private readonly List _buffer = new(); + /// + /// Ordered buffer of live deltas that arrived while a fan-out was in flight, capped at + /// with drop-oldest eviction (WP2.3 — a stuck fan-out + /// used to buffer without bound). Dropping the OLDEST is the right eviction here: the + /// fan-out that follows rebuilds the cache authoritatively, so an evicted delta is + /// superseded rather than lost, while the newest transitions — the ones the snapshot may + /// predate — are the ones kept. Actor-thread only. + /// + private readonly Queue _buffer = new(); + + /// Total deltas evicted from over this actor's life. Actor-thread only. + private long _bufferDropped; + + /// + /// Hard cap on the pre-fan-out delta buffer. Beyond this the oldest entry is evicted and + /// counted; the counter is logged (first drop + every 500th) and exported as + /// scadabridge.site.alarm_cache.buffer_dropped. + /// + private const int MaxBufferedDeltas = 20_000; + + /// + /// True when the next successful (re)connect must run a seed fan-out: set whenever the + /// stream is lost (fault or graceful completion) and cleared by the connect that consumes + /// it. This is what makes the re-seed happen once per successful reconnect rather than + /// once per reconnect ATTEMPT — the old code fanned a full snapshot out on every retry, + /// against a site that was by definition unreachable. Actor-thread only. + /// + private bool _seedOnConnect; + + /// + /// Set whenever a fan-out finishes (success or failure); consumed by the next reconcile + /// tick, which skips its own fan-out when it finds the flag set. That makes the + /// connect-driven seed and the periodic backstop mutually exclusive — the pair used to + /// run BOTH, which is the "full unconditional snapshot every 60s" waste — while bounding + /// staleness at two intervals (a tick can be skipped at most once in a row). + /// Actor-thread only. + /// + private bool _fanoutSinceLastTick; + + /// + /// True between issuing a stream open and its first outcome (connected / error / + /// completed), so a reconcile tick never stacks a second open on an in-flight one. + /// Actor-thread only. + /// + private bool _openInFlight; + + /// Consecutive whole-fan-out failures, driving the seed-retry backoff. Actor-thread only. + private int _consecutiveSeedFailures; + + /// Upper bound on the seed-retry backoff, as a multiple of the reconcile interval. + private const int MaxSeedRetryBackoffMultiplier = 8; /// /// A failover re-seed was requested while a fan-out was already in flight; it must run @@ -144,6 +200,10 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers /// /// Publishes a fresh immutable snapshot of the cache to the owning service, which /// stores it and raises viewer onChanged callbacks. Invoked on the actor thread. + /// The second argument is the live-stream liveness flag: false means the site-wide + /// gRPC stream is currently down (fault-exhausted or gracefully completed), so the cache + /// is only as fresh as the last reconcile and the page must fall back to polling + /// (WP2.3 carried residual — IsLive used to stay true across a dead stream). /// /// Factory caching one gRPC client per (site, endpoint). /// gRPC address of the site's node A. @@ -158,18 +218,24 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers /// /// How long a fresh gRPC stream must stay up before its retry budget recovers (production 60s). /// + /// + /// Fraction of added as a per-tick random offset so + /// N per-site aggregators on one central node do not fan out in lockstep (production 0.2 = + /// up to +20%). Zero disables jitter, which is what tests want for determinism. + /// public SiteAlarmAggregatorActor( string siteIdentifier, string correlationId, Func>> seedFn, - Action> publish, + Action, bool> publish, SiteStreamGrpcClientFactory grpcFactory, string grpcNodeAAddress, string grpcNodeBAddress, TimeSpan reconcileInterval, TimeSpan publishCoalesce, TimeSpan reconnectDelay, - TimeSpan stabilityWindow) + TimeSpan stabilityWindow, + double reconcileJitterFraction = 0.0) { _siteIdentifier = siteIdentifier; _correlationId = correlationId; @@ -182,6 +248,7 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers _publishCoalesce = publishCoalesce; _reconnectDelay = reconnectDelay; _stabilityWindow = stabilityWindow; + _reconcileJitterFraction = Math.Clamp(reconcileJitterFraction, 0.0, 1.0); // Live delta from the site-wide alarm stream (marshalled in via Self.Tell). // A received delta must NOT reset the retry budget (a flapping stream that @@ -198,6 +265,22 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers // Periodic reconcile tick (and the re-seed kicked after a reconnect). Receive(_ => OnReconcileTick()); + // Backoff-scheduled retry of a fan-out that failed as a whole. + Receive(_ => + { + if (_stopped) return; + StartFanout(isInitial: false); + }); + + // The site-wide stream is confirmed established (response headers received). This — + // not the reconnect attempt — is the "successful (re)connect" that earns one re-seed. + Receive(msg => + { + if (_stopped) return; + if (msg.Generation != _streamGeneration) return; + OnStreamConnected(); + }); + // Coalesced-publish tick: one publish for a batch of dirtying deltas (N6). Receive(_ => { @@ -270,11 +353,31 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers // in the seed window are captured (buffered) rather than lost. OpenGrpcStream(); - // Kick the initial seed fan-out. + // Kick the initial seed fan-out. The connect callback for this first stream must + // NOT seed again on top of it, so the flag starts cleared. + _seedOnConnect = false; StartFanout(isInitial: true); - // Periodic reconcile backstop. - Timers.StartPeriodicTimer(ReconcileTimerKey, new RunReconcile(), _reconcileInterval, _reconcileInterval); + // Periodic reconcile backstop. Single-shot and re-armed with fresh jitter each + // tick (a periodic timer would lock every site to the same phase forever). + ArmReconcileTimer(); + } + + /// + /// Arms the next reconcile tick at plus a random + /// offset of up to of it, so N per-site + /// aggregators started together (a central failover restarts them all at once) spread + /// their fan-outs instead of stampeding the same 60s boundary. + /// + private void ArmReconcileTimer() + { + var delay = _reconcileInterval; + if (_reconcileJitterFraction > 0 && _reconcileInterval > TimeSpan.Zero) + { + var jitterTicks = (long)(_reconcileInterval.Ticks * _reconcileJitterFraction * Random.Shared.NextDouble()); + delay += TimeSpan.FromTicks(jitterTicks); + } + Timers.StartSingleTimer(ReconcileTimerKey, new RunReconcile(), delay); } /// @@ -294,16 +397,24 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers // ── Reconcile tick ────────────────────────────────────────────────────────── /// - /// Periodic reconcile: always re-run the snapshot fan-out (corrects drift + missed - /// deltas), and if the live stream was previously given up, self-heal it by resetting - /// the retry budget and reopening — so a sustained outage never permanently kills the - /// live feed. + /// Periodic reconcile backstop: re-run the snapshot fan-out (corrects instance-set drift + /// + any missed delta) UNLESS one already completed inside this window — a reconnect-driven + /// seed makes the tick redundant, and running both was the "full unconditional snapshot + /// every 60s" waste (WP2.3). If the live stream was previously given up, self-heal it by + /// resetting the retry budget and reopening, so a sustained outage never permanently kills + /// the live feed. /// private void OnReconcileTick() { if (_stopped) return; - StartFanout(isInitial: false); - if (_streamDown) + + // Re-arm first: every exit path below must leave the backstop running. + ArmReconcileTimer(); + + // A down stream is reopened in preference to fanning out: the reopen's connect + // callback seeds by itself, so doing both would double the work. + var reopening = false; + if (_streamDown && !_openInFlight) { _log.Info("Site-alarm gRPC stream for {0} was down; reopening on reconcile tick", _siteIdentifier); if (_retryBudgetExhausted) @@ -313,8 +424,24 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers } // Telemetry: a reconcile-driven reopen after the stream was given up is a reconnect. ScadaBridgeTelemetry.RecordLiveAlarmStreamReconnect(); + _seedOnConnect = true; OpenGrpcStream(); + reopening = true; } + + if (_fanoutSinceLastTick) + { + // A connect-driven seed (or the initial seed) already refreshed the cache inside + // this window. Clear the flag so the NEXT tick fans out regardless — staleness + // stays bounded at two intervals even if reconnects keep arriving. + _fanoutSinceLastTick = false; + _log.Debug("Site-alarm reconcile for {0} skipped; a fan-out already ran this window", + _siteIdentifier); + return; + } + + if (!reopening) + StartFanout(isInitial: false); } // ── Seed / reconcile fan-out ──────────────────────────────────────────────── @@ -366,6 +493,12 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers // Rebuild the cache authoritatively from the fresh snapshot (this is what makes // reconcile able to DROP rows for instances/alarms that disappeared — a merge // could never remove a stale row since the live stream sends no "removed" event). + // The previous cache is kept alongside so the reconcile can publish as a DIFF: + // an unchanged snapshot must not wake every viewer's render path once a minute. + var previous = _cache.Count == 0 + ? null + : new Dictionary(_cache); + _cache.Clear(); foreach (var alarm in msg.Alarms) { @@ -378,19 +511,24 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers // buffered delta whose key is in the fresh snapshot with an equal-or-newer // timestamp is already reflected → drop; a strictly-newer (or new-key) delta is // applied. Inclusive-on-snapshot boundary matches DebugStreamBridgeActor. - FlushBuffer(); + var flushChanged = FlushBuffer(); _fanoutInFlight = false; + _fanoutSinceLastTick = true; + _consecutiveSeedFailures = 0; + Timers.Cancel(SeedRetryTimerKey); + var firstSeed = !_seeded; _seeded = true; _log.Debug("Site-alarm {0} {1} complete: {2} alarm row(s)", _siteIdentifier, msg.IsInitial ? "seed" : "reconcile", _cache.Count); // The fresh snapshot already carries the buffered deltas; drop any armed coalesce - // tick so we publish once, immediately. + // tick so we publish once, immediately — but only when something actually moved. Timers.Cancel(PublishTimerKey); _publishPending = false; - Publish(); + if (firstSeed || flushChanged || DiffersFrom(previous)) + Publish(); // A failover re-seed requested while this fan-out was in flight runs now (N7.1). if (_reseedQueued) @@ -400,22 +538,44 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers } } + /// + /// True when the just-installed snapshot differs from in any + /// key or any row value — the reconcile's "is this publish worth a viewer fan-out?" test. + /// A null previous cache (nothing published yet) always counts as different. + /// + private bool DiffersFrom(Dictionary? previous) + { + if (previous is null || previous.Count != _cache.Count) return true; + + foreach (var (key, current) in _cache) + { + if (!previous.TryGetValue(key, out var old)) return true; + // AlarmStateChanged is a record: structural equality covers state, level, + // timestamp and every native-alarm enrichment field. + if (!Equals(old, current)) return true; + } + return false; + } + private void OnSeedFailed(SeedFailed msg) { if (_stopped) return; + _consecutiveSeedFailures++; + _log.Warning(msg.Exception, - "Site-alarm {0} {1} fan-out failed; keeping current cache and relying on the next reconcile", - _siteIdentifier, msg.IsInitial ? "seed" : "reconcile"); + "Site-alarm {0} {1} fan-out failed ({2} consecutive); keeping current cache and retrying with backoff", + _siteIdentifier, msg.IsInitial ? "seed" : "reconcile", _consecutiveSeedFailures); // Don't lose deltas captured during the failed window — apply them pass-through // into the (possibly stale/empty) cache. The next reconcile re-seeds authoritatively. _fanoutInFlight = false; - FlushBuffer(dedupAgainstSeed: false); + _fanoutSinceLastTick = true; + var flushChanged = FlushBuffer(dedupAgainstSeed: false); // Only publish if we already had a seed (so IsLive doesn't flip true on a // failed initial seed — the page keeps its poll fallback until we truly seed). - if (_seeded) + if (_seeded && flushChanged) { Timers.Cancel(PublishTimerKey); _publishPending = false; @@ -427,27 +587,54 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers { _reseedQueued = false; StartFanout(isInitial: false); + return; } + + // Backoff retry (WP2.3): the seed leg used to have none — a site returning errors + // was re-fanned at full reconcile cadence forever. Retry at reconnectDelay doubling + // up to MaxSeedRetryBackoffMultiplier × the reconcile interval; the periodic + // reconcile remains the floor once the backoff saturates. + ScheduleSeedRetry(); + } + + /// + /// Arms the backoff retry for a failed fan-out: reconnectDelay × 2^(failures-1), + /// capped at × the reconcile interval. Once + /// the cap is reached the periodic reconcile tick is doing the same work anyway, so no + /// extra timer is armed. + /// + private void ScheduleSeedRetry() + { + var cap = _reconcileInterval > TimeSpan.Zero + ? TimeSpan.FromTicks(_reconcileInterval.Ticks * MaxSeedRetryBackoffMultiplier) + : TimeSpan.FromMinutes(8); + + var shift = Math.Min(_consecutiveSeedFailures - 1, 16); + var delayTicks = _reconnectDelay.Ticks * (1L << shift); + var delay = TimeSpan.FromTicks(Math.Min(delayTicks, cap.Ticks)); + if (delay <= TimeSpan.Zero) return; + + Timers.StartSingleTimer(SeedRetryTimerKey, new RetrySeed(), delay); } /// /// Flushes the pre-fan-out buffer in arrival order. When /// is true (normal completion) a buffered delta already reflected in the just-installed /// snapshot (same key, timestamp <= cache entry) is dropped; otherwise every buffered - /// delta is applied pass-through. + /// delta is applied pass-through. Returns true when at least one delta changed the + /// cache (the reconcile's publish-only-on-change test must count these too). /// - private void FlushBuffer(bool dedupAgainstSeed = true) + private bool FlushBuffer(bool dedupAgainstSeed = true) { - if (_buffer.Count == 0) return; + if (_buffer.Count == 0) return false; + var changed = false; foreach (var delta in _buffer) { - if (dedupAgainstSeed) - ApplyDelta(delta, requireStrictlyNewer: true); - else - ApplyDelta(delta, requireStrictlyNewer: false); + changed |= ApplyDelta(delta, requireStrictlyNewer: dedupAgainstSeed); } _buffer.Clear(); + return changed; } // ── Live delta handling ───────────────────────────────────────────────────── @@ -458,15 +645,32 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers if (_fanoutInFlight) { - _buffer.Add(delta); - if (!_bufferWarned && _buffer.Count > BufferWarnThreshold) + if (!_bufferWarned && _buffer.Count + 1 > BufferWarnThreshold) { _bufferWarned = true; _log.Warning( "Site-alarm pre-seed buffer for {0} exceeded {1} deltas while a fan-out was in flight " + - "(deltas retained, not dropped).", - _siteIdentifier, BufferWarnThreshold); + "(hard cap {2}, drop-oldest beyond it).", + _siteIdentifier, BufferWarnThreshold, MaxBufferedDeltas); } + + // Hard cap with drop-oldest: an unbounded buffer behind a stuck fan-out was a + // straight memory leak. The evicted rows are superseded by the snapshot that + // ends the fan-out, so the drop costs freshness, never correctness. + while (_buffer.Count >= MaxBufferedDeltas) + { + _buffer.Dequeue(); + _bufferDropped++; + ScadaBridgeTelemetry.RecordLiveAlarmBufferDrop(); + if (_bufferDropped == 1 || _bufferDropped % 500 == 0) + { + _log.Warning( + "Site-alarm pre-seed buffer for {0} is at its {1}-delta cap; {2} delta(s) evicted so far", + _siteIdentifier, MaxBufferedDeltas, _bufferDropped); + } + } + + _buffer.Enqueue(delta); return; } @@ -516,7 +720,10 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers var snapshot = _cache.Values.ToList(); try { - _publish(snapshot); + // Liveness rides every publish so the owning service's IsLive tracks the STREAM, + // not merely "a snapshot was published once". A down stream means the cache is + // only as fresh as the last reconcile and the page must keep polling. + _publish(snapshot, !_streamDown); } catch (Exception ex) { @@ -530,7 +737,8 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers { if (_stopped) return; - _streamDown = false; + // The stream is not live again until the server actually accepts it — liveness flips + // on GrpcAlarmStreamConnected, not on the attempt (WP2.3 carried residual). var endpoint = _useNodeA ? _grpcNodeAAddress : _grpcNodeBAddress; _log.Info("Opening site-alarm gRPC stream for {0} to {1}", _siteIdentifier, endpoint); @@ -540,6 +748,7 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers Timers.StartSingleTimer(StabilityTimerKey, new GrpcAlarmStreamStable(), _stabilityWindow); + _openInFlight = true; var generation = ++_streamGeneration; var client = _grpcFactory.GetOrCreate(_siteIdentifier, endpoint); var self = Self; @@ -556,7 +765,8 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers alarm => self.Tell(alarm), ex => self.Tell(new GrpcAlarmStreamError(ex, generation)), () => self.Tell(new GrpcAlarmStreamCompleted(generation)), - ct); + ct, + () => self.Tell(new GrpcAlarmStreamConnected(generation))); }, ct).ContinueWith(t => { if (t.IsFaulted) @@ -567,6 +777,32 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers }, TaskContinuationOptions.ExecuteSynchronously); } + /// + /// The site accepted the subscription — the only point at which the stream is genuinely + /// live. Consumes the pending re-seed (so exactly ONE fan-out runs per successful + /// (re)connect, instead of one per reconnect attempt against a site that is by definition + /// unreachable) and republishes so IsLive recovers. + /// + private void OnStreamConnected() + { + _openInFlight = false; + + var wasDown = _streamDown; + _streamDown = false; + + if (_seedOnConnect) + { + _seedOnConnect = false; + _log.Info("Site-alarm gRPC stream for {0} (re)connected; running one re-seed", _siteIdentifier); + StartFanout(isInitial: false); + } + + // Liveness recovered — republish so viewers stop falling back to polling even if the + // seed that follows finds nothing changed. + if (wasDown && _seeded) + Publish(); + } + private void HandleGrpcError() { if (_stopped) return; @@ -574,6 +810,19 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers // Stream failed before the stability window — retry budget NOT recovered. Timers.Cancel(StabilityTimerKey); + _openInFlight = false; + var wasLive = !_streamDown; + _streamDown = true; + + // The next successful connect owes exactly one re-seed. Setting it here (rather than + // fanning out now) is the fix for the per-attempt re-fan-out: a site outage used to + // cost one whole-site snapshot per retry. + _seedOnConnect = true; + + // Liveness lost — tell viewers immediately so a dead stream stops reading as live. + if (wasLive && _seeded) + Publish(); + _retryCount++; if (_retryCount > MaxRetries) @@ -585,7 +834,6 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers _log.Error("Site-alarm gRPC stream for {0} exceeded max retries ({1}); leaving stream down, " + "reconcile snapshots continue and the next reconcile tick will retry the stream", _siteIdentifier, MaxRetries); - _streamDown = true; _retryBudgetExhausted = true; CleanupGrpc(); return; @@ -602,11 +850,9 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers // Telemetry: a NodeA↔NodeB failover flip is a reconnect + re-seed. ScadaBridgeTelemetry.RecordLiveAlarmStreamReconnect(); - // A failover flip must RE-SEED (never silently serve stale) — kick a reconcile - // fan-out alongside the reconnect. Buffering during the fan-out keeps the new - // stream's deltas coherent with the fresh snapshot. - StartFanout(isInitial: false); - + // The re-seed is owed to the CONNECT (see _seedOnConnect above), not to this + // attempt: a snapshot fan-out issued while the site is unreachable degrades to + // empty rows and would have to be redone on reconnect anyway. if (_retryCount == 1) Self.Tell(new ReconnectAlarmStream()); else @@ -629,8 +875,17 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers _log.Info("Site-alarm gRPC stream for {0} completed gracefully (server end of stream); " + "reopening on the next reconcile tick", _siteIdentifier); + _openInFlight = false; + var wasLive = !_streamDown; _streamDown = true; + // The reopen owes one re-seed, run when it actually connects. + _seedOnConnect = true; CleanupGrpc(); + + // A completed stream is NOT live: without this the cache kept reporting live for up + // to a full reconcile interval after the site closed the feed (WP2.3 residual). + if (wasLive && _seeded) + Publish(); } private void CleanupGrpc() @@ -676,6 +931,14 @@ internal sealed record SeedFailed(Exception Exception, bool IsInitial); /// Internal: periodic reconcile tick (and the re-seed kicked after a reconnect). internal sealed record RunReconcile; +/// Internal: backoff-scheduled retry of a fan-out that failed as a whole. +internal sealed record RetrySeed; + +/// Internal: the site accepted the site-wide alarm subscription (response headers +/// received), stamped with its stream generation so a late connect from a cancelled stream +/// is ignored. This — not the reconnect attempt — is what earns a re-seed. +internal sealed record GrpcAlarmStreamConnected(int Generation); + /// Internal: coalesced-publish tick — flush the dirty cache to viewers once (N6). internal sealed record PublishCoalesced; diff --git a/src/ZB.MOM.WW.ScadaBridge.Communication/CommunicationOptions.cs b/src/ZB.MOM.WW.ScadaBridge.Communication/CommunicationOptions.cs index 3f91d824..33073cdb 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Communication/CommunicationOptions.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Communication/CommunicationOptions.cs @@ -107,6 +107,21 @@ public class CommunicationOptions /// Maximum number of concurrent gRPC streaming subscriptions per site node. public int GrpcMaxConcurrentStreams { get; set; } = 100; + /// + /// Send-channel capacity for a per-instance Debug View stream (SubscribeInstance). + /// Lossy by design: the Debug View is a diagnostic surface and drops its oldest events + /// under backpressure rather than stalling the site's event hub. + /// + public int GrpcInstanceStreamChannelCapacity { get; set; } = 1000; + + /// + /// Send-channel capacity for the site-wide alarm stream (SubscribeSite). Larger + /// than the Debug View's (WP2.3): that feed backs the operator Alarm Summary, where a + /// silently dropped transition is a missed alarm rather than a missed diagnostic frame, + /// and an alarm burst arriving during a WAN stall must survive the stall. + /// + public int GrpcSiteAlarmStreamChannelCapacity { get; set; } = 20_000; + /// Akka.Remote transport heartbeat interval. public TimeSpan TransportHeartbeatInterval { get; set; } = TimeSpan.FromSeconds(5); @@ -185,4 +200,12 @@ public class CommunicationOptions /// reconcile publishes are always immediate. Default 250 ms. /// public TimeSpan LiveAlarmCachePublishCoalesce { get; set; } = TimeSpan.FromMilliseconds(250); + + /// + /// Random jitter added to each per-site reconcile tick, as a fraction of + /// (WP2.3). Without it every aggregator + /// started by one central failover fans its whole-site snapshot out on the same 60s + /// boundary forever. Zero disables jitter. + /// + public double LiveAlarmCacheReconcileJitterFraction { get; set; } = 0.2; } diff --git a/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteCommandDtoMapper.cs b/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteCommandDtoMapper.cs index 6bbd6a73..97508ba8 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteCommandDtoMapper.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteCommandDtoMapper.cs @@ -1620,7 +1620,10 @@ public static class SiteCommandDtoMapper return new DebugSnapshotRequestDto { InstanceUniqueName = request.InstanceUniqueName, - CorrelationId = request.CorrelationId + CorrelationId = request.CorrelationId, + // Additive field 3 — proto3 false default means an older site that does + // not know the field simply returns the full snapshot (correct, just fatter). + AlarmsOnly = request.AlarmsOnly }; } @@ -1631,7 +1634,7 @@ public static class SiteCommandDtoMapper { ArgumentNullException.ThrowIfNull(dto); - return new DebugSnapshotRequest(dto.InstanceUniqueName, dto.CorrelationId); + return new DebugSnapshotRequest(dto.InstanceUniqueName, dto.CorrelationId, dto.AlarmsOnly); } /// Projects a onto the wire. diff --git a/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcClient.cs b/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcClient.cs index 5763c6c4..667fd1d3 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcClient.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcClient.cs @@ -245,13 +245,21 @@ public class SiteStreamGrpcClient : IAsyncDisposable, IDisposable /// ; see . /// /// Cancellation token to stop the subscription. + /// + /// Optional callback invoked once when the site has ACCEPTED the subscription (response + /// headers received — the site writes them as soon as its relay actor is subscribed, so + /// no event can be missed after this point). The per-site aggregator uses it to run + /// exactly one re-seed per successful (re)connect instead of one per reconnect attempt. + /// Never invoked more than once per call, and never after . + /// /// A task that represents the asynchronous operation. public virtual async Task SubscribeSiteAsync( string correlationId, Action onAlarmEvent, Action onError, Action onCompleted, - CancellationToken ct) + CancellationToken ct, + Action? onConnected = null) { if (_client is null) throw new InvalidOperationException("Cannot subscribe on a test-only client."); @@ -275,7 +283,8 @@ public class SiteStreamGrpcClient : IAsyncDisposable, IDisposable onAlarmEvent(alarm); }, onError, - onCompleted); + onCompleted, + onConnected); } /// @@ -301,6 +310,13 @@ public class SiteStreamGrpcClient : IAsyncDisposable, IDisposable /// Invoked per wire event. /// Invoked once if the stream faulted. /// Invoked once if the server ended the stream with OK. + /// + /// Optional; invoked once when the server's response headers arrive — i.e. the site has + /// accepted the subscription and its relay actor is attached. Bounded by + /// so a peer that defers headers (a pre-WP2.3 site, + /// which only flushes them with its first event) still reports connected instead of + /// leaving the caller waiting for a signal that may never come on a quiet site. + /// /// A task that completes when the stream has ended and its outcome been reported. internal async Task ConsumeStreamAsync( string correlationId, @@ -308,13 +324,20 @@ public class SiteStreamGrpcClient : IAsyncDisposable, IDisposable Func> openCall, Action onEvent, Action onError, - Action onCompleted) + Action onCompleted, + Action? onConnected = null) { var completedGracefully = false; try { using (var call = openCall()) { + if (onConnected is not null) + { + await AwaitHeadersAsync(call, cts.Token).ConfigureAwait(false); + onConnected(); + } + await foreach (var evt in call.ResponseStream.ReadAllAsync(cts.Token)) { onEvent(evt); @@ -348,6 +371,38 @@ public class SiteStreamGrpcClient : IAsyncDisposable, IDisposable onCompleted(); } + /// + /// How long to wait for response headers before treating the stream as connected anyway. + /// A peer that only flushes headers with its first message would otherwise hold the + /// connected signal — and with it the aggregator's re-seed — for as long as the site + /// happens to be quiet. + /// + internal static TimeSpan ConnectedHeaderTimeout { get; set; } = TimeSpan.FromSeconds(10); + + /// + /// Awaits the call's response headers, bounded by . + /// A fault propagates (the caller reports it through onError like any other stream + /// fault); a timeout returns normally. On timeout the abandoned headers task is observed + /// so a later fault on it can never surface as an unobserved task exception. + /// + private static async Task AwaitHeadersAsync( + AsyncServerStreamingCall call, CancellationToken ct) + { + var headers = call.ResponseHeadersAsync; + try + { + await headers.WaitAsync(ConnectedHeaderTimeout, ct).ConfigureAwait(false); + } + catch (TimeoutException) + { + _ = headers.ContinueWith( + t => _ = t.Exception, + CancellationToken.None, + TaskContinuationOptions.OnlyOnFaulted | TaskContinuationOptions.ExecuteSynchronously, + TaskScheduler.Default); + } + } + /// /// Cancels an active subscription by correlation ID. /// diff --git a/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcServer.cs b/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcServer.cs index 0fb89c4a..48178262 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcServer.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcServer.cs @@ -27,6 +27,8 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase private readonly ConcurrentDictionary _activeStreams = new(); private readonly int _maxConcurrentStreams; private readonly TimeSpan _maxStreamLifetime; + private readonly int _instanceChannelCapacity; + private readonly int _siteAlarmChannelCapacity; private volatile bool _ready; // Flipped by CancelAllStreams() when the host enters // CoordinatedShutdown so SubscribeInstance refuses new streams with @@ -72,10 +74,17 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase ISiteStreamSubscriber streamSubscriber, ILogger logger, int maxConcurrentStreams = 100) - : this(streamSubscriber, logger, maxConcurrentStreams, TimeSpan.FromHours(4)) + : this(streamSubscriber, logger, maxConcurrentStreams, TimeSpan.FromHours(4), + DefaultInstanceChannelCapacity, DefaultSiteAlarmChannelCapacity) { } + /// Fallback Debug View send-channel capacity when no options are bound. + internal const int DefaultInstanceChannelCapacity = 1000; + + /// Fallback site-wide alarm send-channel capacity when no options are bound. + internal const int DefaultSiteAlarmChannelCapacity = 20_000; + /// /// DI constructor — binds /// and so the documented @@ -91,7 +100,9 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase IOptions options) : this(streamSubscriber, logger, options.Value.GrpcMaxConcurrentStreams, - options.Value.GrpcMaxStreamLifetime) + options.Value.GrpcMaxStreamLifetime, + options.Value.GrpcInstanceStreamChannelCapacity, + options.Value.GrpcSiteAlarmStreamChannelCapacity) { } @@ -99,12 +110,16 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase ISiteStreamSubscriber streamSubscriber, ILogger logger, int maxConcurrentStreams, - TimeSpan maxStreamLifetime) + TimeSpan maxStreamLifetime, + int instanceChannelCapacity, + int siteAlarmChannelCapacity) { _streamSubscriber = streamSubscriber; _logger = logger; _maxConcurrentStreams = maxConcurrentStreams; _maxStreamLifetime = maxStreamLifetime; + _instanceChannelCapacity = Math.Max(1, instanceChannelCapacity); + _siteAlarmChannelCapacity = Math.Max(1, siteAlarmChannelCapacity); } /// @@ -209,6 +224,21 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase /// Effective per-stream session lifetime. Exposed for tests. internal TimeSpan MaxStreamLifetime => _maxStreamLifetime; + /// Effective Debug View send-channel capacity. Exposed for tests. + internal int InstanceChannelCapacity => _instanceChannelCapacity; + + /// Effective site-wide alarm send-channel capacity. Exposed for tests. + internal int SiteAlarmChannelCapacity => _siteAlarmChannelCapacity; + + /// + /// Total events evicted from stream send channels on this node since start (both stream + /// kinds). Exported as scadabridge.site.stream.events_dropped; exposed here so a + /// test — and an operator via diagnostics — can read the raw count. + /// + public long DroppedStreamEventCount => Interlocked.Read(ref _droppedStreamEvents); + + private long _droppedStreamEvents; + /// public override Task SubscribeInstance( InstanceStreamRequest request, @@ -219,7 +249,9 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase responseStream, context, relay => _streamSubscriber.Subscribe(request.InstanceUniqueName, relay), - request.InstanceUniqueName); + request.InstanceUniqueName, + _instanceChannelCapacity, + streamKind: "instance"); /// public override Task SubscribeSite( @@ -234,7 +266,12 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase // already drops IsConfiguredPlaceholder rows and maps only the // enriched AlarmStateUpdate, so it is reused unchanged. _streamSubscriber.SubscribeSiteAlarms, - "site-wide alarms"); + "site-wide alarms", + // Its OWN, much larger channel (WP2.3): sharing the Debug View's 1000-slot + // DropOldest meant an alarm burst during a WAN stall silently evicted operator- + // visible transitions to make room for diagnostics traffic. + _siteAlarmChannelCapacity, + streamKind: "site-alarms"); /// /// Shared streaming pipeline behind and @@ -250,12 +287,16 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase /// The server call context (carries the client cancellation token). /// Subscribes the relay actor to the hub, returning a subscription id. /// Human-readable subscription description for logging. + /// Send-channel capacity for this stream kind. + /// Telemetry tag for this stream kind (instance/site-alarms). private async Task RunSubscriptionStreamAsync( string correlationId, IServerStreamWriter responseStream, ServerCallContext context, Func subscribe, - string description) + string description, + int channelCapacity, + string streamKind) { if (!_ready) throw new RpcException(new GrpcStatus(StatusCode.Unavailable, "Server not ready")); @@ -302,16 +343,20 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase long dropped = 0; var channel = Channel.CreateBounded( - new BoundedChannelOptions(1000) { FullMode = BoundedChannelFullMode.DropOldest }, + new BoundedChannelOptions(channelCapacity) { FullMode = BoundedChannelFullMode.DropOldest }, _ => { // Lossy-under-backpressure is the debug-view spec; make the real - // loss visible: first eviction + every 500th thereafter. + // loss visible: first eviction + every 500th thereafter, plus a + // per-kind counter on the node (WP2.3). var n = Interlocked.Increment(ref dropped); + Interlocked.Increment(ref _droppedStreamEvents); + ScadaBridgeTelemetry.RecordSiteStreamEventDropped(streamKind); if (n == 1 || n % 500 == 0) _logger.LogWarning( - "Debug stream {CorrelationId} backpressure: {Dropped} oldest event(s) evicted so far", - correlationId, n); + "Stream {CorrelationId} ({StreamKind}) backpressure: {Dropped} oldest event(s) evicted so far " + + "of a {Capacity}-slot channel", + correlationId, streamKind, n, channelCapacity); }); var actorSeq = Interlocked.Increment(ref _actorCounter); @@ -348,6 +393,25 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase "Stream {CorrelationId} started for {Description} (subscription {SubscriptionId})", correlationId, description, subscriptionId); + // Flush response headers NOW, while the relay is attached and before the first event + // (WP2.3). This is the client's "the site accepted my subscription" signal — the + // per-site aggregator hangs its once-per-successful-reconnect re-seed off it. Without + // an explicit flush, ASP.NET Core defers headers to the first written message, so a + // quiet site would never report connected. Best-effort: a client that vanished between + // Subscribe and here fails the write, and the read loop below handles the teardown. + try + { + await context.WriteResponseHeadersAsync(Metadata.Empty); + } + catch (Exception ex) + { + // Deliberately catch-all: an unsent header must never leak the relay actor and + // the _activeStreams entry, which the finally below owns. Any real transport + // failure resurfaces on the read/write loop immediately after. + _logger.LogDebug(ex, + "Could not flush response headers for stream {CorrelationId}; continuing.", correlationId); + } + // Telemetry follow-on: the connection is now fully established (Subscribe // succeeded, so no leak via the catch above). Count it up here and balance // it in the finally below so the scadabridge.site.connection.up gauge is @@ -519,17 +583,46 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase ? DateTime.SpecifyKind(request.SinceUtc.ToDateTime(), DateTimeKind.Utc) : DateTime.MinValue; + // Composite-keyset cursor (WP2.3), mirroring PullSiteCalls: proto3 + // defaults an unset string to "", which the queue treats as "no cursor" + // (legacy inclusive >= behaviour), so an older central is unaffected. + var afterId = string.IsNullOrEmpty(request.AfterId) ? null : request.AfterId; + + // AT-LEAST-ONCE (WP2.3). The incoming cursor is central's receipt for + // everything at or before it — that is the ONLY proof the site accepts. + // Rows are flipped to Reconciled here, at the START of the NEXT pull, + // instead of right after the previous response was projected: a fault + // between the response leaving the site and central committing it used + // to lose the rows outright, because they were already Reconciled and + // ReadPendingSinceAsync would never serve them again. The flip runs + // before the read so the rows it retires do not consume this batch's + // budget. Best-effort — a failure here only costs a re-ship, which + // central dedups on EventId. + if (since > DateTime.MinValue) + { + try + { + await queue.MarkReconciledUpToAsync(since, afterId, context.CancellationToken); + } + catch (Exception ex) + { + _logger.LogWarning(ex, + "MarkReconciledUpToAsync failed for cursor since={Since} afterId={AfterId}; rows stay pending for the next pull.", + since, afterId); + } + } + IReadOnlyList events; try { events = await queue.ReadPendingSinceAsync( - since, request.BatchSize, context.CancellationToken); + since, request.BatchSize, afterId, context.CancellationToken); } catch (Exception ex) { _logger.LogError(ex, - "ReadPendingSinceAsync failed for since={Since} batch={Batch}; returning empty response.", - since, request.BatchSize); + "ReadPendingSinceAsync failed for since={Since} batch={Batch} afterId={AfterId}; returning empty response.", + since, request.BatchSize, afterId); return new PullAuditEventsResponse(); } @@ -537,7 +630,8 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase { // batch_size saturated → tell central to issue a follow-up pull // with an advanced cursor. The site doesn't compute the cursor — - // central walks it forward from the last returned OccurredAtUtc. + // central walks it forward from the last returned OccurredAtUtc + // (plus, once it sets after_id, that row's EventId). MoreAvailable = events.Count >= request.BatchSize, }; foreach (var evt in events) @@ -545,31 +639,6 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase response.Events.Add(AuditEventDtoMapper.ToDto(evt)); } - // Flip to Reconciled AFTER projecting the response so a fault below the - // try/catch (mid-response, mid-flip) leaves the rows in Pending/Forwarded - // and central pulls them again next cycle. The flip itself is - // best-effort — its failure is a warning, not a fault, because central - // will dedup on EventId on the next pull. - var ids = new List(events.Count); - foreach (var evt in events) - { - ids.Add(evt.EventId); - } - - if (ids.Count > 0) - { - try - { - await queue.MarkReconciledAsync(ids, context.CancellationToken); - } - catch (Exception ex) - { - _logger.LogWarning(ex, - "MarkReconciledAsync failed after PullAuditEvents response of {Count} rows; rows stay Pending for retry.", - ids.Count); - } - } - return response; } diff --git a/src/ZB.MOM.WW.ScadaBridge.Communication/Protos/site_command.proto b/src/ZB.MOM.WW.ScadaBridge.Communication/Protos/site_command.proto index 7f86b93f..8c6fa912 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Communication/Protos/site_command.proto +++ b/src/ZB.MOM.WW.ScadaBridge.Communication/Protos/site_command.proto @@ -553,6 +553,14 @@ message EventLogQueryResponseDto { message DebugSnapshotRequestDto { string instance_unique_name = 1; string correlation_id = 2; + // Alarms-only projection (WP2.3 wire efficiency): when true the site builds + // and returns ONLY the alarm rows of the debug snapshot, leaving + // attribute_values empty. Used by the central per-site live alarm cache, + // whose seed/reconcile fan-out discards every attribute row anyway — a full + // snapshot ships the whole attribute surface of every enabled instance once + // per reconcile for nothing. proto3 defaults it to false, so an older + // central that never sets it keeps the full-snapshot behaviour. Additive-only. + bool alarms_only = 3; } message SubscribeDebugViewRequestDto { diff --git a/src/ZB.MOM.WW.ScadaBridge.Communication/Protos/sitestream.proto b/src/ZB.MOM.WW.ScadaBridge.Communication/Protos/sitestream.proto index 27587337..850816da 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Communication/Protos/sitestream.proto +++ b/src/ZB.MOM.WW.ScadaBridge.Communication/Protos/sitestream.proto @@ -164,13 +164,26 @@ message CachedTelemetryBatch { repeated CachedTelemetryPacket packets = 1; } // Audit Log (#23) M6 reconciliation pull: central→site request for any // site-local AuditLog rows with OccurredAtUtc >= since_utc that have not yet -// been ingested centrally (ForwardState in {Pending, Forwarded}). The site -// flips returned rows to Reconciled after the response is on the wire. +// been ingested centrally (ForwardState in {Pending, Forwarded}). Rows are NOT +// flipped to Reconciled when they are served — only when a LATER pull's cursor +// proves central consumed them (see after_id), so a fault between the response +// leaving the site and central committing it re-serves the rows instead of +// silently losing them (at-least-once). // more_available signals batch_size was saturated so the caller knows to // issue a follow-up pull with an advanced since_utc cursor. message PullAuditEventsRequest { google.protobuf.Timestamp since_utc = 1; int32 batch_size = 2; + // Composite-keyset cursor (WP2.3), mirroring PullSiteCallsRequest.after_id: + // the EventId ("D" GUID form) of the last row central has already CONSUMED at + // since_utc. When set, the site returns only rows strictly after the composite + // (OccurredAtUtc, EventId) pair — un-pinning a batch that would otherwise stall + // when more than batch_size rows share one since_utc instant — AND treats the + // cursor as proof of receipt: everything at or before it is flipped to + // Reconciled. Empty (the proto3 string default) preserves the legacy inclusive + // >= behaviour, under which only rows strictly older than since_utc are proven + // received. Additive-only. + string after_id = 3; } message PullAuditEventsResponse { diff --git a/src/ZB.MOM.WW.ScadaBridge.Communication/SiteAlarmLiveCacheService.cs b/src/ZB.MOM.WW.ScadaBridge.Communication/SiteAlarmLiveCacheService.cs index c3fbb7f3..c71071ee 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Communication/SiteAlarmLiveCacheService.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Communication/SiteAlarmLiveCacheService.cs @@ -132,7 +132,7 @@ public sealed class SiteAlarmLiveCacheService : ISiteAlarmLiveCache public bool IsLive(int siteId) { lock (_lock) - return _sites.TryGetValue(siteId, out var entry) && entry.HasPublished; + return _sites.TryGetValue(siteId, out var entry) && entry.HasPublished && entry.StreamLive; } // ── Subscriber teardown ───────────────────────────────────────────────────── @@ -213,8 +213,8 @@ public sealed class SiteAlarmLiveCacheService : ISiteAlarmLiveCache Func>> seedFn = ct => FanOutSnapshotsAsync(entry.SiteId, ct); - Action> publish = - snapshot => OnPublish(entry, snapshot); + Action, bool> publish = + (snapshot, streamLive) => OnPublish(entry, snapshot, streamLive); lock (_lock) { @@ -236,7 +236,8 @@ public sealed class SiteAlarmLiveCacheService : ISiteAlarmLiveCache _options.LiveAlarmCacheReconcileInterval, _options.LiveAlarmCachePublishCoalesce, TimeSpan.FromSeconds(5), // reconnect delay — former ReconnectDelay static default - TimeSpan.FromSeconds(60))); // stability window — former StabilityWindow static default + TimeSpan.FromSeconds(60), // stability window — former StabilityWindow static default + _options.LiveAlarmCacheReconcileJitterFraction)); entry.Actor = system.ActorOf(props, $"site-alarm-aggregator-{entry.SiteId}-{Guid.NewGuid():N}"); @@ -378,7 +379,12 @@ public sealed class SiteAlarmLiveCacheService : ISiteAlarmLiveCache await gate.WaitAsync(ct); try { - var request = new DebugSnapshotRequest(instanceUniqueName, Guid.NewGuid().ToString("N")); + // Alarms-only (WP2.3): this fan-out discards snapshot.AttributeValues, so + // asking the site to build and ship them is pure wire waste. The flag is + // additive on the wire; a pre-WP2.3 site ignores it and returns the full + // snapshot, which this loop reads exactly as before. + var request = new DebugSnapshotRequest( + instanceUniqueName, Guid.NewGuid().ToString("N"), AlarmsOnly: true); var snapshot = await _communicationService.RequestDebugSnapshotAsync(siteIdentifier, request, ct); if (snapshot.InstanceNotFound) return; @@ -402,7 +408,7 @@ public sealed class SiteAlarmLiveCacheService : ISiteAlarmLiveCache // ── Publish (actor thread → viewers) ──────────────────────────────────────── - private void OnPublish(SiteEntry entry, IReadOnlyList snapshot) + private void OnPublish(SiteEntry entry, IReadOnlyList snapshot, bool streamLive) { Subscription[] subscribers; lock (_lock) @@ -410,6 +416,10 @@ public sealed class SiteAlarmLiveCacheService : ISiteAlarmLiveCache // Store the fresh immutable snapshot (readers get it lock-free-ish via GetCurrentAlarms). entry.Current = snapshot; entry.HasPublished = true; + // Liveness is the STREAM's, not the cache's: a completed or given-up stream must + // stop reporting live between reopen ticks, or the page grafts a freezing snapshot + // over fresh poll data (WP2.3 carried residual). + entry.StreamLive = streamLive; subscribers = entry.Subscribers.ToArray(); } @@ -462,6 +472,7 @@ public sealed class SiteAlarmLiveCacheService : ISiteAlarmLiveCache entry.Actor = null; entry.HasPublished = false; + entry.StreamLive = false; entry.Current = Empty; if (entry.Subscribers.Count > 0 && !entry.Starting) @@ -498,6 +509,13 @@ public sealed class SiteAlarmLiveCacheService : ISiteAlarmLiveCache /// True once the aggregator has seeded and published at least once. public bool HasPublished { get; set; } + /// + /// Liveness of the aggregator's site-wide gRPC stream as of the last publish. Both + /// this and must hold for IsLive: a published cache + /// behind a dead stream is stale, not live. + /// + public bool StreamLive { get; set; } + public Timer? LingerTimer { get; set; } public int LingerVersion { get; set; } diff --git a/src/ZB.MOM.WW.ScadaBridge.Communication/SiteCommandGrpc/SiteCommand.cs b/src/ZB.MOM.WW.ScadaBridge.Communication/SiteCommandGrpc/SiteCommand.cs index d8e45300..fda7176e 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Communication/SiteCommandGrpc/SiteCommand.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Communication/SiteCommandGrpc/SiteCommand.cs @@ -260,241 +260,242 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { "cmlkZ2Uuc2l0ZWNvbW1hbmQudjEuRXZlbnRMb2dFbnRyeUR0bxIaChJjb250", "aW51YXRpb25fdG9rZW4YBCABKAkSEAoIaGFzX21vcmUYBSABKAgSDwoHc3Vj", "Y2VzcxgGIAEoCBIVCg1lcnJvcl9tZXNzYWdlGAcgASgJEi0KCXRpbWVzdGFt", - "cBgIIAEoCzIaLmdvb2dsZS5wcm90b2J1Zi5UaW1lc3RhbXAiTwoXRGVidWdT", + "cBgIIAEoCzIaLmdvb2dsZS5wcm90b2J1Zi5UaW1lc3RhbXAiZAoXRGVidWdT", "bmFwc2hvdFJlcXVlc3REdG8SHAoUaW5zdGFuY2VfdW5pcXVlX25hbWUYASAB", - "KAkSFgoOY29ycmVsYXRpb25faWQYAiABKAkiVAocU3Vic2NyaWJlRGVidWdW", - "aWV3UmVxdWVzdER0bxIcChRpbnN0YW5jZV91bmlxdWVfbmFtZRgBIAEoCRIW", - "Cg5jb3JyZWxhdGlvbl9pZBgCIAEoCSJWCh5VbnN1YnNjcmliZURlYnVnVmll", - "d1JlcXVlc3REdG8SHAoUaW5zdGFuY2VfdW5pcXVlX25hbWUYASABKAkSFgoO", - "Y29ycmVsYXRpb25faWQYAiABKAkiHAoaVW5zdWJzY3JpYmVEZWJ1Z1ZpZXdB", - "Y2tEdG8i1AEKFkFsYXJtQ29uZGl0aW9uU3RhdGVEdG8SDgoGYWN0aXZlGAEg", - "ASgIEhQKDGFja25vd2xlZGdlZBgCIAEoCBItCgljb25maXJtZWQYAyABKAsy", - "Gi5nb29nbGUucHJvdG9idWYuQm9vbFZhbHVlEj8KBnNoZWx2ZRgEIAEoDjIv", - "LnNjYWRhYnJpZGdlLnNpdGVjb21tYW5kLnYxLkFsYXJtU2hlbHZlU3RhdGVE", - "dG8SEgoKc3VwcHJlc3NlZBgFIAEoCBIQCghzZXZlcml0eRgGIAEoBSLdAQoW", - "RGVidWdBdHRyaWJ1dGVWYWx1ZUR0bxIcChRpbnN0YW5jZV91bmlxdWVfbmFt", - "ZRgBIAEoCRIWCg5hdHRyaWJ1dGVfcGF0aBgCIAEoCRIWCg5hdHRyaWJ1dGVf", - "bmFtZRgDIAEoCRI1CgV2YWx1ZRgEIAEoCzImLnNjYWRhYnJpZGdlLnNpdGVj", - "b21tYW5kLnYxLkxvb3NlVmFsdWUSDwoHcXVhbGl0eRgFIAEoCRItCgl0aW1l", - "c3RhbXAYBiABKAsyGi5nb29nbGUucHJvdG9idWYuVGltZXN0YW1wIq8FChJE", - "ZWJ1Z0FsYXJtU3RhdGVEdG8SHAoUaW5zdGFuY2VfdW5pcXVlX25hbWUYASAB", - "KAkSEgoKYWxhcm1fbmFtZRgCIAEoCRI4CgVzdGF0ZRgDIAEoDjIpLnNjYWRh", - "YnJpZGdlLnNpdGVjb21tYW5kLnYxLkFsYXJtU3RhdGVEdG8SEAoIcHJpb3Jp", - "dHkYBCABKAUSLQoJdGltZXN0YW1wGAUgASgLMhouZ29vZ2xlLnByb3RvYnVm", - "LlRpbWVzdGFtcBI4CgVsZXZlbBgGIAEoDjIpLnNjYWRhYnJpZGdlLnNpdGVj", - "b21tYW5kLnYxLkFsYXJtTGV2ZWxEdG8SDwoHbWVzc2FnZRgHIAEoCRI2CgRr", - "aW5kGAggASgOMiguc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuQWxhcm1L", - "aW5kRHRvEkUKCWNvbmRpdGlvbhgJIAEoCzIyLnNjYWRhYnJpZGdlLnNpdGVj", - "b21tYW5kLnYxLkFsYXJtQ29uZGl0aW9uU3RhdGVEdG8SGAoQc291cmNlX3Jl", - "ZmVyZW5jZRgKIAEoCRIXCg9hbGFybV90eXBlX25hbWUYCyABKAkSEAoIY2F0", - "ZWdvcnkYDCABKAkSFQoNb3BlcmF0b3JfdXNlchgNIAEoCRIYChBvcGVyYXRv", - "cl9jb21tZW50GA4gASgJEjcKE29yaWdpbmFsX3JhaXNlX3RpbWUYDyABKAsy", - "Gi5nb29nbGUucHJvdG9idWYuVGltZXN0YW1wEhUKDWN1cnJlbnRfdmFsdWUY", - "ECABKAkSEwoLbGltaXRfdmFsdWUYESABKAkSJAocbmF0aXZlX3NvdXJjZV9j", - "YW5vbmljYWxfbmFtZRgSIAEoCRIhChlpc19jb25maWd1cmVkX3BsYWNlaG9s", - "ZGVyGBMgASgIIpwCChREZWJ1Z1ZpZXdTbmFwc2hvdER0bxIcChRpbnN0YW5j", - "ZV91bmlxdWVfbmFtZRgBIAEoCRJMChBhdHRyaWJ1dGVfdmFsdWVzGAIgAygL", - "MjIuc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuRGVidWdBdHRyaWJ1dGVW", - "YWx1ZUR0bxJECgxhbGFybV9zdGF0ZXMYAyADKAsyLi5zY2FkYWJyaWRnZS5z", - "aXRlY29tbWFuZC52MS5EZWJ1Z0FsYXJtU3RhdGVEdG8SNgoSc25hcHNob3Rf", - "dGltZXN0YW1wGAQgASgLMhouZ29vZ2xlLnByb3RvYnVmLlRpbWVzdGFtcBIa", - "ChJpbnN0YW5jZV9ub3RfZm91bmQYBSABKAgi8AIKDFF1ZXJ5UmVxdWVzdBJO", - "Cg9ldmVudF9sb2dfcXVlcnkYASABKAsyMy5zY2FkYWJyaWRnZS5zaXRlY29t", - "bWFuZC52MS5FdmVudExvZ1F1ZXJ5UmVxdWVzdER0b0gAEk0KDmRlYnVnX3Nu", - "YXBzaG90GAIgASgLMjMuc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuRGVi", - "dWdTbmFwc2hvdFJlcXVlc3REdG9IABJYChRzdWJzY3JpYmVfZGVidWdfdmll", - "dxgDIAEoCzI4LnNjYWRhYnJpZGdlLnNpdGVjb21tYW5kLnYxLlN1YnNjcmli", - "ZURlYnVnVmlld1JlcXVlc3REdG9IABJcChZ1bnN1YnNjcmliZV9kZWJ1Z192", - "aWV3GAQgASgLMjouc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuVW5zdWJz", - "Y3JpYmVEZWJ1Z1ZpZXdSZXF1ZXN0RHRvSABCCQoHY29tbWFuZCKRAgoKUXVl", - "cnlSZXBseRJPCg9ldmVudF9sb2dfcXVlcnkYASABKAsyNC5zY2FkYWJyaWRn", - "ZS5zaXRlY29tbWFuZC52MS5FdmVudExvZ1F1ZXJ5UmVzcG9uc2VEdG9IABJP", - "ChNkZWJ1Z192aWV3X3NuYXBzaG90GAIgASgLMjAuc2NhZGFicmlkZ2Uuc2l0", - "ZWNvbW1hbmQudjEuRGVidWdWaWV3U25hcHNob3REdG9IABJYChZ1bnN1YnNj", - "cmliZV9kZWJ1Z192aWV3GAMgASgLMjYuc2NhZGFicmlkZ2Uuc2l0ZWNvbW1h", - "bmQudjEuVW5zdWJzY3JpYmVEZWJ1Z1ZpZXdBY2tEdG9IAEIHCgVyZXBseSKe", - "AQocUGFya2VkTWVzc2FnZVF1ZXJ5UmVxdWVzdER0bxIWCg5jb3JyZWxhdGlv", - "bl9pZBgBIAEoCRIPCgdzaXRlX2lkGAIgASgJEhMKC3BhZ2VfbnVtYmVyGAMg", - "ASgFEhEKCXBhZ2Vfc2l6ZRgEIAEoBRItCgl0aW1lc3RhbXAYBSABKAsyGi5n", - "b29nbGUucHJvdG9idWYuVGltZXN0YW1wIvICChVQYXJrZWRNZXNzYWdlRW50", - "cnlEdG8SEgoKbWVzc2FnZV9pZBgBIAEoCRIVCg10YXJnZXRfc3lzdGVtGAIg", - "ASgJEhMKC21ldGhvZF9uYW1lGAMgASgJEhUKDWVycm9yX21lc3NhZ2UYBCAB", - "KAkSFQoNYXR0ZW1wdF9jb3VudBgFIAEoBRI2ChJvcmlnaW5hbF90aW1lc3Rh", - "bXAYBiABKAsyGi5nb29nbGUucHJvdG9idWYuVGltZXN0YW1wEjoKFmxhc3Rf", - "YXR0ZW1wdF90aW1lc3RhbXAYByABKAsyGi5nb29nbGUucHJvdG9idWYuVGlt", - "ZXN0YW1wEhQKDG1heF9hdHRlbXB0cxgIIAEoBRJICghjYXRlZ29yeRgJIAEo", - "DjI2LnNjYWRhYnJpZGdlLnNpdGVjb21tYW5kLnYxLlN0b3JlQW5kRm9yd2Fy", - "ZENhdGVnb3J5RHRvEhcKD29yaWdpbl9pbnN0YW5jZRgKIAEoCSKhAgodUGFy", - "a2VkTWVzc2FnZVF1ZXJ5UmVzcG9uc2VEdG8SFgoOY29ycmVsYXRpb25faWQY", - "ASABKAkSDwoHc2l0ZV9pZBgCIAEoCRJDCghtZXNzYWdlcxgDIAMoCzIxLnNj", - "YWRhYnJpZGdlLnNpdGVjb21tYW5kLnYxLlBhcmtlZE1lc3NhZ2VFbnRyeUR0", - "bxITCgt0b3RhbF9jb3VudBgEIAEoBRITCgtwYWdlX251bWJlchgFIAEoBRIR", - "CglwYWdlX3NpemUYBiABKAUSDwoHc3VjY2VzcxgHIAEoCBIVCg1lcnJvcl9t", - "ZXNzYWdlGAggASgJEi0KCXRpbWVzdGFtcBgJIAEoCzIaLmdvb2dsZS5wcm90", - "b2J1Zi5UaW1lc3RhbXAiigEKHFBhcmtlZE1lc3NhZ2VSZXRyeVJlcXVlc3RE", - "dG8SFgoOY29ycmVsYXRpb25faWQYASABKAkSDwoHc2l0ZV9pZBgCIAEoCRIS", - "CgptZXNzYWdlX2lkGAMgASgJEi0KCXRpbWVzdGFtcBgEIAEoCzIaLmdvb2ds", - "ZS5wcm90b2J1Zi5UaW1lc3RhbXAiXwodUGFya2VkTWVzc2FnZVJldHJ5UmVz", - "cG9uc2VEdG8SFgoOY29ycmVsYXRpb25faWQYASABKAkSDwoHc3VjY2VzcxgC", - "IAEoCBIVCg1lcnJvcl9tZXNzYWdlGAMgASgJIowBCh5QYXJrZWRNZXNzYWdl", - "RGlzY2FyZFJlcXVlc3REdG8SFgoOY29ycmVsYXRpb25faWQYASABKAkSDwoH", - "c2l0ZV9pZBgCIAEoCRISCgptZXNzYWdlX2lkGAMgASgJEi0KCXRpbWVzdGFt", - "cBgEIAEoCzIaLmdvb2dsZS5wcm90b2J1Zi5UaW1lc3RhbXAiYQofUGFya2Vk", - "TWVzc2FnZURpc2NhcmRSZXNwb25zZUR0bxIWCg5jb3JyZWxhdGlvbl9pZBgB", - "IAEoCRIPCgdzdWNjZXNzGAIgASgIEhUKDWVycm9yX21lc3NhZ2UYAyABKAki", - "TwoXUmV0cnlQYXJrZWRPcGVyYXRpb25EdG8SFgoOY29ycmVsYXRpb25faWQY", - "ASABKAkSHAoUdHJhY2tlZF9vcGVyYXRpb25faWQYAiABKAkiUQoZRGlzY2Fy", - "ZFBhcmtlZE9wZXJhdGlvbkR0bxIWCg5jb3JyZWxhdGlvbl9pZBgBIAEoCRIc", - "ChR0cmFja2VkX29wZXJhdGlvbl9pZBgCIAEoCSJdChtQYXJrZWRPcGVyYXRp", - "b25BY3Rpb25BY2tEdG8SFgoOY29ycmVsYXRpb25faWQYASABKAkSDwoHYXBw", - "bGllZBgCIAEoCBIVCg1lcnJvcl9tZXNzYWdlGAMgASgJIt4DCg1QYXJrZWRS", - "ZXF1ZXN0ElgKFHBhcmtlZF9tZXNzYWdlX3F1ZXJ5GAEgASgLMjguc2NhZGFi", - "cmlkZ2Uuc2l0ZWNvbW1hbmQudjEuUGFya2VkTWVzc2FnZVF1ZXJ5UmVxdWVz", - "dER0b0gAElgKFHBhcmtlZF9tZXNzYWdlX3JldHJ5GAIgASgLMjguc2NhZGFi", - "cmlkZ2Uuc2l0ZWNvbW1hbmQudjEuUGFya2VkTWVzc2FnZVJldHJ5UmVxdWVz", - "dER0b0gAElwKFnBhcmtlZF9tZXNzYWdlX2Rpc2NhcmQYAyABKAsyOi5zY2Fk", - "YWJyaWRnZS5zaXRlY29tbWFuZC52MS5QYXJrZWRNZXNzYWdlRGlzY2FyZFJl", - "cXVlc3REdG9IABJVChZyZXRyeV9wYXJrZWRfb3BlcmF0aW9uGAQgASgLMjMu", - "c2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuUmV0cnlQYXJrZWRPcGVyYXRp", - "b25EdG9IABJZChhkaXNjYXJkX3BhcmtlZF9vcGVyYXRpb24YBSABKAsyNS5z", - "Y2FkYWJyaWRnZS5zaXRlY29tbWFuZC52MS5EaXNjYXJkUGFya2VkT3BlcmF0", - "aW9uRHRvSABCCQoHY29tbWFuZCKHAwoLUGFya2VkUmVwbHkSWQoUcGFya2Vk", - "X21lc3NhZ2VfcXVlcnkYASABKAsyOS5zY2FkYWJyaWRnZS5zaXRlY29tbWFu", - "ZC52MS5QYXJrZWRNZXNzYWdlUXVlcnlSZXNwb25zZUR0b0gAElkKFHBhcmtl", - "ZF9tZXNzYWdlX3JldHJ5GAIgASgLMjkuc2NhZGFicmlkZ2Uuc2l0ZWNvbW1h", - "bmQudjEuUGFya2VkTWVzc2FnZVJldHJ5UmVzcG9uc2VEdG9IABJdChZwYXJr", - "ZWRfbWVzc2FnZV9kaXNjYXJkGAMgASgLMjsuc2NhZGFicmlkZ2Uuc2l0ZWNv", - "bW1hbmQudjEuUGFya2VkTWVzc2FnZURpc2NhcmRSZXNwb25zZUR0b0gAEloK", - "F3BhcmtlZF9vcGVyYXRpb25fYWN0aW9uGAQgASgLMjcuc2NhZGFicmlkZ2Uu", - "c2l0ZWNvbW1hbmQudjEuUGFya2VkT3BlcmF0aW9uQWN0aW9uQWNrRHRvSABC", - "BwoFcmVwbHki7QEKFVJvdXRlVG9DYWxsUmVxdWVzdER0bxIWCg5jb3JyZWxh", - "dGlvbl9pZBgBIAEoCRIcChRpbnN0YW5jZV91bmlxdWVfbmFtZRgCIAEoCRIT", - "CgtzY3JpcHRfbmFtZRgDIAEoCRI9CgpwYXJhbWV0ZXJzGAQgASgLMikuc2Nh", - "ZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuTG9vc2VWYWx1ZU1hcBItCgl0aW1l", - "c3RhbXAYBSABKAsyGi5nb29nbGUucHJvdG9idWYuVGltZXN0YW1wEhsKE3Bh", - "cmVudF9leGVjdXRpb25faWQYBiABKAkixQEKFlJvdXRlVG9DYWxsUmVzcG9u", - "c2VEdG8SFgoOY29ycmVsYXRpb25faWQYASABKAkSDwoHc3VjY2VzcxgCIAEo", - "CBI8CgxyZXR1cm5fdmFsdWUYAyABKAsyJi5zY2FkYWJyaWRnZS5zaXRlY29t", - "bWFuZC52MS5Mb29zZVZhbHVlEhUKDWVycm9yX21lc3NhZ2UYBCABKAkSLQoJ", - "dGltZXN0YW1wGAUgASgLMhouZ29vZ2xlLnByb3RvYnVmLlRpbWVzdGFtcCK7", - "AQoeUm91dGVUb0dldEF0dHJpYnV0ZXNSZXF1ZXN0RHRvEhYKDmNvcnJlbGF0", - "aW9uX2lkGAEgASgJEhwKFGluc3RhbmNlX3VuaXF1ZV9uYW1lGAIgASgJEhcK", - "D2F0dHJpYnV0ZV9uYW1lcxgDIAMoCRItCgl0aW1lc3RhbXAYBCABKAsyGi5n", - "b29nbGUucHJvdG9idWYuVGltZXN0YW1wEhsKE3BhcmVudF9leGVjdXRpb25f", - "aWQYBSABKAkiywEKH1JvdXRlVG9HZXRBdHRyaWJ1dGVzUmVzcG9uc2VEdG8S", - "FgoOY29ycmVsYXRpb25faWQYASABKAkSOQoGdmFsdWVzGAIgASgLMikuc2Nh", - "ZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuTG9vc2VWYWx1ZU1hcBIPCgdzdWNj", - "ZXNzGAMgASgIEhUKDWVycm9yX21lc3NhZ2UYBCABKAkSLQoJdGltZXN0YW1w", - "GAUgASgLMhouZ29vZ2xlLnByb3RvYnVmLlRpbWVzdGFtcCLFAgoeUm91dGVU", - "b1NldEF0dHJpYnV0ZXNSZXF1ZXN0RHRvEhYKDmNvcnJlbGF0aW9uX2lkGAEg", - "ASgJEhwKFGluc3RhbmNlX3VuaXF1ZV9uYW1lGAIgASgJEmkKEGF0dHJpYnV0", - "ZV92YWx1ZXMYAyADKAsyTy5zY2FkYWJyaWRnZS5zaXRlY29tbWFuZC52MS5S", - "b3V0ZVRvU2V0QXR0cmlidXRlc1JlcXVlc3REdG8uQXR0cmlidXRlVmFsdWVz", - "RW50cnkSLQoJdGltZXN0YW1wGAQgASgLMhouZ29vZ2xlLnByb3RvYnVmLlRp", - "bWVzdGFtcBIbChNwYXJlbnRfZXhlY3V0aW9uX2lkGAUgASgJGjYKFEF0dHJp", - "YnV0ZVZhbHVlc0VudHJ5EgsKA2tleRgBIAEoCRINCgV2YWx1ZRgCIAEoCToC", - "OAEikAEKH1JvdXRlVG9TZXRBdHRyaWJ1dGVzUmVzcG9uc2VEdG8SFgoOY29y", - "cmVsYXRpb25faWQYASABKAkSDwoHc3VjY2VzcxgCIAEoCBIVCg1lcnJvcl9t", - "ZXNzYWdlGAMgASgJEi0KCXRpbWVzdGFtcBgEIAEoCzIaLmdvb2dsZS5wcm90", - "b2J1Zi5UaW1lc3RhbXAiwwIKIVJvdXRlVG9XYWl0Rm9yQXR0cmlidXRlUmVx", - "dWVzdER0bxIWCg5jb3JyZWxhdGlvbl9pZBgBIAEoCRIcChRpbnN0YW5jZV91", - "bmlxdWVfbmFtZRgCIAEoCRIWCg5hdHRyaWJ1dGVfbmFtZRgDIAEoCRI6ChR0", - "YXJnZXRfdmFsdWVfZW5jb2RlZBgEIAEoCzIcLmdvb2dsZS5wcm90b2J1Zi5T", - "dHJpbmdWYWx1ZRIqCgd0aW1lb3V0GAUgASgLMhkuZ29vZ2xlLnByb3RvYnVm", - "LkR1cmF0aW9uEi0KCXRpbWVzdGFtcBgGIAEoCzIaLmdvb2dsZS5wcm90b2J1", - "Zi5UaW1lc3RhbXASGwoTcGFyZW50X2V4ZWN1dGlvbl9pZBgHIAEoCRIcChRy", - "ZXF1aXJlX2dvb2RfcXVhbGl0eRgIIAEoCCL/AQoiUm91dGVUb1dhaXRGb3JB", - "dHRyaWJ1dGVSZXNwb25zZUR0bxIWCg5jb3JyZWxhdGlvbl9pZBgBIAEoCRIP", - "CgdtYXRjaGVkGAIgASgIEjUKBXZhbHVlGAMgASgLMiYuc2NhZGFicmlkZ2Uu", - "c2l0ZWNvbW1hbmQudjEuTG9vc2VWYWx1ZRIPCgdxdWFsaXR5GAQgASgJEhEK", - "CXRpbWVkX291dBgFIAEoCBIPCgdzdWNjZXNzGAYgASgIEhUKDWVycm9yX21l", - "c3NhZ2UYByABKAkSLQoJdGltZXN0YW1wGAggASgLMhouZ29vZ2xlLnByb3Rv", - "YnVmLlRpbWVzdGFtcCKJAwoMUm91dGVSZXF1ZXN0EkoKDXJvdXRlX3RvX2Nh", - "bGwYASABKAsyMS5zY2FkYWJyaWRnZS5zaXRlY29tbWFuZC52MS5Sb3V0ZVRv", - "Q2FsbFJlcXVlc3REdG9IABJdChdyb3V0ZV90b19nZXRfYXR0cmlidXRlcxgC", - "IAEoCzI6LnNjYWRhYnJpZGdlLnNpdGVjb21tYW5kLnYxLlJvdXRlVG9HZXRB", - "dHRyaWJ1dGVzUmVxdWVzdER0b0gAEl0KF3JvdXRlX3RvX3NldF9hdHRyaWJ1", - "dGVzGAMgASgLMjouc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuUm91dGVU", - "b1NldEF0dHJpYnV0ZXNSZXF1ZXN0RHRvSAASZAobcm91dGVfdG9fd2FpdF9m", - "b3JfYXR0cmlidXRlGAQgASgLMj0uc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQu", - "djEuUm91dGVUb1dhaXRGb3JBdHRyaWJ1dGVSZXF1ZXN0RHRvSABCCQoHY29t", - "bWFuZCKJAwoKUm91dGVSZXBseRJLCg1yb3V0ZV90b19jYWxsGAEgASgLMjIu", - "c2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuUm91dGVUb0NhbGxSZXNwb25z", - "ZUR0b0gAEl4KF3JvdXRlX3RvX2dldF9hdHRyaWJ1dGVzGAIgASgLMjsuc2Nh", - "ZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuUm91dGVUb0dldEF0dHJpYnV0ZXNS", - "ZXNwb25zZUR0b0gAEl4KF3JvdXRlX3RvX3NldF9hdHRyaWJ1dGVzGAMgASgL", - "Mjsuc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuUm91dGVUb1NldEF0dHJp", - "YnV0ZXNSZXNwb25zZUR0b0gAEmUKG3JvdXRlX3RvX3dhaXRfZm9yX2F0dHJp", - "YnV0ZRgEIAEoCzI+LnNjYWRhYnJpZGdlLnNpdGVjb21tYW5kLnYxLlJvdXRl", - "VG9XYWl0Rm9yQXR0cmlidXRlUmVzcG9uc2VEdG9IAEIHCgVyZXBseSJBChZU", - "cmlnZ2VyU2l0ZUZhaWxvdmVyRHRvEhYKDmNvcnJlbGF0aW9uX2lkGAEgASgJ", - "Eg8KB3NpdGVfaWQYAiABKAkibQoSU2l0ZUZhaWxvdmVyQWNrRHRvEhYKDmNv", - "cnJlbGF0aW9uX2lkGAEgASgJEhAKCGFjY2VwdGVkGAIgASgIEhYKDnRhcmdl", - "dF9hZGRyZXNzGAMgASgJEhUKDWVycm9yX21lc3NhZ2UYBCABKAkqywEKE0Rl", - "cGxveW1lbnRTdGF0dXNEdG8SJQohREVQTE9ZTUVOVF9TVEFUVVNfRFRPX1VO", - "U1BFQ0lGSUVEEAASIQodREVQTE9ZTUVOVF9TVEFUVVNfRFRPX1BFTkRJTkcQ", - "ARIlCiFERVBMT1lNRU5UX1NUQVRVU19EVE9fSU5fUFJPR1JFU1MQAhIhCh1E", - "RVBMT1lNRU5UX1NUQVRVU19EVE9fU1VDQ0VTUxADEiAKHERFUExPWU1FTlRf", - "U1RBVFVTX0RUT19GQUlMRUQQBCrEAQoSQnJvd3NlTm9kZUNsYXNzRHRvEiUK", - "IUJST1dTRV9OT0RFX0NMQVNTX0RUT19VTlNQRUNJRklFRBAAEiAKHEJST1dT", - "RV9OT0RFX0NMQVNTX0RUT19PQkpFQ1QQARIiCh5CUk9XU0VfTk9ERV9DTEFT", - "U19EVE9fVkFSSUFCTEUQAhIgChxCUk9XU0VfTk9ERV9DTEFTU19EVE9fTUVU", - "SE9EEAMSHwobQlJPV1NFX05PREVfQ0xBU1NfRFRPX09USEVSEAQqoQIKFEJy", - "b3dzZUZhaWx1cmVLaW5kRHRvEicKI0JST1dTRV9GQUlMVVJFX0tJTkRfRFRP", - "X1VOU1BFQ0lGSUVEEAASMAosQlJPV1NFX0ZBSUxVUkVfS0lORF9EVE9fQ09O", - "TkVDVElPTl9OT1RfRk9VTkQQARI0CjBCUk9XU0VfRkFJTFVSRV9LSU5EX0RU", - "T19DT05ORUNUSU9OX05PVF9DT05ORUNURUQQAhIpCiVCUk9XU0VfRkFJTFVS", - "RV9LSU5EX0RUT19OT1RfQlJPV1NBQkxFEAMSIwofQlJPV1NFX0ZBSUxVUkVf", - "S0lORF9EVE9fVElNRU9VVBAEEigKJEJST1dTRV9GQUlMVVJFX0tJTkRfRFRP", - "X1NFUlZFUl9FUlJPUhAFKqoCChtSZWFkVGFnVmFsdWVzRmFpbHVyZUtpbmRE", - "dG8SMAosUkVBRF9UQUdfVkFMVUVTX0ZBSUxVUkVfS0lORF9EVE9fVU5TUEVD", - "SUZJRUQQABI5CjVSRUFEX1RBR19WQUxVRVNfRkFJTFVSRV9LSU5EX0RUT19D", - "T05ORUNUSU9OX05PVF9GT1VORBABEj0KOVJFQURfVEFHX1ZBTFVFU19GQUlM", - "VVJFX0tJTkRfRFRPX0NPTk5FQ1RJT05fTk9UX0NPTk5FQ1RFRBACEiwKKFJF", - "QURfVEFHX1ZBTFVFU19GQUlMVVJFX0tJTkRfRFRPX1RJTUVPVVQQAxIxCi1S", - "RUFEX1RBR19WQUxVRVNfRkFJTFVSRV9LSU5EX0RUT19TRVJWRVJfRVJST1IQ", - "BCqTAgoUVmVyaWZ5RmFpbHVyZUtpbmREdG8SJwojVkVSSUZZX0ZBSUxVUkVf", - "S0lORF9EVE9fVU5TUEVDSUZJRUQQABInCiNWRVJJRllfRkFJTFVSRV9LSU5E", - "X0RUT19VTlJFQUNIQUJMRRABEicKI1ZFUklGWV9GQUlMVVJFX0tJTkRfRFRP", - "X0FVVEhfRkFJTEVEEAISMQotVkVSSUZZX0ZBSUxVUkVfS0lORF9EVE9fVU5U", - "UlVTVEVEX0NFUlRJRklDQVRFEAMSIwofVkVSSUZZX0ZBSUxVUkVfS0lORF9E", - "VE9fVElNRU9VVBAEEigKJFZFUklGWV9GQUlMVVJFX0tJTkRfRFRPX1NFUlZF", - "Ul9FUlJPUhAFKuUBChpTdG9yZUFuZEZvcndhcmRDYXRlZ29yeUR0bxIuCipT", - "VE9SRV9BTkRfRk9SV0FSRF9DQVRFR09SWV9EVE9fVU5TUEVDSUZJRUQQABIy", - "Ci5TVE9SRV9BTkRfRk9SV0FSRF9DQVRFR09SWV9EVE9fRVhURVJOQUxfU1lT", - "VEVNEAESLworU1RPUkVfQU5EX0ZPUldBUkRfQ0FURUdPUllfRFRPX05PVElG", - "SUNBVElPThACEjIKLlNUT1JFX0FORF9GT1JXQVJEX0NBVEVHT1JZX0RUT19D", - "QUNIRURfREJfV1JJVEUQAypoCg1BbGFybVN0YXRlRHRvEh8KG0FMQVJNX1NU", - "QVRFX0RUT19VTlNQRUNJRklFRBAAEhoKFkFMQVJNX1NUQVRFX0RUT19BQ1RJ", - "VkUQARIaChZBTEFSTV9TVEFURV9EVE9fTk9STUFMEAIquQEKDUFsYXJtTGV2", - "ZWxEdG8SHwobQUxBUk1fTEVWRUxfRFRPX1VOU1BFQ0lGSUVEEAASGAoUQUxB", - "Uk1fTEVWRUxfRFRPX05PTkUQARIXChNBTEFSTV9MRVZFTF9EVE9fTE9XEAIS", - "GwoXQUxBUk1fTEVWRUxfRFRPX0xPV19MT1cQAxIYChRBTEFSTV9MRVZFTF9E", - "VE9fSElHSBAEEh0KGUFMQVJNX0xFVkVMX0RUT19ISUdIX0hJR0gQBSqSAQoM", - "QWxhcm1LaW5kRHRvEh4KGkFMQVJNX0tJTkRfRFRPX1VOU1BFQ0lGSUVEEAAS", - "GwoXQUxBUk1fS0lORF9EVE9fQ09NUFVURUQQARIgChxBTEFSTV9LSU5EX0RU", - "T19OQVRJVkVfT1BDX1VBEAISIwofQUxBUk1fS0lORF9EVE9fTkFUSVZFX01Y", - "X0FDQ0VTUxADKugBChNBbGFybVNoZWx2ZVN0YXRlRHRvEiYKIkFMQVJNX1NI", - "RUxWRV9TVEFURV9EVE9fVU5TUEVDSUZJRUQQABIkCiBBTEFSTV9TSEVMVkVf", - "U1RBVEVfRFRPX1VOU0hFTFZFRBABEisKJ0FMQVJNX1NIRUxWRV9TVEFURV9E", - "VE9fT05FX1NIT1RfU0hFTFZFRBACEigKJEFMQVJNX1NIRUxWRV9TVEFURV9E", - "VE9fVElNRURfU0hFTFZFRBADEiwKKEFMQVJNX1NIRUxWRV9TVEFURV9EVE9f", - "UEVSTUFORU5UX1NIRUxWRUQQBDKEBQoSU2l0ZUNvbW1hbmRTZXJ2aWNlEmwK", - "EEV4ZWN1dGVMaWZlY3ljbGUSLC5zY2FkYWJyaWRnZS5zaXRlY29tbWFuZC52", - "MS5MaWZlY3ljbGVSZXF1ZXN0Giouc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQu", - "djEuTGlmZWN5Y2xlUmVwbHkSYAoMRXhlY3V0ZU9wY1VhEiguc2NhZGFicmlk", - "Z2Uuc2l0ZWNvbW1hbmQudjEuT3BjVWFSZXF1ZXN0GiYuc2NhZGFicmlkZ2Uu", - "c2l0ZWNvbW1hbmQudjEuT3BjVWFSZXBseRJgCgxFeGVjdXRlUXVlcnkSKC5z", - "Y2FkYWJyaWRnZS5zaXRlY29tbWFuZC52MS5RdWVyeVJlcXVlc3QaJi5zY2Fk", - "YWJyaWRnZS5zaXRlY29tbWFuZC52MS5RdWVyeVJlcGx5EmMKDUV4ZWN1dGVQ", - "YXJrZWQSKS5zY2FkYWJyaWRnZS5zaXRlY29tbWFuZC52MS5QYXJrZWRSZXF1", - "ZXN0Gicuc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuUGFya2VkUmVwbHkS", - "YAoMRXhlY3V0ZVJvdXRlEiguc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEu", - "Um91dGVSZXF1ZXN0GiYuc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuUm91", - "dGVSZXBseRJ1Cg9UcmlnZ2VyRmFpbG92ZXISMi5zY2FkYWJyaWRnZS5zaXRl", - "Y29tbWFuZC52MS5UcmlnZ2VyU2l0ZUZhaWxvdmVyRHRvGi4uc2NhZGFicmlk", - "Z2Uuc2l0ZWNvbW1hbmQudjEuU2l0ZUZhaWxvdmVyQWNrRHRvQiuqAihaQi5N", - "T00uV1cuU2NhZGFCcmlkZ2UuQ29tbXVuaWNhdGlvbi5HcnBjYgZwcm90bzM=")); + "KAkSFgoOY29ycmVsYXRpb25faWQYAiABKAkSEwoLYWxhcm1zX29ubHkYAyAB", + "KAgiVAocU3Vic2NyaWJlRGVidWdWaWV3UmVxdWVzdER0bxIcChRpbnN0YW5j", + "ZV91bmlxdWVfbmFtZRgBIAEoCRIWCg5jb3JyZWxhdGlvbl9pZBgCIAEoCSJW", + "Ch5VbnN1YnNjcmliZURlYnVnVmlld1JlcXVlc3REdG8SHAoUaW5zdGFuY2Vf", + "dW5pcXVlX25hbWUYASABKAkSFgoOY29ycmVsYXRpb25faWQYAiABKAkiHAoa", + "VW5zdWJzY3JpYmVEZWJ1Z1ZpZXdBY2tEdG8i1AEKFkFsYXJtQ29uZGl0aW9u", + "U3RhdGVEdG8SDgoGYWN0aXZlGAEgASgIEhQKDGFja25vd2xlZGdlZBgCIAEo", + "CBItCgljb25maXJtZWQYAyABKAsyGi5nb29nbGUucHJvdG9idWYuQm9vbFZh", + "bHVlEj8KBnNoZWx2ZRgEIAEoDjIvLnNjYWRhYnJpZGdlLnNpdGVjb21tYW5k", + "LnYxLkFsYXJtU2hlbHZlU3RhdGVEdG8SEgoKc3VwcHJlc3NlZBgFIAEoCBIQ", + "CghzZXZlcml0eRgGIAEoBSLdAQoWRGVidWdBdHRyaWJ1dGVWYWx1ZUR0bxIc", + "ChRpbnN0YW5jZV91bmlxdWVfbmFtZRgBIAEoCRIWCg5hdHRyaWJ1dGVfcGF0", + "aBgCIAEoCRIWCg5hdHRyaWJ1dGVfbmFtZRgDIAEoCRI1CgV2YWx1ZRgEIAEo", + "CzImLnNjYWRhYnJpZGdlLnNpdGVjb21tYW5kLnYxLkxvb3NlVmFsdWUSDwoH", + "cXVhbGl0eRgFIAEoCRItCgl0aW1lc3RhbXAYBiABKAsyGi5nb29nbGUucHJv", + "dG9idWYuVGltZXN0YW1wIq8FChJEZWJ1Z0FsYXJtU3RhdGVEdG8SHAoUaW5z", + "dGFuY2VfdW5pcXVlX25hbWUYASABKAkSEgoKYWxhcm1fbmFtZRgCIAEoCRI4", + "CgVzdGF0ZRgDIAEoDjIpLnNjYWRhYnJpZGdlLnNpdGVjb21tYW5kLnYxLkFs", + "YXJtU3RhdGVEdG8SEAoIcHJpb3JpdHkYBCABKAUSLQoJdGltZXN0YW1wGAUg", + "ASgLMhouZ29vZ2xlLnByb3RvYnVmLlRpbWVzdGFtcBI4CgVsZXZlbBgGIAEo", + "DjIpLnNjYWRhYnJpZGdlLnNpdGVjb21tYW5kLnYxLkFsYXJtTGV2ZWxEdG8S", + "DwoHbWVzc2FnZRgHIAEoCRI2CgRraW5kGAggASgOMiguc2NhZGFicmlkZ2Uu", + "c2l0ZWNvbW1hbmQudjEuQWxhcm1LaW5kRHRvEkUKCWNvbmRpdGlvbhgJIAEo", + "CzIyLnNjYWRhYnJpZGdlLnNpdGVjb21tYW5kLnYxLkFsYXJtQ29uZGl0aW9u", + "U3RhdGVEdG8SGAoQc291cmNlX3JlZmVyZW5jZRgKIAEoCRIXCg9hbGFybV90", + "eXBlX25hbWUYCyABKAkSEAoIY2F0ZWdvcnkYDCABKAkSFQoNb3BlcmF0b3Jf", + "dXNlchgNIAEoCRIYChBvcGVyYXRvcl9jb21tZW50GA4gASgJEjcKE29yaWdp", + "bmFsX3JhaXNlX3RpbWUYDyABKAsyGi5nb29nbGUucHJvdG9idWYuVGltZXN0", + "YW1wEhUKDWN1cnJlbnRfdmFsdWUYECABKAkSEwoLbGltaXRfdmFsdWUYESAB", + "KAkSJAocbmF0aXZlX3NvdXJjZV9jYW5vbmljYWxfbmFtZRgSIAEoCRIhChlp", + "c19jb25maWd1cmVkX3BsYWNlaG9sZGVyGBMgASgIIpwCChREZWJ1Z1ZpZXdT", + "bmFwc2hvdER0bxIcChRpbnN0YW5jZV91bmlxdWVfbmFtZRgBIAEoCRJMChBh", + "dHRyaWJ1dGVfdmFsdWVzGAIgAygLMjIuc2NhZGFicmlkZ2Uuc2l0ZWNvbW1h", + "bmQudjEuRGVidWdBdHRyaWJ1dGVWYWx1ZUR0bxJECgxhbGFybV9zdGF0ZXMY", + "AyADKAsyLi5zY2FkYWJyaWRnZS5zaXRlY29tbWFuZC52MS5EZWJ1Z0FsYXJt", + "U3RhdGVEdG8SNgoSc25hcHNob3RfdGltZXN0YW1wGAQgASgLMhouZ29vZ2xl", + "LnByb3RvYnVmLlRpbWVzdGFtcBIaChJpbnN0YW5jZV9ub3RfZm91bmQYBSAB", + "KAgi8AIKDFF1ZXJ5UmVxdWVzdBJOCg9ldmVudF9sb2dfcXVlcnkYASABKAsy", + "My5zY2FkYWJyaWRnZS5zaXRlY29tbWFuZC52MS5FdmVudExvZ1F1ZXJ5UmVx", + "dWVzdER0b0gAEk0KDmRlYnVnX3NuYXBzaG90GAIgASgLMjMuc2NhZGFicmlk", + "Z2Uuc2l0ZWNvbW1hbmQudjEuRGVidWdTbmFwc2hvdFJlcXVlc3REdG9IABJY", + "ChRzdWJzY3JpYmVfZGVidWdfdmlldxgDIAEoCzI4LnNjYWRhYnJpZGdlLnNp", + "dGVjb21tYW5kLnYxLlN1YnNjcmliZURlYnVnVmlld1JlcXVlc3REdG9IABJc", + "ChZ1bnN1YnNjcmliZV9kZWJ1Z192aWV3GAQgASgLMjouc2NhZGFicmlkZ2Uu", + "c2l0ZWNvbW1hbmQudjEuVW5zdWJzY3JpYmVEZWJ1Z1ZpZXdSZXF1ZXN0RHRv", + "SABCCQoHY29tbWFuZCKRAgoKUXVlcnlSZXBseRJPCg9ldmVudF9sb2dfcXVl", + "cnkYASABKAsyNC5zY2FkYWJyaWRnZS5zaXRlY29tbWFuZC52MS5FdmVudExv", + "Z1F1ZXJ5UmVzcG9uc2VEdG9IABJPChNkZWJ1Z192aWV3X3NuYXBzaG90GAIg", + "ASgLMjAuc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuRGVidWdWaWV3U25h", + "cHNob3REdG9IABJYChZ1bnN1YnNjcmliZV9kZWJ1Z192aWV3GAMgASgLMjYu", + "c2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuVW5zdWJzY3JpYmVEZWJ1Z1Zp", + "ZXdBY2tEdG9IAEIHCgVyZXBseSKeAQocUGFya2VkTWVzc2FnZVF1ZXJ5UmVx", + "dWVzdER0bxIWCg5jb3JyZWxhdGlvbl9pZBgBIAEoCRIPCgdzaXRlX2lkGAIg", + "ASgJEhMKC3BhZ2VfbnVtYmVyGAMgASgFEhEKCXBhZ2Vfc2l6ZRgEIAEoBRIt", + "Cgl0aW1lc3RhbXAYBSABKAsyGi5nb29nbGUucHJvdG9idWYuVGltZXN0YW1w", + "IvICChVQYXJrZWRNZXNzYWdlRW50cnlEdG8SEgoKbWVzc2FnZV9pZBgBIAEo", + "CRIVCg10YXJnZXRfc3lzdGVtGAIgASgJEhMKC21ldGhvZF9uYW1lGAMgASgJ", + "EhUKDWVycm9yX21lc3NhZ2UYBCABKAkSFQoNYXR0ZW1wdF9jb3VudBgFIAEo", + "BRI2ChJvcmlnaW5hbF90aW1lc3RhbXAYBiABKAsyGi5nb29nbGUucHJvdG9i", + "dWYuVGltZXN0YW1wEjoKFmxhc3RfYXR0ZW1wdF90aW1lc3RhbXAYByABKAsy", + "Gi5nb29nbGUucHJvdG9idWYuVGltZXN0YW1wEhQKDG1heF9hdHRlbXB0cxgI", + "IAEoBRJICghjYXRlZ29yeRgJIAEoDjI2LnNjYWRhYnJpZGdlLnNpdGVjb21t", + "YW5kLnYxLlN0b3JlQW5kRm9yd2FyZENhdGVnb3J5RHRvEhcKD29yaWdpbl9p", + "bnN0YW5jZRgKIAEoCSKhAgodUGFya2VkTWVzc2FnZVF1ZXJ5UmVzcG9uc2VE", + "dG8SFgoOY29ycmVsYXRpb25faWQYASABKAkSDwoHc2l0ZV9pZBgCIAEoCRJD", + "CghtZXNzYWdlcxgDIAMoCzIxLnNjYWRhYnJpZGdlLnNpdGVjb21tYW5kLnYx", + "LlBhcmtlZE1lc3NhZ2VFbnRyeUR0bxITCgt0b3RhbF9jb3VudBgEIAEoBRIT", + "CgtwYWdlX251bWJlchgFIAEoBRIRCglwYWdlX3NpemUYBiABKAUSDwoHc3Vj", + "Y2VzcxgHIAEoCBIVCg1lcnJvcl9tZXNzYWdlGAggASgJEi0KCXRpbWVzdGFt", + "cBgJIAEoCzIaLmdvb2dsZS5wcm90b2J1Zi5UaW1lc3RhbXAiigEKHFBhcmtl", + "ZE1lc3NhZ2VSZXRyeVJlcXVlc3REdG8SFgoOY29ycmVsYXRpb25faWQYASAB", + "KAkSDwoHc2l0ZV9pZBgCIAEoCRISCgptZXNzYWdlX2lkGAMgASgJEi0KCXRp", + "bWVzdGFtcBgEIAEoCzIaLmdvb2dsZS5wcm90b2J1Zi5UaW1lc3RhbXAiXwod", + "UGFya2VkTWVzc2FnZVJldHJ5UmVzcG9uc2VEdG8SFgoOY29ycmVsYXRpb25f", + "aWQYASABKAkSDwoHc3VjY2VzcxgCIAEoCBIVCg1lcnJvcl9tZXNzYWdlGAMg", + "ASgJIowBCh5QYXJrZWRNZXNzYWdlRGlzY2FyZFJlcXVlc3REdG8SFgoOY29y", + "cmVsYXRpb25faWQYASABKAkSDwoHc2l0ZV9pZBgCIAEoCRISCgptZXNzYWdl", + "X2lkGAMgASgJEi0KCXRpbWVzdGFtcBgEIAEoCzIaLmdvb2dsZS5wcm90b2J1", + "Zi5UaW1lc3RhbXAiYQofUGFya2VkTWVzc2FnZURpc2NhcmRSZXNwb25zZUR0", + "bxIWCg5jb3JyZWxhdGlvbl9pZBgBIAEoCRIPCgdzdWNjZXNzGAIgASgIEhUK", + "DWVycm9yX21lc3NhZ2UYAyABKAkiTwoXUmV0cnlQYXJrZWRPcGVyYXRpb25E", + "dG8SFgoOY29ycmVsYXRpb25faWQYASABKAkSHAoUdHJhY2tlZF9vcGVyYXRp", + "b25faWQYAiABKAkiUQoZRGlzY2FyZFBhcmtlZE9wZXJhdGlvbkR0bxIWCg5j", + "b3JyZWxhdGlvbl9pZBgBIAEoCRIcChR0cmFja2VkX29wZXJhdGlvbl9pZBgC", + "IAEoCSJdChtQYXJrZWRPcGVyYXRpb25BY3Rpb25BY2tEdG8SFgoOY29ycmVs", + "YXRpb25faWQYASABKAkSDwoHYXBwbGllZBgCIAEoCBIVCg1lcnJvcl9tZXNz", + "YWdlGAMgASgJIt4DCg1QYXJrZWRSZXF1ZXN0ElgKFHBhcmtlZF9tZXNzYWdl", + "X3F1ZXJ5GAEgASgLMjguc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuUGFy", + "a2VkTWVzc2FnZVF1ZXJ5UmVxdWVzdER0b0gAElgKFHBhcmtlZF9tZXNzYWdl", + "X3JldHJ5GAIgASgLMjguc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuUGFy", + "a2VkTWVzc2FnZVJldHJ5UmVxdWVzdER0b0gAElwKFnBhcmtlZF9tZXNzYWdl", + "X2Rpc2NhcmQYAyABKAsyOi5zY2FkYWJyaWRnZS5zaXRlY29tbWFuZC52MS5Q", + "YXJrZWRNZXNzYWdlRGlzY2FyZFJlcXVlc3REdG9IABJVChZyZXRyeV9wYXJr", + "ZWRfb3BlcmF0aW9uGAQgASgLMjMuc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQu", + "djEuUmV0cnlQYXJrZWRPcGVyYXRpb25EdG9IABJZChhkaXNjYXJkX3Bhcmtl", + "ZF9vcGVyYXRpb24YBSABKAsyNS5zY2FkYWJyaWRnZS5zaXRlY29tbWFuZC52", + "MS5EaXNjYXJkUGFya2VkT3BlcmF0aW9uRHRvSABCCQoHY29tbWFuZCKHAwoL", + "UGFya2VkUmVwbHkSWQoUcGFya2VkX21lc3NhZ2VfcXVlcnkYASABKAsyOS5z", + "Y2FkYWJyaWRnZS5zaXRlY29tbWFuZC52MS5QYXJrZWRNZXNzYWdlUXVlcnlS", + "ZXNwb25zZUR0b0gAElkKFHBhcmtlZF9tZXNzYWdlX3JldHJ5GAIgASgLMjku", + "c2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuUGFya2VkTWVzc2FnZVJldHJ5", + "UmVzcG9uc2VEdG9IABJdChZwYXJrZWRfbWVzc2FnZV9kaXNjYXJkGAMgASgL", + "Mjsuc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuUGFya2VkTWVzc2FnZURp", + "c2NhcmRSZXNwb25zZUR0b0gAEloKF3BhcmtlZF9vcGVyYXRpb25fYWN0aW9u", + "GAQgASgLMjcuc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuUGFya2VkT3Bl", + "cmF0aW9uQWN0aW9uQWNrRHRvSABCBwoFcmVwbHki7QEKFVJvdXRlVG9DYWxs", + "UmVxdWVzdER0bxIWCg5jb3JyZWxhdGlvbl9pZBgBIAEoCRIcChRpbnN0YW5j", + "ZV91bmlxdWVfbmFtZRgCIAEoCRITCgtzY3JpcHRfbmFtZRgDIAEoCRI9Cgpw", + "YXJhbWV0ZXJzGAQgASgLMikuc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEu", + "TG9vc2VWYWx1ZU1hcBItCgl0aW1lc3RhbXAYBSABKAsyGi5nb29nbGUucHJv", + "dG9idWYuVGltZXN0YW1wEhsKE3BhcmVudF9leGVjdXRpb25faWQYBiABKAki", + "xQEKFlJvdXRlVG9DYWxsUmVzcG9uc2VEdG8SFgoOY29ycmVsYXRpb25faWQY", + "ASABKAkSDwoHc3VjY2VzcxgCIAEoCBI8CgxyZXR1cm5fdmFsdWUYAyABKAsy", + "Ji5zY2FkYWJyaWRnZS5zaXRlY29tbWFuZC52MS5Mb29zZVZhbHVlEhUKDWVy", + "cm9yX21lc3NhZ2UYBCABKAkSLQoJdGltZXN0YW1wGAUgASgLMhouZ29vZ2xl", + "LnByb3RvYnVmLlRpbWVzdGFtcCK7AQoeUm91dGVUb0dldEF0dHJpYnV0ZXNS", + "ZXF1ZXN0RHRvEhYKDmNvcnJlbGF0aW9uX2lkGAEgASgJEhwKFGluc3RhbmNl", + "X3VuaXF1ZV9uYW1lGAIgASgJEhcKD2F0dHJpYnV0ZV9uYW1lcxgDIAMoCRIt", + "Cgl0aW1lc3RhbXAYBCABKAsyGi5nb29nbGUucHJvdG9idWYuVGltZXN0YW1w", + "EhsKE3BhcmVudF9leGVjdXRpb25faWQYBSABKAkiywEKH1JvdXRlVG9HZXRB", + "dHRyaWJ1dGVzUmVzcG9uc2VEdG8SFgoOY29ycmVsYXRpb25faWQYASABKAkS", + "OQoGdmFsdWVzGAIgASgLMikuc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEu", + "TG9vc2VWYWx1ZU1hcBIPCgdzdWNjZXNzGAMgASgIEhUKDWVycm9yX21lc3Nh", + "Z2UYBCABKAkSLQoJdGltZXN0YW1wGAUgASgLMhouZ29vZ2xlLnByb3RvYnVm", + "LlRpbWVzdGFtcCLFAgoeUm91dGVUb1NldEF0dHJpYnV0ZXNSZXF1ZXN0RHRv", + "EhYKDmNvcnJlbGF0aW9uX2lkGAEgASgJEhwKFGluc3RhbmNlX3VuaXF1ZV9u", + "YW1lGAIgASgJEmkKEGF0dHJpYnV0ZV92YWx1ZXMYAyADKAsyTy5zY2FkYWJy", + "aWRnZS5zaXRlY29tbWFuZC52MS5Sb3V0ZVRvU2V0QXR0cmlidXRlc1JlcXVl", + "c3REdG8uQXR0cmlidXRlVmFsdWVzRW50cnkSLQoJdGltZXN0YW1wGAQgASgL", + "MhouZ29vZ2xlLnByb3RvYnVmLlRpbWVzdGFtcBIbChNwYXJlbnRfZXhlY3V0", + "aW9uX2lkGAUgASgJGjYKFEF0dHJpYnV0ZVZhbHVlc0VudHJ5EgsKA2tleRgB", + "IAEoCRINCgV2YWx1ZRgCIAEoCToCOAEikAEKH1JvdXRlVG9TZXRBdHRyaWJ1", + "dGVzUmVzcG9uc2VEdG8SFgoOY29ycmVsYXRpb25faWQYASABKAkSDwoHc3Vj", + "Y2VzcxgCIAEoCBIVCg1lcnJvcl9tZXNzYWdlGAMgASgJEi0KCXRpbWVzdGFt", + "cBgEIAEoCzIaLmdvb2dsZS5wcm90b2J1Zi5UaW1lc3RhbXAiwwIKIVJvdXRl", + "VG9XYWl0Rm9yQXR0cmlidXRlUmVxdWVzdER0bxIWCg5jb3JyZWxhdGlvbl9p", + "ZBgBIAEoCRIcChRpbnN0YW5jZV91bmlxdWVfbmFtZRgCIAEoCRIWCg5hdHRy", + "aWJ1dGVfbmFtZRgDIAEoCRI6ChR0YXJnZXRfdmFsdWVfZW5jb2RlZBgEIAEo", + "CzIcLmdvb2dsZS5wcm90b2J1Zi5TdHJpbmdWYWx1ZRIqCgd0aW1lb3V0GAUg", + "ASgLMhkuZ29vZ2xlLnByb3RvYnVmLkR1cmF0aW9uEi0KCXRpbWVzdGFtcBgG", + "IAEoCzIaLmdvb2dsZS5wcm90b2J1Zi5UaW1lc3RhbXASGwoTcGFyZW50X2V4", + "ZWN1dGlvbl9pZBgHIAEoCRIcChRyZXF1aXJlX2dvb2RfcXVhbGl0eRgIIAEo", + "CCL/AQoiUm91dGVUb1dhaXRGb3JBdHRyaWJ1dGVSZXNwb25zZUR0bxIWCg5j", + "b3JyZWxhdGlvbl9pZBgBIAEoCRIPCgdtYXRjaGVkGAIgASgIEjUKBXZhbHVl", + "GAMgASgLMiYuc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuTG9vc2VWYWx1", + "ZRIPCgdxdWFsaXR5GAQgASgJEhEKCXRpbWVkX291dBgFIAEoCBIPCgdzdWNj", + "ZXNzGAYgASgIEhUKDWVycm9yX21lc3NhZ2UYByABKAkSLQoJdGltZXN0YW1w", + "GAggASgLMhouZ29vZ2xlLnByb3RvYnVmLlRpbWVzdGFtcCKJAwoMUm91dGVS", + "ZXF1ZXN0EkoKDXJvdXRlX3RvX2NhbGwYASABKAsyMS5zY2FkYWJyaWRnZS5z", + "aXRlY29tbWFuZC52MS5Sb3V0ZVRvQ2FsbFJlcXVlc3REdG9IABJdChdyb3V0", + "ZV90b19nZXRfYXR0cmlidXRlcxgCIAEoCzI6LnNjYWRhYnJpZGdlLnNpdGVj", + "b21tYW5kLnYxLlJvdXRlVG9HZXRBdHRyaWJ1dGVzUmVxdWVzdER0b0gAEl0K", + "F3JvdXRlX3RvX3NldF9hdHRyaWJ1dGVzGAMgASgLMjouc2NhZGFicmlkZ2Uu", + "c2l0ZWNvbW1hbmQudjEuUm91dGVUb1NldEF0dHJpYnV0ZXNSZXF1ZXN0RHRv", + "SAASZAobcm91dGVfdG9fd2FpdF9mb3JfYXR0cmlidXRlGAQgASgLMj0uc2Nh", + "ZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuUm91dGVUb1dhaXRGb3JBdHRyaWJ1", + "dGVSZXF1ZXN0RHRvSABCCQoHY29tbWFuZCKJAwoKUm91dGVSZXBseRJLCg1y", + "b3V0ZV90b19jYWxsGAEgASgLMjIuc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQu", + "djEuUm91dGVUb0NhbGxSZXNwb25zZUR0b0gAEl4KF3JvdXRlX3RvX2dldF9h", + "dHRyaWJ1dGVzGAIgASgLMjsuc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEu", + "Um91dGVUb0dldEF0dHJpYnV0ZXNSZXNwb25zZUR0b0gAEl4KF3JvdXRlX3Rv", + "X3NldF9hdHRyaWJ1dGVzGAMgASgLMjsuc2NhZGFicmlkZ2Uuc2l0ZWNvbW1h", + "bmQudjEuUm91dGVUb1NldEF0dHJpYnV0ZXNSZXNwb25zZUR0b0gAEmUKG3Jv", + "dXRlX3RvX3dhaXRfZm9yX2F0dHJpYnV0ZRgEIAEoCzI+LnNjYWRhYnJpZGdl", + "LnNpdGVjb21tYW5kLnYxLlJvdXRlVG9XYWl0Rm9yQXR0cmlidXRlUmVzcG9u", + "c2VEdG9IAEIHCgVyZXBseSJBChZUcmlnZ2VyU2l0ZUZhaWxvdmVyRHRvEhYK", + "DmNvcnJlbGF0aW9uX2lkGAEgASgJEg8KB3NpdGVfaWQYAiABKAkibQoSU2l0", + "ZUZhaWxvdmVyQWNrRHRvEhYKDmNvcnJlbGF0aW9uX2lkGAEgASgJEhAKCGFj", + "Y2VwdGVkGAIgASgIEhYKDnRhcmdldF9hZGRyZXNzGAMgASgJEhUKDWVycm9y", + "X21lc3NhZ2UYBCABKAkqywEKE0RlcGxveW1lbnRTdGF0dXNEdG8SJQohREVQ", + "TE9ZTUVOVF9TVEFUVVNfRFRPX1VOU1BFQ0lGSUVEEAASIQodREVQTE9ZTUVO", + "VF9TVEFUVVNfRFRPX1BFTkRJTkcQARIlCiFERVBMT1lNRU5UX1NUQVRVU19E", + "VE9fSU5fUFJPR1JFU1MQAhIhCh1ERVBMT1lNRU5UX1NUQVRVU19EVE9fU1VD", + "Q0VTUxADEiAKHERFUExPWU1FTlRfU1RBVFVTX0RUT19GQUlMRUQQBCrEAQoS", + "QnJvd3NlTm9kZUNsYXNzRHRvEiUKIUJST1dTRV9OT0RFX0NMQVNTX0RUT19V", + "TlNQRUNJRklFRBAAEiAKHEJST1dTRV9OT0RFX0NMQVNTX0RUT19PQkpFQ1QQ", + "ARIiCh5CUk9XU0VfTk9ERV9DTEFTU19EVE9fVkFSSUFCTEUQAhIgChxCUk9X", + "U0VfTk9ERV9DTEFTU19EVE9fTUVUSE9EEAMSHwobQlJPV1NFX05PREVfQ0xB", + "U1NfRFRPX09USEVSEAQqoQIKFEJyb3dzZUZhaWx1cmVLaW5kRHRvEicKI0JS", + "T1dTRV9GQUlMVVJFX0tJTkRfRFRPX1VOU1BFQ0lGSUVEEAASMAosQlJPV1NF", + "X0ZBSUxVUkVfS0lORF9EVE9fQ09OTkVDVElPTl9OT1RfRk9VTkQQARI0CjBC", + "Uk9XU0VfRkFJTFVSRV9LSU5EX0RUT19DT05ORUNUSU9OX05PVF9DT05ORUNU", + "RUQQAhIpCiVCUk9XU0VfRkFJTFVSRV9LSU5EX0RUT19OT1RfQlJPV1NBQkxF", + "EAMSIwofQlJPV1NFX0ZBSUxVUkVfS0lORF9EVE9fVElNRU9VVBAEEigKJEJS", + "T1dTRV9GQUlMVVJFX0tJTkRfRFRPX1NFUlZFUl9FUlJPUhAFKqoCChtSZWFk", + "VGFnVmFsdWVzRmFpbHVyZUtpbmREdG8SMAosUkVBRF9UQUdfVkFMVUVTX0ZB", + "SUxVUkVfS0lORF9EVE9fVU5TUEVDSUZJRUQQABI5CjVSRUFEX1RBR19WQUxV", + "RVNfRkFJTFVSRV9LSU5EX0RUT19DT05ORUNUSU9OX05PVF9GT1VORBABEj0K", + "OVJFQURfVEFHX1ZBTFVFU19GQUlMVVJFX0tJTkRfRFRPX0NPTk5FQ1RJT05f", + "Tk9UX0NPTk5FQ1RFRBACEiwKKFJFQURfVEFHX1ZBTFVFU19GQUlMVVJFX0tJ", + "TkRfRFRPX1RJTUVPVVQQAxIxCi1SRUFEX1RBR19WQUxVRVNfRkFJTFVSRV9L", + "SU5EX0RUT19TRVJWRVJfRVJST1IQBCqTAgoUVmVyaWZ5RmFpbHVyZUtpbmRE", + "dG8SJwojVkVSSUZZX0ZBSUxVUkVfS0lORF9EVE9fVU5TUEVDSUZJRUQQABIn", + "CiNWRVJJRllfRkFJTFVSRV9LSU5EX0RUT19VTlJFQUNIQUJMRRABEicKI1ZF", + "UklGWV9GQUlMVVJFX0tJTkRfRFRPX0FVVEhfRkFJTEVEEAISMQotVkVSSUZZ", + "X0ZBSUxVUkVfS0lORF9EVE9fVU5UUlVTVEVEX0NFUlRJRklDQVRFEAMSIwof", + "VkVSSUZZX0ZBSUxVUkVfS0lORF9EVE9fVElNRU9VVBAEEigKJFZFUklGWV9G", + "QUlMVVJFX0tJTkRfRFRPX1NFUlZFUl9FUlJPUhAFKuUBChpTdG9yZUFuZEZv", + "cndhcmRDYXRlZ29yeUR0bxIuCipTVE9SRV9BTkRfRk9SV0FSRF9DQVRFR09S", + "WV9EVE9fVU5TUEVDSUZJRUQQABIyCi5TVE9SRV9BTkRfRk9SV0FSRF9DQVRF", + "R09SWV9EVE9fRVhURVJOQUxfU1lTVEVNEAESLworU1RPUkVfQU5EX0ZPUldB", + "UkRfQ0FURUdPUllfRFRPX05PVElGSUNBVElPThACEjIKLlNUT1JFX0FORF9G", + "T1JXQVJEX0NBVEVHT1JZX0RUT19DQUNIRURfREJfV1JJVEUQAypoCg1BbGFy", + "bVN0YXRlRHRvEh8KG0FMQVJNX1NUQVRFX0RUT19VTlNQRUNJRklFRBAAEhoK", + "FkFMQVJNX1NUQVRFX0RUT19BQ1RJVkUQARIaChZBTEFSTV9TVEFURV9EVE9f", + "Tk9STUFMEAIquQEKDUFsYXJtTGV2ZWxEdG8SHwobQUxBUk1fTEVWRUxfRFRP", + "X1VOU1BFQ0lGSUVEEAASGAoUQUxBUk1fTEVWRUxfRFRPX05PTkUQARIXChNB", + "TEFSTV9MRVZFTF9EVE9fTE9XEAISGwoXQUxBUk1fTEVWRUxfRFRPX0xPV19M", + "T1cQAxIYChRBTEFSTV9MRVZFTF9EVE9fSElHSBAEEh0KGUFMQVJNX0xFVkVM", + "X0RUT19ISUdIX0hJR0gQBSqSAQoMQWxhcm1LaW5kRHRvEh4KGkFMQVJNX0tJ", + "TkRfRFRPX1VOU1BFQ0lGSUVEEAASGwoXQUxBUk1fS0lORF9EVE9fQ09NUFVU", + "RUQQARIgChxBTEFSTV9LSU5EX0RUT19OQVRJVkVfT1BDX1VBEAISIwofQUxB", + "Uk1fS0lORF9EVE9fTkFUSVZFX01YX0FDQ0VTUxADKugBChNBbGFybVNoZWx2", + "ZVN0YXRlRHRvEiYKIkFMQVJNX1NIRUxWRV9TVEFURV9EVE9fVU5TUEVDSUZJ", + "RUQQABIkCiBBTEFSTV9TSEVMVkVfU1RBVEVfRFRPX1VOU0hFTFZFRBABEisK", + "J0FMQVJNX1NIRUxWRV9TVEFURV9EVE9fT05FX1NIT1RfU0hFTFZFRBACEigK", + "JEFMQVJNX1NIRUxWRV9TVEFURV9EVE9fVElNRURfU0hFTFZFRBADEiwKKEFM", + "QVJNX1NIRUxWRV9TVEFURV9EVE9fUEVSTUFORU5UX1NIRUxWRUQQBDKEBQoS", + "U2l0ZUNvbW1hbmRTZXJ2aWNlEmwKEEV4ZWN1dGVMaWZlY3ljbGUSLC5zY2Fk", + "YWJyaWRnZS5zaXRlY29tbWFuZC52MS5MaWZlY3ljbGVSZXF1ZXN0Giouc2Nh", + "ZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuTGlmZWN5Y2xlUmVwbHkSYAoMRXhl", + "Y3V0ZU9wY1VhEiguc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuT3BjVWFS", + "ZXF1ZXN0GiYuc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuT3BjVWFSZXBs", + "eRJgCgxFeGVjdXRlUXVlcnkSKC5zY2FkYWJyaWRnZS5zaXRlY29tbWFuZC52", + "MS5RdWVyeVJlcXVlc3QaJi5zY2FkYWJyaWRnZS5zaXRlY29tbWFuZC52MS5R", + "dWVyeVJlcGx5EmMKDUV4ZWN1dGVQYXJrZWQSKS5zY2FkYWJyaWRnZS5zaXRl", + "Y29tbWFuZC52MS5QYXJrZWRSZXF1ZXN0Gicuc2NhZGFicmlkZ2Uuc2l0ZWNv", + "bW1hbmQudjEuUGFya2VkUmVwbHkSYAoMRXhlY3V0ZVJvdXRlEiguc2NhZGFi", + "cmlkZ2Uuc2l0ZWNvbW1hbmQudjEuUm91dGVSZXF1ZXN0GiYuc2NhZGFicmlk", + "Z2Uuc2l0ZWNvbW1hbmQudjEuUm91dGVSZXBseRJ1Cg9UcmlnZ2VyRmFpbG92", + "ZXISMi5zY2FkYWJyaWRnZS5zaXRlY29tbWFuZC52MS5UcmlnZ2VyU2l0ZUZh", + "aWxvdmVyRHRvGi4uc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuU2l0ZUZh", + "aWxvdmVyQWNrRHRvQiuqAihaQi5NT00uV1cuU2NhZGFCcmlkZ2UuQ29tbXVu", + "aWNhdGlvbi5HcnBjYgZwcm90bzM=")); descriptor = pbr::FileDescriptor.FromGeneratedCode(descriptorData, new pbr::FileDescriptor[] { global::Google.Protobuf.WellKnownTypes.TimestampReflection.Descriptor, global::Google.Protobuf.WellKnownTypes.DurationReflection.Descriptor, global::Google.Protobuf.WellKnownTypes.WrappersReflection.Descriptor, }, new pbr::GeneratedClrTypeInfo(new[] {typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.DeploymentStatusDto), typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.BrowseNodeClassDto), typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.BrowseFailureKindDto), typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ReadTagValuesFailureKindDto), typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.VerifyFailureKindDto), typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.StoreAndForwardCategoryDto), typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AlarmStateDto), typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AlarmLevelDto), typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AlarmKindDto), typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AlarmShelveStateDto), }, null, new pbr::GeneratedClrTypeInfo[] { @@ -554,7 +555,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.EventLogQueryRequestDto), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.EventLogQueryRequestDto.Parser, new[]{ "CorrelationId", "SiteId", "From", "To", "EventType", "Severity", "InstanceId", "KeywordFilter", "ContinuationToken", "PageSize", "Timestamp" }, null, null, null, null), new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.EventLogEntryDto), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.EventLogEntryDto.Parser, new[]{ "Id", "Timestamp", "EventType", "Severity", "InstanceId", "Source", "Message", "Details" }, null, null, null, null), new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.EventLogQueryResponseDto), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.EventLogQueryResponseDto.Parser, new[]{ "CorrelationId", "SiteId", "Entries", "ContinuationToken", "HasMore", "Success", "ErrorMessage", "Timestamp" }, null, null, null, null), - new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.DebugSnapshotRequestDto), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.DebugSnapshotRequestDto.Parser, new[]{ "InstanceUniqueName", "CorrelationId" }, null, null, null, null), + new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.DebugSnapshotRequestDto), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.DebugSnapshotRequestDto.Parser, new[]{ "InstanceUniqueName", "CorrelationId", "AlarmsOnly" }, null, null, null, null), new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SubscribeDebugViewRequestDto), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SubscribeDebugViewRequestDto.Parser, new[]{ "InstanceUniqueName", "CorrelationId" }, null, null, null, null), new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.UnsubscribeDebugViewRequestDto), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.UnsubscribeDebugViewRequestDto.Parser, new[]{ "InstanceUniqueName", "CorrelationId" }, null, null, null, null), new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.UnsubscribeDebugViewAckDto), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.UnsubscribeDebugViewAckDto.Parser, null, null, null, null, null), @@ -19509,6 +19510,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { public DebugSnapshotRequestDto(DebugSnapshotRequestDto other) : this() { instanceUniqueName_ = other.instanceUniqueName_; correlationId_ = other.correlationId_; + alarmsOnly_ = other.alarmsOnly_; _unknownFields = pb::UnknownFieldSet.Clone(other._unknownFields); } @@ -19542,6 +19544,27 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { } } + /// Field number for the "alarms_only" field. + public const int AlarmsOnlyFieldNumber = 3; + private bool alarmsOnly_; + /// + /// Alarms-only projection (WP2.3 wire efficiency): when true the site builds + /// and returns ONLY the alarm rows of the debug snapshot, leaving + /// attribute_values empty. Used by the central per-site live alarm cache, + /// whose seed/reconcile fan-out discards every attribute row anyway — a full + /// snapshot ships the whole attribute surface of every enabled instance once + /// per reconcile for nothing. proto3 defaults it to false, so an older + /// central that never sets it keeps the full-snapshot behaviour. Additive-only. + /// + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public bool AlarmsOnly { + get { return alarmsOnly_; } + set { + alarmsOnly_ = value; + } + } + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] public override bool Equals(object other) { @@ -19559,6 +19582,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { } if (InstanceUniqueName != other.InstanceUniqueName) return false; if (CorrelationId != other.CorrelationId) return false; + if (AlarmsOnly != other.AlarmsOnly) return false; return Equals(_unknownFields, other._unknownFields); } @@ -19568,6 +19592,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { int hash = 1; if (InstanceUniqueName.Length != 0) hash ^= InstanceUniqueName.GetHashCode(); if (CorrelationId.Length != 0) hash ^= CorrelationId.GetHashCode(); + if (AlarmsOnly != false) hash ^= AlarmsOnly.GetHashCode(); if (_unknownFields != null) { hash ^= _unknownFields.GetHashCode(); } @@ -19594,6 +19619,10 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { output.WriteRawTag(18); output.WriteString(CorrelationId); } + if (AlarmsOnly != false) { + output.WriteRawTag(24); + output.WriteBool(AlarmsOnly); + } if (_unknownFields != null) { _unknownFields.WriteTo(output); } @@ -19612,6 +19641,10 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { output.WriteRawTag(18); output.WriteString(CorrelationId); } + if (AlarmsOnly != false) { + output.WriteRawTag(24); + output.WriteBool(AlarmsOnly); + } if (_unknownFields != null) { _unknownFields.WriteTo(ref output); } @@ -19628,6 +19661,9 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { if (CorrelationId.Length != 0) { size += 1 + pb::CodedOutputStream.ComputeStringSize(CorrelationId); } + if (AlarmsOnly != false) { + size += 1 + 1; + } if (_unknownFields != null) { size += _unknownFields.CalculateSize(); } @@ -19646,6 +19682,9 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { if (other.CorrelationId.Length != 0) { CorrelationId = other.CorrelationId; } + if (other.AlarmsOnly != false) { + AlarmsOnly = other.AlarmsOnly; + } _unknownFields = pb::UnknownFieldSet.MergeFrom(_unknownFields, other._unknownFields); } @@ -19673,6 +19712,10 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { CorrelationId = input.ReadString(); break; } + case 24: { + AlarmsOnly = input.ReadBool(); + break; + } } } #endif @@ -19700,6 +19743,10 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { CorrelationId = input.ReadString(); break; } + case 24: { + AlarmsOnly = input.ReadBool(); + break; + } } } } diff --git a/src/ZB.MOM.WW.ScadaBridge.Communication/SiteStreamGrpc/Sitestream.cs b/src/ZB.MOM.WW.ScadaBridge.Communication/SiteStreamGrpc/Sitestream.cs index 450aacd3..40173172 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Communication/SiteStreamGrpc/Sitestream.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Communication/SiteStreamGrpc/Sitestream.cs @@ -81,36 +81,36 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { "dEV2ZW50RHRvEjcKC29wZXJhdGlvbmFsGAIgASgLMiIuc2l0ZXN0cmVhbS5T", "aXRlQ2FsbE9wZXJhdGlvbmFsRHRvIkoKFENhY2hlZFRlbGVtZXRyeUJhdGNo", "EjIKB3BhY2tldHMYASADKAsyIS5zaXRlc3RyZWFtLkNhY2hlZFRlbGVtZXRy", - "eVBhY2tldCJbChZQdWxsQXVkaXRFdmVudHNSZXF1ZXN0Ei0KCXNpbmNlX3V0", + "eVBhY2tldCJtChZQdWxsQXVkaXRFdmVudHNSZXF1ZXN0Ei0KCXNpbmNlX3V0", "YxgBIAEoCzIaLmdvb2dsZS5wcm90b2J1Zi5UaW1lc3RhbXASEgoKYmF0Y2hf", - "c2l6ZRgCIAEoBSJcChdQdWxsQXVkaXRFdmVudHNSZXNwb25zZRIpCgZldmVu", - "dHMYASADKAsyGS5zaXRlc3RyZWFtLkF1ZGl0RXZlbnREdG8SFgoObW9yZV9h", - "dmFpbGFibGUYAiABKAgiawoUUHVsbFNpdGVDYWxsc1JlcXVlc3QSLQoJc2lu", - "Y2VfdXRjGAEgASgLMhouZ29vZ2xlLnByb3RvYnVmLlRpbWVzdGFtcBISCgpi", - "YXRjaF9zaXplGAIgASgFEhAKCGFmdGVyX2lkGAMgASgJImkKFVB1bGxTaXRl", - "Q2FsbHNSZXNwb25zZRI4CgxvcGVyYXRpb25hbHMYASADKAsyIi5zaXRlc3Ry", - "ZWFtLlNpdGVDYWxsT3BlcmF0aW9uYWxEdG8SFgoObW9yZV9hdmFpbGFibGUY", - "AiABKAgqXAoHUXVhbGl0eRIXChNRVUFMSVRZX1VOU1BFQ0lGSUVEEAASEAoM", - "UVVBTElUWV9HT09EEAESFQoRUVVBTElUWV9VTkNFUlRBSU4QAhIPCgtRVUFM", - "SVRZX0JBRBADKl0KDkFsYXJtU3RhdGVFbnVtEhsKF0FMQVJNX1NUQVRFX1VO", - "U1BFQ0lGSUVEEAASFgoSQUxBUk1fU1RBVEVfTk9STUFMEAESFgoSQUxBUk1f", - "U1RBVEVfQUNUSVZFEAIqhQEKDkFsYXJtTGV2ZWxFbnVtEhQKEEFMQVJNX0xF", - "VkVMX05PTkUQABITCg9BTEFSTV9MRVZFTF9MT1cQARIXChNBTEFSTV9MRVZF", - "TF9MT1dfTE9XEAISFAoQQUxBUk1fTEVWRUxfSElHSBADEhkKFUFMQVJNX0xF", - "VkVMX0hJR0hfSElHSBAEMoYEChFTaXRlU3RyZWFtU2VydmljZRJVChFTdWJz", - "Y3JpYmVJbnN0YW5jZRIhLnNpdGVzdHJlYW0uSW5zdGFuY2VTdHJlYW1SZXF1", - "ZXN0Ghsuc2l0ZXN0cmVhbS5TaXRlU3RyZWFtRXZlbnQwARJNCg1TdWJzY3Jp", - "YmVTaXRlEh0uc2l0ZXN0cmVhbS5TaXRlU3RyZWFtUmVxdWVzdBobLnNpdGVz", - "dHJlYW0uU2l0ZVN0cmVhbUV2ZW50MAESRwoRSW5nZXN0QXVkaXRFdmVudHMS", - "Gy5zaXRlc3RyZWFtLkF1ZGl0RXZlbnRCYXRjaBoVLnNpdGVzdHJlYW0uSW5n", - "ZXN0QWNrElAKFUluZ2VzdENhY2hlZFRlbGVtZXRyeRIgLnNpdGVzdHJlYW0u", - "Q2FjaGVkVGVsZW1ldHJ5QmF0Y2gaFS5zaXRlc3RyZWFtLkluZ2VzdEFjaxJa", - "Cg9QdWxsQXVkaXRFdmVudHMSIi5zaXRlc3RyZWFtLlB1bGxBdWRpdEV2ZW50", - "c1JlcXVlc3QaIy5zaXRlc3RyZWFtLlB1bGxBdWRpdEV2ZW50c1Jlc3BvbnNl", - "ElQKDVB1bGxTaXRlQ2FsbHMSIC5zaXRlc3RyZWFtLlB1bGxTaXRlQ2FsbHNS", - "ZXF1ZXN0GiEuc2l0ZXN0cmVhbS5QdWxsU2l0ZUNhbGxzUmVzcG9uc2VCK6oC", - "KFpCLk1PTS5XVy5TY2FkYUJyaWRnZS5Db21tdW5pY2F0aW9uLkdycGNiBnBy", - "b3RvMw==")); + "c2l6ZRgCIAEoBRIQCghhZnRlcl9pZBgDIAEoCSJcChdQdWxsQXVkaXRFdmVu", + "dHNSZXNwb25zZRIpCgZldmVudHMYASADKAsyGS5zaXRlc3RyZWFtLkF1ZGl0", + "RXZlbnREdG8SFgoObW9yZV9hdmFpbGFibGUYAiABKAgiawoUUHVsbFNpdGVD", + "YWxsc1JlcXVlc3QSLQoJc2luY2VfdXRjGAEgASgLMhouZ29vZ2xlLnByb3Rv", + "YnVmLlRpbWVzdGFtcBISCgpiYXRjaF9zaXplGAIgASgFEhAKCGFmdGVyX2lk", + "GAMgASgJImkKFVB1bGxTaXRlQ2FsbHNSZXNwb25zZRI4CgxvcGVyYXRpb25h", + "bHMYASADKAsyIi5zaXRlc3RyZWFtLlNpdGVDYWxsT3BlcmF0aW9uYWxEdG8S", + "FgoObW9yZV9hdmFpbGFibGUYAiABKAgqXAoHUXVhbGl0eRIXChNRVUFMSVRZ", + "X1VOU1BFQ0lGSUVEEAASEAoMUVVBTElUWV9HT09EEAESFQoRUVVBTElUWV9V", + "TkNFUlRBSU4QAhIPCgtRVUFMSVRZX0JBRBADKl0KDkFsYXJtU3RhdGVFbnVt", + "EhsKF0FMQVJNX1NUQVRFX1VOU1BFQ0lGSUVEEAASFgoSQUxBUk1fU1RBVEVf", + "Tk9STUFMEAESFgoSQUxBUk1fU1RBVEVfQUNUSVZFEAIqhQEKDkFsYXJtTGV2", + "ZWxFbnVtEhQKEEFMQVJNX0xFVkVMX05PTkUQABITCg9BTEFSTV9MRVZFTF9M", + "T1cQARIXChNBTEFSTV9MRVZFTF9MT1dfTE9XEAISFAoQQUxBUk1fTEVWRUxf", + "SElHSBADEhkKFUFMQVJNX0xFVkVMX0hJR0hfSElHSBAEMoYEChFTaXRlU3Ry", + "ZWFtU2VydmljZRJVChFTdWJzY3JpYmVJbnN0YW5jZRIhLnNpdGVzdHJlYW0u", + "SW5zdGFuY2VTdHJlYW1SZXF1ZXN0Ghsuc2l0ZXN0cmVhbS5TaXRlU3RyZWFt", + "RXZlbnQwARJNCg1TdWJzY3JpYmVTaXRlEh0uc2l0ZXN0cmVhbS5TaXRlU3Ry", + "ZWFtUmVxdWVzdBobLnNpdGVzdHJlYW0uU2l0ZVN0cmVhbUV2ZW50MAESRwoR", + "SW5nZXN0QXVkaXRFdmVudHMSGy5zaXRlc3RyZWFtLkF1ZGl0RXZlbnRCYXRj", + "aBoVLnNpdGVzdHJlYW0uSW5nZXN0QWNrElAKFUluZ2VzdENhY2hlZFRlbGVt", + "ZXRyeRIgLnNpdGVzdHJlYW0uQ2FjaGVkVGVsZW1ldHJ5QmF0Y2gaFS5zaXRl", + "c3RyZWFtLkluZ2VzdEFjaxJaCg9QdWxsQXVkaXRFdmVudHMSIi5zaXRlc3Ry", + "ZWFtLlB1bGxBdWRpdEV2ZW50c1JlcXVlc3QaIy5zaXRlc3RyZWFtLlB1bGxB", + "dWRpdEV2ZW50c1Jlc3BvbnNlElQKDVB1bGxTaXRlQ2FsbHMSIC5zaXRlc3Ry", + "ZWFtLlB1bGxTaXRlQ2FsbHNSZXF1ZXN0GiEuc2l0ZXN0cmVhbS5QdWxsU2l0", + "ZUNhbGxzUmVzcG9uc2VCK6oCKFpCLk1PTS5XVy5TY2FkYUJyaWRnZS5Db21t", + "dW5pY2F0aW9uLkdycGNiBnByb3RvMw==")); descriptor = pbr::FileDescriptor.FromGeneratedCode(descriptorData, new pbr::FileDescriptor[] { global::Google.Protobuf.WellKnownTypes.TimestampReflection.Descriptor, global::Google.Protobuf.WellKnownTypes.WrappersReflection.Descriptor, }, new pbr::GeneratedClrTypeInfo(new[] {typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.Quality), typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AlarmStateEnum), typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AlarmLevelEnum), }, null, new pbr::GeneratedClrTypeInfo[] { @@ -125,7 +125,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteCallOperationalDto), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteCallOperationalDto.Parser, new[]{ "TrackedOperationId", "Channel", "Target", "SourceSite", "Status", "RetryCount", "LastError", "HttpStatus", "CreatedAtUtc", "UpdatedAtUtc", "TerminalAtUtc", "SourceNode" }, null, null, null, null), new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.CachedTelemetryPacket), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.CachedTelemetryPacket.Parser, new[]{ "AuditEvent", "Operational" }, null, null, null, null), new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.CachedTelemetryBatch), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.CachedTelemetryBatch.Parser, new[]{ "Packets" }, null, null, null, null), - new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.PullAuditEventsRequest), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.PullAuditEventsRequest.Parser, new[]{ "SinceUtc", "BatchSize" }, null, null, null, null), + new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.PullAuditEventsRequest), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.PullAuditEventsRequest.Parser, new[]{ "SinceUtc", "BatchSize", "AfterId" }, null, null, null, null), new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.PullAuditEventsResponse), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.PullAuditEventsResponse.Parser, new[]{ "Events", "MoreAvailable" }, null, null, null, null), new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.PullSiteCallsRequest), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.PullSiteCallsRequest.Parser, new[]{ "SinceUtc", "BatchSize", "AfterId" }, null, null, null, null), new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.PullSiteCallsResponse), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.PullSiteCallsResponse.Parser, new[]{ "Operationals", "MoreAvailable" }, null, null, null, null) @@ -4942,8 +4942,11 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { /// /// Audit Log (#23) M6 reconciliation pull: central→site request for any /// site-local AuditLog rows with OccurredAtUtc >= since_utc that have not yet - /// been ingested centrally (ForwardState in {Pending, Forwarded}). The site - /// flips returned rows to Reconciled after the response is on the wire. + /// been ingested centrally (ForwardState in {Pending, Forwarded}). Rows are NOT + /// flipped to Reconciled when they are served — only when a LATER pull's cursor + /// proves central consumed them (see after_id), so a fault between the response + /// leaving the site and central committing it re-serves the rows instead of + /// silently losing them (at-least-once). /// more_available signals batch_size was saturated so the caller knows to /// issue a follow-up pull with an advanced since_utc cursor. /// @@ -4984,6 +4987,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { public PullAuditEventsRequest(PullAuditEventsRequest other) : this() { sinceUtc_ = other.sinceUtc_ != null ? other.sinceUtc_.Clone() : null; batchSize_ = other.batchSize_; + afterId_ = other.afterId_; _unknownFields = pb::UnknownFieldSet.Clone(other._unknownFields); } @@ -5017,6 +5021,29 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { } } + /// Field number for the "after_id" field. + public const int AfterIdFieldNumber = 3; + private string afterId_ = ""; + /// + /// Composite-keyset cursor (WP2.3), mirroring PullSiteCallsRequest.after_id: + /// the EventId ("D" GUID form) of the last row central has already CONSUMED at + /// since_utc. When set, the site returns only rows strictly after the composite + /// (OccurredAtUtc, EventId) pair — un-pinning a batch that would otherwise stall + /// when more than batch_size rows share one since_utc instant — AND treats the + /// cursor as proof of receipt: everything at or before it is flipped to + /// Reconciled. Empty (the proto3 string default) preserves the legacy inclusive + /// >= behaviour, under which only rows strictly older than since_utc are proven + /// received. Additive-only. + /// + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public string AfterId { + get { return afterId_; } + set { + afterId_ = pb::ProtoPreconditions.CheckNotNull(value, "value"); + } + } + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] public override bool Equals(object other) { @@ -5034,6 +5061,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { } if (!object.Equals(SinceUtc, other.SinceUtc)) return false; if (BatchSize != other.BatchSize) return false; + if (AfterId != other.AfterId) return false; return Equals(_unknownFields, other._unknownFields); } @@ -5043,6 +5071,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { int hash = 1; if (sinceUtc_ != null) hash ^= SinceUtc.GetHashCode(); if (BatchSize != 0) hash ^= BatchSize.GetHashCode(); + if (AfterId.Length != 0) hash ^= AfterId.GetHashCode(); if (_unknownFields != null) { hash ^= _unknownFields.GetHashCode(); } @@ -5069,6 +5098,10 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { output.WriteRawTag(16); output.WriteInt32(BatchSize); } + if (AfterId.Length != 0) { + output.WriteRawTag(26); + output.WriteString(AfterId); + } if (_unknownFields != null) { _unknownFields.WriteTo(output); } @@ -5087,6 +5120,10 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { output.WriteRawTag(16); output.WriteInt32(BatchSize); } + if (AfterId.Length != 0) { + output.WriteRawTag(26); + output.WriteString(AfterId); + } if (_unknownFields != null) { _unknownFields.WriteTo(ref output); } @@ -5103,6 +5140,9 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { if (BatchSize != 0) { size += 1 + pb::CodedOutputStream.ComputeInt32Size(BatchSize); } + if (AfterId.Length != 0) { + size += 1 + pb::CodedOutputStream.ComputeStringSize(AfterId); + } if (_unknownFields != null) { size += _unknownFields.CalculateSize(); } @@ -5124,6 +5164,9 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { if (other.BatchSize != 0) { BatchSize = other.BatchSize; } + if (other.AfterId.Length != 0) { + AfterId = other.AfterId; + } _unknownFields = pb::UnknownFieldSet.MergeFrom(_unknownFields, other._unknownFields); } @@ -5154,6 +5197,10 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { BatchSize = input.ReadInt32(); break; } + case 26: { + AfterId = input.ReadString(); + break; + } } } #endif @@ -5184,6 +5231,10 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { BatchSize = input.ReadInt32(); break; } + case 26: { + AfterId = input.ReadString(); + break; + } } } } diff --git a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/InstanceActor.cs b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/InstanceActor.cs index 8285bd41..61e98da8 100644 --- a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/InstanceActor.cs +++ b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/InstanceActor.cs @@ -1372,13 +1372,21 @@ public class InstanceActor : ReceiveActor private void HandleDebugSnapshot(DebugSnapshotRequest request) { var now = DateTimeOffset.UtcNow; - var attributeValues = _attributes.Select(kvp => new AttributeValueChanged( - _instanceUniqueName, - kvp.Key, - kvp.Key, - kvp.Value, - _attributeQualities.GetValueOrDefault(kvp.Key, "Good"), - _attributeTimestamps.GetValueOrDefault(kvp.Key, now))).ToList(); + + // Alarms-only projection (WP2.3): the central live-alarm cache seeds/reconciles + // through this same query surface and throws every attribute row away. Building + // and shipping them is pure waste — one instance with a few hundred attributes + // costs more on the wire than the whole alarm set of the site. The flag is + // additive and defaults false, so the Debug View is untouched. + var attributeValues = request.AlarmsOnly + ? new List() + : _attributes.Select(kvp => new AttributeValueChanged( + _instanceUniqueName, + kvp.Key, + kvp.Key, + kvp.Value, + _attributeQualities.GetValueOrDefault(kvp.Key, "Good"), + _attributeTimestamps.GetValueOrDefault(kvp.Key, now))).ToList(); var snapshot = new DebugViewSnapshot( _instanceUniqueName, diff --git a/tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Integration/OutageReconciliationTests.cs b/tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Integration/OutageReconciliationTests.cs index c59c311c..075fddc6 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Integration/OutageReconciliationTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Integration/OutageReconciliationTests.cs @@ -89,28 +89,25 @@ public class OutageReconciliationTests : TestKit, IClassFixture 0) + // Mirrors SiteStreamGrpcServer.PullAuditEvents exactly (WP2.3): the + // INCOMING cursor is central's receipt, so everything at or before it + // is retired FIRST; the rows this call serves are NOT retired, because + // nothing yet proves central consumed them. A fault between here and + // central's commit therefore re-serves them on the next tick instead of + // losing them. The actor sends no after_id, so the cursor is a bare + // timestamp under the inclusive >= read contract and only rows strictly + // older than it are provably received. + if (sinceUtc > DateTime.MinValue) { - var ids = rows.Select(e => e.EventId).ToList(); - await _siteQueue.MarkReconciledAsync(ids, ct).ConfigureAwait(false); + await _siteQueue.MarkReconciledUpToAsync(sinceUtc, null, ct).ConfigureAwait(false); } + var rows = await _siteQueue + .ReadPendingSinceAsync(sinceUtc, batchSize, afterId: null, ct) + .ConfigureAwait(false); + + // MoreAvailable is true iff the read filled the batch — the actor + // uses this to decide whether to follow up on the next tick. return new PullAuditEventsResponse(rows, MoreAvailable: rows.Count >= batchSize); } } @@ -251,13 +248,19 @@ public class OutageReconciliationTests : TestKit, IClassFixture= read contract a bare timestamp cannot + // prove the rows AT that instant were consumed, so the site keeps + // serving that row until a newer one advances the cursor. Re-serving is + // harmless: central dedups on EventId (asserted in step 5). await AwaitAssertAsync(async () => { var stillPending = await sqliteWriter.ReadPendingAsync(totalEvents + 10); - Assert.Empty(stillPending); + Assert.True(stillPending.Count <= 1, + $"expected at most the boundary row to remain Pending, found {stillPending.Count}"); }, duration: TimeSpan.FromSeconds(10), interval: TimeSpan.FromMilliseconds(100)); diff --git a/tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Site/SiteAuditRetentionServiceTests.cs b/tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Site/SiteAuditRetentionServiceTests.cs index 3a3bd811..57f333e4 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Site/SiteAuditRetentionServiceTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Site/SiteAuditRetentionServiceTests.cs @@ -160,10 +160,13 @@ public class SiteAuditRetentionServiceTests => throw new NotSupportedException(); public Task MarkForwardedAsync(IReadOnlyList eventIds, CancellationToken ct = default) => throw new NotSupportedException(); - public Task> ReadPendingSinceAsync(DateTime sinceUtc, int batchSize, CancellationToken ct = default) + public Task> ReadPendingSinceAsync( + DateTime sinceUtc, int batchSize, string? afterId = null, CancellationToken ct = default) => throw new NotSupportedException(); public Task MarkReconciledAsync(IReadOnlyList eventIds, CancellationToken ct = default) => throw new NotSupportedException(); + public Task MarkReconciledUpToAsync(DateTime sinceUtc, string? afterId, CancellationToken ct = default) + => throw new NotSupportedException(); public Task GetBacklogStatsAsync(CancellationToken ct = default) => throw new NotSupportedException(); } diff --git a/tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Site/SqliteAuditWriterWriteTests.cs b/tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Site/SqliteAuditWriterWriteTests.cs index db18c2a6..73391a30 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Site/SqliteAuditWriterWriteTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Site/SqliteAuditWriterWriteTests.cs @@ -531,6 +531,123 @@ public class SqliteAuditWriterWriteTests Assert.Equal(pending.EventId, rows[0].EventId); } + // ----- WP2.3: composite keyset cursor + cursor-proved retirement ----- // + + [Fact] + public async Task ReadPendingSinceAsync_WithAfterId_SkipsRowsAtOrBeforeTheCompositeCursor() + { + // A batch of rows sharing ONE exact instant used to pin the inclusive-timestamp + // cursor forever: every pull re-served the same page and the backlog never drained. + var (writer, _) = CreateWriter(nameof(ReadPendingSinceAsync_WithAfterId_SkipsRowsAtOrBeforeTheCompositeCursor)); + await using var _w = writer; + + var instant = new DateTime(2026, 5, 20, 12, 0, 0, DateTimeKind.Utc); + var sameInstant = Enumerable.Range(0, 5) + .Select(_ => NewEvent(occurredAtUtc: instant)) + .ToList(); + foreach (var e in sameInstant) await writer.WriteAsync(e); + + // First page under the composite order (OccurredAtUtc, EventId). + var page1 = await writer.ReadPendingSinceAsync(instant, batchSize: 2); + Assert.Equal(2, page1.Count); + + // Second page continues strictly after the last row of the first — no repeats, + // no stall, even though every row shares the same instant. + var page2 = await writer.ReadPendingSinceAsync( + instant, batchSize: 2, afterId: page1[^1].EventId.ToString()); + + Assert.Equal(2, page2.Count); + Assert.Empty(page2.Select(r => r.EventId).Intersect(page1.Select(r => r.EventId))); + + var page3 = await writer.ReadPendingSinceAsync( + instant, batchSize: 2, afterId: page2[^1].EventId.ToString()); + Assert.Single(page3); // the 5th and last + + var allIds = page1.Concat(page2).Concat(page3).Select(r => r.EventId).ToHashSet(); + Assert.Equal(sameInstant.Select(e => e.EventId).ToHashSet(), allIds); + } + + [Fact] + public async Task MarkReconciledUpToAsync_WithCursorId_RetiresOnlyRowsAtOrBeforeIt() + { + var (writer, _) = CreateWriter(nameof(MarkReconciledUpToAsync_WithCursorId_RetiresOnlyRowsAtOrBeforeIt)); + await using var _w = writer; + + var instant = new DateTime(2026, 5, 20, 12, 0, 0, DateTimeKind.Utc); + var older = NewEvent(occurredAtUtc: instant.AddSeconds(-10)); + var atInstant = Enumerable.Range(0, 4).Select(_ => NewEvent(occurredAtUtc: instant)).ToList(); + var newer = NewEvent(occurredAtUtc: instant.AddSeconds(10)); + + await writer.WriteAsync(older); + foreach (var e in atInstant) await writer.WriteAsync(e); + await writer.WriteAsync(newer); + + // Central consumed the older row plus the first two at the shared instant. + var consumed = await writer.ReadPendingSinceAsync(instant, batchSize: 2); + var cursorId = consumed[^1].EventId.ToString(); + + var flipped = await writer.MarkReconciledUpToAsync(instant, cursorId); + + // older + the two consumed at the instant. + Assert.Equal(3, flipped); + + var remaining = await writer.ReadPendingSinceAsync(DateTime.MinValue, batchSize: 100); + var remainingIds = remaining.Select(r => r.EventId).ToHashSet(); + Assert.DoesNotContain(older.EventId, remainingIds); + foreach (var e in consumed) Assert.DoesNotContain(e.EventId, remainingIds); + Assert.Contains(newer.EventId, remainingIds); + Assert.Equal(3, remaining.Count); // the two un-consumed at the instant + newer + } + + [Fact] + public async Task MarkReconciledUpToAsync_WithoutCursorId_LeavesTheBoundaryInstantServable() + { + // With a bare timestamp cursor under the inclusive >= read contract, the rows AT + // the cursor instant may be only half-consumed, so they must stay servable. Only + // strictly-older rows are provably received. + var (writer, _) = CreateWriter(nameof(MarkReconciledUpToAsync_WithoutCursorId_LeavesTheBoundaryInstantServable)); + await using var _w = writer; + + var instant = new DateTime(2026, 5, 20, 12, 0, 0, DateTimeKind.Utc); + var older = NewEvent(occurredAtUtc: instant.AddSeconds(-5)); + var boundary = NewEvent(occurredAtUtc: instant); + await writer.WriteAsync(older); + await writer.WriteAsync(boundary); + + var flipped = await writer.MarkReconciledUpToAsync(instant, afterId: null); + + Assert.Equal(1, flipped); + var remaining = await writer.ReadPendingSinceAsync(DateTime.MinValue, batchSize: 100); + var row = Assert.Single(remaining); + Assert.Equal(boundary.EventId, row.EventId); + } + + [Fact] + public async Task ServingRows_WithoutAnAdvancingCursor_KeepsThemServable_AtLeastOnce() + { + // The at-least-once contract end to end at the storage layer: reading a batch + // changes no state, so a central that faults before committing gets the identical + // batch on its next pull. Only an advanced cursor retires rows. + var (writer, _) = CreateWriter(nameof(ServingRows_WithoutAnAdvancingCursor_KeepsThemServable_AtLeastOnce)); + await using var _w = writer; + + var since = new DateTime(2026, 5, 20, 12, 0, 0, DateTimeKind.Utc); + var events = Enumerable.Range(1, 3) + .Select(i => NewEvent(occurredAtUtc: since.AddSeconds(i))) + .ToList(); + foreach (var e in events) await writer.WriteAsync(e); + + var first = await writer.ReadPendingSinceAsync(since, batchSize: 100); + // ... central faults here; its cursor never moves, so it replays the same call. + await writer.MarkReconciledUpToAsync(since, afterId: null); + var second = await writer.ReadPendingSinceAsync(since, batchSize: 100); + + Assert.Equal(3, first.Count); + Assert.Equal( + first.Select(r => r.EventId).ToHashSet(), + second.Select(r => r.EventId).ToHashSet()); + } + [Fact] public async Task ReadPendingSinceAsync_InvalidBatchSize_Throws() { diff --git a/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/DebugStreamBridgeActorTests.cs b/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/DebugStreamBridgeActorTests.cs index c0832f16..0d122c1f 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/DebugStreamBridgeActorTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/DebugStreamBridgeActorTests.cs @@ -920,6 +920,149 @@ public class DebugStreamBridgeActorTests : TestKit DebugStreamBridgeActor.StabilityWindow = TimeSpan.FromSeconds(30); } } + + // ── WP2.3: bounded pre-snapshot buffer, hard snapshot deadline, timeout hygiene ── + + [Fact] + public void PreSnapshotBuffer_IsCapped_DropsOldest_AndCountsTheDrops() + { + // Before the cap a snapshot that never arrived buffered every live event on the + // CENTRAL node without limit — one wedged session on a chatty instance was enough + // to grow unbounded. Now the oldest are evicted and counted. + var ctx = CreateBridgeActor(); + ctx.CommProbe.ExpectMsg(); // subscribe envelope; no snapshot is ever sent + + const int cap = 20_000; + const int overflow = 250; + var before = Interlocked.Read(ref DebugStreamBridgeActor.TotalPreSnapshotDropped); + + var t = DateTimeOffset.UtcNow; + for (var i = 0; i < cap + overflow; i++) + { + ctx.BridgeActor.Tell(new AttributeValueChanged( + InstanceName, "Modules.IO", $"Attr{i}", i, "Good", t.AddMilliseconds(i))); + } + + // The overflow was evicted (drop-oldest) and counted. + AwaitCondition( + () => Interlocked.Read(ref DebugStreamBridgeActor.TotalPreSnapshotDropped) - before >= overflow, + TimeSpan.FromSeconds(10)); + + // The session is still healthy: the snapshot can still arrive and flush the + // (capped) buffer — the newest events, which the snapshot may predate, survived. + var snapshot = new DebugViewSnapshot( + InstanceName, + new List(), + new List(), + t.AddMilliseconds(-1)); + ctx.BridgeActor.Tell(snapshot); + + AwaitCondition(() => + { + lock (ctx.ReceivedEvents) { return ctx.ReceivedEvents.Count >= cap; } + }, TimeSpan.FromSeconds(10)); + + lock (ctx.ReceivedEvents) + { + // Snapshot + exactly the retained (capped) events, and the newest survived. + Assert.Equal(cap + 1, ctx.ReceivedEvents.Count); + var lastAttr = ctx.ReceivedEvents.OfType().Last(); + Assert.Equal($"Attr{cap + overflow - 1}", lastAttr.AttributeName); + } + } + + [Fact] + public void NoSnapshotWithinDeadline_FailsTheSession_InsteadOfBufferingForever() + { + // Nothing else ends a session wedged in the buffering phase: a lost site reply + // raises no gRPC error, and stream events no longer reset the orphan timeout. + DebugStreamBridgeActor.SnapshotTimeout = TimeSpan.FromMilliseconds(300); + try + { + var ctx = CreateBridgeActor(); + ctx.CommProbe.ExpectMsg(); // subscribe request — never answered + + Watch(ctx.BridgeActor); + ExpectTerminated(ctx.BridgeActor, TimeSpan.FromSeconds(5)); + + // The consumer is told, so the UI can surface the failure and reopen. + Assert.True(ctx.TerminatedFlag[0]); + // And the site-side relay was released rather than left as a zombie. + AwaitCondition( + () => ctx.MockGrpcClient.UnsubscribedCorrelationIds.Contains("corr-1"), + TimeSpan.FromSeconds(3)); + } + finally + { + DebugStreamBridgeActor.SnapshotTimeout = TimeSpan.FromSeconds(60); + } + } + + [Fact] + public void SnapshotArrival_StandsDownTheDeadline() + { + // The deadline must not fire after a healthy snapshot — a live session would + // otherwise be killed mid-stream. + DebugStreamBridgeActor.SnapshotTimeout = TimeSpan.FromMilliseconds(300); + try + { + var ctx = CreateBridgeActor(); + ctx.CommProbe.ExpectMsg(); + + ctx.BridgeActor.Tell(new DebugViewSnapshot( + InstanceName, + new List(), + new List(), + DateTimeOffset.UtcNow)); + + Thread.Sleep(700); // well past the deadline + Assert.False(ctx.TerminatedFlag[0]); + + // Still serving: a post-snapshot event passes straight through. + ctx.MockGrpcClient.SubscribeCalls[0].OnEvent(new AlarmStateChanged( + InstanceName, "PumpFault", Commons.Types.Enums.AlarmState.Active, 500, + DateTimeOffset.UtcNow)); + AwaitCondition(() => + { + lock (ctx.ReceivedEvents) { return ctx.ReceivedEvents.OfType().Any(); } + }, TimeSpan.FromSeconds(3)); + } + finally + { + DebugStreamBridgeActor.SnapshotTimeout = TimeSpan.FromSeconds(60); + } + } + + [Fact] + public void StreamEvents_AreWrapped_SoTheyDoNotResetTheOrphanReceiveTimeout() + { + // Structural pin for the fix: the gRPC callback wraps every event in an envelope + // marked INotInfluenceReceiveTimeout, so a busy site can no longer keep an + // abandoned session alive indefinitely by feeding it events. + Assert.True(typeof(Akka.Actor.INotInfluenceReceiveTimeout) + .IsAssignableFrom(typeof(LiveDebugStreamEvent))); + + var ctx = CreateBridgeActor(); + ctx.CommProbe.ExpectMsg(); + AwaitCondition(() => ctx.MockGrpcClient.SubscribeCalls.Count == 1, TimeSpan.FromSeconds(3)); + + ctx.BridgeActor.Tell(new DebugViewSnapshot( + InstanceName, + new List(), + new List(), + DateTimeOffset.UtcNow)); + + // An event delivered through the real gRPC callback path still reaches the + // consumer — the wrapper is transparent to delivery. + var evt = new AttributeValueChanged( + InstanceName, "Modules.IO", "Temperature", 42.5, "Good", DateTimeOffset.UtcNow); + ctx.MockGrpcClient.SubscribeCalls[0].OnEvent(evt); + + AwaitCondition(() => + { + lock (ctx.ReceivedEvents) { return ctx.ReceivedEvents.OfType().Any(); } + }, TimeSpan.FromSeconds(3)); + } } /// diff --git a/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/SiteAlarmAggregatorActorTests.cs b/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/SiteAlarmAggregatorActorTests.cs index 98aa76a5..d5c85507 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/SiteAlarmAggregatorActorTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/SiteAlarmAggregatorActorTests.cs @@ -79,11 +79,14 @@ public class SiteAlarmAggregatorActorTests : TestKit private readonly object _lock = new(); public List> Snapshots { get; } = new(); - public void Publish(IReadOnlyList snapshot) + public void Publish(IReadOnlyList snapshot, bool streamLive) { - lock (_lock) { Snapshots.Add(snapshot); } + lock (_lock) { Snapshots.Add(snapshot); StreamLive = streamLive; } } + /// Liveness reported with the most recent publish. + public bool StreamLive { get; private set; } + public IReadOnlyList? Latest { get { lock (_lock) { return Snapshots.Count == 0 ? null : Snapshots[^1]; } } @@ -94,7 +97,7 @@ public class SiteAlarmAggregatorActorTests : TestKit private sealed record SiteSub( string CorrelationId, Action OnAlarm, Action OnError, - Action OnCompleted, CancellationToken Ct); + Action OnCompleted, CancellationToken Ct, Action? OnConnected); private sealed class MockSiteAlarmStreamClient : SiteStreamGrpcClient { @@ -107,11 +110,21 @@ public class SiteAlarmAggregatorActorTests : TestKit public MockSiteAlarmStreamClient() : base() { } + /// When false the stream never reports connected — the site accepted the TCP + /// call but never answered, so the aggregator's connect-driven re-seed must not run. + public bool AutoConnect { get; set; } = true; + public override Task SubscribeSiteAsync( string correlationId, Action onAlarmEvent, Action onError, - Action onCompleted, CancellationToken ct) + Action onCompleted, CancellationToken ct, Action? onConnected = null) { - lock (_lock) { _subs.Add(new SiteSub(correlationId, onAlarmEvent, onError, onCompleted, ct)); } + lock (_lock) + { + _subs.Add(new SiteSub(correlationId, onAlarmEvent, onError, onCompleted, ct, onConnected)); + } + // A healthy site accepts the subscription immediately (it flushes response headers + // as soon as its relay is attached), which is the aggregator's connect signal. + if (AutoConnect) onConnected?.Invoke(); var tcs = new TaskCompletionSource(); ct.Register(() => tcs.TrySetResult()); return tcs.Task; // never completes until cancelled (simulates a live stream) @@ -130,8 +143,11 @@ public class SiteAlarmAggregatorActorTests : TestKit public MockSiteAlarmStreamClientFactory() : base(Microsoft.Extensions.Logging.Abstractions.NullLoggerFactory.Instance) { } + /// Applied to every client this factory hands out (set before the actor starts). + public bool AutoConnect { get; set; } = true; + public MockSiteAlarmStreamClient ClientFor(string endpoint) => - _byEndpoint.GetOrAdd(endpoint, _ => new MockSiteAlarmStreamClient()); + _byEndpoint.GetOrAdd(endpoint, _ => new MockSiteAlarmStreamClient { AutoConnect = AutoConnect }); public override SiteStreamGrpcClient GetOrCreate(string siteIdentifier, string grpcEndpoint) => ClientFor(grpcEndpoint); @@ -141,18 +157,19 @@ public class SiteAlarmAggregatorActorTests : TestKit } private (IActorRef Actor, SeedStub Seed, PublishSink Sink, MockSiteAlarmStreamClientFactory Factory) CreateActor( - TimeSpan? reconcileInterval = null, TimeSpan? publishCoalesce = null) + TimeSpan? reconcileInterval = null, TimeSpan? publishCoalesce = null, bool autoConnect = true) { var seed = new SeedStub(); var sink = new PublishSink(); - var factory = new MockSiteAlarmStreamClientFactory(); + var factory = new MockSiteAlarmStreamClientFactory { AutoConnect = autoConnect }; var props = Props.Create(() => new SiteAlarmAggregatorActor( SiteId, "corr-1", seed.Seed, sink.Publish, factory, GrpcNodeA, GrpcNodeB, reconcileInterval ?? TimeSpan.FromMinutes(10), publishCoalesce ?? TimeSpan.Zero, TestReconnectDelay, - TestStabilityWindow)); + TestStabilityWindow, + 0.0)); // no reconcile jitter in tests — determinism var actor = Sys.ActorOf(props); return (actor, seed, sink, factory); @@ -414,12 +431,16 @@ public class SiteAlarmAggregatorActorTests : TestKit AwaitAssert(() => Assert.Equal(1, sink.Count)); AwaitCondition(() => factory.ClientFor(GrpcNodeA).Subs.Count == 1, TimeSpan.FromSeconds(3)); + // The first tick after a fan-out is deliberately skipped (WP2.3: the initial seed + // already covered this window); the second one actually fans out. + actor.Tell(new RunReconcile()); // skipped — consumes the "already fanned out" flag actor.Tell(new RunReconcile()); // reconcile fan-out now in flight (CallCount 2) AwaitAssert(() => Assert.Equal(2, seed.CallCount)); - // Stream error while the reconcile is in flight → the failover re-seed must not be - // silently skipped. Pre-fix: StartFanout no-ops and CallCount stays 2 until the next - // 60s reconcile tick. + // Stream error while the reconcile is in flight: the reconnect that follows + // reconnects (the mock accepts immediately) and its connect-driven re-seed must not + // be silently swallowed. Pre-fix: StartFanout no-ops and CallCount stays 2 until the + // next 60s reconcile tick. factory.ClientFor(GrpcNodeA).Subs.Last().OnError(new Exception("stream fault")); seed.CompleteNext(); // finish the in-flight reconcile @@ -577,4 +598,141 @@ public class SiteAlarmAggregatorActorTests : TestKit AwaitCondition(() => factory.ClientFor(GrpcNodeA).Unsubscribed.Contains("corr-1"), TimeSpan.FromSeconds(3)); } + + // -- WP2.3: seed once per SUCCESSFUL (re)connect, and stream-truthful liveness -- + + [Fact] + public void ReconnectAttempts_DoNotEachReSeed_OnlyTheSuccessfulConnectDoes() + { + // Finding #10's per-attempt re-fan-out: every reconnect ATTEMPT used to kick a + // whole-site snapshot fan-out - against a site that is, by definition of the + // reconnect, unreachable. Now the re-seed is owed to the connect that succeeds. + var (_, seed, _, factory) = CreateActor( + reconcileInterval: TimeSpan.FromMinutes(10), autoConnect: false); + AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3)); // initial seed + AwaitCondition(() => factory.ClientFor(GrpcNodeA).Subs.Count == 1, TimeSpan.FromSeconds(3)); + seed.CompleteNext(); + + // Three failed attempts, each flipping the node - none of them may re-seed. + factory.ClientFor(GrpcNodeA).Subs[0].OnError(new Exception("1")); + AwaitCondition(() => factory.ClientFor(GrpcNodeB).Subs.Count == 1, TimeSpan.FromSeconds(5)); + factory.ClientFor(GrpcNodeB).Subs[0].OnError(new Exception("2")); + AwaitCondition(() => factory.ClientFor(GrpcNodeA).Subs.Count == 2, TimeSpan.FromSeconds(5)); + factory.ClientFor(GrpcNodeA).Subs[1].OnError(new Exception("3")); + AwaitCondition(() => factory.ClientFor(GrpcNodeB).Subs.Count == 2, TimeSpan.FromSeconds(5)); + + Thread.Sleep(300); + Assert.Equal(1, seed.CallCount); // pre-fix: 4 + + // The stream finally comes up - exactly ONE re-seed, on the connect. + factory.ClientFor(GrpcNodeB).Subs[1].OnConnected!(); + AwaitCondition(() => seed.CallCount == 2, TimeSpan.FromSeconds(5)); + seed.CompleteNext(); + Thread.Sleep(300); + Assert.Equal(2, seed.CallCount); // and no second one + } + + [Fact] + public void ReconcileTick_IsSkipped_WhenAFanoutAlreadyRanInTheWindow() + { + // The reconcile is a BACKSTOP, not an unconditional 60s whole-site snapshot: a + // window already covered by a seed costs nothing. Staleness stays bounded because + // the skip consumes the flag, so the next tick always fans out. + var (actor, seed, _, _) = CreateActor(reconcileInterval: TimeSpan.FromMinutes(10)); + AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3)); + seed.CompleteNext(); + Thread.Sleep(200); + + actor.Tell(new RunReconcile()); // skipped - the initial seed covered this window + Thread.Sleep(300); + Assert.Equal(1, seed.CallCount); + + actor.Tell(new RunReconcile()); // and the next one runs + AwaitCondition(() => seed.CallCount == 2, TimeSpan.FromSeconds(3)); + } + + [Fact] + public void UnchangedReconcileSnapshot_DoesNotRepublish() + { + // A reconcile that finds nothing changed used to wake every viewer's render path + // once a minute regardless. It now publishes as a diff. + var (actor, seed, sink, _) = CreateActor(reconcileInterval: TimeSpan.FromMinutes(10)); + AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3)); + + var t = DateTimeOffset.UtcNow; + seed.CompleteNext(Alarm("PumpFault", "", 500, t)); + AwaitAssert(() => Assert.Equal(1, sink.Count)); + + actor.Tell(new RunReconcile()); // skipped (seed covered the window) + actor.Tell(new RunReconcile()); // real fan-out + AwaitCondition(() => seed.CallCount == 2, TimeSpan.FromSeconds(3)); + seed.CompleteNext(Alarm("PumpFault", "", 500, t)); // identical snapshot + + Thread.Sleep(300); + Assert.Equal(1, sink.Count); // no second publish + + // A genuinely changed snapshot still publishes. + actor.Tell(new RunReconcile()); + actor.Tell(new RunReconcile()); + AwaitCondition(() => seed.CallCount == 3, TimeSpan.FromSeconds(3)); + seed.CompleteNext(Alarm("PumpFault", "", 900, t.AddSeconds(1))); + AwaitAssert(() => Assert.Equal(2, sink.Count)); + } + + [Fact] + public void StreamCompletion_DropsLiveness_AndTheReopenRestoresIt() + { + // WP2.3 carried residual: a completed (or given-up) stream kept reporting live + // until the next reconcile publish, so the page grafted a freezing snapshot over + // fresh poll data. + // Reconcile is held far away so the drop can be observed before the reopen + // restores liveness; the reopen is then driven explicitly. + var (actor, seed, sink, factory) = CreateActor(reconcileInterval: TimeSpan.FromMinutes(10)); + AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3)); + AwaitCondition(() => factory.ClientFor(GrpcNodeA).Subs.Count == 1, TimeSpan.FromSeconds(3)); + seed.CompleteNext(); + AwaitCondition(() => sink.Count >= 1 && sink.StreamLive, TimeSpan.FromSeconds(3)); + + // The site ends the stream cleanly (its 4h max lifetime). + factory.ClientFor(GrpcNodeA).Subs[0].OnCompleted(); + + // Liveness drops IMMEDIATELY - not at the next reconcile. + AwaitCondition(() => !sink.StreamLive, TimeSpan.FromSeconds(3)); + + // The reconcile tick reopens; the connect restores liveness. + actor.Tell(new RunReconcile()); + AwaitCondition(() => factory.ClientFor(GrpcNodeA).Subs.Count == 2, TimeSpan.FromSeconds(5)); + AwaitCondition(() => sink.StreamLive, TimeSpan.FromSeconds(5)); + } + + [Fact] + public void StreamFault_DropsLiveness_Immediately() + { + var (_, seed, sink, factory) = CreateActor( + reconcileInterval: TimeSpan.FromMinutes(10), autoConnect: false); + AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3)); + AwaitCondition(() => factory.ClientFor(GrpcNodeA).Subs.Count == 1, TimeSpan.FromSeconds(3)); + factory.ClientFor(GrpcNodeA).Subs[0].OnConnected!(); + seed.CompleteNext(); + AwaitCondition(() => sink.StreamLive, TimeSpan.FromSeconds(3)); + + factory.ClientFor(GrpcNodeA).Subs[0].OnError(new Exception("site gone")); + AwaitCondition(() => !sink.StreamLive, TimeSpan.FromSeconds(3)); + } + + [Fact] + public void FailedSeed_IsRetried_WithBackoff_BeforeTheNextReconcile() + { + // The seed leg had no backoff: a failing fan-out simply waited a full reconcile + // interval. Now it retries on its own timer (reconnectDelay, doubling). + var (_, seed, _, _) = CreateActor(reconcileInterval: TimeSpan.FromMinutes(10)); + AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3)); + + seed.FaultNext(); + // TestReconnectDelay is 50 ms, so the first retry lands far inside the 10-minute + // reconcile interval - pre-fix nothing would run until that interval elapsed. + AwaitCondition(() => seed.CallCount == 2, TimeSpan.FromSeconds(5)); + seed.FaultNext(); + AwaitCondition(() => seed.CallCount == 3, TimeSpan.FromSeconds(5)); + } } diff --git a/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/SiteStreamGrpcServerTests.cs b/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/SiteStreamGrpcServerTests.cs index a44e570e..7ab9bc72 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/SiteStreamGrpcServerTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/SiteStreamGrpcServerTests.cs @@ -530,4 +530,54 @@ public class SiteStreamGrpcServerTests : TestKit Assert.True(condition(), $"Condition not met within {timeoutMs}ms"); } + + // ── WP2.3: the site-wide alarm feed gets its OWN, larger send channel ── + + [Fact] + public void SiteAlarmStream_HasItsOwnLargerChannel_ThanTheDebugView() + { + // Sharing the Debug View's 1000-slot DropOldest channel meant an alarm burst during + // a WAN stall silently evicted operator-visible transitions to make room for + // diagnostics traffic. The two feeds are now sized independently. + var options = Microsoft.Extensions.Options.Options.Create(new CommunicationOptions()); + var server = new SiteStreamGrpcServer(_subscriber, _logger, options); + + Assert.Equal(1000, server.InstanceChannelCapacity); + Assert.Equal(20_000, server.SiteAlarmChannelCapacity); + Assert.True(server.SiteAlarmChannelCapacity > server.InstanceChannelCapacity); + } + + [Fact] + public void ChannelCapacities_AreBoundFromOptions_AndFloorAtOne() + { + var options = Microsoft.Extensions.Options.Options.Create(new CommunicationOptions + { + GrpcInstanceStreamChannelCapacity = 42, + GrpcSiteAlarmStreamChannelCapacity = 4242, + }); + var server = new SiteStreamGrpcServer(_subscriber, _logger, options); + + Assert.Equal(42, server.InstanceChannelCapacity); + Assert.Equal(4242, server.SiteAlarmChannelCapacity); + + // A misconfigured zero/negative capacity must not throw at channel-construction + // time deep inside a live RPC — it floors at one instead. + var degenerate = new SiteStreamGrpcServer(_subscriber, _logger, + Microsoft.Extensions.Options.Options.Create(new CommunicationOptions + { + GrpcInstanceStreamChannelCapacity = 0, + GrpcSiteAlarmStreamChannelCapacity = -5, + })); + Assert.Equal(1, degenerate.InstanceChannelCapacity); + Assert.Equal(1, degenerate.SiteAlarmChannelCapacity); + } + + [Fact] + public void DroppedStreamEventCount_StartsAtZero() + { + // The raw counter behind scadabridge.site.stream.events_dropped — a fresh node has + // evicted nothing. + var server = CreateServer(); + Assert.Equal(0, server.DroppedStreamEventCount); + } } diff --git a/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/SiteAlarmLiveCacheServiceTests.cs b/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/SiteAlarmLiveCacheServiceTests.cs index e386f760..0c94c776 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/SiteAlarmLiveCacheServiceTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/SiteAlarmLiveCacheServiceTests.cs @@ -26,11 +26,34 @@ public class SiteAlarmLiveCacheServiceTests : TestKit // A mock site-wide alarm stream client whose subscription hangs until cancelled. private sealed class HangingClient : SiteStreamGrpcClient { + private readonly object _lock = new(); + private readonly List _connects = new(); + private readonly List _completions = new(); + public HangingClient() : base() { } + + /// When false the site never accepts the subscription (no connect signal). + public bool AutoConnect { get; set; } = true; + + /// Connect callbacks captured from every subscribe, for manual firing. + public List Connects { get { lock (_lock) { return _connects.ToList(); } } } + + /// Graceful-completion callbacks captured from every subscribe. + public List Completions { get { lock (_lock) { return _completions.ToList(); } } } + public override Task SubscribeSiteAsync( string correlationId, Action onAlarmEvent, - Action onError, Action onCompleted, CancellationToken ct) + Action onError, Action onCompleted, CancellationToken ct, + Action? onConnected = null) { + lock (_lock) + { + if (onConnected is not null) _connects.Add(onConnected); + _completions.Add(onCompleted); + } + // A healthy site accepts the subscription immediately (headers flushed on + // subscribe), which is what makes the aggregator report the stream live. + if (AutoConnect) onConnected?.Invoke(); var tcs = new TaskCompletionSource(); ct.Register(() => tcs.TrySetResult()); return tcs.Task; @@ -43,6 +66,9 @@ public class SiteAlarmLiveCacheServiceTests : TestKit private readonly HangingClient _client = new(); public int GetOrCreateCount; + /// The single client this factory hands out, for connect/complete control. + public HangingClient Client => _client; + public CountingFactory() : base(NullLoggerFactory.Instance) { } public override SiteStreamGrpcClient GetOrCreate(string siteIdentifier, string grpcEndpoint) @@ -338,4 +364,43 @@ public class SiteAlarmLiveCacheServiceTests : TestKit Thread.Sleep(300); Assert.Equal(startsAfterStop, factory.GetOrCreateCount); // no self-heal restart fired } + + // ── WP2.3: IsLive tracks the STREAM, not merely "a snapshot was published once" ── + + [Fact] + public void IsLive_StaysFalse_UntilTheSiteAcceptsTheStream() + { + var service = CreateService(TimeSpan.FromMilliseconds(200), out var factory); + factory.Client.AutoConnect = false; // site never answers the subscription + + using var sub = service.Subscribe(SiteId, () => { }); + + // The (empty) seed completes and publishes, but with no accepted stream behind it + // the cache is not live — the page must keep polling. Pre-fix IsLive flipped true + // here and the page grafted a never-updating snapshot over fresh poll data. + AwaitCondition(() => service.GetCurrentAlarms(SiteId) is not null, TimeSpan.FromSeconds(5)); + Thread.Sleep(400); + Assert.False(service.IsLive(SiteId)); + + // The site accepts it → live. + AwaitCondition(() => factory.Client.Connects.Count >= 1, TimeSpan.FromSeconds(5)); + factory.Client.Connects[0](); + AwaitCondition(() => service.IsLive(SiteId), TimeSpan.FromSeconds(5)); + } + + [Fact] + public void IsLive_DropsWhenTheStreamCompletes() + { + // The carried residual from Phase 1: a stream that ended gracefully (the site's 4h + // max lifetime) left IsLive true until the next reconcile publish. + var service = CreateService(TimeSpan.FromMilliseconds(200), out var factory); + + using var sub = service.Subscribe(SiteId, () => { }); + AwaitCondition(() => service.IsLive(SiteId), TimeSpan.FromSeconds(5)); + + AwaitCondition(() => factory.Client.Completions.Count >= 1, TimeSpan.FromSeconds(5)); + factory.Client.Completions[0](); + + AwaitCondition(() => !service.IsLive(SiteId), TimeSpan.FromSeconds(5)); + } } diff --git a/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/SiteStreamPullAuditEventsTests.cs b/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/SiteStreamPullAuditEventsTests.cs index 44e1b03b..ee3e3ad2 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/SiteStreamPullAuditEventsTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/SiteStreamPullAuditEventsTests.cs @@ -13,9 +13,10 @@ using ZB.MOM.WW.ScadaBridge.Communication.Grpc; namespace ZB.MOM.WW.ScadaBridge.Communication.Tests; /// -/// Bundle A A2 tests for . -/// Verifies the request → ISiteAuditQueue.ReadPendingSinceAsync → response → -/// MarkReconciledAsync round-trip through the gRPC handler. The queue is an +/// Tests for : the request → +/// ISiteAuditQueue.ReadPendingSinceAsync → response round-trip, plus the WP2.3 +/// at-least-once contract — rows are retired by the NEXT pull's cursor +/// (MarkReconciledUpToAsync), never by the act of serving them. The queue is an /// NSubstitute stub so the tests never touch SQLite. /// public class SiteStreamPullAuditEventsTests : TestKit @@ -63,11 +64,12 @@ public class SiteStreamPullAuditEventsTests : TestKit } [Fact] - public async Task PullAuditEvents_With5PendingRows_ReturnsAllFiveDtos_AndFlipsToReconciled() + public async Task PullAuditEvents_With5PendingRows_ReturnsAllFiveDtos_AndDoesNotFlipThem() { var queue = Substitute.For(); var events = Enumerable.Range(0, 5).Select(_ => NewEvent()).ToList(); - queue.ReadPendingSinceAsync(Arg.Any(), Arg.Any(), Arg.Any()) + queue.ReadPendingSinceAsync( + Arg.Any(), Arg.Any(), Arg.Any(), Arg.Any()) .Returns((IReadOnlyList)events); var server = CreateServer(); @@ -86,11 +88,100 @@ public class SiteStreamPullAuditEventsTests : TestKit var expectedIds = events.Select(e => e.EventId.ToString()).ToHashSet(); Assert.True(expectedIds.SetEquals(response.Events.Select(d => d.EventId).ToHashSet())); - // Verify MarkReconciledAsync received the same 5 ids (best-effort flip). - await queue.Received(1).MarkReconciledAsync( - Arg.Is>(ids => ids.Count == 5 && - ids.ToHashSet().SetEquals(events.Select(e => e.EventId))), - Arg.Any()); + // AT-LEAST-ONCE: serving rows is NOT proof of receipt. The per-id flip is gone + // entirely; only a later cursor retires rows. + await queue.DidNotReceive().MarkReconciledAsync( + Arg.Any>(), Arg.Any()); + } + + [Fact] + public async Task PullAuditEvents_FaultBetweenResponseAndNextPull_ReservesTheSameRows() + { + // The failure this closes: central receives the batch, then dies before committing + // it, so its cursor never advances. Pre-fix the site had already flipped the rows to + // Reconciled while serving them, and ReadPendingSinceAsync would never return them + // again — the rows were silently lost. Now the unchanged cursor means no flip, and + // the identical batch is served again. + var queue = Substitute.For(); + var since = DateTime.SpecifyKind(new DateTime(2026, 5, 20, 9, 30, 0), DateTimeKind.Utc); + var events = Enumerable.Range(0, 3).Select(_ => NewEvent()).ToList(); + queue.ReadPendingSinceAsync( + Arg.Any(), Arg.Any(), Arg.Any(), Arg.Any()) + .Returns((IReadOnlyList)events); + + var server = CreateServer(); + server.SetSiteAuditQueue(queue); + + var request = new PullAuditEventsRequest + { + SinceUtc = Timestamp.FromDateTime(since), + BatchSize = 100, + }; + + var first = await server.PullAuditEvents(request, NewContext()); + // …central faults here; it never commits, so it re-pulls with the SAME cursor. + var second = await server.PullAuditEvents(request, NewContext()); + + Assert.Equal(3, first.Events.Count); + Assert.Equal( + first.Events.Select(e => e.EventId).ToHashSet(), + second.Events.Select(e => e.EventId).ToHashSet()); + + // Neither pull retired anything past the (unchanged) cursor: the flip is bounded by + // the cursor value, so replaying the same cursor can never retire the served rows. + await queue.Received(2).MarkReconciledUpToAsync( + since, null, Arg.Any()); + } + + [Fact] + public async Task PullAuditEvents_AdvancedCursor_RetiresEverythingUpToIt_BeforeReading() + { + // The cursor central sends back IS the receipt: everything at or before it has been + // ingested, so those rows are flipped — and flipped BEFORE the read, so they do not + // consume this batch's budget. + var queue = Substitute.For(); + queue.ReadPendingSinceAsync( + Arg.Any(), Arg.Any(), Arg.Any(), Arg.Any()) + .Returns((IReadOnlyList)Array.Empty()); + + var server = CreateServer(); + server.SetSiteAuditQueue(queue); + + var cursorTime = DateTime.SpecifyKind(new DateTime(2026, 5, 20, 10, 0, 0), DateTimeKind.Utc); + var cursorId = Guid.NewGuid().ToString(); + var request = new PullAuditEventsRequest + { + SinceUtc = Timestamp.FromDateTime(cursorTime), + BatchSize = 100, + AfterId = cursorId, + }; + + await server.PullAuditEvents(request, NewContext()); + + await queue.Received(1).MarkReconciledUpToAsync( + cursorTime, cursorId, Arg.Any()); + // The keyset cursor is passed straight through to the read as well. + await queue.Received(1).ReadPendingSinceAsync( + cursorTime, 100, cursorId, Arg.Any()); + } + + [Fact] + public async Task PullAuditEvents_FirstEverPull_DoesNotFlipAnything() + { + // since == MinValue means "from the beginning of recorded history" — central has + // consumed nothing yet, so there is nothing to retire. + var queue = Substitute.For(); + queue.ReadPendingSinceAsync( + Arg.Any(), Arg.Any(), Arg.Any(), Arg.Any()) + .Returns((IReadOnlyList)Array.Empty()); + + var server = CreateServer(); + server.SetSiteAuditQueue(queue); + + await server.PullAuditEvents(new PullAuditEventsRequest { BatchSize = 10 }, NewContext()); + + await queue.DidNotReceive().MarkReconciledUpToAsync( + Arg.Any(), Arg.Any(), Arg.Any()); } [Fact] @@ -102,7 +193,8 @@ public class SiteStreamPullAuditEventsTests : TestKit // yields an empty gRPC response. var queue = Substitute.For(); var capturedSince = DateTime.MinValue; - queue.ReadPendingSinceAsync(Arg.Any(), Arg.Any(), Arg.Any()) + queue.ReadPendingSinceAsync( + Arg.Any(), Arg.Any(), Arg.Any(), Arg.Any()) .Returns(call => { capturedSince = call.ArgAt(0); @@ -124,9 +216,6 @@ public class SiteStreamPullAuditEventsTests : TestKit Assert.Empty(response.Events); Assert.False(response.MoreAvailable); Assert.Equal(since, capturedSince); - // Empty result → no MarkReconciledAsync call (no rows to flip). - await queue.DidNotReceive().MarkReconciledAsync( - Arg.Any>(), Arg.Any()); } [Fact] @@ -134,7 +223,8 @@ public class SiteStreamPullAuditEventsTests : TestKit { var queue = Substitute.For(); var events = Enumerable.Range(0, 3).Select(_ => NewEvent()).ToList(); - queue.ReadPendingSinceAsync(Arg.Any(), Arg.Any(), Arg.Any()) + queue.ReadPendingSinceAsync( + Arg.Any(), Arg.Any(), Arg.Any(), Arg.Any()) .Returns((IReadOnlyList)events); var server = CreateServer(); @@ -154,16 +244,17 @@ public class SiteStreamPullAuditEventsTests : TestKit } [Fact] - public async Task PullAuditEvents_MarkReconciledThrows_ResponseStillReturned() + public async Task PullAuditEvents_MarkReconciledUpToThrows_ResponseStillReturned() { - // The Reconciled flip is best-effort — if it fails, the response must - // still surface so central can ingest the rows (and dedup on EventId - // when it pulls them again). + // The retire step is best-effort — if it fails, the pull must still serve rows. + // Worst case the same rows are shipped again and central dedups on EventId. var queue = Substitute.For(); var events = Enumerable.Range(0, 2).Select(_ => NewEvent()).ToList(); - queue.ReadPendingSinceAsync(Arg.Any(), Arg.Any(), Arg.Any()) + queue.ReadPendingSinceAsync( + Arg.Any(), Arg.Any(), Arg.Any(), Arg.Any()) .Returns((IReadOnlyList)events); - queue.MarkReconciledAsync(Arg.Any>(), Arg.Any()) + queue.MarkReconciledUpToAsync( + Arg.Any(), Arg.Any(), Arg.Any()) .ThrowsAsync(new InvalidOperationException("SQLite disposed mid-call")); var server = CreateServer(); @@ -175,8 +266,6 @@ public class SiteStreamPullAuditEventsTests : TestKit BatchSize = 100, }; - // Must NOT throw — the response is built before the flip and returned - // regardless of the flip outcome. var response = await server.PullAuditEvents(request, NewContext()); Assert.Equal(2, response.Events.Count); diff --git a/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Grpc/SiteAlarmStreamEndToEndTests.cs b/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Grpc/SiteAlarmStreamEndToEndTests.cs index 53f47c57..68dda8aa 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Grpc/SiteAlarmStreamEndToEndTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Grpc/SiteAlarmStreamEndToEndTests.cs @@ -265,7 +265,7 @@ public class SiteAlarmStreamEndToEndTests : TestKit { private readonly object _lock = new(); private IReadOnlyList? _latest; - public void Publish(IReadOnlyList snapshot) + public void Publish(IReadOnlyList snapshot, bool streamLive) { lock (_lock) { _latest = snapshot; } } @@ -280,8 +280,9 @@ public class SiteAlarmStreamEndToEndTests : TestKit { public override Task SubscribeSiteAsync( string correlationId, Action onAlarmEvent, Action onError, - Action onCompleted, CancellationToken ct) + Action onCompleted, CancellationToken ct, Action? onConnected = null) { + onConnected?.Invoke(); var tcs = new TaskCompletionSource(); ct.Register(() => tcs.TrySetResult()); return tcs.Task; From d15c5f02ea36b63fe578f5001212372beb556847 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Fri, 14 Aug 2026 21:14:04 -0400 Subject: [PATCH 14/31] perf(dcl): batch subscribe/read/write seam, bounded reconnect, sharded subscriptions --- .../Component-DataConnectionLayer.md | 54 +- .../Protocol/IBatchSubscribableConnection.cs | 60 ++ .../OpcUaEndpointConfigSerializer.cs | 2 + .../DataConnections/OpcUaEndpointConfig.cs | 8 + .../Actors/DataConnectionActor.cs | 801 +++++++++++++++--- .../Adapters/AlarmFilterPrefix.cs | 63 ++ .../Adapters/AsyncSerialGate.cs | 55 ++ .../Adapters/BulkPipeline.cs | 79 ++ .../Adapters/IMxGatewayClient.cs | 32 +- .../Adapters/IOpcUaClient.cs | 115 ++- .../Adapters/MonitoredItemShardPlanner.cs | 63 ++ .../Adapters/MxGatewayDataConnection.cs | 103 ++- .../Adapters/OpcUaDataConnection.cs | 158 +++- .../Adapters/RealMxGatewayClient.cs | 118 +++ .../Adapters/RealOpcUaClient.cs | 557 ++++++++++-- .../DataConnectionFactory.cs | 10 +- .../DataConnectionOptions.cs | 63 +- .../DataConnectionOptionsValidator.cs | 25 + .../Actors/DataConnectionActorBatchTests.cs | 291 +++++++ .../Actors/FakeBatchDataConnection.cs | 161 ++++ .../Adapters/BatchSeamPrimitiveTests.cs | 204 +++++ .../Adapters/FakeMxGatewayClient.cs | 44 +- .../Adapters/MxGatewayBatchSeamTests.cs | 146 ++++ .../DataConnectionActorAlarmIndexTests.cs | 166 ++++ .../OpcUaDataConnectionTests.cs | 77 +- 25 files changed, 3131 insertions(+), 324 deletions(-) create mode 100644 src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Protocol/IBatchSubscribableConnection.cs create mode 100644 src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/AlarmFilterPrefix.cs create mode 100644 src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/AsyncSerialGate.cs create mode 100644 src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/BulkPipeline.cs create mode 100644 src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/MonitoredItemShardPlanner.cs create mode 100644 tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/Actors/DataConnectionActorBatchTests.cs create mode 100644 tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/Actors/FakeBatchDataConnection.cs create mode 100644 tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/Adapters/BatchSeamPrimitiveTests.cs create mode 100644 tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/Adapters/MxGatewayBatchSeamTests.cs create mode 100644 tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/DataConnectionActorAlarmIndexTests.cs diff --git a/docs/requirements/Component-DataConnectionLayer.md b/docs/requirements/Component-DataConnectionLayer.md index 80db53e1..ea167cc5 100644 --- a/docs/requirements/Component-DataConnectionLayer.md +++ b/docs/requirements/Component-DataConnectionLayer.md @@ -38,6 +38,23 @@ IDataConnection : IAsyncDisposable The `Disconnected` event is raised by an adapter when it detects an unexpected connection loss (server offline, network failure, keep-alive timeout). The `DataConnectionActor` subscribes to this event to trigger the reconnection state machine. Additional protocols can be added by implementing this interface. +### Batch Capability Seam + +A protocol whose wire form can subscribe MANY tags in one round trip additionally implements the optional capability interface `IBatchSubscribableConnection` (same pattern as `IBrowsableDataConnection` / `IAlarmSubscribableConnection`): + +``` +IBatchSubscribableConnection +├── SubscribeBatchAsync(tagPaths, callback) → IReadOnlyList +└── UnsubscribeBatchAsync(subscriptionIds) → void +``` + +- **One shared callback, plain tag list.** Every tag in a batch shares one callback; `SubscriptionCallback` already carries the tag path, so per-tag delegates would carry nothing. +- **Partial-failure contract.** A per-tag fault (bad node id, unresolvable path) is a result row with `Success:false` and never aborts the batch — mirroring the gateway's `SubscribeResult` and OPC UA's per-monitored-item create status. A **thrown** exception means the whole batch failed; the actor classifies it exactly as the per-tag path does and drives the Reconnecting state machine on a connection-level fault. +- Implementing the interface also asserts that `ReadBatchAsync` / `WriteBatchAsync` are **true bulk** service calls rather than loops over the single-tag methods. Both shipped adapters (`OpcUaDataConnection`, `MxGatewayDataConnection`) implement it; an adapter that does not keeps the historical per-tag behaviour, so the capability is purely additive. +- Single-tag `SubscribeAsync` / `ReadAsync` / `WriteAsync` are **kept** (heartbeat monitor, interactive read/write paths) and delegate to the batch form as a batch of one. + +Sizing: at the 37,500-tag site target (`docs/deployment/topology-guide.md`), the pre-batch path issued one adapter round trip — and, on OPC UA, one `ApplyChanges` — per tag on every subscribe and every reconnect. + ### Common Value Type All protocols produce the same value tuple consumed by Instance Actors. Before the first value update arrives from the DCL, data-sourced attributes are held at **uncertain** quality by the Instance Actor (see Site Runtime — Initialization): @@ -147,6 +164,7 @@ All settings are parsed from the data connection's configuration JSON dictionari | `MaxNotificationsPerPublish` | int | `100` | Max notifications batched per publish cycle | | `SamplingIntervalMs` | int | `1000` | Per-item server sampling rate in milliseconds | | `QueueSize` | int | `10` | Per-item notification buffer size | +| `MaxMonitoredItemsPerSubscription` | int | `5000` | Monitored-item budget per OPC UA subscription. Above it the adapter **shards** onto an additional subscription on the same session (37,500 tags → 8 shards). Per endpoint, because item-count ceilings are a property of the target server. | | `SecurityMode` | string | `None` | Preferred endpoint security: `None`, `Sign`, or `SignAndEncrypt` | | `AutoAcceptUntrustedCerts` | bool | `true` | Accept untrusted server certificates | @@ -172,9 +190,17 @@ These are configured via `DataConnectionOptions` in `appsettings.json`, not per- | Setting | Default | Description | |---------|---------|-------------| | `ReconnectInterval` | 5s | Fixed interval between reconnection attempts | -| `TagResolutionRetryInterval` | 10s | Retry interval for unresolved tag paths | +| `TagResolutionRetryInterval` | 10s | **Floor** interval for the unresolved-tag retry; the retry backs off from here | +| `TagResolutionRetryMaxInterval` | 5m | Ceiling for the exponential tag-resolution backoff | | `WriteTimeout` | 30s | Timeout for write operations | -| `SeedReadTimeout` | 30s | Per-tag timeout for seed reads on the initial-subscribe (and reconnect re-seed) path. A hung device read is treated as a failed seed: retried up to `SeedReadMaxAttempts`, then the tag stays Uncertain until a change notification arrives. | +| `SeedReadTimeout` | 30s | **Per-chunk** timeout for seed reads on the initial-subscribe (and reconnect re-seed) path. A hung device read is treated as a failed seed: retried up to `SeedReadMaxAttempts`, then the tag stays Uncertain until a change notification arrives. | +| `SeedOverallTimeout` | 120s | Wall-clock deadline for the WHOLE seed (all chunks, all retry rounds). Replaces the old 30s-per-tag serial worst case; remaining tags are logged and stay Uncertain. | +| `SeedReadBatchSize` | 250 | Tags per seed-read chunk | +| `SeedReadMaxParallelism` | 4 | Seed-read chunks in flight concurrently | +| `SubscribeBatchSize` | 500 | Tags per adapter subscribe round trip (initial subscribe, reconnect re-subscribe, resolution probes) | +| `SubscribeBatchDelay` | 50ms | Delay BETWEEN reconnect re-subscribe chunks. Not applied on the instance-driven subscribe path — the Deployment Manager already staggers instance startup there. | +| `QualityFlushInterval` | 1s | Coalescing window for pushing tag-quality counters to the health collector | +| `MxSupervisoryAdviseParallelism` | 16 | Max in-flight supervisory advise commands on the MxGateway bulk-subscribe path when the endpoint has no write-user context | ## Subscription Management @@ -184,6 +210,22 @@ These are configured via `DataConnectionOptions` in `appsettings.json`, not per- - When an Instance Actor is stopped (due to disable, delete, or redeployment), the DCL cleans up the associated subscriptions. - When a new Instance Actor is created for a redeployment, subscriptions are established fresh based on the new configuration. +### Batching, chunking and pacing + +Three subscribe paths, deliberately paced differently: + +| Path | Shape | +|---|---| +| **Instance-driven subscribe** (`SubscribeTagsRequest`, i.e. site failover / deploy) | One `SubscribeBatchAsync` per `SubscribeBatchSize` chunk with **no** added delay. The Deployment Manager already staggers instance startup (`SiteRuntimeOptions.StartupBatchSize`/`StartupBatchDelayMs`), so requests arrive pre-spaced at ~75 tags each — double-staggering would only slow failover. | +| **Reconnect re-subscribe** (`ReSubscribeAll`) | **Sequential** chunks of `SubscribeBatchSize` with `SubscribeBatchDelay` between them, all inside ONE background task; each chunk reports its per-tag results back to the actor as its own message. Replaces a tight loop that fired one fire-and-forget task per tag. | +| **Tag-resolution probe** | The whole not-in-flight unresolved set, chunked at `SubscribeBatchSize`, as one probe round per timer tick. | + +Seeding (the initial-value read that keeps a STATIC tag from staying Uncertain) issues chunked bulk reads — `SeedReadBatchSize` per chunk, `SeedReadMaxParallelism` chunks in flight, `SeedReadTimeout` per chunk — under a single `SeedOverallTimeout` deadline covering every chunk and retry round. + +**OPC UA subscription sharding.** `RealOpcUaClient` places monitored items on the first subscription shard with free capacity (`MaxMonitoredItemsPerSubscription`), creating a shard when all are full and deleting one once it empties. A batch is N × `AddItem` plus **one** `ApplyChanges` per touched shard. Alarms & Conditions event items are pinned to a **dedicated event shard**, so `ConditionRefresh` has a single subscription id to target and `QueueSize:1000` event items never consume data-shard capacity. Every structural mutation (AddItem / RemoveItem / ApplyChanges / Create / Delete) and every shard-list mutation serializes behind ONE per-client gate — the SDK `Subscription` is not safe under concurrent structural mutation, and the subscribe task, resolution probe and alarm subscribe are otherwise unordered. + +**MxGateway.** A batch is one `SubscribeBulk` gateway command (AddItem + Advise per tag in a single worker pass), replacing the historical 2 RPCs per tag. When the endpoint has no write-user context (`WriteUserId == 0`) the worker offers no BULK supervisory advise, so the client issues one `AddItemBulk` and then pipelines the per-item supervisory advises with a bounded window (`MxSupervisoryAdviseParallelism`). A cross-repo follow-up (an additive `supervisory` flag on `SubscribeBulkCommand` in mxaccessgw) would collapse that to one RPC as well. + ## Write-Back Support - When a script calls `Instance.SetAttribute` for an attribute with a data source reference, the Instance Actor sends a write request to the DCL. @@ -263,6 +305,8 @@ The registered `SourceReference` is a **NodeId** for OPC UA (the picker, CSV and - **Capability check**: if `_adapter is not IAlarmSubscribableConnection`, the actor replies `SubscribeAlarmsResponse(Success = false, ...)`. - **Reconnect handling**: on entering **Reconnecting**, the actor pushes a `NativeAlarmSourceUnavailable` to every alarm subscriber (consumers mark mirrored alarms uncertain rather than clearing them). On successful reconnection it re-subscribes the feed; the adapter re-emits a snapshot, reconciling state. - **Shared condition filter (last-subscriber-wins)**: because the feed is opened once per source, it carries a single condition filter. A second subscriber that registers a *different* filter for the same source overwrites it (last writer wins) and the actor logs a warning — co-subscribers to one source are expected to agree on the filter. +- **Routing index**: subscribed sources are bucketed by the **first path segment** of their reference (everything before the first `.`), and a transition is matched only against the bucket for its own first segment, plus a residue list of sources that carry no separator at all (a prefix shorter than one segment, which can match other buckets). This is sound because a source reference containing a separator can only prefix a reference sharing its entire first segment. The per-source `StartsWith` test and the condition-type gate inside the bucket are unchanged, so routing decisions are identical to the previous linear scan over every subscribed source; the `SnapshotComplete` sentinel still bypasses the index entirely and is broadcast to every subscriber. +- **Gateway union filter (MxGateway)**: `StreamAlarms` carries a single `alarm_filter_prefix`, so the adapter opens the stream on the **longest common prefix** of the currently subscribed source references and restarts it only when a NEW source falls outside that prefix (the source then replays a fresh snapshot, which `NativeAlarmActor` already handles). An unsubscribe never restarts the stream — the prefix it leaves behind is at worst too broad, which costs bandwidth, not correctness. The prefix is a bandwidth optimisation only; the actor's per-source + condition-type gate remains authoritative, mirroring the OPC UA server-side WhereClause stance. ### Protocol-Neutral Types & Messages @@ -359,8 +403,9 @@ When the DCL subscribes to a tag path from the flattened configuration but the p 1. The failure is **logged to Site Event Logging**. 2. The attribute is marked with quality `bad`. -3. The DCL **periodically retries resolution** at a configurable interval, accommodating devices that come online in stages or load modules after startup. -4. On successful resolution, the subscription activates normally and quality reflects the live value from the device. +3. The DCL **retries resolution with exponential backoff**, accommodating devices that come online in stages or load modules after startup: the interval starts at `TagResolutionRetryInterval` (10s), doubles after a round in which nothing resolved, and is capped at `TagResolutionRetryMaxInterval` (5m). It resets to the floor as soon as **any** tag resolves or the connection reconnects, so a device that is merely slow to boot is still picked up quickly while a dead one is not probed at full width forever. Each round probes the whole unresolved set in `SubscribeBatchSize` chunks, not one call per tag. +4. The retry timer is **single-shot, rescheduled on probe completion** — a periodic timer cannot back off, and rescheduling only on completion is what keeps a fan-out of failures from resetting the clock (the anti-starvation property previously defended by an `IsTimerActive` gate, which is retained). +5. On successful resolution, the subscription activates normally and quality reflects the live value from the device. Note: Pre-deployment validation at central does **not** verify that tag paths resolve to real tags on physical devices — that is a runtime concern handled here. @@ -370,6 +415,7 @@ The DCL reports the following metrics to the Health Monitoring component via the - **Connection status**: `connected`, `disconnected`, or `reconnecting` per data connection. - **Tag resolution counts**: Per connection, the number of total subscribed tags vs. successfully resolved tags. This gives operators visibility into misconfigured templates without needing to open the debug view for individual instances. +- **Tag quality counters** are pushed on a genuine quality **transition** only, coalesced onto a `QualityFlushInterval` (1s) single-shot timer. Counter arithmetic still runs per message; only the collector push is deferred, and a value whose quality is unchanged moves no counter at all. Health reports poll at 30s, so the coalescing loses nothing. Three paths flush **synchronously** because their correctness depends on it: the bad-quality push on disconnect, unsubscribe, and the reconnect counter reset. ## Dependencies diff --git a/src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Protocol/IBatchSubscribableConnection.cs b/src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Protocol/IBatchSubscribableConnection.cs new file mode 100644 index 00000000..9b848ece --- /dev/null +++ b/src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Protocol/IBatchSubscribableConnection.cs @@ -0,0 +1,60 @@ +namespace ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Protocol; + +/// +/// Per-tag outcome of a batch subscribe. A failed row is a genuine per-tag fault +/// (bad node id, unresolvable path) and never aborts the batch — mirroring the +/// MxAccess Gateway's SubscribeResult and OPC UA's per-monitored-item create +/// status. A batch that fails at connection level throws instead, so the +/// DataConnectionActor can drive the reconnect state machine. +/// +/// The requested tag path. +/// Whether the tag was subscribed. +/// Adapter subscription handle when ; otherwise null. +/// Per-tag failure reason when not successful. +public record TagSubscribeResult(string TagPath, bool Success, string? SubscriptionId, string? ErrorMessage); + +/// +/// Optional capability for an implementation whose +/// protocol can subscribe/unsubscribe MANY tags in one round trip, and whose +/// / +/// are TRUE bulk service calls rather than a loop over the single-tag methods. +/// Mirrors the / +/// capability-interface pattern; consumed by the DataConnectionActor only. +/// +/// +/// Implementing this interface is the adapter's assertion that batching is genuinely +/// cheaper than N single calls: the actor then subscribes, unsubscribes and seed-reads +/// in bounded chunks instead of per tag. An adapter that does NOT implement it keeps +/// the historical per-tag behaviour, so the capability is purely additive. +/// +/// +public interface IBatchSubscribableConnection +{ + /// + /// Subscribes every tag in in as few protocol round + /// trips as the adapter allows, returning one per + /// requested tag (in any order — callers key by ). + /// All tags share ONE ; the callback already carries the + /// tag path, so per-tag delegates would carry nothing extra. + /// + /// The tag paths to subscribe. + /// Callback invoked for every value change on any of the tags. + /// Cancellation token. + /// One result row per requested tag path. + /// + /// A connection-level fault (adapter not connected / transport down) is thrown so the + /// caller classifies it as a connection failure; per-tag faults are result rows. + /// + Task> SubscribeBatchAsync( + IReadOnlyList tagPaths, SubscriptionCallback callback, CancellationToken cancellationToken = default); + + /// + /// Releases every supplied subscription id in as few protocol round trips as the + /// adapter allows. Unknown/stale ids are ignored, mirroring + /// . + /// + /// Subscription ids previously returned by a subscribe call. + /// Cancellation token. + /// A task that represents the asynchronous operation. + Task UnsubscribeBatchAsync(IReadOnlyList subscriptionIds, CancellationToken cancellationToken = default); +} diff --git a/src/ZB.MOM.WW.ScadaBridge.Commons/Serialization/OpcUaEndpointConfigSerializer.cs b/src/ZB.MOM.WW.ScadaBridge.Commons/Serialization/OpcUaEndpointConfigSerializer.cs index 55d1d107..fea3f727 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Commons/Serialization/OpcUaEndpointConfigSerializer.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Commons/Serialization/OpcUaEndpointConfigSerializer.cs @@ -195,6 +195,7 @@ public static class OpcUaEndpointConfigSerializer ["DiscardOldest"] = config.DiscardOldest.ToString(), ["SubscriptionPriority"] = config.SubscriptionPriority.ToString(), ["SubscriptionDisplayName"] = config.SubscriptionDisplayName, + ["MaxMonitoredItemsPerSubscription"] = config.MaxMonitoredItemsPerSubscription.ToString(), ["TimestampsToReturn"] = config.TimestampsToReturn.ToString(), }; if (config.Heartbeat is { } hb) @@ -248,6 +249,7 @@ public static class OpcUaEndpointConfigSerializer TryAssignInt(dict, "KeepAliveCount", v => c.KeepAliveCount = v); TryAssignInt(dict, "LifetimeCount", v => c.LifetimeCount = v); TryAssignInt(dict, "MaxNotificationsPerPublish", v => c.MaxNotificationsPerPublish = v); + TryAssignInt(dict, "MaxMonitoredItemsPerSubscription", v => c.MaxMonitoredItemsPerSubscription = v); if (dict.TryGetValue("DiscardOldest", out var doStr) && bool.TryParse(doStr, out var doVal)) c.DiscardOldest = doVal; diff --git a/src/ZB.MOM.WW.ScadaBridge.Commons/Types/DataConnections/OpcUaEndpointConfig.cs b/src/ZB.MOM.WW.ScadaBridge.Commons/Types/DataConnections/OpcUaEndpointConfig.cs index cad140f6..9627e76b 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Commons/Types/DataConnections/OpcUaEndpointConfig.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Commons/Types/DataConnections/OpcUaEndpointConfig.cs @@ -66,6 +66,14 @@ public sealed class OpcUaEndpointConfig /// Display name for the subscription. /// public string SubscriptionDisplayName { get; set; } = "ScadaBridge"; + /// + /// Maximum monitored items placed on a single OPC UA subscription before the + /// adapter shards onto an additional subscription. Item-count ceilings are a + /// property of the target server, so this is per endpoint rather than global. + /// A site sized at 37,500 tags shards into 8 subscriptions at the default. + /// Values <= 0 are treated as the default. + /// + public int MaxMonitoredItemsPerSubscription { get; set; } = 5000; // Read / filter /// diff --git a/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Actors/DataConnectionActor.cs b/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Actors/DataConnectionActor.cs index a390a481..a5f15429 100644 --- a/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Actors/DataConnectionActor.cs +++ b/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Actors/DataConnectionActor.cs @@ -140,6 +140,38 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers private int _tagsUncertainQuality; private readonly Dictionary _lastTagQuality = new(); + /// + /// Set when a genuine quality TRANSITION has moved the counters since the last push to + /// the health collector. The push itself is coalesced onto a single-shot + /// quality-flush timer () + /// instead of running on every received value: at 37,500 tags the per-message push was + /// pure overhead against a collector that is only read every 30s. Disconnect, + /// unsubscribe and the reconnect reset still flush SYNCHRONOUSLY — their correctness + /// depends on the collector being current at that instant. + /// + private bool _qualityDirty; + + /// + /// Current tag-resolution retry interval. Starts at + /// , doubles after a + /// round in which no tag resolved, and is capped at + /// . Reset to the floor + /// whenever a tag resolves or the connection reconnects, so a dead device backs off + /// while a booting one is still picked up quickly. + /// + private TimeSpan _tagResolutionInterval; + + // ── Alarm subscriber prefix index ── + // HandleAlarmTransitionReceived used to scan EVERY subscribed source (two StartsWith + // per source) on every transition. Sources are bucketed by the FIRST path segment of + // their reference; a transition is matched only against the bucket for its own first + // segment, plus the residue list of sources that carry no separator at all (a prefix + // shorter than one segment, which can match transitions in other buckets). The + // per-source StartsWith + condition-type gate inside the bucket is unchanged, so + // routing decisions are identical to the linear scan. + private readonly Dictionary> _alarmSourcesByFirstSegment = new(StringComparer.Ordinal); + private readonly HashSet _alarmSourcesWithoutSeparator = new(StringComparer.Ordinal); + private IDictionary _connectionDetails; private readonly IDictionary _primaryConfig; private readonly IDictionary? _backupConfig; @@ -200,6 +232,7 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers _failoverRetryCount = failoverRetryCount; _siteEventLogger = siteEventLogger; _connectionDetails = _primaryConfig; + _tagResolutionInterval = _options.TagResolutionRetryInterval; } /// @@ -284,6 +317,17 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers case AlarmTransitionReceived: // No live feed yet in Connecting; ignore (snapshot replays on subscribe). break; + case BatchSubscribeCompleted: + // Re-subscribe / resolution-probe results from a previous connection — + // ReSubscribeAll re-issues everything once the link is back up. + break; + case RetryTagResolution: + // No session yet — a probe would fail for every tag. ReSubscribeAll's chunk + // completions re-arm the timer once the link is up. + break; + case QualityFlushTick: + FlushQualityCountersIfDirty(); + break; case BrowseNodeCommand browse: // Browse is an interactive design-time query; never stash. The // adapter has no session yet in this state, so reply with a @@ -366,11 +410,11 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers case TagValueReceived tvr: HandleTagValueReceived(tvr); break; - case TagResolutionSucceeded trs: - HandleTagResolutionSucceeded(trs); + case BatchSubscribeCompleted bsc: + HandleBatchSubscribeCompleted(bsc); break; - case TagResolutionFailed trf: - HandleTagResolutionFailed(trf); + case QualityFlushTick: + FlushQualityCountersIfDirty(); break; case AdapterDisconnected: HandleDisconnect(); @@ -510,10 +554,16 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers case AlarmTransitionReceived: // Ignore — stale alarm callback from previous connection; ReSubscribeAll re-seeds. break; - case TagResolutionSucceeded: - case TagResolutionFailed: + case BatchSubscribeCompleted: // Ignore — stale results from previous connection; ReSubscribeAll runs after reconnect break; + case RetryTagResolution: + // Ignore — the adapter has no live session; ReSubscribeAll's chunk + // completions re-arm the retry timer after reconnect. + break; + case QualityFlushTick: + FlushQualityCountersIfDirty(); + break; case SubscribeCompleted sc: // A subscribe started while Connected can complete after a transition; // apply it so its state survives into the next ReSubscribeAll. @@ -728,6 +778,13 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers } } + // ONE batch subscribe per chunk instead of one adapter round trip per tag. The + // Deployment Manager already staggers instance startup (SiteRuntimeOptions + // StartupBatchSize/Delay), so requests arrive pre-spaced on this path and the DCL + // deliberately adds NO extra delay between chunks here — double-staggering would + // only slow failover. A typical request is ~75 tags, i.e. a single chunk. + var batchSize = Math.Max(1, _options.SubscribeBatchSize); + Task.Run(async () => { var results = new List(request.TagPaths.Count); @@ -738,27 +795,19 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers tagsToSeed.Add(r.TagPath); } - foreach (var tagPath in tagsToSubscribe) + for (var offset = 0; offset < tagsToSubscribe.Count; offset += batchSize) { - try + var chunk = tagsToSubscribe.GetRange( + offset, Math.Min(batchSize, tagsToSubscribe.Count - offset)); + var chunkResults = await SubscribeTagsAsync(adapter, chunk, (path, value) => { - var subId = await adapter.SubscribeAsync(tagPath, (path, value) => - { - self.Tell(new TagValueReceived(path, value, generation)); - }); - results.Add(new SubscribeTagResult(tagPath, AlreadySubscribed: false, Success: true, subId, null)); - tagsToSeed.Add(tagPath); - } - catch (Exception ex) + self.Tell(new TagValueReceived(path, value, generation)); + }); + results.AddRange(chunkResults); + foreach (var r in chunkResults) { - // Distinguish a connection-level fault - // (adapter not connected / transport down) from a genuine - // node-not-found. Connection-level faults must drive the - // reconnection state machine, not be retried as unresolved tags. - var connectionLevel = IsConnectionLevelFailure(ex); - results.Add(new SubscribeTagResult( - tagPath, AlreadySubscribed: false, Success: false, null, ex.Message, - ConnectionLevelFailure: connectionLevel)); + if (r.Success) + tagsToSeed.Add(r.TagPath); } } @@ -782,18 +831,129 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers }).PipeTo(self); } + /// + /// Subscribes a CHUNK of tags against and returns one row + /// per requested tag. An adapter that advertises + /// does it in one round trip; any other + /// adapter falls back to the historical per-tag loop, so the capability is purely + /// additive. + /// + /// + /// Partial-failure contract: a per-tag fault is a result row with + /// Success:false. A THROWN exception means the whole chunk failed, and is + /// classified with exactly as the per-tag path + /// does — connection-level faults drive the reconnect state machine, everything else + /// is treated as a tag-resolution failure and retried on the backoff timer. + /// + /// + /// Runs on a background task: it touches no actor state. + /// + private static async Task> SubscribeTagsAsync( + IDataConnection adapter, IReadOnlyList tags, SubscriptionCallback callback) + { + var results = new List(tags.Count); + if (tags.Count == 0) + return results; + + if (adapter is IBatchSubscribableConnection batchAdapter) + { + IReadOnlyList rows; + try + { + rows = await batchAdapter.SubscribeBatchAsync(tags, callback); + } + catch (Exception ex) + { + var connectionLevel = IsConnectionLevelFailure(ex); + foreach (var tagPath in tags) + { + results.Add(new SubscribeTagResult( + tagPath, AlreadySubscribed: false, Success: false, null, ex.Message, + ConnectionLevelFailure: connectionLevel)); + } + return results; + } + + var byTag = new Dictionary(rows.Count, StringComparer.Ordinal); + foreach (var row in rows) + byTag[row.TagPath] = row; + + foreach (var tagPath in tags) + { + results.Add(byTag.TryGetValue(tagPath, out var row) + ? new SubscribeTagResult( + tagPath, AlreadySubscribed: false, row.Success, row.SubscriptionId, row.ErrorMessage) + : new SubscribeTagResult( + tagPath, AlreadySubscribed: false, Success: false, null, + "Adapter returned no subscribe result for this tag.")); + } + + return results; + } + + foreach (var tagPath in tags) + { + try + { + var subId = await adapter.SubscribeAsync(tagPath, callback); + results.Add(new SubscribeTagResult(tagPath, AlreadySubscribed: false, Success: true, subId, null)); + } + catch (Exception ex) + { + // Distinguish a connection-level fault + // (adapter not connected / transport down) from a genuine + // node-not-found. Connection-level faults must drive the + // reconnection state machine, not be retried as unresolved tags. + var connectionLevel = IsConnectionLevelFailure(ex); + results.Add(new SubscribeTagResult( + tagPath, AlreadySubscribed: false, Success: false, null, ex.Message, + ConnectionLevelFailure: connectionLevel)); + } + } + + return results; + } + + /// + /// Releases adapter subscription handles, in one round trip where the adapter supports + /// it. Fire-and-forget at every call site (the actor never awaits a release), so + /// faults are swallowed the same way the per-tag UnsubscribeAsync calls were. + /// + private static Task UnsubscribeIdsAsync(IDataConnection adapter, IReadOnlyList subscriptionIds) + { + if (subscriptionIds.Count == 0) + return Task.CompletedTask; + + if (adapter is IBatchSubscribableConnection batchAdapter) + return batchAdapter.UnsubscribeBatchAsync(subscriptionIds); + + return Task.WhenAll(subscriptionIds.Select(id => adapter.UnsubscribeAsync(id))); + } + /// /// Reads the current value of each tag so the Instance Actor /// has an initial value, retrying the still-empty subset a bounded number of times. A /// STATIC tag (one that emits no further OnDataChange after the advise) depends /// entirely on this seed; on a cold/fresh advise the read can race the just-created /// subscription and return an empty/failed result, which pre-fix was swallowed — - /// leaving the attribute Uncertain forever even though the source reads Good. Per-tag - /// (not a single bulk read) is deliberate: - /// some gateways time out on a large batch. The retry delay is applied once per round - /// across the whole pending subset, so total added latency is bounded to - /// (attempts - 1) × SeedReadRetryDelay regardless of tag count. Tags still - /// empty after the budget are logged (named) and left to heal from a future change. + /// leaving the attribute Uncertain forever even though the source reads Good. + /// + /// + /// Reads are issued as CHUNKED bulk reads ( + /// tags per chunk, chunks in + /// flight) against a batch-capable adapter, and per tag otherwise. + /// now bounds a CHUNK rather than a + /// single tag — chunking plus that per-chunk bound answers the "some gateways time out + /// on a large batch" caveat that originally motivated per-tag reads — and the whole + /// seed (all chunks, all retry rounds) is bounded by + /// , replacing the old + /// 30s-per-tag serial worst case. + /// + /// + /// The retry delay is applied once per round across the whole pending subset, so total + /// added latency is bounded to (attempts - 1) × SeedReadRetryDelay regardless of + /// tag count. Tags still empty after the budget (or when the deadline expires) are + /// logged (named) and left to heal from a future change. /// Runs on a background task: it reads only the supplied /// and returns the seeds — all actor-state mutation/delivery stays on the actor thread. /// @@ -805,35 +965,60 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers var pending = new HashSet(tags); var attempts = Math.Max(1, _options.SeedReadMaxAttempts); - for (var attempt = 1; attempt <= attempts && pending.Count > 0; attempt++) + var chunkSize = Math.Max(1, _options.SeedReadBatchSize); + var parallelism = Math.Max(1, _options.SeedReadMaxParallelism); + var useBatch = adapter is IBatchSubscribableConnection; + + // ONE deadline across every chunk and every retry round. + using var overall = new CancellationTokenSource(_options.SeedOverallTimeout); + var deadlineHit = false; + + for (var attempt = 1; attempt <= attempts && pending.Count > 0 && !overall.IsCancellationRequested; attempt++) { - foreach (var tagPath in pending.ToList()) + var round = pending.ToList(); + var chunks = new List>(); + for (var offset = 0; offset < round.Count; offset += chunkSize) + chunks.Add(round.GetRange(offset, Math.Min(chunkSize, round.Count - offset))); + + var gate = new SemaphoreSlim(parallelism); + var chunkTasks = chunks.Select(async chunk => { - // Bound each per-tag read with SeedReadTimeout so a - // hung device read cannot delay SubscribeCompleted/the ack indefinitely. - // On timeout the catch block treats it identically to any other failed seed - // read — the tag stays in pending, is retried up to SeedReadMaxAttempts, and - // left Uncertain if still empty after the budget. Same CancellationTokenSource - // mechanism used by HandleWrite for WriteTimeout. - using var cts = new CancellationTokenSource(_options.SeedReadTimeout); + await gate.WaitAsync(CancellationToken.None); try { - var readResult = await adapter.ReadAsync(tagPath, cts.Token); - if (readResult.Success && readResult.Value is { Value: not null } value) - { - seedValues.Add(new SeededValue(tagPath, value)); - pending.Remove(tagPath); - } + return await ReadSeedChunkAsync(adapter, chunk, useBatch, overall.Token); } - catch + finally { - // Best-effort read — retried below, or logged once the budget is spent. - // Includes OperationCanceledException on SeedReadTimeout expiry. + gate.Release(); } + }).ToList(); + + var chunkResults = await Task.WhenAll(chunkTasks); + foreach (var seeded in chunkResults.SelectMany(r => r)) + { + if (pending.Remove(seeded.TagPath)) + seedValues.Add(seeded); + } + + if (overall.IsCancellationRequested) + { + deadlineHit = true; + break; } if (pending.Count > 0 && attempt < attempts) - await Task.Delay(_options.SeedReadRetryDelay); + { + try + { + await Task.Delay(_options.SeedReadRetryDelay, overall.Token); + } + catch (OperationCanceledException) + { + deadlineHit = true; + break; + } + } } if (pending.Count > 0) @@ -843,13 +1028,68 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers if (pending.Count > maxNamed) named += $", … (+{pending.Count - maxNamed} more)"; _log.Warning( - "[{0}] Seed read returned no value for {1} tag(s) after {2} attempt(s); they stay Uncertain until a change notification arrives: {3}", - _connectionName, pending.Count, attempts, named); + "[{0}] Seed read returned no value for {1} tag(s) after {2} attempt(s){3}; they stay Uncertain until a change notification arrives: {4}", + _connectionName, pending.Count, attempts, + deadlineHit ? $" (overall {_options.SeedOverallTimeout.TotalSeconds:F0}s seed deadline expired)" : string.Empty, + named); } return seedValues; } + /// + /// Reads one seed chunk, bounded by + /// and by the caller's overall seed deadline. Every failure mode (bad read, chunk + /// timeout, deadline) yields "no seed for these tags", which the caller retries or + /// leaves Uncertain — identical to the historical per-tag behaviour. + /// + private async Task> ReadSeedChunkAsync( + IDataConnection adapter, IReadOnlyList chunk, bool useBatch, CancellationToken overallToken) + { + var seeded = new List(chunk.Count); + using var cts = CancellationTokenSource.CreateLinkedTokenSource(overallToken); + cts.CancelAfter(_options.SeedReadTimeout); + + if (useBatch) + { + try + { + var results = await adapter.ReadBatchAsync(chunk, cts.Token); + foreach (var tagPath in chunk) + { + if (results.TryGetValue(tagPath, out var readResult) + && readResult.Success && readResult.Value is { Value: not null } value) + { + seeded.Add(new SeededValue(tagPath, value)); + } + } + } + catch + { + // Best-effort read — retried by the caller, or logged once the budget is + // spent. Includes OperationCanceledException on the chunk/overall deadline. + } + + return seeded; + } + + foreach (var tagPath in chunk) + { + try + { + var readResult = await adapter.ReadAsync(tagPath, cts.Token); + if (readResult.Success && readResult.Value is { Value: not null } value) + seeded.Add(new SeededValue(tagPath, value)); + } + catch + { + // Best-effort read — see above. + } + } + + return seeded; + } + /// /// Applies the result of an asynchronous subscribe on the actor thread. ALL mutation /// of subscription state and counters happens here — never on the background task — @@ -880,6 +1120,7 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers "{2} adapter handle(s) and discarding state mutations.", _connectionName, instanceName, msg.Results.Count(r => r.Success && !r.AlreadySubscribed)); + var orphanedIds = new List(); foreach (var result in msg.Results) { // Clear in-flight markers we placed in HandleSubscribe. @@ -891,10 +1132,12 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers // owns the adapter handle and unsubscribing it would break them. if (result is { Success: true, AlreadySubscribed: false, SubscriptionId: not null }) { - _ = _adapter.UnsubscribeAsync(result.SubscriptionId); + orphanedIds.Add(result.SubscriptionId); } } + _ = UnsubscribeIdsAsync(_adapter, orphanedIds); + // The original sender is already gone (unsubscribed). Telling a dead // ref produces a dead letter, which is the harmless and observable // outcome — but skipping the reply altogether keeps dead-letter noise @@ -1012,20 +1255,7 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers } // Start the tag-resolution retry timer if any tags are unresolved. - // StartPeriodicTimer with an existing key CANCELS - // and replaces the prior timer, so a fan-out of SubscribeTagsRequests - // arriving faster than TagResolutionRetryInterval would keep resetting - // the timer and starve the retry indefinitely. Gating on IsTimerActive - // means the first failure starts the timer and subsequent failures - // simply pile onto _unresolvedTags without restarting the clock. - if (_unresolvedTags.Count > 0 && !Timers.IsTimerActive("tag-resolution-retry")) - { - Timers.StartPeriodicTimer( - "tag-resolution-retry", - new RetryTagResolution(), - _options.TagResolutionRetryInterval, - _options.TagResolutionRetryInterval); - } + ScheduleTagResolutionRetry(); // The response must match the actor's own assessment. // When a connection-level failure is driving the actor into Reconnecting, the @@ -1070,6 +1300,10 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers if (!_subscriptionsByInstance.TryGetValue(request.InstanceUniqueName, out var tags)) return; + // Released adapter handles are collected and released in ONE round trip below + // (batch-capable adapters) instead of one call per tag. + var idsToRelease = new List(); + // Cleanup on Instance Actor stop foreach (var tagPath in tags) { @@ -1095,7 +1329,7 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers // an unconditional Remove was always-true dead logic). if (_subscriptionIds.TryGetValue(tagPath, out var subId)) { - _ = _adapter.UnsubscribeAsync(subId); + idsToRelease.Add(subId); _subscriptionIds.Remove(tagPath); _resolutionInFlight.Remove(tagPath); _totalSubscribed--; @@ -1130,9 +1364,13 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers _subscriptionsByInstance.Remove(request.InstanceUniqueName); _subscribers.Remove(request.InstanceUniqueName); + // Fire-and-forget release of every handle this unsubscribe orphaned. + _ = UnsubscribeIdsAsync(_adapter, idsToRelease); + // Keep the reported quality counters in sync after the - // unsubscribed tags' buckets were decremented above. - _healthCollector.UpdateTagQuality(_connectionName, _tagsGoodQuality, _tagsBadQuality, _tagsUncertainQuality); + // unsubscribed tags' buckets were decremented above. SYNCHRONOUS flush: the + // coalescing timer must never delay a count that just dropped. + FlushQualityCounters(); _healthCollector.UpdateTagResolution(_connectionName, _totalSubscribed, _resolvedTags); } @@ -1544,7 +1782,7 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers var self = Self; // Only dispatch retries for tags that do not already - // have an attempt in flight. A slow SubscribeAsync overlapping the next tick + // have an attempt in flight. A slow subscribe overlapping the next tick // would otherwise produce duplicate concurrent subscribes for the same tag. var toResolve = _unresolvedTags.Where(t => !_resolutionInFlight.Contains(t)).ToList(); @@ -1555,22 +1793,168 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers return; } - _log.Debug("[{0}] Retrying resolution for {1} unresolved tags", _connectionName, toResolve.Count); + _log.Debug("[{0}] Retrying resolution for {1} unresolved tags (interval {2:F0}s)", + _connectionName, toResolve.Count, _tagResolutionInterval.TotalSeconds); var generation = _adapterGeneration; + var adapter = _adapter; + var batchSize = Math.Max(1, _options.SubscribeBatchSize); + foreach (var tagPath in toResolve) - { _resolutionInFlight.Add(tagPath); - _adapter.SubscribeAsync(tagPath, (path, value) => + + // One probe ROUND = chunked batch subscribes over the whole not-in-flight + // unresolved set, reported back as a single completion. Rescheduling happens on + // that completion (never on a periodic tick), which is what makes the backoff + // possible AND preserves the anti-starvation property the old IsTimerActive gate + // defended: a fan-out of failures can never reset the clock. + Task.Run(async () => + { + var rows = new List(toResolve.Count); + for (var offset = 0; offset < toResolve.Count; offset += batchSize) { - self.Tell(new TagValueReceived(path, value, generation)); - }).ContinueWith(t => - { - if (t.IsCompletedSuccessfully) - return new TagResolutionSucceeded(tagPath, t.Result) as object; - return new TagResolutionFailed(tagPath, t.Exception?.GetBaseException().Message ?? "Unknown error"); - }).PipeTo(self); + var chunk = toResolve.GetRange(offset, Math.Min(batchSize, toResolve.Count - offset)); + rows.AddRange(await SubscribeTagsAsync(adapter, chunk, (path, value) => + { + self.Tell(new TagValueReceived(path, value, generation)); + })); + } + + return new BatchSubscribeCompleted(rows, generation, BatchSubscribeSource.ResolutionProbe); + }).PipeTo(self); + } + + /// + /// Arms the single-shot tag-resolution retry timer at the current backoff interval + /// when unresolved tags remain, and cancels it when none do. Single-shot (rescheduled + /// on probe completion) rather than periodic — a periodic timer cannot back off, and + /// re-arming a periodic timer on every failure is what starved the retry before + /// (DataConnectionLayer-022). Gating on IsTimerActive keeps a burst of failed + /// subscribes from pushing the already-running deadline out. + /// + private void ScheduleTagResolutionRetry() + { + if (_unresolvedTags.Count == 0) + { + Timers.Cancel("tag-resolution-retry"); + return; } + + if (Timers.IsTimerActive("tag-resolution-retry")) + return; + + Timers.StartSingleTimer("tag-resolution-retry", new RetryTagResolution(), _tagResolutionInterval); + } + + /// Resets the tag-resolution backoff to its floor (a tag resolved, or the connection reconnected). + private void ResetTagResolutionBackoff() => _tagResolutionInterval = _options.TagResolutionRetryInterval; + + /// + /// Doubles the tag-resolution retry interval, capped at + /// . Called only after + /// a probe round in which NOTHING resolved, so a device that is merely slow to boot is + /// still picked up at the floor interval. + /// + private void BackOffTagResolution() => + _tagResolutionInterval = NextTagResolutionInterval( + _tagResolutionInterval, _options.TagResolutionRetryInterval, _options.TagResolutionRetryMaxInterval); + + /// + /// Pure backoff step: the interval after a fully-failed probe round — double the + /// current one, never above (itself never below + /// , so a misconfigured ceiling degrades to a fixed interval + /// rather than shrinking below the floor). + /// + /// Interval that has just been used. + /// Configured floor interval. + /// Configured ceiling interval. + /// The next retry interval. + internal static TimeSpan NextTagResolutionInterval(TimeSpan current, TimeSpan floor, TimeSpan max) + { + var ceiling = max < floor ? floor : max; + if (current < floor) + current = floor; + + var doubled = current + current; + return doubled > ceiling ? ceiling : doubled; + } + + /// + /// Applies one chunk of batch-subscribe results on the actor thread — the shared tail + /// of the reconnect re-subscribe and the tag-resolution probe. Both paths subscribe + /// tags that are ALREADY counted in (they come from + /// ), so neither touches that counter. + /// + private void HandleBatchSubscribeCompleted(BatchSubscribeCompleted msg) + { + // Results produced by a disposed adapter (post-failover) are dropped, mirroring + // the TagValueReceived generation guard. + if (msg.Generation != _adapterGeneration) + { + _log.Debug("[{0}] Dropping {1} stale batch-subscribe result(s) from adapter generation {2} (current {3})", + _connectionName, msg.Results.Count, msg.Generation, _adapterGeneration); + return; + } + + var anyResolved = false; + var duplicateIds = new List(); + + foreach (var row in msg.Results) + { + _resolutionInFlight.Remove(row.TagPath); + + if (row is { Success: true, SubscriptionId: not null }) + { + var wasUnresolved = _unresolvedTags.Remove(row.TagPath); + if (_subscriptionIds.ContainsKey(row.TagPath)) + { + // Another path already stored a handle for this tag while this one was + // in flight — release the redundant handle instead of leaking it + // (mirrors the duplicate-alarm-feed guard). + duplicateIds.Add(row.SubscriptionId); + } + else + { + _subscriptionIds[row.TagPath] = row.SubscriptionId; + _resolvedTags++; + anyResolved = true; + if (wasUnresolved) + _log.Info("[{0}] Tag resolved: {1}", _connectionName, row.TagPath); + } + } + else if (row.ConnectionLevelFailure) + { + // Connection-level fault: not a tag-resolution problem. The reconnect + // cycle re-issues everything from _subscriptionsByInstance. + _log.Warning("[{0}] Batch subscribe for {1} failed at connection level: {2}", + _connectionName, row.TagPath, row.Error); + } + else + { + _log.Debug("[{0}] Tag resolution still failing for {1}: {2}", + _connectionName, row.TagPath, row.Error); + _unresolvedTags.Add(row.TagPath); + } + } + + _ = UnsubscribeIdsAsync(_adapter, duplicateIds); + _healthCollector.UpdateTagResolution(_connectionName, _totalSubscribed, _resolvedTags); + + // Backoff bookkeeping belongs to the probe round only: a reconnect re-subscribe + // chunk is not a "retry round" and must not double the interval. + if (msg.Source == BatchSubscribeSource.ResolutionProbe) + { + if (anyResolved) + ResetTagResolutionBackoff(); + else + BackOffTagResolution(); + } + else if (anyResolved) + { + ResetTagResolutionBackoff(); + } + + ScheduleTagResolutionRetry(); } // ── Bad Quality Push ── @@ -1592,6 +1976,44 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers _tagsBadQuality = _lastTagQuality.Count; foreach (var key in _lastTagQuality.Keys.ToList()) _lastTagQuality[key] = QualityCode.Bad; + // SYNCHRONOUS — "immediate bad quality on disconnect" must never be deferred to + // the quality-flush coalescing timer. + FlushQualityCounters(); + } + + // ── Quality counter flush ── + + /// + /// Marks the quality counters dirty and arms the coalescing flush timer. Called only + /// on a genuine quality TRANSITION — an unchanged quality moves no counter, so there + /// is nothing to push. + /// + private void MarkQualityDirty() + { + _qualityDirty = true; + if (!Timers.IsTimerActive("quality-flush")) + Timers.StartSingleTimer("quality-flush", new QualityFlushTick(), _options.QualityFlushInterval); + } + + /// + /// Coalescing-timer tick: pushes only when a transition is still pending. A tick that + /// races a synchronous flush (disconnect / unsubscribe) does nothing. + /// + private void FlushQualityCountersIfDirty() + { + if (_qualityDirty) + FlushQualityCounters(); + } + + /// + /// Pushes the current quality counters to the health collector and disarms the + /// coalescing timer. Used both by the timer tick and by the paths whose correctness + /// depends on an immediate push (disconnect, unsubscribe, reconnect reset). + /// + private void FlushQualityCounters() + { + _qualityDirty = false; + Timers.Cancel("quality-flush"); _healthCollector.UpdateTagQuality(_connectionName, _tagsGoodQuality, _tagsBadQuality, _tagsUncertainQuality); } @@ -1632,25 +2054,51 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers _tagsGoodQuality = 0; _tagsBadQuality = 0; _tagsUncertainQuality = 0; - _healthCollector.UpdateTagQuality(_connectionName, _tagsGoodQuality, _tagsBadQuality, _tagsUncertainQuality); + // SYNCHRONOUS reset push — never deferred to the coalescing timer. + FlushQualityCounters(); + + // A reconnect starts from the retry floor: the device just answered, so any + // backoff accumulated against the previous outage is stale. + ResetTagResolutionBackoff(); var generation = _adapterGeneration; - // Capture the adapter up front (S7), symmetric with reseedAdapter below. This loop - // runs on the actor thread so a mid-loop swap can't happen, but reading the local - // keeps every adapter access in this method tied to one generation. + // Capture the adapter up front (S7), symmetric with reseedAdapter below, so every + // chunk runs against the adapter that started the re-subscribe even if a later + // failover swaps the _adapter field mid-flight. var subscribeAdapter = _adapter; - foreach (var tagPath in allTags) + var batchSize = Math.Max(1, _options.SubscribeBatchSize); + var chunkDelay = _options.SubscribeBatchDelay; + + // Bounded, sequential chunks in ONE background task — replacing the previous + // "one fire-and-forget task per tag" storm (37,500 concurrent subscribes at the + // sizing target). Sequential because subscription creation on one session is + // serialized by the adapter's apply lock anyway, and sequencing bounds device load; + // the inter-chunk delay paces it further. Each chunk reports its per-tag results + // back to the actor as its own message, so progress is applied incrementally. + Task.Run(async () => { - subscribeAdapter.SubscribeAsync(tagPath, (path, value) => + for (var offset = 0; offset < allTags.Count; offset += batchSize) { - self.Tell(new TagValueReceived(path, value, generation)); - }).ContinueWith(t => - { - if (t.IsCompletedSuccessfully) - return new TagResolutionSucceeded(tagPath, t.Result) as object; - return new TagResolutionFailed(tagPath, t.Exception?.GetBaseException().Message ?? "Unknown error"); - }).PipeTo(self); - } + if (offset > 0 && chunkDelay > TimeSpan.Zero) + await Task.Delay(chunkDelay); + + var chunk = allTags.GetRange(offset, Math.Min(batchSize, allTags.Count - offset)); + try + { + var rows = await SubscribeTagsAsync(subscribeAdapter, chunk, (path, value) => + { + self.Tell(new TagValueReceived(path, value, generation)); + }); + self.Tell(new BatchSubscribeCompleted(rows, generation, BatchSubscribeSource.Resubscribe)); + } + catch (Exception ex) + { + // SubscribeTagsAsync already converts faults into rows; reaching here is + // unexpected, so guarantee a trace rather than an unobserved task fault. + _log.Warning("[{0}] Reconnect re-subscribe chunk faulted: {1}", _connectionName, ex.Message); + } + } + }); // Re-advising alone does NOT restore a STATIC tag's value. // PushBadQualityForAllTags flipped every tag Bad on disconnect, and a tag that fires @@ -1697,47 +2145,6 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers // ── Internal message handlers for piped async results ── - private void HandleTagResolutionSucceeded(TagResolutionSucceeded msg) - { - // The retry attempt for this tag has completed. - _resolutionInFlight.Remove(msg.TagPath); - - if (_unresolvedTags.Remove(msg.TagPath)) - { - _subscriptionIds[msg.TagPath] = msg.SubscriptionId; - _resolvedTags++; - _healthCollector.UpdateTagResolution(_connectionName, _totalSubscribed, _resolvedTags); - _log.Info("[{0}] Tag resolved: {1}", _connectionName, msg.TagPath); - } - - if (_unresolvedTags.Count == 0) - { - Timers.Cancel("tag-resolution-retry"); - } - } - - private void HandleTagResolutionFailed(TagResolutionFailed msg) - { - _log.Debug("[{0}] Tag resolution still failing for {1}: {2}", - _connectionName, msg.TagPath, msg.Error); - - // The retry attempt for this tag has completed — - // it is eligible for the next retry tick again. - _resolutionInFlight.Remove(msg.TagPath); - - // Track as unresolved so periodic retry picks it up. Gate on - // IsTimerActive so a stream of TagResolutionFailed events doesn't keep - // cancelling and re-starting the timer faster than its own interval. - if (_unresolvedTags.Add(msg.TagPath) && !Timers.IsTimerActive("tag-resolution-retry")) - { - Timers.StartPeriodicTimer( - "tag-resolution-retry", - new RetryTagResolution(), - _options.TagResolutionRetryInterval, - _options.TagResolutionRetryInterval); - } - } - /// /// Adds to the reverse index for . /// Call at every site that adds a tag to an instance's set. @@ -1789,8 +2196,17 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers } } - // Track quality transitions - if (_lastTagQuality.TryGetValue(msg.TagPath, out var prevQuality)) + // Track quality transitions. A value whose quality is UNCHANGED moves no bucket — + // the old code still ran the decrement/increment pair and pushed the identical + // counters to the health collector on every single value. At 37,500 tags that push + // was the dominant cost of the value hot path, against a collector only read every + // 30s. Now: unchanged quality is a no-op; a genuine transition updates the buckets + // and arms the coalescing flush timer. + var hadPrevious = _lastTagQuality.TryGetValue(msg.TagPath, out var prevQuality); + if (hadPrevious && prevQuality == msg.Value.Quality) + return; + + if (hadPrevious) { // Decrement old quality bucket switch (prevQuality) @@ -1808,7 +2224,7 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers case QualityCode.Uncertain: _tagsUncertainQuality++; break; } _lastTagQuality[msg.TagPath] = msg.Value.Quality; - _healthCollector.UpdateTagQuality(_connectionName, _tagsGoodQuality, _tagsBadQuality, _tagsUncertainQuality); + MarkQualityDirty(); } // ── Native alarm subscriptions ── @@ -1832,6 +2248,7 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers { subs = new HashSet(); _alarmSourceSubscribers[request.SourceReference] = subs; + IndexAlarmSource(request.SourceReference); } subs.Add(subscriber); // S10: the adapter feed carries a single shared condition filter per source. @@ -1971,8 +2388,11 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers return; } - foreach (var (sourceRef, subs) in _alarmSourceSubscribers) + foreach (var sourceRef in CandidateAlarmSources(transition)) { + if (!_alarmSourceSubscribers.TryGetValue(sourceRef, out var subs)) + continue; + // A subscriber bound to source S receives a transition whose source // object (or full reference) falls under S. var match = transition.SourceObjectReference.StartsWith(sourceRef, StringComparison.Ordinal) @@ -1996,6 +2416,86 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers } } + /// + /// First path segment of a source reference (everything before the first .), or + /// the whole reference when it carries no separator. + /// + private static string FirstSegment(string reference) + { + var separator = reference.IndexOf('.', StringComparison.Ordinal); + return separator < 0 ? reference : reference[..separator]; + } + + /// + /// Adds a source reference to the routing index. A reference containing a separator + /// is bucketed by its first segment; one WITHOUT a separator is a prefix shorter than + /// a full segment (it can match transitions whose first segment merely starts with it) + /// and goes to the small residue list that every transition is checked against. + /// + private void IndexAlarmSource(string sourceReference) + { + if (sourceReference.IndexOf('.', StringComparison.Ordinal) < 0) + { + _alarmSourcesWithoutSeparator.Add(sourceReference); + return; + } + + var segment = FirstSegment(sourceReference); + if (!_alarmSourcesByFirstSegment.TryGetValue(segment, out var bucket)) + { + bucket = new HashSet(StringComparer.Ordinal); + _alarmSourcesByFirstSegment[segment] = bucket; + } + bucket.Add(sourceReference); + } + + /// Removes a source reference from the routing index, dropping an emptied bucket. + private void UnindexAlarmSource(string sourceReference) + { + if (sourceReference.IndexOf('.', StringComparison.Ordinal) < 0) + { + _alarmSourcesWithoutSeparator.Remove(sourceReference); + return; + } + + var segment = FirstSegment(sourceReference); + if (_alarmSourcesByFirstSegment.TryGetValue(segment, out var bucket) + && bucket.Remove(sourceReference) && bucket.Count == 0) + { + _alarmSourcesByFirstSegment.Remove(segment); + } + } + + /// + /// Source references that could possibly match this transition: the buckets for the + /// first segment of its source-object and full references, plus the separator-less + /// residue. Sound because a source reference containing a separator can only be a + /// prefix of a reference sharing its ENTIRE first segment — everything before the + /// first . must match character for character. The caller still applies the + /// full StartsWith test and the condition-type gate, so routing is identical to the + /// previous linear scan. + /// + private IEnumerable CandidateAlarmSources(NativeAlarmTransition transition) + { + var objectSegment = FirstSegment(transition.SourceObjectReference); + if (_alarmSourcesByFirstSegment.TryGetValue(objectSegment, out var objectBucket)) + { + foreach (var sourceRef in objectBucket) + yield return sourceRef; + } + + var referenceSegment = FirstSegment(transition.SourceReference); + if (!string.Equals(referenceSegment, objectSegment, StringComparison.Ordinal) + && _alarmSourcesByFirstSegment.TryGetValue(referenceSegment, out var referenceBucket)) + { + foreach (var sourceRef in referenceBucket) + yield return sourceRef; + } + + foreach (var sourceRef in _alarmSourcesWithoutSeparator) + yield return sourceRef; + } + private void HandleUnsubscribeAlarms(UnsubscribeAlarmsRequest request) { if (!_alarmSourceSubscribers.TryGetValue(request.SourceReference, out var subs)) @@ -2007,6 +2507,7 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers // No subscribers remain for this source — tear down the adapter feed. _alarmSourceSubscribers.Remove(request.SourceReference); + UnindexAlarmSource(request.SourceReference); _alarmSourceFilter.Remove(request.SourceReference); _alarmSourceFilterPredicate.Remove(request.SourceReference); // Clear the in-flight marker so that if an adapter @@ -2064,9 +2565,27 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers internal record ConnectResult(bool Success, string? Error); internal record AdapterDisconnected; internal record TagValueReceived(string TagPath, TagValue Value, int AdapterGeneration); - internal record TagResolutionFailed(string TagPath, string Error); - internal record TagResolutionSucceeded(string TagPath, string SubscriptionId); internal record RetryTagResolution; + + /// Coalescing-timer tick for the tag-quality counter push. + internal record QualityFlushTick; + + /// Which path produced a . + internal enum BatchSubscribeSource + { + /// One chunk of the post-reconnect transparent re-subscribe. + Resubscribe, + + /// A whole tag-resolution retry round (all chunks), which drives the backoff. + ResolutionProbe + } + + /// + /// Per-tag results of a batch subscribe issued off the actor thread. Carries the + /// adapter generation it ran against so post-failover results are dropped. + /// + internal record BatchSubscribeCompleted( + IReadOnlyList Results, int Generation, BatchSubscribeSource Source); internal record SubscribeTagResult( string TagPath, bool AlreadySubscribed, bool Success, string? SubscriptionId, string? Error, bool ConnectionLevelFailure = false); diff --git a/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/AlarmFilterPrefix.cs b/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/AlarmFilterPrefix.cs new file mode 100644 index 00000000..e5e2e368 --- /dev/null +++ b/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/AlarmFilterPrefix.cs @@ -0,0 +1,63 @@ +namespace ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Adapters; + +/// +/// Computes the single source-reference prefix an MxAccess Gateway alarm stream can be +/// opened with while still covering every subscribed source. StreamAlarms carries +/// ONE alarm_filter_prefix, so the only pushable union of N source references is +/// their longest common prefix. +/// +/// +/// The prefix is a BANDWIDTH optimisation, never correctness: the DataConnectionActor's +/// per-source match plus condition-type gate stays authoritative, mirroring the OPC UA +/// server-side WhereClause stance. A too-broad prefix (including the empty, +/// gateway-wide one) is therefore always safe. +/// +/// +internal static class AlarmFilterPrefix +{ + /// + /// Longest common prefix of the supplied source references, or the empty string when + /// they share none (or when any source is empty — an empty source means "everything"). + /// + /// Currently subscribed source references. + /// The prefix to open the gateway alarm stream with; empty = gateway-wide. + public static string LongestCommonPrefix(IEnumerable sourceReferences) + { + string? prefix = null; + foreach (var source in sourceReferences) + { + if (string.IsNullOrEmpty(source)) + return string.Empty; + + if (prefix is null) + { + prefix = source; + continue; + } + + var max = Math.Min(prefix.Length, source.Length); + var common = 0; + while (common < max && prefix[common] == source[common]) + common++; + + if (common == 0) + return string.Empty; + prefix = prefix[..common]; + } + + return prefix ?? string.Empty; + } + + /// + /// Whether an existing stream opened with already + /// covers . Only a source falling OUTSIDE the + /// current prefix justifies restarting the stream; an unsubscribe never does, + /// because the prefix it leaves behind is at worst too broad — which costs + /// bandwidth, not correctness. + /// + /// Prefix the live stream was opened with. + /// Newly subscribed source reference. + /// true when the live stream already carries this source. + public static bool Covers(string currentPrefix, string sourceReference) => + currentPrefix.Length == 0 || sourceReference.StartsWith(currentPrefix, StringComparison.Ordinal); +} diff --git a/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/AsyncSerialGate.cs b/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/AsyncSerialGate.cs new file mode 100644 index 00000000..88f44cac --- /dev/null +++ b/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/AsyncSerialGate.cs @@ -0,0 +1,55 @@ +namespace ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Adapters; + +/// +/// Serializes async critical sections — one caller inside at a time, FIFO. Used by +/// so every structural mutation of an OPC UA Subscription +/// (AddItem / RemoveItem / ApplyChanges / Create / Delete) and every shard-list mutation +/// runs alone: the OPC Foundation Subscription object is not safe under concurrent +/// structural mutation, and nothing else orders the subscribe background task, the +/// tag-resolution probe and an alarm subscribe against each other. +/// +/// +/// A named type rather than a bare field so the "exactly one +/// gate, released on every path" discipline is visible at each call site and testable on +/// its own. +/// +/// +internal sealed class AsyncSerialGate +{ + private readonly SemaphoreSlim _gate = new(1, 1); + + /// Runs with no other gated section in flight. + /// The critical section. + /// Cancellation token observed while waiting to enter. + /// A task that completes when the section has run and the gate is released. + public async Task RunAsync(Func action, CancellationToken cancellationToken = default) + { + await _gate.WaitAsync(cancellationToken).ConfigureAwait(false); + try + { + await action().ConfigureAwait(false); + } + finally + { + _gate.Release(); + } + } + + /// Value-returning counterpart of . + /// Result type of the critical section. + /// The critical section. + /// Cancellation token observed while waiting to enter. + /// The section's result. + public async Task RunAsync(Func> action, CancellationToken cancellationToken = default) + { + await _gate.WaitAsync(cancellationToken).ConfigureAwait(false); + try + { + return await action().ConfigureAwait(false); + } + finally + { + _gate.Release(); + } + } +} diff --git a/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/BulkPipeline.cs b/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/BulkPipeline.cs new file mode 100644 index 00000000..3ad42328 --- /dev/null +++ b/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/BulkPipeline.cs @@ -0,0 +1,79 @@ +namespace ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Adapters; + +/// +/// Bounded-parallel execution of a per-item operation. Used where a protocol offers no +/// bulk form of a command and the only alternative is N serial round trips — today the +/// MxAccess Gateway's supervisory advise, which exists per item only (the worker's +/// SubscribeBulk issues plain advises). Pipelining 37,500 tags at a window of 16 +/// replaces 37,500 serial round trips with ~2,344 windows. +/// +/// +/// Factored out of the gateway client so the concurrency bound is unit-testable without +/// a live MXAccess session. +/// +/// +internal static class BulkPipeline +{ + /// + /// Runs for every item with at most + /// in flight, preserving result order. A per-item + /// fault is captured into that item's result slot via + /// rather than aborting the pipeline; cancellation propagates. + /// + /// Input item type. + /// Per-item result type. + /// Items to process. + /// Maximum operations in flight; values < 1 are treated as 1. + /// The per-item operation. + /// Maps an item plus its exception onto a result row. + /// Cancellation token. + /// One result per item, in input order. + public static async Task RunAsync( + IReadOnlyList items, + int maxParallelism, + Func> operation, + Func onError, + CancellationToken ct = default) + { + var results = new TResult[items.Count]; + if (items.Count == 0) + return results; + + // Deliberately NOT disposed: on cancellation Task.WhenAll rethrows while sibling + // operations may still be unwinding, and a disposed SemaphoreSlim would turn that + // into an ObjectDisposedException. SemaphoreSlim without AvailableWaitHandle holds + // no unmanaged resource, so letting it be collected is safe. + var gate = new SemaphoreSlim(Math.Max(1, maxParallelism)); + var tasks = new Task[items.Count]; + for (var i = 0; i < items.Count; i++) + { + var index = i; + var item = items[i]; + tasks[i] = RunOneAsync(item, index); + } + + await Task.WhenAll(tasks).ConfigureAwait(false); + return results; + + async Task RunOneAsync(TItem item, int index) + { + await gate.WaitAsync(ct).ConfigureAwait(false); + try + { + results[index] = await operation(item, ct).ConfigureAwait(false); + } + catch (OperationCanceledException) + { + throw; + } + catch (Exception ex) + { + results[index] = onError(item, ex); + } + finally + { + gate.Release(); + } + } + } +} diff --git a/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/IMxGatewayClient.cs b/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/IMxGatewayClient.cs index edb2f9f9..e91f0bae 100644 --- a/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/IMxGatewayClient.cs +++ b/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/IMxGatewayClient.cs @@ -6,7 +6,11 @@ namespace ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Adapters; /// Connection parameters resolved from the flat config dict. public record MxGatewayConnectionOptions( string Endpoint, string ApiKey, string ClientName, int WriteUserId, - bool UseTls, string? CaFile, string? ServerName, int ReadTimeoutMs); + bool UseTls, string? CaFile, string? ServerName, int ReadTimeoutMs, + // Maximum in-flight supervisory advise commands on the bulk-subscribe path when + // WriteUserId == 0 (the gateway worker has no BULK supervisory advise). Sourced from + // DataConnectionOptions.MxSupervisoryAdviseParallelism. + int SupervisoryAdviseParallelism = 16); /// One advised-tag value change pushed from the gateway event stream. public record MxValueUpdate(string TagPath, object? Value, QualityCode Quality, DateTimeOffset Timestamp); @@ -17,6 +21,13 @@ public record MxReadOutcome(string TagPath, bool Success, object? Value, Quality /// Per-tag write outcome. public record MxWriteOutcome(string TagPath, bool Success, string? Error); +/// Per-tag outcome of a bulk subscribe (AddItem + Advise in one gateway command). +/// The requested tag address. +/// Whether the item was added and advised. +/// Gateway item handle (as a string) when successful. +/// Per-tag failure reason when not successful. +public record MxSubscribeOutcome(string TagPath, bool Success, string? SubscriptionId, string? Error); + /// One node in a Galaxy browse level. public record MxBrowseChild(string NodeId, string DisplayName, BrowseNodeClass NodeClass, bool HasChildren, string? DataType = null); @@ -51,6 +62,25 @@ public interface IMxGatewayClient : IAsyncDisposable /// A task that represents the asynchronous operation. Task UnsubscribeAsync(string subscriptionId, CancellationToken ct = default); + /// + /// Adds and advises MANY tags in as few gateway commands as the worker allows — + /// ONE SubscribeBulk round trip in plain-advise mode, or one + /// AddItemBulk plus bounded-parallel supervisory advises when the connection + /// has no write-user context (the worker has no bulk supervisory advise). + /// Replaces the historical 2-RPC-per-tag AddItem + Advise pair. + /// + /// Tag addresses to subscribe. + /// Cancellation token. + /// One outcome per requested tag path, in request order. + Task> SubscribeBulkAsync( + IReadOnlyList tagPaths, CancellationToken ct = default); + + /// UnAdvise + RemoveItem for many subscription ids in one gateway command. + /// Subscription ids previously returned by a subscribe call. + /// Cancellation token. + /// A task that represents the asynchronous operation. + Task UnsubscribeBulkAsync(IReadOnlyList subscriptionIds, CancellationToken ct = default); + /// Snapshot read of one or more tags (ReadBulk). /// Tag addresses to read. /// Cancellation token. diff --git a/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/IOpcUaClient.cs b/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/IOpcUaClient.cs index 8430d58a..82c1d67d 100644 --- a/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/IOpcUaClient.cs +++ b/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/IOpcUaClient.cs @@ -26,7 +26,11 @@ public record OpcUaConnectionOptions( string SubscriptionDisplayName = "ScadaBridge", string TimestampsToReturn = "Source", OpcUaDeadbandOptions? Deadband = null, - OpcUaUserIdentityOptions? UserIdentity = null); + OpcUaUserIdentityOptions? UserIdentity = null, + // Monitored items above this ceiling are sharded onto an additional + // subscription on the same session (see RealOpcUaClient). Per endpoint, because + // item-count limits are a property of the target server. + int MaxMonitoredItemsPerSubscription = 5000); public record OpcUaDeadbandOptions(string Type, double Value); @@ -37,6 +41,32 @@ public record OpcUaUserIdentityOptions( string CertificatePath, string CertificatePassword); +/// +/// Per-node outcome of a batch monitored-item create. A node that fails to resolve or +/// whose monitored item comes back with a bad create status is reported as a failed row +/// — never by aborting the whole batch (a connection-level fault still throws). +/// +/// The requested node id, verbatim. +/// Whether the monitored item was created. +/// Handle for when successful. +/// Failure reason when not successful. +public record OpcUaSubscribeOutcome(string NodeId, bool Success, string? SubscriptionHandle, string? Error); + +/// Per-node outcome of a batch read. +/// The requested node id, verbatim. +/// The value read, when the read produced one. +/// Source timestamp reported by the server. +/// OPC UA status code for this node's read. +/// Set when the node could not be read at all (e.g. an unresolvable node id). +public record OpcUaReadOutcome( + string NodeId, object? Value, DateTime SourceTimestamp, uint StatusCode, string? Error); + +/// Per-node outcome of a batch write. +/// The requested node id, verbatim. +/// OPC UA status code for this node's write (0 = Good). +/// Set when the node could not be written at all (e.g. an unresolvable node id). +public record OpcUaWriteOutcome(string NodeId, uint StatusCode, string? Error); + /// /// Abstraction over OPC UA client library for testability. /// The real implementation would wrap an OPC UA SDK (e.g., OPC Foundation .NET Standard Library). @@ -74,6 +104,22 @@ public interface IOpcUaClient : IAsyncDisposable Action onValueChanged, CancellationToken cancellationToken = default); + /// + /// Creates monitored items for MANY nodes with ONE ApplyChanges per touched + /// subscription — the batch counterpart of and + /// the reason the DCL no longer issues one ApplyChanges per tag. All nodes share the + /// single callback (its first argument is the node + /// id, so per-node delegates would carry nothing extra). + /// + /// The node ids to monitor. + /// Callback invoked with (nodeId, value, sourceTimestamp, statusCode) on each change. + /// A cancellation token that can be used to cancel the operation. + /// One outcome per requested node id, in request order. + Task> CreateSubscriptionsAsync( + IReadOnlyList nodeIds, + Action onValueChanged, + CancellationToken cancellationToken = default); + /// /// Removes a monitored item subscription by handle. /// @@ -82,6 +128,16 @@ public interface IOpcUaClient : IAsyncDisposable /// A task representing the asynchronous operation. Task RemoveSubscriptionAsync(string subscriptionHandle, CancellationToken cancellationToken = default); + /// + /// Removes many monitored items, grouping ONE ApplyChanges per touched subscription. + /// Unknown handles are ignored. + /// + /// Handles previously returned by a create call. + /// A cancellation token that can be used to cancel the operation. + /// A task representing the asynchronous operation. + Task RemoveSubscriptionsAsync( + IReadOnlyList subscriptionHandles, CancellationToken cancellationToken = default); + /// /// Subscribes to OPC UA Alarms & Conditions events under /// (or the Server object when null). On @@ -124,6 +180,26 @@ public interface IOpcUaClient : IAsyncDisposable /// A task that completes with the OPC UA status code of the write operation. Task WriteValueAsync(string nodeId, object? value, CancellationToken cancellationToken = default); + /// + /// Reads many nodes in ONE OPC UA Read service call (chunked internally against the + /// server's operation limits). + /// + /// The node ids to read. + /// A cancellation token that can be used to cancel the operation. + /// One outcome per requested node id, in request order. + Task> ReadValuesAsync( + IReadOnlyList nodeIds, CancellationToken cancellationToken = default); + + /// + /// Writes many nodes in ONE OPC UA Write service call (chunked internally against the + /// server's operation limits). + /// + /// The node id / value pairs to write. + /// A cancellation token that can be used to cancel the operation. + /// One outcome per requested node id, in request order. + Task> WriteValuesAsync( + IReadOnlyList<(string NodeId, object? Value)> values, CancellationToken cancellationToken = default); + /// /// Raised when the OPC UA session detects a keep-alive failure or the server /// becomes unreachable. The adapter layer uses this to trigger reconnection. @@ -320,12 +396,29 @@ internal class StubOpcUaClient : IOpcUaClient return Task.FromResult(Guid.NewGuid().ToString()); } + /// + public Task> CreateSubscriptionsAsync( + IReadOnlyList nodeIds, + Action onValueChanged, + CancellationToken cancellationToken = default) + { + IReadOnlyList outcomes = nodeIds + .Select(n => new OpcUaSubscribeOutcome(n, true, Guid.NewGuid().ToString(), null)) + .ToList(); + return Task.FromResult(outcomes); + } + /// public Task RemoveSubscriptionAsync(string subscriptionHandle, CancellationToken cancellationToken = default) { return Task.CompletedTask; } + /// + public Task RemoveSubscriptionsAsync( + IReadOnlyList subscriptionHandles, CancellationToken cancellationToken = default) + => Task.CompletedTask; + /// public Task CreateAlarmSubscriptionAsync( string? sourceNodeId, string? conditionFilter, @@ -352,6 +445,26 @@ internal class StubOpcUaClient : IOpcUaClient return Task.FromResult(0); // Good status } + /// + public Task> ReadValuesAsync( + IReadOnlyList nodeIds, CancellationToken cancellationToken = default) + { + IReadOnlyList outcomes = nodeIds + .Select(n => new OpcUaReadOutcome(n, null, DateTime.UtcNow, 0u, null)) + .ToList(); + return Task.FromResult(outcomes); + } + + /// + public Task> WriteValuesAsync( + IReadOnlyList<(string NodeId, object? Value)> values, CancellationToken cancellationToken = default) + { + IReadOnlyList outcomes = values + .Select(v => new OpcUaWriteOutcome(v.NodeId, 0u, null)) + .ToList(); + return Task.FromResult(outcomes); + } + /// public Task BrowseChildrenAsync( string? parentNodeId, string? continuationToken = null, CancellationToken cancellationToken = default) diff --git a/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/MonitoredItemShardPlanner.cs b/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/MonitoredItemShardPlanner.cs new file mode 100644 index 00000000..ad0e53d0 --- /dev/null +++ b/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/MonitoredItemShardPlanner.cs @@ -0,0 +1,63 @@ +namespace ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Adapters; + +/// +/// Pure placement policy for monitored items across OPC UA subscription shards: first +/// shard with free capacity, otherwise a new shard. Factored out of +/// so the budget arithmetic is testable without a live +/// server (the SDK's Session/Subscription types cannot be faked). +/// +internal static class MonitoredItemShardPlanner +{ + /// + /// Plans where new items go given the current per-shard + /// item counts. Returns one shard index per item, in order; an index equal to or above + /// .Count means "a shard that must be created first". + /// + /// Item count of each existing shard, in shard order. + /// Maximum items per shard; values <= 0 fall back to 1. + /// Number of items to place. + /// Shard index per item, in request order. + public static IReadOnlyList Plan(IReadOnlyList currentCounts, int budget, int itemCount) + { + var effectiveBudget = budget > 0 ? budget : 1; + var counts = currentCounts.ToList(); + var placement = new List(itemCount); + + for (var i = 0; i < itemCount; i++) + { + var target = -1; + for (var shard = 0; shard < counts.Count; shard++) + { + if (counts[shard] < effectiveBudget) + { + target = shard; + break; + } + } + + if (target < 0) + { + counts.Add(0); + target = counts.Count - 1; + } + + counts[target]++; + placement.Add(target); + } + + return placement; + } + + /// + /// Number of shards needed to hold items at + /// items per shard, starting from nothing. + /// + /// Total monitored items. + /// Maximum items per shard; values <= 0 fall back to 1. + /// The shard count. + public static int ShardCountFor(int itemCount, int budget) + { + var effectiveBudget = budget > 0 ? budget : 1; + return (itemCount + effectiveBudget - 1) / effectiveBudget; + } +} diff --git a/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/MxGatewayDataConnection.cs b/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/MxGatewayDataConnection.cs index 3b29c7fe..20c22f61 100644 --- a/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/MxGatewayDataConnection.cs +++ b/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/MxGatewayDataConnection.cs @@ -21,11 +21,13 @@ namespace ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Adapters; /// , the actor disposes this adapter, creates a fresh one, /// reconnects and re-subscribes all tags. /// -public class MxGatewayDataConnection : IDataConnection, IBrowsableDataConnection, IAlarmSubscribableConnection +public class MxGatewayDataConnection + : IDataConnection, IBrowsableDataConnection, IAlarmSubscribableConnection, IBatchSubscribableConnection { private readonly IMxGatewayClientFactory _clientFactory; private readonly ILogger _logger; private readonly ISecretResolver? _secretResolver; + private readonly int _supervisoryAdviseParallelism; private IMxGatewayClient? _client; private ConnectionHealth _status = ConnectionHealth.Disconnected; private CancellationTokenSource? _eventLoopCts; @@ -39,6 +41,13 @@ public class MxGatewayDataConnection : IDataConnection, IBrowsableDataConnection private int _alarmSubCount; private readonly object _alarmLock = new(); + // Source references currently subscribed (alarm subscription id → source reference), + // and the prefix the live stream was opened with. StreamAlarms carries ONE prefix, so + // the pushable union is the longest common prefix of the active sources: the stream is + // restarted only when a NEW source falls outside it. Guarded by _alarmLock. + private readonly Dictionary _alarmSubSources = new(StringComparer.Ordinal); + private string _alarmStreamPrefix = string.Empty; + // subscriptionId → (tagPath, callback) so the event loop can route updates by tag, // plus tagPath → subscriptionId for reverse lookup. Concurrent because the event // loop reads from a background thread while Subscribe/Unsubscribe mutate. @@ -58,14 +67,20 @@ public class MxGatewayDataConnection : IDataConnection, IBrowsableDataConnection /// connect time. When null, a literal ApiKey still works, but a secret: reference /// fails closed (a connection is never established with an unresolved key). /// + /// + /// Maximum in-flight supervisory advise commands on the bulk-subscribe path when the + /// endpoint has no write-user context (DataConnectionOptions.MxSupervisoryAdviseParallelism). + /// public MxGatewayDataConnection( IMxGatewayClientFactory clientFactory, ILogger logger, - ISecretResolver? secretResolver = null) + ISecretResolver? secretResolver = null, + int supervisoryAdviseParallelism = 16) { _clientFactory = clientFactory; _logger = logger; _secretResolver = secretResolver; + _supervisoryAdviseParallelism = Math.Max(1, supervisoryAdviseParallelism); } /// @@ -129,7 +144,8 @@ public class MxGatewayDataConnection : IDataConnection, IBrowsableDataConnection cfg.UseTls, string.IsNullOrWhiteSpace(cfg.CaFile) ? null : cfg.CaFile, string.IsNullOrWhiteSpace(cfg.ServerName) ? null : cfg.ServerName, - cfg.ReadTimeoutMs), cancellationToken); + cfg.ReadTimeoutMs, + _supervisoryAdviseParallelism), cancellationToken); _status = ConnectionHealth.Connected; @@ -192,6 +208,8 @@ public class MxGatewayDataConnection : IDataConnection, IBrowsableDataConnection _alarmCts?.Dispose(); _alarmCts = null; _alarmSubCount = 0; + _alarmSubSources.Clear(); + _alarmStreamPrefix = string.Empty; } if (_client is not null) await _client.DisconnectAsync(cancellationToken); @@ -215,28 +233,79 @@ public class MxGatewayDataConnection : IDataConnection, IBrowsableDataConnection await _client!.UnsubscribeAsync(subscriptionId, cancellationToken); } + /// + public async Task> SubscribeBatchAsync( + IReadOnlyList tagPaths, SubscriptionCallback callback, CancellationToken cancellationToken = default) + { + // ONE gateway round trip per batch (SubscribeBulk = AddItem + Advise per tag in a + // single worker pass) instead of the historical 2 RPCs per tag. + var outcomes = await _client!.SubscribeBulkAsync(tagPaths, cancellationToken); + var results = new List(outcomes.Count); + foreach (var outcome in outcomes) + { + if (outcome is { Success: true, SubscriptionId: not null }) + { + _subs[outcome.SubscriptionId] = (outcome.TagPath, callback); + _tagToSub[outcome.TagPath] = outcome.SubscriptionId; + } + + results.Add(new TagSubscribeResult( + outcome.TagPath, outcome.Success, outcome.SubscriptionId, outcome.Error)); + } + + return results; + } + + /// + public async Task UnsubscribeBatchAsync( + IReadOnlyList subscriptionIds, CancellationToken cancellationToken = default) + { + foreach (var id in subscriptionIds) + { + if (_subs.TryRemove(id, out var s)) + _tagToSub.TryRemove(s.TagPath, out _); + } + + await _client!.UnsubscribeBulkAsync(subscriptionIds, cancellationToken); + } + /// public Task SubscribeAlarmsAsync( string sourceReference, string? conditionFilter, AlarmTransitionCallback callback, CancellationToken cancellationToken = default) { + var subscriptionId = Guid.NewGuid().ToString(); lock (_alarmLock) { _alarmSubCount++; - if (_alarmCts == null) + _alarmSubSources[subscriptionId] = sourceReference; + + // Push the UNION of the subscribed source references to the gateway as the + // stream's single filter prefix. It is a bandwidth optimisation only — the + // MxGateway has no server-side CONDITION filter either, so conditionFilter is + // still not forwarded: the DataConnectionActor's per-source + per-condition-type + // gate stays authoritative, uniform with the OPC UA WhereClause stance. + if (_alarmCts != null && AlarmFilterPrefix.Covers(_alarmStreamPrefix, sourceReference)) + return Task.FromResult(subscriptionId); + + // A new source outside the live prefix (or the first subscriber): (re)open the + // stream on the recomputed union. The source replays a fresh snapshot, which + // NativeAlarmActor already handles. + if (_alarmCts != null) { - _alarmCts = new CancellationTokenSource(); - var token = _alarmCts.Token; - var client = _client!; - // Gateway-wide feed (null prefix). The MxGateway has no server-side - // condition filter, so conditionFilter is intentionally NOT forwarded - // here: the DataConnectionActor applies it as the authoritative - // client-side gate per source reference AND per condition type - // (AlarmConditionFilter), uniform with the OPC UA path. - _ = Task.Run(() => client.RunAlarmStreamAsync(null, t => callback(t), token), token); + _alarmCts.Cancel(); + _alarmCts.Dispose(); } + + _alarmStreamPrefix = AlarmFilterPrefix.LongestCommonPrefix(_alarmSubSources.Values); + _alarmCts = new CancellationTokenSource(); + var token = _alarmCts.Token; + var client = _client!; + var prefix = _alarmStreamPrefix.Length == 0 ? null : _alarmStreamPrefix; + _ = Task.Run(() => client.RunAlarmStreamAsync(prefix, t => callback(t), token), token); } - return Task.FromResult(Guid.NewGuid().ToString()); + + return Task.FromResult(subscriptionId); } /// @@ -244,13 +313,17 @@ public class MxGatewayDataConnection : IDataConnection, IBrowsableDataConnection { lock (_alarmLock) { + _alarmSubSources.Remove(subscriptionId); if (_alarmSubCount > 0) _alarmSubCount--; + // Deliberately no stream restart here: dropping a source can only leave the + // prefix too BROAD, which costs bandwidth, never correctness. if (_alarmSubCount == 0) { _alarmCts?.Cancel(); _alarmCts?.Dispose(); _alarmCts = null; + _alarmStreamPrefix = string.Empty; } } return Task.CompletedTask; @@ -354,6 +427,8 @@ public class MxGatewayDataConnection : IDataConnection, IBrowsableDataConnection _alarmCts?.Dispose(); _alarmCts = null; _alarmSubCount = 0; + _alarmSubSources.Clear(); + _alarmStreamPrefix = string.Empty; } if (_client is not null) await _client.DisposeAsync(); diff --git a/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/OpcUaDataConnection.cs b/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/OpcUaDataConnection.cs index 7176492c..4cabfc02 100644 --- a/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/OpcUaDataConnection.cs +++ b/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/OpcUaDataConnection.cs @@ -18,7 +18,9 @@ namespace ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Adapters; /// - Read/Write → Read/Write service calls /// - Quality → OPC UA StatusCode mapping /// -public class OpcUaDataConnection : IDataConnection, IBrowsableDataConnection, IAlarmSubscribableConnection, IAddressSpaceSearchable +public class OpcUaDataConnection + : IDataConnection, IBrowsableDataConnection, IAlarmSubscribableConnection, IAddressSpaceSearchable, + IBatchSubscribableConnection { private readonly IOpcUaClientFactory _clientFactory; private readonly ILogger _logger; @@ -93,7 +95,8 @@ public class OpcUaDataConnection : IDataConnection, IBrowsableDataConnection, IA ? new OpcUaUserIdentityOptions( ui.TokenType.ToString(), ui.Username, ui.Password, ui.CertificatePath, ui.CertificatePassword) - : null); + : null, + MaxMonitoredItemsPerSubscription: config.MaxMonitoredItemsPerSubscription); _status = ConnectionHealth.Connecting; @@ -193,6 +196,36 @@ public class OpcUaDataConnection : IDataConnection, IBrowsableDataConnection, IA cancellationToken); } + /// + public async Task> SubscribeBatchAsync( + IReadOnlyList tagPaths, SubscriptionCallback callback, CancellationToken cancellationToken = default) + { + EnsureConnected(); + + // ONE shared callback for the whole batch — the client hands back the node id + // (== tag path for OPC UA), so no per-tag closure is needed. + var outcomes = await _client!.CreateSubscriptionsAsync( + tagPaths, + (nodeId, value, timestamp, statusCode) => + { + var quality = MapStatusCode(statusCode); + callback(nodeId, new TagValue(value, quality, new DateTimeOffset(timestamp, TimeSpan.Zero))); + }, + cancellationToken); + + return outcomes + .Select(o => new TagSubscribeResult(o.NodeId, o.Success, o.SubscriptionHandle, o.Error)) + .ToList(); + } + + /// + public async Task UnsubscribeBatchAsync( + IReadOnlyList subscriptionIds, CancellationToken cancellationToken = default) + { + if (_client != null) + await _client.RemoveSubscriptionsAsync(subscriptionIds, cancellationToken); + } + /// public async Task SubscribeAlarmsAsync( string sourceReference, string? conditionFilter, @@ -249,27 +282,59 @@ public class OpcUaDataConnection : IDataConnection, IBrowsableDataConnection, IA /// public async Task> ReadBatchAsync(IEnumerable tagPaths, CancellationToken cancellationToken = default) { - // A single failing tag must not abort the whole batch. - // ReadAsync re-throws non-cancellation exceptions; catch them per tag and record - // a failed ReadResult so the caller receives a complete result map for every - // requested tag (the ReadResult shape already carries per-tag Success/error). - var results = new Dictionary(); - foreach (var tagPath in tagPaths) + EnsureConnected(); + + // TRUE bulk: one OPC UA Read service call (chunked inside the client against the + // server's operation limits), not a loop over ReadAsync. A single failing tag + // still comes back as a failed ReadResult row rather than aborting the batch; + // OperationCanceledException still aborts the whole batch. + var requested = tagPaths as IReadOnlyList ?? tagPaths.ToList(); + var results = new Dictionary(requested.Count); + if (requested.Count == 0) + return results; + + try { - try + var outcomes = await _client!.ReadValuesAsync(requested, cancellationToken); + foreach (var outcome in outcomes) { - results[tagPath] = await ReadAsync(tagPath, cancellationToken); - } - catch (OperationCanceledException) - { - // Cancellation aborts the whole batch — propagate it. - throw; - } - catch (Exception ex) - { - results[tagPath] = new ReadResult(false, null, ex.Message); + if (outcome.Error != null) + { + results[outcome.NodeId] = new ReadResult(false, null, outcome.Error); + continue; + } + + var quality = MapStatusCode(outcome.StatusCode); + results[outcome.NodeId] = quality == QualityCode.Bad + ? new ReadResult(false, null, $"OPC UA read returned bad status: 0x{outcome.StatusCode:X8}") + : new ReadResult(true, + new TagValue(outcome.Value, quality, new DateTimeOffset(outcome.SourceTimestamp, TimeSpan.Zero)), + null); } } + catch (OperationCanceledException) + { + // Cancellation aborts the whole batch — propagate it. + throw; + } + catch (Exception ex) + { + // A batch-level fault (session dropped mid-call) mirrors ReadAsync: signal the + // disconnect, then report every requested tag as failed so the caller still + // receives a complete map instead of an exception. + _logger.LogWarning(ex, "OPC UA batch read failed — connection may be lost"); + RaiseDisconnected(); + foreach (var tagPath in requested) + results[tagPath] = new ReadResult(false, null, ex.Message); + } + + // Defensive: a non-conformant client/server that omits a requested tag. + foreach (var tagPath in requested) + { + if (!results.ContainsKey(tagPath)) + results[tagPath] = new ReadResult(false, null, "Tag missing from OPC UA read result."); + } + return results; } @@ -288,29 +353,46 @@ public class OpcUaDataConnection : IDataConnection, IBrowsableDataConnection, IA /// public async Task> WriteBatchAsync(IDictionary values, CancellationToken cancellationToken = default) { - // A mid-batch fault must not abort the whole batch. - // WriteAsync calls EnsureConnected(), which throws InvalidOperationException when - // the connection drops partway through; catch per-tag exceptions and record a - // failed WriteResult so the caller (including WriteBatchAndWaitAsync) receives a - // complete result map. OperationCanceledException is still propagated so a - // cancelled batch aborts as a whole — mirrors the ReadBatchAsync fix. - var results = new Dictionary(); - foreach (var (tagPath, value) in values) + EnsureConnected(); + + // TRUE bulk: one OPC UA Write service call (chunked inside the client). A mid-batch + // fault must not abort the batch — every requested tag gets a WriteResult row — + // while OperationCanceledException still aborts as a whole. + var requested = values.ToList(); + var results = new Dictionary(requested.Count); + if (requested.Count == 0) + return results; + + try { - try + var outcomes = await _client!.WriteValuesAsync( + requested.Select(kv => (kv.Key, kv.Value)).ToList(), cancellationToken); + foreach (var outcome in outcomes) { - results[tagPath] = await WriteAsync(tagPath, value, cancellationToken); - } - catch (OperationCanceledException) - { - // Cancellation aborts the whole batch — propagate it. - throw; - } - catch (Exception ex) - { - results[tagPath] = new WriteResult(false, ex.Message); + results[outcome.NodeId] = outcome.Error != null + ? new WriteResult(false, outcome.Error) + : outcome.StatusCode != 0 + ? new WriteResult(false, $"OPC UA write failed with status: 0x{outcome.StatusCode:X8}") + : new WriteResult(true, null); } } + catch (OperationCanceledException) + { + // Cancellation aborts the whole batch — propagate it. + throw; + } + catch (Exception ex) + { + foreach (var (tagPath, _) in requested) + results[tagPath] = new WriteResult(false, ex.Message); + } + + foreach (var (tagPath, _) in requested) + { + if (!results.ContainsKey(tagPath)) + results[tagPath] = new WriteResult(false, "Tag missing from OPC UA write result."); + } + return results; } diff --git a/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/RealMxGatewayClient.cs b/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/RealMxGatewayClient.cs index 5a5d2c4e..5684fd22 100644 --- a/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/RealMxGatewayClient.cs +++ b/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/RealMxGatewayClient.cs @@ -26,6 +26,7 @@ public sealed class RealMxGatewayClient : IMxGatewayClient private int _serverHandle; private int _writeUserId; private int _readTimeoutMs; + private int _supervisoryAdviseParallelism = 16; private ulong _lastSeq; // tag ↔ MXAccess item handle, maintained across subscribe/write. @@ -65,6 +66,7 @@ public sealed class RealMxGatewayClient : IMxGatewayClient { _writeUserId = options.WriteUserId; _readTimeoutMs = options.ReadTimeoutMs; + _supervisoryAdviseParallelism = Math.Max(1, options.SupervisoryAdviseParallelism); var clientOptions = new MxGatewayClientOptions { @@ -102,6 +104,122 @@ public sealed class RealMxGatewayClient : IMxGatewayClient return handle.ToString(CultureInfo.InvariantCulture); } + /// + public async Task> SubscribeBulkAsync( + IReadOnlyList tagPaths, CancellationToken ct = default) + { + if (tagPaths.Count == 0) + return []; + + // Plain-advise mode (a configured write user): the worker's SubscribeBulk does + // AddItem + Advise for every tag in ONE gateway command, collapsing the historical + // 2 RPCs per tag to 1 RPC per chunk. + if (!UseSupervisoryAdvise) + { + var replies = await _session!.SubscribeBulkAsync(_serverHandle, tagPaths, ct).ConfigureAwait(false); + return MapSubscribeResults(tagPaths, replies); + } + + // Supervisory mode (WriteUserId == 0): the worker has no BULK supervisory advise + // (SubscribeBulk issues plain Advise), so bulk-add the items in one command and + // pipeline the per-item AdviseSupervisory commands with a bounded window instead + // of issuing them serially. Cross-repo follow-up: an additive `supervisory` flag on + // SubscribeBulkCommand in mxaccessgw would collapse this to one RPC as well. + var added = await _session!.AddItemBulkAsync(_serverHandle, tagPaths, ct).ConfigureAwait(false); + var outcomes = MapSubscribeResults(tagPaths, added); + + var advisable = outcomes + .Select((o, i) => (Outcome: o, Index: i)) + .Where(x => x.Outcome.Success) + .ToList(); + + var advised = await BulkPipeline.RunAsync( + advisable, + _supervisoryAdviseParallelism, + async (x, token) => + { + var handle = int.Parse(x.Outcome.SubscriptionId!, NumberStyles.Integer, CultureInfo.InvariantCulture); + await EnsureSupervisoryAdvisedAsync(handle, token).ConfigureAwait(false); + return x.Outcome; + }, + (x, ex) => x.Outcome with { Success = false, SubscriptionId = null, Error = ex.Message }, + ct).ConfigureAwait(false); + + var result = outcomes.ToArray(); + for (var i = 0; i < advisable.Count; i++) + result[advisable[i].Index] = advised[i]; + + // A failed advise leaves an added-but-unadvised item; drop our handle mapping so a + // later retry re-adds cleanly rather than reusing a half-subscribed handle. + foreach (var failed in result.Where(r => !r.Success)) + { + if (_tagToHandle.TryRemove(failed.TagPath, out var staleHandle)) + _handleToTag.TryRemove(staleHandle, out _); + } + + return result; + } + + /// + /// Maps the gateway's per-item bulk results back onto the requested tag paths. Results + /// are returned in request order; the tag address on the reply is preferred when the + /// gateway echoes it. Successful rows populate the tag ↔ item-handle maps that the + /// event loop and the write path key off. + /// + private MxSubscribeOutcome[] MapSubscribeResults( + IReadOnlyList tagPaths, IReadOnlyList replies) + { + var outcomes = new MxSubscribeOutcome[tagPaths.Count]; + for (var i = 0; i < tagPaths.Count; i++) + { + var tag = tagPaths[i]; + if (i >= replies.Count) + { + outcomes[i] = new MxSubscribeOutcome(tag, false, null, "Gateway returned no result for this tag."); + continue; + } + + var reply = replies[i]; + if (!reply.WasSuccessful) + { + outcomes[i] = new MxSubscribeOutcome(tag, false, null, + string.IsNullOrEmpty(reply.ErrorMessage) ? "Bulk subscribe failed." : reply.ErrorMessage); + continue; + } + + _tagToHandle[tag] = reply.ItemHandle; + _handleToTag[reply.ItemHandle] = tag; + outcomes[i] = new MxSubscribeOutcome( + tag, true, reply.ItemHandle.ToString(CultureInfo.InvariantCulture), null); + } + + return outcomes; + } + + /// + public async Task UnsubscribeBulkAsync(IReadOnlyList subscriptionIds, CancellationToken ct = default) + { + var handles = new List(subscriptionIds.Count); + foreach (var id in subscriptionIds) + { + if (int.TryParse(id, NumberStyles.Integer, CultureInfo.InvariantCulture, out var handle)) + handles.Add(handle); + } + + if (handles.Count == 0) + return; + + await _session!.UnsubscribeBulkAsync(_serverHandle, handles, ct).ConfigureAwait(false); + + foreach (var handle in handles) + { + // UnsubscribeBulk unadvises AND removes the item, releasing every advice kind. + _supervisoryAdvised.TryRemove(handle, out _); + if (_handleToTag.TryRemove(handle, out var tag)) + _tagToHandle.TryRemove(tag, out _); + } + } + /// public async Task UnsubscribeAsync(string subscriptionId, CancellationToken ct = default) { diff --git a/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/RealOpcUaClient.cs b/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/RealOpcUaClient.cs index 0fdf8db6..26dd75bf 100644 --- a/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/RealOpcUaClient.cs +++ b/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/RealOpcUaClient.cs @@ -19,7 +19,42 @@ namespace ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Adapters; public class RealOpcUaClient : IOpcUaClient { private ISession? _session; - private Subscription? _subscription; + + /// + /// One data shard = one OPC UA Subscription holding at most + /// monitored + /// items. The count is tracked here rather than read back off the SDK object so + /// placement decisions are made from state this class owns (and mutates only under + /// ). + /// + private sealed class DataShard(Subscription subscription) + { + public Subscription Subscription { get; } = subscription; + public int ItemCount { get; set; } + } + + // Data-item shards, in creation order. A monitored item is placed on the first shard + // with free capacity; when all are full a new shard is created, and a shard is + // deleted once it empties. Mutated ONLY under _applyGate. + private readonly List _dataShards = new(); + + // Alarms & Conditions event items live on their own shard: TriggerConditionRefresh + // needs a single subscription id to target, and QueueSize:1000 event items would + // otherwise consume data-shard capacity. Created lazily, torn down on disconnect. + private Subscription? _eventShard; + + // subscription handle → owning shard's Subscription, so a removal can group ONE + // ApplyChanges per touched shard. + private readonly ConcurrentDictionary _itemShard = new(); + + // The OPC Foundation Subscription object is not safe under concurrent structural + // mutation, and nothing else orders HandleSubscribe's background task, the + // tag-resolution probe and an alarm subscribe against each other. Every + // AddItem/RemoveItem/ApplyChanges/Create/Delete — and every shard-list mutation — + // serializes behind this ONE per-client gate. Batching makes contention rare; a + // single gate (rather than per-shard) also serializes shard creation and is the + // simplest thing that is correct. + private readonly AsyncSerialGate _applyGate = new(); // These maps are read from the OPC Foundation SDK's // internal publish threads (the MonitoredItem.Notification handler reads @@ -146,20 +181,68 @@ public class RealOpcUaClient : IOpcUaClient // Store options for monitored item creation _options = opts; - // Create a default subscription for all monitored items - _subscription = new Subscription(_session.DefaultSubscription) + // Create the first data shard up front so a session is immediately usable; + // further shards are added on demand as the item budget fills. + await _applyGate.RunAsync(async () => { - DisplayName = opts.SubscriptionDisplayName, - Priority = opts.SubscriptionPriority, + _dataShards.Clear(); + _itemShard.Clear(); + _eventShard = null; + await CreateDataShardAsync(cancellationToken); + }, cancellationToken); + } + + /// Item budget per subscription, guarding against a non-positive configured value. + private int ItemsPerShard => _options.MaxMonitoredItemsPerSubscription > 0 + ? _options.MaxMonitoredItemsPerSubscription + : 5000; + + /// + /// Builds a Subscription from the current connection options. Callers must hold + /// . + /// + private Subscription NewSubscription(string displayName) => + new(_session!.DefaultSubscription) + { + DisplayName = displayName, + Priority = _options.SubscriptionPriority, PublishingEnabled = true, - PublishingInterval = opts.PublishingIntervalMs, - KeepAliveCount = (uint)opts.KeepAliveCount, - LifetimeCount = (uint)opts.LifetimeCount, - MaxNotificationsPerPublish = (uint)opts.MaxNotificationsPerPublish + PublishingInterval = _options.PublishingIntervalMs, + KeepAliveCount = (uint)_options.KeepAliveCount, + LifetimeCount = (uint)_options.LifetimeCount, + MaxNotificationsPerPublish = (uint)_options.MaxNotificationsPerPublish }; - _session.AddSubscription(_subscription); - await _subscription.CreateAsync(cancellationToken); + /// + /// Adds and creates a new data shard. Callers must hold . + /// + private async Task CreateDataShardAsync(CancellationToken cancellationToken) + { + var suffix = _dataShards.Count == 0 ? string.Empty : $"-{_dataShards.Count + 1}"; + var subscription = NewSubscription(_options.SubscriptionDisplayName + suffix); + _session!.AddSubscription(subscription); + await subscription.CreateAsync(cancellationToken); + var shard = new DataShard(subscription); + _dataShards.Add(shard); + _logger.LogDebug("OPC UA data shard {Shard} created (budget {Budget} items)", + subscription.DisplayName, ItemsPerShard); + return shard; + } + + /// + /// Returns the dedicated Alarms & Conditions event shard, creating it on first + /// use. Callers must hold . + /// + private async Task GetOrCreateEventShardAsync(CancellationToken cancellationToken) + { + if (_eventShard != null) + return _eventShard; + + var subscription = NewSubscription(_options.SubscriptionDisplayName + "-events"); + _session!.AddSubscription(subscription); + await subscription.CreateAsync(cancellationToken); + _eventShard = subscription; + return subscription; } /// @@ -439,11 +522,19 @@ public class RealOpcUaClient : IOpcUaClient /// public async Task DisconnectAsync(CancellationToken cancellationToken = default) { - if (_subscription != null) + await _applyGate.RunAsync(async () => { - await _subscription.DeleteAsync(true); - _subscription = null; - } + foreach (var shard in _dataShards) + await shard.Subscription.DeleteAsync(true); + _dataShards.Clear(); + _itemShard.Clear(); + if (_eventShard != null) + { + await _eventShard.DeleteAsync(true); + _eventShard = null; + } + }, cancellationToken); + if (_session != null) { _session.KeepAlive -= OnSessionKeepAlive; @@ -459,55 +550,230 @@ public class RealOpcUaClient : IOpcUaClient string nodeId, Action onValueChanged, CancellationToken cancellationToken = default) { - if (_subscription == null || _session == null) - throw new InvalidOperationException("Not connected."); + // Batch-of-one delegation: the single-node entry point is kept (heartbeat monitor, + // interactive paths) but shares the batch implementation so there is exactly one + // placement / ApplyChanges code path. A per-item failure throws here — preserving + // the caller's historical "create throws on failure" contract AND the exception + // TYPE, so DataConnectionActor.IsConnectionLevelFailure still classifies a + // resolution failure as a tag-resolution failure rather than a connection fault. + var (outcomes, errors) = await CreateSubscriptionsCoreAsync( + [nodeId], onValueChanged, cancellationToken); - var handle = Guid.NewGuid().ToString(); - var monitoredItem = new MonitoredItem(_subscription.DefaultItem) - { - DisplayName = nodeId, - StartNodeId = OpcUaNodeReference.Resolve(nodeId, _session.NamespaceUris), - AttributeId = Attributes.Value, - SamplingInterval = _options.SamplingIntervalMs, - QueueSize = (uint)_options.QueueSize, - DiscardOldest = _options.DiscardOldest, - Filter = BuildDataChangeFilter(_options.Deadband) - }; + var outcome = outcomes[0]; + if (outcome.Success) + return outcome.SubscriptionHandle!; - _callbacks[handle] = onValueChanged; - - monitoredItem.Notification += (item, e) => - { - if (e.NotificationValue is MonitoredItemNotification notification) - { - var value = notification.Value?.Value; - var timestamp = notification.Value?.SourceTimestamp ?? DateTime.UtcNow; - var statusCode = notification.Value?.StatusCode.Code ?? 0; - - if (_callbacks.TryGetValue(handle, out var cb)) - { - cb(nodeId, value, timestamp, statusCode); - } - } - }; - - _subscription.AddItem(monitoredItem); - await _subscription.ApplyChangesAsync(cancellationToken); - - _monitoredItems[handle] = monitoredItem; - return handle; + if (errors[0] is { } captured) + System.Runtime.ExceptionServices.ExceptionDispatchInfo.Capture(captured).Throw(); + throw new ServiceResultException(StatusCodes.BadNodeIdUnknown, + outcome.Error ?? $"Monitored item for '{nodeId}' could not be created."); } /// - public async Task RemoveSubscriptionAsync(string subscriptionHandle, CancellationToken cancellationToken = default) + public async Task> CreateSubscriptionsAsync( + IReadOnlyList nodeIds, + Action onValueChanged, + CancellationToken cancellationToken = default) { - if (_subscription != null && _monitoredItems.TryGetValue(subscriptionHandle, out var item)) + var (outcomes, _) = await CreateSubscriptionsCoreAsync(nodeIds, onValueChanged, cancellationToken); + return outcomes; + } + + /// + /// Creates monitored items for every requested node with ONE ApplyChanges per touched + /// shard — the core of the batch subscribe seam. Node-id resolution runs outside the + /// apply lock (it is pure over the session's namespace table); placement, AddItem and + /// ApplyChanges run under it. Returns per-node outcomes plus the captured exception + /// (when any) per node, so the single-node entry point can rethrow the original. + /// + private async Task<(OpcUaSubscribeOutcome[] Outcomes, Exception?[] Errors)> CreateSubscriptionsCoreAsync( + IReadOnlyList nodeIds, + Action onValueChanged, + CancellationToken cancellationToken) + { + var session = _session; + if (session == null) + throw new InvalidOperationException("Not connected."); + + var outcomes = new OpcUaSubscribeOutcome[nodeIds.Count]; + var errors = new Exception?[nodeIds.Count]; + + // Resolve first: a bad node id is a per-tag fault and must not abort the batch. + var resolved = new List<(int Index, string NodeId, NodeId Resolved)>(nodeIds.Count); + for (var i = 0; i < nodeIds.Count; i++) { - _subscription.RemoveItem(item); - await _subscription.ApplyChangesAsync(cancellationToken); - _monitoredItems.TryRemove(subscriptionHandle, out _); - _callbacks.TryRemove(subscriptionHandle, out _); + try + { + resolved.Add((i, nodeIds[i], OpcUaNodeReference.Resolve(nodeIds[i], session.NamespaceUris))); + } + catch (Exception ex) + { + outcomes[i] = new OpcUaSubscribeOutcome(nodeIds[i], false, null, ex.Message); + errors[i] = ex; + } } + + if (resolved.Count == 0) + return (outcomes, errors); + + return await _applyGate.RunAsync(async () => + { + var created = new List<(int Index, string NodeId, string Handle, MonitoredItem Item)>(resolved.Count); + var touched = new List(); + + foreach (var (index, nodeId, startNode) in resolved) + { + var shard = await PlaceOnDataShardAsync(cancellationToken); + var handle = Guid.NewGuid().ToString(); + var monitoredItem = new MonitoredItem(shard.Subscription.DefaultItem) + { + DisplayName = nodeId, + StartNodeId = startNode, + AttributeId = Attributes.Value, + SamplingInterval = _options.SamplingIntervalMs, + QueueSize = (uint)_options.QueueSize, + DiscardOldest = _options.DiscardOldest, + Filter = BuildDataChangeFilter(_options.Deadband) + }; + + _callbacks[handle] = onValueChanged; + monitoredItem.Notification += (_, e) => + { + if (e.NotificationValue is MonitoredItemNotification notification) + { + var value = notification.Value?.Value; + var timestamp = notification.Value?.SourceTimestamp ?? DateTime.UtcNow; + var statusCode = notification.Value?.StatusCode.Code ?? 0; + + if (_callbacks.TryGetValue(handle, out var cb)) + { + cb(nodeId, value, timestamp, statusCode); + } + } + }; + + shard.Subscription.AddItem(monitoredItem); + shard.ItemCount++; + _itemShard[handle] = shard.Subscription; + created.Add((index, nodeId, handle, monitoredItem)); + if (!touched.Contains(shard)) + touched.Add(shard); + } + + // ONE ApplyChanges per touched shard — the whole point of the batch seam. + foreach (var shard in touched) + await shard.Subscription.ApplyChangesAsync(cancellationToken); + + foreach (var (index, nodeId, handle, item) in created) + { + var error = item.Status?.Error; + if (error != null && ServiceResult.IsBad(error)) + { + // Per-item create failure (unknown node, access denied): release the + // half-created item so the shard budget stays honest, and report a + // failed row. The actor treats it as an unresolved tag and retries it + // on the backoff timer. + ReleaseFailedItem(handle, item); + outcomes[index] = new OpcUaSubscribeOutcome(nodeId, false, null, error.ToString()); + } + else + { + _monitoredItems[handle] = item; + outcomes[index] = new OpcUaSubscribeOutcome(nodeId, true, handle, null); + } + } + + return (outcomes, errors); + }, cancellationToken); + } + + /// + /// Returns the shard this item belongs on per + /// — first shard with free capacity, else a + /// freshly created one. Callers must hold . + /// + private async Task PlaceOnDataShardAsync(CancellationToken cancellationToken) + { + var index = MonitoredItemShardPlanner.Plan( + _dataShards.Select(s => s.ItemCount).ToList(), ItemsPerShard, 1)[0]; + return index < _dataShards.Count + ? _dataShards[index] + : await CreateDataShardAsync(cancellationToken); + } + + /// + /// Drops a monitored item that failed to create from the shard it was placed on. + /// Callers must hold . No ApplyChanges is issued: the server + /// never created the item, so removing it locally keeps the budget honest. + /// + private void ReleaseFailedItem(string handle, MonitoredItem item) + { + _callbacks.TryRemove(handle, out _); + if (!_itemShard.TryRemove(handle, out var subscription)) + return; + + subscription.RemoveItem(item); + var shard = _dataShards.FirstOrDefault(s => ReferenceEquals(s.Subscription, subscription)); + if (shard is { ItemCount: > 0 }) + shard.ItemCount--; + } + + /// + public Task RemoveSubscriptionAsync(string subscriptionHandle, CancellationToken cancellationToken = default) + => RemoveSubscriptionsAsync([subscriptionHandle], cancellationToken); + + /// + public async Task RemoveSubscriptionsAsync( + IReadOnlyList subscriptionHandles, CancellationToken cancellationToken = default) + { + if (subscriptionHandles.Count == 0) + return; + + await _applyGate.RunAsync(async () => + { + var touched = new List(); + foreach (var handle in subscriptionHandles) + { + if (!_monitoredItems.TryGetValue(handle, out var item)) + continue; + if (!_itemShard.TryGetValue(handle, out var subscription)) + continue; + + subscription.RemoveItem(item); + _monitoredItems.TryRemove(handle, out _); + _callbacks.TryRemove(handle, out _); + _itemShard.TryRemove(handle, out _); + + var shard = _dataShards.FirstOrDefault(s => ReferenceEquals(s.Subscription, subscription)); + if (shard == null) + continue; + if (shard.ItemCount > 0) + shard.ItemCount--; + if (!touched.Contains(shard)) + touched.Add(shard); + } + + // ONE ApplyChanges per touched shard, then drop the shards that emptied. + foreach (var shard in touched) + { + await shard.Subscription.ApplyChangesAsync(cancellationToken); + if (shard.ItemCount != 0) + continue; + + _dataShards.Remove(shard); + try + { + await shard.Subscription.DeleteAsync(true); + } + catch (Exception ex) + { + // Best-effort: an empty shard the server already dropped must not fail + // the unsubscribe path. + _logger.LogDebug(ex, "Deleting emptied OPC UA data shard {Shard} failed (ignored).", + shard.Subscription.DisplayName); + } + } + }, cancellationToken); } // ── Native alarm (Alarms & Conditions) subscription ── @@ -523,7 +789,7 @@ public class RealOpcUaClient : IOpcUaClient string? sourceNodeId, string? conditionFilter, Action onTransition, CancellationToken cancellationToken = default) { - if (_subscription == null || _session == null) + if (_session == null) throw new InvalidOperationException("Not connected."); var handle = Guid.NewGuid().ToString(); @@ -533,7 +799,14 @@ public class RealOpcUaClient : IOpcUaClient var startNode = string.IsNullOrEmpty(sourceNodeId) ? ObjectIds.Server : OpcUaNodeReference.Resolve(sourceNodeId, _session.NamespaceUris); - var item = new MonitoredItem(_subscription.DefaultItem) + + // A&C event items live on their own shard (see _eventShard): ConditionRefresh + // needs one subscription id to target, and QueueSize:1000 event items must not + // occupy data-shard capacity. + var eventShard = await _applyGate.RunAsync( + () => GetOrCreateEventShardAsync(cancellationToken), cancellationToken); + + var item = new MonitoredItem(eventShard.DefaultItem) { DisplayName = $"alarm:{sourceNodeId ?? "Server"}", StartNodeId = startNode, @@ -554,22 +827,31 @@ public class RealOpcUaClient : IOpcUaClient HandleAlarmEvent(handle, sourceNodeId, efl, onTransition); }; - _subscription.AddItem(item); - await _subscription.ApplyChangesAsync(cancellationToken); + await _applyGate.RunAsync(async () => + { + eventShard.AddItem(item); + await eventShard.ApplyChangesAsync(cancellationToken); + }, cancellationToken); _alarmItems[handle] = item; // Replay currently-active conditions as a Snapshot…SnapshotComplete sequence. - await TriggerConditionRefreshAsync(handle, cancellationToken); + await TriggerConditionRefreshAsync(handle, eventShard, cancellationToken); return handle; } /// public async Task RemoveAlarmSubscriptionAsync(string subscriptionHandle, CancellationToken cancellationToken = default) { - if (_subscription != null && _alarmItems.TryRemove(subscriptionHandle, out var item)) + if (_alarmItems.TryRemove(subscriptionHandle, out var item)) { - _subscription.RemoveItem(item); - await _subscription.ApplyChangesAsync(cancellationToken); + await _applyGate.RunAsync(async () => + { + if (_eventShard != null) + { + _eventShard.RemoveItem(item); + await _eventShard.ApplyChangesAsync(cancellationToken); + } + }, cancellationToken); } _alarmInRefresh.TryRemove(subscriptionHandle, out _); _alarmLastState.TryRemove(subscriptionHandle, out _); @@ -775,15 +1057,17 @@ public class RealOpcUaClient : IOpcUaClient }; } - private async Task TriggerConditionRefreshAsync(string handle, CancellationToken cancellationToken) + private async Task TriggerConditionRefreshAsync( + string handle, Subscription eventShard, CancellationToken cancellationToken) { try { // ConditionRefresh replays active conditions; RefreshStart/End events - // bracket the replay so HandleAlarmEvent can mark them Snapshot. + // bracket the replay so HandleAlarmEvent can mark them Snapshot. It targets + // the EVENT shard's subscription id — the shard every A&C item is pinned to. await _session!.CallAsync( ObjectTypeIds.ConditionType, MethodIds.ConditionType_ConditionRefresh, - cancellationToken, _subscription!.Id); + cancellationToken, eventShard.Id); } catch (Exception ex) { @@ -906,42 +1190,143 @@ public class RealOpcUaClient : IOpcUaClient new Commons.Types.Alarms.AlarmConditionState(false, true, null, AlarmShelveState.Unshelved, false, 0), "", "", "", "", "", null, DateTimeOffset.UtcNow, "", ""); + /// + /// Nodes per OPC UA Read/Write service call. Stays under the + /// MaxNodesPerRead / MaxNodesPerWrite operation limits typical servers + /// advertise, so a 37,500-tag re-seed never builds one oversized request. + /// + private const int MaxNodesPerServiceCall = 1000; + /// public async Task<(object? Value, DateTime SourceTimestamp, uint StatusCode)> ReadValueAsync( string nodeId, CancellationToken cancellationToken = default) { - if (_session == null) throw new InvalidOperationException("Not connected."); + // Batch-of-one delegation; a resolution failure still surfaces as the thrown + // exception the single-node callers expect. + var outcomes = await ReadValuesAsync([nodeId], cancellationToken); + var outcome = outcomes[0]; + if (outcome.Error != null) + throw new ServiceResultException(StatusCodes.BadNodeIdUnknown, outcome.Error); + return (outcome.Value, outcome.SourceTimestamp, outcome.StatusCode); + } - var readValue = new ReadValueId + /// + public async Task> ReadValuesAsync( + IReadOnlyList nodeIds, CancellationToken cancellationToken = default) + { + var session = _session; + if (session == null) throw new InvalidOperationException("Not connected."); + + var outcomes = new OpcUaReadOutcome[nodeIds.Count]; + + // Resolve first — an unresolvable node id is a per-node fault, not a batch abort. + var resolvable = new List<(int Index, string NodeId, ReadValueId Read)>(nodeIds.Count); + for (var i = 0; i < nodeIds.Count; i++) { - NodeId = OpcUaNodeReference.Resolve(nodeId, _session.NamespaceUris), - AttributeId = Attributes.Value - }; + try + { + resolvable.Add((i, nodeIds[i], new ReadValueId + { + NodeId = OpcUaNodeReference.Resolve(nodeIds[i], session.NamespaceUris), + AttributeId = Attributes.Value + })); + } + catch (Exception ex) + { + outcomes[i] = new OpcUaReadOutcome(nodeIds[i], null, DateTime.UtcNow, StatusCodes.BadNodeIdUnknown, ex.Message); + } + } - var response = await _session.ReadAsync( - null, 0, MapTimestampsToReturn(_options.TimestampsToReturn), - new ReadValueIdCollection { readValue }, cancellationToken); + for (var offset = 0; offset < resolvable.Count; offset += MaxNodesPerServiceCall) + { + var chunk = resolvable.GetRange(offset, Math.Min(MaxNodesPerServiceCall, resolvable.Count - offset)); + var collection = new ReadValueIdCollection(chunk.Select(c => c.Read)); + var response = await session.ReadAsync( + null, 0, MapTimestampsToReturn(_options.TimestampsToReturn), collection, cancellationToken); - var result = response.Results[0]; - return (result.Value, result.SourceTimestamp, result.StatusCode.Code); + for (var i = 0; i < chunk.Count; i++) + { + var (index, nodeId, _) = chunk[i]; + if (response.Results is { Count: > 0 } results && i < results.Count) + { + var result = results[i]; + outcomes[index] = new OpcUaReadOutcome( + nodeId, result.Value, result.SourceTimestamp, result.StatusCode.Code, null); + } + else + { + // Non-conformant server: fewer results than requested nodes. + outcomes[index] = new OpcUaReadOutcome( + nodeId, null, DateTime.UtcNow, StatusCodes.BadUnexpectedError, + "OPC UA read returned no result for this node."); + } + } + } + + return outcomes; } /// public async Task WriteValueAsync(string nodeId, object? value, CancellationToken cancellationToken = default) { - if (_session == null) throw new InvalidOperationException("Not connected."); + // Batch-of-one delegation, mirroring ReadValueAsync. + var outcomes = await WriteValuesAsync([(nodeId, value)], cancellationToken); + var outcome = outcomes[0]; + if (outcome.Error != null) + throw new ServiceResultException(StatusCodes.BadNodeIdUnknown, outcome.Error); + return outcome.StatusCode; + } - var writeValue = new WriteValue + /// + public async Task> WriteValuesAsync( + IReadOnlyList<(string NodeId, object? Value)> values, CancellationToken cancellationToken = default) + { + var session = _session; + if (session == null) throw new InvalidOperationException("Not connected."); + + var outcomes = new OpcUaWriteOutcome[values.Count]; + + var resolvable = new List<(int Index, string NodeId, WriteValue Write)>(values.Count); + for (var i = 0; i < values.Count; i++) { - NodeId = OpcUaNodeReference.Resolve(nodeId, _session.NamespaceUris), - AttributeId = Attributes.Value, - Value = new DataValue(new Variant(value)) - }; + try + { + resolvable.Add((i, values[i].NodeId, new WriteValue + { + NodeId = OpcUaNodeReference.Resolve(values[i].NodeId, session.NamespaceUris), + AttributeId = Attributes.Value, + Value = new DataValue(new Variant(values[i].Value)) + })); + } + catch (Exception ex) + { + outcomes[i] = new OpcUaWriteOutcome(values[i].NodeId, StatusCodes.BadNodeIdUnknown, ex.Message); + } + } - var response = await _session.WriteAsync( - null, new WriteValueCollection { writeValue }, cancellationToken); + for (var offset = 0; offset < resolvable.Count; offset += MaxNodesPerServiceCall) + { + var chunk = resolvable.GetRange(offset, Math.Min(MaxNodesPerServiceCall, resolvable.Count - offset)); + var collection = new WriteValueCollection(chunk.Select(c => c.Write)); + var response = await session.WriteAsync(null, collection, cancellationToken); - return response.Results[0].Code; + for (var i = 0; i < chunk.Count; i++) + { + var (index, nodeId, _) = chunk[i]; + if (response.Results is { Count: > 0 } results && i < results.Count) + { + outcomes[index] = new OpcUaWriteOutcome(nodeId, results[i].Code, null); + } + else + { + outcomes[index] = new OpcUaWriteOutcome( + nodeId, StatusCodes.BadUnexpectedError, + "OPC UA write returned no result for this node."); + } + } + } + + return outcomes; } /// diff --git a/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/DataConnectionFactory.cs b/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/DataConnectionFactory.cs index 81b1f53f..759233d2 100644 --- a/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/DataConnectionFactory.cs +++ b/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/DataConnectionFactory.cs @@ -37,11 +37,16 @@ public class DataConnectionFactory : IDataConnectionFactory public DataConnectionFactory( ILoggerFactory loggerFactory, IOptions opcUaGlobalOptions, - ISecretResolver? secretResolver = null) + ISecretResolver? secretResolver = null, + IOptions? dataConnectionOptions = null) { _loggerFactory = loggerFactory; _secretResolver = secretResolver; var globalOptions = opcUaGlobalOptions.Value; + // Only the MxGateway supervisory-advise window is adapter-level today; the rest of + // DataConnectionOptions is consumed by the actor, which is constructed elsewhere. + var supervisoryAdviseParallelism = + (dataConnectionOptions?.Value ?? new DataConnectionOptions()).MxSupervisoryAdviseParallelism; // Register built-in protocols. // Pass the ILoggerFactory into RealOpcUaClientFactory so @@ -57,7 +62,8 @@ public class DataConnectionFactory : IDataConnectionFactory RegisterAdapter("MxGateway", details => new MxGatewayDataConnection( new RealMxGatewayClientFactory(_loggerFactory), _loggerFactory.CreateLogger(), - _secretResolver)); + _secretResolver, + supervisoryAdviseParallelism)); } /// diff --git a/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/DataConnectionOptions.cs b/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/DataConnectionOptions.cs index aefe2f67..bbc6bec2 100644 --- a/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/DataConnectionOptions.cs +++ b/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/DataConnectionOptions.cs @@ -8,9 +8,70 @@ public class DataConnectionOptions /// Fixed interval between reconnect attempts after disconnect. public TimeSpan ReconnectInterval { get; set; } = TimeSpan.FromSeconds(5); - /// Interval for retrying failed tag path resolution. + /// + /// Floor interval for retrying failed tag path resolution. The retry backs off + /// exponentially (doubling per fully-failed round) up to + /// , and resets to this floor as soon as + /// any tag resolves or the connection reconnects. + /// public TimeSpan TagResolutionRetryInterval { get; set; } = TimeSpan.FromSeconds(10); + /// + /// Ceiling for the exponential tag-resolution retry backoff. A dead device with + /// thousands of unresolved tags would otherwise probe forever at full width every + /// . + /// + public TimeSpan TagResolutionRetryMaxInterval { get; set; } = TimeSpan.FromMinutes(5); + + /// + /// Number of tags per adapter subscribe round trip on the batch subscribe path + /// (initial subscribe, reconnect re-subscribe, tag-resolution probes). + /// + public int SubscribeBatchSize { get; set; } = 500; + + /// + /// Delay inserted BETWEEN reconnect re-subscribe chunks so a site-wide reconnect + /// paces the device instead of issuing one task per tag in a tight loop. Not applied + /// on the instance-driven subscribe path — the Deployment Manager already staggers + /// instance startup there, and double-staggering would only slow failover. + /// + public TimeSpan SubscribeBatchDelay { get; set; } = TimeSpan.FromMilliseconds(50); + + /// + /// Number of tags per seed-read chunk. Moderate on purpose: chunking plus the + /// per-chunk answers the "some gateways time out on a + /// large batch" caveat that originally motivated per-tag seed reads. + /// + public int SeedReadBatchSize { get; set; } = 250; + + /// Maximum seed-read chunks in flight concurrently. + public int SeedReadMaxParallelism { get; set; } = 4; + + /// + /// Wall-clock deadline for the WHOLE seed (all chunks and all retry rounds). On + /// expiry the remaining tags are logged and stay Uncertain until a change + /// notification arrives — the same outcome as exhausting + /// . + /// + public TimeSpan SeedOverallTimeout { get; set; } = TimeSpan.FromSeconds(120); + + /// + /// Coalescing window for pushing tag-quality counters to the health collector. + /// Counter math stays per message; only the collector push is deferred, and only + /// when a genuine quality transition occurred. Health reports poll at 30s, so a + /// short window loses nothing. Disconnect / unsubscribe / reconnect-reset flushes + /// stay synchronous. + /// + public TimeSpan QualityFlushInterval { get; set; } = TimeSpan.FromSeconds(1); + + /// + /// Maximum in-flight supervisory advise commands when the MxGateway adapter has no + /// write-user context. The gateway worker has no BULK supervisory advise, so the + /// client pipelines per-item advises after one bulk AddItem instead of issuing them + /// serially. + /// + public int MxSupervisoryAdviseParallelism { get; set; } = 16; + /// Timeout for synchronous write operations to devices. public TimeSpan WriteTimeout { get; set; } = TimeSpan.FromSeconds(30); diff --git a/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/DataConnectionOptionsValidator.cs b/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/DataConnectionOptionsValidator.cs index e80dce81..c74e4ed8 100644 --- a/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/DataConnectionOptionsValidator.cs +++ b/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/DataConnectionOptionsValidator.cs @@ -35,5 +35,30 @@ public sealed class DataConnectionOptionsValidator : OptionsValidatorBase 0, $"ScadaBridge:DataConnection:SeedReadMaxAttempts must be positive (was {options.SeedReadMaxAttempts})."); + + builder.RequireThat(options.TagResolutionRetryMaxInterval >= options.TagResolutionRetryInterval, + "ScadaBridge:DataConnection:TagResolutionRetryMaxInterval must be at least " + + $"TagResolutionRetryInterval (was {options.TagResolutionRetryMaxInterval} vs {options.TagResolutionRetryInterval})."); + + builder.RequireThat(options.SubscribeBatchSize > 0, + $"ScadaBridge:DataConnection:SubscribeBatchSize must be positive (was {options.SubscribeBatchSize})."); + + builder.RequireThat(options.SubscribeBatchDelay >= TimeSpan.Zero, + $"ScadaBridge:DataConnection:SubscribeBatchDelay must not be negative (was {options.SubscribeBatchDelay})."); + + builder.RequireThat(options.SeedReadBatchSize > 0, + $"ScadaBridge:DataConnection:SeedReadBatchSize must be positive (was {options.SeedReadBatchSize})."); + + builder.RequireThat(options.SeedReadMaxParallelism > 0, + $"ScadaBridge:DataConnection:SeedReadMaxParallelism must be positive (was {options.SeedReadMaxParallelism})."); + + builder.RequireThat(options.SeedOverallTimeout > TimeSpan.Zero, + $"ScadaBridge:DataConnection:SeedOverallTimeout must be a positive duration (was {options.SeedOverallTimeout})."); + + builder.RequireThat(options.QualityFlushInterval > TimeSpan.Zero, + $"ScadaBridge:DataConnection:QualityFlushInterval must be a positive duration (was {options.QualityFlushInterval})."); + + builder.RequireThat(options.MxSupervisoryAdviseParallelism > 0, + $"ScadaBridge:DataConnection:MxSupervisoryAdviseParallelism must be positive (was {options.MxSupervisoryAdviseParallelism})."); } } diff --git a/tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/Actors/DataConnectionActorBatchTests.cs b/tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/Actors/DataConnectionActorBatchTests.cs new file mode 100644 index 00000000..e681de59 --- /dev/null +++ b/tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/Actors/DataConnectionActorBatchTests.cs @@ -0,0 +1,291 @@ +using Akka.Actor; +using Akka.TestKit.Xunit2; +using NSubstitute; +using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Protocol; +using ZB.MOM.WW.ScadaBridge.Commons.Messages.DataConnection; +using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; +using ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Actors; +using ZB.MOM.WW.ScadaBridge.HealthMonitoring; + +namespace ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests.Actors; + +/// +/// WP2.1b — the DataConnectionActor against a batch-capable adapter: chunked subscribe, +/// chunked/bounded re-subscribe, the seed deadline, batched tag-resolution probes, the +/// partial-failure surface and the coalesced quality push. The existing suite covers the +/// same actor against a NON batch-capable adapter, which keeps the per-tag fallback pinned. +/// +public class DataConnectionActorBatchTests : TestKit +{ + private readonly ISiteHealthCollector _health = Substitute.For(); + private readonly IDataConnectionFactory _factory = Substitute.For(); + + private static DataConnectionOptions Options() => new() + { + ReconnectInterval = TimeSpan.FromMilliseconds(100), + TagResolutionRetryInterval = TimeSpan.FromMilliseconds(150), + TagResolutionRetryMaxInterval = TimeSpan.FromMilliseconds(600), + WriteTimeout = TimeSpan.FromSeconds(5), + SeedReadMaxAttempts = 1, + SeedReadRetryDelay = TimeSpan.FromMilliseconds(10), + SubscribeBatchSize = 5, + SubscribeBatchDelay = TimeSpan.FromMilliseconds(50), + SeedReadBatchSize = 4, + SeedReadMaxParallelism = 2, + SeedOverallTimeout = TimeSpan.FromSeconds(30), + QualityFlushInterval = TimeSpan.FromMilliseconds(200) + }; + + private IActorRef CreateActor(FakeBatchDataConnection adapter, DataConnectionOptions options, string name) => + Sys.ActorOf(Props.Create(() => new DataConnectionActor( + name, adapter, options, _health, _factory, "OpcUa")), name); + + private static string[] Tags(int count) => Enumerable.Range(1, count).Select(i => $"tag{i}").ToArray(); + + [Fact] + public void Subscribe_IssuesOneAdapterRoundTripPerChunk_NotPerTag() + { + // 12 tags at SubscribeBatchSize 5 → 3 batch calls (5/5/2) and ZERO per-tag + // subscribes. This is finding #3: the old path was one adapter call (and one OPC UA + // ApplyChanges) per tag. + var adapter = new FakeBatchDataConnection(); + var actor = CreateActor(adapter, Options(), "batch-chunking"); + + actor.Tell(new SubscribeTagsRequest("c1", "inst1", "batch-chunking", Tags(12), DateTimeOffset.UtcNow)); + ExpectMsg(m => m.Success, TimeSpan.FromSeconds(5)); + + var batches = adapter.SubscribeBatches.ToList(); + Assert.Equal(3, batches.Count); + Assert.Equal([5, 5, 2], batches.Select(b => b.Count)); + Assert.Equal(0, Volatile.Read(ref adapter.SingleSubscribeCalls)); + } + + [Fact] + public void Seed_UsesChunkedBulkReads_NotPerTagReads() + { + // Seeding 6 tags at SeedReadBatchSize 4 → 2 bulk reads, no single-tag reads. + var adapter = new FakeBatchDataConnection(); + var actor = CreateActor(adapter, Options(), "batch-seed"); + + actor.Tell(new SubscribeTagsRequest("c1", "inst1", "batch-seed", Tags(6), DateTimeOffset.UtcNow)); + ExpectMsg(m => m.Success, TimeSpan.FromSeconds(5)); + + var reads = adapter.ReadBatches.ToList(); + Assert.Equal(2, reads.Count); + Assert.Equal(6, reads.Sum(r => r.Count)); + Assert.Equal(0, Volatile.Read(ref adapter.SingleReadCalls)); + // The seeded value still reaches the instance actor after registration. + ExpectMsg(u => u.Quality == QualityCode.Good, TimeSpan.FromSeconds(5)); + } + + [Fact] + public void Seed_HangingReads_ReturnAtTheOverallDeadline_AndTheSubscribeIsStillAcked() + { + // A device whose reads never answer must not hold SubscribeCompleted (and therefore + // the instance's ack) open: the whole seed is bounded by SeedOverallTimeout. + var options = Options(); + options.SeedOverallTimeout = TimeSpan.FromMilliseconds(600); + options.SeedReadTimeout = TimeSpan.FromSeconds(30); // deliberately longer than the deadline + options.SeedReadMaxAttempts = 3; + + var adapter = new FakeBatchDataConnection { HangReads = true }; + var actor = CreateActor(adapter, options, "batch-seed-deadline"); + + var started = DateTimeOffset.UtcNow; + actor.Tell(new SubscribeTagsRequest("c1", "inst1", "batch-seed-deadline", Tags(3), DateTimeOffset.UtcNow)); + ExpectMsg(m => m.Success, TimeSpan.FromSeconds(10)); + var elapsed = DateTimeOffset.UtcNow - started; + + // Bounded by the overall deadline, not by SeedReadTimeout (30s) or the retry budget. + Assert.True(elapsed < TimeSpan.FromSeconds(5), $"seed took {elapsed.TotalSeconds:F1}s"); + // No value was seeded — the tags stay Uncertain until a change notification. + ExpectNoMsg(TimeSpan.FromMilliseconds(200)); + } + + [Fact] + public void PartialFailure_PerTagRow_MarksTagBad_ButTheRequestStillSucceeds() + { + // A per-tag failure row is a tag-resolution problem: Bad quality is pushed for that + // tag and the subscribe is still acked Success (the surviving tags are live). + var adapter = new FakeBatchDataConnection(); + adapter.FailingTags.Add("tag2"); + var actor = CreateActor(adapter, Options(), "batch-partial"); + + actor.Tell(new SubscribeTagsRequest("c1", "inst1", "batch-partial", Tags(3), DateTimeOffset.UtcNow)); + + ExpectMsg(u => u.TagPath == "tag2" && u.Quality == QualityCode.Bad, TimeSpan.FromSeconds(5)); + ExpectMsg(m => m.Success, TimeSpan.FromSeconds(5)); + } + + [Fact] + public void ThrownBatch_AtConnectionLevel_FailsTheRequestAndDrivesReconnect() + { + // A THROWN batch means the whole chunk failed at connection level: the response must + // say so, and the actor must enter Reconnecting (which pushes bad quality for the + // connection). + var adapter = new FakeBatchDataConnection + { + BatchSubscribeThrows = () => new InvalidOperationException("client is not connected") + }; + var actor = CreateActor(adapter, Options(), "batch-connection-fault"); + + actor.Tell(new SubscribeTagsRequest("c1", "inst1", "batch-connection-fault", Tags(3), DateTimeOffset.UtcNow)); + + ExpectMsg(m => !m.Success && m.ErrorMessage != null, TimeSpan.FromSeconds(5)); + ExpectMsg(q => q.Quality == QualityCode.Bad, TimeSpan.FromSeconds(5)); + } + + [Fact] + public void ResolutionProbes_AreBatched_NotOnePerTag() + { + // Every unresolved tag is probed in ONE batch call per chunk. Pre-fix the retry tick + // fired one SubscribeAsync task per unresolved tag, every tick, forever. + var options = Options(); + options.SubscribeBatchSize = 10; + var adapter = new FakeBatchDataConnection(); + foreach (var tag in Tags(6)) + adapter.FailingTags.Add(tag); + + var actor = CreateActor(adapter, options, "batch-probe"); + actor.Tell(new SubscribeTagsRequest("c1", "inst1", "batch-probe", Tags(6), DateTimeOffset.UtcNow)); + + for (var i = 0; i < 6; i++) + ExpectMsg(u => u.Quality == QualityCode.Bad, TimeSpan.FromSeconds(5)); + ExpectMsg(TimeSpan.FromSeconds(5)); + + var initialCalls = adapter.SubscribeBatches.Count; + AwaitCondition(() => adapter.SubscribeBatches.Count > initialCalls, TimeSpan.FromSeconds(5)); + + // The probe round carried all six unresolved tags in a single call. + var probe = adapter.SubscribeBatches.Skip(initialCalls).First(); + Assert.Equal(6, probe.Count); + Assert.Equal(0, Volatile.Read(ref adapter.SingleSubscribeCalls)); + } + + [Fact] + public void ResolutionProbes_BackOff_SoADeadDeviceIsNotProbedAtFullRate() + { + // Floor 150ms, ceiling 600ms: a fixed-interval retry would fire ~13 rounds in 2s; + // the backoff sequence (150, 300, 600, 600 …) fires far fewer. + var options = Options(); + options.SubscribeBatchSize = 10; + var adapter = new FakeBatchDataConnection(); + adapter.FailingTags.Add("tag1"); + + var actor = CreateActor(adapter, options, "batch-backoff"); + actor.Tell(new SubscribeTagsRequest("c1", "inst1", "batch-backoff", ["tag1"], DateTimeOffset.UtcNow)); + ExpectMsg(u => u.Quality == QualityCode.Bad, TimeSpan.FromSeconds(5)); + ExpectMsg(TimeSpan.FromSeconds(5)); + + var initialCalls = adapter.SubscribeBatches.Count; + Thread.Sleep(2000); + var rounds = adapter.SubscribeBatches.Count - initialCalls; + + Assert.InRange(rounds, 2, 8); + } + + [Fact] + public void QualityCounters_ArePushedOnTransitionOnly_AndCoalesced() + { + // Repeated values at UNCHANGED quality move no counter and must produce no collector + // push at all; a genuine transition produces exactly one push per coalescing window. + var adapter = new FakeBatchDataConnection(); + var actor = CreateActor(adapter, Options(), "batch-quality"); + + actor.Tell(new SubscribeTagsRequest("c1", "inst1", "batch-quality", ["tag1"], DateTimeOffset.UtcNow)); + ExpectMsg(m => m.Success, TimeSpan.FromSeconds(5)); + AwaitCondition(() => adapter.ValueCallback != null, TimeSpan.FromSeconds(5)); + + // Let the seed's own transition flush, then start counting. + Thread.Sleep(400); + _health.ClearReceivedCalls(); + + for (var i = 0; i < 20; i++) + adapter.ValueCallback!("tag1", new TagValue(i, QualityCode.Good, DateTimeOffset.UtcNow)); + + Thread.Sleep(400); + Assert.DoesNotContain(_health.ReceivedCalls(), c => c.GetMethodInfo().Name == "UpdateTagQuality"); + + // One genuine transition → exactly one coalesced push. + adapter.ValueCallback!("tag1", new TagValue(1, QualityCode.Bad, DateTimeOffset.UtcNow)); + AwaitCondition( + () => _health.ReceivedCalls().Count(c => c.GetMethodInfo().Name == "UpdateTagQuality") == 1, + TimeSpan.FromSeconds(3)); + Thread.Sleep(300); + Assert.Single(_health.ReceivedCalls(), c => c.GetMethodInfo().Name == "UpdateTagQuality"); + } + + [Fact] + public void QualityCounters_FlushImmediatelyOnDisconnect() + { + // "Immediate bad quality on disconnect" must never wait for the coalescing timer. + var options = Options(); + options.QualityFlushInterval = TimeSpan.FromSeconds(30); // would mask a deferred push + var adapter = new FakeBatchDataConnection(); + var actor = CreateActor(adapter, options, "batch-quality-disconnect"); + + actor.Tell(new SubscribeTagsRequest("c1", "inst1", "batch-quality-disconnect", ["tag1"], DateTimeOffset.UtcNow)); + ExpectMsg(m => m.Success, TimeSpan.FromSeconds(5)); + ExpectMsg(TimeSpan.FromSeconds(5)); // the seed + _health.ClearReceivedCalls(); + + adapter.RaiseDisconnected(); + + ExpectMsg(q => q.Quality == QualityCode.Bad, TimeSpan.FromSeconds(5)); + AwaitCondition( + () => _health.ReceivedCalls().Any(c => c.GetMethodInfo().Name == "UpdateTagQuality"), + TimeSpan.FromSeconds(3)); + } + + [Fact] + public void Unsubscribe_ReleasesEveryHandleInOneRoundTrip() + { + var adapter = new FakeBatchDataConnection(); + var actor = CreateActor(adapter, Options(), "batch-unsubscribe"); + + actor.Tell(new SubscribeTagsRequest("c1", "inst1", "batch-unsubscribe", Tags(6), DateTimeOffset.UtcNow)); + ExpectMsg(m => m.Success, TimeSpan.FromSeconds(5)); + + actor.Tell(new UnsubscribeTagsRequest("c2", "inst1", "batch-unsubscribe", DateTimeOffset.UtcNow)); + + AwaitCondition(() => adapter.UnsubscribeBatches.Count == 1, TimeSpan.FromSeconds(5)); + Assert.True(adapter.UnsubscribeBatches.TryDequeue(out var released)); + Assert.Equal(6, released!.Count); + } + + [Fact] + public void Reconnect_ReSubscribesInPacedChunks_AndRestoresTheResolvedCount() + { + // The reconnect re-subscribe is chunked and paced (SubscribeBatchDelay between + // chunks) rather than firing one task per tag, and the per-tag results repopulate + // _subscriptionIds so a later unsubscribe still releases every adapter handle. + var options = Options(); + options.SubscribeBatchSize = 4; + options.SubscribeBatchDelay = TimeSpan.FromMilliseconds(120); + + var adapter = new FakeBatchDataConnection(); + var actor = CreateActor(adapter, options, "batch-resubscribe"); + + actor.Tell(new SubscribeTagsRequest("c1", "inst1", "batch-resubscribe", Tags(8), DateTimeOffset.UtcNow)); + ExpectMsg(m => m.Success, TimeSpan.FromSeconds(5)); + + var beforeReconnect = adapter.SubscribeBatches.Count; + adapter.RaiseDisconnected(); + + // Two re-subscribe chunks of 4. + AwaitCondition(() => adapter.SubscribeBatches.Count >= beforeReconnect + 2, TimeSpan.FromSeconds(10)); + var reconnectChunks = adapter.SubscribeBatches.Skip(beforeReconnect).Take(2).ToList(); + Assert.Equal([4, 4], reconnectChunks.Select(c => c.Count)); + + // Paced: the second chunk is not issued back-to-back with the first. + var times = adapter.SubscribeBatchTimes.Skip(beforeReconnect).Take(2).ToList(); + Assert.True(times[1] - times[0] >= TimeSpan.FromMilliseconds(100), + $"chunks were {(times[1] - times[0]).TotalMilliseconds:F0}ms apart"); + + // Every tag is registered again, so unsubscribe releases all 8 handles. + actor.Tell(new UnsubscribeTagsRequest("c2", "inst1", "batch-resubscribe", DateTimeOffset.UtcNow)); + AwaitCondition( + () => adapter.UnsubscribeBatches.Any(b => b.Count == 8), + TimeSpan.FromSeconds(10)); + } +} diff --git a/tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/Actors/FakeBatchDataConnection.cs b/tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/Actors/FakeBatchDataConnection.cs new file mode 100644 index 00000000..ad4067c9 --- /dev/null +++ b/tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/Actors/FakeBatchDataConnection.cs @@ -0,0 +1,161 @@ +using System.Collections.Concurrent; +using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Protocol; +using ZB.MOM.WW.ScadaBridge.Commons.Types.Alarms; +using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; + +namespace ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests.Actors; + +/// +/// In-memory batch-capable for the WP2.1b seam tests. It +/// records every batch call (so "one round trip per chunk, not per tag" is assertable), +/// can fail individual tags or a whole batch, and can hang its bulk read so the seed +/// deadline is observable. +/// +public sealed class FakeBatchDataConnection + : IDataConnection, IBatchSubscribableConnection, IAlarmSubscribableConnection +{ + private int _nextId; + + /// Tag lists handed to , one entry per call. + public readonly ConcurrentQueue> SubscribeBatches = new(); + /// Id lists handed to , one entry per call. + public readonly ConcurrentQueue> UnsubscribeBatches = new(); + /// Tag lists handed to , one entry per call. + public readonly ConcurrentQueue> ReadBatches = new(); + /// Count of SINGLE-tag subscribe calls; must stay 0 on a batch-capable adapter. + public int SingleSubscribeCalls; + /// Count of SINGLE-tag read calls; must stay 0 on a batch-capable adapter. + public int SingleReadCalls; + /// Wall-clock instant of each call. + public readonly ConcurrentQueue SubscribeBatchTimes = new(); + + /// Tags reported as failed rows (per-tag resolution failure). + public readonly HashSet FailingTags = new(StringComparer.Ordinal); + /// When set, every batch subscribe throws this — a batch-level fault. + public Func? BatchSubscribeThrows; + /// When true, bulk reads never return until the caller's token cancels. + public bool HangReads; + /// Value returned for every readable tag. + public object? SeedValue = 42; + + /// Callback the last batch subscribe registered; drives value pushes in tests. + public SubscriptionCallback? ValueCallback; + /// Callback the last alarm subscribe registered. + public AlarmTransitionCallback? AlarmCallback; + + /// + public ConnectionHealth Status { get; private set; } = ConnectionHealth.Disconnected; + + /// + public event Action? Disconnected; + + /// Raises as a real adapter would on a transport fault. + public void RaiseDisconnected() => Disconnected?.Invoke(); + + /// + public Task ConnectAsync(IDictionary connectionDetails, CancellationToken cancellationToken = default) + { + Status = ConnectionHealth.Connected; + return Task.CompletedTask; + } + + /// + public Task DisconnectAsync(CancellationToken cancellationToken = default) + { + Status = ConnectionHealth.Disconnected; + return Task.CompletedTask; + } + + /// + public Task> SubscribeBatchAsync( + IReadOnlyList tagPaths, SubscriptionCallback callback, CancellationToken cancellationToken = default) + { + SubscribeBatches.Enqueue(tagPaths.ToList()); + SubscribeBatchTimes.Enqueue(DateTimeOffset.UtcNow); + ValueCallback = callback; + + if (BatchSubscribeThrows is { } factory) + throw factory(); + + IReadOnlyList rows = tagPaths + .Select(t => FailingTags.Contains(t) + ? new TagSubscribeResult(t, false, null, "node not found") + : new TagSubscribeResult(t, true, $"sub-{Interlocked.Increment(ref _nextId)}", null)) + .ToList(); + return Task.FromResult(rows); + } + + /// + public Task UnsubscribeBatchAsync(IReadOnlyList subscriptionIds, CancellationToken cancellationToken = default) + { + UnsubscribeBatches.Enqueue(subscriptionIds.ToList()); + return Task.CompletedTask; + } + + /// + public Task SubscribeAsync(string tagPath, SubscriptionCallback callback, CancellationToken cancellationToken = default) + { + Interlocked.Increment(ref SingleSubscribeCalls); + ValueCallback = callback; + return Task.FromResult($"sub-{Interlocked.Increment(ref _nextId)}"); + } + + /// + public Task UnsubscribeAsync(string subscriptionId, CancellationToken cancellationToken = default) + => Task.CompletedTask; + + /// + public async Task ReadAsync(string tagPath, CancellationToken cancellationToken = default) + { + Interlocked.Increment(ref SingleReadCalls); + if (HangReads) + await Task.Delay(Timeout.Infinite, cancellationToken); + return new ReadResult(true, new TagValue(SeedValue, QualityCode.Good, DateTimeOffset.UtcNow), null); + } + + /// + public async Task> ReadBatchAsync( + IEnumerable tagPaths, CancellationToken cancellationToken = default) + { + var tags = tagPaths.ToList(); + ReadBatches.Enqueue(tags); + if (HangReads) + await Task.Delay(Timeout.Infinite, cancellationToken); + + return tags.ToDictionary( + t => t, + t => new ReadResult(true, new TagValue(SeedValue, QualityCode.Good, DateTimeOffset.UtcNow), null)); + } + + /// + public Task WriteAsync(string tagPath, object? value, CancellationToken cancellationToken = default) + => Task.FromResult(new WriteResult(true, null)); + + /// + public Task> WriteBatchAsync( + IDictionary values, CancellationToken cancellationToken = default) + => Task.FromResult>( + values.ToDictionary(kv => kv.Key, _ => new WriteResult(true, null))); + + /// + public Task WriteBatchAndWaitAsync( + IDictionary values, string flagPath, object? flagValue, string responsePath, + object? responseValue, TimeSpan timeout, CancellationToken cancellationToken = default) + => Task.FromResult(true); + + /// + public Task SubscribeAlarmsAsync( + string sourceReference, string? conditionFilter, AlarmTransitionCallback callback, + CancellationToken cancellationToken = default) + { + AlarmCallback = callback; + return Task.FromResult($"alarm-{Interlocked.Increment(ref _nextId)}"); + } + + /// + public Task UnsubscribeAlarmsAsync(string subscriptionId, CancellationToken cancellationToken = default) + => Task.CompletedTask; + + /// + public ValueTask DisposeAsync() => ValueTask.CompletedTask; +} diff --git a/tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/Adapters/BatchSeamPrimitiveTests.cs b/tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/Adapters/BatchSeamPrimitiveTests.cs new file mode 100644 index 00000000..65a71abb --- /dev/null +++ b/tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/Adapters/BatchSeamPrimitiveTests.cs @@ -0,0 +1,204 @@ +using ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Actors; +using ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Adapters; + +namespace ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests.Adapters; + +/// +/// WP2.1b — the primitives the batch seam is built from, tested in isolation because the +/// OPC Foundation Session/Subscription types they drive cannot be faked without a live +/// server: monitored-item shard placement, the apply-serialization gate, the bounded +/// pipeline behind the MxGateway supervisory advise, the alarm-stream union prefix, and +/// the tag-resolution backoff step. +/// +public class BatchSeamPrimitiveTests +{ + // ── Shard placement ── + + [Fact] + public void Sharding_SplitsItemsAtTheBudget() + { + // 12,001 items at the 5,000 default → 3 shards (memo §2 sizing example). + Assert.Equal(3, MonitoredItemShardPlanner.ShardCountFor(12_001, 5_000)); + // 37,500 tags → 8 shards. + Assert.Equal(8, MonitoredItemShardPlanner.ShardCountFor(37_500, 5_000)); + + var placement = MonitoredItemShardPlanner.Plan([], 5_000, 12_001); + Assert.Equal(12_001, placement.Count); + Assert.Equal(0, placement[0]); + Assert.Equal(0, placement[4_999]); + Assert.Equal(1, placement[5_000]); + Assert.Equal(2, placement[10_000]); + Assert.Equal(2, placement[12_000]); + } + + [Fact] + public void Sharding_FillsTheFirstShardWithFreeCapacityBeforeCreatingOne() + { + // Shard 0 full, shard 1 has one free slot: the next two items fill shard 1 then + // open shard 2 — the "first shard with free capacity, else a new shard" policy, + // which is also what makes an emptied-and-deleted shard's capacity reusable. + var placement = MonitoredItemShardPlanner.Plan([5_000, 4_999], 5_000, 2); + Assert.Equal([1, 2], placement); + } + + [Fact] + public void Sharding_NonPositiveBudgetDegradesToOneItemPerShard() + { + Assert.Equal([0, 1, 2], MonitoredItemShardPlanner.Plan([], 0, 3)); + } + + // ── Apply serialization ── + + [Fact] + public async Task ApplyGate_NeverRunsTwoSectionsConcurrently() + { + var gate = new AsyncSerialGate(); + var inFlight = 0; + var maxObserved = 0; + + var tasks = Enumerable.Range(0, 32).Select(_ => gate.RunAsync(async () => + { + var now = Interlocked.Increment(ref inFlight); + InterlockedMax(ref maxObserved, now); + await Task.Delay(5); + Interlocked.Decrement(ref inFlight); + })).ToArray(); + + await Task.WhenAll(tasks); + + Assert.Equal(1, maxObserved); + } + + [Fact] + public async Task ApplyGate_ReleasesWhenASectionThrows() + { + var gate = new AsyncSerialGate(); + + await Assert.ThrowsAsync(() => + gate.RunAsync(() => throw new InvalidOperationException("apply failed"))); + + // The gate must still admit the next caller — a faulted ApplyChanges must not + // wedge every later subscribe/unsubscribe on this client. + var ran = false; + await gate.RunAsync(() => + { + ran = true; + return Task.CompletedTask; + }); + Assert.True(ran); + } + + // ── Bounded pipeline (MxGateway supervisory advise) ── + + [Fact] + public async Task BulkPipeline_KeepsInFlightCountWithinTheWindow() + { + const int parallelism = 16; + var inFlight = 0; + var maxObserved = 0; + var items = Enumerable.Range(0, 200).ToList(); + + var results = await BulkPipeline.RunAsync( + items, + parallelism, + async (item, _) => + { + var now = Interlocked.Increment(ref inFlight); + InterlockedMax(ref maxObserved, now); + await Task.Delay(2); + Interlocked.Decrement(ref inFlight); + return item * 2; + }, + (item, _) => -item); + + Assert.Equal(items.Count, results.Length); + Assert.Equal(items.Select(i => i * 2), results); + Assert.InRange(maxObserved, 1, parallelism); + } + + [Fact] + public async Task BulkPipeline_CapturesPerItemFaultsWithoutAbortingTheBatch() + { + var results = await BulkPipeline.RunAsync( + [1, 2, 3], + 4, + (item, _) => item == 2 + ? throw new InvalidOperationException("advise failed") + : Task.FromResult(item), + (item, _) => -item); + + Assert.Equal([1, -2, 3], results); + } + + // ── Alarm-stream union prefix ── + + [Theory] + [InlineData(new[] { "Area1.Tank1", "Area1.Tank2" }, "Area1.Tank")] + [InlineData(new[] { "Area1.Tank1" }, "Area1.Tank1")] + [InlineData(new[] { "Area1.Tank1", "Area2.Tank1" }, "Area")] + [InlineData(new[] { "Plant.A", "Zone.B" }, "")] + [InlineData(new[] { "Area1.Tank1", "" }, "")] + [InlineData(new string[0], "")] + public void AlarmPrefix_IsTheLongestCommonPrefix(string[] sources, string expected) + { + Assert.Equal(expected, AlarmFilterPrefix.LongestCommonPrefix(sources)); + } + + [Fact] + public void AlarmPrefix_CoversOnlySourcesUnderTheLivePrefix() + { + Assert.True(AlarmFilterPrefix.Covers("Area1.", "Area1.Tank1")); + Assert.False(AlarmFilterPrefix.Covers("Area1.", "Area2.Tank1")); + // A gateway-wide stream covers everything. + Assert.True(AlarmFilterPrefix.Covers("", "Anything.At.All")); + } + + // ── Tag-resolution backoff ── + + [Fact] + public void Backoff_DoublesPerFailedRoundAndCapsAtTheMaximum() + { + var floor = TimeSpan.FromSeconds(10); + var max = TimeSpan.FromMinutes(5); + + var sequence = new List(); + var current = floor; + for (var round = 0; round < 8; round++) + { + current = DataConnectionActor.NextTagResolutionInterval(current, floor, max); + sequence.Add(current); + } + + Assert.Equal( + [ + TimeSpan.FromSeconds(20), + TimeSpan.FromSeconds(40), + TimeSpan.FromSeconds(80), + TimeSpan.FromSeconds(160), + TimeSpan.FromSeconds(300), + TimeSpan.FromSeconds(300), + TimeSpan.FromSeconds(300), + TimeSpan.FromSeconds(300), + ], sequence); + } + + [Fact] + public void Backoff_MisconfiguredCeilingBelowFloorDegradesToAFixedInterval() + { + var floor = TimeSpan.FromSeconds(10); + var next = DataConnectionActor.NextTagResolutionInterval(floor, floor, TimeSpan.FromSeconds(1)); + Assert.Equal(floor, next); + } + + private static void InterlockedMax(ref int target, int value) + { + int seen; + do + { + seen = Volatile.Read(ref target); + if (value <= seen) + return; + } + while (Interlocked.CompareExchange(ref target, value, seen) != seen); + } +} diff --git a/tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/Adapters/FakeMxGatewayClient.cs b/tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/Adapters/FakeMxGatewayClient.cs index b1d1982e..995d9574 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/Adapters/FakeMxGatewayClient.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/Adapters/FakeMxGatewayClient.cs @@ -11,6 +11,14 @@ public sealed class FakeMxGatewayClient : IMxGatewayClient, IMxGatewayClientFact public MxGatewayConnectionOptions? ConnectedWith; public readonly List Subscribed = new(); public readonly List Unsubscribed = new(); + /// One entry per SubscribeBulkAsync call, carrying that call's tag list. + public readonly List> BulkSubscribeCalls = new(); + /// One entry per UnsubscribeBulkAsync call, carrying that call's id list. + public readonly List> BulkUnsubscribeCalls = new(); + /// Tags the fake reports as failed rows from a bulk subscribe. + public readonly HashSet BulkSubscribeFailures = new(StringComparer.Ordinal); + /// Every alarm-stream prefix the adapter opened a stream with (null = gateway-wide). + public readonly List AlarmStreamPrefixes = new(); public readonly TaskCompletionSource EventLoopGate = new(TaskCreationOptions.RunContinuationsAsynchronously); public Action? OnUpdate; public Func, IReadOnlyList>? ReadHandler; @@ -41,6 +49,34 @@ public sealed class FakeMxGatewayClient : IMxGatewayClient, IMxGatewayClientFact return Task.CompletedTask; } + public Task> SubscribeBulkAsync( + IReadOnlyList tagPaths, CancellationToken ct = default) + { + BulkSubscribeCalls.Add(tagPaths.ToList()); + var outcomes = new List(tagPaths.Count); + foreach (var tag in tagPaths) + { + if (BulkSubscribeFailures.Contains(tag)) + { + outcomes.Add(new MxSubscribeOutcome(tag, false, null, "not found")); + continue; + } + + Subscribed.Add(tag); + outcomes.Add(new MxSubscribeOutcome( + tag, true, (++_nextHandle).ToString(), null)); + } + + return Task.FromResult>(outcomes); + } + + public Task UnsubscribeBulkAsync(IReadOnlyList subscriptionIds, CancellationToken ct = default) + { + BulkUnsubscribeCalls.Add(subscriptionIds.ToList()); + Unsubscribed.AddRange(subscriptionIds); + return Task.CompletedTask; + } + public Task> ReadAsync(IReadOnlyList tags, CancellationToken ct = default) => Task.FromResult(ReadHandler!(tags)); @@ -62,7 +98,13 @@ public sealed class FakeMxGatewayClient : IMxGatewayClient, IMxGatewayClientFact string? alarmFilterPrefix, Action onTransition, CancellationToken ct = default) - => Task.CompletedTask; // no alarm feed in the fake + { + // No alarm feed in the fake — but the prefix each stream is opened with is + // recorded so the union-filter (longest-common-prefix) behaviour is testable. + lock (AlarmStreamPrefixes) + AlarmStreamPrefixes.Add(alarmFilterPrefix); + return Task.CompletedTask; + } public ValueTask DisposeAsync() => ValueTask.CompletedTask; diff --git a/tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/Adapters/MxGatewayBatchSeamTests.cs b/tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/Adapters/MxGatewayBatchSeamTests.cs new file mode 100644 index 00000000..fbee3bf2 --- /dev/null +++ b/tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/Adapters/MxGatewayBatchSeamTests.cs @@ -0,0 +1,146 @@ +using Microsoft.Extensions.Logging.Abstractions; +using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Protocol; +using ZB.MOM.WW.ScadaBridge.Commons.Types.Alarms; +using ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Adapters; + +namespace ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests.Adapters; + +/// +/// WP2.1b — the MxGateway adapter's batch subscribe seam and the alarm-stream union +/// filter. The plain-vs-supervisory advise choice itself lives inside +/// RealMxGatewayClient (it needs a live MXAccess session); its bounded-pipeline +/// mechanism is pinned separately by . +/// +[Collection("DataConnectionManagerActor")] +public class MxGatewayBatchSeamTests +{ + private static MxGatewayDataConnection NewAdapter(FakeMxGatewayClient fake) => + new(fake, NullLogger.Instance); + + private static Dictionary Details() => new() + { + ["Endpoint"] = "http://gw:5000", + ["ApiKey"] = "key", + ["ClientName"] = "client-a", + ["WriteUserId"] = "0", + ["ReadTimeoutMs"] = "2000", + }; + + [Fact] + public async Task SubscribeBatch_IssuesExactlyOneBulkCallPerBatch() + { + var fake = new FakeMxGatewayClient(); + var adapter = NewAdapter(fake); + await adapter.ConnectAsync(Details()); + + var results = await adapter.SubscribeBatchAsync( + ["A.x", "A.y", "A.z"], (_, _) => { }); + + // ONE bulk RPC for the whole batch — replacing the historical AddItem + Advise + // pair PER TAG (6 RPCs for these three tags). + Assert.Single(fake.BulkSubscribeCalls); + Assert.Equal(3, fake.BulkSubscribeCalls[0].Count); + Assert.Equal(3, results.Count); + Assert.All(results, r => Assert.True(r.Success)); + Assert.All(results, r => Assert.NotNull(r.SubscriptionId)); + } + + [Fact] + public async Task SubscribeBatch_ReportsPerTagFailuresWithoutFailingTheBatch() + { + var fake = new FakeMxGatewayClient(); + fake.BulkSubscribeFailures.Add("A.y"); + var adapter = NewAdapter(fake); + await adapter.ConnectAsync(Details()); + + var results = await adapter.SubscribeBatchAsync(["A.x", "A.y"], (_, _) => { }); + + Assert.True(results.Single(r => r.TagPath == "A.x").Success); + var failed = results.Single(r => r.TagPath == "A.y"); + Assert.False(failed.Success); + Assert.Null(failed.SubscriptionId); + Assert.NotNull(failed.ErrorMessage); + } + + [Fact] + public async Task UnsubscribeBatch_ReleasesEveryIdInOneBulkCall() + { + var fake = new FakeMxGatewayClient(); + var adapter = NewAdapter(fake); + await adapter.ConnectAsync(Details()); + + var results = await adapter.SubscribeBatchAsync(["A.x", "A.y"], (_, _) => { }); + await adapter.UnsubscribeBatchAsync(results.Select(r => r.SubscriptionId!).ToList()); + + Assert.Single(fake.BulkUnsubscribeCalls); + Assert.Equal(2, fake.BulkUnsubscribeCalls[0].Count); + } + + [Fact] + public async Task AlarmStream_OpensOnTheLongestCommonPrefix_AndOnlyRestartsForAnEscapingSource() + { + var fake = new FakeMxGatewayClient(); + var adapter = NewAdapter(fake); + await adapter.ConnectAsync(Details()); + + void NoTransition(NativeAlarmTransition _) { } + + // First source → the stream opens scoped to it. + await adapter.SubscribeAlarmsAsync("Area1.Tank1", null, NoTransition); + await WaitForPrefixCountAsync(fake, 1); + Assert.Equal("Area1.Tank1", fake.AlarmStreamPrefixes[0]); + + // A sibling under the same prefix widens the union → one restart on "Area1.Tank". + await adapter.SubscribeAlarmsAsync("Area1.Tank2", null, NoTransition); + await WaitForPrefixCountAsync(fake, 2); + Assert.Equal("Area1.Tank", fake.AlarmStreamPrefixes[1]); + + // A source ALREADY covered by the live prefix must NOT restart the stream. + await adapter.SubscribeAlarmsAsync("Area1.Tank3.Sub", null, NoTransition); + await Task.Delay(100); + Assert.Equal(2, fake.AlarmStreamPrefixes.Count); + + // A source outside the prefix widens it again — here down to gateway-wide. + await adapter.SubscribeAlarmsAsync("Zone9.Pump", null, NoTransition); + await WaitForPrefixCountAsync(fake, 3); + Assert.Null(fake.AlarmStreamPrefixes[2]); // "" → gateway-wide + } + + [Fact] + public async Task AlarmStream_UnsubscribeNeverRestartsTheStream() + { + var fake = new FakeMxGatewayClient(); + var adapter = NewAdapter(fake); + await adapter.ConnectAsync(Details()); + + var first = await adapter.SubscribeAlarmsAsync("Area1.Tank1", null, _ => { }); + // Wait for the first stream to actually open before widening: the open runs on a + // Task.Run whose token the restart cancels, so a restart racing an unstarted task + // would leave the first prefix unrecorded. + await WaitForPrefixCountAsync(fake, 1); + await adapter.SubscribeAlarmsAsync("Area1.Tank2", null, _ => { }); + await WaitForPrefixCountAsync(fake, 2); + + // Dropping a source leaves the prefix too BROAD at worst — bandwidth, not + // correctness — so no restart is issued. + await adapter.UnsubscribeAlarmsAsync(first); + await Task.Delay(100); + Assert.Equal(2, fake.AlarmStreamPrefixes.Count); + } + + private static async Task WaitForPrefixCountAsync(FakeMxGatewayClient fake, int expected) + { + var deadline = DateTime.UtcNow + TimeSpan.FromSeconds(5); + while (DateTime.UtcNow < deadline) + { + lock (fake.AlarmStreamPrefixes) + { + if (fake.AlarmStreamPrefixes.Count >= expected) + return; + } + await Task.Delay(20); + } + + Assert.Fail($"alarm stream was opened {fake.AlarmStreamPrefixes.Count} time(s), expected {expected}"); + } +} diff --git a/tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/DataConnectionActorAlarmIndexTests.cs b/tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/DataConnectionActorAlarmIndexTests.cs new file mode 100644 index 00000000..4b403ac9 --- /dev/null +++ b/tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/DataConnectionActorAlarmIndexTests.cs @@ -0,0 +1,166 @@ +using Akka.Actor; +using Akka.TestKit.Xunit2; +using NSubstitute; +using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Protocol; +using ZB.MOM.WW.ScadaBridge.Commons.Messages.DataConnection; +using ZB.MOM.WW.ScadaBridge.Commons.Types.Alarms; +using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; +using ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Actors; +using ZB.MOM.WW.ScadaBridge.HealthMonitoring; + +namespace ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests; + +/// +/// WP2.1b — the alarm subscriber first-segment bucket index must make EXACTLY the routing +/// decisions the previous linear scan made. The expectation in each case is computed with +/// the original rule (plain StartsWith against every subscribed source), so the +/// test pins equivalence rather than restating the new implementation. +/// +public class DataConnectionActorAlarmIndexTests : TestKit +{ + private readonly ISiteHealthCollector _health = Substitute.For(); + private readonly IDataConnectionFactory _factory = Substitute.For(); + private readonly DataConnectionOptions _options = new() + { + ReconnectInterval = TimeSpan.FromMilliseconds(100), + TagResolutionRetryInterval = TimeSpan.FromMilliseconds(200), + WriteTimeout = TimeSpan.FromSeconds(5) + }; + + // Sources chosen to exercise every index path: + // - "Area1" → NO separator: the residue list (can match other buckets) + // - "Area1.Tank" → bucket "Area1", shared prefix with the next one + // - "Area1.Tank1.Sub" → bucket "Area1", deeper + // - "Area2.Tank" → a different bucket + private static readonly string[] Sources = ["Area1", "Area1.Tank", "Area1.Tank1.Sub", "Area2.Tank"]; + + private static readonly (string SourceRef, string SourceObjectRef)[] Transitions = + [ + ("Area1.Tank1.Hi", "Area1.Tank1"), // Area1 (residue) + Area1.Tank + ("Area1.Tank1.Sub.Hi", "Area1.Tank1.Sub"), // + Area1.Tank1.Sub + ("Area1X.Pump.Hi", "Area1X.Pump"), // sub-segment prefix: Area1 ONLY + ("Area2.Tank9.Hi", "Area2.Tank9"), // Area2.Tank only + ("Zone.A.Hi", "Zone.A"), // nobody + ]; + + private static NativeAlarmTransition Transition(string sourceRef, string sourceObj) => + new(sourceRef, sourceObj, "AnalogLimit.Hi", AlarmTransitionKind.Raise, + new AlarmConditionState(true, false, null, AlarmShelveState.Unshelved, false, 500), + "Process", "hi", "hi", "", "", null, DateTimeOffset.UtcNow, "92", "90"); + + /// The ORIGINAL linear-scan rule, kept here as the oracle. + private static bool LinearScanMatches(string sourceRef, string transitionSourceRef, string transitionSourceObjectRef) => + transitionSourceObjectRef.StartsWith(sourceRef, StringComparison.Ordinal) + || transitionSourceRef.StartsWith(sourceRef, StringComparison.Ordinal); + + [Fact] + public void BucketIndex_RoutesExactlyLikeTheLinearScan() + { + AlarmTransitionCallback? cb = null; + var adapter = Substitute.For(); + adapter.ConnectAsync(Arg.Any>(), Arg.Any()) + .Returns(Task.CompletedTask); + ((IAlarmSubscribableConnection)adapter) + .SubscribeAlarmsAsync(Arg.Any(), Arg.Any(), + Arg.Do(c => cb = c), Arg.Any()) + .Returns(ci => Task.FromResult("alarm-" + ci.ArgAt(0))); + + var actor = Sys.ActorOf(Props.Create(() => new DataConnectionActor( + "conn", adapter, _options, _health, _factory, "OpcUa")), "alarm-index"); + + // One probe per source so the routing decision per source is observable. + var probes = Sources.ToDictionary(s => s, _ => CreateTestProbe()); + foreach (var source in Sources) + { + actor.Tell(new SubscribeAlarmsRequest("c", "inst-" + source, "conn", source, null, DateTimeOffset.UtcNow), + probes[source].Ref); + probes[source].ExpectMsg(m => m.Success, TimeSpan.FromSeconds(5)); + } + + Assert.NotNull(cb); + + foreach (var (sourceRef, sourceObjectRef) in Transitions) + { + cb!(Transition(sourceRef, sourceObjectRef)); + + foreach (var source in Sources) + { + if (LinearScanMatches(source, sourceRef, sourceObjectRef)) + { + probes[source].ExpectMsg( + u => u.Transition.SourceReference == sourceRef, TimeSpan.FromSeconds(5)); + } + } + } + + // Nothing extra was delivered to anybody. + foreach (var source in Sources) + probes[source].ExpectNoMsg(TimeSpan.FromMilliseconds(200)); + } + + [Fact] + public void SnapshotComplete_StillBroadcastsToEverySubscriber_BypassingTheIndex() + { + // The framing sentinel carries an EMPTY source reference, so it matches no bucket. + // It must still reach every alarm subscriber, or statically-active conditions + // delivered only in the snapshot would buffer forever. + AlarmTransitionCallback? cb = null; + var adapter = Substitute.For(); + adapter.ConnectAsync(Arg.Any>(), Arg.Any()) + .Returns(Task.CompletedTask); + ((IAlarmSubscribableConnection)adapter) + .SubscribeAlarmsAsync(Arg.Any(), Arg.Any(), + Arg.Do(c => cb = c), Arg.Any()) + .Returns(ci => Task.FromResult("alarm-" + ci.ArgAt(0))); + + var actor = Sys.ActorOf(Props.Create(() => new DataConnectionActor( + "conn", adapter, _options, _health, _factory, "OpcUa")), "alarm-index-snapshot"); + + var probes = Sources.ToDictionary(s => s, _ => CreateTestProbe()); + foreach (var source in Sources) + { + actor.Tell(new SubscribeAlarmsRequest("c", "inst-" + source, "conn", source, null, DateTimeOffset.UtcNow), + probes[source].Ref); + probes[source].ExpectMsg(m => m.Success, TimeSpan.FromSeconds(5)); + } + + Assert.NotNull(cb); + cb!(new NativeAlarmTransition( + "", "", "", AlarmTransitionKind.SnapshotComplete, + new AlarmConditionState(false, true, null, AlarmShelveState.Unshelved, false, 0), + "", "", "", "", "", null, DateTimeOffset.UtcNow, "", "")); + + foreach (var source in Sources) + { + probes[source].ExpectMsg( + u => u.Transition.Kind == AlarmTransitionKind.SnapshotComplete, TimeSpan.FromSeconds(5)); + } + } + + [Fact] + public void UnsubscribedSource_IsDroppedFromTheIndex() + { + AlarmTransitionCallback? cb = null; + var adapter = Substitute.For(); + adapter.ConnectAsync(Arg.Any>(), Arg.Any()) + .Returns(Task.CompletedTask); + ((IAlarmSubscribableConnection)adapter) + .SubscribeAlarmsAsync(Arg.Any(), Arg.Any(), + Arg.Do(c => cb = c), Arg.Any()) + .Returns(ci => Task.FromResult("alarm-" + ci.ArgAt(0))); + + var actor = Sys.ActorOf(Props.Create(() => new DataConnectionActor( + "conn", adapter, _options, _health, _factory, "OpcUa")), "alarm-index-unsub"); + + var probe = CreateTestProbe(); + actor.Tell(new SubscribeAlarmsRequest("c", "inst", "conn", "Area1.Tank", null, DateTimeOffset.UtcNow), probe.Ref); + probe.ExpectMsg(m => m.Success, TimeSpan.FromSeconds(5)); + + actor.Tell(new UnsubscribeAlarmsRequest("c2", "inst", "conn", "Area1.Tank", DateTimeOffset.UtcNow), probe.Ref); + Thread.Sleep(200); + + Assert.NotNull(cb); + cb!(Transition("Area1.Tank1.Hi", "Area1.Tank1")); + probe.ExpectNoMsg(TimeSpan.FromMilliseconds(300)); + } +} diff --git a/tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/OpcUaDataConnectionTests.cs b/tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/OpcUaDataConnectionTests.cs index cec163cb..af02990c 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/OpcUaDataConnectionTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/OpcUaDataConnectionTests.cs @@ -323,33 +323,41 @@ public class OpcUaDataConnectionTests [Fact] public async Task ReadBatch_ReadsAllTags() { + // WP2.1b: ReadBatchAsync is TRUE bulk — ONE client ReadValuesAsync call carrying + // every requested node, not a per-tag loop. _mockClient.IsConnected.Returns(true); - _mockClient.ReadValueAsync(Arg.Any(), Arg.Any()) - .Returns((1.0, DateTime.UtcNow, 0u)); + _mockClient.ReadValuesAsync(Arg.Any>(), Arg.Any()) + .Returns(ci => Task.FromResult>( + ci.Arg>() + .Select(n => new OpcUaReadOutcome(n, 1.0, DateTime.UtcNow, 0u, null)) + .ToList())); await _adapter.ConnectAsync(new Dictionary()); var results = await _adapter.ReadBatchAsync(["tag1", "tag2", "tag3"]); Assert.Equal(3, results.Count); Assert.All(results.Values, r => Assert.True(r.Success)); + Assert.Single(_mockClient.ReceivedCalls(), c => c.GetMethodInfo().Name == "ReadValuesAsync"); } [Fact] public async Task DCL007_ReadBatch_ReturnsPerTagResults_WhenOneTagFails() { - // Regression test for DataConnectionLayer-007. ReadBatchAsync looped calling - // ReadAsync per tag; ReadAsync re-throws any non-cancellation exception, so a - // single failing tag aborted the whole batch and the caller got NO results for - // the tags that did read successfully — even though ReadResult already carries - // a per-tag Success/ErrorMessage shape. After the fix the batch catches per-tag - // exceptions and returns a complete map. + // Regression test for DataConnectionLayer-007. ReadBatchAsync originally looped + // calling ReadAsync per tag; ReadAsync re-throws any non-cancellation exception, so + // a single failing tag aborted the whole batch and the caller got NO results for + // the tags that did read successfully — even though ReadResult already carries a + // per-tag Success/ErrorMessage shape. The batch is now one bulk service call, and + // the same invariant holds: a per-node failure row never aborts the batch, and + // every requested tag comes back in the map. _mockClient.IsConnected.Returns(true); - _mockClient.ReadValueAsync("good1", Arg.Any()) - .Returns((1.0, DateTime.UtcNow, 0u)); - _mockClient.ReadValueAsync("bad", Arg.Any()) - .Returns<(object?, DateTime, uint)>(_ => throw new InvalidOperationException("node not found")); - _mockClient.ReadValueAsync("good2", Arg.Any()) - .Returns((2.0, DateTime.UtcNow, 0u)); + _mockClient.ReadValuesAsync(Arg.Any>(), Arg.Any()) + .Returns(ci => Task.FromResult>( + ci.Arg>() + .Select(n => n == "bad" + ? new OpcUaReadOutcome(n, null, DateTime.UtcNow, 0x80340000u, "node not found") + : new OpcUaReadOutcome(n, 1.0, DateTime.UtcNow, 0u, null)) + .ToList())); await _adapter.ConnectAsync(new Dictionary()); @@ -365,28 +373,25 @@ public class OpcUaDataConnectionTests } [Fact] - public async Task DCL017_WriteBatch_ReturnsPerTagResults_WhenConnectionDropsMidBatch() + public async Task DCL017_WriteBatch_ReturnsPerTagResults_WhenSomeTagsFail() { - // Regression test for DataConnectionLayer-017. WriteBatchAsync looped calling - // WriteAsync per tag; WriteAsync first calls EnsureConnected(), which throws - // InvalidOperationException when the client is disconnected. WriteBatchAsync did - // not catch that, so a connection dropping partway through a batch made the whole - // WriteBatchAsync throw — the caller lost the per-tag outcomes for the tags that - // already wrote. After the fix (mirroring DCL-007's ReadBatchAsync) each per-tag - // failure is recorded as a failed WriteResult and the batch returns a complete map. - var writeCount = 0; - // First write succeeds; then the client "disconnects" so EnsureConnected throws. - _mockClient.IsConnected.Returns(_ => Interlocked.Increment(ref writeCount) <= 1); - _mockClient.WriteValueAsync(Arg.Any(), Arg.Any(), Arg.Any()) - .Returns((uint)0); - - // Connect leaves IsConnected true for the first WriteAsync's EnsureConnected check. + // Regression test for DataConnectionLayer-017. WriteBatchAsync originally looped + // calling WriteAsync per tag; a mid-batch fault made the whole call throw and the + // caller lost the per-tag outcomes for the tags that already wrote. The batch is + // now ONE bulk service call (WP2.1b), and the invariant is unchanged: per-node + // failures are reported as failed WriteResult rows and every requested tag is + // present in the returned map. _mockClient.IsConnected.Returns(true); await _adapter.ConnectAsync(new Dictionary()); - // Re-arm: IsConnected true for tag1's check, false for tag2 and tag3. - var checks = 0; - _mockClient.IsConnected.Returns(_ => Interlocked.Increment(ref checks) <= 1); + _mockClient.WriteValuesAsync( + Arg.Any>(), Arg.Any()) + .Returns(ci => Task.FromResult>( + ci.Arg>() + .Select(v => v.NodeId == "tag1" + ? new OpcUaWriteOutcome(v.NodeId, 0u, null) + : new OpcUaWriteOutcome(v.NodeId, 0x80AE0000u, null)) + .ToList())); var results = await _adapter.WriteBatchAsync(new Dictionary { @@ -398,11 +403,12 @@ public class OpcUaDataConnectionTests // Every requested tag is present in the result map — the batch was not aborted. Assert.Equal(3, results.Count); Assert.True(results["tag1"].Success); - // tag2 and tag3 fail at the connection check but are reported per-tag. Assert.False(results["tag2"].Success); Assert.NotNull(results["tag2"].ErrorMessage); Assert.False(results["tag3"].Success); Assert.NotNull(results["tag3"].ErrorMessage); + // ONE bulk write, not three single writes. + Assert.Single(_mockClient.ReceivedCalls(), c => c.GetMethodInfo().Name == "WriteValuesAsync"); } [Fact] @@ -415,8 +421,9 @@ public class OpcUaDataConnectionTests using var cts = new CancellationTokenSource(); cts.Cancel(); - _mockClient.WriteValueAsync(Arg.Any(), Arg.Any(), Arg.Any()) - .Returns(_ => throw new OperationCanceledException()); + _mockClient.WriteValuesAsync( + Arg.Any>(), Arg.Any()) + .Returns>(_ => throw new OperationCanceledException()); await Assert.ThrowsAnyAsync(() => _adapter.WriteBatchAsync(new Dictionary { ["tag1"] = 1 }, cts.Token)); From 48b3c40a7f37911bce3df5deaaf831e2cf7c4475 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Fri, 14 Aug 2026 21:14:22 -0400 Subject: [PATCH 15/31] perf(deploy): flatten-session caching, bulk DeploySiteAsync, paged management queries --- .../Component-DeploymentManager.md | 51 ++ .../Component-ManagementService.md | 11 +- docs/requirements/Component-ScriptAnalysis.md | 13 + docs/requirements/Component-TemplateEngine.md | 58 ++ .../Commands/DeployCommands.cs | 38 + src/ZB.MOM.WW.ScadaBridge.CLI/README.md | 31 + .../IDeploymentManagerRepository.cs | 42 ++ .../Repositories/ITemplateEngineRepository.cs | 35 + .../Services/ITemplateGraphWatermark.cs | 82 +++ .../Messages/Management/DeploymentCommands.cs | 21 + .../Deployment/DeploymentRecordSummary.cs | 54 ++ .../Types/Templates/TemplateGraphWatermark.cs | 72 ++ .../Types/Templates/TemplateSummary.cs | 53 ++ .../DeploymentManagerRepository.cs | 83 +++ .../Repositories/TemplateEngineRepository.cs | 226 +++++- .../ServiceCollectionExtensions.cs | 9 + .../DeploymentManagerOptions.cs | 39 ++ .../DeploymentManagerOptionsValidator.cs | 15 + .../DeploymentService.cs | 654 ++++++++++++++++-- .../FlattenSession.cs | 257 +++++++ .../FlatteningPipeline.cs | 86 ++- .../IFlatteningPipeline.cs | 20 +- .../ServiceCollectionExtensions.cs | 8 + .../StaleInstanceProbe.cs | 111 ++- .../ManagementActor.cs | 158 ++++- .../ManagementEndpoints.cs | 4 + .../TemplateService.cs | 55 +- .../Validation/ScriptCompileVerdictCache.cs | 105 ++- .../CommandTreeTests.cs | 17 + .../SchemaConfigurationTests.cs | 3 +- .../TemplateEngineRepositoryTests.cs | 3 +- .../DeploySiteAsyncTests.cs | 279 ++++++++ .../DeploymentComparisonTests.cs | 4 +- .../DeploymentServiceTests.cs | 13 +- .../FlattenSessionCacheTests.cs | 153 ++++ ...latteningPipelineConnectionBindingTests.cs | 4 +- ...eningPipelineNativeAlarmCapabilityTests.cs | 4 +- .../ManagementActorTests.cs | 138 ++-- .../RequiredRoleMatrixTests.cs | 3 + .../TemplateServiceTests.cs | 97 +++ .../ScriptCompileVerdictCacheEvictionTests.cs | 135 ++++ .../Validation/ScriptCompilerTests.cs | 4 + .../CompositionImportTests.cs | 2 + .../ConflictResolutionTests.cs | 2 + .../Export/BundleExporterTests.cs | 2 + .../Import/AreaTransportTests.cs | 2 + .../Import/BundleImporterApplyTests.cs | 2 + .../Import/BundleImporterPreviewTests.cs | 2 + .../BundleImporterRetryingStrategyTests.cs | 2 + .../BundleImporterRollbackFailureTests.cs | 2 + .../CreateMissingSiteRelationalTests.cs | 2 + .../Import/InheritanceImportTests.cs | 2 + .../Import/NativeAlarmSourceImportTests.cs | 2 + .../Import/SiteInstanceImportTests.cs | 2 + .../Import/TemplateScriptFidelityTests.cs | 2 + .../RoundTripEquivalenceTests.cs | 2 + .../RoundTripTests.cs | 2 + .../SemanticValidatorImportTests.cs | 2 + .../ValidationFailureTests.cs | 2 + 59 files changed, 3051 insertions(+), 231 deletions(-) create mode 100644 src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Services/ITemplateGraphWatermark.cs create mode 100644 src/ZB.MOM.WW.ScadaBridge.Commons/Types/Deployment/DeploymentRecordSummary.cs create mode 100644 src/ZB.MOM.WW.ScadaBridge.Commons/Types/Templates/TemplateGraphWatermark.cs create mode 100644 src/ZB.MOM.WW.ScadaBridge.Commons/Types/Templates/TemplateSummary.cs create mode 100644 src/ZB.MOM.WW.ScadaBridge.DeploymentManager/FlattenSession.cs create mode 100644 tests/ZB.MOM.WW.ScadaBridge.DeploymentManager.Tests/DeploySiteAsyncTests.cs create mode 100644 tests/ZB.MOM.WW.ScadaBridge.DeploymentManager.Tests/FlattenSessionCacheTests.cs create mode 100644 tests/ZB.MOM.WW.ScadaBridge.TemplateEngine.Tests/Validation/ScriptCompileVerdictCacheEvictionTests.cs diff --git a/docs/requirements/Component-DeploymentManager.md b/docs/requirements/Component-DeploymentManager.md index 2f90d2ed..406dadca 100644 --- a/docs/requirements/Component-DeploymentManager.md +++ b/docs/requirements/Component-DeploymentManager.md @@ -137,6 +137,57 @@ When deploying artifacts (shared scripts, external system definitions, etc.) to - Deployment is performed at the **individual instance level**. - The UI may provide convenience operations (e.g., "deploy all out-of-date instances at Site A"), but these decompose into individual instance deployments. +### Bulk site deployment (`DeploySiteAsync`, WP2.5) + +`DeploySiteAsync(siteId, user)` deploys every instance at one site in a single +operation, surfaced as the `MgmtDeploySite` management command and the CLI +`deploy site --site-id`. It is the "deploy all at Site A" convenience above, made +first-class — it still decomposes into individual instance deployments, and each +instance keeps its own deployment id, revision hash, operation lock, and +optimistically-concurrent status record. + +It runs the ordinary deployment pipeline in three phases, of which only the middle +one is parallel: + +1. **Prepare (serial).** Validate transition, take the operation lock, flatten + + validate, run query-before-redeploy reconciliation, stage the + `PendingDeployment`, insert the `InProgress` record. Every step here touches the + scoped, non-thread-safe `DbContext`, so the phase is strictly serial. All + instances share ONE `FlattenSession`, so a template chain common to N instances + is walked once and the session-global queries (shared scripts, schema library, + the site's data connections) run once for the batch. +2. **Send (bounded parallel).** The `RefreshDeploymentCommand` round-trips run + concurrently up to `SiteDeploymentMaxParallelism` (default 4), each under a + `SiteDeploymentTimeoutPerInstance` deadline (default 120 s). This phase touches + no repository — that is exactly why it is the only phase allowed to run in + parallel. Shape mirrors `ArtifactDeploymentService.DeployCoreAsync`. +3. **Finalize (serial).** Commit terminal statuses, apply post-success side + effects, write audit rows, release each operation lock. + +**Not all-or-nothing.** An instance that fails for any reason (wrong state, failed +validation, lock already held by another operation, site round-trip timed out) is +reported as a failed row while the rest proceed, and is individually retryable via +the ordinary single-instance deploy. This matches the artifact-deployment policy: +successful targets are never rolled back because another target failed. + +`DeployInstanceAsync` is composed from the same three phase helpers with a batch of +one, so the two entry points cannot drift on deployment identity, idempotency, lock +coverage, or optimistic concurrency. + +### Terminal deployment-record retention + +Deployment records are insert-only — one row per deploy attempt — so without a +window the table grows for the life of the system and every unfiltered deployment +query degrades with age rather than with page size. +`TerminalDeploymentRecordRetention` (default **365 days**, deliberately generous +because deployment history is operator forensics) bounds it, swept opportunistically +and rate-limited on the deployment-list read path. + +Only **terminal** rows (`Success` / `Failed`) are eligible. An `InProgress` row is +never purged regardless of age: it is precisely the row the query-before-redeploy +reconciliation reads to decide whether a prior deploy actually landed at the site, +and expiring it by age would silently disable that idempotency guard. + ## Diff View Before deploying, the Deployment Manager can request a diff from the Template Engine showing: diff --git a/docs/requirements/Component-ManagementService.md b/docs/requirements/Component-ManagementService.md index 20c7219a..ac5fa61b 100644 --- a/docs/requirements/Component-ManagementService.md +++ b/docs/requirements/Component-ManagementService.md @@ -87,7 +87,7 @@ Both endpoints honour any site-scope rules attached to the caller's audit role b ### Templates -- **ListTemplates** / **GetTemplate**: Query template definitions. `ListTemplates` supports additive offset paging via `Skip` / `Take` (arch-review P2): `Take = null` (the default) preserves the historical unpaged behaviour (unlimited), a set `Take` is clamped to `1..1000` and `Skip` floors at `0`. +- **ListTemplates** / **GetTemplate**: Query template definitions. `ListTemplates` supports additive offset paging via `Skip` / `Take` (arch-review P2): `Take = null` (the default) preserves the historical unpaged behaviour (unlimited), a set `Take` is clamped to `1..1000` and `Skip` floors at `0`. **Paging and projection are DB-side (WP2.5):** the handler reads `TemplateSummary` rows (child collections reduced to counts, no script bodies) via `GetTemplateSummariesAsync`, instead of materialising every template's full five-Include child graph and then discarding all but one page in memory. A caller that needs a template's members already fetches it with `GetTemplate`. - **CreateTemplate** / **UpdateTemplate** / **DeleteTemplate**: Manage templates. - **ValidateTemplate**: Run on-demand pre-deployment validation (flattening, naming collisions, script compilation). - **GetTemplateDiff**: Compare deployed vs. template-derived configuration for an instance. @@ -115,9 +115,10 @@ The whole folder surface is reachable from the CLI as `template folder list|crea ### Instances -- **ListInstances** / **GetInstance**: Query instances, with filtering by site and area. `ListInstances` supports additive offset paging via `Skip` / `Take` (arch-review P2), applied **after** the in-memory site-scope filter so a site-scoped user never sees an out-of-scope instance surface into their page window; same `Take = null` unlimited default and `1..1000` clamp as `ListTemplates`. (`QueryDeployments` and `ExportBundle` still load the full table internally — a logged scale follow-up, arch-review P2 deferred.) +- **ListInstances** / **GetInstance**: Query instances, with filtering by site and area. `ListInstances` supports additive offset paging via `Skip` / `Take` (arch-review P2), applied **after** the in-memory site-scope filter so a site-scoped user never sees an out-of-scope instance surface into their page window; same `Take = null` unlimited default and `1..1000` clamp as `ListTemplates`. (`ExportBundle` still loads the full table internally — a logged scale follow-up, arch-review P2 deferred. `QueryDeployments` was closed by WP2.5, below.) - **CreateInstance**: Create a new instance from a template. -- **UpdateInstanceOverrides**: Set attribute overrides on an instance. +- **UpdateInstanceOverrides** (`SetInstanceOverrides`): Set attribute overrides on an instance. Every requested attribute is validated against the instance's template up front (unknown or locked ⇒ the whole batch is rejected before any write), then the batch is applied as **one bulk read of the existing override rows plus one commit** (WP2.5). It previously ran K independent read-modify-commit cycles — each re-reading the instance, the template's attributes and the existing overrides, then committing and writing its own audit row — so the all-or-nothing promise was only as good as the pre-validation. It is now a real single-transaction apply, with one audit row summarising the batch. +- **MgmtDeploySite** (`SiteId`, WP2.5): Bulk-deploy every deployable instance at one site. Site scope is enforced on the SITE (the command's target); every instance the batch touches belongs to that site by construction. Same `Deployer` authority as `MgmtDeployInstance` — it is N of those, not a new class of privilege — and registered as a long-running command so the caller does not time out mid-batch. Semantics, phasing and failure policy: see `Component-DeploymentManager.md` → Bulk site deployment. - **SetInstanceAlarmOverride** / **DeleteInstanceAlarmOverride** / **ListInstanceAlarmOverrides**: Manage per-instance computed-alarm overrides. - **SetInstanceNativeAlarmSourceOverride** / **DeleteInstanceNativeAlarmSourceOverride**: Retarget or clear a per-instance native alarm source binding, keyed by `SourceCanonicalName` — `ConnectionNameOverride` / `SourceReferenceOverride` / `ConditionFilterOverride` each apply only when non-null. Gated to the **Deployment** role. - **ListInstanceNativeAlarmSourceOverrides** (`InstanceId`): List an instance's native alarm source overrides (read-only). @@ -144,7 +145,9 @@ The whole folder surface is reachable from the CLI as `template folder list|crea - **DeployInstance**: Deploy configuration to a specific instance (includes pre-deployment validation). - **DeployArtifacts**: Deploy system-wide artifacts (shared scripts, external system definitions, DB connections, data connections) to all sites or a specific site. The command's `SiteId` is honored (arch-review C2): a value routes to the single-site deploy path (`ArtifactDeploymentService.DeployToSiteAsync`), while `null` deploys fleet-wide (`DeployToAllSitesAsync`). Site scope is enforced — a site-scoped (non-Administrator) Deployer may only target a site within its `PermittedSiteIds`, and may **not** deploy fleet-wide (`SiteId is null` is rejected with `SiteScopeViolationException` → `ManagementUnauthorized`, since `EnforceSiteScope(null)` is a deliberate no-op); fleet-wide deployment requires a system-wide Deployer or Administrator. +- **DeploySite** (`MgmtDeploySite`, WP2.5): Bulk-deploy every deployable instance at one site — see the Instances section above and `Component-DeploymentManager.md`. - **GetDeploymentStatus**: Query deployment status. +- **QueryDeployments** (`InstanceId?`, `Status?`, `Page`, `PageSize`): List deployment records. **All four arguments are honoured DB-side (WP2.5)** and the result is a `DeploymentRecordSummary` projection. The handler previously ignored `Status`, `Page` and `PageSize` entirely (the CLI had been sending them all along) and loaded the whole insert-only `DeploymentRecords` table; for a site-scoped user it additionally loaded every instance and intersected the two sets in memory. Site scope is now resolved to the set of in-scope instance ids and pushed into the query as an id filter — an **empty** scope stays a real filter (a user permitted no in-scope instances sees nothing), never a no-op. This read path also drives the opportunistic, rate-limited terminal-record retention sweep described in `Component-DeploymentManager.md`. ### Secured Writes (MxGateway, two-person) @@ -259,7 +262,7 @@ The ManagementActor receives the following services and repositories via DI (inj | Section | Options Class | Contents | |---------|--------------|----------| | `ScadaBridge:ManagementService` | `ManagementServiceOptions` | `CommandTimeout` (`TimeSpan`, default 30 s) — Ask timeout the HTTP endpoint applies when forwarding to the `ManagementActor`. A non-positive configured value falls back to the 30 s default. | -| | | `LongRunningCommandTimeout` (`TimeSpan`, default 5 min) — Ask timeout applied to long-running commands (`ImportBundle`, `PreviewBundle`, `ExportBundle`, `MgmtDeployArtifacts`, `MgmtDeployInstance`); all other commands use `CommandTimeout`. A non-positive configured value falls back to the 5 min default. | +| | | `LongRunningCommandTimeout` (`TimeSpan`, default 5 min) — Ask timeout applied to long-running commands (`ImportBundle`, `PreviewBundle`, `ExportBundle`, `MgmtDeployArtifacts`, `MgmtDeployInstance`, `MgmtDeploySite`); all other commands use `CommandTimeout`. A non-positive configured value falls back to the 5 min default. | | | | `SecuredWritePendingTtl` (`TimeSpan`, default 24 h) — age after which a `Pending` secured write is transitioned to `Expired` and can no longer be approved/executed; enforced at approve/reject and swept opportunistically on list. A non-positive value disables expiry (arch-review S2). | ## Dependencies diff --git a/docs/requirements/Component-ScriptAnalysis.md b/docs/requirements/Component-ScriptAnalysis.md index 86319200..44fea050 100644 --- a/docs/requirements/Component-ScriptAnalysis.md +++ b/docs/requirements/Component-ScriptAnalysis.md @@ -158,6 +158,19 @@ Mirrors `TriggerExpressionGlobals` in the same way. Used by `ValidationService.C `CheckExpressionSyntax` memoises its verdict in the Template Engine's process-wide `ScriptCompileVerdictCache`, whose key is the pair **(globals surface, SHA-256 of the code)** — not the code alone. The surface discriminator is load-bearing: a trigger expression vetted against `TriggerCompileSurface` is **not** interchangeable with a `ScriptCompileSurface` script-body verdict (different globals resolve different identifiers), so a code-only key could return a stale "clean" for code never compiled against the caller's surface. Keying by surface makes that cross-surface verdict reuse structurally impossible. +**Eviction is segmented, never wholesale (WP2.5).** The cache is bounded at two +generations of 2048 entries. New verdicts land in the *hot* generation; a hit in +*cold* is promoted back into hot; on overflow hot becomes the new cold and only the +old cold — the half nothing has touched for a full generation — is dropped. + +The bound must never be enforced by clearing the cache outright. That reads as +harmless ("a verdict is cheap to recompute") but is not: a recompute is a fresh +Roslyn compile, and every script compile loads an assembly through a +**non-collectible `InteractiveAssemblyLoader`**. Bounding that leak is the entire +reason this cache exists, so dropping every hot entry at the 4096th distinct script +would re-open it for the whole working set at once. Segmenting retains anything in +active use across an eviction while keeping the same ceiling. + #### Parity guard A reflection-based parity test in `SiteRuntime.Tests` compares the public member names on `ScriptCompileSurface` against `ScriptGlobals` (and `TriggerCompileSurface` against `TriggerExpressionGlobals`). Any drift between the stub and the real globals causes this test to fail, ensuring the stubs cannot silently fall out of sync. diff --git a/docs/requirements/Component-TemplateEngine.md b/docs/requirements/Component-TemplateEngine.md index 9b7abf88..875a4c1a 100644 --- a/docs/requirements/Component-TemplateEngine.md +++ b/docs/requirements/Component-TemplateEngine.md @@ -137,6 +137,64 @@ When an instance is deployed, the Template Engine resolves the full configuratio 5. Resolve data connection bindings — replace connection name references with concrete connection details from the site. 6. Output a flat structure: list of attributes with resolved values and data source addresses, list of alarms with resolved trigger definitions, list of scripts with resolved code and triggers. +### Flatten-session caching and the graph watermark (WP2.5) + +The flatten above is driven per instance, so an unmemoised implementation re-walks +the same template chain — one query per link, plus the compositions of every +template it reaches and of every composed chain those reach — once per instance, +and re-issues the three session-global queries (shared scripts, the shared-schema +library, the target site's data connections) each time too. + +A **`FlattenSession`** memoises all of that for the lifetime of ONE +flatten/validate operation. Callers that flatten a batch (`DeploySiteAsync`) pass +one shared session; a caller that passes none gets a private single-use session, +which still collapses the repeated composed-chain loads inside a single instance's +flatten. + +Cache validity is decided by **`ITemplateGraphWatermark`**, a process-wide set of +monotonic counters bumped by the configuration-database unit of work — the only +place every template-graph writer funnels through (`TemplateService`, the +`ManagementActor` native-alarm-source handlers, and the Transport bundle importer +all commit via the same `SaveChangesAsync`, and the change tracker is inspected +pre-commit to attribute each change to its owning template or instance): + +- a memoised **template** is keyed on `(id, template version)`; +- a memoised **chain** is valid only while BOTH the graph's `StructureVersion` + (bumped on add/remove, re-parent, or any composition change — i.e. anything that + can alter chain MEMBERSHIP) and every member's own version are unchanged. Both + halves are load-bearing: structure alone misses an ordinary member edit, member + versions alone miss a re-parent. + +Sessions are short-lived by construction, so the design's "template state is +captured at the time of flatten" guarantee is unchanged — the session narrows the +capture window, it never widens it. + +The watermark is **in-memory and process-local**, deliberately not persisted and +not replicated between central nodes. A restart, a failover, or any mutation simply +misses and falls back to the authoritative full flatten, so the watermark can only +ever cause EXTRA work, never stale work. + +The same watermark backs a **staleness fast path** in `StaleInstanceProbe`: a +previously computed revision hash is reused when the instance's version, the +structure version, and the version of every template that flatten walked are all +unchanged. This matters most where the probe is called per-instance across a whole +bundle import or a fleet-wide staleness sweep. + +### Design-time analysis reads + +The design-time checks — acyclicity (`CycleDetector`), naming collisions +(`CollisionDetector`) and canonical-name resolution (`TemplateResolver`) — read the +whole template graph but write through none of it and read no script bodies. They +use `GetAllTemplatesForAnalysisAsync`: `AsNoTracking`, with `TemplateScript.Code` +projected away. + +The tracked, body-bearing `GetAllTemplatesAsync` remains the read for the two paths +that genuinely need it — the inheritance reconciler (which compares and copies +script bodies and writes through the loaded entities) and the flattener (which must +observe rows an in-flight bundle import has staged on the shared change tracker). +`ReconcileDescendantsAsync` additionally accepts an already-loaded tracked graph so +a caller holding one does not force a second load. + ### Native Alarm Source Resolution The `FlatteningService` resolves native alarm sources alongside alarms, emitting a `ResolvedNativeAlarmSource` (CanonicalName, ConnectionName, SourceReference, ConditionFilter *(optional)*, and `Source` ∈ `Template` | `Inherited` | `Composed` | `Override`) for each. The resolved set is attached to `FlattenedConfiguration.NativeAlarmSources`. diff --git a/src/ZB.MOM.WW.ScadaBridge.CLI/Commands/DeployCommands.cs b/src/ZB.MOM.WW.ScadaBridge.CLI/Commands/DeployCommands.cs index f014f784..08b43a3d 100644 --- a/src/ZB.MOM.WW.ScadaBridge.CLI/Commands/DeployCommands.cs +++ b/src/ZB.MOM.WW.ScadaBridge.CLI/Commands/DeployCommands.cs @@ -19,6 +19,7 @@ public static class DeployCommands var command = new Command("deploy") { Description = "Deployment operations" }; command.Add(BuildInstance(urlOption, formatOption, usernameOption, passwordOption)); + command.Add(BuildSite(urlOption, formatOption, usernameOption, passwordOption)); command.Add(BuildArtifacts(urlOption, formatOption, usernameOption, passwordOption)); command.Add(BuildStatus(urlOption, formatOption, usernameOption, passwordOption)); @@ -39,6 +40,43 @@ public static class DeployCommands return cmd; } + /// + /// Builds deploy site — bulk-deploy every instance at one site. + /// + /// + /// Sits under deploy alongside deploy instance and + /// deploy artifacts, naming the SCOPE of the deploy as the verb, which + /// is the convention the group already follows. Note it is deliberately NOT + /// fleet-wide when --site-id is omitted (the pattern deploy + /// artifacts uses): a bulk instance deploy is far more consequential than + /// an artifact push, so the target site is required rather than defaulted. + /// + /// + private static Command BuildSite(Option urlOption, Option formatOption, Option usernameOption, Option passwordOption) + { + var siteIdOption = new Option("--site-id") { Description = "Target site ID", Required = true }; + var cmd = new Command("site") { Description = "Deploy every deployable instance at a site" }; + cmd.Add(siteIdOption); + cmd.SetAction(async (ParseResult result) => + { + var siteId = result.GetValue(siteIdOption); + return await CommandHelpers.ExecuteCommandAsync( + result, urlOption, formatOption, usernameOption, passwordOption, + new MgmtDeploySiteCommand(siteId), + // A bulk deploy is N instance round-trips; the default 30 s client + // timeout would abandon the request while the server is still + // applying. Matches the server-side long-running Ask window. + timeout: BulkDeployTimeout); + }); + return cmd; + } + + /// + /// Client-side timeout for the bulk site deploy, matching the management + /// service's long-running command Ask window. + /// + internal static readonly TimeSpan BulkDeployTimeout = TimeSpan.FromMinutes(5); + private static Command BuildArtifacts(Option urlOption, Option formatOption, Option usernameOption, Option passwordOption) { var siteIdOption = new Option("--site-id") { Description = "Target site ID (all sites if omitted)" }; diff --git a/src/ZB.MOM.WW.ScadaBridge.CLI/README.md b/src/ZB.MOM.WW.ScadaBridge.CLI/README.md index 28c7b912..e737dd63 100644 --- a/src/ZB.MOM.WW.ScadaBridge.CLI/README.md +++ b/src/ZB.MOM.WW.ScadaBridge.CLI/README.md @@ -952,6 +952,37 @@ scadabridge --url deploy instance --id |--------|----------|-------------| | `--id` | yes | Instance ID | +#### `deploy site` + +Deploy **every deployable instance** at one site in a single batched operation. + +Each instance keeps the full single-instance semantics — its own deployment ID and +revision hash, its own per-instance operation lock, its own optimistically-concurrent +status record, and the same query-before-redeploy idempotency check. The batch is +**not** all-or-nothing: an instance that cannot be deployed (wrong state, failed +validation, lock already held, site round-trip timed out) is reported as a failed row +and the rest proceed. Each failed instance is individually retryable with +`instance deploy --id`. + +Site round-trips run with bounded concurrency +(`ScadaBridge:DeploymentManager:SiteDeploymentMaxParallelism`, default 4), each under +`SiteDeploymentTimeoutPerInstance` (default 120 s), so one wedged instance cannot stall +the batch. The client waits up to 5 minutes. + +`--site-id` is **required** — unlike `deploy artifacts`, omitting it does not mean +fleet-wide. + +```sh +scadabridge --url deploy site --site-id +``` + +| Option | Required | Description | +|--------|----------|-------------| +| `--site-id` | yes | Target site ID. A site-scoped (non-Administrator) Deployer must supply an in-scope site. | + +Output is a per-instance result matrix (`instanceId`, `uniqueName`, `deploymentId`, +`success`, `errorMessage`) plus `successCount` / `failureCount`. + #### `deploy artifacts` Deploy compiled artifacts to one or all sites (same as `site deploy-artifacts`). diff --git a/src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/IDeploymentManagerRepository.cs b/src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/IDeploymentManagerRepository.cs index 01b55d60..b7c59ced 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/IDeploymentManagerRepository.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/IDeploymentManagerRepository.cs @@ -22,6 +22,48 @@ public interface IDeploymentManagerRepository /// A read-only list of all deployment records. Task> GetAllDeploymentRecordsAsync(CancellationToken cancellationToken = default); /// + /// Database-side paged and filtered deployment query returning row summaries. + /// Backs the QueryDeployments management command, which previously + /// loaded the ENTIRE deployment-record table (insert-only, one row per deploy + /// attempt for the life of the system) and ignored the page arguments the CLI + /// was already sending. + /// + /// Restrict to one instance's history, or for all. + /// Restrict to one deployment status, or for all. + /// + /// When non-null, restrict to these instance ids. Used to apply a site-scoped + /// user's permitted-site filter IN THE DATABASE rather than by loading every + /// record and every instance and filtering in memory. + /// + /// Rows to skip; negative values are floored at 0. + /// Page size; non-positive or null returns every remaining row. + /// A cancellation token that can be used to cancel the operation. + /// One page of deployment summaries, newest first. + Task> QueryDeploymentSummariesAsync( + int? instanceId, + DeploymentStatus? status, + IReadOnlyCollection? instanceIdScope, + int skip, + int? take, + CancellationToken cancellationToken = default); + /// + /// Deletes TERMINAL deployment records ( / + /// ) that completed before + /// , in one bounded batch. + /// + /// + /// Non-terminal rows are never touched: an InProgress record is what the + /// query-before-redeploy reconciliation path reads to decide whether a prior + /// deploy actually landed at the site, so purging one by age would silently + /// disable that idempotency guard. + /// + /// + /// Records completing before this instant are eligible. + /// Maximum rows deleted in this call. + /// A cancellation token that can be used to cancel the operation. + /// The number of rows deleted; fewer than means the sweep is complete. + Task PurgeTerminalDeploymentRecordsAsync(DateTimeOffset cutoffUtc, int batchSize, CancellationToken cancellationToken = default); + /// /// Gets all deployment records for a specific instance. /// /// The instance ID. diff --git a/src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/ITemplateEngineRepository.cs b/src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/ITemplateEngineRepository.cs index 3cc1d1d2..dc2d294f 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/ITemplateEngineRepository.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/ITemplateEngineRepository.cs @@ -35,6 +35,41 @@ public interface ITemplateEngineRepository /// A task that resolves to a read-only list of all templates. Task> GetAllTemplatesAsync(CancellationToken cancellationToken = default); /// + /// Read-only, no-tracking variant of for the + /// design-time ANALYSIS walks — acyclicity (CycleDetector), naming + /// collisions (CollisionDetector) and canonical-name resolution + /// (TemplateResolver). Script bodies are NOT loaded + /// ( comes back empty) because no analysis + /// consumer reads them; everything those walks do read — ids, parent and + /// composition edges, member names and lock flags — is present. + /// + /// + /// Do NOT use this for the inheritance reconciler or the flattener: the + /// reconciler compares and copies script bodies and writes through tracked + /// entities, and the flattener must observe rows an in-flight bundle import + /// has staged on the shared change tracker. Both keep using + /// / . + /// + /// + /// Cancellation token. + /// A task that resolves to a read-only list of all templates without script bodies. + Task> GetAllTemplatesForAnalysisAsync(CancellationToken cancellationToken = default); + /// + /// Database-side paged listing of templates as row summaries (child + /// collections reduced to counts, no script bodies). Backs the + /// ListTemplates management command, which previously materialised + /// every template's full child graph and then paged the list in memory. + /// + /// Number of rows to skip; negative values are floored at 0. + /// + /// Page size. or non-positive returns every remaining + /// row (the historical unpaged behaviour); otherwise clamped to + /// . + /// + /// Cancellation token. + /// A task that resolves to one page of template summaries, ordered by id. + Task> GetTemplateSummariesAsync(int skip, int? take, CancellationToken cancellationToken = default); + /// /// Returns every template that contains a composition referencing /// . Each result is eager-loaded with /// its Attributes / Scripts / Compositions so the caller can build a diff --git a/src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Services/ITemplateGraphWatermark.cs b/src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Services/ITemplateGraphWatermark.cs new file mode 100644 index 00000000..2d2952d5 --- /dev/null +++ b/src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Services/ITemplateGraphWatermark.cs @@ -0,0 +1,82 @@ +namespace ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services; + +/// +/// Process-wide monotonic version watermark over the template/instance +/// configuration graph. +/// +/// +/// Every mutation that lands through the configuration database bumps the +/// version of the affected template (or instance), plus a global counter and — +/// for edge-shaped changes (template added/removed, parent or composition edge +/// changed) — a separate . Consumers use the +/// watermark two ways: +/// +/// +/// +/// as the discriminator in a flatten-session cache key +/// (templateId + version), so a memoised template chain can never be +/// served after that template changed; and +/// +/// +/// as a staleness fast path — when neither the instance nor any template in +/// its chain has moved since a previously computed revision hash, the hash +/// is still current and a full re-flatten can be skipped. +/// +/// +/// +/// +/// Versions are in-memory and process-local. They are deliberately NOT +/// persisted: a restart (or the standby central node) simply starts from zero, +/// which invalidates every cached entry and falls back to the authoritative +/// full flatten. The watermark is therefore only ever able to cause EXTRA work, +/// never stale work — a correctness property that matters because the value is +/// not replicated between the two central nodes. +/// +/// +public interface ITemplateGraphWatermark +{ + /// + /// Monotonic counter bumped on every template- or instance-graph mutation. + /// Useful as a coarse "did anything change at all" gate. + /// + long Global { get; } + + /// + /// Monotonic counter bumped only when the SHAPE of the graph changes — a + /// template row added or removed, a parent-template edge changed, or a + /// composition row added/removed/repointed. A cached template-chain + /// MEMBERSHIP is valid only while this value is unchanged; the per-template + /// versions then cover the contents of each member. + /// + long StructureVersion { get; } + + /// Current version of a single template. Zero when never mutated in this process. + /// Template id to read. + /// The template's monotonic version. + long GetTemplateVersion(int templateId); + + /// Current version of a single instance (its override/binding rows). Zero when never mutated in this process. + /// Instance id to read. + /// The instance's monotonic version. + long GetInstanceVersion(int instanceId); + + /// Records a mutation of the given template. + /// Template whose version is bumped. + /// + /// When true the change altered the graph shape (add/remove/parent or + /// composition edge) and is bumped too. + /// + void BumpTemplate(int templateId, bool structural = false); + + /// Records a mutation of the given instance. + /// Instance whose version is bumped. + void BumpInstance(int instanceId); + + /// + /// Records a mutation whose affected template/instance could not be + /// attributed to a specific id. Bumps and + /// , invalidating every chain-membership and + /// staleness fast path — the safe, conservative fallback. + /// + void BumpAll(); +} diff --git a/src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Management/DeploymentCommands.cs b/src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Management/DeploymentCommands.cs index 3896a455..e193e292 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Management/DeploymentCommands.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Management/DeploymentCommands.cs @@ -1,5 +1,26 @@ namespace ZB.MOM.WW.ScadaBridge.Commons.Messages.Management; public record MgmtDeployArtifactsCommand(int? SiteId = null); + +/// +/// Deploys every deployable instance at one site in a single operation. +/// +/// +/// Distinct from , which ships the +/// SYSTEM-WIDE artifact sets (shared scripts, external systems, DB/data +/// connections) to a site. This command deploys the site's INSTANCE +/// configurations — the same per-instance pipeline +/// MgmtDeployInstanceCommand runs, batched. +/// +/// +/// +/// Not all-or-nothing: each instance keeps its own deployment id, revision hash, +/// operation lock and status record, and a failure on one is reported as a failed +/// row while the rest proceed. Individually retryable via the single-instance +/// deploy. Additive contract; new fields go on the end with a default. +/// +/// +/// Database id of the target site. Required — this command is never fleet-wide. +public record MgmtDeploySiteCommand(int SiteId); public record QueryDeploymentsCommand(int? InstanceId = null, string? Status = null, int Page = 1, int PageSize = 50); public record GetDeploymentDiffCommand(int InstanceId); diff --git a/src/ZB.MOM.WW.ScadaBridge.Commons/Types/Deployment/DeploymentRecordSummary.cs b/src/ZB.MOM.WW.ScadaBridge.Commons/Types/Deployment/DeploymentRecordSummary.cs new file mode 100644 index 00000000..804ba256 --- /dev/null +++ b/src/ZB.MOM.WW.ScadaBridge.Commons/Types/Deployment/DeploymentRecordSummary.cs @@ -0,0 +1,54 @@ +using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; + +namespace ZB.MOM.WW.ScadaBridge.Commons.Types.Deployment; + +/// +/// Row-shaped projection of a DeploymentRecord for list surfaces (CLI +/// deploy list / instance history, the Central UI deployment-status +/// page). +/// +/// +/// Exists so a deployment listing can be paged and filtered in the DATABASE. +/// Deployment records are insert-only — one row per deploy attempt, retained for +/// the configured retention window — so the previous "load the whole table, then +/// filter in memory" read path degraded with the age of the system rather than +/// with the size of the requested page. +/// +/// +/// +/// The RowVersion optimistic-concurrency token is deliberately absent: this +/// is a READ projection, and a summary must never be mistaken for something that +/// can be written back. Mutating paths keep loading the tracked entity. +/// +/// +/// +/// Message-contract evolution rule: additive-only. New fields go on the end with a +/// default. +/// +/// +/// Deployment record row id. +/// The logical deployment id (GUID, "N" format). +/// Instance the deployment targeted. +/// Terminal or in-flight deployment status. +/// Revision hash of the deployed configuration. +/// User who initiated the deployment. +/// When the deployment was initiated. +/// When the deployment reached a terminal status, if it has. +/// Failure detail when the deployment did not succeed. +public record DeploymentRecordSummary( + int Id, + string DeploymentId, + int InstanceId, + DeploymentStatus Status, + string? RevisionHash, + string DeployedBy, + DateTimeOffset DeployedAt, + DateTimeOffset? CompletedAt, + string? ErrorMessage) +{ + /// + /// Hard ceiling applied to a requested page size, mirroring the in-memory + /// Page clamp the management actor already applies elsewhere. + /// + public const int MaxPageSize = 1000; +} diff --git a/src/ZB.MOM.WW.ScadaBridge.Commons/Types/Templates/TemplateGraphWatermark.cs b/src/ZB.MOM.WW.ScadaBridge.Commons/Types/Templates/TemplateGraphWatermark.cs new file mode 100644 index 00000000..0aced5b1 --- /dev/null +++ b/src/ZB.MOM.WW.ScadaBridge.Commons/Types/Templates/TemplateGraphWatermark.cs @@ -0,0 +1,72 @@ +using System.Collections.Concurrent; +using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services; + +namespace ZB.MOM.WW.ScadaBridge.Commons.Types.Templates; + +/// +/// In-memory . Registered as a singleton so +/// the writer (the configuration-database repository's unit-of-work commit) and +/// the readers (the flatten-session cache, the staleness fast path) observe the +/// same counters within a process. +/// +/// +/// Lives in Commons rather than a component project because both the +/// ConfigurationDatabase writer and the DeploymentManager readers need it and +/// neither references the other. It is a pure in-memory primitive with no +/// infrastructure dependency, so it does not violate the "implementations live +/// with their component" convention the way a repository would. +/// +/// +/// +/// All counters are -updated and the per-id maps are +/// , so concurrent management +/// commands (which each run on their own DI scope) can bump safely. Reads are +/// deliberately unsynchronised relative to each other: a reader that observes a +/// half-applied batch of bumps sees a CHANGED value for at least one member, +/// which invalidates rather than falsely validates a cache entry. +/// +/// +public sealed class TemplateGraphWatermark : ITemplateGraphWatermark +{ + private readonly ConcurrentDictionary _templateVersions = new(); + private readonly ConcurrentDictionary _instanceVersions = new(); + private long _global; + private long _structure; + + /// + public long Global => Interlocked.Read(ref _global); + + /// + public long StructureVersion => Interlocked.Read(ref _structure); + + /// + public long GetTemplateVersion(int templateId) => + _templateVersions.TryGetValue(templateId, out var v) ? v : 0L; + + /// + public long GetInstanceVersion(int instanceId) => + _instanceVersions.TryGetValue(instanceId, out var v) ? v : 0L; + + /// + public void BumpTemplate(int templateId, bool structural = false) + { + _templateVersions.AddOrUpdate(templateId, 1L, static (_, current) => current + 1); + Interlocked.Increment(ref _global); + if (structural) + Interlocked.Increment(ref _structure); + } + + /// + public void BumpInstance(int instanceId) + { + _instanceVersions.AddOrUpdate(instanceId, 1L, static (_, current) => current + 1); + Interlocked.Increment(ref _global); + } + + /// + public void BumpAll() + { + Interlocked.Increment(ref _global); + Interlocked.Increment(ref _structure); + } +} diff --git a/src/ZB.MOM.WW.ScadaBridge.Commons/Types/Templates/TemplateSummary.cs b/src/ZB.MOM.WW.ScadaBridge.Commons/Types/Templates/TemplateSummary.cs new file mode 100644 index 00000000..c9294eca --- /dev/null +++ b/src/ZB.MOM.WW.ScadaBridge.Commons/Types/Templates/TemplateSummary.cs @@ -0,0 +1,53 @@ +namespace ZB.MOM.WW.ScadaBridge.Commons.Types.Templates; + +/// +/// Row-shaped projection of a template for list surfaces (CLI template list, +/// the Central UI template tree, the ListTemplates management command). +/// +/// +/// Exists so a template listing can be paged in the DATABASE rather than by +/// materialising every template's full child graph — attributes, alarms, script +/// bodies, compositions and native alarm sources — and then discarding all but +/// one page of it in memory. Child collections are reduced to counts, which is +/// what every list surface actually renders. +/// +/// +/// +/// Message-contract evolution rule: this record is additive-only. New fields go +/// on the end with a default so an older peer deserialising a newer payload is +/// unaffected. +/// +/// +/// Template id. +/// Template name. +/// Optional description. +/// Parent template id when this template inherits. +/// Containing folder id. +/// True when the template was auto-derived to back a composition slot. +/// Owning composition row for a derived template. +/// Number of directly declared attributes. +/// Number of directly declared alarms. +/// Number of directly declared scripts. +/// Number of composition slots. +/// Number of native alarm source bindings. +public record TemplateSummary( + int Id, + string Name, + string? Description, + int? ParentTemplateId, + int? FolderId, + bool IsDerived, + int? OwnerCompositionId, + int AttributeCount, + int AlarmCount, + int ScriptCount, + int CompositionCount, + int NativeAlarmSourceCount) +{ + /// + /// Hard ceiling applied to a requested page size, mirroring the existing + /// in-memory Page clamp in the management actor so DB-side paging + /// cannot be used to pull an unbounded result set. + /// + public const int MaxPageSize = 1000; +} diff --git a/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/DeploymentManagerRepository.cs b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/DeploymentManagerRepository.cs index 1b5e5413..0383adcf 100644 --- a/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/DeploymentManagerRepository.cs +++ b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/DeploymentManagerRepository.cs @@ -277,6 +277,89 @@ public class DeploymentManagerRepository : IDeploymentManagerRepository return expired.Count; } + /// + public async Task> QueryDeploymentSummariesAsync( + int? instanceId, + DeploymentStatus? status, + IReadOnlyCollection? instanceIdScope, + int skip, + int? take, + CancellationToken cancellationToken = default) + { + var query = _dbContext.DeploymentRecords.AsNoTracking(); + + if (instanceId.HasValue) + query = query.Where(d => d.InstanceId == instanceId.Value); + + if (status.HasValue) + query = query.Where(d => d.Status == status.Value); + + if (instanceIdScope != null) + { + // Site scoping applied DB-side. An empty scope is a real, meaningful + // filter — a user permitted no in-scope instances sees nothing — so it + // must not be short-circuited into "no filter". + var scope = instanceIdScope as int[] ?? instanceIdScope.ToArray(); + query = query.Where(d => scope.Contains(d.InstanceId)); + } + + // ThenByDescending(Id) for the same reason GetCurrentDeploymentStatusAsync + // does it: DeployedAt ties on rapid redeploys, and an unstable sort key + // makes Skip/Take paging non-deterministic (rows repeat or vanish between + // pages). + var paged = query + .OrderByDescending(d => d.DeployedAt) + .ThenByDescending(d => d.Id) + .Skip(Math.Max(0, skip)); + + if (take is > 0) + paged = paged.Take(Math.Min(take.Value, DeploymentRecordSummary.MaxPageSize)); + + return await paged + .Select(d => new DeploymentRecordSummary( + d.Id, + d.DeploymentId, + d.InstanceId, + d.Status, + d.RevisionHash, + d.DeployedBy, + d.DeployedAt, + d.CompletedAt, + d.ErrorMessage)) + .ToListAsync(cancellationToken); + } + + /// + public async Task PurgeTerminalDeploymentRecordsAsync( + DateTimeOffset cutoffUtc, + int batchSize, + CancellationToken cancellationToken = default) + { + // Terminal statuses ONLY. An InProgress (or Pending) row is never purged + // regardless of age: it is precisely the row TryReconcileWithSiteAsync + // looks for when deciding whether a prior deploy actually landed, and + // deleting it would silently disable the query-before-redeploy idempotency + // guard for that instance. + // + // Bounded batch rather than one unbounded DELETE so the first purge on a + // long-lived system cannot take a table-scale lock on DeploymentRecords. + // The caller re-invokes until it returns fewer rows than the batch size. + var stale = await _dbContext.DeploymentRecords + .Where(d => d.CompletedAt != null + && d.CompletedAt < cutoffUtc + && (d.Status == DeploymentStatus.Success || d.Status == DeploymentStatus.Failed)) + .OrderBy(d => d.Id) + .Take(batchSize) + .ToListAsync(cancellationToken); + + if (stale.Count == 0) + return 0; + + _dbContext.DeploymentRecords.RemoveRange(stale); + await _dbContext.SaveChangesAsync(cancellationToken); + return stale.Count; + } + // --- Startup reconciliation --- /// diff --git a/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/TemplateEngineRepository.cs b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/TemplateEngineRepository.cs index a0a1e632..83e7d017 100644 --- a/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/TemplateEngineRepository.cs +++ b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/TemplateEngineRepository.cs @@ -3,20 +3,32 @@ using ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances; using ZB.MOM.WW.ScadaBridge.Commons.Entities.Scripts; using ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates; using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories; +using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services; +using ZB.MOM.WW.ScadaBridge.Commons.Types.Templates; namespace ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Repositories; public class TemplateEngineRepository : ITemplateEngineRepository { private readonly ScadaBridgeDbContext _context; + private readonly ITemplateGraphWatermark _watermark; /// /// Initializes a new instance of the TemplateEngineRepository class. /// /// The database context used to access template and instance data. - public TemplateEngineRepository(ScadaBridgeDbContext context) + /// + /// Process-wide template/instance version watermark. This repository is the + /// single unit-of-work through which EVERY template-graph mutation commits — + /// TemplateService, the ManagementActor native-alarm-source + /// handlers, and the Transport bundle importer all funnel through the same + /// — so bumping here (rather than in each + /// caller) is the only placement that cannot be bypassed. + /// + public TemplateEngineRepository(ScadaBridgeDbContext context, ITemplateGraphWatermark watermark) { _context = context ?? throw new ArgumentNullException(nameof(context)); + _watermark = watermark ?? throw new ArgumentNullException(nameof(watermark)); } // Template @@ -78,6 +90,95 @@ public class TemplateEngineRepository : ITemplateEngineRepository .ToListAsync(cancellationToken); } + /// + /// + /// Two deliberate departures from : + /// + /// + /// AsNoTracking — the analysis walks (cycle detection, collision + /// detection, canonical-name resolution) never write through the loaded + /// graph, so paying for a change-tracker snapshot of every attribute, + /// alarm, script and composition in the database is pure waste. + /// + /// + /// TemplateScript.Code is projected away (left empty). Script + /// bodies are by far the largest column in the graph and NONE of the + /// analysis consumers read them — CycleDetector reads only ids and + /// edges, CollisionDetector and TemplateResolver read only + /// member names and lock flags. + /// + /// + /// Callers that DO need bodies or tracked entities — the inheritance + /// reconciler and the flattener — must keep using + /// / . + /// + public async Task> GetAllTemplatesForAnalysisAsync(CancellationToken cancellationToken = default) + { + return await _context.Templates + .AsNoTracking() + .Select(t => new Template(t.Name) + { + Id = t.Id, + Description = t.Description, + ParentTemplateId = t.ParentTemplateId, + FolderId = t.FolderId, + IsDerived = t.IsDerived, + OwnerCompositionId = t.OwnerCompositionId, + Attributes = t.Attributes.ToList(), + Alarms = t.Alarms.ToList(), + NativeAlarmSources = t.NativeAlarmSources.ToList(), + Compositions = t.Compositions.ToList(), + Scripts = t.Scripts + .Select(s => new TemplateScript(s.Name, string.Empty) + { + Id = s.Id, + TemplateId = s.TemplateId, + IsLocked = s.IsLocked, + TriggerType = s.TriggerType, + TriggerConfiguration = s.TriggerConfiguration, + ParameterDefinitions = s.ParameterDefinitions, + ReturnDefinition = s.ReturnDefinition, + MinTimeBetweenRuns = s.MinTimeBetweenRuns, + ExecutionTimeoutSeconds = s.ExecutionTimeoutSeconds, + IsInherited = s.IsInherited, + LockedInDerived = s.LockedInDerived + }) + .ToList() + }) + .ToListAsync(cancellationToken); + } + + /// + public async Task> GetTemplateSummariesAsync( + int skip, + int? take, + CancellationToken cancellationToken = default) + { + var query = _context.Templates + .AsNoTracking() + .OrderBy(t => t.Id) + .Skip(Math.Max(0, skip)); + + if (take is > 0) + query = query.Take(Math.Min(take.Value, TemplateSummary.MaxPageSize)); + + return await query + .Select(t => new TemplateSummary( + t.Id, + t.Name, + t.Description, + t.ParentTemplateId, + t.FolderId, + t.IsDerived, + t.OwnerCompositionId, + t.Attributes.Count, + t.Alarms.Count, + t.Scripts.Count, + t.Compositions.Count, + t.NativeAlarmSources.Count)) + .ToListAsync(cancellationToken); + } + /// public async Task> GetTemplatesComposingAsync(int composedTemplateId, CancellationToken cancellationToken = default) { @@ -659,8 +760,129 @@ public class TemplateEngineRepository : ITemplateEngineRepository } /// + /// + /// Inspects the change tracker BEFORE committing (afterwards every entry is + /// and the attribution is lost) and bumps + /// the for each affected template / + /// instance. The bump is applied only AFTER a successful commit, so a failed + /// save does not invalidate caches for a change that never landed. + /// public async Task SaveChangesAsync(CancellationToken cancellationToken = default) { - return await _context.SaveChangesAsync(cancellationToken); + var pending = CollectWatermarkBumps(); + var written = await _context.SaveChangesAsync(cancellationToken); + ApplyWatermarkBumps(pending); + return written; + } + + /// + /// Snapshot of the watermark bumps implied by the current change-tracker + /// contents. Captured pre-commit, applied post-commit. + /// + private readonly record struct WatermarkBumps( + HashSet Templates, + HashSet StructuralTemplates, + HashSet Instances, + bool Unattributed); + + /// + /// Walks the change tracker and maps every added/modified/deleted + /// template-graph or instance-graph entry back to the template (or instance) + /// whose flattened output it can affect. + /// + /// + /// A change is treated as STRUCTURAL — invalidating cached chain membership, + /// not just chain contents — when it adds or removes a + /// row, changes a , or touches a + /// row. Those are exactly the edges the + /// flattener walks to build a chain. + /// + /// + /// + /// An entry whose owning id cannot be resolved (a child row deleted before + /// its FK was materialised, or an entity type added later that this switch + /// does not know) sets Unattributed, which conservatively invalidates + /// every cached chain rather than silently letting a stale entry survive. + /// + /// + private WatermarkBumps CollectWatermarkBumps() + { + var templates = new HashSet(); + var structural = new HashSet(); + var instances = new HashSet(); + var unattributed = false; + + foreach (var entry in _context.ChangeTracker.Entries()) + { + if (entry.State is not (EntityState.Added or EntityState.Modified or EntityState.Deleted)) + continue; + + switch (entry.Entity) + { + case Template t: + templates.Add(t.Id); + // Add/Delete changes chain membership; a re-parent changes it too. + if (entry.State != EntityState.Modified + || entry.Property(nameof(Template.ParentTemplateId)).IsModified) + { + structural.Add(t.Id); + } + break; + case TemplateAttribute a: + templates.Add(a.TemplateId); + break; + case TemplateAlarm al: + templates.Add(al.TemplateId); + break; + case TemplateScript s: + templates.Add(s.TemplateId); + break; + case TemplateNativeAlarmSource ns: + templates.Add(ns.TemplateId); + break; + case TemplateComposition c: + templates.Add(c.TemplateId); + structural.Add(c.TemplateId); + break; + case Instance i: + instances.Add(i.Id); + break; + case InstanceAttributeOverride ao: + instances.Add(ao.InstanceId); + break; + case InstanceAlarmOverride alo: + instances.Add(alo.InstanceId); + break; + case InstanceNativeAlarmSourceOverride nso: + instances.Add(nso.InstanceId); + break; + case InstanceConnectionBinding cb: + instances.Add(cb.InstanceId); + break; + case SharedScript: + // Shared scripts are a session-global input to validation, not + // owned by any one template — invalidate everything. + unattributed = true; + break; + default: + // Areas, folders and anything else do not feed the flattener. + break; + } + } + + return new WatermarkBumps(templates, structural, instances, unattributed); + } + + /// Applies a previously collected snapshot. + private void ApplyWatermarkBumps(WatermarkBumps bumps) + { + if (bumps.Unattributed) + _watermark.BumpAll(); + + foreach (var templateId in bumps.Templates) + _watermark.BumpTemplate(templateId, bumps.StructuralTemplates.Contains(templateId)); + + foreach (var instanceId in bumps.Instances) + _watermark.BumpInstance(instanceId); } } diff --git a/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/ServiceCollectionExtensions.cs b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/ServiceCollectionExtensions.cs index 1752e17e..e8718dac 100644 --- a/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/ServiceCollectionExtensions.cs +++ b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/ServiceCollectionExtensions.cs @@ -1,10 +1,12 @@ using Microsoft.AspNetCore.DataProtection; using Microsoft.EntityFrameworkCore; using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.DependencyInjection.Extensions; using ZB.MOM.WW.ScadaBridge.Commons.Interfaces; using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories; using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services; using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Transport; +using ZB.MOM.WW.ScadaBridge.Commons.Types.Templates; using ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Maintenance; using ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Repositories; using ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Services; @@ -49,6 +51,13 @@ public static class ServiceCollectionExtensions return new ScadaBridgeDbContext(options, protectionProvider); }); + // Process-wide template/instance version watermark. Singleton because the + // writer (TemplateEngineRepository.SaveChangesAsync, scoped) and the readers + // (the flatten-session cache and the staleness fast path, also scoped) must + // observe the same counters across DI scopes. TryAdd so the DeploymentManager + // registration of the same interface is idempotent. + services.TryAddSingleton(); + services.AddScoped(); services.AddScoped(); services.AddScoped(); diff --git a/src/ZB.MOM.WW.ScadaBridge.DeploymentManager/DeploymentManagerOptions.cs b/src/ZB.MOM.WW.ScadaBridge.DeploymentManager/DeploymentManagerOptions.cs index 7fc4dabf..46f905c7 100644 --- a/src/ZB.MOM.WW.ScadaBridge.DeploymentManager/DeploymentManagerOptions.cs +++ b/src/ZB.MOM.WW.ScadaBridge.DeploymentManager/DeploymentManagerOptions.cs @@ -17,4 +17,43 @@ public class DeploymentManagerOptions /// Timeout for acquiring an operation lock on an instance. public TimeSpan OperationLockTimeout { get; set; } = TimeSpan.FromSeconds(5); + + /// + /// Maximum number of instance deployments whose SITE ROUND-TRIP runs + /// concurrently during a bulk DeploySiteAsync. The flatten/validate and + /// persistence phases stay serial regardless (they share one non-thread-safe + /// DbContext); only the network wait is fanned out. + /// + /// + /// Bounded rather than unbounded so a large site cannot open one in-flight + /// gRPC command per instance against a two-node site pair. Four mirrors the + /// conservative end of the site's own apply concurrency; raise it only with + /// evidence from the target-scale load test. + /// + /// + public int SiteDeploymentMaxParallelism { get; set; } = 4; + + /// + /// Per-instance deadline applied to the site round-trip inside a bulk + /// DeploySiteAsync. One unreachable or wedged instance is recorded as a + /// failed row and the rest of the batch continues — the bulk operation never + /// inherits a single instance's hang. + /// + public TimeSpan SiteDeploymentTimeoutPerInstance { get; set; } = TimeSpan.FromSeconds(120); + + /// + /// Retention window for TERMINAL deployment records (Success, + /// Failed). Deployments are insert-only — one row per attempt, forever — + /// so without a window the table grows without bound and every unfiltered + /// deployment query gets slower with the age of the system. + /// + /// + /// Deliberately generous (one year) because deployment history is + /// operator-facing forensics, and deliberately terminal-only: an + /// InProgress or Pending row is never purged regardless of age, + /// because it is exactly the row the query-before-redeploy reconciliation + /// path needs to find. Set to to disable purging. + /// + /// + public TimeSpan TerminalDeploymentRecordRetention { get; set; } = TimeSpan.FromDays(365); } diff --git a/src/ZB.MOM.WW.ScadaBridge.DeploymentManager/DeploymentManagerOptionsValidator.cs b/src/ZB.MOM.WW.ScadaBridge.DeploymentManager/DeploymentManagerOptionsValidator.cs index d041088e..a4431212 100644 --- a/src/ZB.MOM.WW.ScadaBridge.DeploymentManager/DeploymentManagerOptionsValidator.cs +++ b/src/ZB.MOM.WW.ScadaBridge.DeploymentManager/DeploymentManagerOptionsValidator.cs @@ -27,5 +27,20 @@ public sealed class DeploymentManagerOptionsValidator : OptionsValidatorBase TimeSpan.Zero, $"ScadaBridge:DeploymentManager:OperationLockTimeout must be a positive duration " + $"(was {options.OperationLockTimeout}); it bounds acquiring an instance operation lock."); + + builder.RequireThat(options.SiteDeploymentMaxParallelism > 0, + $"ScadaBridge:DeploymentManager:SiteDeploymentMaxParallelism must be at least 1 " + + $"(was {options.SiteDeploymentMaxParallelism}); it is the SemaphoreSlim bound on a bulk " + + $"site deployment's site round-trips, and a non-positive value would deadlock the fan-out."); + + builder.RequireThat(options.SiteDeploymentTimeoutPerInstance > TimeSpan.Zero, + $"ScadaBridge:DeploymentManager:SiteDeploymentTimeoutPerInstance must be a positive duration " + + $"(was {options.SiteDeploymentTimeoutPerInstance}); it bounds each instance's site round-trip " + + $"inside a bulk site deployment."); + + builder.RequireThat(options.TerminalDeploymentRecordRetention >= TimeSpan.Zero, + $"ScadaBridge:DeploymentManager:TerminalDeploymentRecordRetention must not be negative " + + $"(was {options.TerminalDeploymentRecordRetention}); use TimeSpan.Zero to disable purging of " + + $"terminal deployment records."); } } diff --git a/src/ZB.MOM.WW.ScadaBridge.DeploymentManager/DeploymentService.cs b/src/ZB.MOM.WW.ScadaBridge.DeploymentManager/DeploymentService.cs index 321d6fe6..3fef99e7 100644 --- a/src/ZB.MOM.WW.ScadaBridge.DeploymentManager/DeploymentService.cs +++ b/src/ZB.MOM.WW.ScadaBridge.DeploymentManager/DeploymentService.cs @@ -57,6 +57,24 @@ public class DeploymentService /// private const string TimeoutFailurePrefix = "Communication failure:"; + /// Rows deleted per terminal-record purge batch. + private const int TerminalPurgeBatchSize = 500; + + /// Batches deleted per opportunistic sweep, bounding one caller's latency cost. + private const int TerminalPurgeBatchesPerSweep = 4; + + /// Minimum interval between opportunistic terminal-record sweeps. + private static readonly TimeSpan TerminalPurgeMinInterval = TimeSpan.FromHours(6); + + /// + /// Ticks of the last opportunistic terminal-record sweep, process-wide. + /// is scoped, so this must be static to + /// rate-limit across requests; makes the + /// claim-the-sweep step a compare-and-swap so concurrent readers do not all + /// sweep at once. + /// + private static long _lastTerminalPurgeTicks; + /// /// Initializes a new instance of with all required dependencies. /// @@ -161,28 +179,113 @@ public class DeploymentService int instanceId, string user, CancellationToken cancellationToken = default) + { + // Single-instance deploy = the same three phases the bulk path runs, with a + // batch of one and a private flatten session. Composing both entry points + // out of the same phase helpers is what keeps deployment identity, + // idempotency, lock coverage and optimistic concurrency from drifting apart + // between them. + var prepared = await PrepareDeploymentAsync( + instanceId, user, session: null, cancellationToken); + + if (prepared.EarlyResult is { } early) + return early; + + using (prepared.LockHandle) + { + var outcome = await SendDeploymentAsync(prepared, cancellationToken); + return await FinalizeDeploymentAsync(prepared, outcome, user, cancellationToken); + } + } + + /// + /// Phase 1 of a deployment (serial, database-bound): validate the state + /// transition, take the per-instance operation lock, mint the deployment id, + /// flatten + validate, run query-before-redeploy reconciliation, stage the + /// PendingDeployment row and insert the InProgress + /// . + /// + /// + /// Everything here touches the scoped, non-thread-safe DbContext, so it + /// must run serially — a bulk site deploy loops this phase before fanning out + /// the phase-2 network waits. On success the returned + /// owns the operation lock; the CALLER is + /// responsible for disposing it once phase 3 has completed. + /// + /// + /// Instance being deployed. + /// User attributed on the deployment record and audit rows. + /// + /// Shared flatten session for a batch, or for a private + /// one. Sharing it across a batch is what collapses N identical template-chain + /// walks into one. + /// + /// Cancellation token. + /// + /// A prepared deployment ready to send, or one carrying + /// when the deploy resolved before + /// any site round-trip (validation failure, or a reconciled prior deployment). + /// + private async Task PrepareDeploymentAsync( + int instanceId, + string user, + FlattenSession? session, + CancellationToken cancellationToken) { // Load instance var instance = await _repository.GetInstanceByIdAsync(instanceId, cancellationToken); if (instance == null) - return Result.Failure($"Instance with ID {instanceId} not found."); + return PreparedDeployment.Resolved(Result.Failure($"Instance with ID {instanceId} not found.")); // Validate state transition var transitionError = StateTransitionValidator.ValidateTransition(instance.State, "deploy"); if (transitionError != null) - return Result.Failure(transitionError); + return PreparedDeployment.Resolved(Result.Failure(transitionError)); // Acquire per-instance operation lock - using var lockHandle = await _lockManager.AcquireAsync( + var lockHandle = await _lockManager.AcquireAsync( instance.UniqueName, _options.OperationLockTimeout, cancellationToken); + try + { + return await PrepareUnderLockAsync(instance, user, session, lockHandle, cancellationToken); + } + catch + { + // The lock is handed to the caller only on the success path; any fault + // during preparation must release it here or a failed prepare would + // wedge the instance until the process restarts. + lockHandle.Dispose(); + throw; + } + } + + /// + /// The body of that runs while the + /// per-instance operation lock is held. Split out so the lock can be released + /// on every fault path with one try/catch rather than a nested one per step. + /// + private async Task PrepareUnderLockAsync( + Instance instance, + string user, + FlattenSession? session, + IDisposable lockHandle, + CancellationToken cancellationToken) + { + var instanceId = instance.Id; + // Generate unique deployment ID var deploymentId = Guid.NewGuid().ToString("N"); // Flatten configuration (captures template state at this point in time) - var flattenResult = await _flatteningPipeline.FlattenAndValidateAsync(instanceId, cancellationToken); + var flattenResult = await _flatteningPipeline.FlattenAndValidateAsync( + instanceId, cancellationToken, validateScripts: true, session); if (flattenResult.IsFailure) - return Result.Failure($"Validation failed: {flattenResult.Error}"); + { + lockHandle.Dispose(); + return PreparedDeployment.Resolved( + Result.Failure($"Validation failed: {flattenResult.Error}")); + } var flattenedConfig = flattenResult.Value.Configuration; var revisionHash = flattenResult.Value.RevisionHash; @@ -200,8 +303,9 @@ public class DeploymentService validationResult.Errors.Count, string.Join("; ", validationResult.Errors.Select(e => e.Message))); - return Result.Failure( - $"Pre-deployment validation failed: {validationResult.SummarizeErrors()}"); + lockHandle.Dispose(); + return PreparedDeployment.Resolved(Result.Failure( + $"Pre-deployment validation failed: {validationResult.SummarizeErrors()}")); } // Serialize for transmission (also the payload stored in the deployed @@ -218,7 +322,10 @@ public class DeploymentService var reconciled = await TryReconcileWithSiteAsync( instance, revisionHash, configJson, user, cancellationToken); if (reconciled != null) - return Result.Success(reconciled); + { + lockHandle.Dispose(); + return PreparedDeployment.Resolved(Result.Success(reconciled)); + } // Notify-and-fetch: the site fetches the staged config from // CentralFetchBaseUrl, so a deploy is impossible without it. Fail fast @@ -227,8 +334,11 @@ public class DeploymentService // confusing downstream site-fetch failure (and no InProgress record is // stranded). if (string.IsNullOrEmpty(_commOptions.CentralFetchBaseUrl)) - return Result.Failure( - "CentralFetchBaseUrl is not configured — required for deployment (notify-and-fetch)."); + { + lockHandle.Dispose(); + return PreparedDeployment.Resolved(Result.Failure( + "CentralFetchBaseUrl is not configured — required for deployment (notify-and-fetch).")); + } // Create the deployment record directly in InProgress. // @@ -275,12 +385,8 @@ public class DeploymentService deploymentId, instance.UniqueName, revisionHash, user, stagedAt, _commOptions.CentralFetchBaseUrl, token); - _logger.LogInformation( - "Sending deployment {DeploymentId} for instance {Instance} to site {SiteId} (notify-and-fetch)", - deploymentId, instance.UniqueName, siteId); - // Cleanup of the staged PendingDeployment is TTL-based ONLY — the row - // is deliberately NOT deleted on success or in the catch. On a + // is deliberately NOT deleted on success or on failure. On a // central-side Ask timeout the site may have applied AND told the // standby node to fetch; deleting now would 404 that in-flight // standby fetch and break failover. Supersession bounds pending rows @@ -288,8 +394,89 @@ public class DeploymentService // leaving rows for TTL purge is safe. Expired rows are swept by the // central PendingDeploymentPurgeActor singleton on its // CommunicationOptions.PendingDeploymentPurgeInterval cadence. - var response = await _communicationService.RefreshDeploymentAsync(siteId, command, cancellationToken); + return new PreparedDeployment( + instance, deploymentId, record, revisionHash, configJson, + siteId, command, lockHandle, EarlyResult: null); + } + catch (Exception ex) + { + // Staging failed before anything was sent. Record the failure exactly + // as the post-send path does (never leave the record InProgress) and + // release the lock — there is no phase 2 or 3 to run. + await MarkDeploymentFailedAsync(record, instance, deploymentId, user, ex); + lockHandle.Dispose(); + return PreparedDeployment.Resolved(FailureResultFor(ex)); + } + } + /// + /// Phase 2 of a deployment (parallelisable, network-bound): the + /// RefreshDeploymentCommand round-trip to the site. Touches no + /// repository and mutates no shared state, which is what makes it safe to fan + /// out across a batch while phases 1 and 3 stay serial. + /// + /// The prepared deployment to send. + /// Cancellation token, already carrying any per-instance deadline. + /// The site's response, or the exception that prevented one. + private async Task SendDeploymentAsync( + PreparedDeployment prepared, + CancellationToken cancellationToken) + { + try + { + _logger.LogInformation( + "Sending deployment {DeploymentId} for instance {Instance} to site {SiteId} (notify-and-fetch)", + prepared.DeploymentId, prepared.Instance.UniqueName, prepared.SiteIdentifier); + + var response = await _communicationService.RefreshDeploymentAsync( + prepared.SiteIdentifier, prepared.Command, cancellationToken); + return new SendOutcome(response, null); + } + catch (Exception ex) + { + return new SendOutcome(null, ex); + } + } + + /// + /// Phase 3 of a deployment (serial, database-bound): commit the terminal + /// status, apply post-success side effects, and write the audit row. + /// + /// The prepared deployment whose send just completed. + /// The site response or the fault from phase 2. + /// User attributed on the audit rows. + /// Cancellation token. + /// The deployment result for the caller. + private async Task> FinalizeDeploymentAsync( + PreparedDeployment prepared, + SendOutcome outcome, + string user, + CancellationToken cancellationToken) + { + var instance = prepared.Instance; + var record = prepared.Record; + var deploymentId = prepared.DeploymentId; + var instanceId = instance.Id; + + if (outcome.Error is { } sendError) + { + // Any fault out of the site round-trip (timeout, cancellation, + // transport, serialization) must leave the deployment record as + // Failed -- the design requires an interrupted deployment to be + // treated as failed, never stuck in InProgress. + await MarkDeploymentFailedAsync(record, instance, deploymentId, user, sendError); + + _logger.LogError(sendError, + "Deployment {DeploymentId} for instance {Instance} failed", + deploymentId, instance.UniqueName); + + return FailureResultFor(sendError); + } + + var response = outcome.Response!; + + try + { // Update status based on site response. record.Status = response.Status; record.ErrorMessage = response.ErrorMessage; @@ -322,7 +509,7 @@ public class DeploymentService // logged loudly for operator reconciliation but must not flip // the already-committed Success record back to Failed. await ApplyPostSuccessSideEffectsAsync( - instance, deploymentId, revisionHash, configJson, + instance, deploymentId, prepared.RevisionHash, prepared.ConfigJson, forceEnabledState: true, cancellationToken); } @@ -345,61 +532,366 @@ public class DeploymentService } catch (Exception ex) { - // Any exception out of the try (timeout, - // cancellation, transport, serialization, DB) must leave the - // deployment record as Failed -- the design requires an interrupted - // deployment to be treated as failed, never stuck in InProgress. - // - // The failure-status write must NOT use the - // operation's cancellation token. If the operation was cancelled or - // timed out, that token is already cancelled and the cleanup writes - // would themselves throw before the Failed status is persisted. - // Use CancellationToken.None so the failure is durably recorded. - var isTimeout = ex is TimeoutException or OperationCanceledException or Akka.Actor.AskTimeoutException; - - record.Status = DeploymentStatus.Failed; - record.ErrorMessage = isTimeout - ? $"{TimeoutFailurePrefix} {ex.Message}" - : $"Deployment error: {ex.Message}"; - record.CompletedAt = DateTimeOffset.UtcNow; - - try - { - await _repository.UpdateDeploymentRecordAsync(record, CancellationToken.None); - // Note: if the staging SaveChangesAsync above was interrupted, an - // Added PendingDeployment may still be tracked and will be - // committed by this cleanup save. That row is orphaned (no - // RefreshDeploymentCommand was sent, so no site holds its token) - // and is removed by TTL purge / superseded by the next deploy -- - // harmless. - await _repository.SaveChangesAsync(CancellationToken.None); - NotifyStatusChange(record); - - await _auditService.LogAsync(user, "DeployFailed", "Instance", instanceId.ToString(), - instance.UniqueName, new { DeploymentId = deploymentId, Error = ex.Message }, - CancellationToken.None); - } - catch (Exception cleanupEx) - { - // The deployment already failed; a failed cleanup write must not - // mask the original error. Log loudly so an operator can reconcile. - _logger.LogError(cleanupEx, - "Failed to persist Failed status for deployment {DeploymentId} of instance {Instance} " + - "after deployment error: {Error}", - deploymentId, instance.UniqueName, ex.Message); - } + // A fault in the post-send persistence path is handled identically to + // a send fault: the record must never be left InProgress. + await MarkDeploymentFailedAsync(record, instance, deploymentId, user, ex); _logger.LogError(ex, "Deployment {DeploymentId} for instance {Instance} failed", deploymentId, instance.UniqueName); - return Result.Failure( - isTimeout - ? $"Deployment timed out: {ex.Message}" - : $"Deployment failed: {ex.Message}"); + return FailureResultFor(ex); } } + /// + /// Writes the terminal status (plus the + /// failure audit row) for a deployment that faulted, on any phase. + /// + /// + /// The failure-status write deliberately uses : + /// if the operation was cancelled or timed out, the operation's own token is + /// already cancelled and these cleanup writes would themselves throw before + /// the Failed status was persisted, leaving the record stuck InProgress. + /// + /// + /// + /// A fault DURING cleanup is logged loudly and swallowed — it must not mask + /// the original error the caller is about to report. + /// + /// + private async Task MarkDeploymentFailedAsync( + DeploymentRecord record, + Instance instance, + string deploymentId, + string user, + Exception cause) + { + record.Status = DeploymentStatus.Failed; + record.ErrorMessage = IsTimeoutFault(cause) + ? $"{TimeoutFailurePrefix} {cause.Message}" + : $"Deployment error: {cause.Message}"; + record.CompletedAt = DateTimeOffset.UtcNow; + + try + { + await _repository.UpdateDeploymentRecordAsync(record, CancellationToken.None); + // Note: if the staging SaveChangesAsync was interrupted, an Added + // PendingDeployment may still be tracked and will be committed by this + // cleanup save. That row is orphaned (no RefreshDeploymentCommand was + // sent, so no site holds its token) and is removed by TTL purge / + // superseded by the next deploy -- harmless. + await _repository.SaveChangesAsync(CancellationToken.None); + NotifyStatusChange(record); + + await _auditService.LogAsync(user, "DeployFailed", "Instance", instance.Id.ToString(), + instance.UniqueName, new { DeploymentId = deploymentId, Error = cause.Message }, + CancellationToken.None); + } + catch (Exception cleanupEx) + { + _logger.LogError(cleanupEx, + "Failed to persist Failed status for deployment {DeploymentId} of instance {Instance} " + + "after deployment error: {Error}", + deploymentId, instance.UniqueName, cause.Message); + } + } + + /// + /// True when a deployment fault is a timeout/cancellation rather than a hard + /// error. Drives both the marker (which the + /// query-before-redeploy trigger reads) and the caller-facing wording. + /// + private static bool IsTimeoutFault(Exception ex) => + ex is TimeoutException or OperationCanceledException or Akka.Actor.AskTimeoutException; + + /// Maps a deployment fault to the caller-facing failure result. + private static Result FailureResultFor(Exception ex) => + Result.Failure( + IsTimeoutFault(ex) + ? $"Deployment timed out: {ex.Message}" + : $"Deployment failed: {ex.Message}"); + + /// + /// A deployment that has cleared phase 1 and is ready for its site round-trip, + /// OR one that resolved during phase 1 (carrying ). + /// + /// + /// When is this record OWNS the + /// per-instance operation lock in , and the caller must + /// dispose it after phase 3. When is set the lock has + /// already been released and every other member is a placeholder. + /// + /// + private sealed record PreparedDeployment( + Instance Instance, + string DeploymentId, + DeploymentRecord Record, + string RevisionHash, + string ConfigJson, + string SiteIdentifier, + RefreshDeploymentCommand Command, + IDisposable? LockHandle, + Result? EarlyResult) + { + /// Builds a phase-1-resolved deployment carrying the given result and holding no lock. + /// The result to hand back to the caller. + /// A prepared deployment whose is set. + public static PreparedDeployment Resolved(Result result) => + new(null!, string.Empty, null!, string.Empty, string.Empty, string.Empty, null!, null, result); + } + + /// + /// Result of phase 2: either the site's response or the exception that + /// prevented one. Exactly one is non-null. + /// + private readonly record struct SendOutcome(DeploymentStatusResponse? Response, Exception? Error); + + /// + /// Deploy every deployable instance at a site in one operation. + /// + /// + /// Structured as the three deployment phases with the middle one fanned out: + /// + /// + /// + /// Prepare (serial). Every instance is flattened, validated, staged + /// and given an InProgress record on the caller's single scoped + /// DbContext. All instances share ONE , + /// so a template chain common to N instances is walked once, and the + /// session-global queries (shared scripts, schema library, the site's data + /// connections) run once for the whole batch instead of once per instance. + /// + /// + /// Send (bounded parallel). Site round-trips run concurrently up to + /// , + /// each under its own + /// + /// deadline, so one wedged instance cannot stall the batch. This phase + /// touches no repository — that is precisely why it is the only phase that + /// may run in parallel against a non-thread-safe DbContext. + /// + /// + /// Finalize (serial). Terminal statuses, post-success side effects + /// and audit rows are committed one instance at a time, then each + /// instance's operation lock is released. + /// + /// + /// + /// + /// Semantics per instance are byte-for-byte those of + /// — its own deployment id and revision hash, + /// its own operation lock held across the whole operation, its own + /// optimistically-concurrent status record, and the same query-before-redeploy + /// idempotency check. Instances that cannot be deployed (wrong state, failed + /// validation, lock already held) are reported as failed rows; the batch is + /// NOT all-or-nothing, mirroring the artifact deployment path. + /// + /// + /// The database ID of the site whose instances are deployed. + /// The username initiating the deployment, recorded on every record and audit row. + /// Cancellation token for the operation. + /// A per-instance result matrix, or a failure result when the site itself cannot be resolved. + public async Task> DeploySiteAsync( + int siteId, + string user, + CancellationToken cancellationToken = default) + { + var site = await _siteRepository.GetSiteByIdAsync(siteId, cancellationToken); + if (site == null) + return Result.Failure($"Site with ID {siteId} not found."); + + var instances = await _siteRepository.GetInstancesBySiteIdAsync(siteId, cancellationToken); + if (instances.Count == 0) + return Result.Success(new SiteDeploymentSummary(site.SiteIdentifier, [], 0, 0)); + + // ---- Phase 1: prepare, serially, on ONE shared flatten session. ---- + var session = _flatteningPipeline.CreateSession(); + var prepared = new List(instances.Count); + var results = new List(instances.Count); + + foreach (var instance in instances) + { + cancellationToken.ThrowIfCancellationRequested(); + + PreparedDeployment step; + try + { + step = await PrepareDeploymentAsync(instance.Id, user, session, cancellationToken); + } + catch (Exception ex) + { + // A prepare fault (most commonly a TimeoutException from the + // operation lock because another operation holds this instance) + // fails only this instance. Recording it and moving on is what + // makes a bulk deploy usable while individual instances are busy. + _logger.LogWarning(ex, + "Preparing instance {Instance} for bulk deployment of site {SiteId} failed", + instance.UniqueName, site.SiteIdentifier); + results.Add(new InstanceDeploymentResult( + instance.Id, instance.UniqueName, null, false, ex.Message)); + continue; + } + + if (step.EarlyResult is { } early) + { + results.Add(ToInstanceResult(instance.Id, instance.UniqueName, early)); + continue; + } + + prepared.Add(step); + } + + // ---- Phase 2: bounded-parallel site round-trips. ---- + var outcomes = await SendPreparedAsync(prepared, cancellationToken); + + // ---- Phase 3: finalize, serially, releasing each lock as we go. ---- + for (var i = 0; i < prepared.Count; i++) + { + var step = prepared[i]; + try + { + var result = await FinalizeDeploymentAsync(step, outcomes[i], user, cancellationToken); + results.Add(ToInstanceResult(step.Instance.Id, step.Instance.UniqueName, result, step.DeploymentId)); + } + finally + { + step.LockHandle?.Dispose(); + } + } + + var summary = new SiteDeploymentSummary( + site.SiteIdentifier, + results, + results.Count(r => r.Success), + results.Count(r => !r.Success)); + + _logger.LogInformation( + "Bulk deployment of site {SiteId} complete: {SuccessCount} succeeded, {FailureCount} failed " + + "({ChainLoads} template-chain load(s) across {InstanceCount} instance(s))", + site.SiteIdentifier, summary.SuccessCount, summary.FailureCount, + session.ChainLoads, instances.Count); + + return Result.Success(summary); + } + + /// + /// Runs phase 2 for a whole batch: every prepared deployment's site round-trip, + /// concurrent up to + /// and each bounded by + /// . + /// Outcomes are returned positionally so phase 3 can pair them back with their + /// prepared deployment. + /// + private async Task SendPreparedAsync( + IReadOnlyList prepared, + CancellationToken cancellationToken) + { + var outcomes = new SendOutcome[prepared.Count]; + if (prepared.Count == 0) + return outcomes; + + using var gate = new SemaphoreSlim(_options.SiteDeploymentMaxParallelism); + + var sends = prepared.Select(async (step, index) => + { + await gate.WaitAsync(cancellationToken); + try + { + using var cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + cts.CancelAfter(_options.SiteDeploymentTimeoutPerInstance); + outcomes[index] = await SendDeploymentAsync(step, cts.Token); + } + finally + { + gate.Release(); + } + }); + + await Task.WhenAll(sends); + return outcomes; + } + + /// Maps a per-instance deployment result into the bulk summary row shape. + private static InstanceDeploymentResult ToInstanceResult( + int instanceId, + string uniqueName, + Result result, + string? deploymentId = null) => + result.IsSuccess + ? new InstanceDeploymentResult( + instanceId, uniqueName, result.Value.DeploymentId, true, null) + : new InstanceDeploymentResult( + instanceId, uniqueName, deploymentId, false, result.Error); + + /// + /// Opportunistic retention sweep for TERMINAL deployment records, rate-limited + /// to once every per process and bounded + /// to batches per call. + /// + /// + /// Swept on read (the deployment-list path) rather than from a timer for the + /// same reason expired secured writes are: the component owns no scheduling + /// infrastructure of its own, and the read path is exactly the operation that + /// suffers when the table is allowed to grow without bound. Both central nodes + /// may sweep; the delete is idempotent, so a duplicate pass is a no-op. + /// + /// + /// + /// Best-effort: a failed sweep is logged and swallowed. Retention maintenance + /// must never fail the operator's actual query. + /// + /// + /// Cancellation token. + /// The number of records purged by this call (0 when the sweep was skipped). + public async Task TryPurgeTerminalDeploymentRecordsAsync(CancellationToken cancellationToken = default) + { + var retention = _options.TerminalDeploymentRecordRetention; + if (retention <= TimeSpan.Zero) + return 0; + + var now = DateTimeOffset.UtcNow; + var last = Interlocked.Read(ref _lastTerminalPurgeTicks); + if (last != 0 && now.UtcTicks - last < TerminalPurgeMinInterval.Ticks) + return 0; + + // Claim the sweep. A loser of this CAS returns immediately rather than + // running a redundant second pass. + if (Interlocked.CompareExchange(ref _lastTerminalPurgeTicks, now.UtcTicks, last) != last) + return 0; + + var cutoff = now - retention; + var purged = 0; + + try + { + for (var batch = 0; batch < TerminalPurgeBatchesPerSweep; batch++) + { + var deleted = await _repository.PurgeTerminalDeploymentRecordsAsync( + cutoff, TerminalPurgeBatchSize, cancellationToken); + purged += deleted; + if (deleted < TerminalPurgeBatchSize) + break; + } + + if (purged > 0) + { + _logger.LogInformation( + "Purged {Count} terminal deployment record(s) completed before {Cutoff} " + + "(retention {Retention}).", + purged, cutoff, retention); + } + } + catch (Exception ex) + { + _logger.LogWarning(ex, + "Terminal deployment-record retention sweep failed; retrying after {Interval}.", + TerminalPurgeMinInterval); + } + + return purged; + } + /// /// Disable an instance. Stops Instance Actor, retains config, S&F drains. /// @@ -1178,3 +1670,37 @@ public record DeploymentComparisonResult( bool IsStale, DateTimeOffset DeployedAt, ConfigurationDiff? Diff = null); + +/// +/// One instance's outcome inside a bulk site deployment. +/// +/// Database id of the instance. +/// The instance's unique name (also its operation-lock key). +/// +/// The deployment id minted for this instance, or when the +/// instance never got that far (state-transition rejection, lock contention). +/// +/// Whether the site confirmed the apply. +/// Failure detail; on success. +public record InstanceDeploymentResult( + int InstanceId, + string UniqueName, + string? DeploymentId, + bool Success, + string? ErrorMessage); + +/// +/// Result matrix of a bulk site deployment. Mirrors the shape of +/// ArtifactDeploymentSummary: successes are not rolled back when other +/// instances fail, and each failed instance is individually retryable through the +/// ordinary single-instance deploy. +/// +/// The site's string identifier (e.g. site-a). +/// Per-instance outcomes. +/// Number of instances the site confirmed. +/// Number of instances that failed for any reason. +public record SiteDeploymentSummary( + string SiteIdentifier, + IReadOnlyList InstanceResults, + int SuccessCount, + int FailureCount); diff --git a/src/ZB.MOM.WW.ScadaBridge.DeploymentManager/FlattenSession.cs b/src/ZB.MOM.WW.ScadaBridge.DeploymentManager/FlattenSession.cs new file mode 100644 index 00000000..6c0fe23c --- /dev/null +++ b/src/ZB.MOM.WW.ScadaBridge.DeploymentManager/FlattenSession.cs @@ -0,0 +1,257 @@ +using ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites; +using ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates; +using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services; +using ZB.MOM.WW.ScadaBridge.Commons.Types.Flattening; + +namespace ZB.MOM.WW.ScadaBridge.DeploymentManager; + +/// +/// Memoises the repository reads that performs, +/// for the lifetime of ONE flatten/validate session. +/// +/// +/// Without a session every instance flatten re-walks its whole template +/// inheritance chain (one query per link), re-loads the compositions of every +/// template in that chain and of every composed chain it reaches, and re-issues +/// the three session-global queries — shared scripts, the shared-schema library, +/// and the target site's data connections. For a bulk deploy of N instances off +/// the same template that is N× the same work; even a single instance re-loads a +/// composed chain once per composing template. +/// +/// +/// +/// Cache key. Templates are keyed on (id, ITemplateGraphWatermark +/// version) and chain MEMBERSHIP additionally on the watermark's +/// , so a template edited +/// mid-session is re-read rather than served from the memo. The session is also +/// short-lived by construction — it is created per deploy/validate operation and +/// discarded with it — so a cached graph can never outlive the operation that +/// captured it. That is the design's "template state is captured at the time of +/// flatten" guarantee, unchanged: the session narrows the capture window, it does +/// not widen it. +/// +/// +/// +/// Threading. A session is NOT thread-safe and must not be shared across +/// threads. It caches entities materialised by one scoped DbContext, which +/// is itself single-threaded, and it hands out tracked entities that the +/// in-flight bundle importer relies on observing. DeploySiteAsync honours +/// this by running the whole prepare (flatten) phase serially on one scope and +/// parallelising only the site round-trips, which touch no repository at all. +/// +/// +public sealed class FlattenSession +{ + private readonly ITemplateGraphWatermark _watermark; + + private readonly Dictionary _templates = new(); + private readonly Dictionary _chains = new(); + private readonly Dictionary Compositions)> _compositions = new(); + private readonly Dictionary> _dataConnections = new(); + + private IReadOnlyList? _sharedScripts; + private IReadOnlyDictionary? _schemaLibrary; + + /// Initializes a session bound to the process-wide graph watermark. + /// Watermark supplying the per-template versions used as cache keys. + public FlattenSession(ITemplateGraphWatermark watermark) + { + _watermark = watermark ?? throw new ArgumentNullException(nameof(watermark)); + } + + /// + /// Number of template-chain walks that actually hit the repository. Reset + /// never; used by tests to assert that N instances sharing a template load + /// that chain exactly once. + /// + public int ChainLoads { get; private set; } + + /// Number of single-template repository reads that missed the memo. + public int TemplateLoads { get; private set; } + + /// Number of per-template composition queries that missed the memo. + public int CompositionLoads { get; private set; } + + /// Number of times the session-global queries (shared scripts, schema library) ran. + public int GlobalLoads { get; private set; } + + /// Number of per-site data-connection queries that missed the memo. + public int DataConnectionLoads { get; private set; } + + /// + /// Every template id this session resolved — the union of all inheritance and + /// composed chains it walked. The staleness fast path records the watermark + /// version of exactly this set, so a change to ANY template that fed a + /// flatten invalidates the memoised revision hash. + /// + public IReadOnlyCollection VisitedTemplateIds => _templates.Keys; + + /// + /// Returns the inheritance chain for (the + /// template itself, then each ancestor), loading it through + /// on a miss. + /// + /// Root of the chain to build. + /// Repository read for one template with its children. + /// The chain, ordered leaf-first. + public async Task> GetChainAsync( + int templateId, + Func> loadTemplate) + { + var structure = _watermark.StructureVersion; + if (_chains.TryGetValue(templateId, out var cached) && IsChainCurrent(cached, structure)) + return cached.Chain; + + var chain = new List