diff --git a/CLAUDE.md b/CLAUDE.md index da8b3dd8..9cf7a310 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -64,7 +64,7 @@ spec for each is `docs/requirements/Component-.md`, and `README.md` carrie ### Architecture & Runtime - Instance modeled as Akka actor (Instance Actor) — single source of truth for runtime state. - Site Runtime actor hierarchy: Deployment Manager singleton → Instance Actors → Script Actors + Alarm Actors. -- Script Actors spawn short-lived Script Execution Actors on a dedicated blocking I/O dispatcher. +- Script/Alarm Actors launch script runs directly onto a dedicated blocking-I/O thread pool (`ScriptExecutionScheduler`) — the short-lived `ScriptExecutionActor`/`AlarmExecutionActor` children were **deleted** by WP3.1 (they had no `Receive` handler, no `PostStop`, and their `IActorRef` was never a message target; `ScriptRunLauncher` carries the run body verbatim). Same pool, same telemetry, same audit `ExecutionId`/`ParentExecutionId` threading, same "a stop does NOT cancel an in-flight run" semantics — minus a per-run actor cell. WP3.1 also: **trigger-expression evaluation left that pool** for the shared .NET thread pool behind `TriggerEvalGate` (arch-review finding #4 — blocked script bodies used to stall every Expression trigger on the node indefinitely); script and eval deadlines are armed **at enqueue** so queue wait burns the budget (a run dequeuing past its deadline is shed unrun); the pool is instance-scaled grow-only `clamp(max(ScriptExecutionThreadCount, ceil(instances/8)), 1, ScriptExecutionMaxThreadCount)`; the stuck-script watchdog now **detaches and replaces** a wedged worker (capped at pool size, gauge `DetachedScriptThreads`); `MaxConcurrentRunsPerScript` (4) sheds the newest run with a counter + rate-limited site event + an explicit error reply to Ask callers; `SiteScriptCompileCache` evicts an oldest-⅛ batch by approximate LRU instead of clearing wholesale; and deploy/startup compiles are warmed off-thread before the (ordering-critical, now cache-hit-only) synchronous deploy gate. Design memo: `docs/plans/2026-08-15-script-pool-split-design.md`. - Alarm Actors are separate peer subsystem from scripts (not inside Script Engine). - Shared scripts execute inline as compiled code (no separate actors). - Site-wide Akka stream for attribute value and alarm state changes with per-subscriber buffering. @@ -79,7 +79,7 @@ spec for each is `docs/requirements/Component-.md`, and `README.md` carrie - DCL write failures returned synchronously to calling script. - Tag path resolution retried periodically for devices still booting. - Static attribute writes persisted to local SQLite (survive restart/failover, reset on redeployment). -- **Consolidated site database (LocalDb Phase 1 + 2, complete 2026-07-20).** Ten tables now live in ONE `ZB.MOM.WW.LocalDb`-managed SQLite file (enumerate them with `grep -r RegisterReplicated src/`), configured by the **required** `LocalDb:Path` (`/app/data/site-localdb.db` on the rig; validated with `ValidateOnStart`, so a site config missing it fails to boot). Consequences worth knowing: +- **Consolidated site database (LocalDb Phase 1 + 2, complete 2026-07-20).** Ten tables now live in ONE `ZB.MOM.WW.LocalDb`-managed SQLite file (the `SiteLocalDbSetup.ReplicatedTables` list), configured by the **required** `LocalDb:Path` (`/app/data/site-localdb.db` on the rig; validated with `ValidateOnStart`, so a site config missing it fails to boot). Consequences worth knowing: - `site_events.id` changed from autoincrement INTEGER to an application-minted **GUID**. Last-writer-wins keys on the primary key, so two nodes independently minting `id=1,2,3…` would destroy each other's events rather than merge them. The event-log read path uses a composite `(timestamp, id)` keyset cursor with an **opaque string** continuation token; `EventLogEntry.Id` and both `ContinuationToken`s are `string`/`string?` on the site↔central Akka DTOs. - `ScadaBridge:OperationTracking:ConnectionString`, `ScadaBridge:SiteEventLog:DatabasePath` and — as of Phase 2 — `ScadaBridge:StoreAndForward:SqliteDbPath` + `ScadaBridge:Database:SiteDbPath` are all **migration-only** — nothing reads them but `SiteLocalDbLegacyMigrator`, which copies a pre-Phase-1 file in once (deterministic `mig-{NodeName}-{legacyId}` event ids, `INSERT OR IGNORE`, runs AFTER `RegisterReplicated` so migrated rows replicate) and renames it `.migrated`. Delete the keys once a node has migrated. - This incidentally fixes a data-loss bug: both legacy databases defaulted to CWD-relative paths **outside** the mounted volume and were discarded on every container recreate. @@ -89,14 +89,15 @@ spec for each is `docs/requirements/Component-.md`, and `README.md` carrie - CDC replication does all three jobs now: config deploys reach the standby as ordinary row changes — **the standby makes no fetch at all** during a deploy (`SiteReconciliationActor`'s node-STARTUP fetch when central reports gaps is a different, surviving path) — and buffer mutations replicate via triggers on `sf_messages`. `ReplaceAllAsync` was a destructive delete-all-then-insert-all resync and is **unsafe to reintroduce**: a mass DELETE on a replicated table would be captured and shipped to the peer. LocalDb's snapshot resync merges per row under LWW and never deletes, which is also why the old N1 directional-authority guard is gone — there is no wipe left to gate. - **`notification_lists` and `smtp_configurations` are created but deliberately NOT registered.** They are permanently empty on a site (no writer since 2026-07-10, the migrator skips them, the active-node purge keeps them empty), and registering them would open a standing replication channel whose only historical payload was plaintext SMTP passwords. Pinned by a security-named test, and verified live: those two tables have **no CDC triggers** on either rig node. - **Operational constraints (read before upgrading a site pair):** stop and start both nodes TOGETHER — rolling one at a time is no longer supported, since the legacy `SfBufferSnapshot` compatibility handler went with the replicator. And a node offline longer than `LocalDb:Replication:TombstoneRetention` (default 7 days) can resurrect deleted rows on rejoin. See `docs/deployment/topology-guide.md`. - - `LocalDb:Replication:MaxBatchSize` batches by ROW COUNT, not bytes, against a 4 MB gRPC cap — the rig pins it to **16** (~70 KB worst-case `config_json` x 16 ~= 1.1 MB). The 500 default would allow ~35 MB. + - **Batching is by BYTE BUDGET as of LocalDb 0.2.0** — `LocalDb:Replication:MaxBatchBytes` (default **2 MB**, sized under the 4 MB gRPC cap) bounds a delta/snapshot message by summed serialized size via a per-message split in `SyncSession.PumpLoopAsync`, with `MaxBatchSize` demoted to a secondary row cap; a single row over budget is sent alone rather than stalling the stream. **The rig's old `MaxBatchSize = 16` pin (a hand-computed byte-budget proxy: ~70 KB worst-case `config_json` x the 500 default is ~35 MB) is retired, but `MaxBatchSize` is NOT fully redundant with `MaxBatchBytes`** — it also bounds the separate DB READ page in `OplogStore.ReadBatchAboveAsync`/`SnapshotStreamer`, which materializes the whole page into memory *before* the byte-budget split runs, so an unset (500-default) `MaxBatchSize` still lets a reconnect drain transiently allocate ~35 MB per read even though every wire message stays under `MaxBatchBytes` (arch-review adversarial finding F2). Both site-a nodes on `docker/` therefore pin an explicit `"MaxBatchSize": 64` to bound that transient allocation, while `MaxBatchBytes` stays unset (its 2 MB default) to bound the wire message; site-b/site-c stay unreplicated so the key doesn't apply there. + - **CDC registration is conditional, and both directions self-heal at boot** (arch-review WP1.3 + WP3.3, `Host/SiteLocalDbSetup.cs`). Capture triggers are installed only when this node has replication configured — `PeerAddress` **or** `ApiKey`, an OR because only the dialling half sets `PeerAddress` while the passive half carries the key alone. An unreplicated node calls **`DeregisterReplicated`** on all ten tables at boot, dropping triggers an earlier build left behind and pruning their oplog/row-version rows (idempotent; logs once at Information when something was actually cleaned). A replicating node registers with **`baselineExistingRows: true`**, which seeds `__localdb_row_version` for pre-existing rows at the LWW floor (HLC `0`, this node's id) and flags a snapshot resync — so turning replication ON for a site that has been running without it now converges on the rows already in the file instead of only on writes made after the restart. Deregistration must be symmetric (the handshake compares registered-table digests fail-closed), so replication is a both-nodes-together change in either direction. LocalDb 0.2.0 also makes backlog depth O(1) and drops the unused `__localdb_oplog_hlc` index; the on-disk bookkeeping schema goes to v2, upgraded in place on open, with no wire change (0.1.x peers still sync). **LocalDb 0.2.1** (two adversarial-review findings against 0.2.0, wire-compatible, no schema change) fixes: (1) `DeregisterReplicated` was deleting the HLC clock's only durable crash anchor along with the table rows — an ungraceful exit (crash, SIGKILL, power loss) on a fully-deregistered file left nothing to recover from, so the reopened clock could restart from wall-clock time and re-issue stamps below HLCs peers already hold (silently discarded by LWW); the current clock is now flushed inside the deregistration transaction, before the prune. (2) The sync inbox was an **unbounded** channel, so inbound memory was a function of what the peer sends — a mutual full snapshot (both sides of a pair baseline unconditionally when replication is enabled late) buffered whole, in memory, on both nodes at once; it is now **bounded at 64 messages** with wait-mode backpressure (unrelated to, and not to be confused with, the rig's separate `MaxBatchSize: 64` DB-read-page pin above), which required moving the snapshot to run alongside the receive loop (not inside the pre-loop handshake) so the inbox is drained for the whole session and can't deadlock itself. - All timestamps are UTC throughout the system. - Inter-cluster communication uses **three** transports, not two — **all cross-cluster command/control and data now rides gRPC** after the ClusterClient→gRPC migration's Phase 4 (`docs/plans/2026-07-22-clusterclient-to-grpc-plan.md`) deleted Akka `ClusterClient`/`ClusterClientReceptionist`: (1) **gRPC command/control** — site→central over the central-hosted `CentralControlService` (`GrpcCentralTransport`, sticky central-a→central-b channel pair; deployments/notifications/health/heartbeat/audit-ingest/reconcile), and central→site over the site-hosted `SiteCommandService` (`GrpcSiteTransport`, per-site NodeA→NodeB channel pair; the 28 lifecycle/OPC-UA/query/parked/route/failover commands); (2) **gRPC** server-streaming for real-time data (attribute values, alarm states, `SiteStreamService`); and (3) **plain token-gated HTTP** for the deployment config itself — notify-and-fetch, the site pulls the config from `DeploymentConfigEndpoints` (`ManagementService/DeploymentConfigEndpoints.cs`) with an `X-Deployment-Token` header, `AllowAnonymous` with the per-deployment token as the entire security boundary. The gRPC boundary is per-site PSK-authenticated (`ControlPlaneAuthInterceptor`, unchanged). There is **no receptionist registration** — discovery is by dialling configured endpoints; central builds one `SitePairChannelProvider` per site (addresses from `Site.GrpcNodeAAddress`/`GrpcNodeBAddress`, refreshed from the DB every 60s and on admin changes), sites dial `ScadaBridge:Communication:CentralGrpcEndpoints` (both central nodes, h2c on `CentralGrpcPort` 8083, **NOT** via Traefik). **Discovery is asymmetric by design:** central discovers site gRPC addresses from the *database* (refreshable at runtime), sites discover central from *appsettings* (`CentralGrpcEndpoints`, static — restart required; `StartupValidator` requires a Site node to list at least one). `Akka.Cluster.Tools` stays for ClusterSingleton; only the ClusterClient part is gone. **Central never buffers for an unreachable site** — the send fails with the caller's Ask/deadline timing out; a `ConnectionStateChanged` mechanism built for this was deleted as dead code. - **All clusters share ONE ActorSystem name**, `"scadabridge"` — hardcoded in `AkkaHostedService` at the `ActorSystem.Create` call. Central and each site are separate clusters *only* by seed-node partitioning. The constraint originated with ClusterClient (Akka.Remote address matching meant it could not reach a differently-named system); whether it is still load-bearing after the gRPC migration has **not** been re-verified, so treat the name as fixed until someone checks. - **`ActiveNodeEvaluator.SelfIsOldestUp` is THE single definition of "active node"** (`Communication/ClusterState/ActiveNodeEvaluator.cs`) — the **oldest Up member** in a role scope, and explicitly **never `cluster.State.Leader`**: leadership (lowest address) is an Akka-internal concept that diverges from singleton placement permanently once the original first node restarts and rejoins, and both sides claim it during a partition. The equivalence *oldest-Up == where `ClusterSingletonManager` places singletons* **is** the design. `ClusterActivityEvaluator.SelfIsOldest`, the S&F delivery gate, `/health/active` and the heartbeat `IsActive` stamp all delegate here. - Site nodes carry **two Akka roles**: the base `Site` plus a site-specific `site-{SiteId}` (`AkkaHostedService.BuildRoles`). Singletons scope to the **site-specific** role. - **The gRPC boundary is authenticated (PSK) as of 2026-07-22; Akka remoting still is not, and nothing is encrypted.** Akka remoting sets no `enable-ssl`, no secure cookie, no `trusted-selection-paths` — so intra-cluster Akka remoting remains open to anyone who can reach the remoting port, and that boundary still assumes a trusted network. The gRPC listener stays **h2c**, but `SiteStreamService` is no longer open: `ControlPlaneAuthInterceptor` (`Host/ControlPlaneAuthInterceptor.cs`) gates `/sitestream.SiteStreamService/` — including the `PullAuditEvents`/`PullSiteCalls` RPCs that return audit rows — against a **per-site preshared key**, fail-closed, constant-time compared, alongside the separate `LocalDbSyncAuthInterceptor` on `/localdb_sync.v1.LocalDbSync/` with its own separate key. **Site side:** `ScadaBridge:Communication:GrpcPsk`, in production `${secret:SB-GRPC-PSK-}`, and **`StartupValidator` refuses to boot a site node without it** (an unset key would leave the node healthy-looking but serving nothing). **Central side:** `SitePskProvider` resolves `SB-GRPC-PSK-{siteId}` from the secrets store at channel-build time (sites are added at runtime, so no boot-time expansion is possible), with `ScadaBridge:Communication:SitePsks:{siteId}` as an override for hosts running without a master key — the docker rig uses the latter. One key per site, never fleet-wide. A bearer token over h2c is readable and replayable on-path; TLS is the follow-on hardening and needs no change to this design. Introduced by Phase 0 of the ClusterClient→gRPC migration (`docs/plans/2026-07-22-clusterclient-to-grpc-plan.md`). -- gRPC streaming channel — **note the direction is inverted from the data flow**: data moves site→central, but each **site node hosts the gRPC server** (`SiteStreamGrpcServer`, Kestrel h2c, port 8083, mapped **only in the Site branch** of `Program.cs`) and **central is the client**, dialling in. Central creates per-site `SiteStreamGrpcClient` via `SiteStreamGrpcClientFactory`, keyed **`(siteId, endpoint)`** — the key was widened from site-only to fix an arch-review High where one session's NodeA→NodeB flip disposed a channel another session was still using. Proto evolution is **additive only** and field numbers are never reused (`AlarmStateUpdate` grew 7→23 fields for the native-alarm mirror). Generated C# is **vendored** under `Communication/SiteStreamGrpc/` with the `` include commented out — regeneration is a manual toggle-build-copy-untoggle. +- gRPC streaming channel — **note the direction is inverted from the data flow**: data moves site→central, but each **site node hosts the gRPC server** (`SiteStreamGrpcServer`, Kestrel h2c, port 8083, mapped **only in the Site branch** of `Program.cs`) and **central is the client**, dialling in. Central creates per-site `SiteStreamGrpcClient` via `SiteStreamGrpcClientFactory`, keyed **`(siteId, endpoint)`** — the key was widened from site-only to fix an arch-review High where one session's NodeA→NodeB flip disposed a channel another session was still using. Proto evolution is **additive only** and field numbers are never reused (`AlarmStateUpdate` grew 7→23 fields for the native-alarm mirror). Generated C# is **vendored** under `Communication/SiteStreamGrpc/` with the `` include commented out — regeneration is a manual toggle-build-copy-untoggle. **The client reconnects on graceful (OK-status) stream completion, not just on fault (arch-review remediation, commit `34a3f4bb`)** — Kestrel's `MaxStreamLifetime`/`Grpc.AspNetCore.Server` max-connection-age periodically ends a healthy stream with a normal completion, which the client used to treat as terminal (no reconnect attempt), silently killing a site's live feed until the next process restart (observed up to ~4h on the rig); live-probed at a 2-minute forced lifetime, reconnect lands within one reconcile tick and `IsLive` reflects the gap in between. - Native alarms are a **read-only** mirror of OPC UA Alarms & Conditions and MxAccess Gateway alarms — **no ack-back, no central tables**; state lives in the site's `native_alarm_state`, survives failover, and is cleared on redeploy/undeploy (mirrors static overrides). Central's per-site live alarm cache (`ISiteAlarmLiveCache`) is **transient in-memory only** — there is deliberately no persisted central alarm store, so the 15s poll remains the NotReporting authority behind the live stream. See `Component-DataConnectionLayer.md` / `Component-CentralUI.md` for the model and the authoring surface. - **`AckTime` mirror enrichment + the `Alarms` script accessor (MES alarm-status API Phase 1, 2026-08-01).** `AlarmStateChanged` carries an additive `AckTime` (`DateTimeOffset?`), mirrored on the vendored `AlarmStateUpdate` proto as **field 24** and persisted inside `native_alarm_state`'s `metadata_json` — deliberately NOT a new column, because that table is `RegisterReplicated` and LocalDb builds its CDC triggers from the column list at registration time. Set only while a condition is active AND acknowledged (so it is null while unacked and cleared on re-raise); the DCL stamps the source's own ack instant for OPC UA (new SelectClause **index 18** = `AckedState/TransitionTime`) and its observation time of the ack transition for MxGateway, which supplies none. Site `Call` scripts read alarms via the new **`Alarms.CurrentAsync()`** accessor (`ScriptRuntimeContext` + `ScriptGlobals`, local Ask on `GetAlarmSnapshotRequest`, returns `Commons.Types.Scripts.ScriptAlarm`), mirrored on `ScriptCompileSurface` AND the Central UI `SandboxScriptHost` editor surface. The trust model needed no change — it is a deny-list over API roots, not an allow-list of context members. Plan: `docs/plans/2026-06-30-mes-alarm-status-api.md` (Phases 2–4 are deployed config, not repo). - OPC UA cert trust is **site-local and not persisted centrally** (follow-up): the verify-endpoint probe captures an untrusted server cert but **NEVER trusts it**, and DeploymentManager broadcasts `TrustServerCertCommand`/`RemoveServerCertCommand` to **BOTH** site nodes — `CertStoreActor` runs on every site node, not as a singleton, so PKI stores stay consistent across failover. @@ -121,6 +122,7 @@ spec for each is `docs/requirements/Component-.md`, and `README.md` carrie - Flattened configs include a revision hash for staleness detection. - Deployment identity: unique deployment ID + revision hash for idempotency. - Per-instance operation lock covers all mutating commands (deploy, disable, enable, delete). +- **Bulk site deployment (`DeploymentService.DeploySiteAsync`, arch-review WP2.5)** redeploys every out-of-date instance at a site through one flatten session (a template chain shared by N instances is walked once) in three phases — serial Prepare, bounded-parallel Send (`SiteDeploymentMaxParallelism`, default 4), serial Finalize — still **not all-or-nothing**: any one instance's failure doesn't block the rest. The `PendingDeployment` staging row (5-minute TTL) is written per-instance immediately before that instance's round-trip in the Send phase, not upfront in Prepare, so a large batch's tail instances don't have their TTL clock already half-spent by the time they're reached. - Site-side apply is all-or-nothing per instance. - System-wide artifact version skew across sites is supported. - Last-write-wins for concurrent template editing (no optimistic concurrency on templates). @@ -152,7 +154,7 @@ spec for each is `docs/requirements/Component-.md`, and `README.md` carrie - One row per lifecycle event; cached calls produce 4+ rows per operation (`Submitted`, `Forwarded`, `Attempted`, `Delivered`/`Parked`/`Discarded`). - `ExecutionId` (`uniqueidentifier NULL`) is the universal per-run correlation value — every audit row emitted by one script execution / inbound request shares it; `CorrelationId` remains the per-operation lifecycle id (NULL for sync one-shots). - `ParentExecutionId` (`uniqueidentifier NULL`) is the cross-execution spawn pointer — every row of a spawned run carries the spawner's `ExecutionId`; bridges inbound API → routed-site-script, alarm-triggered on-trigger scripts, and nested `CallScript`/`CallShared` invocations; `IX_AuditLog_ParentExecution` backs the filter + the recursive execution-tree walk. **Tag-cascade (alarm leg) is populated, not just plumbed** — M5.4 T4 threaded the `parentExecutionId` parameter but every `AlarmActor.SpawnAlarmExecution` call site passed null, so alarm runs were silently always roots; the id now rides site-locally as `SetStaticAttributeCommand.SourceExecutionId` → `AttributeValueChanged.SourceExecutionId` → `SpawnAlarmExecution` (all additive/nullable, no wire/proto/schema change; `Expression` triggers capture the writer *with* the evaluated snapshot since the eval completes off-dispatcher). Sources are `ScriptRuntimeContext.SetAttribute` (the run's own `ExecutionId`) and `Route.To(...).SetAttributes(...)` (the inbound request's). **Still roots by design, not omission:** alarms fired by DCL data (external values have no spawning execution — including the device echo of a script write to a *data-sourced* attribute, so only **static** writes cascade), and `ScriptActor` value-change/conditional/expression/timer trigger runs (a timer tick has no spawner; a `WhileTrue`/interval run has no single identifiable write). -- Site SQLite hot-path first, then gRPC telemetry to central; ingest is idempotent on `EventId`; periodic reconciliation pull as fallback when telemetry is lost. +- Site SQLite hot-path first, then gRPC telemetry to central; ingest is idempotent on `EventId`; periodic reconciliation pull as fallback when telemetry is lost. **The site audit DB is a separate, unreplicated SQLite file — NOT one of LocalDb's ten consolidated tables** — at the **required** `AuditLog:SiteWriter:DatabasePath` (arch-review WP1.2, finding #2 High), `/app/data/auditlog.db` on the rig; `StartupValidator` refuses to boot a Site node without it. `DatabasePath` used to default to CWD-relative `auditlog.db`, which on the docker rig landed on the container's ephemeral overlay, not the mounted volume — the same data-loss pattern the pre-LocalDb legacy S&F/tracking databases hit (see the LocalDb bullet above), independently rediscovered here because this file sits outside LocalDb's management. The writer loop now also honors `FlushIntervalMs` (was validated but never read — one fsync'ing commit per event even at trickle rate) and sets `PRAGMA synchronous = NORMAL` (audit is best-effort by design, so the narrower power-loss window is an acceptable trade for far fewer fsyncs; WAL still guarantees no corruption). - Cached operations: site emits a single additively-extended `CachedCallTelemetry` packet carrying both audit events and operational state; central writes `AuditLog` + `SiteCalls` in one transaction. - Payload cap 8 KB by default / 64 KB on error rows; auth headers redacted by default; SQL parameter values captured by default; per-target redaction opt-in. Inbound API: full verbatim capture up to `InboundMaxBytes` (default 1 MiB); request headers stored in `Extra.requestHeaders` (post-redaction); per-method `SkipBodyCapture` flag suppresses bodies while still recording headers + metadata; `AuditInboundCeilingHits` counter surfaced on health snapshot. (M5.3 T7) - Audit-write failure NEVER aborts the user-facing action — audit is best-effort, the action's own success/failure path is authoritative. @@ -213,6 +215,7 @@ spec for each is `docs/requirements/Component-.md`, and `README.md` carrie ## Tool Usage - When a task requires setting up or controlling system state (sites, templates, instances, data connections, deployments, security, etc.) and the Central UI is not needed, prefer the ScadaBridge CLI over manual DB edits or UI navigation. See [`src/ZB.MOM.WW.ScadaBridge.CLI/README.md`](src/ZB.MOM.WW.ScadaBridge.CLI/README.md) for the full command reference. +- **CLI HTTP timeouts are honest (arch-review remediation, finding #1 High).** `ManagementHttpClient` used to cap every call at `min(30s, caller timeout)`, silently truncating `deploy site`'s 5-minute `BulkDeployTimeout` and the 5-minute bundle export/preview/import calls — the CLI printed a fake `504 Request timed out` while the server kept working to completion. `HttpClient.Timeout` is now `Timeout.InfiniteTimeSpan`; the per-call `CancellationTokenSource` is the single overall deadline (connect phase included), with connect bounded separately via `SocketsHttpHandler.ConnectTimeout` (env override renamed `SCADABRIDGE_HTTP_CONNECT_TIMEOUT_SECONDS` to match its narrower new meaning). ### CLI Quick Reference (Docker / OrbStack) diff --git a/Directory.Packages.props b/Directory.Packages.props index b06434db..80ee60d0 100644 --- a/Directory.Packages.props +++ b/Directory.Packages.props @@ -109,9 +109,9 @@ - - - + + + AA2 IA1 --> NAA1 - SA1 --> SEA1 - SA2 --> SEA2 - AA1 --> AEA1 + SA1 -.-> SEA1 + SA2 -.-> SEA2 + AA1 -.-> AEA1 IA2 -.-> IA2CHILD @@ -96,9 +96,11 @@ flowchart TD > **Startup config load is retried (S5)**: step 1's deployed-config read is best-effort against local SQLite, which can transiently fail (locked/busy). A failed load must **not** leave the site as a silent zero-instance node — it is re-attempted on a fixed interval until it succeeds, then the retry self-cancels. -> **Per-instance compilation during staggered startup (P6, follow-up)**: the Roslyn compile of each instance's scripts still runs inside Instance Actor start during staggered startup; moving it off-thread is a deferred optimization (it affects failover time-to-recover only, not correctness). As of the round-2 hardening, a process-wide compile cache dedupes identical script bodies within a node's process lifetime (the deploy gate's compile is reused by Instance Actor start), shrinking the recompile cost; the *first* compile after process start still runs inside Instance Actor start, so the deferral stands. +> **Per-instance compilation during staggered startup — CLOSED (WP3.1)**: this was a deferred optimization (P6) affecting failover time-to-recover; each Instance Actor Roslyn-compiled its own scripts inside its own start, serialising a site's whole recovery behind compilation. Each staggered startup batch now runs a **pre-warm step**: before the batch's Instance Actors are created, the Deployment Manager compiles that batch's distinct script bodies and trigger expressions on a background task and pipes `BatchCompileWarmed` back, so every `PreStart` compile is a cache hit. Instance Actor start still calls the compilation service — it remains the correctness backstop — but in every warmed path those calls are memoised lookups. Cost: one extra message per batch. -> **The Roslyn `ScriptOptions` are process-static, and must stay that way**: the reference set backing every site-side compile is built once per process, never per compile. `ScriptOptions.WithReferences(Assembly[])` resolves each assembly through `MetadataReference.CreateFromFile`, which does **not** cache — each call mints a fresh `MetadataReference` owning an `AssemblyMetadata` → `PEReader` → `NativeHeapMemoryBlock`, an unmanaged copy of the assembly metadata that nothing disposes. Building the options per compile leaks native memory permanently: no GC reclaims it, and it is invisible to gcdump and to the managed allocation counters, so the node's working set grows without the GC heap growing. This was a live defect (a Site node at 2,885 MB working set with 150 MB of live GC heap, ~2,469 MB of it on the default process heap across ~6,700 undisposed `AssemblyMetadata` instances). Note this is orthogonal to the compile cache above — the cache dedupes *identical* script bodies, so it bounds nothing when the bodies differ, and it clears wholesale on overflow, after which every script recompiles. Pinned by a reference-equality regression test rather than by watching memory. +> **The Roslyn `ScriptOptions` are process-static, and must stay that way**: the reference set backing every site-side compile is built once per process, never per compile. `ScriptOptions.WithReferences(Assembly[])` resolves each assembly through `MetadataReference.CreateFromFile`, which does **not** cache — each call mints a fresh `MetadataReference` owning an `AssemblyMetadata` → `PEReader` → `NativeHeapMemoryBlock`, an unmanaged copy of the assembly metadata that nothing disposes. Building the options per compile leaks native memory permanently: no GC reclaims it, and it is invisible to gcdump and to the managed allocation counters, so the node's working set grows without the GC heap growing. This was a live defect (a Site node at 2,885 MB working set with 150 MB of live GC heap, ~2,469 MB of it on the default process heap across ~6,700 undisposed `AssemblyMetadata` instances). Note this is orthogonal to the compile cache above — the cache dedupes *identical* script bodies, so it bounds nothing when the bodies differ. (It no longer clears wholesale on overflow: WP3.1 replaced that latency cliff with approximate-LRU batch eviction, see **Compile cache eviction** below.) Pinned by a reference-equality regression test rather than by watching memory. + +> **Compile cache eviction is approximate LRU, not a wholesale clear (WP3.1)**: `SiteScriptCompileCache` is bounded at 1024 entries — deliberately smaller than the TemplateEngine verdict cache's 4096, because entries pin compiled assemblies rather than verdict strings. It used to `Clear()` **everything** on overflow. Instance scripts *and* trigger expressions share this cache, so on a site large enough to cross the bound every overflow discarded up to 1023 live compiled scripts and the next deploy or Instance-Actor start paid a full recompile storm **on actor threads**. Overflow now evicts only the oldest ⅛ (128 entries) by last-access stamp: one 1024-element scan instead of 1023 future recompiles, and hot entries survive. Recency is an `Interlocked` access sequence rather than a clock — deterministic for tests and immune to clock steps; hits update it lock-free, and only the (rare) eviction sweep takes a lock, double-checked so concurrent inserts do not stampede it. The sweep snapshots through `ConcurrentDictionary.ToArray()` and **not** LINQ over the dictionary: LINQ's `ToArray` picks the `ICollection.CopyTo` fast path, which throws when a concurrent insert lands between its Count read and its copy, and inserts are deliberately lock-free. > **Static options were only half the fix — the compile must also use the shared caching metadata resolver**: the static `ScriptOptions` carry only the *direct* API-surface references (5 assemblies here), and each `script.Compile()` still binds their **transitive closure** (~105 assemblies on a site node). Every transitively-referenced assembly is resolved through the options' `MetadataReferenceResolver`, and Roslyn's default one has no cross-compilation cache — it calls `MetadataReference.CreateFromFile` afresh on **every compile**, minting the same undisposed `AssemblyMetadata` → `PEReader` → `NativeHeapMemoryBlock` triple per assembly per compiled script, pinned for the process lifetime by the compile cache. That per-compile *re-resolution* — not the per-compile options construction — was the dominant term (measured live: 21 site scripts holding 6,640 metadata objects, counts bit-identical before and after the static-options fix). `ScriptCompilationService` therefore also attaches the process-wide `CachingScriptMetadataResolver.Instance` (owned by the Script Analysis component, see `Component-ScriptAnalysis.md`) via `ScriptOptions.WithMetadataResolver`, so each distinct assembly is materialized once per process. The decorator returns exactly what the undecorated resolver returned — only object identity is de-duplicated — so compile results and the trust gate are unchanged. @@ -171,7 +173,7 @@ flowchart TD ### Debug View Support - On request from central (via Communication Layer), the Instance Actor provides a **snapshot** of all current attribute values and alarm states. - Subsequent changes are delivered via the **SiteStreamManager** → **SiteStreamGrpcServer** → gRPC stream to central. The Instance Actor publishes attribute value and alarm state changes to the SiteStreamManager; it does not forward events directly to the Communication Layer. -- The Instance Actor also handles one-shot `DebugSnapshotRequest` messages: it builds the same snapshot (attribute values and alarm states) and replies directly to the sender. Unlike `SubscribeDebugViewRequest`, no subscriber is registered and no stream is established. +- The Instance Actor also handles one-shot `DebugSnapshotRequest` messages: it builds the same snapshot (attribute values and alarm states) and replies directly to the sender. Unlike `SubscribeDebugViewRequest`, no subscriber is registered and no stream is established. When the request sets `AlarmsOnly` (WP2.3 — the central live alarm cache's seed/reconcile fan-out, which discards attribute rows), only the alarm half is built and `AttributeValues` comes back empty; the flag defaults to `false`, so the Debug View path is unchanged. ### Supervision Strategy @@ -179,10 +181,10 @@ The Instance Actor supervises all child Script and Alarm Actors with explicit st | Child Actor | Exception Type | Strategy | Rationale | |-------------|---------------|----------|-----------| -| Script Actor | Any exception | Resume | Script Actor is a coordinator — its state (trigger timers, last execution time) should survive child failures. Script Execution Actor failures are isolated. | +| Script Actor | Any exception | Resume | Script Actor is a coordinator — its state (trigger timers, last execution time) should survive run failures. | | Alarm Actor | Any exception | Resume | Alarm Actor holds alarm state. Resume preserves state and continues evaluation on next value update. | -| Script Execution Actor | Unhandled exception | Stop | Short-lived, per-invocation. Failure is logged; the Script Actor coordinator remains active for future triggers. | -| Alarm Execution Actor | Unhandled exception | Stop | Short-lived, per on-trigger invocation. Same as Script Execution Actor. | + +Script Actor and Alarm Actor have **no children of their own** since WP3.1 removed the per-run execution actors, so neither declares a supervision strategy any more. A run's own exceptions were always contained inside the run's try/catch (they never reached supervision); a failure of the *launch* is caught by the coordinator, which answers the caller and releases the run slot rather than letting the throw escalate to the Instance Actor. The Deployment Manager singleton supervises Instance Actors with a **OneForOneStrategy** — one Instance Actor's failure does not affect other instances. @@ -196,11 +198,11 @@ When the Instance Actor is stopped (due to disable, delete, or redeployment), Ak - **Coordinator** for a single script definition on an instance. - Holds the compiled script code and trigger configuration. - Manages trigger evaluation (interval timer, value change detection, conditional evaluation). -- Spawns short-lived Script Execution Actors for each invocation. +- Launches script runs directly for each invocation (no per-run child actor since WP3.1). ### Trigger Management -- **Interval**: The Script Actor manages an internal timer. When the timer fires, it spawns a Script Execution Actor. -- **Value Change**: The Script Actor subscribes to attribute change notifications from its parent Instance Actor for the specific monitored attribute. When the attribute changes, it spawns a Script Execution Actor. +- **Interval**: The Script Actor manages an internal timer. When the timer fires, it launches a run. +- **Value Change**: The Script Actor subscribes to attribute change notifications from its parent Instance Actor for the specific monitored attribute. When the attribute changes, it launches a run. - **Conditional**: The Script Actor subscribes to attribute change notifications for the monitored attribute. On each update, it evaluates the condition (compares the attribute against a threshold). Firing depends on the **fire mode** (see below). - **Expression**: The Script Actor evaluates a compiled boolean expression against an attribute snapshot on each attribute change. Firing depends on the **fire mode** (see below). - **Fire mode (Conditional + Expression)**: @@ -211,20 +213,24 @@ When the Instance Actor is stopped (due to disable, delete, or redeployment), Ak > **Per-attribute change routing (P2).** Rather than broadcasting every attribute change to all child Script/Alarm Actors, the Instance Actor routes each change only to the children that monitor that attribute (built from their trigger configs at spawn time); **Expression-trigger children, which read an attribute snapshot, receive all changes**. Each child still keeps its own trigger gate as defense-in-depth — the routing is an optimization, not the authority. ### Concurrent Execution -- Each invocation spawns a **new Script Execution Actor** as a child. -- Multiple Script Execution Actors can run concurrently (e.g., a trigger fires while a previous `Instance.CallScript` invocation is still running). -- The Script Actor coordinates but does not block on child completion. +- Each invocation launches a **new run** on the script-execution scheduler. +- Multiple runs can execute concurrently (e.g., a trigger fires while a previous `Instance.CallScript` invocation is still running), up to `MaxConcurrentRunsPerScript` — beyond which the newest is shed (see below). +- The Script Actor coordinates but does not block on run completion. -### Script Execution Actor -- **Short-lived** child actor created per invocation. -- Receives: compiled script code, input parameters, reference to the parent Instance Actor, current call depth. -- Executes the script in the Akka actor context. -- Has access to the full Script Runtime API (see below). -- Returns the script's return value (if defined) to the caller, then stops. -- The script body itself runs on the **dedicated `ScriptExecutionScheduler`** (a bounded set of dedicated threads), not the shared .NET thread pool, so blocking script I/O cannot starve the global pool or stall Akka dispatchers. The scheduler is **process-wide by default** (one pool per host, sized from `SiteRuntimeOptions.ScriptExecutionThreadCount`), but each script/alarm actor takes it through an **optional injection seam** rather than reaching for the static directly: the Host injects nothing and gets the shared pool, while tests (or a future multi-site host) can hand an actor its own instance. The shared accessor also recreates a disposed pool rather than returning it, so a disposed scheduler can never silently poison later executions. +### Script Run Launch (WP3.1 — replaces the Script Execution Actor) +- A run is launched **directly by the Script Actor** through `ScriptRunLauncher`. There is no per-run child actor. + - The former `ScriptExecutionActor` / `AlarmExecutionActor` were already inert shells: neither declared a `Receive` handler (they executed from their constructor), neither had a `PostStop`, state, or stash, and neither's `IActorRef` was ever a message target — the whole lifecycle lived inside a detached task the actor never observed. What they cost was an actor cell, mailbox, and name registration **per run**, plus a per-spawn expression-tree `Props.Create`. Removing them changed no semantics; the run body moved verbatim into the shared launcher. + - Everything the shells provided is preserved: exception and timeout containment, one DI scope per run disposed on every path, the site-event/health telemetry, the Ask reply, the completion notification (now the coordinator's own `Self`), and the audit `ExecutionId` / `ParentExecutionId` threading. A run still in flight when its Script Actor is stopped runs to completion and its completion message dead-letters, exactly as before — **stopping does NOT cancel in-flight runs**; redeploy/undeploy semantics are unchanged. + - **Per-run Started/Completed site events are sampled, not unconditional (WP3.2).** The two Info rows an instance script run used to emit unconditionally were the dominant `site_events` writer under load (design memo: `docs/plans/2026-08-15-site-events-policy-design.md`). They are now **off by default** (`ScadaBridge:SiteRuntime:PerRunScriptEvents`, default `false`); a `ScriptRunSummaryRecorder` accumulates per-`(instance, script)` run counters from the same call sites and a site-only hosted flush service emits **one aggregate "script" Info row per interval** (`ScriptRunSummaryIntervalSeconds`, default 300s) with a top-50-scripts breakdown, capped so a pathological deployment cannot mint an oversized row — an idle interval emits nothing. `ScadaBridge:SiteRuntime:PerRunScriptEventScripts` restores the legacy per-run rows for named scripts (`"Instance/Script"` exact or `"Instance/*"` wildcard) for live debugging. All three keys are read from `IOptionsMonitor` per run, so they are hot-togglable without a restart. **Error-level rows (timeout, failure, stuck-watchdog, recursion-limit) are unaffected — always emitted, regardless of this policy.** See Component-SiteEventLogging.md → Volume Policy for the full design. + - One deliberate improvement: a failure of the *launch itself* (e.g. queueing onto a disposed scheduler) is caught by the Script Actor, which replies to the Ask caller and releases the run slot. The old per-run child's constructor throw was handled by a Stop supervision directive that sent no reply, leaving the caller to hang to its Ask timeout. +- The script body runs on the **dedicated `ScriptExecutionScheduler`** (a bounded set of dedicated threads), not the shared .NET thread pool, so blocking script I/O cannot starve the global pool or stall Akka dispatchers. The scheduler is **process-wide by default** (one pool per host), but each script/alarm actor takes it through an **optional injection seam** rather than reaching for the static directly: the Host injects nothing and gets the shared pool, while tests (or a future multi-site host) can hand an actor its own instance. The shared accessor also recreates a disposed pool rather than returning it, so a disposed scheduler can never silently poison later executions. +- **Pool sizing is instance-scaled and grow-only (WP3.1).** The pool was a fixed 8 threads regardless of load. It is now `clamp(max(ScriptExecutionThreadCount, ceil(enabledInstances / 8)), 1, ScriptExecutionMaxThreadCount)` — the existing `ScriptExecutionThreadCount` (default 8) becomes the **floor**, so configurations at or below 64 instances behave exactly as before, and the new `ScriptExecutionMaxThreadCount` (default 32) is the ceiling. Beyond the ceiling the per-script cap below is the real regulator. `DeploymentManagerActor.UpdateInstanceCounts` calls `EnsureCapacity` on every deploy / undeploy / enable / disable and once per staggered startup batch. Growth only: undeploying leaves idle threads, which cost nothing measurable and avoid drain/steal complexity. +- **The deadline is armed at enqueue, not at dequeue (WP3.1).** The run's timeout `CancellationTokenSource` is created on the actor thread *before* the body is queued, so queue wait consumes the script's own budget. A body that dequeues past its deadline **skips execution entirely** and takes the existing timeout path (site event, script-error counter, error reply, completion message): a saturated pool sheds stale work instead of running it late with a fresh full budget. +- **Concurrent runs per script are capped (WP3.1).** `MaxConcurrentRunsPerScript` (default 4) bounds runs in flight — queued or executing — for any one script or alarm on-trigger script. Over the cap the **newest** run is shed: the four already in flight are closest to their own deadlines and already charged against them, so nothing is ever reordered and no extra queue is needed (the scheduler's FIFO already is the queue). A shed increments `ISiteHealthCollector.IncrementScriptRunShed` (surfaced as `ScriptRunShedCount`), emits a `script`/`Warning` site event **rate-limited to one per script per minute** so a hot trigger cannot flood `site_events`, and — for an Ask-based `CallScript` — replies with an explicit error so a nested call or inbound-API route fails fast instead of hanging. + - **The cap gates only depth-0 launches (review fix).** A trigger fire or a depth-0 `CallScript`/inbound-API route counts against `MaxConcurrentRunsPerScript`; a *nested* `Instance.CallScript` self-recursion (`callDepth > 0`) is exempt — it is already bounded by `MaxScriptCallDepth` instead, and the calling run is holding one of the four cap slots itself while it awaits the callee, so counting the nested launch too would spuriously shed legitimate self-recursion. ### Handling `Instance.CallScript` -- When an external caller (another Script Execution Actor, an Alarm Execution Actor, or a routed call from the Inbound API) sends a `CallScript` message to the Script Actor, it spawns a Script Execution Actor to handle the call. +- When an external caller (another script run, an alarm on-trigger run, or a routed call from the Inbound API) sends a `CallScript` message to the Script Actor, it launches a run to handle the call. - The caller uses the **Akka ask pattern** and receives the return value when the execution completes. --- @@ -244,10 +250,10 @@ When the Instance Actor is stopped (due to disable, delete, or redeployment), Ak - **Range Violation**: Value is outside the allowed min/max range. - **Rate of Change**: Value change rate exceeds the defined threshold over a configurable time window. Direction filter (rising / falling / either) restricts which side of the rate triggers. - **HiLo**: Multi-setpoint level alarm with up to four configurable setpoints (LoLo, Lo, Hi, HiHi). Any subset may be configured. Each setpoint may carry its own priority that overrides the alarm-level priority for that band. -- **Expression** trigger evaluation runs on the bounded script-execution thread pool shared with script bodies (and Expression *script* triggers): scheduler saturation — e.g. stuck scripts occupying all threads — delays Expression alarm transitions site-wide until a thread frees. This is a deliberate trade-off (evaluation no longer blocks dispatcher threads; per-actor coalescing bounds queue growth) and the scheduler gauges + stuck-script watchdog make the saturated state visible. +- **Expression** trigger evaluation runs on the shared .NET thread pool behind a process-wide concurrency gate (`TriggerEvalGate`, sized by `TriggerEvalMaxConcurrency`, default `max(2, ProcessorCount)`) — **not** on the bounded script-execution pool. This is the WP3.1 fix for arch-review finding #4 (High): when evaluation shared that pool, N script bodies blocked in synchronous I/O stalled *every* Expression trigger on the node — scripts and alarms alike — for an unbounded time, and the evaluation's own 2 s timeout was constructed inside the queued body, so it did not start ticking until dequeue. An alarm that should have raised in milliseconds simply never raised, with neither a raise nor a timeout visible to the operator. Trigger expressions are non-blocking **by construction** (`TriggerExpressionGlobals` exposes only reads over an in-memory snapshot, and the script trust gate has already denied I/O, network, threading, and reflection), so the shared pool is where they belong; a second dedicated pool was considered and rejected as adding threads, gauges, and a second starvation surface for no isolation gain. The evaluation deadline (`TriggerEvalTimeoutSeconds`, default 2, previously hardcoded) is now armed **at enqueue**, so gate-wait time burns the same budget and a saturated gate yields a timely `false` rather than an unbounded stall. Per-actor coalescing (one evaluation in flight, one pending) is unchanged and caps waiters at one per Expression trigger, so the gate queue is bounded by trigger count. - For binary trigger types (ValueMatch / RangeViolation / RateOfChange), when the condition is met and the alarm is currently in **normal** state, the alarm transitions to **active**: - Updates the alarm state on the parent Instance Actor (which publishes to the Akka stream). - - If an on-trigger script is defined, spawns an Alarm Execution Actor to execute it. + - If an on-trigger script is defined, launches its run via `ScriptRunLauncher` (see Alarm On-Trigger Run below) — no per-run child actor. - When the condition clears and the alarm is in **active** state, the alarm transitions to **normal**. - For HiLo triggers, the actor tracks the current `AlarmLevel` (None / Low / LowLow / High / HighHigh). Each level transition emits a fresh `AlarmStateChanged` with the new level and its priority; level escalations (e.g., High → HighHigh) and de-escalations (HighHigh → High) both produce events. The on-trigger script fires only on the Normal → non-None edge, not on escalations between alarm bands. - No script execution on clear in any trigger type. @@ -256,9 +262,8 @@ When the Instance Actor is stopped (due to disable, delete, or redeployment), Ak - Held **in memory** only — not persisted to SQLite. State comprises `AlarmState` (Active / Normal) and `AlarmLevel` (None for binary triggers; the active band for HiLo). - On restart (or failover), alarm states are re-evaluated from incoming values. All alarms start in normal state with level None and transition based on incoming values. -### Alarm Execution Actor -- **Short-lived** child actor created when an on-trigger script needs to execute. -- Same pattern as Script Execution Actor — receives compiled code, executes, returns, and stops. +### Alarm On-Trigger Run +- Launched directly by the Alarm Actor when an on-trigger script needs to execute — the same `ScriptRunLauncher` path as an instance script run, with the firing alarm's name/level/priority/message exposed through the `Alarm` global. Bounded by the same `MaxConcurrentRunsPerScript` cap. - Has access to the Instance Actor for `GetAttribute`/`SetAttribute`. - **Can** call instance scripts via `Instance.CallScript()` — sends an ask message to the appropriate sibling Script Actor. - Instance scripts **cannot** call alarm on-trigger scripts — the call direction is one-way. @@ -275,7 +280,7 @@ change: |---|---| | `ScriptRuntimeContext.SetAttribute` (this run's `ExecutionId`) / `RouteToSetAttributesRequest.ParentExecutionId` (the inbound request's) | → `SetStaticAttributeCommand.SourceExecutionId` | | Instance Actor static-write path (`HandleSetStaticAttributeCore`) | → `AttributeValueChanged.SourceExecutionId` | -| Alarm Actor trigger evaluation → `SpawnAlarmExecution` | → `AlarmExecutionActor` → `ScriptRuntimeContext.ParentExecutionId` | +| Alarm Actor trigger evaluation → `SpawnAlarmExecution` | → `ScriptRunLauncher.LaunchAlarmScript` → `ScriptRuntimeContext.ParentExecutionId` | All four computed trigger types participate. `Expression` triggers evaluate a whole attribute snapshot **off the dispatcher**, so the firing change is no @@ -382,7 +387,7 @@ The enriched message flows Instance Actor → site-wide Akka stream → `SiteStr - Shared scripts are compiled at the site when received from central. - Compiled code is stored in memory and made available to all Script Actors. -- When a Script Execution Actor calls `Scripts.CallShared("scriptName", params)`, the shared script code executes **inline** in the Script Execution Actor's context — it is a direct method invocation, not an actor message. +- When a script run calls `Scripts.CallShared("scriptName", params)`, the shared script code executes **inline** in that run's context — it is a direct method invocation, not an actor message. - This avoids serialization bottlenecks since there is no shared script actor to contend for. - Shared scripts have access to the same runtime API as instance scripts (GetAttribute, SetAttribute, external systems, notifications, databases). - **Shared scripts always execute with Root scope (C5)**: `Attributes[...]` inside a shared script addresses **root** attributes even when the shared script is invoked from a composed-module script. Callers needing module-scoped access must pass the values in as parameters. @@ -417,14 +422,14 @@ All script types can be updated without restarting the cluster, but the mechanis ## Script Runtime API -Available to all Script Execution Actors and Alarm Execution Actors: +Available to all script runs and alarm on-trigger runs: ### Instance Attributes - `Instance.GetAttribute("name")` — Read an attribute value from the parent Instance Actor. - `Instance.SetAttribute("name", value)` — Write an attribute value. For data-connected attributes, writes to the DCL; for static attributes, updates in-memory and persists to local SQLite (survives restart/failover, reset on redeployment). ### Other Scripts -- `Instance.CallScript("scriptName", parameters)` — Send an ask message to a sibling Script Actor. The target Script Actor spawns a Script Execution Actor, executes, and returns the result. The call includes the current recursion depth. +- `Instance.CallScript("scriptName", parameters)` — Send an ask message to a sibling Script Actor. The target Script Actor launches a run, executes, and returns the result (or an explicit shed error if that script is already at its concurrent-run cap). The call includes the current recursion depth. - `Scripts.CallShared("scriptName", parameters)` — Execute shared script code inline (direct method invocation). The call includes the current recursion depth. ### External Systems @@ -474,7 +479,7 @@ Scripts execute **in-process** with constrained access. The following restrictio - **Allowed**: Access to the Script Runtime API (GetAttribute, SetAttribute, CallScript, CallShared, ExternalSystem, Notify, Database, Tracking, Alarms), standard C# language features, basic .NET types (collections, string manipulation, math, date/time). `System.Diagnostics.Stopwatch`, `Debug`, and `Activity` are permitted. - **Forbidden**: File system access (`System.IO`), process spawning (`System.Diagnostics.Process`), threading (`System.Threading` — except `Tasks`, `CancellationToken`, and `CancellationTokenSource`), reflection (`System.Reflection`), all raw network access (`System.Net` — must use `ExternalSystem.Call`), native interop (`System.Runtime.InteropServices`, `Microsoft.Win32`), assembly loading, unsafe code, `dynamic`, `Activator`. -- **Execution timeout**: Configurable per-script maximum execution time. Exceeding the timeout cancels the script **cooperatively** (S2/UA5): a script blocked in synchronous I/O or a tight CPU loop does not observe cancellation and continues to occupy its dedicated script-execution thread. A **watchdog logs the script by name** once its thread has not returned within a grace period after the timeout, and the script-execution scheduler's **queue depth / busy-thread / oldest-busy-age gauges** are reported through Health Monitoring so a saturated or stuck scheduler is visible. +- **Execution timeout**: Configurable per-script maximum execution time, **measured from enqueue** (WP3.1) — a run that spends its whole budget queued behind blocked scripts is shed at dequeue without executing, rather than being granted a fresh full budget for work whose triggering condition is already stale. Exceeding the timeout cancels the script **cooperatively** (S2/UA5): a script blocked in synchronous I/O or a tight CPU loop does not observe cancellation and continues to occupy its dedicated script-execution thread. A **watchdog names the script** once its thread has not returned within a grace period after the timeout — and, as of WP3.1, **detaches and replaces that worker thread**, so a wedged script costs the pool a thread only until the watchdog fires rather than for the rest of the process. The detached worker exits (instead of pulling more work) when its body finally returns, so capacity never silently doubles. Live detached workers are capped at the pool size — at the cap no replacement is started and an Error site event says so, because bounded starvation is preferable to unbounded thread growth when scripts wedge en masse. The scheduler's **queue depth / busy-thread / oldest-busy-age / detached-thread gauges** and the **per-interval shed count** are reported through Health Monitoring so a saturated, stuck, or shedding scheduler is visible. - **Memory**: Scripts share the host process memory. No per-script memory limit, but the execution timeout prevents runaway allocations. The forbidden-API policy is defined authoritatively in `ScriptTrustPolicy` (Script Analysis component, #25). `ScriptCompilationService.ValidateTrustModel` delegates to `ScriptTrustValidator.FindViolations` for the trust verdict; the Site Runtime also performs a `CSharpScript.Compile` against the real `ScriptGlobals` for execution. This is defence-in-depth static enforcement, not a true runtime sandbox. @@ -503,14 +508,14 @@ Per Akka.NET best practices, internal actor communication uses **Tell** (fire-an **Ask** is reserved for system boundaries where a synchronous response is needed: -- **`Instance.CallScript()`**: Ask pattern from Script Execution Actor to sibling Script Actor. The caller needs the return value. Acceptable because script calls are infrequent relative to tag updates. +- **`Instance.CallScript()`**: Ask pattern from the running script (via `ScriptRuntimeContext.CallScript`) to sibling Script Actor. The caller needs the return value. Acceptable because script calls are infrequent relative to tag updates. - **`Route.To().Call()`**: Ask from Inbound API to site Instance Actor via Communication Layer. External caller needs a response. - **Debug view snapshot**: Ask from Communication Layer to Instance Actor for initial state. ## Concurrency & Serialization -- The Instance Actor processes messages **sequentially** (standard Akka actor model). This means `SetAttribute` calls from concurrent Script Execution Actors are serialized at the Instance Actor, preventing race conditions on attribute state. -- Script Execution Actors may run concurrently, but all state mutations (attribute reads/writes, alarm state updates) are mediated through the parent Instance Actor's message queue. +- The Instance Actor processes messages **sequentially** (standard Akka actor model). This means `SetAttribute` calls from concurrent script runs are serialized at the Instance Actor, preventing race conditions on attribute state. +- Script runs (launched via `ScriptRunLauncher`, no per-run child actor) may run concurrently, but all state mutations (attribute reads/writes, alarm state updates) are mediated through the parent Instance Actor's message queue. - External side effects (external system calls, notifications, database writes) are not serialized — concurrent scripts may produce interleaved side effects. This is acceptable because each side effect is independent. ## SiteStreamManager and gRPC Integration @@ -528,7 +533,7 @@ Per Akka.NET best practices, internal actor communication uses **Tell** (fire-an ## Error Handling ### Script Errors -- Unhandled exceptions and timeouts in Script Execution Actors are **logged locally** to the site event log. +- Unhandled exceptions and timeouts in a script run are **logged locally** to the site event log — unconditionally, regardless of the WP3.2 per-run Started/Completed sampling policy (see Script Run Launch above). - The Script Actor (coordinator) is **not affected** — it remains active for future trigger events. - Script failures are **not reported to central** (except as aggregated error rate metrics via Health Monitoring). @@ -540,7 +545,7 @@ Per Akka.NET best practices, internal actor communication uses **Tell** (fire-an ### Script Compilation Errors - If script compilation fails when a deployment is received, the entire deployment for that instance is **rejected**. No partial state is applied. - The failure is reported back to central as a failed deployment. -- **Enforced by a site-side pre-compile validation gate (S3)**: before the Instance Actor is created or the config persisted, the Deployment Manager compiles **all** of the instance's scripts, alarm on-trigger scripts, and trigger expressions **synchronously, as a pure prefix step of the deploy handler on the singleton's actor thread** — deliberately not off-thread, because piping the verdict back to self would reorder concurrent deploys against delete/disable and break the redeploy-supersede mailbox-FIFO ordering (see the in-code rationale in `DeploymentManagerActor.HandleDeploy`); unchanged script bodies hit the process-wide compile cache, so the gate recompiles only genuinely new code. Any compile failure rejects the deployment with the collected compile errors. This makes the deployment result honest even if central's pre-deployment validation was bypassed or skew let a bad artifact through. +- **Enforced by a site-side pre-compile validation gate (S3)**: before the Instance Actor is created or the config persisted, the Deployment Manager compiles **all** of the instance's scripts, alarm on-trigger scripts, and trigger expressions **synchronously, as a pure prefix step of the deploy application on the singleton's actor thread** — the verdict is never derived off-thread, because piping a verdict back to self would reorder concurrent deploys against delete/disable and break the redeploy-supersede mailbox-FIFO ordering. WP3.1 made it **warm-then-gate**: the same compile is first run off-thread purely to populate the process-wide compile cache (`DeployCompileWarmed` pipes back), so the authoritative synchronous gate is then all cache hits and no longer holds the singleton for a Roslyn compile. Ordering is preserved by a **per-instance in-flight guard** — while a warm is in flight for instance X, further mutating commands for X (deploy / delete / disable / enable) are queued and re-dispatched in arrival order once the deploy applies; a second deploy with nothing queued behind it supersedes the pending one last-write-wins and its displaced deployer is answered `Failed`-superseded rather than left to Ask-timeout. Commands for **other** instances flow freely, which is the point, and is safe because cross-instance ordering was never guaranteed to callers. Unchanged script bodies hit the compile cache, so the gate recompiles only genuinely new code. Any compile failure rejects the deployment with the collected compile errors. This makes the deployment result honest even if central's pre-deployment validation was bypassed or skew let a bad artifact through. - Note: Pre-deployment validation at central should still catch compilation errors before they reach the site; the site gate is the authoritative backstop. --- diff --git a/docs/requirements/Component-StoreAndForward.md b/docs/requirements/Component-StoreAndForward.md index 8bc3edf3..58054c1b 100644 --- a/docs/requirements/Component-StoreAndForward.md +++ b/docs/requirements/Component-StoreAndForward.md @@ -108,6 +108,12 @@ Notifications are unaffected: they have no tracking table. Their `NotificationId On every tracking-table status transition, the site emits a `CachedCallTelemetry` message to the central Site Call Audit component over the site→central channel. Emission is best-effort, at-least-once, and idempotent on `TrackedOperationId`. Because telemetry is best-effort, the site also responds to `CachedCallReconcileRequest` reconciliation pulls — cursor-based per-site reads of tracking rows changed since a cursor — so any missed telemetry self-heals. The site never depends on central; central converges to the site. +The telemetry-emitting `ICachedCallLifecycleObserver` hook is dispatched through a bounded, single-reader **observer queue** (`StoreAndForwardOptions.ObserverQueueCapacity`, default 10,000, `DropOldest`) — the one unbounded `Channel` left in the system before this bound was added (arch-review WP2.6c). A slow or stuck observer (e.g. a SQLite audit write wedged behind disk contention) can no longer grow this queue without limit; it instead sheds the oldest unprocessed notification, incrementing an `ObserverQueueDroppedCount` counter that counts every drop. **Logging is separately rate-limited (adversarial review finding F3):** a Warning fires for the first drop of an episode, then at most one rollup Warning per minute while drops keep happening — never one Warning-per-dropped-item, which would itself have been a log-flood risk under the exact sustained-drop condition the bound exists to survive. This bounds memory, not delivery: telemetry loss here is covered by the reconciliation pull above, same as any other missed telemetry. + +### Retry Sweep Indexing + +`GetMessagesForRetryAsync` orders candidates `ORDER BY created_at ASC` within the due `status`. The existing `idx_sf_messages_status_due (status, last_attempt_at_ms)` matches the status filter and due-time predicate but not this ordering, forcing a sort/scan on a large sweep. A second covering index, `idx_sf_messages_status_created (status, created_at)`, lets the sweep walk matching rows already in `created_at` order and stop at the batch limit without re-sorting or touching non-pending rows (arch-review WP1.4). Both indexes are retained — `idx_sf_messages_status_due` still backs status+due-time lookups that don't order by `created_at`. + ## Parked Message Management - Parked messages remain stored at the site in SQLite. diff --git a/docs/requirements/Component-TemplateEngine.md b/docs/requirements/Component-TemplateEngine.md index 9b7abf88..b4160753 100644 --- a/docs/requirements/Component-TemplateEngine.md +++ b/docs/requirements/Component-TemplateEngine.md @@ -137,6 +137,72 @@ When an instance is deployed, the Template Engine resolves the full configuratio 5. Resolve data connection bindings — replace connection name references with concrete connection details from the site. 6. Output a flat structure: list of attributes with resolved values and data source addresses, list of alarms with resolved trigger definitions, list of scripts with resolved code and triggers. +### Flatten-session caching and the graph watermark (WP2.5) + +The flatten above is driven per instance, so an unmemoised implementation re-walks +the same template chain — one query per link, plus the compositions of every +template it reaches and of every composed chain those reach — once per instance, +and re-issues the three session-global queries (shared scripts, the shared-schema +library, the target site's data connections) each time too. + +A **`FlattenSession`** memoises all of that for the lifetime of ONE +flatten/validate operation. Callers that flatten a batch (`DeploySiteAsync`) pass +one shared session; a caller that passes none gets a private single-use session, +which still collapses the repeated composed-chain loads inside a single instance's +flatten. + +Cache validity is decided by **`ITemplateGraphWatermark`**, a process-wide set of +monotonic counters. `TemplateService` and the `ManagementActor` native-alarm-source +handlers bump it via `TemplateEngineRepository.SaveChangesAsync`'s change-tracker +inspection, which attributes each pending change to its owning template or +instance pre-commit. The Transport bundle importer (`BundleImporter`) commits +through the raw `DbContext` instead — bypassing that repository — so it calls +`ITemplateGraphWatermark.BumpAll()` itself, once per import: after commit on +success, after rollback on failure (a bundle import can touch templates, +instances, and connections in one transaction, so a scoped bump isn't +worthwhile). A data connection edit (`SiteRepository.SaveChangesAsync`) also bumps — via `BumpAll()`, +since a connection has no owning template — because `Protocol`/`PrimaryConfiguration`/ +`BackupConfiguration`/`FailoverRetryCount` are revision-hash inputs (review fix, commit +`e0e4b246`, F2c). All three writers converge on the same watermark, so a cached flatten +still invalidates correctly regardless of which one changed the graph: + +- a memoised **template** is keyed on `(id, template version)`; +- a memoised **chain** is valid only while BOTH the graph's `StructureVersion` + (bumped on add/remove, re-parent, or any composition change — i.e. anything that + can alter chain MEMBERSHIP) and every member's own version are unchanged. Both + halves are load-bearing: structure alone misses an ordinary member edit, member + versions alone miss a re-parent. + +Sessions are short-lived by construction, so the design's "template state is +captured at the time of flatten" guarantee is unchanged — the session narrows the +capture window, it never widens it. + +The watermark is **in-memory and process-local**, deliberately not persisted and +not replicated between central nodes. A restart, a failover, or any mutation simply +misses and falls back to the authoritative full flatten, so the watermark can only +ever cause EXTRA work, never stale work. + +The same watermark backs a **staleness fast path** in `StaleInstanceProbe`: a +previously computed revision hash is reused when the instance's version, the +structure version, and the version of every template that flatten walked are all +unchanged. This matters most where the probe is called per-instance across a whole +bundle import or a fleet-wide staleness sweep. + +### Design-time analysis reads + +The design-time checks — acyclicity (`CycleDetector`), naming collisions +(`CollisionDetector`) and canonical-name resolution (`TemplateResolver`) — read the +whole template graph but write through none of it and read no script bodies. They +use `GetAllTemplatesForAnalysisAsync`: `AsNoTracking`, with `TemplateScript.Code` +projected away. + +The tracked, body-bearing `GetAllTemplatesAsync` remains the read for the two paths +that genuinely need it — the inheritance reconciler (which compares and copies +script bodies and writes through the loaded entities) and the flattener (which must +observe rows an in-flight bundle import has staged on the shared change tracker). +`ReconcileDescendantsAsync` additionally accepts an already-loaded tracked graph so +a caller holding one does not force a second load. + ### Native Alarm Source Resolution The `FlatteningService` resolves native alarm sources alongside alarms, emitting a `ResolvedNativeAlarmSource` (CanonicalName, ConnectionName, SourceReference, ConditionFilter *(optional)*, and `Source` ∈ `Template` | `Inherited` | `Composed` | `Override`) for each. The resolved set is attached to `FlattenedConfiguration.NativeAlarmSources`. @@ -167,6 +233,18 @@ A derived template stores `IsInherited` **placeholder rows** mirroring every mem Both delegate to one **order-independent reconcile** that compares a template's stored inherited rows against the inheritance resolver's effective set (the same precedence + HiLo merge the editor preview and deploy use) and only ever touches `IsInherited` placeholder rows — never an authored override (`IsInherited == false`). Because the resolver ignores placeholder rows when picking winners, reconciling one template never changes what another resolves, so the operation needs no particular ordering. The effective value mirrored into each placeholder matches the staleness comparison key per member type, so after a reconcile the "base changed" banner clears. Reconcile is best-effort housekeeping for the *stored authoring rows*: a deploy always re-resolves the chain fresh regardless, so a not-yet-resynced template still deploys correctly. +### Slim List Projections (WP2.5) + +The `ListTemplates` management command (backing the CLI `template list` and the Central UI +template tree) previously materialised every template's full child graph — attributes, alarms, +script bodies, compositions, native alarm sources — to page it in memory and discard all but one +page. `ITemplateEngineRepository.GetTemplateSummariesAsync(skip, take)` instead pages **in the +database** against `TemplateSummary`, a row-shaped projection (id, name, description, parent/folder +ids, derived flag) with child collections reduced to **counts** — the only thing a list surface +renders. `TemplateSummary.MaxPageSize` (1000) mirrors the existing in-memory page clamp so DB-side +paging cannot be used to pull an unbounded result set. Additive-only message-contract evolution, +same rule as other Commons message types. + ## Diff Calculation The Template Engine can compare: diff --git a/src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/AuditLogIngestActor.cs b/src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/AuditLogIngestActor.cs index 8cece24d..1ca8e692 100644 --- a/src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/AuditLogIngestActor.cs +++ b/src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/AuditLogIngestActor.cs @@ -18,8 +18,9 @@ namespace ZB.MOM.WW.ScadaBridge.AuditLog.Central; /// ingest timestamp into DetailsJson (there is no promoted IngestedAtUtc /// column — the value is a DetailsJson field set via /// ) and inserted idempotently -/// via — duplicates are -/// silently swallowed (first-write-wins). +/// via — duplicates +/// are silently swallowed (first-write-wins), whether they repeat inside one +/// packet or across packets. /// /// /// @@ -28,9 +29,11 @@ namespace ZB.MOM.WW.ScadaBridge.AuditLog.Central; /// consistent and the site is free to flip its local row to Forwarded. /// /// -/// Audit-write failures must NEVER abort the user-facing action. The actor -/// wraps each repository call in its own try/catch so a single bad row cannot -/// cause the rest of the batch to be lost, and it guards scope/repository +/// Audit-write failures must NEVER abort the user-facing action. Each message +/// is written as ONE set-based statement (or one transaction, for the cached +/// dual-write); if that fails the actor retries the same work row-by-row inside +/// per-row try/catch, so a single bad row still cannot cause the rest of the +/// batch to be lost. It also guards scope/repository /// resolution so a transient DI fault cannot restart the singleton — those /// catches are what keep this actor alive across handler throws, not the /// supervisor strategy. The override returns @@ -50,6 +53,53 @@ namespace ZB.MOM.WW.ScadaBridge.AuditLog.Central; /// public class AuditLogIngestActor : ReceiveActor { + /// + /// Overall budget for one ingest message's database work, deliberately + /// SHORTER than the gRPC Ask that wraps it. + /// + /// + /// The path used to stack three identical 30 s budgets — the site's Ask, the + /// central gRPC handler's Ask (SiteStreamGrpcServer.AuditIngestAskTimeout) + /// and the ADO.NET command default — so they all expired at the same instant. The + /// caller therefore learned nothing except "it took 30 s": no partial ack, no + /// distinction between a slow database and a wedged singleton. Making the + /// innermost budget strictly smallest means a slow batch is abandoned by the + /// actor FIRST, with the accepted-so-far ids still replied, while the outer + /// Asks are still waiting. + /// + /// The full ladder is now strictly monotonic end to end: + /// CommunicationOptions.AuditForwardTimeout (35 s, the site-side forward Ask) > + /// SiteStreamGrpcServer.AuditIngestAskTimeout (30 s, the gRPC deadline AND central's + /// Ask of this singleton) > (20 s) > + /// (15 s). + /// + /// + internal static readonly TimeSpan IngestBudget = TimeSpan.FromSeconds(20); + + /// + /// Per-statement SQL timeout for the ingest write, strictly inside + /// so a single wedged statement surfaces as a SQL + /// timeout the per-batch catch can log, rather than consuming the whole + /// actor-level budget and starving the rest of the batch. + /// + internal static readonly TimeSpan IngestSqlCommandTimeout = TimeSpan.FromSeconds(15); + + /// + /// Budget for the per-row fallback loop, granted as a FRESH + /// rather than reusing the batch's. + /// + /// + /// The fallback used to run on the batch's own token. When the batch failed + /// BECAUSE that token expired, every fallback insert was handed an + /// already-cancelled token and failed instantly: N logged errors, N counter + /// bumps, zero rows accepted, and the site retried the whole packet anyway. + /// A fresh, deliberately SHORT budget (shorter than , + /// so the reply still beats the caller's Ask even when the batch consumed its + /// whole allowance) gives the poison-row isolation the fallback exists for a + /// real chance to land the good rows. + /// + internal static readonly TimeSpan IngestFallbackBudget = TimeSpan.FromSeconds(5); + private readonly IServiceProvider? _serviceProvider; private readonly IAuditLogRepository? _injectedRepository; private readonly ILogger _logger; @@ -185,39 +235,101 @@ public class AuditLogIngestActor : ReceiveActor DateTime nowUtc, List accepted) { + // Stamp the ingest timestamp here, not at the site. Redact BEFORE the + // IngestedAtUtc stamp so the redacted copy carries the central-side + // ingest timestamp. The redactor is contract-bound to never throw; a null + // redactor (test composition root, no IAuditRedactor registered) falls + // back to the SafeDefault rather than pass-through, so HTTP header + // redaction always runs. IngestedAtUtc is a DetailsJson field on the + // canonical record, so stamp it via the projection helper. + var safeRedactor = redactor ?? SafeDefaultAuditRedactor.Instance; + var projected = new List(cmd.Events.Count); foreach (var evt in cmd.Events) { - try + var filtered = safeRedactor.Apply(evt); + projected.Add(AuditRowProjection.WithIngestedAtUtc(filtered, nowUtc)); + } + + // ONE set-based statement for the whole packet (WP2.2) instead of one + // IF NOT EXISTS … INSERT round trip per event. A telemetry packet of 200 + // rows used to cost 200 sequential round trips on the central singleton's + // dispatcher; it now costs two. Idempotency is unchanged — duplicates + // within the packet collapse first-write-wins, duplicates against + // committed rows are eliminated by the anti-semi-join, and a concurrent + // writer degrades to the repository's per-row fallback. + using var budget = new CancellationTokenSource(IngestBudget); + try + { + await repository + .InsertManyIfNotExistsAsync(projected, IngestSqlCommandTimeout, budget.Token) + .ConfigureAwait(false); + + // A batch that returned without throwing means every row is now + // present — inserted here or already committed by an earlier delivery. + // Both count as accepted: the storage state is consistent and the site + // is free to flip its local rows to Forwarded. + foreach (var evt in cmd.Events) { - // Stamp the ingest timestamp here, not at the site. The - // repository's duplicate-key hardening already swallows - // duplicate-key races, so the same id arriving twice (site - // retry, reconciliation) is a silent no-op. - // Redact BEFORE the IngestedAtUtc stamp so the redacted - // copy carries the central-side ingest timestamp. The redactor - // is contract-bound to never throw. A null - // redactor (test composition root, no IAuditRedactor - // registered) now falls back to the SafeDefault rather than - // pass-through, so HTTP header redaction always runs. - // IngestedAtUtc is a DetailsJson field on - // the canonical record, so stamp it via the projection helper. - var safeRedactor = redactor ?? SafeDefaultAuditRedactor.Instance; - var filtered = safeRedactor.Apply(evt); - var ingested = AuditRowProjection.WithIngestedAtUtc(filtered, nowUtc); - await repository.InsertIfNotExistsAsync(ingested).ConfigureAwait(false); accepted.Add(evt.EventId); } - catch (Exception ex) + } + catch (Exception ex) + { + // The batch is a throughput optimisation, NOT a change to the + // failure grain. The documented invariant — "a single bad row cannot + // cause the rest of the batch to be lost" — is preserved by falling + // back to the per-row path here, so a poison row still costs only + // itself. Re-running rows the failed batch may already have committed + // is safe: every insert is idempotent on EventId. + // + // The fallback gets its OWN token. Reusing the batch's meant that a + // batch which failed because the budget EXPIRED handed the loop an + // already-cancelled token, so every row failed instantly and the + // fallback did nothing but multiply the log and counter noise by the + // row count. + var budgetExpired = budget.IsCancellationRequested; + using var fallbackBudget = new CancellationTokenSource(IngestFallbackBudget); + + _logger.LogWarning(ex, + "Set-based ingest of {Count} audit event(s) failed{BudgetNote}; falling back to per-row inserts (fresh {FallbackSeconds}s budget) so one bad row does not sink the batch.", + cmd.Events.Count, + budgetExpired ? " after exhausting the ingest budget" : string.Empty, + IngestFallbackBudget.TotalSeconds); + + if (budgetExpired) { - // Per-row catch — one bad row never sinks the whole batch. - // The row stays Pending at the site; the next drain retries. - // Bump the central health counter so a - // sustained insert-throw failure surfaces on the dashboard. + // A blown budget is ONE failure event for the batch, not one per + // row — bump the health counter once here and suppress the + // per-row bumps below so the dashboard sees a timeout as a + // timeout rather than as N independent write failures. try { failureCounter?.Increment(); } catch { /* counter must never throw — defence in depth */ } - _logger.LogError(ex, - "Failed to persist audit event {EventId} during batch ingest; row will be retried by the site.", - evt.EventId); + } + + for (var i = 0; i < projected.Count; i++) + { + try + { + await repository.InsertIfNotExistsAsync(projected[i], fallbackBudget.Token).ConfigureAwait(false); + accepted.Add(cmd.Events[i].EventId); + } + catch (Exception rowEx) + { + // Per-row catch — one bad row never sinks the whole batch. + // The row stays Pending at the site; the next drain retries. + // Bump the central health counter so a sustained insert-throw + // failure surfaces on the dashboard — unless the batch already + // counted itself as a single budget-exhaustion failure. + if (!budgetExpired) + { + try { failureCounter?.Increment(); } + catch { /* counter must never throw — defence in depth */ } + } + + _logger.LogError(rowEx, + "Failed to persist audit event {EventId} during batch ingest; row will be retried by the site.", + cmd.Events[i].EventId); + } } } } @@ -262,6 +374,24 @@ public class AuditLogIngestActor : ReceiveActor var strategy = dbContext.Database.CreateExecutionStrategy(); + // Fast path (WP2.2): the WHOLE packet in ONE transaction — one + // set-based audit insert plus one single-statement upsert per entry. + // The predecessor opened a transaction PER entry and issued three + // statements inside it (IF NOT EXISTS insert, then the two-statement + // SiteCalls upsert), so a 50-entry packet cost ~200 round trips and 50 + // commits. If anything faults, the per-entry loop below re-runs the + // packet with its original per-entry isolation, so the documented + // "one entry's failure does not abort the others" invariant survives — + // it is simply no longer paid for on the healthy path. + using var budget = new CancellationTokenSource(IngestBudget); + if (await TryIngestCachedBatchAsync( + strategy, dbContext, auditRepo, siteCallRepo, redactor, cmd, accepted, budget.Token) + .ConfigureAwait(false)) + { + replyTo.Tell(new IngestCachedTelemetryReply(accepted)); + return; + } + foreach (var entry in cmd.Entries) { try @@ -330,6 +460,90 @@ public class AuditLogIngestActor : ReceiveActor replyTo.Tell(new IngestCachedTelemetryReply(accepted)); } + /// + /// Attempts the whole cached-telemetry packet as ONE transaction: a single + /// set-based audit insert followed by one monotonic SiteCalls upsert + /// per entry. Returns when it committed (and only then + /// appends to ), when the + /// caller should fall back to the per-entry transaction loop. + /// + /// + /// The batch is all-or-nothing by construction — it is a single transaction — + /// which is why a failure MUST fall back rather than report partial success: + /// the per-entry loop is what turns one poison entry into one lost entry + /// instead of a lost packet. Nothing is appended to + /// until the commit returns, so a failed attempt + /// leaves the caller's list untouched and the retry starts from a clean slate. + /// + private async Task TryIngestCachedBatchAsync( + Microsoft.EntityFrameworkCore.Storage.IExecutionStrategy strategy, + ScadaBridgeDbContext dbContext, + IAuditLogRepository auditRepo, + ISiteCallAuditRepository siteCallRepo, + IAuditRedactor? redactor, + IngestCachedTelemetryCommand cmd, + List accepted, + CancellationToken ct) + { + try + { + await strategy.ExecuteAsync(async () => + { + await using var tx = await dbContext.Database + .BeginTransactionAsync(ct) + .ConfigureAwait(false); + + // One central-side instant for the whole packet so a join across + // the two tables sees matching timestamps (debugging convenience, + // not a correctness invariant). + var ingestedAt = DateTime.UtcNow; + var safeRedactor = redactor ?? SafeDefaultAuditRedactor.Instance; + + var auditRows = new List(cmd.Entries.Count); + foreach (var entry in cmd.Entries) + { + // Only the AuditLog row's payload columns are redactable; + // SiteCalls carries operational state only (status, retry + // count) and is left untouched. + var filteredAudit = safeRedactor.Apply(entry.Audit); + auditRows.Add(AuditRowProjection.WithIngestedAtUtc(filteredAudit, ingestedAt)); + } + + await auditRepo + .InsertManyIfNotExistsAsync(auditRows, IngestSqlCommandTimeout, ct) + .ConfigureAwait(false); + + foreach (var entry in cmd.Entries) + { + await siteCallRepo + .UpsertAsync(entry.SiteCall with { IngestedAtUtc = ingestedAt }, ct) + .ConfigureAwait(false); + } + + await tx.CommitAsync(ct).ConfigureAwait(false); + }).ConfigureAwait(false); + } + catch (Exception ex) + { + // No health-counter bump here — the per-entry retry is the authority + // on whether this packet genuinely failed, and double-counting a + // batch that the fallback then writes successfully would make the + // dashboard read as a sustained fault during normal contention. + _logger.LogWarning( + ex, + "Batched cached-telemetry dual-write of {Count} entr(ies) failed; falling back to per-entry transactions.", + cmd.Entries.Count); + return false; + } + + foreach (var entry in cmd.Entries) + { + accepted.Add(entry.Audit.EventId); + } + + return true; + } + /// /// Fallback handler installed on the single-repository test ctor — that /// ctor has no DbContext and no , so diff --git a/src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/AuditLogPurgeActor.cs b/src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/AuditLogPurgeActor.cs index dc95d731..d824c32d 100644 --- a/src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/AuditLogPurgeActor.cs +++ b/src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/AuditLogPurgeActor.cs @@ -17,8 +17,9 @@ namespace ZB.MOM.WW.ScadaBridge.AuditLog.Central; /// for monthly boundaries whose latest OccurredAtUtc is older /// than DateTime.UtcNow - RetentionDays. /// For each eligible boundary, calls -/// which runs -/// the drop-and-rebuild dance around UX_AuditLog_EventId. +/// , a +/// metadata-only staging-table switch (WP2.2 removed the index +/// drop/rebuild that used to bracket it). /// Publishes on the actor-system /// EventStream so the central health collector + ops surfaces /// can subscribe without coupling to this actor. @@ -26,11 +27,10 @@ namespace ZB.MOM.WW.ScadaBridge.AuditLog.Central; /// /// /// -/// Daily cadence. Partition switch is metadata-only but the -/// drop-and-rebuild dance briefly removes UX_AuditLog_EventId; running -/// more often than necessary trades unique-index rebuild outages for -/// negligible freshness wins. The default 24-hour interval matches -/// alog.md §10's retention policy. +/// Daily cadence. The partition switch is metadata-only, but it still +/// takes schema-modification locks on a table the ingest path is writing to; +/// running more often than necessary trades contention for negligible freshness +/// wins. The default 24-hour interval matches alog.md §10's retention policy. /// /// /// Continue-on-error. A single boundary that throws (transient SQL diff --git a/src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/AuditLogPurgeOptions.cs b/src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/AuditLogPurgeOptions.cs index 5621cd56..0d97571c 100644 --- a/src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/AuditLogPurgeOptions.cs +++ b/src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/AuditLogPurgeOptions.cs @@ -10,12 +10,12 @@ namespace ZB.MOM.WW.ScadaBridge.AuditLog.Central; /// /// /// -/// The purge actor is a daily-cadence singleton, not a hot-loop, because -/// partition-switch I/O is metadata-only but the drop-and-rebuild dance -/// briefly removes the UX_AuditLog_EventId unique index — running -/// more often than necessary trades index-rebuild outages for marginal -/// freshness gains. Lower this only when an operator can prove they need -/// sub-daily purge granularity. +/// The purge actor is a daily-cadence singleton, not a hot-loop. The +/// partition switch itself is metadata-only (since WP2.2's +/// AlignAuditLogEventIdUniqueness it no longer drops and rebuilds a +/// non-aligned unique index around it), but it still takes schema-modification +/// locks and competes with ingest for the same table. Lower this only when an +/// operator can prove they need sub-daily purge granularity. /// /// /// exists for tests to drop the cadence to @@ -58,15 +58,14 @@ public sealed class AuditLogPurgeOptions /// /// Per-command timeout (in minutes) for the maintenance SQL the purge tick issues — - /// both the partition switch-out drop-and-rebuild dance + /// both the partition switch-out staging batch /// () /// and each per-channel DELETE TOP batch. Default 30 minutes. /// /// /// The ADO.NET default command timeout is ~30 seconds. On a large or contended partition the - /// SWITCH dance (which briefly drops UX_AuditLog_EventId) can exceed that and abort - /// mid-flight — leaving the live table without its idempotency-supporting unique index until a - /// later tick's CATCH branch rebuilds it (arch-review 04, S2). A generous maintenance timeout + /// SWITCH batch can exceed that and abort mid-flight, leaving an orphaned staging table for the + /// next tick's CATCH branch to clean up (arch-review 04, S2). A generous maintenance timeout /// lets the metadata-only switch complete rather than self-locking. Resolved via /// , clamped to a 1-minute floor. /// diff --git a/src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/GrpcPullAuditEventsClient.cs b/src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/GrpcPullAuditEventsClient.cs index 7001f9df..3020e57f 100644 --- a/src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/GrpcPullAuditEventsClient.cs +++ b/src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/GrpcPullAuditEventsClient.cs @@ -73,6 +73,7 @@ public sealed class GrpcPullAuditEventsClient : IPullAuditEventsClient public async Task PullAsync( string siteId, DateTime sinceUtc, + string? afterId, int batchSize, CancellationToken ct) { @@ -93,6 +94,10 @@ public sealed class GrpcPullAuditEventsClient : IPullAuditEventsClient // EnsureUtc keeps Timestamp.FromDateTime happy (it requires UTC kind). SinceUtc = Timestamp.FromDateTime(EnsureUtc(sinceUtc)), BatchSize = batchSize, + // Composite-keyset tiebreak (proto field 3), mirroring PullSiteCalls exactly. + // proto3 has no nullable string — an unset/empty AfterId is the site's signal to + // keep the legacy inclusive-timestamp contract (also what a first pull sends). + AfterId = afterId ?? string.Empty, }; var (reply, transportFault) = await TryInvokeAsync(endpoint, request, siteId, ct) @@ -121,10 +126,13 @@ public sealed class GrpcPullAuditEventsClient : IPullAuditEventsClient // Map proto DTOs to canonical AuditEvent records and order oldest-first // (the wire is already ordered by the site queue, but the - // IPullAuditEventsClient contract is explicit, so sort defensively). + // IPullAuditEventsClient contract is explicit, so sort defensively). The EventId + // tiebreak matches the site's own composite ordering — ordinal over the "D" GUID + // text, which is what SQLite's BINARY collation compares. var events = reply.Events .Select(AuditEventDtoMapper.FromDto) .OrderBy(e => e.OccurredAtUtc) + .ThenBy(e => e.EventId.ToString(), StringComparer.Ordinal) .ToList(); return new PullAuditEventsResponse(events, reply.MoreAvailable); diff --git a/src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/IPullAuditEventsClient.cs b/src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/IPullAuditEventsClient.cs index 52ebf083..e04357c7 100644 --- a/src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/IPullAuditEventsClient.cs +++ b/src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/IPullAuditEventsClient.cs @@ -39,12 +39,24 @@ public interface IPullAuditEventsClient /// /// The identifier of the site to pull audit events from. /// Only events with an OccurredAtUtc at or after this cursor time are returned. + /// + /// The composite-keyset tiebreak cursor, mirroring + /// . When non-null it is the + /// EventId ("D" GUID form) of the last row already consumed at + /// ; the site returns only rows strictly greater than the + /// composite (OccurredAtUtc, EventId) pair, so a burst sharing one exact instant + /// drains via the id tiebreak instead of pinning the inclusive-timestamp cursor — and the + /// site's MarkReconciledUpToAsync can retire the rows AT that instant, which a bare + /// timestamp can never prove received. Null on the first pull (or against a legacy site) + /// preserves the inclusive >= contract. + /// /// Maximum number of events to return per call. /// Cancellation token. /// A task that resolves to the next reconciliation batch with a MoreAvailable flag. Task PullAsync( string siteId, DateTime sinceUtc, + string? afterId, int batchSize, CancellationToken ct); } diff --git a/src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/SiteAuditReconciliationActor.cs b/src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/SiteAuditReconciliationActor.cs index 11423553..a68f7ee5 100644 --- a/src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/SiteAuditReconciliationActor.cs +++ b/src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/SiteAuditReconciliationActor.cs @@ -76,14 +76,28 @@ public class SiteAuditReconciliationActor : ReceiveActor private readonly ILogger _logger; /// - /// Per-site reconciliation watermark — the highest - /// seen for that site on a previous - /// tick. Asking for OccurredAtUtc >= cursor rather than > - /// is the site contract (); - /// duplicate-with-same-timestamp rows are filtered out by the idempotent - /// repository write. + /// Per-site reconciliation watermark — the COMPOSITE + /// (OccurredAtUtc, EventId) of the highest row seen for that site on a previous tick, + /// mirroring SiteCallAuditActor's PullSiteCalls cursor. /// - private readonly Dictionary _cursors = new(); + /// + /// + /// The site's + /// + /// serves rows strictly after the pair when AfterId is set, and falls back to the + /// legacy inclusive OccurredAtUtc >= since when it is null (the first pull). + /// Sending the id half matters twice over: a burst sharing one exact instant drains via the + /// tiebreak instead of pinning the timestamp forever, and + /// + /// can retire the rows AT the cursor instant — a bare timestamp can only prove receipt of + /// rows strictly older than itself, so the boundary rows stayed servable forever. + /// + /// + /// Duplicates that a re-pull does produce are still filtered by the idempotent repository + /// write, so the cursor is an optimization for exactness, never a correctness crutch. + /// + /// + private readonly Dictionary _cursors = new(); /// /// Per-site count of consecutive non-draining cycles. Resets to zero on the @@ -239,20 +253,33 @@ public class SiteAuditReconciliationActor : ReceiveActor /// /// Issues one PullAuditEvents RPC against the site, ingests the /// returned rows idempotently into the central repository, and advances - /// the cursor based on the maximum - /// observed. The brief's "saturate until backlog clears" intent is met by - /// the natural cadence — each tick issues one pull, and a backed-up site + /// the composite (, ) + /// cursor to the maximum row observed. The brief's "saturate until backlog clears" intent is + /// met by the natural cadence — each tick issues one pull, and a backed-up site /// drains across consecutive ticks. The stalled signal (two non-draining /// ticks in a row) surfaces when that drain isn't keeping up. /// + /// + /// Unlike SiteCallAuditActor this does NOT page within a tick; one pull per tick is + /// deliberate (the stalled signal is how a lagging drain surfaces). The composite cursor is + /// what makes that safe against a same-instant burst larger than one batch: the id tiebreak + /// advances even when the timestamp cannot, so consecutive ticks make real progress instead + /// of re-serving the same window forever. + /// private async Task PullSiteAsync(SiteEntry site, IAuditLogRepository repository, Akka.Event.EventStream eventStream) { - var since = _cursors.TryGetValue(site.SiteId, out var c) ? c : DateTime.MinValue; + var cursor = _cursors.TryGetValue(site.SiteId, out var c) + ? c + : (Since: DateTime.MinValue, AfterId: (string?)null); + var since = cursor.Since; + var afterId = cursor.AfterId; + var response = await _client.PullAsync( - site.SiteId, since, _options.BatchSize, CancellationToken.None) + site.SiteId, since, afterId, _options.BatchSize, CancellationToken.None) .ConfigureAwait(false); var maxOccurred = since; + var maxAfterId = afterId; var hasUnresolvedFailure = false; var nowUtc = DateTime.UtcNow; foreach (var evt in response.Events) @@ -310,10 +337,18 @@ public class SiteAuditReconciliationActor : ReceiveActor } // Canonical OccurredAtUtc is a DateTimeOffset; the cursor is a UTC DateTime. + // Advance the COMPOSITE max, exactly as SiteCallAuditActor does: a greater + // timestamp wins; on a tie the greater EventId (ordinal over the "D" GUID text, + // matching the site's SQLite BINARY collation) wins. CompareOrdinal handles the + // null seed — CompareOrdinal(x, null) > 0 for any non-null x. var occurredUtc = evt.OccurredAtUtc.UtcDateTime; - if (advanceForThisRow && occurredUtc > maxOccurred) + var rowId = evt.EventId.ToString(); + if (advanceForThisRow && + (occurredUtc > maxOccurred || + (occurredUtc == maxOccurred && string.CompareOrdinal(rowId, maxAfterId) > 0))) { maxOccurred = occurredUtc; + maxAfterId = rowId; } } @@ -322,8 +357,11 @@ public class SiteAuditReconciliationActor : ReceiveActor // the whole batch next tick — successful rows are no-ops thanks to // InsertIfNotExistsAsync's idempotency, and the failing row gets // another attempt. Once it succeeds (or hits the permanent-abandon - // threshold) the cursor unblocks naturally. - _cursors[site.SiteId] = hasUnresolvedFailure ? since : maxOccurred; + // threshold) the cursor unblocks naturally. Both halves move together: holding the + // timestamp back while advancing the id would skip the very rows being retried. + _cursors[site.SiteId] = hasUnresolvedFailure + ? (since, afterId) + : (maxOccurred, maxAfterId); var nonDraining = response.MoreAvailable && response.Events.Count > 0; UpdateStalledState(site.SiteId, draining: !nonDraining, eventStream); diff --git a/src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/SiteAuditBacklogReporter.cs b/src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/SiteAuditBacklogReporter.cs index ce1d1ec7..9350e1b1 100644 --- a/src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/SiteAuditBacklogReporter.cs +++ b/src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/SiteAuditBacklogReporter.cs @@ -45,12 +45,27 @@ public sealed class SiteAuditBacklogReporter : IHostedService, IDisposable /// internal static readonly TimeSpan DefaultRefreshInterval = TimeSpan.FromSeconds(30); + /// + /// Age at which the oldest still-Pending row is called out in the log. A Pending row is + /// one central has not acknowledged through EITHER path (telemetry push or reconciliation + /// pull), and PurgeExpiredAsync deliberately never purges Pending — so the site + /// store's floor depends on reconciliation actually running. Rows served by a pull but + /// never covered by a later cursor sit in exactly this state, which is why the age is + /// worth a signal rather than only a dashboard number. One day is comfortably longer than + /// any normal drain/reconcile outage and well inside the ~7-day site retention window. + /// + internal static readonly TimeSpan StalePendingThreshold = TimeSpan.FromHours(24); + + /// How often the stale-pending warning may repeat (it is a standing condition). + internal static readonly TimeSpan StalePendingWarnInterval = TimeSpan.FromHours(1); + private readonly ISiteAuditQueue _queue; private readonly ISiteHealthCollector _collector; private readonly ILogger _logger; private readonly TimeSpan _refreshInterval; private CancellationTokenSource? _cts; private Task? _loop; + private DateTime _lastStalePendingWarnUtc = DateTime.MinValue; /// Initializes a new instance of . /// The site audit queue used to probe the backlog count. @@ -151,6 +166,7 @@ public sealed class SiteAuditBacklogReporter : IHostedService, IDisposable { var snapshot = await _queue.GetBacklogStatsAsync(ct).ConfigureAwait(false); _collector.UpdateSiteAuditBacklog(snapshot); + WarnIfPendingIsStale(snapshot.OldestPendingUtc, snapshot.PendingCount); } catch (OperationCanceledException) { @@ -166,6 +182,34 @@ public sealed class SiteAuditBacklogReporter : IHostedService, IDisposable } } + /// + /// Logs a rate-limited warning when the oldest still-Pending audit row is older than + /// . Pending rows are exempt from the retention purge + /// by design, so a standing Pending backlog is the one site-store condition that does not + /// self-heal on age — it clears only when central acknowledges the rows (telemetry ack or + /// a reconciliation cursor that covers them). Internal so tests can drive it directly. + /// + /// Oldest pending row's occurrence instant, or null when none. + /// Number of rows currently pending. + internal void WarnIfPendingIsStale(DateTime? oldestPendingUtc, int pendingCount) + { + if (oldestPendingUtc is null) return; + + var age = DateTime.UtcNow - DateTime.SpecifyKind(oldestPendingUtc.Value, DateTimeKind.Utc); + if (age < StalePendingThreshold) return; + + var now = DateTime.UtcNow; + if (now - _lastStalePendingWarnUtc < StalePendingWarnInterval) return; + _lastStalePendingWarnUtc = now; + + _logger.LogWarning( + "Site audit backlog: oldest pending row is {AgeHours:F1}h old ({PendingCount} pending). " + + "Pending rows are never purged on age, so this backlog only clears when central " + + "acknowledges them — check the site→central audit telemetry drain and the central " + + "reconciliation pull.", + age.TotalHours, pendingCount); + } + /// Signals the polling loop to stop and waits for it to complete. /// Cancellation token (not used; the internal CTS governs shutdown). /// A task that represents the asynchronous operation. diff --git a/src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/SqliteAuditWriter.cs b/src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/SqliteAuditWriter.cs index 62108a76..98bf7423 100644 --- a/src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/SqliteAuditWriter.cs +++ b/src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/SqliteAuditWriter.cs @@ -174,6 +174,24 @@ public class SqliteAuditWriter : IAuditWriter, ISiteAuditQueue, IAsyncDisposable pragmaCmd.ExecuteNonQuery(); } + // synchronous=NORMAL (arch-review remediation WP1.2): SQLite's FULL default + // fsyncs on every transaction commit, which is the dominant cost of the + // per-event write path this same change coalesces via FlushIntervalMs — FULL + // would still fsync once per FLUSH even after that fix. NORMAL fsyncs at + // fewer, WAL-checkpoint-aligned points instead of every commit; the window + // this opens is a handful of the most recent commits lost on OS crash / power + // loss (WAL mode itself still guarantees no *corruption*, only a possible + // short rollback on unclean restart). That trade is acceptable here because + // audit writes are explicitly best-effort by design (CLAUDE.md: "Audit-write + // failure NEVER aborts the user-facing action — audit is best-effort, the + // action's own success/failure path is authoritative") — this is the same + // durability class as a dropped audit row, not a new risk class. + using (var pragmaCmd = _connection.CreateCommand()) + { + pragmaCmd.CommandText = "PRAGMA synchronous = NORMAL"; + pragmaCmd.ExecuteNonQuery(); + } + // Enable FK enforcement on the WRITE connection. PRAGMA foreign_keys is // a per-connection, per-session setting in SQLite — it is NOT persisted // in the database file, so every new connection that may INSERT into @@ -306,20 +324,67 @@ public class SqliteAuditWriter : IAuditWriter, ISiteAuditQueue, IAsyncDisposable { batch.Clear(); batch.Add(first); + var deadline = DateTime.UtcNow.AddMilliseconds(_options.FlushIntervalMs); // Pull additional ready events up to BatchSize. TryRead is non- // blocking and lets us amortise the transaction overhead across a - // burst of concurrent enqueues. - while (batch.Count < _options.BatchSize && - _writeQueue.Reader.TryRead(out var next)) + // burst of concurrent enqueues. Once the immediately-available + // burst is drained, keep the transaction open for up to + // FlushIntervalMs waiting for MORE events to trickle in — this is + // the soft-flush coalescing arch-review remediation WP1.2 adds: + // without it, a low-rate trickle of writes (one script call every + // few ms, well under BatchSize) fsyncs once PER EVENT, because the + // writer loop reaches an empty channel and flushes immediately. + // FlushIntervalMs bounds the worst-case added latency any single + // event can see from this coalescing. + while (batch.Count < _options.BatchSize) { - batch.Add(next); + if (_writeQueue.Reader.TryRead(out var next)) + { + batch.Add(next); + continue; + } + + var remaining = deadline - DateTime.UtcNow; + if (remaining <= TimeSpan.Zero) + { + break; + } + + var waitToRead = _writeQueue.Reader.WaitToReadAsync().AsTask(); + var completed = await Task.WhenAny( + waitToRead, Task.Delay(remaining)).ConfigureAwait(false); + + if (completed != waitToRead) + { + // Deadline elapsed with nothing new — flush what we have. + break; + } + + // WaitToReadAsync resolving false means the channel completed + // (Dispose) with nothing left to read; resolving true means at + // least one item is available, so the top of the loop's + // TryRead will pick it up. + if (!await waitToRead.ConfigureAwait(false)) + { + break; + } } FlushBatch(batch); } } + /// + /// Total number of transactions committed since + /// construction. Test-only observability seam (arch-review remediation + /// WP1.2) for asserting that FlushIntervalMs coalesces a trickle of + /// near-simultaneous writes into a single transaction rather than one + /// commit per event; production code never reads this. + /// + internal int FlushCountForTests => _flushCountForTests; + private int _flushCountForTests; + private void FlushBatch(IReadOnlyList batch) { lock (_writeLock) @@ -436,6 +501,7 @@ public class SqliteAuditWriter : IAuditWriter, ISiteAuditQueue, IAsyncDisposable } transaction.Commit(); + _flushCountForTests++; } catch (Exception ex) { @@ -692,10 +758,17 @@ public class SqliteAuditWriter : IAuditWriter, ISiteAuditQueue, IAsyncDisposable /// /// Lower bound timestamp (UTC) for event occurrence. /// Maximum number of rows to return. + /// + /// Composite-keyset tiebreak: the EventId of the last row already consumed at + /// . Non-null switches the predicate from the inclusive + /// OccurredAtUtc >= $since to the strict composite + /// (OccurredAtUtc, EventId) > ($since, $afterId), matching the query's own + /// ORDER BY so a batch cannot stall on rows sharing one instant. + /// /// Cancellation token. /// A task that resolves to a read-only list of audit events since the given timestamp. public Task> ReadPendingSinceAsync( - DateTime sinceUtc, int batchSize, CancellationToken ct = default) + DateTime sinceUtc, int batchSize, string? afterId = null, CancellationToken ct = default) { if (batchSize <= 0) { @@ -713,13 +786,24 @@ public class SqliteAuditWriter : IAuditWriter, ISiteAuditQueue, IAsyncDisposable ObjectDisposedException.ThrowIf(_disposed, this); using var cmd = _readConnection.CreateCommand(); - cmd.CommandText = """ + var cursorPredicate = afterId is null + ? "fs.OccurredAtUtc >= $since" + // Composite keyset, lexicographic on (OccurredAtUtc, EventId) — the same + // ordering the query applies, so it is a strict "everything after this row". + : "(fs.OccurredAtUtc > $since OR (fs.OccurredAtUtc = $since AND ae.EventId > $afterId))"; + + // fs.rowid rides along as an 11th column purely to maintain + // _maxServedRowId — the insertion-order high-water mark that bounds + // MarkReconciledUpToAsync (see that method). It is never mapped onto + // the returned AuditEvent; rowid stays a storage-layer concept. + cmd.CommandText = $""" SELECT ae.EventId, ae.OccurredAtUtc, ae.Actor, ae.Action, ae.Outcome, - ae.Category, ae.Target, ae.SourceNode, ae.CorrelationId, ae.DetailsJson + ae.Category, ae.Target, ae.SourceNode, ae.CorrelationId, ae.DetailsJson, + fs.rowid FROM audit_event ae JOIN audit_forward_state fs ON fs.EventId = ae.EventId WHERE fs.ForwardState IN ($pending, $forwarded) - AND fs.OccurredAtUtc >= $since + AND {cursorPredicate} ORDER BY fs.OccurredAtUtc ASC, ae.EventId ASC LIMIT $limit; """; @@ -730,9 +814,121 @@ public class SqliteAuditWriter : IAuditWriter, ISiteAuditQueue, IAsyncDisposable // that encoding so we can index-scan against it. cmd.Parameters.AddWithValue("$since", EnsureUtc(sinceUtc).ToString( "o", System.Globalization.CultureInfo.InvariantCulture)); + if (afterId is not null) + { + // EventIds are stored as Guid.ToString() ("D"), so compare in that form. + cmd.Parameters.AddWithValue("$afterId", NormalizeEventId(afterId)); + } cmd.Parameters.AddWithValue("$limit", batchSize); - return Task.FromResult(ReadRows(cmd, batchSize)); + return Task.FromResult(ReadServedRows(cmd, batchSize)); + } + } + + /// + /// Highest audit_forward_state.rowid this instance has ever SERVED from + /// — i.e. the insertion-order high-water mark of + /// rows central could possibly have received through the pull path. Bounds + /// so a row inserted after that point can never be + /// retired by a cursor that happens to sit above its timestamp. Monotonic (a later, smaller + /// batch never lowers it) except for the purge clamp. Written under _readLock (the + /// read path) and read under _writeLock (the flip/purge paths) — two different + /// locks, so the accesses go through for visibility. A stale read + /// can only make the bound smaller, i.e. more conservative, never unsafe. + /// + private long _maxServedRowId; + + /// + /// Executes a reconciliation-pull read whose projection carries fs.rowid as an + /// 11th column, materialising the canonical rows while advancing + /// . Serving a row is what makes it eligible for + /// cursor-proved retirement, so the two must happen in the same step. + /// + private IReadOnlyList ReadServedRows(SqliteCommand cmd, int capacityHint) + { + var rows = new List(Math.Min(capacityHint, 256)); + var maxRowId = Volatile.Read(ref _maxServedRowId); + using (var reader = cmd.ExecuteReader()) + { + while (reader.Read()) + { + rows.Add(MapRow(reader)); + var rowId = reader.GetInt64(10); + if (rowId > maxRowId) maxRowId = rowId; + } + } + + Volatile.Write(ref _maxServedRowId, maxRowId); + return rows; + } + + /// + /// Normalises a wire-supplied event id to the exact textual form stored in + /// audit_event.EventId so the keyset comparison is apples-to-apples. An + /// unparseable value is passed through verbatim rather than throwing — a malformed + /// cursor must degrade to "serves a bit too much", never to a failed pull. + /// + private static string NormalizeEventId(string afterId) => + Guid.TryParse(afterId, out var g) ? g.ToString() : afterId; + + /// + public Task MarkReconciledUpToAsync( + DateTime sinceUtc, string? afterId, CancellationToken ct = default) + { + lock (_writeLock) + { + ObjectDisposedException.ThrowIf(_disposed, this); + + // Everything at or before central's cursor is proven received. With no + // afterId the read contract is inclusive (>= $since), so only rows STRICTLY + // older than the cursor instant are proven — the boundary instant may be + // half-consumed and must stay servable. + var predicate = afterId is null + ? "fs.OccurredAtUtc < $since" + : "(fs.OccurredAtUtc < $since OR (fs.OccurredAtUtc = $since AND fs.EventId <= $afterId))"; + + // INSERTION-ORDER BOUND — "only rows we actually served can retire". + // + // The timestamp cursor alone is not enough. OccurredAtUtc is stamped by the + // caller, so a row can be INSERTED after a batch was served yet carry a + // timestamp BELOW central's (by then advanced) cursor — a late-stamped insert. + // The blanket cursor UPDATE retired exactly those rows: never served, never + // servable again (ReadPendingSinceAsync's keyset has moved past them), and, + // being Reconciled, purged on age — silent audit loss, a failure mode the old + // explicit id-set flip could not produce. + // + // fs.rowid is insertion order, so bounding the flip at the highest rowid this + // instance has ever served restores the invariant exactly: rows inserted after + // that point are out of the flip's reach no matter where the cursor sits. A + // FORWARDED row is exempt from the bound — central ACKED it through the + // telemetry push path, which is receipt-proof independent of the pull. + // + // Bound state is per-process: after a restart it is 0 until this instance serves + // a batch, so the first pull retires only Forwarded rows and the pull after it + // resumes normal retirement. Conservative in the safe direction (delays + // retirement, never loses a row); the liveness note in MarkReconciledUpToAsync's + // interface doc + Component-AuditLog.md covers what a permanently halted + // reconciliation means for such rows. + using var cmd = _connection.CreateCommand(); + cmd.CommandText = $""" + UPDATE audit_forward_state AS fs + SET ForwardState = $reconciled + WHERE fs.ForwardState IN ($pending, $forwarded) + AND (fs.ForwardState = $forwarded OR fs.rowid <= $maxServedRowId) + AND {predicate}; + """; + cmd.Parameters.AddWithValue("$reconciled", AuditForwardState.Reconciled.ToString()); + cmd.Parameters.AddWithValue("$pending", AuditForwardState.Pending.ToString()); + cmd.Parameters.AddWithValue("$forwarded", AuditForwardState.Forwarded.ToString()); + cmd.Parameters.AddWithValue("$maxServedRowId", Volatile.Read(ref _maxServedRowId)); + cmd.Parameters.AddWithValue("$since", EnsureUtc(sinceUtc).ToString( + "o", System.Globalization.CultureInfo.InvariantCulture)); + if (afterId is not null) + { + cmd.Parameters.AddWithValue("$afterId", NormalizeEventId(afterId)); + } + + return Task.FromResult(cmd.ExecuteNonQuery()); } } @@ -933,6 +1129,22 @@ public class SqliteAuditWriter : IAuditWriter, ISiteAuditQueue, IAsyncDisposable dropCmd.ExecuteNonQuery(); } + // Clamp the served-rows high-water mark to what is still in the table. + // SQLite hands a fresh insert MAX(rowid)+1, so a purge that empties the + // sidecar (or removes its top rows) makes rowids REUSABLE — a stale-high + // _maxServedRowId would then vouch for rows nobody has served. Cheap: + // MAX(rowid) is an index-less O(1) lookup. + if (purged > 0) + { + using var maxCmd = _connection.CreateCommand(); + maxCmd.Transaction = transaction; + maxCmd.CommandText = "SELECT IFNULL(MAX(rowid), 0) FROM audit_forward_state;"; + var liveMax = Convert.ToInt64(maxCmd.ExecuteScalar(), + System.Globalization.CultureInfo.InvariantCulture); + if (liveMax < Volatile.Read(ref _maxServedRowId)) + Volatile.Write(ref _maxServedRowId, liveMax); + } + transaction.Commit(); } catch diff --git a/src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/SqliteAuditWriterOptions.cs b/src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/SqliteAuditWriterOptions.cs index 3f558a54..ab8c1f8f 100644 --- a/src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/SqliteAuditWriterOptions.cs +++ b/src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/SqliteAuditWriterOptions.cs @@ -9,8 +9,23 @@ namespace ZB.MOM.WW.ScadaBridge.AuditLog.Site; /// public sealed class SqliteAuditWriterOptions { - /// SQLite database path (or in-memory URI for tests). - public string DatabasePath { get; set; } = "auditlog.db"; + /// + /// SQLite database path (or in-memory URI for tests). No default — + /// mirrors LocalDb:Path (ZB.MOM.WW.LocalDb.LocalDbOptions.Path): a + /// bare filename here would resolve relative to the process CWD, which on the + /// docker rig is the container's ephemeral overlayfs, not the mounted + /// /app/data volume — the file (and its pending forward-state backlog) + /// would be silently discarded on every container recreate. Site nodes MUST set + /// this explicitly (e.g. /app/data/auditlog.db); StartupValidator + /// in ZB.MOM.WW.ScadaBridge.Host enforces the requirement pre-host for Site nodes only, + /// the same way it enforces Communication:GrpcPsk. Central composition roots also + /// bind this options type (AddAuditLog is shared between roles) but never resolve + /// , so the empty default is inert there and deliberately + /// left unvalidated by (which only checks + /// the role-agnostic channel/batch/flush knobs) — enforcing it there too would fail + /// central's boot via ValidateOnStart, since AddAuditLog runs on both roles. + /// + public string DatabasePath { get; set; } = ""; /// /// Capacity of the bounded write queue. Set high enough that ordinary diff --git a/src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/SqliteAuditWriterOptionsValidator.cs b/src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/SqliteAuditWriterOptionsValidator.cs index 68c586ce..cc79e76b 100644 --- a/src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/SqliteAuditWriterOptionsValidator.cs +++ b/src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/SqliteAuditWriterOptionsValidator.cs @@ -6,19 +6,25 @@ namespace ZB.MOM.WW.ScadaBridge.AuditLog.Site; /// Validates at host startup /// (arch-review 08 round 2 NF4). The channel/batch/flush knobs feed the /// background writer task; a zero anywhere stalls it (nothing drains, nothing -/// flushes), and an empty DatabasePath leaves the SQLite writer with no -/// backing store. +/// flushes). /// is intentionally NOT validated — a non-positive value has a documented /// fall-back-to-30s contract in SiteAuditBacklogReporter. /// +/// +/// is deliberately NOT +/// validated here (arch-review remediation WP1.2). AddAuditLog runs on +/// BOTH Central and Site composition roots and binds this options type with +/// ValidateOnStart either way, but only Site nodes ever resolve +/// — requiring a non-empty path in this +/// role-agnostic validator would fail Central's boot too. The requirement is +/// instead enforced pre-host, Site-role-only, by StartupValidator in +/// ZB.MOM.WW.ScadaBridge.Host (mirrors how it gates Communication:GrpcPsk). +/// public sealed class SqliteAuditWriterOptionsValidator : OptionsValidatorBase { /// protected override void Validate(ValidationBuilder builder, SqliteAuditWriterOptions options) { - builder.RequireThat(!string.IsNullOrWhiteSpace(options.DatabasePath), - $"AuditLog:SiteWriter:{nameof(SqliteAuditWriterOptions.DatabasePath)} must be a non-empty path."); - builder.RequireThat(options.ChannelCapacity > 0, $"AuditLog:SiteWriter:{nameof(SqliteAuditWriterOptions.ChannelCapacity)} " + $"({options.ChannelCapacity}) must be > 0."); diff --git a/src/ZB.MOM.WW.ScadaBridge.CLI/Commands/DeployCommands.cs b/src/ZB.MOM.WW.ScadaBridge.CLI/Commands/DeployCommands.cs index f014f784..08b43a3d 100644 --- a/src/ZB.MOM.WW.ScadaBridge.CLI/Commands/DeployCommands.cs +++ b/src/ZB.MOM.WW.ScadaBridge.CLI/Commands/DeployCommands.cs @@ -19,6 +19,7 @@ public static class DeployCommands var command = new Command("deploy") { Description = "Deployment operations" }; command.Add(BuildInstance(urlOption, formatOption, usernameOption, passwordOption)); + command.Add(BuildSite(urlOption, formatOption, usernameOption, passwordOption)); command.Add(BuildArtifacts(urlOption, formatOption, usernameOption, passwordOption)); command.Add(BuildStatus(urlOption, formatOption, usernameOption, passwordOption)); @@ -39,6 +40,43 @@ public static class DeployCommands return cmd; } + /// + /// Builds deploy site — bulk-deploy every instance at one site. + /// + /// + /// Sits under deploy alongside deploy instance and + /// deploy artifacts, naming the SCOPE of the deploy as the verb, which + /// is the convention the group already follows. Note it is deliberately NOT + /// fleet-wide when --site-id is omitted (the pattern deploy + /// artifacts uses): a bulk instance deploy is far more consequential than + /// an artifact push, so the target site is required rather than defaulted. + /// + /// + private static Command BuildSite(Option urlOption, Option formatOption, Option usernameOption, Option passwordOption) + { + var siteIdOption = new Option("--site-id") { Description = "Target site ID", Required = true }; + var cmd = new Command("site") { Description = "Deploy every deployable instance at a site" }; + cmd.Add(siteIdOption); + cmd.SetAction(async (ParseResult result) => + { + var siteId = result.GetValue(siteIdOption); + return await CommandHelpers.ExecuteCommandAsync( + result, urlOption, formatOption, usernameOption, passwordOption, + new MgmtDeploySiteCommand(siteId), + // A bulk deploy is N instance round-trips; the default 30 s client + // timeout would abandon the request while the server is still + // applying. Matches the server-side long-running Ask window. + timeout: BulkDeployTimeout); + }); + return cmd; + } + + /// + /// Client-side timeout for the bulk site deploy, matching the management + /// service's long-running command Ask window. + /// + internal static readonly TimeSpan BulkDeployTimeout = TimeSpan.FromMinutes(5); + private static Command BuildArtifacts(Option urlOption, Option formatOption, Option usernameOption, Option passwordOption) { var siteIdOption = new Option("--site-id") { Description = "Target site ID (all sites if omitted)" }; diff --git a/src/ZB.MOM.WW.ScadaBridge.CLI/Commands/TemplateCommands.cs b/src/ZB.MOM.WW.ScadaBridge.CLI/Commands/TemplateCommands.cs index 2b883bde..598c3bbc 100644 --- a/src/ZB.MOM.WW.ScadaBridge.CLI/Commands/TemplateCommands.cs +++ b/src/ZB.MOM.WW.ScadaBridge.CLI/Commands/TemplateCommands.cs @@ -159,14 +159,20 @@ public static class TemplateCommands private static Command BuildList(Option urlOption, Option formatOption, Option usernameOption, Option passwordOption) { + // The server projects this listing in the database and returns + // TemplateSummary rows (counts, no child collections), so --detail cannot + // produce full definitions here — it only skips the CLI-side column + // projection and renders the summary payload verbatim. Full definitions + // come from `template get --id`. var detailOption = new Option("--detail") { - Description = "Include full template definitions (all attributes/alarms/scripts) in table output. " - + "Without it, table output is a compact summary (counts only). JSON output is always full." + Description = "Render the raw list payload in table output instead of the compact column projection. " + + "The server returns summary rows (member counts only) for a listing, so this does NOT include " + + "attribute/alarm/script definitions — use 'template get --id ' for those. No effect on JSON output." }; var skipOption = new Option("--skip") { Description = "Offset paging: number of items to skip (default 0)" }; var takeOption = new Option("--take") { Description = "Offset paging: max items to return (1..1000; omit for unlimited)" }; - var cmd = new Command("list") { Description = "List all templates (compact table summary; use --detail for the full dump)" }; + var cmd = new Command("list") { Description = "List all templates (compact table summary with member counts; definitions via 'template get')" }; cmd.Add(detailOption); cmd.Add(skipOption); cmd.Add(takeOption); diff --git a/src/ZB.MOM.WW.ScadaBridge.CLI/Commands/TemplateTableProjection.cs b/src/ZB.MOM.WW.ScadaBridge.CLI/Commands/TemplateTableProjection.cs index 107ec22a..e5c46aaa 100644 --- a/src/ZB.MOM.WW.ScadaBridge.CLI/Commands/TemplateTableProjection.cs +++ b/src/ZB.MOM.WW.ScadaBridge.CLI/Commands/TemplateTableProjection.cs @@ -5,12 +5,30 @@ namespace ZB.MOM.WW.ScadaBridge.CLI.Commands; /// /// Compact table projection for template list / template get. -/// The management API returns full Template entities — every attribute, alarm, +/// +/// +/// template get returns a full Template entity — every attribute, alarm, /// script, and composition inline — which the generic table renderer dumps as one giant -/// cell per template (~171 KB for a real catalogue, unusable in a terminal). This -/// projector reduces each template to id / name / description / parent / derived plus -/// member counts, leaving JSON output untouched (callers pass this only on the -/// table path) and the full dump available via the command's --detail flag. +/// cell per template (~171 KB for a real catalogue, unusable in a terminal). +/// template list no longer returns entities at all: the management +/// ListTemplates handler projects in the DATABASE and returns +/// TemplateSummary rows whose children are already reduced to +/// attributeCount/alarmCount/… scalars. +/// +/// +/// +/// This projector handles BOTH shapes: it prefers the pre-computed *Count +/// scalar and falls back to the length of the matching child array, so the same +/// column set renders for a summary row and for a full entity. Reading only the +/// arrays (as it originally did) made template list print a wall of zeros +/// once the server switched to summaries. +/// +/// +/// +/// JSON output is left untouched (callers pass this only on the table path), and +/// the command's --detail flag skips the projection to render the raw +/// payload as-is. +/// /// internal static class TemplateTableProjection { @@ -67,11 +85,11 @@ internal static class TemplateTableProjection ["description"] = Str(element, "description"), ["parentTemplateId"] = Int(element, "parentTemplateId"), ["isDerived"] = Bool(element, "isDerived"), - ["#attrs"] = Count(element, "attributes"), - ["#alarms"] = Count(element, "alarms"), - ["#scripts"] = Count(element, "scripts"), - ["#comps"] = Count(element, "compositions"), - ["#nativeAlarms"] = Count(element, "nativeAlarmSources"), + ["#attrs"] = Count(element, "attributeCount", "attributes"), + ["#alarms"] = Count(element, "alarmCount", "alarms"), + ["#scripts"] = Count(element, "scriptCount", "scripts"), + ["#comps"] = Count(element, "compositionCount", "compositions"), + ["#nativeAlarms"] = Count(element, "nativeAlarmSourceCount", "nativeAlarmSources"), }; } @@ -104,9 +122,28 @@ internal static class TemplateTableProjection ? JsonValue.Create(v.GetBoolean()) : null; - private static JsonNode Count(JsonElement obj, string name) - => JsonValue.Create( - TryGetPropertyCI(obj, name, out var v) && v.ValueKind == JsonValueKind.Array + /// + /// Member count for a column, preferring the summary payload's pre-computed + /// scalar () and falling back to the length of the + /// full entity's child array (). Zero when neither + /// is present. + /// + /// The template object being projected. + /// Scalar count property on a TemplateSummary row. + /// Child-collection property on a full Template entity. + /// The member count as a JSON number. + private static JsonNode Count(JsonElement obj, string countName, string arrayName) + { + if (TryGetPropertyCI(obj, countName, out var scalar) + && scalar.ValueKind == JsonValueKind.Number + && scalar.TryGetInt32(out var n)) + { + return JsonValue.Create(n); + } + + return JsonValue.Create( + TryGetPropertyCI(obj, arrayName, out var v) && v.ValueKind == JsonValueKind.Array ? v.GetArrayLength() : 0); + } } diff --git a/src/ZB.MOM.WW.ScadaBridge.CLI/ManagementHttpClient.cs b/src/ZB.MOM.WW.ScadaBridge.CLI/ManagementHttpClient.cs index e1b62e62..a65f5f09 100644 --- a/src/ZB.MOM.WW.ScadaBridge.CLI/ManagementHttpClient.cs +++ b/src/ZB.MOM.WW.ScadaBridge.CLI/ManagementHttpClient.cs @@ -9,13 +9,79 @@ public class ManagementHttpClient : IDisposable private readonly HttpClient _httpClient; /// - /// Initializes a new instance of the class. + /// Default bound on the CONNECT phase only (30 s) — how long a TCP/TLS + /// connection attempt to a black-holed management endpoint may hang before the + /// call fails. + /// + /// + /// It is deliberately NOT an overall request timeout. + /// caps the whole request/response, so setting + /// it to any fixed value silently truncates every caller whose own per-call + /// TimeSpan timeout argument is longer: the effective deadline becomes + /// min(HttpClient.Timeout, caller timeout). That is exactly what a 30 s + /// client timeout did to deploy site's 5-minute bulk deploy and to the + /// five-minute bundle export/preview/import calls — each printed a fake + /// 504 Request timed out at 30 s while the server carried on working. + /// is therefore + /// and the per-call + /// in + /// // + /// is the SINGLE overall deadline — it bounds connect too, since the token is + /// passed into the send itself. + /// + /// + /// + /// The connect bound lives on + /// instead, which is connect-scoped and so cannot truncate a long-running + /// request that has already reached the server. Overridable via the + /// SCADABRIDGE_HTTP_CONNECT_TIMEOUT_SECONDS environment variable, + /// consistent with how every other CLI setting is overridden (see + /// ) — kept self-contained here (no + /// /command-file plumbing) since CLI commands are owned + /// by a separate work package this phase. + /// + /// + public static readonly TimeSpan DefaultConnectTimeout = TimeSpan.FromSeconds(30); + + /// Test seam — the effective this instance was constructed with. + internal TimeSpan EffectiveTimeout { get; } + + /// + /// Resolves the effective connect timeout: the + /// SCADABRIDGE_HTTP_CONNECT_TIMEOUT_SECONDS environment variable when set + /// to a positive integer, otherwise . + /// + /// The connect timeout to apply to the socket handler. + internal static TimeSpan ResolveConnectTimeout() + { + var env = Environment.GetEnvironmentVariable("SCADABRIDGE_HTTP_CONNECT_TIMEOUT_SECONDS"); + if (!string.IsNullOrWhiteSpace(env) + && int.TryParse(env, out var seconds) + && seconds > 0) + { + return TimeSpan.FromSeconds(seconds); + } + + return DefaultConnectTimeout; + } + + /// + /// Initializes a new instance of the class with + /// an INFINITE (each call supplies its own + /// deadline) over a whose + /// is + /// . /// /// The base URL for the management API. /// The username for HTTP Basic authentication. /// The password for HTTP Basic authentication. public ManagementHttpClient(string baseUrl, string username, string password) - : this(new HttpClient(), baseUrl, username, password) + : this( + new HttpClient(new SocketsHttpHandler { ConnectTimeout = ResolveConnectTimeout() }) + { + Timeout = Timeout.InfiniteTimeSpan + }, + baseUrl, username, password) { } @@ -31,6 +97,9 @@ public class ManagementHttpClient : IDisposable internal ManagementHttpClient(HttpClient httpClient, string baseUrl, string username, string password) { _httpClient = httpClient; + // Test seam (WP2.6e): exposes the constructed HttpClient's effective Timeout + // without requiring reflection. + EffectiveTimeout = httpClient.Timeout; _httpClient.BaseAddress = new Uri(baseUrl.TrimEnd('/') + "/"); var credentials = Convert.ToBase64String(Encoding.UTF8.GetBytes($"{username}:{password}")); _httpClient.DefaultRequestHeaders.Authorization = diff --git a/src/ZB.MOM.WW.ScadaBridge.CLI/README.md b/src/ZB.MOM.WW.ScadaBridge.CLI/README.md index 28c7b912..c4d7f802 100644 --- a/src/ZB.MOM.WW.ScadaBridge.CLI/README.md +++ b/src/ZB.MOM.WW.ScadaBridge.CLI/README.md @@ -86,28 +86,33 @@ Exit codes: #### `template list` -List all templates. **Table** output (`--format table`) shows a compact summary — id, -name, description, parent, and member **counts** (`#attrs`, `#alarms`, `#scripts`, -`#comps`, `#nativeAlarms`) — so it stays readable in a terminal. Add `--detail` to dump -the full attribute/alarm/script/composition definitions in the table. **JSON** output -(`--format json`) is always the full, unmodified payload regardless of `--detail`. +List all templates. The server projects this listing in the database and returns +**summary rows** — no child collections, member counts only. **Table** output +(`--format table`) renders them as id, name, description, parent, and member **counts** +(`#attrs`, `#alarms`, `#scripts`, `#comps`, `#nativeAlarms`), so it stays readable in a +terminal. `--detail` skips that column projection and renders the summary payload +verbatim; it does **not** add attribute/alarm/script definitions — a listing never +carries them. Use `template get --id ` for a template's full definition. **JSON** +output (`--format json`) is always the unmodified payload regardless of `--detail`. ```sh -scadabridge --url template list # compact table -scadabridge --url --format table template list --detail # full table dump -scadabridge --url --format json template list # full JSON (always) +scadabridge --url template list # compact table +scadabridge --url --format table template list --detail # raw summary rows +scadabridge --url --format json template list # unmodified JSON (always) ``` | Option | Required | Description | |--------|----------|-------------| -| `--detail` | no | Include full definitions in table output (no effect on JSON) | +| `--detail` | no | Render the raw summary payload instead of the compact column projection (no effect on JSON; does not add definitions) | | `--skip` | no | Offset paging: number of items to skip (default 0) | | `--take` | no | Offset paging: max items to return (clamped 1..1000; omit for unlimited) | #### `template get` -Get a single template by ID. Like `template list`, table output is a compact summary -unless `--detail` is supplied; JSON output is always full. +Get a single template by ID. Unlike `template list`, this returns the **full** template +entity with every attribute, alarm, script and composition inline. Table output is a +compact summary (the same counts as `template list`) unless `--detail` is supplied, which +dumps the full definitions; JSON output is always full. ```sh scadabridge --url template get --id [--detail] @@ -952,6 +957,37 @@ scadabridge --url deploy instance --id |--------|----------|-------------| | `--id` | yes | Instance ID | +#### `deploy site` + +Deploy **every deployable instance** at one site in a single batched operation. + +Each instance keeps the full single-instance semantics — its own deployment ID and +revision hash, its own per-instance operation lock, its own optimistically-concurrent +status record, and the same query-before-redeploy idempotency check. The batch is +**not** all-or-nothing: an instance that cannot be deployed (wrong state, failed +validation, lock already held, site round-trip timed out) is reported as a failed row +and the rest proceed. Each failed instance is individually retryable with +`instance deploy --id`. + +Site round-trips run with bounded concurrency +(`ScadaBridge:DeploymentManager:SiteDeploymentMaxParallelism`, default 4), each under +`SiteDeploymentTimeoutPerInstance` (default 120 s), so one wedged instance cannot stall +the batch. The client waits up to 5 minutes. + +`--site-id` is **required** — unlike `deploy artifacts`, omitting it does not mean +fleet-wide. + +```sh +scadabridge --url deploy site --site-id +``` + +| Option | Required | Description | +|--------|----------|-------------| +| `--site-id` | yes | Target site ID. A site-scoped (non-Administrator) Deployer must supply an in-scope site. | + +Output is a per-instance result matrix (`instanceId`, `uniqueName`, `deploymentId`, +`success`, `errorMessage`) plus `successCount` / `failureCount`. + #### `deploy artifacts` Deploy compiled artifacts to one or all sites (same as `site deploy-artifacts`). diff --git a/src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Deployment/DebugView.razor b/src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Deployment/DebugView.razor index 7238786c..0051a151 100644 --- a/src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Deployment/DebugView.razor +++ b/src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Deployment/DebugView.razor @@ -291,13 +291,56 @@ .OrderBy(a => a.AttributeName) .ToList(); - /// Attribute composition forest, rebuilt from the live latest-per-name dictionary. - private IReadOnlyList AttributeForest => - DebugTreeBuilder.BuildAttributeTree(_attributeValues.Values, _attrFilter); + // ── Tree memoization (arch-review WP2.4) ────────────────────────────────── + // Both forests used to be rebuilt from scratch on EVERY render — and a render + // fired per streamed event. A busy instance therefore rebuilt two whole + // composition trees hundreds of times a second, on the circuit's single + // dispatcher thread. The rebuild now sits behind a version stamp bumped + // whenever the underlying dictionaries change (a coalesced stream flush, + // Connect, Disconnect), so an unrelated re-render (tab switch, toast, status + // strip tick) reuses the forest instead of rebuilding it. Reusing the same + // instance also lets Blazor skip re-rendering the TreeView subtree entirely. + // + // The cached key includes the dictionary Count as well as the version: the + // trees must never go stale if the dictionaries are populated by some path + // that forgets to bump the stamp. + private int _dataVersion; - /// Alarm composition forest, rebuilt from the live latest-per-name dictionary. - private IReadOnlyList AlarmForest => - DebugTreeBuilder.BuildAlarmTree(_alarmStates.Values, _alarmFilter); + private IReadOnlyList? _attrForestCache; + private (int Version, int Count, string Filter) _attrForestKey = (-1, -1, string.Empty); + + private IReadOnlyList? _alarmForestCache; + private (int Version, int Count, string Filter) _alarmForestKey = (-1, -1, string.Empty); + + /// Attribute composition forest, rebuilt only when the underlying data or filter changed. + private IReadOnlyList AttributeForest + { + get + { + var key = (_dataVersion, _attributeValues.Count, _attrFilter); + if (_attrForestCache is null || _attrForestKey != key) + { + _attrForestCache = DebugTreeBuilder.BuildAttributeTree(_attributeValues.Values, _attrFilter); + _attrForestKey = key; + } + return _attrForestCache; + } + } + + /// Alarm composition forest, rebuilt only when the underlying data or filter changed. + private IReadOnlyList AlarmForest + { + get + { + var key = (_dataVersion, _alarmStates.Count, _alarmFilter); + if (_alarmForestCache is null || _alarmForestKey != key) + { + _alarmForestCache = DebugTreeBuilder.BuildAlarmTree(_alarmStates.Values, _alarmFilter); + _alarmForestKey = key; + } + return _alarmForestCache; + } + } private DebugStreamSession? _session; private ToastNotification _toast = default!; @@ -458,6 +501,9 @@ foreach (var al in session.InitialSnapshot.AlarmStates) _alarmStates[al.AlarmName] = al; + // Snapshot seeding is a data change like any other — bump the stamp so + // the memoized forests rebuild from it. + _dataVersion++; _snapshot = session.InitialSnapshot; _connected = true; @@ -492,6 +538,11 @@ _snapshot = null; _attributeValues.Clear(); _alarmStates.Clear(); + // Drop anything the stream buffered for the session we just tore down, so a + // late flush can't repopulate a disconnected view. + _pendingAttributes.Clear(); + _pendingAlarms.Clear(); + _dataVersion++; } /// @@ -508,37 +559,108 @@ _toast.ShowInfo("Cleared previous session — select a site and instance to begin.", autoDismissMs: 5000); } + // ── Render coalescing (arch-review WP2.4) ───────────────────────────────── + // Every streamed event used to marshal onto the dispatcher and call + // StateHasChanged individually, so a chatty instance drove one full render + // (and two full tree rebuilds) per value change. Events now land in a + // thread-safe pending map keyed by name — repeated updates to the same tag + // inside one window collapse to the latest — and exactly ONE dispatcher + // marshal per window drains them, bumps the version stamp and renders once. + private const int RenderCoalesceMs = 250; + + private readonly System.Collections.Concurrent.ConcurrentDictionary _pendingAttributes = new(); + private readonly System.Collections.Concurrent.ConcurrentDictionary _pendingAlarms = new(); + + /// 0/1 flag: whether a coalesce window is already armed. Interlocked-guarded. + private int _flushArmed; + /// /// Handles one debug-stream event. The callback is invoked on an Akka/gRPC /// thread, but / are /// instances also enumerated by the /// render thread (the tree forests + are /// built from them). Dictionary is not thread-safe (CentralUI-021): a - /// write racing an enumeration can throw or corrupt the buckets. The mutation - /// is therefore marshalled onto the renderer's dispatcher via - /// so every access to the dictionaries — read - /// and write — happens on the render thread. + /// write racing an enumeration can throw or corrupt the buckets. The event is + /// therefore parked in a concurrent pending map here and applied on the + /// renderer's dispatcher by , so every access to + /// the render dictionaries — read and write — still happens on one thread. /// private void HandleStreamEvent(object evt) { // CentralUI-009: the component may have been disposed while this event // was in flight on the Akka/gRPC thread. if (_disposed) return; - _ = SafeInvokeAsync(() => + + switch (evt) { + case AttributeValueChanged av: + _pendingAttributes[av.AttributeName] = av; + break; + case AlarmStateChanged al: + _pendingAlarms[al.AlarmName] = al; + break; + default: + // Unknown event type — nothing to apply, no render needed. + return; + } + + // Arm the window if it isn't already. Losers of the CAS have nothing to do: + // their event is in the pending map and the armed flush will pick it up. + if (Interlocked.CompareExchange(ref _flushArmed, 1, 0) == 0) + { + _ = FlushPendingAsync(); + } + } + + /// + /// Waits out the coalesce window, then applies everything buffered in one + /// dispatcher pass and renders once. + /// + private async Task FlushPendingAsync() + { + try + { + await Task.Delay(RenderCoalesceMs); + } + catch + { + // Delay never faults in practice; if it ever did, fall through and drain. + } + + if (_disposed) + { + Volatile.Write(ref _flushArmed, 0); + return; + } + + await SafeInvokeAsync(() => + { + // Disarm BEFORE draining: an event arriving mid-drain then arms a fresh + // window rather than being stranded until the next unrelated event. + Volatile.Write(ref _flushArmed, 0); if (_disposed) return; - switch (evt) + + var applied = false; + foreach (var key in _pendingAttributes.Keys) { - case AttributeValueChanged av: + if (_pendingAttributes.TryRemove(key, out var av)) + { _attributeValues[av.AttributeName] = av; - break; - case AlarmStateChanged al: - _alarmStates[al.AlarmName] = al; - break; - default: - // Unknown event type — no re-render needed. - return; + applied = true; + } } + foreach (var key in _pendingAlarms.Keys) + { + if (_pendingAlarms.TryRemove(key, out var al)) + { + _alarmStates[al.AlarmName] = al; + applied = true; + } + } + + if (!applied) return; + + _dataVersion++; StateHasChanged(); }); } diff --git a/src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Deployment/Deployments.razor b/src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Deployment/Deployments.razor index 33b8ccd8..69bedd7a 100644 --- a/src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Deployment/Deployments.razor +++ b/src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Deployment/Deployments.razor @@ -236,13 +236,66 @@ DeploymentStatusNotifier.StatusChanged += OnDeploymentStatusChanged; } + // ── Push coalescing (arch-review WP2.4) ─────────────────────────────────── + // One reload = every deployment record + every instance, re-read and re-filtered. + // The notifier fires per status WRITE, so a site deploy of N instances produced + // 2N+ of those full reloads back to back, per circuit. The reload is now + // leading-edge debounced: the first push after an idle gap still reloads + // immediately (a single deployment stays as responsive as before), and every + // push inside the window is absorbed into ONE trailing reload. + private const int ReloadDebounceMs = 500; + + private DateTimeOffset _lastReloadAt = DateTimeOffset.MinValue; + + /// Coalescing timer for pushes arriving inside the debounce window. Disposed with the component. + private Timer? _coalesceTimer; + private void OnDeploymentStatusChanged(ZB.MOM.WW.ScadaBridge.DeploymentManager.DeploymentStatusChange change) { // CentralUI-022: a callback racing disposal must not touch the component. if (_disposed || !_autoRefresh) return; + + lock (_coalesceLock) + { + if (_disposed) return; + + var sinceLast = DateTimeOffset.UtcNow - _lastReloadAt; + if (sinceLast >= TimeSpan.FromMilliseconds(ReloadDebounceMs)) + { + // Idle — reload straight away (leading edge). + _lastReloadAt = DateTimeOffset.UtcNow; + } + else + { + // Inside the window: arm one trailing reload for the remainder, and + // let any further push in this window ride it. + if (_coalesceTimer is not null) return; + + var delay = TimeSpan.FromMilliseconds(ReloadDebounceMs) - sinceLast; + _coalesceTimer = new Timer(_ => OnCoalesceElapsed(), null, delay, Timeout.InfiniteTimeSpan); + return; + } + } + _ = DispatchReloadAsync(); } + /// Trailing edge of the debounce: disarm, stamp, and run the one coalesced reload. + private void OnCoalesceElapsed() + { + lock (_coalesceLock) + { + _coalesceTimer?.Dispose(); + _coalesceTimer = null; + if (_disposed) return; + _lastReloadAt = DateTimeOffset.UtcNow; + } + + _ = DispatchReloadAsync(); + } + + private readonly object _coalesceLock = new(); + /// /// Reloads the deployment table on the renderer's dispatcher, guarded /// against the component being disposed mid-flight (CentralUI-022): @@ -361,5 +414,10 @@ // status change reaches this disposed component. _disposed = true; DeploymentStatusNotifier.StatusChanged -= OnDeploymentStatusChanged; + lock (_coalesceLock) + { + _coalesceTimer?.Dispose(); + _coalesceTimer = null; + } } } diff --git a/src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Monitoring/AlarmSummary.razor b/src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Monitoring/AlarmSummary.razor index c84aeb59..284486e2 100644 --- a/src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Monitoring/AlarmSummary.razor +++ b/src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Monitoring/AlarmSummary.razor @@ -1,5 +1,6 @@ @page "/monitoring/alarms" @attribute [Authorize(Policy = ZB.MOM.WW.ScadaBridge.Security.AuthorizationPolicies.RequireDeployment)] +@using Microsoft.AspNetCore.Components.Web.Virtualization @using ZB.MOM.WW.ScadaBridge.CentralUI.Components.Shared @using ZB.MOM.WW.ScadaBridge.CentralUI.Services @using ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites @@ -196,15 +197,25 @@ @onkeydown='e => OnHeaderKeyDown(e, "severity")' @onkeydown:preventDefault="_preventHeaderDefault">Severity @SortGlyph("severity") + @* Row markup is identical on both paths — the same , the same cells, + the same data-test hook. Only WHO enumerates it differs: a plain + foreach for the ordinary case, and Virtualize once the flat table + gets long enough that rendering every row per delta dominates the + circuit (arch-review WP2.4). Virtualize needs spacers or the + browser hoists its
s out of the table and the layout collapses. *@ - @foreach (var row in _visibleRows) + @if (_visibleRows.Count <= VirtualizeThreshold) { - - @row.InstanceUniqueName - @row.Alarm.AlarmName - - @row.Alarm.Condition.Severity - + @foreach (var row in _visibleRows) + { + @AlarmRow(row) + } + } + else + { + + @AlarmRow(row) + } @@ -222,7 +233,23 @@ // P4 (arch-review): the filtered+sorted view is memoized rather than recomputed // on every render. RecomputeVisibleRows() refreshes it whenever the inputs change // (a fresh snapshot in RefreshAsync, a filter change via @bind:after, or a sort). - private IReadOnlyList _visibleRows = Array.Empty(); + // Concrete List because Virtualize binds ICollection, not IReadOnlyList. + private List _visibleRows = new(); + + // Above this many visible rows the flat table switches to Virtualize. Below it the + // plain foreach is cheaper than the virtualization machinery (and needs no JS), and + // an operator filtered down to a handful of alarms should never pay for either. + private const int VirtualizeThreshold = 150; + + // One row template, shared by the plain and virtualized paths so the two can never + // drift visually. + private RenderFragment AlarmRow => row => + @ + @row.InstanceUniqueName + @row.Alarm.AlarmName + + @row.Alarm.Condition.Severity + ; private IReadOnlyList _notReporting = Array.Empty(); private AlarmRollup _rollup = new(0, 0, 0, new Dictionary()); @@ -304,8 +331,10 @@ return; } - // _notReporting is the poll's unique authority — the alarm-only live cache - // cannot compute it — so it is always refreshed. + // _notReporting always comes from this call, live or not. While the cache is + // serving the site the façade sources it from the aggregator's own fan-out (no + // second fan-out is run); while it is cold the poll computes it. Either way the + // shape and ordering are identical. _notReporting = result.NotReportingInstances; // While the cache is live, the live deltas own the row set: a poll whose fan-out diff --git a/src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Monitoring/Health.razor b/src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Monitoring/Health.razor index beb3743e..bbcd16a6 100644 --- a/src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Monitoring/Health.razor +++ b/src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Monitoring/Health.razor @@ -15,8 +15,7 @@ @implements IDisposable @inject ICentralHealthAggregator HealthAggregator @inject ISiteRepository SiteRepository -@inject CommunicationService CommunicationService -@inject IAuditLogQueryService AuditLogQueryService +@inject IKpiSnapshotCache KpiSnapshots @inject IKpiHistoryQueryService KpiHistory @inject Microsoft.Extensions.Options.IOptions HealthOptions @@ -640,7 +639,9 @@ // Non-fatal — fall back to showing siteId only } - await RefreshNow(); + // First load rides whatever the shared KPI cache already has fresh — a second + // operator opening the dashboard inside the window costs no extra query. + await RefreshAllAsync(forceRefresh: false); // Site Health Trends (M6) load on their own path — never from the // timer tick below — so a trend-query fault can't disturb the live tile @@ -655,7 +656,7 @@ { try { - await RefreshNow(); + await RefreshAllAsync(forceRefresh: false); StateHasChanged(); } finally @@ -769,20 +770,38 @@ ? "Central Cluster" : $"{GetSiteName(siteKey)} ({siteKey})"; - private async Task RefreshNow() + // Per-tile budget for one refresh tick. The underlying KPI Asks carry the + // 30s CommunicationOptions.QueryTimeout, so without this a single hung + // singleton stalled the WHOLE tick (and, with a 10s timer, stacked ticks) + // rather than degrading its own tile. Cancelling here abandons only this + // circuit's wait — the shared KPI flight continues and its result lands in + // the process-level cache for the next tick. + private static readonly TimeSpan TileTimeout = TimeSpan.FromSeconds(5); + + // The operator's explicit "Refresh Now" bypasses the shared cache's freshness + // window; the timer tick and the first load ride whatever is already fresh + // (which is the whole point — N dashboards share one query per window). + private Task RefreshNow() => RefreshAllAsync(forceRefresh: true); + + // Every tile loads concurrently, each with its own short budget, so one slow + // KPI degrades one tile instead of the tick. Every loader swallows its own + // faults, so Task.WhenAll here can never observe an exception. + private async Task RefreshAllAsync(bool forceRefresh) { _siteStates = HealthAggregator.GetAllSiteStates(); - await LoadOutboxKpis(); - await Task.WhenAll(LoadSiteCallKpis(), LoadSiteCallNodeKpis()); - await LoadAuditKpis(); + await Task.WhenAll( + LoadOutboxKpis(forceRefresh), + LoadSiteCallKpis(forceRefresh), + LoadSiteCallNodeKpis(forceRefresh), + LoadAuditKpis(forceRefresh)); } - private async Task LoadOutboxKpis() + private async Task LoadOutboxKpis(bool forceRefresh) { try { - var response = await CommunicationService.GetNotificationKpisAsync( - new NotificationKpiRequest(Guid.NewGuid().ToString("N"))); + using var timeout = new CancellationTokenSource(TileTimeout); + var response = await KpiSnapshots.GetNotificationKpisAsync(forceRefresh, timeout.Token); if (response.Success) { _outboxKpi = response; @@ -795,6 +814,11 @@ _outboxKpiError = response.ErrorMessage ?? "KPI query failed."; } } + catch (OperationCanceledException) + { + _outboxKpiAvailable = false; + _outboxKpiError = "KPI query timed out."; + } catch (Exception ex) { _outboxKpiAvailable = false; @@ -807,12 +831,12 @@ // killing the dashboard. Mirrors LoadOutboxKpis's error handling shape — a // response with Success == false (repository fault) and an Ask that threw // (transport fault) both collapse to "unavailable". - private async Task LoadSiteCallKpis() + private async Task LoadSiteCallKpis(bool forceRefresh) { try { - var response = await CommunicationService.GetSiteCallKpisAsync( - new SiteCallKpiRequest(Guid.NewGuid().ToString("N"))); + using var timeout = new CancellationTokenSource(TileTimeout); + var response = await KpiSnapshots.GetSiteCallKpisAsync(forceRefresh, timeout.Token); if (response.Success) { _siteCallKpi = response; @@ -825,6 +849,11 @@ _siteCallKpiError = response.ErrorMessage ?? "KPI query failed."; } } + catch (OperationCanceledException) + { + _siteCallKpiAvailable = false; + _siteCallKpiError = "KPI query timed out."; + } catch (Exception ex) { _siteCallKpiAvailable = false; @@ -834,12 +863,12 @@ // Per-node site-call KPI loader (M5.2). Best-effort; a fault silently // suppresses the per-node sub-table rather than degrading the dashboard. - private async Task LoadSiteCallNodeKpis() + private async Task LoadSiteCallNodeKpis(bool forceRefresh) { try { - var response = await CommunicationService.GetPerNodeSiteCallKpisAsync( - new PerNodeSiteCallKpiRequest(Guid.NewGuid().ToString("N"))); + using var timeout = new CancellationTokenSource(TileTimeout); + var response = await KpiSnapshots.GetPerNodeSiteCallKpisAsync(forceRefresh, timeout.Token); if (response.Success) { _siteCallNodeKpis = response.Nodes; @@ -864,14 +893,20 @@ // Audit KPI loader: wraps the service call so a transient DB outage degrades // the three tiles to em dashes with an inline error rather than killing the // dashboard. Mirrors LoadOutboxKpis's error handling shape. - private async Task LoadAuditKpis() + private async Task LoadAuditKpis(bool forceRefresh) { try { - _auditKpi = await AuditLogQueryService.GetKpiSnapshotAsync(); + using var timeout = new CancellationTokenSource(TileTimeout); + _auditKpi = await KpiSnapshots.GetAuditKpiSnapshotAsync(forceRefresh, timeout.Token); _auditKpiAvailable = true; _auditKpiError = null; } + catch (OperationCanceledException) + { + _auditKpiAvailable = false; + _auditKpiError = "KPI query timed out."; + } catch (Exception ex) { _auditKpiAvailable = false; diff --git a/src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Notifications/NotificationKpis.razor b/src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Notifications/NotificationKpis.razor index 93df1e40..a8793aab 100644 --- a/src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Notifications/NotificationKpis.razor +++ b/src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Notifications/NotificationKpis.razor @@ -7,7 +7,7 @@ @using ZB.MOM.WW.ScadaBridge.Commons.Types.Kpi @using ZB.MOM.WW.ScadaBridge.Communication @using ZB.MOM.WW.ScadaBridge.CentralUI.Services -@inject CommunicationService CommunicationService +@inject IKpiSnapshotCache KpiSnapshots @inject ISiteRepository SiteRepository @inject IKpiHistoryQueryService KpiHistory @inject ILogger Logger @@ -250,15 +250,24 @@ Logger.LogWarning(ex, "Failed to load sites for the KPI per-site breakdown."); } - await RefreshAll(); + // First load rides whatever the shared KPI cache already holds fresh; the + // Refresh button forces a new query (see RefreshAll). + await LoadAllAsync(forceRefresh: false); } - private async Task RefreshAll() + // The operator's explicit Refresh bypasses the shared cache's freshness window. + private Task RefreshAll() => LoadAllAsync(forceRefresh: true); + + private async Task LoadAllAsync(bool forceRefresh) { _loading = true; // Race-free despite all tasks mutating component fields: Blazor Server runs // every continuation on the circuit's single-threaded synchronization context. - await Task.WhenAll(LoadGlobalKpis(), LoadPerSiteKpis(), LoadPerNodeKpis(), LoadTrends()); + await Task.WhenAll( + LoadGlobalKpis(forceRefresh), + LoadPerSiteKpis(forceRefresh), + LoadPerNodeKpis(forceRefresh), + LoadTrends()); _loading = false; } @@ -320,12 +329,11 @@ } } - private async Task LoadGlobalKpis() + private async Task LoadGlobalKpis(bool forceRefresh) { try { - var response = await CommunicationService.GetNotificationKpisAsync( - new NotificationKpiRequest(Guid.NewGuid().ToString("N"))); + var response = await KpiSnapshots.GetNotificationKpisAsync(forceRefresh); if (response.Success) { _kpi = response; @@ -342,12 +350,11 @@ } } - private async Task LoadPerSiteKpis() + private async Task LoadPerSiteKpis(bool forceRefresh) { try { - var response = await CommunicationService.GetPerSiteNotificationKpisAsync( - new PerSiteNotificationKpiRequest(Guid.NewGuid().ToString("N"))); + var response = await KpiSnapshots.GetPerSiteNotificationKpisAsync(forceRefresh); if (response.Success) { _perSite = response.Sites; @@ -364,12 +371,11 @@ } } - private async Task LoadPerNodeKpis() + private async Task LoadPerNodeKpis(bool forceRefresh) { try { - var response = await CommunicationService.GetPerNodeNotificationKpisAsync( - new PerNodeNotificationKpiRequest(Guid.NewGuid().ToString("N"))); + var response = await KpiSnapshots.GetPerNodeNotificationKpisAsync(forceRefresh); if (response.Success) { _perNode = response.Nodes; diff --git a/src/ZB.MOM.WW.ScadaBridge.CentralUI/ScriptAnalysis/ScriptAnalysisService.cs b/src/ZB.MOM.WW.ScadaBridge.CentralUI/ScriptAnalysis/ScriptAnalysisService.cs index 5157ed09..6b6a0882 100644 --- a/src/ZB.MOM.WW.ScadaBridge.CentralUI/ScriptAnalysis/ScriptAnalysisService.cs +++ b/src/ZB.MOM.WW.ScadaBridge.CentralUI/ScriptAnalysis/ScriptAnalysisService.cs @@ -177,8 +177,42 @@ public class ScriptAnalysisService private const int SandboxMaxConsoleChars = 32_000; private const int SandboxMaxReturnJsonChars = 32_000; + /// + /// Shared options for the per-Test-Run return-value JSON serialize in + /// (perf remediation, arch-review WP1.5) — avoids + /// allocating a new per run. Settings preserved + /// exactly. + /// + private static readonly JsonSerializerOptions SandboxReturnJsonOptions = + new() { WriteIndented = true }; + private const int SandboxMaxCallSharedDepth = 16; + /// + /// Process-wide cap on concurrently executing Test Runs (arch-review WP2.4). + /// + /// A sandbox run occupies a thread-pool thread for its whole duration — the + /// SandboxScriptHost attribute accessors are synchronous by contract and block on + /// cross-site I/O (GetAwaiter().GetResult()), so the thread is parked, not + /// merely busy, for up to the 10s sandbox timeout. Unbounded, a handful of Designers + /// each hitting Run on a slow-bound script could park enough pool threads to starve the + /// Blazor circuits sharing this process. The gate bounds that: excess runs wait for a + /// slot instead of taking a thread, and the wait is charged to the caller's own timeout. + /// + /// + /// Static because ScriptAnalysisService is registered scoped — a per-instance gate + /// would be per-request and bound nothing. + /// + /// + private static readonly SemaphoreSlim SandboxRunGate = + new(SandboxMaxConcurrentRuns, SandboxMaxConcurrentRuns); + + /// + /// Slot count for . Deliberately small: Test Run is an + /// authoring convenience on a node whose real job is serving circuits. + /// + private const int SandboxMaxConcurrentRuns = 4; + /// /// Compiles and runs a script in the central process. The globals surface /// depends on : template and shared @@ -428,8 +462,15 @@ public class ScriptAnalysisService using var errorScope = captureError.BeginCapture(captured); var stopwatch = Stopwatch.StartNew(); + var gateAcquired = false; try { + // Bound concurrent Test Runs (see SandboxRunGate). The wait is charged to the + // caller's own linked token, so queue time counts against the sandbox timeout + // and a queued run reports Timeout rather than hanging. + await SandboxRunGate.WaitAsync(linkedCts.Token).ConfigureAwait(false); + gateAcquired = true; + // Run on a thread-pool thread with no SynchronizationContext: a // bound script's Instance.SetAttribute / Attributes[...] block // synchronously on cross-site I/O (the API surface is sync by @@ -480,6 +521,10 @@ public class ScriptAnalysisService $"{inner.GetType().Name}: {inner.Message}", SandboxErrorKind.RuntimeError, stopwatch.ElapsedMilliseconds, null); } + finally + { + if (gateAcquired) SandboxRunGate.Release(); + } // outScope / errorScope are disposed by their `using` declarations when the // method returns, restoring the previous capture scope on this call-tree // without touching process-global Console state. @@ -520,7 +565,7 @@ public class ScriptAnalysisService var typeName = value.GetType().Name; try { - var json = JsonSerializer.Serialize(value, new JsonSerializerOptions { WriteIndented = true }); + var json = JsonSerializer.Serialize(value, SandboxReturnJsonOptions); if (json.Length > SandboxMaxReturnJsonChars) json = json[..SandboxMaxReturnJsonChars] + "\n… (truncated)"; return (json, typeName); diff --git a/src/ZB.MOM.WW.ScadaBridge.CentralUI/ServiceCollectionExtensions.cs b/src/ZB.MOM.WW.ScadaBridge.CentralUI/ServiceCollectionExtensions.cs index 7ba4f291..2509313a 100644 --- a/src/ZB.MOM.WW.ScadaBridge.CentralUI/ServiceCollectionExtensions.cs +++ b/src/ZB.MOM.WW.ScadaBridge.CentralUI/ServiceCollectionExtensions.cs @@ -99,7 +99,17 @@ public static class ServiceCollectionExtensions // facade over CommunicationService.RequestDebugSnapshotAsync (the same // single-shot Ask the Debug View uses) — and flattens the alarm states. services.AddScoped(); - services.AddScoped(); + // The fan-out itself stays scoped (it holds scoped repositories + the snapshot + // client), but the page resolves it through SharedAlarmSummaryService — a process + // singleton that memoizes the fan-out per site with single-flight, so N circuits + // watching one site cost ONE fan-out per window instead of N (arch-review WP2.4). + services.AddScoped(); + services.AddSingleton(); + + // Process-level memoized point-in-time KPI snapshots (arch-review WP2.4, finding + // #8's per-circuit half). Singleton on purpose: the memo has to outlive any one + // circuit for N dashboards to share a single SQL round per refresh window. + services.AddSingleton(); // Secured Writes: dispatches the two-person secured-write commands // (submit / approve / reject / list) to the central ManagementActor through the diff --git a/src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/AlarmSummaryService.cs b/src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/AlarmSummaryService.cs index 0e2bcc9d..1d33fc82 100644 --- a/src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/AlarmSummaryService.cs +++ b/src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/AlarmSummaryService.cs @@ -128,7 +128,20 @@ public sealed class AlarmSummaryService : IAlarmSummaryService } /// - public AlarmSummaryResult BuildFromLiveAlarms(IReadOnlyList alarms) + public AlarmSummaryResult BuildFromLiveAlarms(IReadOnlyList alarms) => + BuildFromLiveAlarmsCore(alarms); + + /// + public AlarmRollup ComputeRollup(IReadOnlyList rows) => ComputeRollupCore(rows); + + /// + /// Pure flattening of a live-cache snapshot into summary rows. Shared with + /// , which memoizes the fan-out but must produce + /// byte-identical rows on the live path — one implementation, no drift. + /// + /// The live-cache alarm snapshot. + /// The flattened, deterministically ordered rows (never any not-reporting names). + internal static AlarmSummaryResult BuildFromLiveAlarmsCore(IReadOnlyList alarms) { ArgumentNullException.ThrowIfNull(alarms); @@ -141,14 +154,22 @@ public sealed class AlarmSummaryService : IAlarmSummaryService .ThenBy(r => r.Alarm.AlarmName, StringComparer.OrdinalIgnoreCase) .ToList(); - // The live cache is alarm-only, so it cannot enumerate "not reporting" - // instances — left empty here; the periodic poll (GetSiteAlarmsAsync) - // remains the authority for that list. + // Alarm rows alone cannot name the instances that failed to answer, so this pure + // flattening leaves the list empty. The names come from the fan-out that produced the + // snapshot: while the live cache is serving a site, SharedAlarmSummaryService pairs + // these rows with ISiteAlarmLiveCache.GetNotReportingInstances (the aggregator's own + // seed/reconcile result); while it is cold, GetSiteAlarmsAsync's poll supplies both. return new AlarmSummaryResult(orderedRows, Array.Empty()); } - /// - public AlarmRollup ComputeRollup(IReadOnlyList rows) + /// + /// Pure roll-up over already-flattened rows. Shared with + /// for the same one-implementation reason as + /// . + /// + /// The rows to roll up. + /// The active/worst-severity/unacked/by-kind roll-up. + internal static AlarmRollup ComputeRollupCore(IReadOnlyList rows) { ArgumentNullException.ThrowIfNull(rows); diff --git a/src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/IKpiSnapshotCache.cs b/src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/IKpiSnapshotCache.cs new file mode 100644 index 00000000..4785c4a1 --- /dev/null +++ b/src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/IKpiSnapshotCache.cs @@ -0,0 +1,70 @@ +using ZB.MOM.WW.ScadaBridge.Commons.Messages.Audit; +using ZB.MOM.WW.ScadaBridge.Commons.Messages.Notification; +using ZB.MOM.WW.ScadaBridge.Commons.Types; + +namespace ZB.MOM.WW.ScadaBridge.CentralUI.Services; + +/// +/// Process-level memoized point-in-time KPI snapshots shared by every Blazor circuit on +/// this node (arch-review WP2.4, finding #8 — the per-circuit polling half). +/// +/// Every KPI here is a global, point-in-time aggregate: it does not vary by user, +/// site scope, or role, so N circuits asking for it inside one refresh window are asking +/// the identical question. Without memoization, ten Health dashboards meant ten Asks → +/// ten aggregate SQL round trips every ten seconds. Each accessor is independently +/// memoized with a short TTL and single-flight production, so the fleet costs ONE round +/// trip per KPI per window regardless of viewer count. +/// +/// +/// Faults are never memoized: a failed query is re-attempted by the next caller, and the +/// caller still sees the exception exactly as it would from a direct query — the pages' +/// existing per-tile "unavailable" degradation is unchanged. +/// +/// +public interface IKpiSnapshotCache +{ + /// Global Notification Outbox KPIs (queue depth, stuck, parked, delivered-last-interval). + /// Bypass a still-fresh value (operator "Refresh"); an in-flight round is still joined. + /// Cancels this caller's wait, never the shared flight. + /// The memoized notification KPI response. + Task GetNotificationKpisAsync( + bool forceRefresh = false, CancellationToken cancellationToken = default); + + /// Per-source-site Notification Outbox KPI breakdown. + /// Bypass a still-fresh value; an in-flight round is still joined. + /// Cancels this caller's wait, never the shared flight. + /// The memoized per-site notification KPI response. + Task GetPerSiteNotificationKpisAsync( + bool forceRefresh = false, CancellationToken cancellationToken = default); + + /// Per-node Notification Outbox KPI breakdown (grouped by SourceNode). + /// Bypass a still-fresh value; an in-flight round is still joined. + /// Cancels this caller's wait, never the shared flight. + /// The memoized per-node notification KPI response. + Task GetPerNodeNotificationKpisAsync( + bool forceRefresh = false, CancellationToken cancellationToken = default); + + /// Global Site Call Audit KPIs (buffered, stuck, parked). + /// Bypass a still-fresh value; an in-flight round is still joined. + /// Cancels this caller's wait, never the shared flight. + /// The memoized site call KPI response. + Task GetSiteCallKpisAsync( + bool forceRefresh = false, CancellationToken cancellationToken = default); + + /// Per-node Site Call Audit KPI breakdown (grouped by SourceNode). + /// Bypass a still-fresh value; an in-flight round is still joined. + /// Cancels this caller's wait, never the shared flight. + /// The memoized per-node site call KPI response. + Task GetPerNodeSiteCallKpisAsync( + bool forceRefresh = false, CancellationToken cancellationToken = default); + + /// + /// Audit Log KPI snapshot (1h volume + error rate over the central AuditLog table, + /// plus the summed per-site backlog). + /// + /// Bypass a still-fresh value; an in-flight round is still joined. + /// Cancels this caller's wait, never the shared flight. + /// The memoized audit KPI snapshot. + Task GetAuditKpiSnapshotAsync( + bool forceRefresh = false, CancellationToken cancellationToken = default); +} diff --git a/src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/KpiSnapshotCache.cs b/src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/KpiSnapshotCache.cs new file mode 100644 index 00000000..f18b56d7 --- /dev/null +++ b/src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/KpiSnapshotCache.cs @@ -0,0 +1,122 @@ +using Microsoft.Extensions.DependencyInjection; +using ZB.MOM.WW.ScadaBridge.Commons.Messages.Audit; +using ZB.MOM.WW.ScadaBridge.Commons.Messages.Notification; +using ZB.MOM.WW.ScadaBridge.Commons.Types; +using ZB.MOM.WW.ScadaBridge.Communication; + +namespace ZB.MOM.WW.ScadaBridge.CentralUI.Services; + +/// +/// Default — a DI singleton holding one +/// per KPI query. +/// +/// Registered as a singleton on purpose: the whole point is that the memo outlives any one +/// circuit. It depends only on (itself a singleton) and +/// — the audit KPI resolves its scoped query service from +/// a fresh scope per flight, the same shape AuditLogQueryService uses internally so a +/// KPI round never shares a circuit-scoped DbContext. +/// +/// +public sealed class KpiSnapshotCache : IKpiSnapshotCache +{ + /// + /// Freshness window. Sized just under the Health dashboard's 10s tick so each tick still + /// yields ONE fresh query for the whole node (rather than one per circuit), while a + /// circuit whose tick lands mid-window is served the memoized value instead of duplicating it. + /// + internal static readonly TimeSpan Ttl = TimeSpan.FromSeconds(8); + + private readonly CommunicationService _communication; + private readonly IServiceScopeFactory _scopeFactory; + + private readonly SingleFlightMemo _notificationKpis; + private readonly SingleFlightMemo _perSiteNotificationKpis; + private readonly SingleFlightMemo _perNodeNotificationKpis; + private readonly SingleFlightMemo _siteCallKpis; + private readonly SingleFlightMemo _perNodeSiteCallKpis; + private readonly SingleFlightMemo _auditKpis; + + /// + /// Initializes the shared KPI snapshot cache. + /// + /// Central-side comms used to Ask the outbox / site-call singletons. + /// Opens a fresh DI scope per audit-KPI flight. + public KpiSnapshotCache(CommunicationService communication, IServiceScopeFactory scopeFactory) + : this(communication, scopeFactory, Ttl, clock: null) + { + } + + /// + /// Test seam: same cache with an explicit TTL and clock so freshness can be asserted + /// without real delays. + /// + /// Central-side comms used to Ask the outbox / site-call singletons. + /// Opens a fresh DI scope per audit-KPI flight. + /// Freshness window. + /// Clock used for freshness. + internal KpiSnapshotCache( + CommunicationService communication, + IServiceScopeFactory scopeFactory, + TimeSpan ttl, + Func? clock) + { + _communication = communication ?? throw new ArgumentNullException(nameof(communication)); + _scopeFactory = scopeFactory ?? throw new ArgumentNullException(nameof(scopeFactory)); + + _notificationKpis = new SingleFlightMemo(ttl, clock); + _perSiteNotificationKpis = new SingleFlightMemo(ttl, clock); + _perNodeNotificationKpis = new SingleFlightMemo(ttl, clock); + _siteCallKpis = new SingleFlightMemo(ttl, clock); + _perNodeSiteCallKpis = new SingleFlightMemo(ttl, clock); + _auditKpis = new SingleFlightMemo(ttl, clock); + } + + /// + public Task GetNotificationKpisAsync( + bool forceRefresh = false, CancellationToken cancellationToken = default) => + _notificationKpis.GetAsync( + () => _communication.GetNotificationKpisAsync(new NotificationKpiRequest(NewCorrelationId())), + forceRefresh, cancellationToken); + + /// + public Task GetPerSiteNotificationKpisAsync( + bool forceRefresh = false, CancellationToken cancellationToken = default) => + _perSiteNotificationKpis.GetAsync( + () => _communication.GetPerSiteNotificationKpisAsync(new PerSiteNotificationKpiRequest(NewCorrelationId())), + forceRefresh, cancellationToken); + + /// + public Task GetPerNodeNotificationKpisAsync( + bool forceRefresh = false, CancellationToken cancellationToken = default) => + _perNodeNotificationKpis.GetAsync( + () => _communication.GetPerNodeNotificationKpisAsync(new PerNodeNotificationKpiRequest(NewCorrelationId())), + forceRefresh, cancellationToken); + + /// + public Task GetSiteCallKpisAsync( + bool forceRefresh = false, CancellationToken cancellationToken = default) => + _siteCallKpis.GetAsync( + () => _communication.GetSiteCallKpisAsync(new SiteCallKpiRequest(NewCorrelationId())), + forceRefresh, cancellationToken); + + /// + public Task GetPerNodeSiteCallKpisAsync( + bool forceRefresh = false, CancellationToken cancellationToken = default) => + _perNodeSiteCallKpis.GetAsync( + () => _communication.GetPerNodeSiteCallKpisAsync(new PerNodeSiteCallKpiRequest(NewCorrelationId())), + forceRefresh, cancellationToken); + + /// + public Task GetAuditKpiSnapshotAsync( + bool forceRefresh = false, CancellationToken cancellationToken = default) => + _auditKpis.GetAsync(LoadAuditKpisAsync, forceRefresh, cancellationToken); + + private async Task LoadAuditKpisAsync() + { + await using var scope = _scopeFactory.CreateAsyncScope(); + var queryService = scope.ServiceProvider.GetRequiredService(); + return await queryService.GetKpiSnapshotAsync(); + } + + private static string NewCorrelationId() => Guid.NewGuid().ToString("N"); +} diff --git a/src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/SharedAlarmSummaryService.cs b/src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/SharedAlarmSummaryService.cs new file mode 100644 index 00000000..3457ded2 --- /dev/null +++ b/src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/SharedAlarmSummaryService.cs @@ -0,0 +1,132 @@ +using System.Collections.Concurrent; +using Microsoft.Extensions.DependencyInjection; +using ZB.MOM.WW.ScadaBridge.Commons.Messages.Streaming; +using ZB.MOM.WW.ScadaBridge.Communication; + +namespace ZB.MOM.WW.ScadaBridge.CentralUI.Services; + +/// +/// Process-level memoizing façade over (arch-review WP2.4). +/// +/// The Alarm Summary page polls every 15s per circuit, and each poll fans one debug +/// snapshot Ask out to every Enabled instance on the site. Ten operators watching one site +/// meant ten independent fan-outs per 15s across the same instances — on top of the shared +/// live-alarm aggregator already seeding and reconciling from the identical fan-out. The +/// question is not user-specific, so this service collapses it: one memo slot per site, +/// single-flight, so the node costs ONE fan-out per site per window no matter how many +/// viewers are watching. +/// +/// +/// While the live cache is serving the site there is NO fan-out at all. The page already +/// ignores the poll's alarm rows in that state (the live deltas own them — arch-review R2 N5), +/// and the aggregator's own seed/reconcile fan-out now publishes the not-reporting set through +/// — the one thing the poll used to +/// still supply. So the live path is answered entirely from the cache: rows flattened from +/// through the same +/// AlarmSummaryService.BuildFromLiveAlarmsCore the page's live subscription uses, plus the +/// cached not-reporting names. One fan-out per site per reconcile window, run by the aggregator, +/// instead of two. +/// +/// +/// The rows are returned rather than left empty on purpose: liveness can flip to false in the +/// window between this call and the page's own IsLive re-check, and the page would then +/// adopt whatever came back. Handing it the live snapshot makes that race land on last-known +/// state instead of a blank grid. +/// +/// +/// While the cache is cold the poll is the page's full-rebuild safety net, so it still fans out, +/// memoized with a window just under the page's 15s tick so every tick gets fresh data. +/// +/// +public sealed class SharedAlarmSummaryService : IAlarmSummaryService +{ + /// + /// Freshness window while the live alarm cache is NOT serving this site: the poll is the + /// page's authoritative rebuild path, so it must stay under the 15s page tick. + /// + internal static readonly TimeSpan ColdCacheTtl = TimeSpan.FromSeconds(12); + + private readonly IServiceScopeFactory _scopeFactory; + private readonly ISiteAlarmLiveCache _liveCache; + private readonly Func? _clock; + + private readonly ConcurrentDictionary> _bySite = new(); + + /// + /// Initializes the shared alarm summary façade. + /// + /// + /// There is no live-cache freshness window any more: the live path reads the aggregator's + /// cache directly and never fans out, so the only window left is . + /// The reconcile interval this ctor used to read from CommunicationOptions is now + /// implicit — it IS the cadence at which the aggregator refreshes what the live path returns. + /// + /// Opens a fresh DI scope per fan-out (fresh repositories, off any circuit scope). + /// The shared live alarm cache: liveness, alarm rows, and not-reporting names. + public SharedAlarmSummaryService( + IServiceScopeFactory scopeFactory, + ISiteAlarmLiveCache liveCache) + : this(scopeFactory, liveCache, clock: null) + { + } + + /// + /// Test seam: same façade with an explicit clock. + /// + /// Opens a fresh DI scope per fan-out. + /// The shared live alarm cache. + /// Clock used for freshness. + internal SharedAlarmSummaryService( + IServiceScopeFactory scopeFactory, + ISiteAlarmLiveCache liveCache, + Func? clock) + { + _scopeFactory = scopeFactory ?? throw new ArgumentNullException(nameof(scopeFactory)); + _liveCache = liveCache ?? throw new ArgumentNullException(nameof(liveCache)); + _clock = clock; + } + + /// + public Task GetSiteAlarmsAsync( + int siteId, CancellationToken cancellationToken = default) + { + // Live: answer from the shared aggregator's cache and skip the fan-out entirely. Not + // memoized — there is nothing to amortize, and reading the cache directly keeps the + // answer as fresh as the aggregator itself. + if (_liveCache.IsLive(siteId)) + { + var live = AlarmSummaryService.BuildFromLiveAlarmsCore(_liveCache.GetCurrentAlarms(siteId)); + return Task.FromResult(new AlarmSummaryResult( + live.Alarms, _liveCache.GetNotReportingInstances(siteId))); + } + + var memo = _bySite.GetOrAdd( + siteId, + _ => new SingleFlightMemo(ColdCacheTtl, _clock)); + + return memo.GetAsync( + () => FanOutAsync(siteId), + forceRefresh: false, + cancellationToken, + ttlOverride: ColdCacheTtl); + } + + /// + public AlarmSummaryResult BuildFromLiveAlarms(IReadOnlyList alarms) => + AlarmSummaryService.BuildFromLiveAlarmsCore(alarms); + + /// + public AlarmRollup ComputeRollup(IReadOnlyList rows) => + AlarmSummaryService.ComputeRollupCore(rows); + + /// + /// One shared fan-out. Deliberately runs with no caller cancellation token: the flight is + /// shared, so one circuit navigating away must not cancel the round the others are awaiting. + /// + private async Task FanOutAsync(int siteId) + { + await using var scope = _scopeFactory.CreateAsyncScope(); + var inner = scope.ServiceProvider.GetRequiredService(); + return await inner.GetSiteAlarmsAsync(siteId, CancellationToken.None); + } +} diff --git a/src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/SingleFlightMemo.cs b/src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/SingleFlightMemo.cs new file mode 100644 index 00000000..a95731e8 --- /dev/null +++ b/src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/SingleFlightMemo.cs @@ -0,0 +1,136 @@ +namespace ZB.MOM.WW.ScadaBridge.CentralUI.Services; + +/// +/// A process-level memo slot: one value, a time-to-live, and single-flight production. +/// +/// Blazor Server runs one circuit per connected browser, so a page that polls on a timer +/// runs its query per circuit. Ten operators on the Health dashboard meant ten +/// identical KPI round trips every ten seconds. This slot collapses them: the first caller +/// in a TTL window starts the flight, every concurrent caller awaits the same +/// , and callers arriving after it completes are served the +/// memoized value until the TTL expires. +/// +/// +/// Failures are never cached — a faulted flight is discarded so the next caller retries. +/// The caller's own exception handling is therefore unchanged: it still sees the fault +/// (all callers of that one flight see it), and the next window starts clean. +/// +/// +/// The memoized value type. +internal sealed class SingleFlightMemo +{ + private readonly object _lock = new(); + private readonly TimeSpan _ttl; + private readonly Func _clock; + + /// The current flight, completed or not. Null until the first call. + private Task? _current; + + /// Wall-clock instant after which is stale. Only meaningful when it completed successfully. + private DateTimeOffset _freshUntil; + + /// + /// Initializes a memo slot. + /// + /// How long a successfully produced value stays fresh. + /// Clock used for freshness, injectable so tests need no real delay. + public SingleFlightMemo(TimeSpan ttl, Func? clock = null) + { + _ttl = ttl; + _clock = clock ?? (() => DateTimeOffset.UtcNow); + } + + /// + /// Number of times the factory has actually been invoked. Diagnostics and tests only. + /// + public int FlightCount { get; private set; } + + /// + /// Returns the memoized value, starting a flight if none is fresh. + /// + /// Produces a fresh value. Invoked outside the slot's lock. + /// + /// When , a completed-but-still-fresh value is discarded and a new + /// flight starts. An in-flight round is still joined rather than duplicated, so a + /// burst of operator "Refresh" clicks remains one round trip. + /// + /// + /// Cancels this caller's wait only — never the shared flight, which other callers may still + /// be awaiting. + /// + /// + /// Freshness window for this flight, overriding the slot's default. Used where the + /// acceptable staleness depends on state the caller knows (the Alarm Summary poll needs a + /// far shorter window while the live alarm cache is cold than while it is serving deltas). + /// + /// The memoized (or freshly produced) value. + public Task GetAsync( + Func> factory, + bool forceRefresh = false, + CancellationToken cancellationToken = default, + TimeSpan? ttlOverride = null) + { + ArgumentNullException.ThrowIfNull(factory); + + TaskCompletionSource? started = null; + Task result; + + lock (_lock) + { + var current = _current; + if (current is { IsCompleted: false }) + { + // A round is already in progress — join it, force or not. + result = current; + } + else if (!forceRefresh + && current is { IsCompletedSuccessfully: true } + && _clock() < _freshUntil) + { + result = current; + } + else + { + // RunContinuationsAsynchronously: a caller's continuation must never run + // inline on the thread that completes the flight (that thread is another + // circuit's dispatcher). + started = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + _current = started.Task; + FlightCount++; + result = started.Task; + } + } + + // The factory runs OUTSIDE the lock: it does I/O (an Ask, a SQL round trip) and must + // never be able to re-enter this slot on the calling thread while the lock is held. + if (started is not null) + { + _ = RunAsync(factory, started, ttlOverride ?? _ttl); + } + + return cancellationToken.CanBeCanceled ? result.WaitAsync(cancellationToken) : result; + } + + private async Task RunAsync(Func> factory, TaskCompletionSource completion, TimeSpan ttl) + { + try + { + var value = await factory().ConfigureAwait(false); + lock (_lock) + { + _freshUntil = _clock() + ttl; + } + completion.SetResult(value); + } + catch (Exception ex) + { + // A failure is not memoized: leave _freshUntil in the past so the next caller + // starts a new flight instead of re-serving the fault for the whole TTL. + lock (_lock) + { + _freshUntil = default; + } + completion.SetException(ex); + } + } +} diff --git a/src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Protocol/IBatchSubscribableConnection.cs b/src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Protocol/IBatchSubscribableConnection.cs new file mode 100644 index 00000000..9b848ece --- /dev/null +++ b/src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Protocol/IBatchSubscribableConnection.cs @@ -0,0 +1,60 @@ +namespace ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Protocol; + +/// +/// Per-tag outcome of a batch subscribe. A failed row is a genuine per-tag fault +/// (bad node id, unresolvable path) and never aborts the batch — mirroring the +/// MxAccess Gateway's SubscribeResult and OPC UA's per-monitored-item create +/// status. A batch that fails at connection level throws instead, so the +/// DataConnectionActor can drive the reconnect state machine. +/// +/// The requested tag path. +/// Whether the tag was subscribed. +/// Adapter subscription handle when ; otherwise null. +/// Per-tag failure reason when not successful. +public record TagSubscribeResult(string TagPath, bool Success, string? SubscriptionId, string? ErrorMessage); + +/// +/// Optional capability for an implementation whose +/// protocol can subscribe/unsubscribe MANY tags in one round trip, and whose +/// / +/// are TRUE bulk service calls rather than a loop over the single-tag methods. +/// Mirrors the / +/// capability-interface pattern; consumed by the DataConnectionActor only. +/// +/// +/// Implementing this interface is the adapter's assertion that batching is genuinely +/// cheaper than N single calls: the actor then subscribes, unsubscribes and seed-reads +/// in bounded chunks instead of per tag. An adapter that does NOT implement it keeps +/// the historical per-tag behaviour, so the capability is purely additive. +/// +/// +public interface IBatchSubscribableConnection +{ + /// + /// Subscribes every tag in in as few protocol round + /// trips as the adapter allows, returning one per + /// requested tag (in any order — callers key by ). + /// All tags share ONE ; the callback already carries the + /// tag path, so per-tag delegates would carry nothing extra. + /// + /// The tag paths to subscribe. + /// Callback invoked for every value change on any of the tags. + /// Cancellation token. + /// One result row per requested tag path. + /// + /// A connection-level fault (adapter not connected / transport down) is thrown so the + /// caller classifies it as a connection failure; per-tag faults are result rows. + /// + Task> SubscribeBatchAsync( + IReadOnlyList tagPaths, SubscriptionCallback callback, CancellationToken cancellationToken = default); + + /// + /// Releases every supplied subscription id in as few protocol round trips as the + /// adapter allows. Unknown/stale ids are ignored, mirroring + /// . + /// + /// Subscription ids previously returned by a subscribe call. + /// Cancellation token. + /// A task that represents the asynchronous operation. + Task UnsubscribeBatchAsync(IReadOnlyList subscriptionIds, CancellationToken cancellationToken = default); +} diff --git a/src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/IAuditLogRepository.cs b/src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/IAuditLogRepository.cs index 6aa0e707..72e31a66 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/IAuditLogRepository.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/IAuditLogRepository.cs @@ -34,6 +34,62 @@ public interface IAuditLogRepository /// A task that represents the asynchronous operation. Task InsertIfNotExistsAsync(AuditEvent evt, CancellationToken ct = default); + /// + /// Set-based form of : inserts every + /// event in that does not already exist, in as few + /// round trips as the provider allows, and returns the number of rows + /// actually written. First-write-wins idempotency is unchanged — an EventId + /// already present is silently skipped. + /// + /// + /// + /// Duplicate EventIds are tolerated both within and across packets. + /// Duplicates inside one call collapse to the first occurrence before the + /// statement is built (an append-only row is immutable, so later copies of an + /// EventId carry the same content); duplicates against already-committed rows + /// are eliminated by the anti-semi-join. A concurrent writer that commits a + /// row mid-statement is handled by falling back to the per-row path, so the + /// batch is a throughput optimisation and never a correctness dependency. + /// + /// + /// Default implementation. The interface supplies a per-row loop so a + /// test double or an alternative store keeps working unchanged; the EF Core + /// implementation overrides it with a genuine set-based statement. Callers on + /// the ingest hot path should always prefer this method — the per-row form + /// cost one IF NOT EXISTS … INSERT round trip per audit event. + /// + /// + /// Audit events to insert; may contain duplicates. + /// + /// Optional per-statement timeout. The ingest actor passes a budget strictly + /// SHORTER than the gRPC Ask that wraps it, so the reply is produced before + /// the caller gives up rather than at the same instant. + /// + /// Cancellation token. + /// A task that resolves to the number of rows inserted (duplicates excluded). + async Task InsertManyIfNotExistsAsync( + IReadOnlyList events, + TimeSpan? commandTimeout = null, + CancellationToken ct = default) + { + ArgumentNullException.ThrowIfNull(events); + + var inserted = 0; + var seen = new HashSet(events.Count); + foreach (var evt in events) + { + if (evt is null || !seen.Add(evt.EventId)) + { + continue; + } + + await InsertIfNotExistsAsync(evt, ct).ConfigureAwait(false); + inserted++; + } + + return inserted; + } + /// /// Returns up to rows matching /// , ordered by (OccurredAtUtc DESC, EventId DESC). @@ -60,36 +116,32 @@ public interface IAuditLogRepository /// /// /// - /// Drop-and-rebuild dance. UX_AuditLog_EventId is intentionally - /// non-partition-aligned (it lives on [PRIMARY] so single-column - /// EventId uniqueness — required by — - /// can be enforced cheaply). SQL Server rejects - /// ALTER TABLE … SWITCH PARTITION while a non-aligned unique index - /// is present, so the implementation drops the index, creates a staging - /// table with byte-identical schema, switches the partition's data into - /// staging, drops staging (discarding the rows), and rebuilds the unique - /// index. The CATCH branch guarantees the index is rebuilt even on partial - /// failure so the table never returns to live traffic without its - /// idempotency-supporting index. + /// Partition-aligned uniqueness — no index drop. EventId uniqueness is + /// enforced by the clustered PK_AuditLog (EventId, OccurredAtUtc), + /// which is aligned on ps_AuditLog_Month(OccurredAtUtc), so + /// ALTER TABLE … SWITCH PARTITION has no non-aligned unique index to + /// object to. The implementation creates a staging table with byte-identical + /// schema, switches the partition's data into staging, and drops staging + /// (discarding the rows). Nothing is dropped from the live table, so there is + /// no window in which the idempotency enforcement is absent and no offline + /// index rebuild inside the switch transaction. /// /// - /// Outage window. The dance briefly removes the unique index, so - /// concurrent calls during the switch - /// could in principle race past the IF NOT EXISTS check without the index - /// catching the duplicate. This is acceptable for the daily purge cadence - /// — the inserts that the IF NOT EXISTS check guards are themselves rare - /// enough that a sub-second collision window is operationally negligible, - /// and the composite PK still rejects same-(EventId, OccurredAtUtc) rows. + /// The predecessor design carried a non-aligned + /// UX_AuditLog_EventId on [PRIMARY] that had to be dropped and + /// rebuilt around every switch. A defensive guarded DROP INDEX remains + /// in the batch so a database restored from a pre-alignment backup still + /// purges; it is a one-way cleanup, never rebuilt. /// /// /// Lower-bound datetime of the monthly partition to switch out. /// /// Optional per-command timeout for the maintenance SQL (the row-count sample plus the - /// drop-and-rebuild dance). When null the provider default applies. The purge actor passes + /// staging/switch batch). When null the provider default applies. The purge actor passes /// - /// (default 30 min) because the ~30s ADO.NET default can abort the switch mid-dance on a large - /// or contended partition, leaving the table without UX_AuditLog_EventId until the next - /// tick recovers (arch-review 04, S2). + /// (default 30 min) because the ~30s ADO.NET default can abort the switch mid-batch on a large + /// or contended partition, leaving an orphaned staging table for the next tick's CATCH to clean + /// up (arch-review 04, S2). /// /// Cancellation token. /// A task that resolves to the approximate number of rows discarded by the partition switch. diff --git a/src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/IDeploymentManagerRepository.cs b/src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/IDeploymentManagerRepository.cs index 01b55d60..b7c59ced 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/IDeploymentManagerRepository.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/IDeploymentManagerRepository.cs @@ -22,6 +22,48 @@ public interface IDeploymentManagerRepository /// A read-only list of all deployment records. Task> GetAllDeploymentRecordsAsync(CancellationToken cancellationToken = default); /// + /// Database-side paged and filtered deployment query returning row summaries. + /// Backs the QueryDeployments management command, which previously + /// loaded the ENTIRE deployment-record table (insert-only, one row per deploy + /// attempt for the life of the system) and ignored the page arguments the CLI + /// was already sending. + /// + /// Restrict to one instance's history, or for all. + /// Restrict to one deployment status, or for all. + /// + /// When non-null, restrict to these instance ids. Used to apply a site-scoped + /// user's permitted-site filter IN THE DATABASE rather than by loading every + /// record and every instance and filtering in memory. + /// + /// Rows to skip; negative values are floored at 0. + /// Page size; non-positive or null returns every remaining row. + /// A cancellation token that can be used to cancel the operation. + /// One page of deployment summaries, newest first. + Task> QueryDeploymentSummariesAsync( + int? instanceId, + DeploymentStatus? status, + IReadOnlyCollection? instanceIdScope, + int skip, + int? take, + CancellationToken cancellationToken = default); + /// + /// Deletes TERMINAL deployment records ( / + /// ) that completed before + /// , in one bounded batch. + /// + /// + /// Non-terminal rows are never touched: an InProgress record is what the + /// query-before-redeploy reconciliation path reads to decide whether a prior + /// deploy actually landed at the site, so purging one by age would silently + /// disable that idempotency guard. + /// + /// + /// Records completing before this instant are eligible. + /// Maximum rows deleted in this call. + /// A cancellation token that can be used to cancel the operation. + /// The number of rows deleted; fewer than means the sweep is complete. + Task PurgeTerminalDeploymentRecordsAsync(DateTimeOffset cutoffUtc, int batchSize, CancellationToken cancellationToken = default); + /// /// Gets all deployment records for a specific instance. /// /// The instance ID. diff --git a/src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/INotificationOutboxRepository.cs b/src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/INotificationOutboxRepository.cs index fa8299de..b0d40b22 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/INotificationOutboxRepository.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/INotificationOutboxRepository.cs @@ -46,13 +46,42 @@ public interface INotificationOutboxRepository Task GetByIdAsync(string notificationId, CancellationToken cancellationToken = default); /// - /// Marks modified and persists it (status transitions). - /// Commits internally — this call is its own transaction. + /// Persists 's delivery-state columns — + /// Status, RetryCount, LastError, ResolvedTargets, + /// LastAttemptAt, NextAttemptAt, DeliveredAt. Commits + /// internally — this call is its own transaction. /// - /// The notification to update. + /// + /// Scope is deliberately narrow. Those seven columns are the ONLY + /// mutable state a notification has: everything else (identity, type, list, + /// subject, body, type data, source/origin attribution, enqueue and creation + /// timestamps) is written once at ingest and is immutable by contract. Every + /// caller — the dispatcher's per-attempt write and the operator retry/discard + /// one-shots — touches only this set. Implementations are therefore free to + /// issue a targeted UPDATE of these columns rather than rewriting the whole + /// row, which matters because the row carries nvarchar(max) body and + /// payload columns and the dispatcher writes on EVERY attempt. A future + /// caller that needs to change an immutable column must add its own method + /// rather than widening this one. + /// + /// The return value is the not-found signal. A targeted UPDATE against + /// a row that no longer exists (the retention purge removed it between the + /// read and the write) affects zero rows and is otherwise indistinguishable + /// from success — which is how an operator Retry/Discard came to report + /// success against a vanished notification. Implementations MUST return + /// when no row matched. Callers that speak to a human + /// (the operator one-shots) must surface that as "not found"; the dispatcher + /// treats it as a lost race and logs. + /// + /// + /// The notification whose delivery state should be persisted. /// Cancellation token. - /// A task that completes when the notification has been persisted. - Task UpdateAsync(Notification n, CancellationToken cancellationToken = default); + /// + /// when the row was found and its delivery state + /// persisted; when no row with that + /// NotificationId exists any more. + /// + Task UpdateAsync(Notification n, CancellationToken cancellationToken = default); /// /// Returns a page of notifications matching , ordered by diff --git a/src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/ITemplateEngineRepository.cs b/src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/ITemplateEngineRepository.cs index 3cc1d1d2..dc2d294f 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/ITemplateEngineRepository.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/ITemplateEngineRepository.cs @@ -35,6 +35,41 @@ public interface ITemplateEngineRepository /// A task that resolves to a read-only list of all templates. Task> GetAllTemplatesAsync(CancellationToken cancellationToken = default); /// + /// Read-only, no-tracking variant of for the + /// design-time ANALYSIS walks — acyclicity (CycleDetector), naming + /// collisions (CollisionDetector) and canonical-name resolution + /// (TemplateResolver). Script bodies are NOT loaded + /// ( comes back empty) because no analysis + /// consumer reads them; everything those walks do read — ids, parent and + /// composition edges, member names and lock flags — is present. + /// + /// + /// Do NOT use this for the inheritance reconciler or the flattener: the + /// reconciler compares and copies script bodies and writes through tracked + /// entities, and the flattener must observe rows an in-flight bundle import + /// has staged on the shared change tracker. Both keep using + /// / . + /// + /// + /// Cancellation token. + /// A task that resolves to a read-only list of all templates without script bodies. + Task> GetAllTemplatesForAnalysisAsync(CancellationToken cancellationToken = default); + /// + /// Database-side paged listing of templates as row summaries (child + /// collections reduced to counts, no script bodies). Backs the + /// ListTemplates management command, which previously materialised + /// every template's full child graph and then paged the list in memory. + /// + /// Number of rows to skip; negative values are floored at 0. + /// + /// Page size. or non-positive returns every remaining + /// row (the historical unpaged behaviour); otherwise clamped to + /// . + /// + /// Cancellation token. + /// A task that resolves to one page of template summaries, ordered by id. + Task> GetTemplateSummariesAsync(int skip, int? take, CancellationToken cancellationToken = default); + /// /// Returns every template that contains a composition referencing /// . Each result is eager-loaded with /// its Attributes / Scripts / Compositions so the caller can build a diff --git a/src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Services/ISiteAuditQueue.cs b/src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Services/ISiteAuditQueue.cs index 17058c91..be3fc2ff 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Services/ISiteAuditQueue.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Services/ISiteAuditQueue.cs @@ -109,10 +109,70 @@ public interface ISiteAuditQueue /// /// Lower bound timestamp (UTC). /// Maximum number of rows to return. + /// + /// Composite-keyset tiebreak cursor (WP2.3), mirroring + /// : when non-null it is the + /// ("D" GUID form) of the last row already consumed at + /// , and only rows strictly after the composite + /// (OccurredAtUtc, EventId) pair are returned — so a burst sharing one exact + /// instant drains via the id tiebreak instead of pinning the inclusive-timestamp cursor. + /// Null (the first pull, or a central that never sets it) keeps the inclusive + /// >= contract. + /// /// Cancellation token. /// A task that resolves to audit events at or after in pending or forwarded state, up to . Task> ReadPendingSinceAsync( - DateTime sinceUtc, int batchSize, CancellationToken ct = default); + DateTime sinceUtc, int batchSize, string? afterId = null, CancellationToken ct = default); + + /// + /// Reconciliation-pull commit surface, cursor form (WP2.3): flips every row at or before + /// the composite cursor (, ) + /// from / + /// to + /// . + /// + /// + /// The cursor is central's proof of receipt: it only advances past rows central has + /// actually ingested. Flipping on THAT — rather than on having merely served the rows — + /// is what makes the pull at-least-once: a fault between the response leaving the site + /// and central committing it leaves the rows unflipped, so the next pull re-serves them + /// (central dedups on ). + /// + /// With null the cursor is a bare timestamp under the legacy + /// inclusive >= read contract, so only rows STRICTLY older than + /// are proven received; rows at the boundary instant are left + /// alone. Idempotent; already-Reconciled rows are untouched. + /// + /// + /// Only-served-rows-retire. The cursor is a timestamp, and + /// is stamped by the caller, so a row can be + /// INSERTED after a batch was served yet carry a timestamp below the (by then advanced) + /// cursor. Implementations MUST NOT retire such a row: it was never served, the keyset + /// read has moved past it, and retiring it would make it purgeable — silent audit loss. + /// The SQLite implementation bounds the flip by insertion order (rowid high-water + /// mark of rows it has served), exempting rows already + /// + /// because central ACKED those through the telemetry push path. + /// + /// + /// Liveness dependency (documented, accepted). A row that WAS served but is never + /// covered by a later cursor — central reconciliation stopped for good, or the site node + /// restarted and its bound reset — stays + /// + /// indefinitely, and never purges Pending. That is the + /// retention invariant working as designed (an unacknowledged row is not droppable), but + /// it means the site store's floor is bounded by reconciliation actually running. The + /// backlog is observable: reports the pending count and + /// oldest-pending instant on every site health report, and the site reporter logs a + /// warning once the oldest pending row exceeds its stale threshold. + /// + /// + /// The cursor timestamp central has consumed up to (UTC). + /// The last consumed at that instant, or null. + /// Cancellation token. + /// A task that resolves to the number of rows flipped. + Task MarkReconciledUpToAsync( + DateTime sinceUtc, string? afterId, CancellationToken ct = default); /// /// Reconciliation-pull commit surface: flips the supplied EventIds to diff --git a/src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Services/ITemplateGraphWatermark.cs b/src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Services/ITemplateGraphWatermark.cs new file mode 100644 index 00000000..2d2952d5 --- /dev/null +++ b/src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Services/ITemplateGraphWatermark.cs @@ -0,0 +1,82 @@ +namespace ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services; + +/// +/// Process-wide monotonic version watermark over the template/instance +/// configuration graph. +/// +/// +/// Every mutation that lands through the configuration database bumps the +/// version of the affected template (or instance), plus a global counter and — +/// for edge-shaped changes (template added/removed, parent or composition edge +/// changed) — a separate . Consumers use the +/// watermark two ways: +/// +/// +/// +/// as the discriminator in a flatten-session cache key +/// (templateId + version), so a memoised template chain can never be +/// served after that template changed; and +/// +/// +/// as a staleness fast path — when neither the instance nor any template in +/// its chain has moved since a previously computed revision hash, the hash +/// is still current and a full re-flatten can be skipped. +/// +/// +/// +/// +/// Versions are in-memory and process-local. They are deliberately NOT +/// persisted: a restart (or the standby central node) simply starts from zero, +/// which invalidates every cached entry and falls back to the authoritative +/// full flatten. The watermark is therefore only ever able to cause EXTRA work, +/// never stale work — a correctness property that matters because the value is +/// not replicated between the two central nodes. +/// +/// +public interface ITemplateGraphWatermark +{ + /// + /// Monotonic counter bumped on every template- or instance-graph mutation. + /// Useful as a coarse "did anything change at all" gate. + /// + long Global { get; } + + /// + /// Monotonic counter bumped only when the SHAPE of the graph changes — a + /// template row added or removed, a parent-template edge changed, or a + /// composition row added/removed/repointed. A cached template-chain + /// MEMBERSHIP is valid only while this value is unchanged; the per-template + /// versions then cover the contents of each member. + /// + long StructureVersion { get; } + + /// Current version of a single template. Zero when never mutated in this process. + /// Template id to read. + /// The template's monotonic version. + long GetTemplateVersion(int templateId); + + /// Current version of a single instance (its override/binding rows). Zero when never mutated in this process. + /// Instance id to read. + /// The instance's monotonic version. + long GetInstanceVersion(int instanceId); + + /// Records a mutation of the given template. + /// Template whose version is bumped. + /// + /// When true the change altered the graph shape (add/remove/parent or + /// composition edge) and is bumped too. + /// + void BumpTemplate(int templateId, bool structural = false); + + /// Records a mutation of the given instance. + /// Instance whose version is bumped. + void BumpInstance(int instanceId); + + /// + /// Records a mutation whose affected template/instance could not be + /// attributed to a specific id. Bumps and + /// , invalidating every chain-membership and + /// staleness fast path — the safe, conservative fallback. + /// + void BumpAll(); +} diff --git a/src/ZB.MOM.WW.ScadaBridge.Commons/Messages/DebugView/DebugSnapshotRequest.cs b/src/ZB.MOM.WW.ScadaBridge.Commons/Messages/DebugView/DebugSnapshotRequest.cs index d6d5668a..306eb9fb 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Commons/Messages/DebugView/DebugSnapshotRequest.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Commons/Messages/DebugView/DebugSnapshotRequest.cs @@ -1,5 +1,19 @@ namespace ZB.MOM.WW.ScadaBridge.Commons.Messages.DebugView; +/// +/// Asks one instance for a point-in-time . +/// +/// Unique name of the instance to snapshot. +/// Correlation id echoed on the reply. +/// +/// When true the site builds ONLY the alarm half of the snapshot and returns an +/// empty list (wire efficiency, WP2.3). +/// Set by the central per-site live alarm cache, whose seed/reconcile fan-out discards +/// every attribute row anyway. Defaults to false so the Debug View and every other +/// caller keep the full snapshot; additive on the wire +/// (DebugSnapshotRequestDto.alarms_only, field 3). +/// public record DebugSnapshotRequest( string InstanceUniqueName, - string CorrelationId); + string CorrelationId, + bool AlarmsOnly = false); diff --git a/src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Health/HeartbeatMessage.cs b/src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Health/HeartbeatMessage.cs index 893e4718..7407c909 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Health/HeartbeatMessage.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Health/HeartbeatMessage.cs @@ -1,7 +1,27 @@ namespace ZB.MOM.WW.ScadaBridge.Commons.Messages.Health; +/// +/// Application-level heartbeat a site node sends to central on its heartbeat interval. +/// Central's health aggregator stamps the site's last-seen time from it, and the offline +/// timeout fires when they stop landing. +/// +/// The reporting site's identifier. +/// The reporting node's hostname. +/// Whether the reporting node is the site's active (oldest-Up) node. +/// When the heartbeat was produced (UTC). +/// +/// true marks a heartbeat that carries NO liveness meaning and that central must skip for +/// all health bookkeeping — it neither stamps a last-seen time nor replicates to the peer node. +/// The only producer today is CentralChannelProvider's failback probe, which reuses the +/// heartbeat RPC purely to ask "does the preferred central endpoint answer again?"; it is emitted +/// by a SITE's transport layer, not by any node's heartbeat timer, and counting it as liveness +/// would keep a site looking alive on the health dashboard from the probe alone. Additive with a +/// false default (and proto field 5 on HeartbeatDto), so every real heartbeat — +/// including one from a node that predates the flag — is unaffected. +/// public record HeartbeatMessage( string SiteId, string NodeHostname, bool IsActive, - DateTimeOffset Timestamp); + DateTimeOffset Timestamp, + bool Synthetic = false); diff --git a/src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Health/SiteHealthReport.cs b/src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Health/SiteHealthReport.cs index 8306ffdc..ec43d5a1 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Health/SiteHealthReport.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Health/SiteHealthReport.cs @@ -74,6 +74,25 @@ public record SiteHealthReport( /// public double? ScriptOldestBusyAgeSeconds { get; init; } + /// + /// WP3.1: script-execution worker threads the stuck-script watchdog has DETACHED and + /// replaced — a script wedged past its deadline plus grace in uninterruptible blocking + /// I/O, so the pool started a fresh thread and the wedged one will exit only when its + /// body finally returns. Point-in-time, refreshed by ScriptSchedulerStatsReporter. + /// Zero is the healthy state; a value that climbs and never drains means script bodies + /// are permanently consuming threads and the pool is being repeatedly rebuilt around them. + /// + public int DetachedScriptThreads { get; init; } + + /// + /// WP3.1: per-interval count of script and alarm on-trigger runs SHED because + /// MaxConcurrentRunsPerScript runs were already in flight for that script. Raw + /// per-interval count (drained on collect) like . A + /// sustained non-zero value means a trigger is firing faster than its script completes; + /// the shed itself is the designed back-pressure, not an error. + /// + public int ScriptRunShedCount { get; init; } + // LocalDb 2-node replication of the consolidated site database (Phase 1). // Additive init properties for the same reason as the scheduler gauges above: // the positional constructor stays untouched. Refreshed on the site by @@ -104,6 +123,17 @@ public record SiteHealthReport( /// pair as perfectly healthy. Collapsing null to 0 anywhere on this path is a bug. /// public long? LocalDbOplogBacklog { get; init; } + + /// + /// WP2.6d (arch-review misc — site stream alarm-vs-attribute shared buffer): + /// cumulative count of alarm state changes dropped from + /// SiteStreamManager's dedicated alarm publish queue because it was at + /// capacity — i.e. the site-wide alarm hand-off itself fell behind, not a + /// per-subscriber buffer. Point-in-time (not reset on collect), refreshed by + /// the site-side SiteStreamAlarmDropReporter hosted service. Defaults to + /// 0 so existing producers/tests that don't populate it stay valid. + /// + public long SiteStreamAlarmDropCount { get; init; } } /// diff --git a/src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Management/DeploymentCommands.cs b/src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Management/DeploymentCommands.cs index 3896a455..e193e292 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Management/DeploymentCommands.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Management/DeploymentCommands.cs @@ -1,5 +1,26 @@ namespace ZB.MOM.WW.ScadaBridge.Commons.Messages.Management; public record MgmtDeployArtifactsCommand(int? SiteId = null); + +/// +/// Deploys every deployable instance at one site in a single operation. +/// +/// +/// Distinct from , which ships the +/// SYSTEM-WIDE artifact sets (shared scripts, external systems, DB/data +/// connections) to a site. This command deploys the site's INSTANCE +/// configurations — the same per-instance pipeline +/// MgmtDeployInstanceCommand runs, batched. +/// +/// +/// +/// Not all-or-nothing: each instance keeps its own deployment id, revision hash, +/// operation lock and status record, and a failure on one is reported as a failed +/// row while the rest proceed. Individually retryable via the single-instance +/// deploy. Additive contract; new fields go on the end with a default. +/// +/// +/// Database id of the target site. Required — this command is never fleet-wide. +public record MgmtDeploySiteCommand(int SiteId); public record QueryDeploymentsCommand(int? InstanceId = null, string? Status = null, int Page = 1, int PageSize = 50); public record GetDeploymentDiffCommand(int InstanceId); diff --git a/src/ZB.MOM.WW.ScadaBridge.Commons/Observability/ScadaBridgeTelemetry.cs b/src/ZB.MOM.WW.ScadaBridge.Commons/Observability/ScadaBridgeTelemetry.cs index 0d1474c2..aa5a60ae 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Commons/Observability/ScadaBridgeTelemetry.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Commons/Observability/ScadaBridgeTelemetry.cs @@ -64,6 +64,35 @@ public static class ScadaBridgeTelemetry Meter.CreateCounter("scadabridge.site.alarm_cache.reconnects", unit: "1", description: "Live-alarm aggregator site-wide gRPC stream reconnects (NodeA↔NodeB flip or reconcile-driven reopen)."); + /// + /// Incremented for each live delta evicted from a per-site live-alarm aggregator's + /// bounded pre-seed buffer (drop-oldest, WP2.3). Non-zero means a seed/reconcile + /// fan-out ran long enough for the delta storm behind it to exceed the cap — the + /// dropped transitions are recovered by the fan-out's authoritative snapshot, but a + /// sustained climb points at a slow site. + /// + private static readonly Counter _liveAlarmBufferDrops = + Meter.CreateCounter("scadabridge.site.alarm_cache.buffer_dropped", unit: "1", + description: "Live deltas evicted from a live-alarm aggregator's bounded pre-seed buffer (drop-oldest)."); + + /// + /// Incremented for each debug event evicted from a central debug session's bounded + /// pre-snapshot buffer (drop-oldest, WP2.3). The Debug View is lossy-under-backpressure + /// by design; this makes the loss measurable instead of unbounded memory growth. + /// + private static readonly Counter _debugPreSnapshotDrops = + Meter.CreateCounter("scadabridge.central.debug_view.presnapshot_dropped", unit: "1", + description: "Debug events evicted from a central debug session's bounded pre-snapshot buffer (drop-oldest)."); + + /// + /// Incremented for each event evicted from a site-hosted gRPC stream's bounded send + /// channel, tagged by stream kind (instance = Debug View, site-alarms = + /// the site-wide alarm feed behind the operator Alarm Summary). + /// + private static readonly Counter _siteStreamEventDrops = + Meter.CreateCounter("scadabridge.site.stream.events_dropped", unit: "1", + description: "Events evicted from a site gRPC stream's bounded send channel, tagged by stream kind."); + // ---------------- Observable gauges ---------------- /// Current count of open site connections, mutated via . @@ -136,6 +165,19 @@ public static class ScadaBridgeTelemetry /// Records that a per-site live-alarm aggregator re-established its site-wide gRPC stream. public static void RecordLiveAlarmStreamReconnect() => _liveAlarmStreamReconnects.Add(1); + /// Records live deltas evicted from a live-alarm aggregator's bounded pre-seed buffer. + /// Number of deltas evicted. + public static void RecordLiveAlarmBufferDrop(long count = 1) => _liveAlarmBufferDrops.Add(count); + + /// Records debug events evicted from a debug session's bounded pre-snapshot buffer. + /// Number of events evicted. + public static void RecordDebugPreSnapshotDrop(long count = 1) => _debugPreSnapshotDrops.Add(count); + + /// Records an event evicted from a site gRPC stream's bounded send channel. + /// Stream kind tag (instance or site-alarms). + public static void RecordSiteStreamEventDropped(string streamKind) => + _siteStreamEventDrops.Add(1, new KeyValuePair("stream", streamKind)); + /// /// Registers the provider the StoreAndForward queue-depth gauge reads on each observation. /// A later task supplies a provider that reads the real StoreAndForward depth. A null diff --git a/src/ZB.MOM.WW.ScadaBridge.Commons/Serialization/OpcUaEndpointConfigSerializer.cs b/src/ZB.MOM.WW.ScadaBridge.Commons/Serialization/OpcUaEndpointConfigSerializer.cs index 55d1d107..fea3f727 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Commons/Serialization/OpcUaEndpointConfigSerializer.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Commons/Serialization/OpcUaEndpointConfigSerializer.cs @@ -195,6 +195,7 @@ public static class OpcUaEndpointConfigSerializer ["DiscardOldest"] = config.DiscardOldest.ToString(), ["SubscriptionPriority"] = config.SubscriptionPriority.ToString(), ["SubscriptionDisplayName"] = config.SubscriptionDisplayName, + ["MaxMonitoredItemsPerSubscription"] = config.MaxMonitoredItemsPerSubscription.ToString(), ["TimestampsToReturn"] = config.TimestampsToReturn.ToString(), }; if (config.Heartbeat is { } hb) @@ -248,6 +249,7 @@ public static class OpcUaEndpointConfigSerializer TryAssignInt(dict, "KeepAliveCount", v => c.KeepAliveCount = v); TryAssignInt(dict, "LifetimeCount", v => c.LifetimeCount = v); TryAssignInt(dict, "MaxNotificationsPerPublish", v => c.MaxNotificationsPerPublish = v); + TryAssignInt(dict, "MaxMonitoredItemsPerSubscription", v => c.MaxMonitoredItemsPerSubscription = v); if (dict.TryGetValue("DiscardOldest", out var doStr) && bool.TryParse(doStr, out var doVal)) c.DiscardOldest = doVal; diff --git a/src/ZB.MOM.WW.ScadaBridge.Commons/Types/DataConnections/OpcUaEndpointConfig.cs b/src/ZB.MOM.WW.ScadaBridge.Commons/Types/DataConnections/OpcUaEndpointConfig.cs index cad140f6..9627e76b 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Commons/Types/DataConnections/OpcUaEndpointConfig.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Commons/Types/DataConnections/OpcUaEndpointConfig.cs @@ -66,6 +66,14 @@ public sealed class OpcUaEndpointConfig /// Display name for the subscription. /// public string SubscriptionDisplayName { get; set; } = "ScadaBridge"; + /// + /// Maximum monitored items placed on a single OPC UA subscription before the + /// adapter shards onto an additional subscription. Item-count ceilings are a + /// property of the target server, so this is per endpoint rather than global. + /// A site sized at 37,500 tags shards into 8 subscriptions at the default. + /// Values <= 0 are treated as the default. + /// + public int MaxMonitoredItemsPerSubscription { get; set; } = 5000; // Read / filter /// diff --git a/src/ZB.MOM.WW.ScadaBridge.Commons/Types/Deployment/DeploymentRecordSummary.cs b/src/ZB.MOM.WW.ScadaBridge.Commons/Types/Deployment/DeploymentRecordSummary.cs new file mode 100644 index 00000000..804ba256 --- /dev/null +++ b/src/ZB.MOM.WW.ScadaBridge.Commons/Types/Deployment/DeploymentRecordSummary.cs @@ -0,0 +1,54 @@ +using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; + +namespace ZB.MOM.WW.ScadaBridge.Commons.Types.Deployment; + +/// +/// Row-shaped projection of a DeploymentRecord for list surfaces (CLI +/// deploy list / instance history, the Central UI deployment-status +/// page). +/// +/// +/// Exists so a deployment listing can be paged and filtered in the DATABASE. +/// Deployment records are insert-only — one row per deploy attempt, retained for +/// the configured retention window — so the previous "load the whole table, then +/// filter in memory" read path degraded with the age of the system rather than +/// with the size of the requested page. +/// +/// +/// +/// The RowVersion optimistic-concurrency token is deliberately absent: this +/// is a READ projection, and a summary must never be mistaken for something that +/// can be written back. Mutating paths keep loading the tracked entity. +/// +/// +/// +/// Message-contract evolution rule: additive-only. New fields go on the end with a +/// default. +/// +/// +/// Deployment record row id. +/// The logical deployment id (GUID, "N" format). +/// Instance the deployment targeted. +/// Terminal or in-flight deployment status. +/// Revision hash of the deployed configuration. +/// User who initiated the deployment. +/// When the deployment was initiated. +/// When the deployment reached a terminal status, if it has. +/// Failure detail when the deployment did not succeed. +public record DeploymentRecordSummary( + int Id, + string DeploymentId, + int InstanceId, + DeploymentStatus Status, + string? RevisionHash, + string DeployedBy, + DateTimeOffset DeployedAt, + DateTimeOffset? CompletedAt, + string? ErrorMessage) +{ + /// + /// Hard ceiling applied to a requested page size, mirroring the in-memory + /// Page clamp the management actor already applies elsewhere. + /// + public const int MaxPageSize = 1000; +} diff --git a/src/ZB.MOM.WW.ScadaBridge.Commons/Types/Templates/TemplateGraphWatermark.cs b/src/ZB.MOM.WW.ScadaBridge.Commons/Types/Templates/TemplateGraphWatermark.cs new file mode 100644 index 00000000..0aced5b1 --- /dev/null +++ b/src/ZB.MOM.WW.ScadaBridge.Commons/Types/Templates/TemplateGraphWatermark.cs @@ -0,0 +1,72 @@ +using System.Collections.Concurrent; +using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services; + +namespace ZB.MOM.WW.ScadaBridge.Commons.Types.Templates; + +/// +/// In-memory . Registered as a singleton so +/// the writer (the configuration-database repository's unit-of-work commit) and +/// the readers (the flatten-session cache, the staleness fast path) observe the +/// same counters within a process. +/// +/// +/// Lives in Commons rather than a component project because both the +/// ConfigurationDatabase writer and the DeploymentManager readers need it and +/// neither references the other. It is a pure in-memory primitive with no +/// infrastructure dependency, so it does not violate the "implementations live +/// with their component" convention the way a repository would. +/// +/// +/// +/// All counters are -updated and the per-id maps are +/// , so concurrent management +/// commands (which each run on their own DI scope) can bump safely. Reads are +/// deliberately unsynchronised relative to each other: a reader that observes a +/// half-applied batch of bumps sees a CHANGED value for at least one member, +/// which invalidates rather than falsely validates a cache entry. +/// +/// +public sealed class TemplateGraphWatermark : ITemplateGraphWatermark +{ + private readonly ConcurrentDictionary _templateVersions = new(); + private readonly ConcurrentDictionary _instanceVersions = new(); + private long _global; + private long _structure; + + /// + public long Global => Interlocked.Read(ref _global); + + /// + public long StructureVersion => Interlocked.Read(ref _structure); + + /// + public long GetTemplateVersion(int templateId) => + _templateVersions.TryGetValue(templateId, out var v) ? v : 0L; + + /// + public long GetInstanceVersion(int instanceId) => + _instanceVersions.TryGetValue(instanceId, out var v) ? v : 0L; + + /// + public void BumpTemplate(int templateId, bool structural = false) + { + _templateVersions.AddOrUpdate(templateId, 1L, static (_, current) => current + 1); + Interlocked.Increment(ref _global); + if (structural) + Interlocked.Increment(ref _structure); + } + + /// + public void BumpInstance(int instanceId) + { + _instanceVersions.AddOrUpdate(instanceId, 1L, static (_, current) => current + 1); + Interlocked.Increment(ref _global); + } + + /// + public void BumpAll() + { + Interlocked.Increment(ref _global); + Interlocked.Increment(ref _structure); + } +} diff --git a/src/ZB.MOM.WW.ScadaBridge.Commons/Types/Templates/TemplateSummary.cs b/src/ZB.MOM.WW.ScadaBridge.Commons/Types/Templates/TemplateSummary.cs new file mode 100644 index 00000000..c9294eca --- /dev/null +++ b/src/ZB.MOM.WW.ScadaBridge.Commons/Types/Templates/TemplateSummary.cs @@ -0,0 +1,53 @@ +namespace ZB.MOM.WW.ScadaBridge.Commons.Types.Templates; + +/// +/// Row-shaped projection of a template for list surfaces (CLI template list, +/// the Central UI template tree, the ListTemplates management command). +/// +/// +/// Exists so a template listing can be paged in the DATABASE rather than by +/// materialising every template's full child graph — attributes, alarms, script +/// bodies, compositions and native alarm sources — and then discarding all but +/// one page of it in memory. Child collections are reduced to counts, which is +/// what every list surface actually renders. +/// +/// +/// +/// Message-contract evolution rule: this record is additive-only. New fields go +/// on the end with a default so an older peer deserialising a newer payload is +/// unaffected. +/// +/// +/// Template id. +/// Template name. +/// Optional description. +/// Parent template id when this template inherits. +/// Containing folder id. +/// True when the template was auto-derived to back a composition slot. +/// Owning composition row for a derived template. +/// Number of directly declared attributes. +/// Number of directly declared alarms. +/// Number of directly declared scripts. +/// Number of composition slots. +/// Number of native alarm source bindings. +public record TemplateSummary( + int Id, + string Name, + string? Description, + int? ParentTemplateId, + int? FolderId, + bool IsDerived, + int? OwnerCompositionId, + int AttributeCount, + int AlarmCount, + int ScriptCount, + int CompositionCount, + int NativeAlarmSourceCount) +{ + /// + /// Hard ceiling applied to a requested page size, mirroring the existing + /// in-memory Page clamp in the management actor so DB-side paging + /// cannot be used to pull an unbounded result set. + /// + public const int MaxPageSize = 1000; +} diff --git a/src/ZB.MOM.WW.ScadaBridge.Communication/Actors/CentralCommunicationActor.cs b/src/ZB.MOM.WW.ScadaBridge.Communication/Actors/CentralCommunicationActor.cs index b2cbc71f..ca63ca68 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Communication/Actors/CentralCommunicationActor.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Communication/Actors/CentralCommunicationActor.cs @@ -67,36 +67,6 @@ public class CentralCommunicationActor : ReceiveActor /// private IActorRef? _notificationOutboxProxy; - /// - /// Proxy for the central AuditLogIngestActor cluster - /// singleton. Set via — the Host creates the - /// singleton proxy after this actor and registers it (mirrors - /// ). Null until registration completes; - /// an audit ingest command arriving before then is answered with an empty - /// reply so the site keeps its rows Pending and retries. - /// - /// Once registered, the handler Asks this proxy and pipes the reply straight - /// back to the caller. On an Ask timeout or a faulted reply, PipeTo forwards a - /// to the caller — the fault propagates rather - /// than being swallowed. This differs from the gRPC handler - /// (SiteStreamGrpcServer), which catches the exception and returns an - /// empty ack; here the faulted Ask is the transient signal the site relies on - /// (see ). - /// - private IActorRef? _auditIngestProxy; - - /// - /// Default Ask timeout for routing audit ingest commands to the - /// Effective Ask timeout for audit ingest routing. Defaults to - /// (30 s) — the two - /// audit-ingest entry points (the site stream server and the control plane) share one source of truth - /// for the timeout. Overridable via the constructor so tests can exercise the - /// timeout/fault path without waiting 30 s. When the window is exceeded the Ask - /// faults and that fault is piped back to the caller as a - /// (see ). - /// - private readonly TimeSpan _auditIngestAskTimeout; - /// /// DistributedPubSub topic used to fan health reports out to the peer /// central node so both per-node aggregators stay in sync. See @@ -111,12 +81,10 @@ public class CentralCommunicationActor : ReceiveActor /// /// DI service provider for scoped repository and aggregator access. /// The central→site command transport to route every through. - /// Optional override for the audit-ingest Ask timeout (test hook). public CentralCommunicationActor( IServiceProvider serviceProvider, - ISiteCommandTransport transport, - TimeSpan? auditIngestAskTimeout = null) - : this(serviceProvider, auditIngestAskTimeout) + ISiteCommandTransport transport) + : this(serviceProvider) { _transport = transport ?? throw new ArgumentNullException(nameof(transport)); } @@ -125,13 +93,9 @@ public class CentralCommunicationActor : ReceiveActor /// is assigned by the delegating public constructor before any message /// is dispatched. /// DI service provider. - /// Optional audit-ingest Ask timeout override. - private CentralCommunicationActor( - IServiceProvider serviceProvider, - TimeSpan? auditIngestAskTimeout) + private CentralCommunicationActor(IServiceProvider serviceProvider) { _serviceProvider = serviceProvider; - _auditIngestAskTimeout = auditIngestAskTimeout ?? Grpc.SiteStreamGrpcServer.AuditIngestAskTimeout; // Site address cache loaded from database Receive(HandleSiteAddressCacheLoaded); @@ -176,24 +140,11 @@ public class CentralCommunicationActor : ReceiveActor // so the NotificationStatusResponse routes back to the querying site. Receive(HandleNotificationStatusQuery); - // Audit Log: the Host registers the AuditLogIngestActor singleton - // proxy after this actor is created (the proxy cannot exist before this - // actor's construction). - Receive(msg => - { - _auditIngestProxy = msg.AuditIngestActor; - _log.Info("Registered audit ingest proxy"); - }); - - // Audit Log site→central ingest: a site forwards a batch of audit - // events to the central cluster. Ask the ingest proxy - // and pipe the IngestAuditEventsReply back to the original Sender (the - // site's transport path) so the site can flip its rows to Forwarded. - Receive(HandleIngestAuditEvents); - - // Audit Log combined-telemetry ingest: routes to the same proxy - // the same way; the proxy replies with an IngestCachedTelemetryReply. - Receive(HandleIngestCachedTelemetry); + // Audit Log site→central ingest is NOT relayed here. The central-hosted + // CentralControlGrpcService Asks the audit-log-ingest singleton proxy directly + // (as SiteStreamGrpcServer always has), so this actor no longer sits in that + // path — the relay was a second hop with an identical 30 s Ask timeout that + // could only add latency. Do not reintroduce it. // Startup reconciliation: a site node forwards its local deployed inventory on // startup. Resolve the scoped ReconcileService, diff the @@ -239,51 +190,6 @@ public class CentralCommunicationActor : ReceiveActor _notificationOutboxProxy.Forward(msg); } - private void HandleIngestAuditEvents(IngestAuditEventsCommand msg) - { - if (_auditIngestProxy == null) - { - // No ingest proxy registered yet (host startup race). Reply with an - // empty IngestAuditEventsReply so the site keeps its rows Pending and - // retries — the same behaviour as the gRPC handler's wiring-race path. - _log.Warning( - "Cannot route IngestAuditEventsCommand ({0} events) — audit ingest not available", - msg.Events.Count); - Sender.Tell(new IngestAuditEventsReply(Array.Empty())); - return; - } - - // Capture Sender before the async/PipeTo — Akka resets Sender between - // dispatches. The reply is piped straight back to the calling site node. - // On an Ask timeout or a faulted reply, PipeTo delivers a Status.Failure to - // replyTo: the fault propagates to the caller rather than being swallowed. - // The site's own Ask through this path then faults, and the site drain loop - // treats that as a transient failure — rows stay Pending and are retried on - // the next tick. (The gRPC handler instead returns an empty ack on fault; - // propagating the fault here is the cleaner transient signal.) - var replyTo = Sender; - _log.Debug("Routing IngestAuditEventsCommand ({0} events) to the audit ingest actor", msg.Events.Count); - _auditIngestProxy.Ask(msg, _auditIngestAskTimeout) - .PipeTo(replyTo); - } - - private void HandleIngestCachedTelemetry(IngestCachedTelemetryCommand msg) - { - if (_auditIngestProxy == null) - { - _log.Warning( - "Cannot route IngestCachedTelemetryCommand ({0} entries) — audit ingest not available", - msg.Entries.Count); - Sender.Tell(new IngestCachedTelemetryReply(Array.Empty())); - return; - } - - var replyTo = Sender; - _log.Debug("Routing IngestCachedTelemetryCommand ({0} entries) to the audit ingest actor", msg.Entries.Count); - _auditIngestProxy.Ask(msg, _auditIngestAskTimeout) - .PipeTo(replyTo); - } - /// /// Startup reconciliation (site→central): resolve the scoped /// in a DI scope, diff the node's reported inventory @@ -291,7 +197,7 @@ public class CentralCommunicationActor : ReceiveActor /// back to the site node's transport path. The actor stays thin — all the diff /// and staging logic lives in the service. Mirrors the DB-access pattern used by /// (Task.Run + CreateScope + PipeTo) and the - /// Sender-preservation pattern of . + /// Sender-preservation pattern of . /// /// On a faulted task PipeTo delivers a to the node; its /// Ask faults and it simply retries reconcile on the next startup — reconcile is @@ -328,6 +234,21 @@ public class CentralCommunicationActor : ReceiveActor private void HandleHeartbeat(HeartbeatMessage heartbeat) { + // Synthetic heartbeats carry no liveness meaning — today only + // CentralChannelProvider's failback probe, which reuses this RPC to ask whether the + // preferred central endpoint answers again. It is emitted by a site's TRANSPORT layer, + // not by any node's heartbeat timer, so stamping it would let a site whose real + // heartbeats had stopped keep looking alive on the health dashboard for as long as its + // transport kept probing. Dropped before the local mark AND before the peer fan-out, + // so neither central node's aggregator ever sees it. + if (heartbeat.Synthetic) + { + _log.Debug( + "Ignoring synthetic heartbeat from site {0} (node '{1}') — probe traffic, not liveness", + heartbeat.SiteId, heartbeat.NodeHostname); + return; + } + MarkHeartbeatLocally(heartbeat); // Fan the heartbeat out to the peer central node so BOTH aggregators mark @@ -350,9 +271,17 @@ public class CentralCommunicationActor : ReceiveActor /// /// Marks a site heartbeat on the local aggregator without re-broadcasting. /// Used for both site-originated heartbeats and peer-replicated ones. + /// A synthetic heartbeat is never marked — already drops it + /// before the fan-out, so a replica can only carry one if a peer node predates the flag; + /// the guard here is belt-and-braces on the last hop before the aggregator. /// private void MarkHeartbeatLocally(HeartbeatMessage heartbeat) { + if (heartbeat.Synthetic) + { + return; + } + var aggregator = _serviceProvider.GetService(); aggregator?.MarkHeartbeat(heartbeat.SiteId, heartbeat.Timestamp); } @@ -623,13 +552,7 @@ public record DebugStreamTerminated(string SiteId, string CorrelationId); /// public record RegisterNotificationOutbox(IActorRef OutboxProxy); -/// -/// Registers the central AuditLogIngestActor singleton proxy with the -/// so site-forwarded -/// and -/// messages can be routed to it. Sent by the Host after the audit-ingest -/// singleton proxy is created. Lives here (not in Commons) because -/// ZB.MOM.WW.ScadaBridge.Commons has no Akka package reference and cannot hold an -/// field. -/// -public sealed record RegisterAuditIngest(IActorRef AuditIngestActor); +// NOTE: there is deliberately no RegisterAuditIngest counterpart. The audit-log-ingest +// singleton proxy goes STRAIGHT to the two gRPC servers that need it +// (CentralControlGrpcService.SetAuditIngestActor / SiteStreamGrpcServer.SetAuditIngestActor); +// routing audit batches through this actor was a redundant hop and was removed. diff --git a/src/ZB.MOM.WW.ScadaBridge.Communication/Actors/DebugStreamBridgeActor.cs b/src/ZB.MOM.WW.ScadaBridge.Communication/Actors/DebugStreamBridgeActor.cs index 7496b053..8d105103 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Communication/Actors/DebugStreamBridgeActor.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Communication/Actors/DebugStreamBridgeActor.cs @@ -2,6 +2,7 @@ using Akka.Actor; using Akka.Event; using ZB.MOM.WW.ScadaBridge.Commons.Messages.DebugView; using ZB.MOM.WW.ScadaBridge.Commons.Messages.Streaming; +using ZB.MOM.WW.ScadaBridge.Commons.Observability; using ZB.MOM.WW.ScadaBridge.Communication.Grpc; namespace ZB.MOM.WW.ScadaBridge.Communication.Actors; @@ -45,9 +46,20 @@ public class DebugStreamBridgeActor : ReceiveActor, IWithTimers private const int MaxRetries = 3; private const string ReconnectTimerKey = "grpc-reconnect"; private const string StabilityTimerKey = "grpc-stability"; + private const string SnapshotTimerKey = "debug-snapshot-deadline"; + private const string ConsumerLivenessTimerKey = "debug-consumer-liveness"; /// Delay between gRPC reconnection attempts. internal static TimeSpan ReconnectDelay { get; set; } = TimeSpan.FromSeconds(5); + /// + /// Hard deadline on the initial (WP2.3). The site builds + /// it in milliseconds; if none arrives inside this window the site never answered (the + /// Ask was lost, the singleton moved mid-request, the instance actor is wedged) and the + /// session must FAIL rather than sit in the buffering phase accumulating live events + /// behind a snapshot that is never coming. Settable for tests. + /// + internal static TimeSpan SnapshotTimeout { get; set; } = TimeSpan.FromSeconds(60); + /// /// How long a freshly-opened gRPC stream must stay up before its retry budget /// is considered "recovered" and is reset to 0. @@ -58,11 +70,46 @@ public class DebugStreamBridgeActor : ReceiveActor, IWithTimers /// internal static TimeSpan StabilityWindow { get; set; } = TimeSpan.FromSeconds(60); + /// + /// Orphan window: how long the session may go without ANY sign of life from its CONSUMER + /// before it self-terminates. Renewed by , which + /// DebugStreamService Tells on a timer for every session still attached to a + /// consumer (Blazor debug view or the SignalR hub) — so it measures the consumer, never + /// the stream. Settable for tests. + /// + internal static TimeSpan ConsumerIdleTimeout { get; set; } = TimeSpan.FromMinutes(5); + + /// + /// How often the actor checks the consumer-last-seen stamp against + /// . A self-tick rather than SetReceiveTimeout: + /// the receive timeout measures the MAILBOX, which conflates site chatter with consumer + /// liveness — and once stream events were correctly excluded from it (via + /// ) nothing recurring reset it at all, so every healthy + /// session self-terminated one window after its snapshot with a false "Site disconnected". + /// Settable for tests. + /// + internal static TimeSpan ConsumerLivenessCheckInterval { get; set; } = TimeSpan.FromSeconds(30); + + /// + /// When the consumer was last known to be attached (UTC). Seeded in + /// so a session gets a full window to receive its first keepalive, then refreshed by every + /// . Actor-thread only. + /// + private DateTime _consumerLastSeenUtc = DateTime.UtcNow; + private int _retryCount; private bool _useNodeA = true; private bool _stopped; private CancellationTokenSource? _grpcCts; + /// + /// Monotonic stream generation stamped on each opened gRPC stream and echoed back on its + /// error/completion callbacks: a late callback raced out of a previous (cancelled) stream + /// carries a stale generation and is ignored, so it can neither burn retry budget nor + /// open a duplicate stream. Mirrors SiteAlarmAggregatorActor. Actor-thread only. + /// + private int _streamGeneration; + /// /// Phase flag. until the initial /// has been delivered and the pre-snapshot buffer @@ -77,17 +124,35 @@ public class DebugStreamBridgeActor : ReceiveActor, IWithTimers /// Ordered buffer of live gRPC events (/ /// ) that arrived before the snapshot was delivered. /// Flushed (with per-entity dedup against the snapshot) when the snapshot arrives, - /// then never used again. Mutated only on the actor thread. + /// then never used again. Bounded by with drop-oldest + /// eviction (WP2.3): a snapshot that never arrives used to buffer without limit on the + /// central node. Mutated only on the actor thread. /// - private readonly List _preSnapshotBuffer = new(); + private readonly Queue _preSnapshotBuffer = new(); /// - /// Defensive log threshold: if the pre-snapshot buffer grows past this many events - /// during a slow snapshot we log once (events are NOT dropped — the window is short). + /// Defensive log threshold: the first warning fires when the pre-snapshot buffer grows + /// past this many events during a slow snapshot, before the hard cap starts evicting. /// private const int BufferWarnThreshold = 10_000; private bool _bufferWarned; + /// + /// Hard cap on the pre-snapshot buffer. Beyond it the OLDEST event is evicted — the + /// snapshot that ends the buffering phase is authoritative for anything that old, so + /// keeping the newest events is what preserves the post-snapshot delta chain. + /// + private const int MaxPreSnapshotBuffer = 20_000; + + /// Events evicted from the pre-snapshot buffer in this session. Actor-thread only. + private long _preSnapshotDropped; + + /// + /// Total pre-snapshot events dropped across all debug sessions on this node — the raw + /// counter behind scadabridge.central.debug_view.presnapshot_dropped. + /// + internal static long TotalPreSnapshotDropped; + /// Timer scheduler for reconnect and stability window timers. public ITimerScheduler Timers { get; set; } = null!; @@ -151,7 +216,7 @@ public class DebugStreamBridgeActor : ReceiveActor, IWithTimers // non-deployed instance — cancel it (and any buffered gap events are // discarded with the actor). No pass-through. // _stopped is set AFTER CleanupGrpc() to match the ordering in the - // DebugStreamTerminated and ReceiveTimeout handlers (cosmetic consistency). + // DebugStreamTerminated and consumer-liveness handlers (cosmetic consistency). CleanupGrpc(); _stopped = true; _preSnapshotBuffer.Clear(); @@ -168,6 +233,9 @@ public class DebugStreamBridgeActor : ReceiveActor, IWithTimers _instanceUniqueName, snapshot.AttributeValues.Count, snapshot.AlarmStates.Count, _preSnapshotBuffer.Count); + // The snapshot arrived — stand the hard deadline down. + Timers.Cancel(SnapshotTimerKey); + // Deliver the snapshot, then flush the gap-window buffer (deduped), then // switch to pass-through. Order matters: snapshot first, buffered events next. _onEvent(snapshot); @@ -175,14 +243,38 @@ public class DebugStreamBridgeActor : ReceiveActor, IWithTimers _snapshotDelivered = true; }); - // Domain events arriving via Self.Tell from gRPC callback. - // Receiving an event must NOT reset _retryCount — a - // flapping stream that delivers a single event between failures would - // otherwise never trip MaxRetries. The retry budget is recovered only by - // GrpcStreamStable (a stream that has stayed up for StabilityWindow). - // Before the snapshot has been delivered, BUFFER (in arrival order) - // rather than deliver — these may be gap-window events. After the snapshot has - // been flushed, pass through directly (same handler, phase-dependent behavior). + // Hard snapshot deadline (WP2.3). Nothing else ends a session stuck in the + // buffering phase: the site's reply was lost, so no gRPC error fires, the stream + // keeps delivering events, and stream traffic does not renew the orphan net (which + // measures the consumer). Fail the session so the consumer is told and can reopen. + Receive(_ => + { + if (_stopped || _snapshotDelivered) return; + _log.Error( + "No debug snapshot for {0} within {1}s ({2} event(s) buffered, {3} dropped); failing the session", + _instanceUniqueName, SnapshotTimeout.TotalSeconds, + _preSnapshotBuffer.Count, _preSnapshotDropped); + CleanupGrpc(); + SendUnsubscribe(); + _stopped = true; + _preSnapshotBuffer.Clear(); + _onTerminated(); + Context.Stop(Self); + }); + + // Domain events arriving via Self.Tell from the gRPC callback. Stream traffic never + // proves the CONSUMER is still there, so it deliberately does not touch the orphan + // net (which now measures the consumer keepalive, not the mailbox). Receiving an + // event must not reset _retryCount either: a flapping stream that delivers a single + // event between failures would otherwise never trip MaxRetries. The retry budget is + // recovered only by GrpcStreamStable (a stream that has stayed up for + // StabilityWindow). Before the snapshot has been delivered, BUFFER (in arrival order) + // rather than deliver — these may be gap-window events; after the snapshot has been + // flushed, pass through directly (phase-dependent behavior). + Receive(wrapped => HandleStreamEvent(wrapped.Event)); + + // Unwrapped forms are still accepted (a direct Tell from a test or a future + // in-process producer) and take the identical path. Receive(changed => HandleStreamEvent(changed)); Receive(changed => HandleStreamEvent(changed)); @@ -198,10 +290,33 @@ public class DebugStreamBridgeActor : ReceiveActor, IWithTimers // gRPC stream error — attempt reconnection Receive(msg => { + // Ignore a late error raced out of a previous (cancelled) stream: it must not + // burn retry budget or flip the node a second time. + if (msg.Generation != _streamGeneration) + { + _log.Debug("Ignoring stale gRPC error from stream generation {0} (current {1})", + msg.Generation, _streamGeneration); + return; + } _log.Warning("gRPC stream error for {0}: {1}", _instanceUniqueName, msg.Exception.Message); HandleGrpcError(); }); + // gRPC stream ended GRACEFULLY (server status OK) — the site's 4h max stream + // lifetime elapsing or a graceful site shutdown. Not a fault: reopen on the SAME + // node without spending retry budget. Without this the session went silently deaf. + Receive(msg => + { + if (_stopped) return; + if (msg.Generation != _streamGeneration) + { + _log.Debug("Ignoring stale gRPC completion from stream generation {0} (current {1})", + msg.Generation, _streamGeneration); + return; + } + HandleGrpcCompleted(); + }); + // Scheduled reconnection Receive(_ => OpenGrpcStream()); @@ -226,11 +341,32 @@ public class DebugStreamBridgeActor : ReceiveActor, IWithTimers Context.Stop(Self); }); - // Orphan safety net — if nobody stops us within 5 minutes, self-terminate - Context.SetReceiveTimeout(TimeSpan.FromMinutes(5)); - Receive(_ => + // Consumer keepalive: DebugStreamService Tells this on a timer for every session it + // still holds (i.e. still attached to a Blazor debug view / SignalR connection). It + // is the ONLY thing that renews the orphan window — deliberately, so neither a chatty + // site nor a silent one can influence it. + Receive(_ => { - _log.Warning("Debug stream for {0} timed out (orphaned session), stopping", _instanceUniqueName); + if (_stopped) return; + _consumerLastSeenUtc = DateTime.UtcNow; + }); + + // Orphan safety net, CONSUMER-measured (WP2.3 follow-up). A periodic self-tick + // compares the consumer-last-seen stamp against ConsumerIdleTimeout; the previous + // SetReceiveTimeout(5 min) measured the mailbox instead, and once stream events were + // (correctly) marked INotInfluenceReceiveTimeout nothing recurring reset it — the + // snapshot arrives once and GrpcStreamStable once, so EVERY healthy session died at + // ~6 minutes and the consumer was told "Site disconnected". + Receive(_ => + { + if (_stopped) return; + var idle = DateTime.UtcNow - _consumerLastSeenUtc; + if (idle < ConsumerIdleTimeout) return; + + _log.Warning( + "Debug stream for {0} has had no consumer activity for {1:F0}s (orphaned session), stopping", + _instanceUniqueName, idle.TotalSeconds); + Timers.Cancel(ConsumerLivenessTimerKey); CleanupGrpc(); SendUnsubscribe(); _stopped = true; @@ -255,15 +391,30 @@ public class DebugStreamBridgeActor : ReceiveActor, IWithTimers return; } - _preSnapshotBuffer.Add(evt); - if (!_bufferWarned && _preSnapshotBuffer.Count > BufferWarnThreshold) + if (!_bufferWarned && _preSnapshotBuffer.Count + 1 > BufferWarnThreshold) { _bufferWarned = true; _log.Warning( - "Pre-snapshot debug-event buffer for {0} exceeded {1} events while awaiting the snapshot; " + - "events are still retained (not dropped).", - _instanceUniqueName, BufferWarnThreshold); + "Pre-snapshot debug-event buffer for {0} exceeded {1} events while awaiting the snapshot " + + "(hard cap {2}, drop-oldest beyond it).", + _instanceUniqueName, BufferWarnThreshold, MaxPreSnapshotBuffer); } + + while (_preSnapshotBuffer.Count >= MaxPreSnapshotBuffer) + { + _preSnapshotBuffer.Dequeue(); + _preSnapshotDropped++; + Interlocked.Increment(ref TotalPreSnapshotDropped); + ScadaBridgeTelemetry.RecordDebugPreSnapshotDrop(); + if (_preSnapshotDropped == 1 || _preSnapshotDropped % 500 == 0) + { + _log.Warning( + "Pre-snapshot debug-event buffer for {0} is at its {1}-event cap; {2} event(s) evicted so far", + _instanceUniqueName, MaxPreSnapshotBuffer, _preSnapshotDropped); + } + } + + _preSnapshotBuffer.Enqueue(evt); } /// @@ -321,8 +472,9 @@ public class DebugStreamBridgeActor : ReceiveActor, IWithTimers if (dropped > 0 || flushed > 0) { - _log.Debug("Flushed {0} buffered debug event(s) for {1}, dropped {2} as already-in-snapshot", - flushed, _instanceUniqueName, dropped); + _log.Debug("Flushed {0} buffered debug event(s) for {1}, dropped {2} as already-in-snapshot" + + " ({3} previously evicted at the buffer cap)", + flushed, _instanceUniqueName, dropped, _preSnapshotDropped); } _preSnapshotBuffer.Clear(); @@ -398,6 +550,19 @@ public class DebugStreamBridgeActor : ReceiveActor, IWithTimers var request = new SubscribeDebugViewRequest(_instanceUniqueName, _correlationId); var envelope = new SiteEnvelope(_siteIdentifier, request); _centralCommunicationActor.Tell(envelope, Self); + + // Arm the hard snapshot deadline alongside the request. + if (SnapshotTimeout > TimeSpan.Zero) + Timers.StartSingleTimer(SnapshotTimerKey, new DebugSnapshotDeadline(), SnapshotTimeout); + + // Arm the consumer-liveness net. The stamp starts now, so the session always gets a + // full ConsumerIdleTimeout to see its first keepalive from DebugStreamService. + _consumerLastSeenUtc = DateTime.UtcNow; + if (ConsumerIdleTimeout > TimeSpan.Zero && ConsumerLivenessCheckInterval > TimeSpan.Zero) + { + Timers.StartPeriodicTimer( + ConsumerLivenessTimerKey, new ConsumerLivenessTick(), ConsumerLivenessCheckInterval); + } } /// @@ -424,20 +589,58 @@ public class DebugStreamBridgeActor : ReceiveActor, IWithTimers // retry budget is recovered. Cancelled by HandleGrpcError. Timers.StartSingleTimer(StabilityTimerKey, new GrpcStreamStable(), StabilityWindow); + var generation = ++_streamGeneration; var client = _grpcFactory.GetOrCreate(_siteIdentifier, endpoint); var self = Self; var ct = _grpcCts.Token; - // Launch as background task — onEvent and onError marshal back to actor via Tell + // Launch as background task — the callbacks marshal back to the actor via Tell. + // The task itself is observed below: a fault escaping SubscribeAsync would otherwise + // leave the session waiting on a stream that does not exist, exception unobserved. Task.Run(async () => { await client.SubscribeAsync( _correlationId, _instanceUniqueName, - evt => self.Tell(evt), - ex => self.Tell(new GrpcStreamError(ex)), + // Wrapped so the stream path is explicit (it never renews the orphan net). + evt => self.Tell(new LiveDebugStreamEvent(evt)), + ex => self.Tell(new GrpcStreamError(ex, generation)), + () => self.Tell(new GrpcStreamCompleted(generation)), ct); - }, ct); + }, ct).ContinueWith(t => + { + if (t.IsFaulted) + self.Tell(new GrpcStreamError(t.Exception!.GetBaseException(), generation)); + else if (t.IsCanceled && !ct.IsCancellationRequested) + self.Tell(new GrpcStreamCompleted(generation)); + // RanToCompletion: SubscribeAsync already reported its own outcome. + }, TaskContinuationOptions.ExecuteSynchronously); + } + + /// + /// Handles a graceful end of stream (server status OK). The stream simply expired or the + /// site shut down cleanly, so the retry budget is left untouched and the endpoint is not + /// flipped; the reopen is scheduled through the existing reconnect timer, which also + /// rate-limits a pathological site that keeps closing streams immediately. + /// + private void HandleGrpcCompleted() + { + // The stream is gone, so its armed stability timer must not later "recover" a + // budget that its successor has since spent. + Timers.Cancel(StabilityTimerKey); + + _log.Info("gRPC stream for {0} completed gracefully (server end of stream); reopening", + _instanceUniqueName); + + // Release the site-side relay for the finished stream before reopening, so the site + // is not left with a zombie relay actor for this correlation id. + _grpcCts?.Cancel(); + _grpcCts?.Dispose(); + _grpcCts = null; + _grpcFactory.TryGet(_siteIdentifier, _useNodeA ? _grpcNodeAAddress : _grpcNodeBAddress) + ?.Unsubscribe(_correlationId); + + Timers.StartSingleTimer(ReconnectTimerKey, new ReconnectGrpcStream(), ReconnectDelay); } private void HandleGrpcError() @@ -512,9 +715,45 @@ public class DebugStreamBridgeActor : ReceiveActor, IWithTimers public record StopDebugStream; /// -/// Internal message indicating a gRPC stream error occurred. +/// Envelope for a live gRPC stream event (AttributeValueChanged/ +/// AlarmStateChanged). Kept as a distinct envelope so the high-volume stream path is +/// explicit at the call site; the orphan net no longer keys off the mailbox at all (it +/// measures the consumer keepalive), so a busy site's event flood can neither hold a dead +/// session open nor — as briefly happened — be the only thing keeping a healthy one alive. /// -internal record GrpcStreamError(Exception Exception); +internal record LiveDebugStreamEvent(object Event); + +/// +/// Consumer keepalive: DebugStreamService Tells one of these to every bridge actor +/// whose session is still attached to a consumer, on +/// -scale cadence. Renewing +/// the consumer-last-seen stamp is its ONLY effect. +/// +public record DebugStreamConsumerAlive; + +/// +/// Internal self-tick that checks the consumer-last-seen stamp against +/// . +/// +internal record ConsumerLivenessTick; + +/// +/// Internal message: the hard deadline for the initial DebugViewSnapshot expired. +/// +internal record DebugSnapshotDeadline; + +/// +/// Internal message indicating a gRPC stream error occurred, stamped with the stream +/// generation it came from so a late error out of a cancelled stream can be ignored. +/// +internal record GrpcStreamError(Exception Exception, int Generation); + +/// +/// Internal message indicating the gRPC stream ended gracefully (server status OK — the +/// site's max stream lifetime elapsed, or the site shut down cleanly), stamped with its +/// stream generation. +/// +internal record GrpcStreamCompleted(int Generation); /// /// Internal message to trigger gRPC stream reconnection. diff --git a/src/ZB.MOM.WW.ScadaBridge.Communication/Actors/ISiteCommandTransport.cs b/src/ZB.MOM.WW.ScadaBridge.Communication/Actors/ISiteCommandTransport.cs index 89545aa6..1b3efc0f 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Communication/Actors/ISiteCommandTransport.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Communication/Actors/ISiteCommandTransport.cs @@ -24,8 +24,10 @@ public interface ISiteCommandTransport /// Routes 's message to its site. Any reply the site produces is /// delivered to — for an Ask that is the temporary ask actor /// (completing the caller's task); for a Tell-with-sender (the debug bridge) that is the - /// originating actor. A message with no route (an unknown site) is warned and dropped so the - /// caller's Ask times out — central never buffers for an unreachable site. + /// originating actor. A message with no route (an unknown/unconfigured site) fails fast: a + /// is delivered to immediately instead of + /// letting the caller's Ask burn its full timeout — central still never buffers for an + /// unreachable site. /// /// The site-addressed command envelope. /// Where a reply (or a ) is delivered. diff --git a/src/ZB.MOM.WW.ScadaBridge.Communication/Actors/SiteAlarmAggregatorActor.cs b/src/ZB.MOM.WW.ScadaBridge.Communication/Actors/SiteAlarmAggregatorActor.cs index a320479a..e2e0b439 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Communication/Actors/SiteAlarmAggregatorActor.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Communication/Actors/SiteAlarmAggregatorActor.cs @@ -31,10 +31,14 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Actors; /// /// /// Failover + drift: a gRPC error flips NodeA↔NodeB with the same retry budget + -/// stability window as , and each reconnect triggers -/// a RE-SEED (never silently serve stale). A periodic reconcile snapshot -/// (, default 60s) corrects instance-set drift and any -/// missed delta. +/// stability window as . A re-seed runs once per +/// successful (re)connect — driven by the stream's connected callback, not by each +/// reconnect ATTEMPT (WP2.3: a site outage used to fan a full snapshot out per retry, all +/// of them against the very site that is unreachable). A periodic reconcile snapshot +/// (, default 60s, jittered per site so N aggregators do +/// not fan out in lockstep) remains the drift/backstop: it corrects instance-set drift and +/// any missed delta, but it is skipped when a fan-out already completed inside the +/// window, and it publishes to viewers only when the snapshot actually changed the cache. /// /// All state is mutated only on the actor thread: gRPC callbacks and fan-out results are /// marshalled back via Self.Tell, so the cache needs no internal lock. The @@ -47,18 +51,20 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers private readonly string _siteIdentifier; private readonly string _correlationId; private readonly Func>> _seedFn; - private readonly Action> _publish; + private readonly Action, bool> _publish; private readonly SiteStreamGrpcClientFactory _grpcFactory; private readonly string _grpcNodeAAddress; private readonly string _grpcNodeBAddress; private readonly TimeSpan _reconcileInterval; private readonly TimeSpan _publishCoalesce; + private readonly double _reconcileJitterFraction; private const int MaxRetries = 3; private const string ReconnectTimerKey = "alarm-grpc-reconnect"; private const string StabilityTimerKey = "alarm-grpc-stability"; private const string ReconcileTimerKey = "alarm-reconcile"; private const string PublishTimerKey = "alarm-publish-coalesce"; + private const string SeedRetryTimerKey = "alarm-seed-retry"; /// True while a coalesced publish is armed (dirty deltas awaiting one tick). Actor-thread only. private bool _publishPending; @@ -78,12 +84,24 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers private bool _stopped; /// - /// True once the live gRPC stream has been given up (retry budget exhausted). Reconcile - /// snapshots keep serving in the meantime; the next reconcile tick self-heals the stream - /// by resetting the retry budget and reopening it, so a sustained site outage does not - /// permanently drop the live feed. Actor-thread only. + /// True while there is no live gRPC stream — either it was given up (retry budget + /// exhausted) or it ended gracefully (server status OK at the site's max stream + /// lifetime). Reconcile snapshots keep serving in the meantime; the next reconcile tick + /// self-heals the stream by resetting the retry budget and reopening it, so neither a + /// sustained site outage nor a routine 4h stream expiry permanently drops the live feed. + /// Starts true: until the site accepts the first subscription there is no live + /// feed, and the owning cache must not advertise one. Actor-thread only. /// - private bool _streamDown; + private bool _streamDown = true; + + /// + /// Why the stream is down: true = the retry budget was exhausted, so the + /// self-healing reopen must also reset it; false = it ended gracefully, and the + /// budget — which a completion neither spends nor refunds — is carried across the reopen + /// untouched (a stream flapping between faults and clean closes must still trip + /// MaxRetries). Actor-thread only. + /// + private bool _retryBudgetExhausted; private CancellationTokenSource? _grpcCts; private CancellationTokenSource? _lifetimeCts; @@ -96,8 +114,80 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers /// True while a seed/reconcile snapshot fan-out is in flight (deltas buffer). Actor-thread only. private bool _fanoutInFlight; - /// Ordered buffer of live deltas that arrived while a fan-out was in flight. Actor-thread only. - private readonly List _buffer = new(); + /// + /// Ordered buffer of live deltas that arrived while a fan-out was in flight, capped at + /// with drop-oldest eviction (WP2.3 — a stuck fan-out + /// used to buffer without bound). Dropping the OLDEST is the right eviction here: the + /// fan-out that follows rebuilds the cache authoritatively, so an evicted delta is + /// superseded rather than lost, while the newest transitions — the ones the snapshot may + /// predate — are the ones kept. Actor-thread only. + /// + private readonly Queue _buffer = new(); + + /// Total deltas evicted from over this actor's life. Actor-thread only. + private long _bufferDropped; + + /// + /// Hard cap on the pre-fan-out delta buffer. Beyond this the oldest entry is evicted and + /// counted; the counter is logged (first drop + every 500th) and exported as + /// scadabridge.site.alarm_cache.buffer_dropped. + /// + private const int MaxBufferedDeltas = 20_000; + + /// + /// True when the next successful (re)connect must run a seed fan-out: set whenever the + /// stream is lost (fault or graceful completion) and cleared by the connect that consumes + /// it. This is what makes the re-seed happen once per successful reconnect rather than + /// once per reconnect ATTEMPT — the old code fanned a full snapshot out on every retry, + /// against a site that was by definition unreachable. Actor-thread only. + /// + private bool _seedOnConnect; + + /// + /// Set when a CONNECT- OR FAILOVER-DRIVEN fan-out finishes (the initial seed, a re-seed + /// consumed by a successful (re)connect, or a re-seed queued behind one); consumed by the + /// next reconcile tick, which skips its own fan-out when it finds the flag set. That makes + /// the connect-driven seed and the periodic backstop mutually exclusive — the pair used to + /// run BOTH, which is the "full unconditional snapshot every 60s" waste — while bounding + /// staleness at two intervals (a tick can be skipped at most once in a row). + /// + /// A TICK-driven fan-out deliberately does NOT set it. It used to: the tick's own fan-out + /// completion armed the skip, so steady state ran fan-out → skip → fan-out → skip, i.e. + /// one reconcile per TWO intervals — halving the not-reporting refresh rate and the alarm + /// reconcile backstop for no reason (nothing was duplicated to suppress). + /// + /// Actor-thread only. + /// + private bool _fanoutSinceLastTick; + + /// + /// Whether the in-flight fan-out is connect/failover-driven and therefore makes the next + /// reconcile tick redundant (see ). Stable for the + /// fan-out's lifetime: never overwrites it while one is in + /// flight (a colliding request queues instead). Actor-thread only. + /// + private bool _fanoutSuppressesNextTick; + + /// + /// Same flag for a fan-out that was queued behind an in-flight one + /// () — OR-ed across colliding requests so a queued + /// connect/failover re-seed keeps its suppression even if a tick collided too. + /// Actor-thread only. + /// + private bool _queuedFanoutSuppressesNextTick; + + /// + /// True between issuing a stream open and its first outcome (connected / error / + /// completed), so a reconcile tick never stacks a second open on an in-flight one. + /// Actor-thread only. + /// + private bool _openInFlight; + + /// Consecutive whole-fan-out failures, driving the seed-retry backoff. Actor-thread only. + private int _consecutiveSeedFailures; + + /// Upper bound on the seed-retry backoff, as a multiple of the reconcile interval. + private const int MaxSeedRetryBackoffMultiplier = 8; /// /// A failover re-seed was requested while a fan-out was already in flight; it must run @@ -133,6 +223,10 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers /// /// Publishes a fresh immutable snapshot of the cache to the owning service, which /// stores it and raises viewer onChanged callbacks. Invoked on the actor thread. + /// The second argument is the live-stream liveness flag: false means the site-wide + /// gRPC stream is currently down (fault-exhausted or gracefully completed), so the cache + /// is only as fresh as the last reconcile and the page must fall back to polling + /// (WP2.3 carried residual — IsLive used to stay true across a dead stream). /// /// Factory caching one gRPC client per (site, endpoint). /// gRPC address of the site's node A. @@ -147,18 +241,24 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers /// /// How long a fresh gRPC stream must stay up before its retry budget recovers (production 60s). /// + /// + /// Fraction of added as a per-tick random offset so + /// N per-site aggregators on one central node do not fan out in lockstep (production 0.2 = + /// up to +20%). Zero disables jitter, which is what tests want for determinism. + /// public SiteAlarmAggregatorActor( string siteIdentifier, string correlationId, Func>> seedFn, - Action> publish, + Action, bool> publish, SiteStreamGrpcClientFactory grpcFactory, string grpcNodeAAddress, string grpcNodeBAddress, TimeSpan reconcileInterval, TimeSpan publishCoalesce, TimeSpan reconnectDelay, - TimeSpan stabilityWindow) + TimeSpan stabilityWindow, + double reconcileJitterFraction = 0.0) { _siteIdentifier = siteIdentifier; _correlationId = correlationId; @@ -171,6 +271,7 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers _publishCoalesce = publishCoalesce; _reconnectDelay = reconnectDelay; _stabilityWindow = stabilityWindow; + _reconcileJitterFraction = Math.Clamp(reconcileJitterFraction, 0.0, 1.0); // Live delta from the site-wide alarm stream (marshalled in via Self.Tell). // A received delta must NOT reset the retry budget (a flapping stream that @@ -187,6 +288,24 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers // Periodic reconcile tick (and the re-seed kicked after a reconnect). Receive(_ => OnReconcileTick()); + // Backoff-scheduled retry of a fan-out that failed as a whole. + Receive(_ => + { + if (_stopped) return; + // A backoff retry of a failed fan-out is not a connect/failover seed: it must + // not stand the next reconcile tick down. + StartFanout(isInitial: false, suppressesNextTick: false); + }); + + // The site-wide stream is confirmed established (response headers received). This — + // not the reconnect attempt — is the "successful (re)connect" that earns one re-seed. + Receive(msg => + { + if (_stopped) return; + if (msg.Generation != _streamGeneration) return; + OnStreamConnected(); + }); + // Coalesced-publish tick: one publish for a batch of dirtying deltas (N6). Receive(_ => { @@ -218,6 +337,22 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers HandleGrpcError(); }); + // gRPC stream ended GRACEFULLY (server status OK) — the site's 4h max stream + // lifetime elapsing or a graceful site shutdown. Not a fault: the stream is marked + // down so the reconcile tick reopens it (which also re-seeds), but the retry budget + // is untouched and the node is not flipped. Same generation fence as the error path. + Receive(msg => + { + if (_stopped) return; + if (msg.Generation != _streamGeneration) + { + _log.Debug("Ignoring stale gRPC completion from stream generation {0} (current {1})", + msg.Generation, _streamGeneration); + return; + } + HandleGrpcCompleted(); + }); + Receive(_ => OpenGrpcStream()); // Owning service asks us to stop (last viewer left + linger elapsed). @@ -243,11 +378,31 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers // in the seed window are captured (buffered) rather than lost. OpenGrpcStream(); - // Kick the initial seed fan-out. - StartFanout(isInitial: true); + // Kick the initial seed fan-out. The connect callback for this first stream must + // NOT seed again on top of it, so the flag starts cleared. + _seedOnConnect = false; + StartFanout(isInitial: true, suppressesNextTick: true); - // Periodic reconcile backstop. - Timers.StartPeriodicTimer(ReconcileTimerKey, new RunReconcile(), _reconcileInterval, _reconcileInterval); + // Periodic reconcile backstop. Single-shot and re-armed with fresh jitter each + // tick (a periodic timer would lock every site to the same phase forever). + ArmReconcileTimer(); + } + + /// + /// Arms the next reconcile tick at plus a random + /// offset of up to of it, so N per-site + /// aggregators started together (a central failover restarts them all at once) spread + /// their fan-outs instead of stampeding the same 60s boundary. + /// + private void ArmReconcileTimer() + { + var delay = _reconcileInterval; + if (_reconcileJitterFraction > 0 && _reconcileInterval > TimeSpan.Zero) + { + var jitterTicks = (long)(_reconcileInterval.Ticks * _reconcileJitterFraction * Random.Shared.NextDouble()); + delay += TimeSpan.FromTicks(jitterTicks); + } + Timers.StartSingleTimer(ReconcileTimerKey, new RunReconcile(), delay); } /// @@ -267,22 +422,56 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers // ── Reconcile tick ────────────────────────────────────────────────────────── /// - /// Periodic reconcile: always re-run the snapshot fan-out (corrects drift + missed - /// deltas), and if the live stream was previously given up, self-heal it by resetting - /// the retry budget and reopening — so a sustained outage never permanently kills the - /// live feed. + /// Periodic reconcile backstop: re-run the snapshot fan-out (corrects instance-set drift + /// + any missed delta) UNLESS a CONNECT- OR FAILOVER-DRIVEN seed already completed inside + /// this window — that seed makes the tick redundant, and running both was the "full + /// unconditional snapshot every 60s" waste (WP2.3). A tick's OWN fan-out never arms that + /// skip: it used to, which made steady state alternate fan-out/skip and halved the + /// effective reconcile rate. If the live stream was previously given up, self-heal it by + /// resetting the retry budget and reopening, so a sustained outage never permanently kills + /// the live feed. /// private void OnReconcileTick() { if (_stopped) return; - StartFanout(isInitial: false); - if (_streamDown) + + // Re-arm first: every exit path below must leave the backstop running. + ArmReconcileTimer(); + + // A down stream is reopened in preference to fanning out: the reopen's connect + // callback seeds by itself, so doing both would double the work. + var reopening = false; + if (_streamDown && !_openInFlight) { _log.Info("Site-alarm gRPC stream for {0} was down; reopening on reconcile tick", _siteIdentifier); - _retryCount = 0; + if (_retryBudgetExhausted) + { + _retryBudgetExhausted = false; + _retryCount = 0; + } // Telemetry: a reconcile-driven reopen after the stream was given up is a reconnect. ScadaBridgeTelemetry.RecordLiveAlarmStreamReconnect(); + _seedOnConnect = true; OpenGrpcStream(); + reopening = true; + } + + if (_fanoutSinceLastTick) + { + // A connect-driven seed (or the initial seed) already refreshed the cache inside + // this window. Clear the flag so the NEXT tick fans out regardless — staleness + // stays bounded at two intervals even if reconnects keep arriving. + _fanoutSinceLastTick = false; + _log.Debug("Site-alarm reconcile for {0} skipped; a fan-out already ran this window", + _siteIdentifier); + return; + } + + if (!reopening) + { + // Tick-driven: deliberately does NOT set the skip flag, so the NEXT tick fans + // out too. Setting it here was the "one reconcile per two intervals" bug. + StartFanout(isInitial: false, suppressesNextTick: false); } } @@ -293,7 +482,7 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers /// Self.Tell. While in flight, live deltas buffer. A reconcile that arrives /// while a fan-out is already running is skipped (no stacking). /// - private void StartFanout(bool isInitial) + private void StartFanout(bool isInitial, bool suppressesNextTick) { if (_stopped) return; if (_fanoutInFlight) @@ -302,11 +491,16 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers // one — its snapshot read-time predates the stream death, so skipping it would // serve stale up to the next 60s reconcile (N7.1). An initial-seed collision // never queues (there is only ever one). - if (!isInitial) _reseedQueued = true; + if (!isInitial) + { + _reseedQueued = true; + _queuedFanoutSuppressesNextTick |= suppressesNextTick; + } return; } _fanoutInFlight = true; + _fanoutSuppressesNextTick = suppressesNextTick; var self = Self; var ct = _lifetimeCts?.Token ?? CancellationToken.None; @@ -335,6 +529,12 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers // Rebuild the cache authoritatively from the fresh snapshot (this is what makes // reconcile able to DROP rows for instances/alarms that disappeared — a merge // could never remove a stale row since the live stream sends no "removed" event). + // The previous cache is kept alongside so the reconcile can publish as a DIFF: + // an unchanged snapshot must not wake every viewer's render path once a minute. + var previous = _cache.Count == 0 + ? null + : new Dictionary(_cache); + _cache.Clear(); foreach (var alarm in msg.Alarms) { @@ -347,44 +547,85 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers // buffered delta whose key is in the fresh snapshot with an equal-or-newer // timestamp is already reflected → drop; a strictly-newer (or new-key) delta is // applied. Inclusive-on-snapshot boundary matches DebugStreamBridgeActor. - FlushBuffer(); + var flushChanged = FlushBuffer(); _fanoutInFlight = false; + // Only a connect/failover-driven seed makes the next tick redundant — a tick's own + // fan-out must never suppress the tick after it (that halved the reconcile rate). + if (_fanoutSuppressesNextTick) _fanoutSinceLastTick = true; + _consecutiveSeedFailures = 0; + Timers.Cancel(SeedRetryTimerKey); + var firstSeed = !_seeded; _seeded = true; _log.Debug("Site-alarm {0} {1} complete: {2} alarm row(s)", _siteIdentifier, msg.IsInitial ? "seed" : "reconcile", _cache.Count); // The fresh snapshot already carries the buffered deltas; drop any armed coalesce - // tick so we publish once, immediately. + // tick so we publish once, immediately — but only when something actually moved. Timers.Cancel(PublishTimerKey); _publishPending = false; - Publish(); + if (firstSeed || flushChanged || DiffersFrom(previous)) + Publish(); // A failover re-seed requested while this fan-out was in flight runs now (N7.1). - if (_reseedQueued) + RunQueuedFanoutIfAny(); + } + + /// + /// Runs a fan-out that collided with an in-flight one, carrying the suppression flag it + /// was queued with so a queued connect/failover re-seed still stands the next tick down + /// (and a queued TICK fan-out still does not). + /// + private void RunQueuedFanoutIfAny() + { + if (!_reseedQueued) return; + _reseedQueued = false; + var suppresses = _queuedFanoutSuppressesNextTick; + _queuedFanoutSuppressesNextTick = false; + StartFanout(isInitial: false, suppressesNextTick: suppresses); + } + + /// + /// True when the just-installed snapshot differs from in any + /// key or any row value — the reconcile's "is this publish worth a viewer fan-out?" test. + /// A null previous cache (nothing published yet) always counts as different. + /// + private bool DiffersFrom(Dictionary? previous) + { + if (previous is null || previous.Count != _cache.Count) return true; + + foreach (var (key, current) in _cache) { - _reseedQueued = false; - StartFanout(isInitial: false); + if (!previous.TryGetValue(key, out var old)) return true; + // AlarmStateChanged is a record: structural equality covers state, level, + // timestamp and every native-alarm enrichment field. + if (!Equals(old, current)) return true; } + return false; } private void OnSeedFailed(SeedFailed msg) { if (_stopped) return; + _consecutiveSeedFailures++; + _log.Warning(msg.Exception, - "Site-alarm {0} {1} fan-out failed; keeping current cache and relying on the next reconcile", - _siteIdentifier, msg.IsInitial ? "seed" : "reconcile"); + "Site-alarm {0} {1} fan-out failed ({2} consecutive); keeping current cache and retrying with backoff", + _siteIdentifier, msg.IsInitial ? "seed" : "reconcile", _consecutiveSeedFailures); // Don't lose deltas captured during the failed window — apply them pass-through // into the (possibly stale/empty) cache. The next reconcile re-seeds authoritatively. _fanoutInFlight = false; - FlushBuffer(dedupAgainstSeed: false); + // Same rule as the success path: only a connect/failover-driven fan-out stands the + // next reconcile tick down. + if (_fanoutSuppressesNextTick) _fanoutSinceLastTick = true; + var flushChanged = FlushBuffer(dedupAgainstSeed: false); // Only publish if we already had a seed (so IsLive doesn't flip true on a // failed initial seed — the page keeps its poll fallback until we truly seed). - if (_seeded) + if (_seeded && flushChanged) { Timers.Cancel(PublishTimerKey); _publishPending = false; @@ -394,29 +635,55 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers // A failover re-seed requested while this fan-out was in flight runs now (N7.1). if (_reseedQueued) { - _reseedQueued = false; - StartFanout(isInitial: false); + RunQueuedFanoutIfAny(); + return; } + + // Backoff retry (WP2.3): the seed leg used to have none — a site returning errors + // was re-fanned at full reconcile cadence forever. Retry at reconnectDelay doubling + // up to MaxSeedRetryBackoffMultiplier × the reconcile interval; the periodic + // reconcile remains the floor once the backoff saturates. + ScheduleSeedRetry(); + } + + /// + /// Arms the backoff retry for a failed fan-out: reconnectDelay × 2^(failures-1), + /// capped at × the reconcile interval. Once + /// the cap is reached the periodic reconcile tick is doing the same work anyway, so no + /// extra timer is armed. + /// + private void ScheduleSeedRetry() + { + var cap = _reconcileInterval > TimeSpan.Zero + ? TimeSpan.FromTicks(_reconcileInterval.Ticks * MaxSeedRetryBackoffMultiplier) + : TimeSpan.FromMinutes(8); + + var shift = Math.Min(_consecutiveSeedFailures - 1, 16); + var delayTicks = _reconnectDelay.Ticks * (1L << shift); + var delay = TimeSpan.FromTicks(Math.Min(delayTicks, cap.Ticks)); + if (delay <= TimeSpan.Zero) return; + + Timers.StartSingleTimer(SeedRetryTimerKey, new RetrySeed(), delay); } /// /// Flushes the pre-fan-out buffer in arrival order. When /// is true (normal completion) a buffered delta already reflected in the just-installed /// snapshot (same key, timestamp <= cache entry) is dropped; otherwise every buffered - /// delta is applied pass-through. + /// delta is applied pass-through. Returns true when at least one delta changed the + /// cache (the reconcile's publish-only-on-change test must count these too). /// - private void FlushBuffer(bool dedupAgainstSeed = true) + private bool FlushBuffer(bool dedupAgainstSeed = true) { - if (_buffer.Count == 0) return; + if (_buffer.Count == 0) return false; + var changed = false; foreach (var delta in _buffer) { - if (dedupAgainstSeed) - ApplyDelta(delta, requireStrictlyNewer: true); - else - ApplyDelta(delta, requireStrictlyNewer: false); + changed |= ApplyDelta(delta, requireStrictlyNewer: dedupAgainstSeed); } _buffer.Clear(); + return changed; } // ── Live delta handling ───────────────────────────────────────────────────── @@ -427,15 +694,32 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers if (_fanoutInFlight) { - _buffer.Add(delta); - if (!_bufferWarned && _buffer.Count > BufferWarnThreshold) + if (!_bufferWarned && _buffer.Count + 1 > BufferWarnThreshold) { _bufferWarned = true; _log.Warning( "Site-alarm pre-seed buffer for {0} exceeded {1} deltas while a fan-out was in flight " + - "(deltas retained, not dropped).", - _siteIdentifier, BufferWarnThreshold); + "(hard cap {2}, drop-oldest beyond it).", + _siteIdentifier, BufferWarnThreshold, MaxBufferedDeltas); } + + // Hard cap with drop-oldest: an unbounded buffer behind a stuck fan-out was a + // straight memory leak. The evicted rows are superseded by the snapshot that + // ends the fan-out, so the drop costs freshness, never correctness. + while (_buffer.Count >= MaxBufferedDeltas) + { + _buffer.Dequeue(); + _bufferDropped++; + ScadaBridgeTelemetry.RecordLiveAlarmBufferDrop(); + if (_bufferDropped == 1 || _bufferDropped % 500 == 0) + { + _log.Warning( + "Site-alarm pre-seed buffer for {0} is at its {1}-delta cap; {2} delta(s) evicted so far", + _siteIdentifier, MaxBufferedDeltas, _bufferDropped); + } + } + + _buffer.Enqueue(delta); return; } @@ -485,7 +769,10 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers var snapshot = _cache.Values.ToList(); try { - _publish(snapshot); + // Liveness rides every publish so the owning service's IsLive tracks the STREAM, + // not merely "a snapshot was published once". A down stream means the cache is + // only as fresh as the last reconcile and the page must keep polling. + _publish(snapshot, !_streamDown); } catch (Exception ex) { @@ -499,7 +786,8 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers { if (_stopped) return; - _streamDown = false; + // The stream is not live again until the server actually accepts it — liveness flips + // on GrpcAlarmStreamConnected, not on the attempt (WP2.3 carried residual). var endpoint = _useNodeA ? _grpcNodeAAddress : _grpcNodeBAddress; _log.Info("Opening site-alarm gRPC stream for {0} to {1}", _siteIdentifier, endpoint); @@ -509,19 +797,59 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers Timers.StartSingleTimer(StabilityTimerKey, new GrpcAlarmStreamStable(), _stabilityWindow); + _openInFlight = true; var generation = ++_streamGeneration; var client = _grpcFactory.GetOrCreate(_siteIdentifier, endpoint); var self = Self; var ct = _grpcCts.Token; + // The subscription task itself is observed (below): a fault escaping + // SubscribeSiteAsync — or a Task.Run that never started because ct was already + // cancelled — would otherwise leave the actor waiting on a stream that does not + // exist, with the exception silently unobserved. Task.Run(async () => { await client.SubscribeSiteAsync( _correlationId, alarm => self.Tell(alarm), ex => self.Tell(new GrpcAlarmStreamError(ex, generation)), - ct); - }, ct); + () => self.Tell(new GrpcAlarmStreamCompleted(generation)), + ct, + () => self.Tell(new GrpcAlarmStreamConnected(generation))); + }, ct).ContinueWith(t => + { + if (t.IsFaulted) + self.Tell(new GrpcAlarmStreamError(t.Exception!.GetBaseException(), generation)); + else if (t.IsCanceled && !ct.IsCancellationRequested) + self.Tell(new GrpcAlarmStreamCompleted(generation)); + // RanToCompletion: SubscribeSiteAsync already reported its own outcome. + }, TaskContinuationOptions.ExecuteSynchronously); + } + + /// + /// The site accepted the subscription — the only point at which the stream is genuinely + /// live. Consumes the pending re-seed (so exactly ONE fan-out runs per successful + /// (re)connect, instead of one per reconnect attempt against a site that is by definition + /// unreachable) and republishes so IsLive recovers. + /// + private void OnStreamConnected() + { + _openInFlight = false; + + var wasDown = _streamDown; + _streamDown = false; + + if (_seedOnConnect) + { + _seedOnConnect = false; + _log.Info("Site-alarm gRPC stream for {0} (re)connected; running one re-seed", _siteIdentifier); + StartFanout(isInitial: false, suppressesNextTick: true); + } + + // Liveness recovered — republish so viewers stop falling back to polling even if the + // seed that follows finds nothing changed. + if (wasDown && _seeded) + Publish(); } private void HandleGrpcError() @@ -531,6 +859,19 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers // Stream failed before the stability window — retry budget NOT recovered. Timers.Cancel(StabilityTimerKey); + _openInFlight = false; + var wasLive = !_streamDown; + _streamDown = true; + + // The next successful connect owes exactly one re-seed. Setting it here (rather than + // fanning out now) is the fix for the per-attempt re-fan-out: a site outage used to + // cost one whole-site snapshot per retry. + _seedOnConnect = true; + + // Liveness lost — tell viewers immediately so a dead stream stops reading as live. + if (wasLive && _seeded) + Publish(); + _retryCount++; if (_retryCount > MaxRetries) @@ -542,7 +883,7 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers _log.Error("Site-alarm gRPC stream for {0} exceeded max retries ({1}); leaving stream down, " + "reconcile snapshots continue and the next reconcile tick will retry the stream", _siteIdentifier, MaxRetries); - _streamDown = true; + _retryBudgetExhausted = true; CleanupGrpc(); return; } @@ -558,17 +899,44 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers // Telemetry: a NodeA↔NodeB failover flip is a reconnect + re-seed. ScadaBridgeTelemetry.RecordLiveAlarmStreamReconnect(); - // A failover flip must RE-SEED (never silently serve stale) — kick a reconcile - // fan-out alongside the reconnect. Buffering during the fan-out keeps the new - // stream's deltas coherent with the fresh snapshot. - StartFanout(isInitial: false); - + // The re-seed is owed to the CONNECT (see _seedOnConnect above), not to this + // attempt: a snapshot fan-out issued while the site is unreachable degrades to + // empty rows and would have to be redone on reconnect anyway. if (_retryCount == 1) Self.Tell(new ReconnectAlarmStream()); else Timers.StartSingleTimer(ReconnectTimerKey, new ReconnectAlarmStream(), _reconnectDelay); } + /// + /// Handles a graceful end of stream. The stream is torn down and left down for the + /// reconcile tick to reopen — deliberately NOT reopened inline, so a site that keeps + /// completing streams immediately can never spin this actor into a hot reconnect loop. + /// The retry budget is neither spent nor reset here (completion is not a fault), and the + /// endpoint is not flipped: the node that just closed a stream cleanly is healthy. + /// + private void HandleGrpcCompleted() + { + // The stream is gone, so its armed stability timer must not later "recover" a + // budget that its successor has since spent. + Timers.Cancel(StabilityTimerKey); + + _log.Info("Site-alarm gRPC stream for {0} completed gracefully (server end of stream); " + + "reopening on the next reconcile tick", _siteIdentifier); + + _openInFlight = false; + var wasLive = !_streamDown; + _streamDown = true; + // The reopen owes one re-seed, run when it actually connects. + _seedOnConnect = true; + CleanupGrpc(); + + // A completed stream is NOT live: without this the cache kept reporting live for up + // to a full reconcile interval after the site closed the feed (WP2.3 residual). + if (wasLive && _seeded) + Publish(); + } + private void CleanupGrpc() { _grpcCts?.Cancel(); @@ -612,6 +980,14 @@ internal sealed record SeedFailed(Exception Exception, bool IsInitial); /// Internal: periodic reconcile tick (and the re-seed kicked after a reconnect). internal sealed record RunReconcile; +/// Internal: backoff-scheduled retry of a fan-out that failed as a whole. +internal sealed record RetrySeed; + +/// Internal: the site accepted the site-wide alarm subscription (response headers +/// received), stamped with its stream generation so a late connect from a cancelled stream +/// is ignored. This — not the reconnect attempt — is what earns a re-seed. +internal sealed record GrpcAlarmStreamConnected(int Generation); + /// Internal: coalesced-publish tick — flush the dirty cache to viewers once (N6). internal sealed record PublishCoalesced; @@ -619,6 +995,11 @@ internal sealed record PublishCoalesced; /// generation it came from so a late error from a cancelled stream can be ignored (N7.2). internal sealed record GrpcAlarmStreamError(Exception Exception, int Generation); +/// Internal: the site-alarm gRPC stream ended gracefully (server status OK — max +/// stream lifetime or site shutdown), stamped with its stream generation so a late completion +/// from a cancelled stream can be ignored. +internal sealed record GrpcAlarmStreamCompleted(int Generation); + /// Internal: reconnect the site-alarm gRPC stream (flip node). internal sealed record ReconnectAlarmStream; diff --git a/src/ZB.MOM.WW.ScadaBridge.Communication/CentralControlGrpc/CentralControl.cs b/src/ZB.MOM.WW.ScadaBridge.Communication/CentralControlGrpc/CentralControl.cs index e8675699..f40a9138 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Communication/CentralControlGrpc/CentralControl.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Communication/CentralControlGrpc/CentralControl.cs @@ -114,33 +114,34 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { "c0R0bzoCOAEaQgogU3RvcmVBbmRGb3J3YXJkQnVmZmVyRGVwdGhzRW50cnkS", "CwoDa2V5GAEgASgJEg0KBXZhbHVlGAIgASgFOgI4ASJjChZTaXRlSGVhbHRo", "UmVwb3J0QWNrRHRvEg8KB3NpdGVfaWQYASABKAkSFwoPc2VxdWVuY2VfbnVt", - "YmVyGAIgASgDEhAKCGFjY2VwdGVkGAMgASgIEg0KBWVycm9yGAQgASgJIngK", - "DEhlYXJ0YmVhdER0bxIPCgdzaXRlX2lkGAEgASgJEhUKDW5vZGVfaG9zdG5h", - "bWUYAiABKAkSEQoJaXNfYWN0aXZlGAMgASgIEi0KCXRpbWVzdGFtcBgEIAEo", - "CzIaLmdvb2dsZS5wcm90b2J1Zi5UaW1lc3RhbXAqvQEKFENvbm5lY3Rpb25I", - "ZWFsdGhFbnVtEiEKHUNPTk5FQ1RJT05fSEVBTFRIX1VOU1BFQ0lGSUVEEAAS", - "HwobQ09OTkVDVElPTl9IRUFMVEhfQ09OTkVDVEVEEAESIgoeQ09OTkVDVElP", - "Tl9IRUFMVEhfRElTQ09OTkVDVEVEEAISIAocQ09OTkVDVElPTl9IRUFMVEhf", - "Q09OTkVDVElORxADEhsKF0NPTk5FQ1RJT05fSEVBTFRIX0VSUk9SEAQyoQYK", - "FUNlbnRyYWxDb250cm9sU2VydmljZRKDAQoSU3VibWl0Tm90aWZpY2F0aW9u", - "EjQuc2NhZGFicmlkZ2UuY2VudHJhbGNvbnRyb2wudjEuTm90aWZpY2F0aW9u", - "U3VibWl0RHRvGjcuc2NhZGFicmlkZ2UuY2VudHJhbGNvbnRyb2wudjEuTm90", - "aWZpY2F0aW9uU3VibWl0QWNrRHRvEpIBChdRdWVyeU5vdGlmaWNhdGlvblN0", - "YXR1cxI5LnNjYWRhYnJpZGdlLmNlbnRyYWxjb250cm9sLnYxLk5vdGlmaWNh", - "dGlvblN0YXR1c1F1ZXJ5RHRvGjwuc2NhZGFicmlkZ2UuY2VudHJhbGNvbnRy", - "b2wudjEuTm90aWZpY2F0aW9uU3RhdHVzUmVzcG9uc2VEdG8SRwoRSW5nZXN0", - "QXVkaXRFdmVudHMSGy5zaXRlc3RyZWFtLkF1ZGl0RXZlbnRCYXRjaBoVLnNp", - "dGVzdHJlYW0uSW5nZXN0QWNrElAKFUluZ2VzdENhY2hlZFRlbGVtZXRyeRIg", - "LnNpdGVzdHJlYW0uQ2FjaGVkVGVsZW1ldHJ5QmF0Y2gaFS5zaXRlc3RyZWFt", - "LkluZ2VzdEFjaxKAAQoNUmVjb25jaWxlU2l0ZRI2LnNjYWRhYnJpZGdlLmNl", - "bnRyYWxjb250cm9sLnYxLlJlY29uY2lsZVNpdGVSZXF1ZXN0RHRvGjcuc2Nh", - "ZGFicmlkZ2UuY2VudHJhbGNvbnRyb2wudjEuUmVjb25jaWxlU2l0ZVJlc3Bv", - "bnNlRHRvEn0KEFJlcG9ydFNpdGVIZWFsdGgSMi5zY2FkYWJyaWRnZS5jZW50", - "cmFsY29udHJvbC52MS5TaXRlSGVhbHRoUmVwb3J0RHRvGjUuc2NhZGFicmlk", - "Z2UuY2VudHJhbGNvbnRyb2wudjEuU2l0ZUhlYWx0aFJlcG9ydEFja0R0bxJQ", - "CglIZWFydGJlYXQSKy5zY2FkYWJyaWRnZS5jZW50cmFsY29udHJvbC52MS5I", - "ZWFydGJlYXREdG8aFi5nb29nbGUucHJvdG9idWYuRW1wdHlCK6oCKFpCLk1P", - "TS5XVy5TY2FkYUJyaWRnZS5Db21tdW5pY2F0aW9uLkdycGNiBnByb3RvMw==")); + "YmVyGAIgASgDEhAKCGFjY2VwdGVkGAMgASgIEg0KBWVycm9yGAQgASgJIosB", + "CgxIZWFydGJlYXREdG8SDwoHc2l0ZV9pZBgBIAEoCRIVCg1ub2RlX2hvc3Ru", + "YW1lGAIgASgJEhEKCWlzX2FjdGl2ZRgDIAEoCBItCgl0aW1lc3RhbXAYBCAB", + "KAsyGi5nb29nbGUucHJvdG9idWYuVGltZXN0YW1wEhEKCXN5bnRoZXRpYxgF", + "IAEoCCq9AQoUQ29ubmVjdGlvbkhlYWx0aEVudW0SIQodQ09OTkVDVElPTl9I", + "RUFMVEhfVU5TUEVDSUZJRUQQABIfChtDT05ORUNUSU9OX0hFQUxUSF9DT05O", + "RUNURUQQARIiCh5DT05ORUNUSU9OX0hFQUxUSF9ESVNDT05ORUNURUQQAhIg", + "ChxDT05ORUNUSU9OX0hFQUxUSF9DT05ORUNUSU5HEAMSGwoXQ09OTkVDVElP", + "Tl9IRUFMVEhfRVJST1IQBDKhBgoVQ2VudHJhbENvbnRyb2xTZXJ2aWNlEoMB", + "ChJTdWJtaXROb3RpZmljYXRpb24SNC5zY2FkYWJyaWRnZS5jZW50cmFsY29u", + "dHJvbC52MS5Ob3RpZmljYXRpb25TdWJtaXREdG8aNy5zY2FkYWJyaWRnZS5j", + "ZW50cmFsY29udHJvbC52MS5Ob3RpZmljYXRpb25TdWJtaXRBY2tEdG8SkgEK", + "F1F1ZXJ5Tm90aWZpY2F0aW9uU3RhdHVzEjkuc2NhZGFicmlkZ2UuY2VudHJh", + "bGNvbnRyb2wudjEuTm90aWZpY2F0aW9uU3RhdHVzUXVlcnlEdG8aPC5zY2Fk", + "YWJyaWRnZS5jZW50cmFsY29udHJvbC52MS5Ob3RpZmljYXRpb25TdGF0dXNS", + "ZXNwb25zZUR0bxJHChFJbmdlc3RBdWRpdEV2ZW50cxIbLnNpdGVzdHJlYW0u", + "QXVkaXRFdmVudEJhdGNoGhUuc2l0ZXN0cmVhbS5Jbmdlc3RBY2sSUAoVSW5n", + "ZXN0Q2FjaGVkVGVsZW1ldHJ5EiAuc2l0ZXN0cmVhbS5DYWNoZWRUZWxlbWV0", + "cnlCYXRjaBoVLnNpdGVzdHJlYW0uSW5nZXN0QWNrEoABCg1SZWNvbmNpbGVT", + "aXRlEjYuc2NhZGFicmlkZ2UuY2VudHJhbGNvbnRyb2wudjEuUmVjb25jaWxl", + "U2l0ZVJlcXVlc3REdG8aNy5zY2FkYWJyaWRnZS5jZW50cmFsY29udHJvbC52", + "MS5SZWNvbmNpbGVTaXRlUmVzcG9uc2VEdG8SfQoQUmVwb3J0U2l0ZUhlYWx0", + "aBIyLnNjYWRhYnJpZGdlLmNlbnRyYWxjb250cm9sLnYxLlNpdGVIZWFsdGhS", + "ZXBvcnREdG8aNS5zY2FkYWJyaWRnZS5jZW50cmFsY29udHJvbC52MS5TaXRl", + "SGVhbHRoUmVwb3J0QWNrRHRvElAKCUhlYXJ0YmVhdBIrLnNjYWRhYnJpZGdl", + "LmNlbnRyYWxjb250cm9sLnYxLkhlYXJ0YmVhdER0bxoWLmdvb2dsZS5wcm90", + "b2J1Zi5FbXB0eUIrqgIoWkIuTU9NLldXLlNjYWRhQnJpZGdlLkNvbW11bmlj", + "YXRpb24uR3JwY2IGcHJvdG8z")); descriptor = pbr::FileDescriptor.FromGeneratedCode(descriptorData, new pbr::FileDescriptor[] { global::Google.Protobuf.WellKnownTypes.EmptyReflection.Descriptor, global::Google.Protobuf.WellKnownTypes.TimestampReflection.Descriptor, global::Google.Protobuf.WellKnownTypes.WrappersReflection.Descriptor, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor, }, new pbr::GeneratedClrTypeInfo(new[] {typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ConnectionHealthEnum), }, null, new pbr::GeneratedClrTypeInfo[] { @@ -160,7 +161,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NodeStatusListDto), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NodeStatusListDto.Parser, new[]{ "Nodes" }, null, null, null, null), new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportDto), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportDto.Parser, new[]{ "SiteId", "SequenceNumber", "ReportTimestamp", "DataConnectionStatuses", "TagResolutionCounts", "ScriptErrorCount", "AlarmEvaluationErrorCount", "StoreAndForwardBufferDepths", "DeadLetterCount", "DeployedInstanceCount", "EnabledInstanceCount", "DisabledInstanceCount", "NodeRole", "NodeHostname", "DataConnectionEndpoints", "DataConnectionTagQuality", "ParkedMessageCount", "ClusterNodes", "SiteAuditWriteFailures", "AuditRedactionFailure", "SiteAuditBacklog", "SiteEventLogWriteFailures", "OldestParkedMessageAgeSeconds", "ScriptQueueDepth", "ScriptBusyThreads", "ScriptOldestBusyAgeSeconds", "LocalDbReplicationConnected", "LocalDbOplogBacklog" }, null, null, null, new pbr::GeneratedClrTypeInfo[] { null, null, null, }), new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportAckDto), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportAckDto.Parser, new[]{ "SiteId", "SequenceNumber", "Accepted", "Error" }, null, null, null, null), - new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.HeartbeatDto), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.HeartbeatDto.Parser, new[]{ "SiteId", "NodeHostname", "IsActive", "Timestamp" }, null, null, null, null) + new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.HeartbeatDto), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.HeartbeatDto.Parser, new[]{ "SiteId", "NodeHostname", "IsActive", "Timestamp", "Synthetic" }, null, null, null, null) })); } #endregion @@ -5826,6 +5827,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { nodeHostname_ = other.nodeHostname_; isActive_ = other.isActive_; timestamp_ = other.timestamp_ != null ? other.timestamp_.Clone() : null; + synthetic_ = other.synthetic_; _unknownFields = pb::UnknownFieldSet.Clone(other._unknownFields); } @@ -5883,6 +5885,27 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { } } + /// Field number for the "synthetic" field. + public const int SyntheticFieldNumber = 5; + private bool synthetic_; + /// + /// Additive (field 5): true marks a SYNTHETIC heartbeat that carries no liveness + /// meaning — today only CentralChannelProvider's failback probe, which reuses this + /// RPC purely to test whether the preferred central endpoint answers again. Central + /// must skip ALL liveness/health bookkeeping for such a heartbeat: it does not come + /// from a site node's heartbeat timer, and treating it as one would keep a site + /// looking alive on the health dashboard from the probe alone. proto3 defaults it to + /// false, so a pre-existing site's heartbeat is a real one, as before. + /// + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public bool Synthetic { + get { return synthetic_; } + set { + synthetic_ = value; + } + } + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] public override bool Equals(object other) { @@ -5902,6 +5925,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { if (NodeHostname != other.NodeHostname) return false; if (IsActive != other.IsActive) return false; if (!object.Equals(Timestamp, other.Timestamp)) return false; + if (Synthetic != other.Synthetic) return false; return Equals(_unknownFields, other._unknownFields); } @@ -5913,6 +5937,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { if (NodeHostname.Length != 0) hash ^= NodeHostname.GetHashCode(); if (IsActive != false) hash ^= IsActive.GetHashCode(); if (timestamp_ != null) hash ^= Timestamp.GetHashCode(); + if (Synthetic != false) hash ^= Synthetic.GetHashCode(); if (_unknownFields != null) { hash ^= _unknownFields.GetHashCode(); } @@ -5947,6 +5972,10 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { output.WriteRawTag(34); output.WriteMessage(Timestamp); } + if (Synthetic != false) { + output.WriteRawTag(40); + output.WriteBool(Synthetic); + } if (_unknownFields != null) { _unknownFields.WriteTo(output); } @@ -5973,6 +6002,10 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { output.WriteRawTag(34); output.WriteMessage(Timestamp); } + if (Synthetic != false) { + output.WriteRawTag(40); + output.WriteBool(Synthetic); + } if (_unknownFields != null) { _unknownFields.WriteTo(ref output); } @@ -5995,6 +6028,9 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { if (timestamp_ != null) { size += 1 + pb::CodedOutputStream.ComputeMessageSize(Timestamp); } + if (Synthetic != false) { + size += 1 + 1; + } if (_unknownFields != null) { size += _unknownFields.CalculateSize(); } @@ -6022,6 +6058,9 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { } Timestamp.MergeFrom(other.Timestamp); } + if (other.Synthetic != false) { + Synthetic = other.Synthetic; + } _unknownFields = pb::UnknownFieldSet.MergeFrom(_unknownFields, other._unknownFields); } @@ -6060,6 +6099,10 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { input.ReadMessage(Timestamp); break; } + case 40: { + Synthetic = input.ReadBool(); + break; + } } } #endif @@ -6098,6 +6141,10 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { input.ReadMessage(Timestamp); break; } + case 40: { + Synthetic = input.ReadBool(); + break; + } } } } diff --git a/src/ZB.MOM.WW.ScadaBridge.Communication/CommunicationOptions.cs b/src/ZB.MOM.WW.ScadaBridge.Communication/CommunicationOptions.cs index 3f91d824..b8ff2a20 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Communication/CommunicationOptions.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Communication/CommunicationOptions.cs @@ -47,6 +47,36 @@ public class CommunicationOptions /// public TimeSpan NotificationForwardTimeout { get; set; } = TimeSpan.FromSeconds(30); + /// + /// Audit Log: timeout for the SITE-side Ask that forwards one audit-telemetry batch + /// (IngestAuditEventsCommand / IngestCachedTelemetryCommand) through the site's + /// SiteCommunicationActor and awaits central's ack. Deliberately the LONGEST rung of the + /// ingest timeout ladder. + /// + /// + /// + /// The ladder is strictly monotonic, outermost first: + /// AuditForwardTimeout (35 s, this option) > + /// SiteStreamGrpcServer.AuditIngestAskTimeout (30 s — both the gRPC call deadline and + /// central's own Ask of the ingest singleton) > + /// AuditLogIngestActor.IngestBudget (20 s) > + /// AuditLogIngestActor.IngestSqlCommandTimeout (15 s). + /// + /// + /// Strictness matters: it used to reuse (30 s), the + /// same value as the rung below it, so a slow-but-succeeding central write could be acked to a + /// caller whose Ask had already expired — the drain loop would treat the batch as unsent and + /// re-ship it. Central dedups on EventId, so the duplicate was harmless but the retry + /// traffic and the "stalled" telemetry signal were not. A timeout here is still transient: the + /// rows stay Pending and drain on the next tick. + /// + /// + /// enforces the outermost inequality — this value + /// must be strictly greater than the 30 s gRPC/central Ask rung. + /// + /// + public TimeSpan AuditForwardTimeout { get; set; } = TimeSpan.FromSeconds(35); + /// /// Preshared key authenticating this node's gRPC control plane — the site↔central /// boundary. On a site node this is the key its inbound gate @@ -107,6 +137,21 @@ public class CommunicationOptions /// Maximum number of concurrent gRPC streaming subscriptions per site node. public int GrpcMaxConcurrentStreams { get; set; } = 100; + /// + /// Send-channel capacity for a per-instance Debug View stream (SubscribeInstance). + /// Lossy by design: the Debug View is a diagnostic surface and drops its oldest events + /// under backpressure rather than stalling the site's event hub. + /// + public int GrpcInstanceStreamChannelCapacity { get; set; } = 1000; + + /// + /// Send-channel capacity for the site-wide alarm stream (SubscribeSite). Larger + /// than the Debug View's (WP2.3): that feed backs the operator Alarm Summary, where a + /// silently dropped transition is a missed alarm rather than a missed diagnostic frame, + /// and an alarm burst arriving during a WAN stall must survive the stall. + /// + public int GrpcSiteAlarmStreamChannelCapacity { get; set; } = 20_000; + /// Akka.Remote transport heartbeat interval. public TimeSpan TransportHeartbeatInterval { get; set; } = TimeSpan.FromSeconds(5); @@ -185,4 +230,12 @@ public class CommunicationOptions /// reconcile publishes are always immediate. Default 250 ms. /// public TimeSpan LiveAlarmCachePublishCoalesce { get; set; } = TimeSpan.FromMilliseconds(250); + + /// + /// Random jitter added to each per-site reconcile tick, as a fraction of + /// (WP2.3). Without it every aggregator + /// started by one central failover fans its whole-site snapshot out on the same 60s + /// boundary forever. Zero disables jitter. + /// + public double LiveAlarmCacheReconcileJitterFraction { get; set; } = 0.2; } diff --git a/src/ZB.MOM.WW.ScadaBridge.Communication/CommunicationOptionsValidator.cs b/src/ZB.MOM.WW.ScadaBridge.Communication/CommunicationOptionsValidator.cs index 00f70888..7bee6e27 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Communication/CommunicationOptionsValidator.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Communication/CommunicationOptionsValidator.cs @@ -39,6 +39,15 @@ public sealed class CommunicationOptionsValidator : OptionsValidatorBase TimeSpan.Zero, $"ScadaBridge:Communication:NotificationForwardTimeout must be a positive duration (was {options.NotificationForwardTimeout})."); + // Outermost rung of the audit-ingest timeout ladder. It must be strictly greater than the + // gRPC deadline / central Ask beneath it, or a slow-but-succeeding central write gets acked + // to a caller that already gave up and re-sent the batch. See CommunicationOptions. + builder.RequireThat( + options.AuditForwardTimeout > Grpc.SiteStreamGrpcServer.AuditIngestAskTimeout, + $"ScadaBridge:Communication:AuditForwardTimeout must be strictly greater than the " + + $"central audit-ingest Ask timeout ({Grpc.SiteStreamGrpcServer.AuditIngestAskTimeout}) " + + $"so the ingest timeout ladder stays monotonic (was {options.AuditForwardTimeout})."); + builder.RequireThat(options.GrpcKeepAlivePingDelay > TimeSpan.Zero, $"ScadaBridge:Communication:GrpcKeepAlivePingDelay must be a positive duration (was {options.GrpcKeepAlivePingDelay})."); diff --git a/src/ZB.MOM.WW.ScadaBridge.Communication/DebugStreamService.cs b/src/ZB.MOM.WW.ScadaBridge.Communication/DebugStreamService.cs index 454ab8d3..b467efce 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Communication/DebugStreamService.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Communication/DebugStreamService.cs @@ -13,14 +13,34 @@ namespace ZB.MOM.WW.ScadaBridge.Communication; /// Manages debug stream sessions by creating DebugStreamBridgeActors that persist /// as subscribers on the site side. Both the Blazor debug view and the SignalR hub /// use this service to start/stop streams. +/// +/// Consumer keepalive. This service is the session registry, and holding a session +/// here IS what "a consumer is attached" means: the Blazor debug view and the SignalR hub +/// both release their session () on dispose/disconnect. A single +/// shared timer therefore Tells to every registered +/// bridge actor, which is the only thing that renews each actor's orphan window. A bridge +/// actor that outlives its registration — the leak the orphan net exists for — stops +/// receiving keepalives and self-terminates. The keepalive is independent of stream traffic +/// by design: neither a chatty nor a silent site can influence the orphan decision. +/// /// -public class DebugStreamService +public class DebugStreamService : IDisposable { + /// + /// Cadence of the shared consumer keepalive. Comfortably shorter than + /// so a few missed ticks (GC + /// pause, thread-pool starvation) can never orphan a live session. Settable for tests. + /// + internal static TimeSpan KeepaliveInterval { get; set; } = TimeSpan.FromSeconds(30); + private readonly CommunicationService _communicationService; private readonly IServiceProvider _serviceProvider; private readonly SiteStreamGrpcClientFactory _grpcClientFactory; private readonly ILogger _logger; private readonly ConcurrentDictionary _sessions = new(); + private readonly object _keepaliveLock = new(); + private Timer? _keepaliveTimer; + private bool _disposed; private ActorSystem? _actorSystem; /// @@ -135,6 +155,7 @@ public class DebugStreamService var bridgeActor = system.ActorOf(props, $"debug-stream-{sessionId}"); _sessions[sessionId] = bridgeActor; + EnsureKeepaliveTimer(); // Wait for the initial snapshot (with timeout) using var timeoutCts = CancellationTokenSource.CreateLinkedTokenSource(ct); @@ -184,6 +205,73 @@ public class DebugStreamService _logger.LogInformation("Debug stream {SessionId} stopped", sessionId); } } + + /// + /// Sends one keepalive round to every attached session. Exposed (internal) so tests can + /// drive the keepalive deterministically instead of waiting on the timer. + /// + internal void SendConsumerKeepalives() + { + foreach (var session in _sessions) + { + session.Value.Tell(new DebugStreamConsumerAlive()); + } + } + + /// + /// Starts the shared keepalive timer on first use. One timer for the whole service (not + /// one per session): the payload is a Tell per attached session, so a single periodic + /// callback is the cheapest shape that covers every consumer surface. + /// + private void EnsureKeepaliveTimer() + { + if (_keepaliveTimer is not null) return; + lock (_keepaliveLock) + { + if (_keepaliveTimer is not null || _disposed) return; + _keepaliveTimer = new Timer( + _ => + { + try + { + SendConsumerKeepalives(); + } + catch (Exception ex) + { + // A keepalive round must never take the timer down — a thrown Tell + // would silently stop renewing EVERY session's orphan window. + _logger.LogWarning(ex, "Debug stream consumer keepalive round failed"); + } + }, + null, + KeepaliveInterval, + KeepaliveInterval); + } + } + + /// Stops the keepalive timer. + public void Dispose() + { + Dispose(true); + GC.SuppressFinalize(this); + } + + /// Stops the keepalive timer. + /// True when called from . + protected virtual void Dispose(bool disposing) + { + if (_disposed) return; + lock (_keepaliveLock) + { + if (_disposed) return; + _disposed = true; + if (disposing) + { + _keepaliveTimer?.Dispose(); + _keepaliveTimer = null; + } + } + } } public record DebugStreamSession(string SessionId, DebugViewSnapshot InitialSnapshot); diff --git a/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/CentralChannelProvider.cs b/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/CentralChannelProvider.cs index 9b8f96fd..ff407971 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/CentralChannelProvider.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/CentralChannelProvider.cs @@ -38,6 +38,15 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc; /// public sealed class CentralChannelProvider : IDisposable { + /// + /// Hostname stamped on the failback probe's heartbeat, so a central-side log line names it + /// for what it is. It is a LABEL only — the machine-readable contract is + /// + /// (HeartbeatDto.synthetic, proto field 5), which central keys its skip on. Never + /// filter on this string. + /// + public const string SyntheticProbeHostname = "failback-probe"; + private static readonly TimeSpan DefaultBackoffBase = TimeSpan.FromSeconds(1); private static readonly TimeSpan DefaultBackoffCap = TimeSpan.FromSeconds(60); private static readonly TimeSpan DefaultProbeDeadline = TimeSpan.FromSeconds(5); @@ -202,9 +211,16 @@ public sealed class CentralChannelProvider : IDisposable new HeartbeatDto { SiteId = _siteId, - NodeHostname = "failback-probe", + NodeHostname = SyntheticProbeHostname, IsActive = false, Timestamp = Timestamp.FromDateTimeOffset(DateTimeOffset.UtcNow), + // Explicit contract, not a hostname convention: central skips ALL + // liveness/health bookkeeping for a synthetic heartbeat. Without it the + // probe would stamp the site's last-seen time on the central health + // aggregator every backoff tick, so a site whose real heartbeats had + // stopped could still look alive purely because its transport layer was + // probing for failback. + Synthetic = true, }, deadline: DateTime.UtcNow.Add(_probeDeadline)).ConfigureAwait(false); diff --git a/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/CentralControlDtoMapper.cs b/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/CentralControlDtoMapper.cs index 2393cff1..d675845b 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/CentralControlDtoMapper.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/CentralControlDtoMapper.cs @@ -695,6 +695,7 @@ public static class CentralControlDtoMapper NodeHostname = msg.NodeHostname, IsActive = msg.IsActive, Timestamp = Timestamp.FromDateTimeOffset(msg.Timestamp), + Synthetic = msg.Synthetic, }; } @@ -709,7 +710,8 @@ public static class CentralControlDtoMapper SiteId: dto.SiteId, NodeHostname: dto.NodeHostname, IsActive: dto.IsActive, - Timestamp: dto.Timestamp.ToDateTimeOffset()); + Timestamp: dto.Timestamp.ToDateTimeOffset(), + Synthetic: dto.Synthetic); } /// Projects a onto its wire enum. diff --git a/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/CentralControlGrpcService.cs b/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/CentralControlGrpcService.cs index e0b821bf..bf74635d 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/CentralControlGrpcService.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/CentralControlGrpcService.cs @@ -33,13 +33,23 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc; /// host's Akka bootstrap, not by the container. /// /// +/// The two audit-ingest RPCs are the exception: they Ask the ingest singleton DIRECTLY. +/// CentralCommunicationActor used to relay them — an extra hop that re-Asked the +/// same audit-log-ingest proxy with the same 30 s +/// , so the inner Ask expired at the same +/// instant as the outer one and could only add latency plus a +/// repackaging. The proxy now arrives through +/// , exactly as has always +/// taken it, and the relay handlers were deleted. +/// +/// /// Fault semantics deliberately differ from 's ingest /// RPCs. That server answers a failed audit ingest with an EMPTY IngestAck; this one /// fails the call with a non-OK status. Both leave the site's rows Pending for the next /// drain, so the outcome is the same — but this service replaced the ClusterClient path, whose -/// documented behaviour was to propagate the fault (CentralCommunicationActor's -/// HandleIngestAuditEvents pipes a Status.Failure back), and preserving that keeps -/// a lost batch visible as a failure rather than as a successful call that acked nothing. +/// documented behaviour was to propagate the fault, and preserving that keeps a lost batch +/// visible as a failure rather than as a successful call that acked nothing. A proxy that has +/// not been handed over yet is NOT a fault and still answers with an empty ack. /// /// /// Status mapping, and why it is not uniform. A site transport may safely re-send a call @@ -66,6 +76,14 @@ public sealed class CentralControlGrpcService : CentralControlService.CentralCon // Kestrel's thread pool. private volatile IActorRef? _central; + // Central-side audit-ingest singleton proxy (audit-log-ingest). Handed over by the host + // AFTER the singleton starts, so it arrives strictly later than _central. The two ingest + // RPCs Ask THIS proxy directly rather than relaying through CentralCommunicationActor — + // see the class remarks. Null until registration completes; an ingest arriving before then + // gets an empty ack (leave the site's rows Pending, retry next drain), which is exactly + // what the removed relay replied in the same window. + private volatile IActorRef? _auditIngest; + /// /// Creates the service. This must remain the only public constructor — see /// for how the actor arrives, and the Host's @@ -105,9 +123,32 @@ public sealed class CentralControlGrpcService : CentralControlService.CentralCon _central = centralCommunicationActor; } + /// + /// Hands the central AuditLogIngestActor singleton proxy to the service so the two + /// ingest RPCs can Ask it DIRECTLY. Mirrors + /// — the site-stream server has always + /// dispatched straight to the singleton, and this service now matches it. + /// + /// + /// Removing the CentralCommunicationActor relay removes an actor hop, a second Ask, + /// and a second timeout from every audit batch: the relay's own Ask was armed with the same + /// 30 s as this service's, so the two + /// expired together and the inner one could only ever add latency and a + /// repackaging on the way out. + /// + /// The audit-log-ingest singleton proxy. + public void SetAuditIngestActor(IActorRef auditIngestProxy) + { + ArgumentNullException.ThrowIfNull(auditIngestProxy); + _auditIngest = auditIngestProxy; + } + /// Exposed for wiring assertions in tests. internal bool IsReady => _central is not null; + /// Exposed for wiring assertions in tests. + internal bool IsAuditIngestBound => _auditIngest is not null; + /// public override async Task SubmitNotification( NotificationSubmitDto request, ServerCallContext context) @@ -147,9 +188,16 @@ public sealed class CentralControlGrpcService : CentralControlService.CentralCon return new IngestAck(); } - var central = RequireReady(context); + // Node-readiness gate first (the actor system must be up), then the ingest proxy. + RequireReady(context); + var ingest = RequireAuditIngest(request.Events.Count, nameof(IngestAuditEvents)); + if (ingest is null) + { + return new IngestAck(); + } + var reply = await AskAsync( - central, + ingest, CentralControlDtoMapper.FromDto(request), SiteStreamGrpcServer.AuditIngestAskTimeout, context).ConfigureAwait(false); @@ -166,9 +214,15 @@ public sealed class CentralControlGrpcService : CentralControlService.CentralCon return new IngestAck(); } - var central = RequireReady(context); + RequireReady(context); + var ingest = RequireAuditIngest(request.Packets.Count, nameof(IngestCachedTelemetry)); + if (ingest is null) + { + return new IngestAck(); + } + var reply = await AskAsync( - central, + ingest, CentralControlDtoMapper.FromDto(request), SiteStreamGrpcServer.AuditIngestAskTimeout, context).ConfigureAwait(false); @@ -176,6 +230,30 @@ public sealed class CentralControlGrpcService : CentralControlService.CentralCon return CentralControlDtoMapper.ToIngestAck(reply.AcceptedEventIds); } + /// + /// Returns the audit-ingest singleton proxy, or null when the host has not handed it + /// over yet (it starts moments after ). A null answer means the caller + /// must reply with an EMPTY — nothing accepted, rows stay + /// Pending on the site and drain on the next tick. That is deliberately NOT + /// : readiness here is intentionally narrow (see + /// ), and this is the exact reply the removed + /// CentralCommunicationActor relay produced in the same window. + /// + private IActorRef? RequireAuditIngest(int itemCount, string rpc) + { + var ingest = _auditIngest; + if (ingest is not null) + { + return ingest; + } + + _logger.LogWarning( + "{Rpc} received {Count} item(s) before SetAuditIngestActor was called; returning an " + + "empty ack so the site keeps its rows Pending and retries.", + rpc, itemCount); + return null; + } + /// public override async Task ReconcileSite( ReconcileSiteRequestDto request, ServerCallContext context) diff --git a/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/GrpcSiteTransport.cs b/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/GrpcSiteTransport.cs index 33322fe3..7d3bffad 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/GrpcSiteTransport.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/GrpcSiteTransport.cs @@ -22,9 +22,10 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc; /// /// /// Per-call deadlines match today's Ask timeouts exactly — see . -/// Behaviour is otherwise unchanged from the Akka path: an unknown/unconfigured site is warned and -/// dropped (the caller's Ask times out), and a transport fault surfaces to the caller as a -/// , which the S&F/audit layers already treat as transient. +/// An unknown/unconfigured site now fails fast: it surfaces to the caller as a +/// immediately (WP1.6) instead of being warned-and-dropped to let the +/// caller's Ask burn its full timeout. A transport fault takes the same +/// path, which the S&F/audit layers already treat as transient. /// /// /// Cross-node retry is the channel provider's job and happens only on @@ -79,13 +80,21 @@ public sealed class GrpcSiteTransport : ISiteCommandTransport replyTo.Tell(reply, ActorRefs.NoSender); } } - catch (SiteChannelUnavailableException) + catch (SiteChannelUnavailableException ex) { - // Parity with the removed Akka "no ClusterClient for site" path: warn and drop, so the caller's - // Ask times out. Central never buffers. + // WP1.6: a known-dead send (no configured channel for this site) used to warn-and-drop, + // leaving the caller's Ask to burn its full timeout for a failure we already know about. + // Fail fast instead — same Status.Failure completion path a transport fault takes below — + // so S&F/audit treat it as transient immediately rather than tens of seconds later. + // Central still never buffers for an unreachable site; only the completion timing changed. _logger.LogWarning( - "No gRPC channel for site {SiteId}; dropping {Message} (caller's Ask will time out)", + "No gRPC channel for site {SiteId}; failing {Message} immediately instead of dropping it", envelope.SiteId, envelope.Message.GetType().Name); + + if (!fireAndForget && !replyTo.IsNobody()) + { + replyTo.Tell(new Status.Failure(ex), ActorRefs.NoSender); + } } catch (Exception ex) { diff --git a/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteCommandDtoMapper.cs b/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteCommandDtoMapper.cs index 6bbd6a73..97508ba8 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteCommandDtoMapper.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteCommandDtoMapper.cs @@ -1620,7 +1620,10 @@ public static class SiteCommandDtoMapper return new DebugSnapshotRequestDto { InstanceUniqueName = request.InstanceUniqueName, - CorrelationId = request.CorrelationId + CorrelationId = request.CorrelationId, + // Additive field 3 — proto3 false default means an older site that does + // not know the field simply returns the full snapshot (correct, just fatter). + AlarmsOnly = request.AlarmsOnly }; } @@ -1631,7 +1634,7 @@ public static class SiteCommandDtoMapper { ArgumentNullException.ThrowIfNull(dto); - return new DebugSnapshotRequest(dto.InstanceUniqueName, dto.CorrelationId); + return new DebugSnapshotRequest(dto.InstanceUniqueName, dto.CorrelationId, dto.AlarmsOnly); } /// Projects a onto the wire. diff --git a/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SitePairChannelProvider.cs b/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SitePairChannelProvider.cs index f2cf359e..b8128f9f 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SitePairChannelProvider.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SitePairChannelProvider.cs @@ -8,9 +8,11 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc; /// /// Raised when a site has no usable gRPC channel — an unknown site, or a site with neither -/// GrpcNodeAAddress nor GrpcNodeBAddress configured. The gRPC transport treats this -/// the way the removed Akka path treated "no ClusterClient for site": warn and drop, so the caller's Ask -/// times out (central never buffers). +/// GrpcNodeAAddress nor GrpcNodeBAddress configured. +/// treats this as a known-dead send: it fails the caller's Ask fast with an +/// Akka.Actor.Status.Failure wrapping this exception (WP1.6) instead of warning and dropping +/// the message, so the caller does not burn its full Ask timeout on a failure already known at send +/// time (central still never buffers). /// public sealed class SiteChannelUnavailableException(string siteId) : Exception($"No gRPC channel is configured for site '{siteId}'.") diff --git a/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcClient.cs b/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcClient.cs index 52584b40..2789cabb 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcClient.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcClient.cs @@ -167,13 +167,19 @@ public class SiteStreamGrpcClient : IAsyncDisposable, IDisposable /// /// Opens a server-streaming subscription for a specific instance. /// This is a long-running async method; the caller launches it as a background task. - /// The callback delivers domain events, and - /// lets the caller handle reconnection. + /// The callback delivers domain events, + /// lets the caller handle reconnection after a fault, and + /// reports a graceful end of stream. /// /// Unique identifier for this subscription. /// Unique name of the instance to subscribe to. /// Callback invoked for each domain event received from the stream. /// Callback invoked when the subscription encounters an error. + /// + /// Callback invoked when the server ended the RPC with OK — the site's max stream + /// lifetime elapsing or a graceful site shutdown. Mutually exclusive with + /// ; see . + /// /// Cancellation token to stop the subscription. /// A task that represents the asynchronous operation. public virtual async Task SubscribeAsync( @@ -181,6 +187,7 @@ public class SiteStreamGrpcClient : IAsyncDisposable, IDisposable string instanceUniqueName, Action onEvent, Action onError, + Action onCompleted, CancellationToken ct) { if (_client is null) @@ -195,30 +202,18 @@ public class SiteStreamGrpcClient : IAsyncDisposable, IDisposable InstanceUniqueName = instanceUniqueName }; - try - { - using var call = _client.SubscribeInstance(request, cancellationToken: cts.Token); - - await foreach (var evt in call.ResponseStream.ReadAllAsync(cts.Token)) + await ConsumeStreamAsync( + correlationId, + cts, + () => _client.SubscribeInstance(request, cancellationToken: cts.Token), + evt => { var domainEvent = ConvertToDomainEvent(evt); if (domainEvent != null) onEvent(domainEvent); - } - } - catch (RpcException ex) when (ex.StatusCode == StatusCode.Cancelled) - { - // Normal cancellation — not an error - } - catch (Exception ex) - { - onError(ex); - } - finally - { - // Remove only our own entry -- a racing reconnect may already own the slot. - RemoveSubscription(correlationId, cts); - } + }, + onError, + onCompleted); } /// @@ -244,13 +239,28 @@ public class SiteStreamGrpcClient : IAsyncDisposable, IDisposable /// Unique identifier for this subscription. /// Callback invoked for each alarm delta received from the site-wide stream. /// Callback invoked when the subscription encounters an error. + /// + /// Callback invoked when the server ended the RPC with OK — the site's max stream + /// lifetime elapsing or a graceful site shutdown. Mutually exclusive with + /// ; see . + /// /// Cancellation token to stop the subscription. + /// + /// Optional callback invoked once when the site has ACCEPTED the subscription — response + /// headers received (the site writes them as soon as its relay actor is subscribed, so no + /// event can be missed after this point), or the first event received if the peer defers + /// its headers. A header timeout is NOT treated as connected. The per-site aggregator uses + /// this to run exactly one re-seed per successful (re)connect instead of one per reconnect + /// attempt. Never invoked more than once per call, and never after . + /// /// A task that represents the asynchronous operation. public virtual async Task SubscribeSiteAsync( string correlationId, Action onAlarmEvent, Action onError, - CancellationToken ct) + Action onCompleted, + CancellationToken ct, + Action? onConnected = null) { if (_client is null) throw new InvalidOperationException("Cannot subscribe on a test-only client."); @@ -263,20 +273,109 @@ public class SiteStreamGrpcClient : IAsyncDisposable, IDisposable CorrelationId = correlationId }; - try - { - using var call = _client.SubscribeSite(request, cancellationToken: cts.Token); - - await foreach (var evt in call.ResponseStream.ReadAllAsync(cts.Token)) + await ConsumeStreamAsync( + correlationId, + cts, + () => _client.SubscribeSite(request, cancellationToken: cts.Token), + evt => { // Site-wide stream is alarm-only by contract; defensively ignore anything else. if (ConvertToAlarmEvent(evt) is { } alarm) onAlarmEvent(alarm); - } - } - catch (RpcException ex) when (ex.StatusCode == StatusCode.Cancelled) + }, + onError, + onCompleted, + onConnected); + } + + /// + /// Drains a server-streaming call to its end, classifying the outcome into exactly one + /// of three terminations: a fault (), a graceful server-side + /// end of stream (), or our own cancellation (neither). + /// + /// The graceful case is the one that matters operationally: the site server caps every + /// stream at CommunicationOptions.GrpcMaxStreamLifetime (4 h) and, when that + /// elapses, ends the RPC with status OK. The client-side await foreach then simply + /// finishes, so before onCompleted existed the consuming actor observed nothing at + /// all and the stream stayed silently dead until the process restarted. + /// + /// Internal so the completion/fault classification can be unit-tested with a fake + /// and no live channel. + /// + /// Unique identifier for this subscription. + /// The subscription's linked token source (owned by the caller). + /// + /// Opens the server-streaming call. Invoked inside the guarded region so a throw while + /// opening is reported through like any other stream fault. + /// + /// Invoked per wire event. + /// Invoked once if the stream faulted. + /// Invoked once if the server ended the stream with OK. + /// + /// Optional; invoked AT MOST ONCE when the site has demonstrably accepted the + /// subscription — either the server's response headers arrived (bounded by + /// ) or, for a peer that defers headers until its + /// first message, the first event was received. A header timeout alone is never + /// reported as connected: that is also exactly what an unreachable site looks like. + /// + /// A task that completes when the stream has ended and its outcome been reported. + internal async Task ConsumeStreamAsync( + string correlationId, + CancellationTokenSource cts, + Func> openCall, + Action onEvent, + Action onError, + Action onCompleted, + Action? onConnected = null) + { + var completedGracefully = false; + var connectedReported = false; + + // Fires onConnected AT MOST ONCE, from whichever proof of a live peer arrives + // first: the response headers, or (for a peer that defers headers until its first + // message) the first event. A header TIMEOUT is deliberately NOT such a proof — + // see AwaitHeadersAsync. + void ReportConnected() { - // Normal cancellation — not an error + if (connectedReported || onConnected is null) return; + connectedReported = true; + onConnected(); + } + + try + { + using (var call = openCall()) + { + if (onConnected is not null && await AwaitHeadersAsync(call, cts.Token).ConfigureAwait(false)) + { + ReportConnected(); + } + + await foreach (var evt in call.ResponseStream.ReadAllAsync(cts.Token)) + { + // Fallback connected signal: an event can only come from a peer that + // accepted the subscription, so it proves what the headers would have. + // Raised BEFORE the event is delivered so the consumer sees + // connected-then-event ordering. + ReportConnected(); + onEvent(evt); + } + } + + completedGracefully = true; + } + catch (RpcException ex) when (ex.StatusCode == StatusCode.Cancelled && cts.IsCancellationRequested) + { + // OUR OWN cancellation (Unsubscribe / reconnect / channel teardown we asked + // for) — not an error. The IsCancellationRequested guard matters: a Cancelled + // status can also originate at the PEER or from a channel disposed underneath + // us, and swallowing THAT fired none of onError/onCompleted/onConnected, so + // the consuming actor kept a dead stream marked live forever. Foreign + // Cancelled now falls through to the onError path below. + } + catch (OperationCanceledException) when (cts.IsCancellationRequested) + { + // Our own Unsubscribe/reconnect cancelled the read — not an error either. } catch (Exception ex) { @@ -287,6 +386,56 @@ public class SiteStreamGrpcClient : IAsyncDisposable, IDisposable // Remove only our own entry -- a racing reconnect may already own the slot. RemoveSubscription(correlationId, cts); } + + // Raised outside the try so a throwing callback is not misreported as a stream + // fault. Suppressed when our own token was cancelled as the stream ended: that is + // a teardown, and the caller has already moved on to a newer stream. + if (completedGracefully && !cts.IsCancellationRequested) + onCompleted(); + } + + /// + /// How long to wait for response headers before giving up on them as the connected + /// signal and falling back to the first received event. A peer that only flushes headers + /// with its first message would otherwise hold the connected signal — and with it the + /// aggregator's re-seed — for as long as the site happens to be quiet. + /// + internal static TimeSpan ConnectedHeaderTimeout { get; set; } = TimeSpan.FromSeconds(10); + + /// + /// Awaits the call's response headers, bounded by . + /// Returns when the headers arrived (the site accepted the + /// subscription) and on timeout. A fault propagates (the caller + /// reports it through onError like any other stream fault); on timeout the + /// abandoned headers task is observed so a later fault on it can never surface as an + /// unobserved task exception. + /// + /// A timeout must NOT be reported as connected: an unreachable/wedged site produces + /// exactly that shape, and calling onConnected for it made the aggregator clear + /// _streamDown, consume its pending re-seed and fan a full snapshot out at a site + /// that never answered. The caller instead treats the FIRST RECEIVED EVENT as the + /// fallback connected signal — real proof of a live peer, and the quiet-site case the + /// timeout was added for is covered by the reconcile backstop. + /// + /// + private static async Task AwaitHeadersAsync( + AsyncServerStreamingCall call, CancellationToken ct) + { + var headers = call.ResponseHeadersAsync; + try + { + await headers.WaitAsync(ConnectedHeaderTimeout, ct).ConfigureAwait(false); + return true; + } + catch (TimeoutException) + { + _ = headers.ContinueWith( + t => _ = t.Exception, + CancellationToken.None, + TaskContinuationOptions.OnlyOnFaulted | TaskContinuationOptions.ExecuteSynchronously, + TaskScheduler.Default); + return false; + } } /// diff --git a/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcServer.cs b/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcServer.cs index 0fb89c4a..48178262 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcServer.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcServer.cs @@ -27,6 +27,8 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase private readonly ConcurrentDictionary _activeStreams = new(); private readonly int _maxConcurrentStreams; private readonly TimeSpan _maxStreamLifetime; + private readonly int _instanceChannelCapacity; + private readonly int _siteAlarmChannelCapacity; private volatile bool _ready; // Flipped by CancelAllStreams() when the host enters // CoordinatedShutdown so SubscribeInstance refuses new streams with @@ -72,10 +74,17 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase ISiteStreamSubscriber streamSubscriber, ILogger logger, int maxConcurrentStreams = 100) - : this(streamSubscriber, logger, maxConcurrentStreams, TimeSpan.FromHours(4)) + : this(streamSubscriber, logger, maxConcurrentStreams, TimeSpan.FromHours(4), + DefaultInstanceChannelCapacity, DefaultSiteAlarmChannelCapacity) { } + /// Fallback Debug View send-channel capacity when no options are bound. + internal const int DefaultInstanceChannelCapacity = 1000; + + /// Fallback site-wide alarm send-channel capacity when no options are bound. + internal const int DefaultSiteAlarmChannelCapacity = 20_000; + /// /// DI constructor — binds /// and so the documented @@ -91,7 +100,9 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase IOptions options) : this(streamSubscriber, logger, options.Value.GrpcMaxConcurrentStreams, - options.Value.GrpcMaxStreamLifetime) + options.Value.GrpcMaxStreamLifetime, + options.Value.GrpcInstanceStreamChannelCapacity, + options.Value.GrpcSiteAlarmStreamChannelCapacity) { } @@ -99,12 +110,16 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase ISiteStreamSubscriber streamSubscriber, ILogger logger, int maxConcurrentStreams, - TimeSpan maxStreamLifetime) + TimeSpan maxStreamLifetime, + int instanceChannelCapacity, + int siteAlarmChannelCapacity) { _streamSubscriber = streamSubscriber; _logger = logger; _maxConcurrentStreams = maxConcurrentStreams; _maxStreamLifetime = maxStreamLifetime; + _instanceChannelCapacity = Math.Max(1, instanceChannelCapacity); + _siteAlarmChannelCapacity = Math.Max(1, siteAlarmChannelCapacity); } /// @@ -209,6 +224,21 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase /// Effective per-stream session lifetime. Exposed for tests. internal TimeSpan MaxStreamLifetime => _maxStreamLifetime; + /// Effective Debug View send-channel capacity. Exposed for tests. + internal int InstanceChannelCapacity => _instanceChannelCapacity; + + /// Effective site-wide alarm send-channel capacity. Exposed for tests. + internal int SiteAlarmChannelCapacity => _siteAlarmChannelCapacity; + + /// + /// Total events evicted from stream send channels on this node since start (both stream + /// kinds). Exported as scadabridge.site.stream.events_dropped; exposed here so a + /// test — and an operator via diagnostics — can read the raw count. + /// + public long DroppedStreamEventCount => Interlocked.Read(ref _droppedStreamEvents); + + private long _droppedStreamEvents; + /// public override Task SubscribeInstance( InstanceStreamRequest request, @@ -219,7 +249,9 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase responseStream, context, relay => _streamSubscriber.Subscribe(request.InstanceUniqueName, relay), - request.InstanceUniqueName); + request.InstanceUniqueName, + _instanceChannelCapacity, + streamKind: "instance"); /// public override Task SubscribeSite( @@ -234,7 +266,12 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase // already drops IsConfiguredPlaceholder rows and maps only the // enriched AlarmStateUpdate, so it is reused unchanged. _streamSubscriber.SubscribeSiteAlarms, - "site-wide alarms"); + "site-wide alarms", + // Its OWN, much larger channel (WP2.3): sharing the Debug View's 1000-slot + // DropOldest meant an alarm burst during a WAN stall silently evicted operator- + // visible transitions to make room for diagnostics traffic. + _siteAlarmChannelCapacity, + streamKind: "site-alarms"); /// /// Shared streaming pipeline behind and @@ -250,12 +287,16 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase /// The server call context (carries the client cancellation token). /// Subscribes the relay actor to the hub, returning a subscription id. /// Human-readable subscription description for logging. + /// Send-channel capacity for this stream kind. + /// Telemetry tag for this stream kind (instance/site-alarms). private async Task RunSubscriptionStreamAsync( string correlationId, IServerStreamWriter responseStream, ServerCallContext context, Func subscribe, - string description) + string description, + int channelCapacity, + string streamKind) { if (!_ready) throw new RpcException(new GrpcStatus(StatusCode.Unavailable, "Server not ready")); @@ -302,16 +343,20 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase long dropped = 0; var channel = Channel.CreateBounded( - new BoundedChannelOptions(1000) { FullMode = BoundedChannelFullMode.DropOldest }, + new BoundedChannelOptions(channelCapacity) { FullMode = BoundedChannelFullMode.DropOldest }, _ => { // Lossy-under-backpressure is the debug-view spec; make the real - // loss visible: first eviction + every 500th thereafter. + // loss visible: first eviction + every 500th thereafter, plus a + // per-kind counter on the node (WP2.3). var n = Interlocked.Increment(ref dropped); + Interlocked.Increment(ref _droppedStreamEvents); + ScadaBridgeTelemetry.RecordSiteStreamEventDropped(streamKind); if (n == 1 || n % 500 == 0) _logger.LogWarning( - "Debug stream {CorrelationId} backpressure: {Dropped} oldest event(s) evicted so far", - correlationId, n); + "Stream {CorrelationId} ({StreamKind}) backpressure: {Dropped} oldest event(s) evicted so far " + + "of a {Capacity}-slot channel", + correlationId, streamKind, n, channelCapacity); }); var actorSeq = Interlocked.Increment(ref _actorCounter); @@ -348,6 +393,25 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase "Stream {CorrelationId} started for {Description} (subscription {SubscriptionId})", correlationId, description, subscriptionId); + // Flush response headers NOW, while the relay is attached and before the first event + // (WP2.3). This is the client's "the site accepted my subscription" signal — the + // per-site aggregator hangs its once-per-successful-reconnect re-seed off it. Without + // an explicit flush, ASP.NET Core defers headers to the first written message, so a + // quiet site would never report connected. Best-effort: a client that vanished between + // Subscribe and here fails the write, and the read loop below handles the teardown. + try + { + await context.WriteResponseHeadersAsync(Metadata.Empty); + } + catch (Exception ex) + { + // Deliberately catch-all: an unsent header must never leak the relay actor and + // the _activeStreams entry, which the finally below owns. Any real transport + // failure resurfaces on the read/write loop immediately after. + _logger.LogDebug(ex, + "Could not flush response headers for stream {CorrelationId}; continuing.", correlationId); + } + // Telemetry follow-on: the connection is now fully established (Subscribe // succeeded, so no leak via the catch above). Count it up here and balance // it in the finally below so the scadabridge.site.connection.up gauge is @@ -519,17 +583,46 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase ? DateTime.SpecifyKind(request.SinceUtc.ToDateTime(), DateTimeKind.Utc) : DateTime.MinValue; + // Composite-keyset cursor (WP2.3), mirroring PullSiteCalls: proto3 + // defaults an unset string to "", which the queue treats as "no cursor" + // (legacy inclusive >= behaviour), so an older central is unaffected. + var afterId = string.IsNullOrEmpty(request.AfterId) ? null : request.AfterId; + + // AT-LEAST-ONCE (WP2.3). The incoming cursor is central's receipt for + // everything at or before it — that is the ONLY proof the site accepts. + // Rows are flipped to Reconciled here, at the START of the NEXT pull, + // instead of right after the previous response was projected: a fault + // between the response leaving the site and central committing it used + // to lose the rows outright, because they were already Reconciled and + // ReadPendingSinceAsync would never serve them again. The flip runs + // before the read so the rows it retires do not consume this batch's + // budget. Best-effort — a failure here only costs a re-ship, which + // central dedups on EventId. + if (since > DateTime.MinValue) + { + try + { + await queue.MarkReconciledUpToAsync(since, afterId, context.CancellationToken); + } + catch (Exception ex) + { + _logger.LogWarning(ex, + "MarkReconciledUpToAsync failed for cursor since={Since} afterId={AfterId}; rows stay pending for the next pull.", + since, afterId); + } + } + IReadOnlyList events; try { events = await queue.ReadPendingSinceAsync( - since, request.BatchSize, context.CancellationToken); + since, request.BatchSize, afterId, context.CancellationToken); } catch (Exception ex) { _logger.LogError(ex, - "ReadPendingSinceAsync failed for since={Since} batch={Batch}; returning empty response.", - since, request.BatchSize); + "ReadPendingSinceAsync failed for since={Since} batch={Batch} afterId={AfterId}; returning empty response.", + since, request.BatchSize, afterId); return new PullAuditEventsResponse(); } @@ -537,7 +630,8 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase { // batch_size saturated → tell central to issue a follow-up pull // with an advanced cursor. The site doesn't compute the cursor — - // central walks it forward from the last returned OccurredAtUtc. + // central walks it forward from the last returned OccurredAtUtc + // (plus, once it sets after_id, that row's EventId). MoreAvailable = events.Count >= request.BatchSize, }; foreach (var evt in events) @@ -545,31 +639,6 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase response.Events.Add(AuditEventDtoMapper.ToDto(evt)); } - // Flip to Reconciled AFTER projecting the response so a fault below the - // try/catch (mid-response, mid-flip) leaves the rows in Pending/Forwarded - // and central pulls them again next cycle. The flip itself is - // best-effort — its failure is a warning, not a fault, because central - // will dedup on EventId on the next pull. - var ids = new List(events.Count); - foreach (var evt in events) - { - ids.Add(evt.EventId); - } - - if (ids.Count > 0) - { - try - { - await queue.MarkReconciledAsync(ids, context.CancellationToken); - } - catch (Exception ex) - { - _logger.LogWarning(ex, - "MarkReconciledAsync failed after PullAuditEvents response of {Count} rows; rows stay Pending for retry.", - ids.Count); - } - } - return response; } diff --git a/src/ZB.MOM.WW.ScadaBridge.Communication/ISiteAlarmLiveCache.cs b/src/ZB.MOM.WW.ScadaBridge.Communication/ISiteAlarmLiveCache.cs index 5d094a47..09042357 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Communication/ISiteAlarmLiveCache.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Communication/ISiteAlarmLiveCache.cs @@ -62,4 +62,29 @@ public interface ISiteAlarmLiveCache /// The numeric site id. /// true while a living aggregator has seeded and published; reverts to false if the aggregator terminates (R2 N6) — a frozen snapshot is never reported as live. bool IsLive(int siteId); + + /// + /// The Enabled instances on the site that failed to answer the aggregator's most recent + /// snapshot fan-out (seed or reconcile) — i.e. that reported InstanceNotFound, timed + /// out, or faulted. Deterministically ordered by instance name, and empty when the site has + /// no aggregator or has not yet completed a fan-out. + /// + /// + /// + /// Published alongside the alarm snapshot on every fan-out, because the aggregator already + /// knows it: the seed/reconcile is the SAME per-instance debug-snapshot fan-out the Alarm + /// Summary poll used to run a second time purely to compute this list. Exposing it here lets + /// the UI-side poll be skipped entirely while is true, which is the + /// whole point — one fan-out per site per reconcile window, not two. + /// + /// + /// It is refreshed at the aggregator's fan-out cadence (seed, then each reconcile), NOT on + /// every stream delta: an instance going silent is precisely what the stream cannot tell you. + /// Callers must therefore treat it as reconcile-fresh, not delta-fresh — which matches the + /// poll's own 15 s freshness, so nothing on screen changes. + /// + /// + /// The numeric site id. + /// The not-reporting instance unique names (possibly empty), never null. + IReadOnlyList GetNotReportingInstances(int siteId); } diff --git a/src/ZB.MOM.WW.ScadaBridge.Communication/Protos/central_control.proto b/src/ZB.MOM.WW.ScadaBridge.Communication/Protos/central_control.proto index 97d4daf1..7faa1170 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Communication/Protos/central_control.proto +++ b/src/ZB.MOM.WW.ScadaBridge.Communication/Protos/central_control.proto @@ -244,4 +244,12 @@ message HeartbeatDto { string node_hostname = 2; bool is_active = 3; google.protobuf.Timestamp timestamp = 4; + // Additive (field 5): true marks a SYNTHETIC heartbeat that carries no liveness + // meaning — today only CentralChannelProvider's failback probe, which reuses this + // RPC purely to test whether the preferred central endpoint answers again. Central + // must skip ALL liveness/health bookkeeping for such a heartbeat: it does not come + // from a site node's heartbeat timer, and treating it as one would keep a site + // looking alive on the health dashboard from the probe alone. proto3 defaults it to + // false, so a pre-existing site's heartbeat is a real one, as before. + bool synthetic = 5; } diff --git a/src/ZB.MOM.WW.ScadaBridge.Communication/Protos/site_command.proto b/src/ZB.MOM.WW.ScadaBridge.Communication/Protos/site_command.proto index 7f86b93f..8c6fa912 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Communication/Protos/site_command.proto +++ b/src/ZB.MOM.WW.ScadaBridge.Communication/Protos/site_command.proto @@ -553,6 +553,14 @@ message EventLogQueryResponseDto { message DebugSnapshotRequestDto { string instance_unique_name = 1; string correlation_id = 2; + // Alarms-only projection (WP2.3 wire efficiency): when true the site builds + // and returns ONLY the alarm rows of the debug snapshot, leaving + // attribute_values empty. Used by the central per-site live alarm cache, + // whose seed/reconcile fan-out discards every attribute row anyway — a full + // snapshot ships the whole attribute surface of every enabled instance once + // per reconcile for nothing. proto3 defaults it to false, so an older + // central that never sets it keeps the full-snapshot behaviour. Additive-only. + bool alarms_only = 3; } message SubscribeDebugViewRequestDto { diff --git a/src/ZB.MOM.WW.ScadaBridge.Communication/Protos/sitestream.proto b/src/ZB.MOM.WW.ScadaBridge.Communication/Protos/sitestream.proto index 27587337..850816da 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Communication/Protos/sitestream.proto +++ b/src/ZB.MOM.WW.ScadaBridge.Communication/Protos/sitestream.proto @@ -164,13 +164,26 @@ message CachedTelemetryBatch { repeated CachedTelemetryPacket packets = 1; } // Audit Log (#23) M6 reconciliation pull: central→site request for any // site-local AuditLog rows with OccurredAtUtc >= since_utc that have not yet -// been ingested centrally (ForwardState in {Pending, Forwarded}). The site -// flips returned rows to Reconciled after the response is on the wire. +// been ingested centrally (ForwardState in {Pending, Forwarded}). Rows are NOT +// flipped to Reconciled when they are served — only when a LATER pull's cursor +// proves central consumed them (see after_id), so a fault between the response +// leaving the site and central committing it re-serves the rows instead of +// silently losing them (at-least-once). // more_available signals batch_size was saturated so the caller knows to // issue a follow-up pull with an advanced since_utc cursor. message PullAuditEventsRequest { google.protobuf.Timestamp since_utc = 1; int32 batch_size = 2; + // Composite-keyset cursor (WP2.3), mirroring PullSiteCallsRequest.after_id: + // the EventId ("D" GUID form) of the last row central has already CONSUMED at + // since_utc. When set, the site returns only rows strictly after the composite + // (OccurredAtUtc, EventId) pair — un-pinning a batch that would otherwise stall + // when more than batch_size rows share one since_utc instant — AND treats the + // cursor as proof of receipt: everything at or before it is flipped to + // Reconciled. Empty (the proto3 string default) preserves the legacy inclusive + // >= behaviour, under which only rows strictly older than since_utc are proven + // received. Additive-only. + string after_id = 3; } message PullAuditEventsResponse { diff --git a/src/ZB.MOM.WW.ScadaBridge.Communication/SiteAlarmLiveCacheService.cs b/src/ZB.MOM.WW.ScadaBridge.Communication/SiteAlarmLiveCacheService.cs index c3fbb7f3..49d55bf6 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Communication/SiteAlarmLiveCacheService.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Communication/SiteAlarmLiveCacheService.cs @@ -30,6 +30,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication; public sealed class SiteAlarmLiveCacheService : ISiteAlarmLiveCache { private static readonly IReadOnlyList Empty = Array.Empty(); + private static readonly IReadOnlyList EmptyNames = Array.Empty(); private readonly IServiceProvider _serviceProvider; private readonly CommunicationService _communicationService; @@ -132,7 +133,14 @@ public sealed class SiteAlarmLiveCacheService : ISiteAlarmLiveCache public bool IsLive(int siteId) { lock (_lock) - return _sites.TryGetValue(siteId, out var entry) && entry.HasPublished; + return _sites.TryGetValue(siteId, out var entry) && entry.HasPublished && entry.StreamLive; + } + + /// + public IReadOnlyList GetNotReportingInstances(int siteId) + { + lock (_lock) + return _sites.TryGetValue(siteId, out var entry) ? entry.NotReporting : EmptyNames; } // ── Subscriber teardown ───────────────────────────────────────────────────── @@ -213,8 +221,8 @@ public sealed class SiteAlarmLiveCacheService : ISiteAlarmLiveCache Func>> seedFn = ct => FanOutSnapshotsAsync(entry.SiteId, ct); - Action> publish = - snapshot => OnPublish(entry, snapshot); + Action, bool> publish = + (snapshot, streamLive) => OnPublish(entry, snapshot, streamLive); lock (_lock) { @@ -236,7 +244,8 @@ public sealed class SiteAlarmLiveCacheService : ISiteAlarmLiveCache _options.LiveAlarmCacheReconcileInterval, _options.LiveAlarmCachePublishCoalesce, TimeSpan.FromSeconds(5), // reconnect delay — former ReconnectDelay static default - TimeSpan.FromSeconds(60))); // stability window — former StabilityWindow static default + TimeSpan.FromSeconds(60), // stability window — former StabilityWindow static default + _options.LiveAlarmCacheReconcileJitterFraction)); entry.Actor = system.ActorOf(props, $"site-alarm-aggregator-{entry.SiteId}-{Guid.NewGuid():N}"); @@ -331,6 +340,15 @@ public sealed class SiteAlarmLiveCacheService : ISiteAlarmLiveCache /// contributes no rows rather than /// failing the whole seed (mirrors AlarmSummaryService). Re-enumerates instances /// each call so a reconcile picks up enable/disable/deploy/delete drift. + /// + /// It also records WHICH instances failed to answer into + /// , published through + /// . That set was previously + /// computed and thrown away here, forcing the Alarm Summary page to run a second, identical + /// fan-out through AlarmSummaryService purely to rebuild it. Same classification as + /// that service (InstanceNotFound or any non-cancellation fault), same + /// ordinal-ignore-case ordering, so the page renders identically from either source. + /// /// private async Task> FanOutSnapshotsAsync(int siteId, CancellationToken ct) { @@ -342,7 +360,10 @@ public sealed class SiteAlarmLiveCacheService : ISiteAlarmLiveCache var siteRepo = scope.ServiceProvider.GetRequiredService(); var site = await siteRepo.GetSiteByIdAsync(siteId, ct); if (site is null) + { + PublishNotReporting(siteId, EmptyNames); return Empty; + } siteIdentifier = site.SiteIdentifier; var instanceRepo = scope.ServiceProvider.GetRequiredService(); @@ -354,34 +375,65 @@ public sealed class SiteAlarmLiveCacheService : ISiteAlarmLiveCache } if (enabledInstanceNames.Count == 0) + { + PublishNotReporting(siteId, EmptyNames); return Empty; + } var rows = new ConcurrentBag(); + var notReporting = new ConcurrentBag(); using var gate = new SemaphoreSlim( Math.Max(1, _options.LiveAlarmCacheSeedConcurrency), Math.Max(1, _options.LiveAlarmCacheSeedConcurrency)); var fetches = enabledInstanceNames.Select(name => - FetchInstanceSnapshotAsync(siteIdentifier, name, gate, rows, ct)); + FetchInstanceSnapshotAsync(siteIdentifier, name, gate, rows, notReporting, ct)); await Task.WhenAll(fetches); + PublishNotReporting( + siteId, + notReporting.OrderBy(n => n, StringComparer.OrdinalIgnoreCase).ToList()); + return rows.ToList(); } + /// + /// Stores the fan-out's not-reporting set on the site entry. Silently no-ops when the entry + /// has been torn down mid-fan-out (last viewer left), matching every other late-write path + /// in this service. + /// + private void PublishNotReporting(int siteId, IReadOnlyList notReporting) + { + lock (_lock) + { + if (_sites.TryGetValue(siteId, out var entry)) + entry.NotReporting = notReporting; + } + } + private async Task FetchInstanceSnapshotAsync( string siteIdentifier, string instanceUniqueName, SemaphoreSlim gate, ConcurrentBag rows, + ConcurrentBag notReporting, CancellationToken ct) { await gate.WaitAsync(ct); try { - var request = new DebugSnapshotRequest(instanceUniqueName, Guid.NewGuid().ToString("N")); + // Alarms-only (WP2.3): this fan-out discards snapshot.AttributeValues, so + // asking the site to build and ship them is pure wire waste. The flag is + // additive on the wire; a pre-WP2.3 site ignores it and returns the full + // snapshot, which this loop reads exactly as before. + var request = new DebugSnapshotRequest( + instanceUniqueName, Guid.NewGuid().ToString("N"), AlarmsOnly: true); var snapshot = await _communicationService.RequestDebugSnapshotAsync(siteIdentifier, request, ct); if (snapshot.InstanceNotFound) + { + notReporting.Add(instanceUniqueName); return; + } foreach (var alarm in snapshot.AlarmStates) rows.Add(alarm); @@ -392,7 +444,9 @@ public sealed class SiteAlarmLiveCacheService : ISiteAlarmLiveCache } catch { - // Degrade this one instance to "no rows" rather than failing the whole seed. + // Degrade this one instance to "no rows, not reporting" rather than failing the + // whole seed. Same rule as AlarmSummaryService.FetchInstanceAsync. + notReporting.Add(instanceUniqueName); } finally { @@ -402,7 +456,7 @@ public sealed class SiteAlarmLiveCacheService : ISiteAlarmLiveCache // ── Publish (actor thread → viewers) ──────────────────────────────────────── - private void OnPublish(SiteEntry entry, IReadOnlyList snapshot) + private void OnPublish(SiteEntry entry, IReadOnlyList snapshot, bool streamLive) { Subscription[] subscribers; lock (_lock) @@ -410,6 +464,10 @@ public sealed class SiteAlarmLiveCacheService : ISiteAlarmLiveCache // Store the fresh immutable snapshot (readers get it lock-free-ish via GetCurrentAlarms). entry.Current = snapshot; entry.HasPublished = true; + // Liveness is the STREAM's, not the cache's: a completed or given-up stream must + // stop reporting live between reopen ticks, or the page grafts a freezing snapshot + // over fresh poll data (WP2.3 carried residual). + entry.StreamLive = streamLive; subscribers = entry.Subscribers.ToArray(); } @@ -462,7 +520,11 @@ public sealed class SiteAlarmLiveCacheService : ISiteAlarmLiveCache entry.Actor = null; entry.HasPublished = false; + entry.StreamLive = false; entry.Current = Empty; + // A dead aggregator's last fan-out result is stale too — clear it so the page's + // poll fallback (now authoritative again) owns the not-reporting list outright. + entry.NotReporting = EmptyNames; if (entry.Subscribers.Count > 0 && !entry.Starting) { @@ -498,6 +560,20 @@ public sealed class SiteAlarmLiveCacheService : ISiteAlarmLiveCache /// True once the aggregator has seeded and published at least once. public bool HasPublished { get; set; } + /// + /// Enabled instances that failed to answer the most recent snapshot fan-out, ordered by + /// name. Refreshed on every seed/reconcile (see FanOutSnapshotsAsync) and cleared + /// when liveness resets, so a dead aggregator never leaves a stale list behind. + /// + public IReadOnlyList NotReporting { get; set; } = EmptyNames; + + /// + /// Liveness of the aggregator's site-wide gRPC stream as of the last publish. Both + /// this and must hold for IsLive: a published cache + /// behind a dead stream is stale, not live. + /// + public bool StreamLive { get; set; } + public Timer? LingerTimer { get; set; } public int LingerVersion { get; set; } diff --git a/src/ZB.MOM.WW.ScadaBridge.Communication/SiteCommandGrpc/SiteCommand.cs b/src/ZB.MOM.WW.ScadaBridge.Communication/SiteCommandGrpc/SiteCommand.cs index d8e45300..fda7176e 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Communication/SiteCommandGrpc/SiteCommand.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Communication/SiteCommandGrpc/SiteCommand.cs @@ -260,241 +260,242 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { "cmlkZ2Uuc2l0ZWNvbW1hbmQudjEuRXZlbnRMb2dFbnRyeUR0bxIaChJjb250", "aW51YXRpb25fdG9rZW4YBCABKAkSEAoIaGFzX21vcmUYBSABKAgSDwoHc3Vj", "Y2VzcxgGIAEoCBIVCg1lcnJvcl9tZXNzYWdlGAcgASgJEi0KCXRpbWVzdGFt", - "cBgIIAEoCzIaLmdvb2dsZS5wcm90b2J1Zi5UaW1lc3RhbXAiTwoXRGVidWdT", + "cBgIIAEoCzIaLmdvb2dsZS5wcm90b2J1Zi5UaW1lc3RhbXAiZAoXRGVidWdT", "bmFwc2hvdFJlcXVlc3REdG8SHAoUaW5zdGFuY2VfdW5pcXVlX25hbWUYASAB", - "KAkSFgoOY29ycmVsYXRpb25faWQYAiABKAkiVAocU3Vic2NyaWJlRGVidWdW", - "aWV3UmVxdWVzdER0bxIcChRpbnN0YW5jZV91bmlxdWVfbmFtZRgBIAEoCRIW", - "Cg5jb3JyZWxhdGlvbl9pZBgCIAEoCSJWCh5VbnN1YnNjcmliZURlYnVnVmll", - "d1JlcXVlc3REdG8SHAoUaW5zdGFuY2VfdW5pcXVlX25hbWUYASABKAkSFgoO", - "Y29ycmVsYXRpb25faWQYAiABKAkiHAoaVW5zdWJzY3JpYmVEZWJ1Z1ZpZXdB", - "Y2tEdG8i1AEKFkFsYXJtQ29uZGl0aW9uU3RhdGVEdG8SDgoGYWN0aXZlGAEg", - "ASgIEhQKDGFja25vd2xlZGdlZBgCIAEoCBItCgljb25maXJtZWQYAyABKAsy", - "Gi5nb29nbGUucHJvdG9idWYuQm9vbFZhbHVlEj8KBnNoZWx2ZRgEIAEoDjIv", - "LnNjYWRhYnJpZGdlLnNpdGVjb21tYW5kLnYxLkFsYXJtU2hlbHZlU3RhdGVE", - "dG8SEgoKc3VwcHJlc3NlZBgFIAEoCBIQCghzZXZlcml0eRgGIAEoBSLdAQoW", - "RGVidWdBdHRyaWJ1dGVWYWx1ZUR0bxIcChRpbnN0YW5jZV91bmlxdWVfbmFt", - "ZRgBIAEoCRIWCg5hdHRyaWJ1dGVfcGF0aBgCIAEoCRIWCg5hdHRyaWJ1dGVf", - "bmFtZRgDIAEoCRI1CgV2YWx1ZRgEIAEoCzImLnNjYWRhYnJpZGdlLnNpdGVj", - "b21tYW5kLnYxLkxvb3NlVmFsdWUSDwoHcXVhbGl0eRgFIAEoCRItCgl0aW1l", - "c3RhbXAYBiABKAsyGi5nb29nbGUucHJvdG9idWYuVGltZXN0YW1wIq8FChJE", - "ZWJ1Z0FsYXJtU3RhdGVEdG8SHAoUaW5zdGFuY2VfdW5pcXVlX25hbWUYASAB", - "KAkSEgoKYWxhcm1fbmFtZRgCIAEoCRI4CgVzdGF0ZRgDIAEoDjIpLnNjYWRh", - "YnJpZGdlLnNpdGVjb21tYW5kLnYxLkFsYXJtU3RhdGVEdG8SEAoIcHJpb3Jp", - "dHkYBCABKAUSLQoJdGltZXN0YW1wGAUgASgLMhouZ29vZ2xlLnByb3RvYnVm", - "LlRpbWVzdGFtcBI4CgVsZXZlbBgGIAEoDjIpLnNjYWRhYnJpZGdlLnNpdGVj", - "b21tYW5kLnYxLkFsYXJtTGV2ZWxEdG8SDwoHbWVzc2FnZRgHIAEoCRI2CgRr", - "aW5kGAggASgOMiguc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuQWxhcm1L", - "aW5kRHRvEkUKCWNvbmRpdGlvbhgJIAEoCzIyLnNjYWRhYnJpZGdlLnNpdGVj", - "b21tYW5kLnYxLkFsYXJtQ29uZGl0aW9uU3RhdGVEdG8SGAoQc291cmNlX3Jl", - "ZmVyZW5jZRgKIAEoCRIXCg9hbGFybV90eXBlX25hbWUYCyABKAkSEAoIY2F0", - "ZWdvcnkYDCABKAkSFQoNb3BlcmF0b3JfdXNlchgNIAEoCRIYChBvcGVyYXRv", - "cl9jb21tZW50GA4gASgJEjcKE29yaWdpbmFsX3JhaXNlX3RpbWUYDyABKAsy", - "Gi5nb29nbGUucHJvdG9idWYuVGltZXN0YW1wEhUKDWN1cnJlbnRfdmFsdWUY", - "ECABKAkSEwoLbGltaXRfdmFsdWUYESABKAkSJAocbmF0aXZlX3NvdXJjZV9j", - "YW5vbmljYWxfbmFtZRgSIAEoCRIhChlpc19jb25maWd1cmVkX3BsYWNlaG9s", - "ZGVyGBMgASgIIpwCChREZWJ1Z1ZpZXdTbmFwc2hvdER0bxIcChRpbnN0YW5j", - "ZV91bmlxdWVfbmFtZRgBIAEoCRJMChBhdHRyaWJ1dGVfdmFsdWVzGAIgAygL", - "MjIuc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuRGVidWdBdHRyaWJ1dGVW", - "YWx1ZUR0bxJECgxhbGFybV9zdGF0ZXMYAyADKAsyLi5zY2FkYWJyaWRnZS5z", - "aXRlY29tbWFuZC52MS5EZWJ1Z0FsYXJtU3RhdGVEdG8SNgoSc25hcHNob3Rf", - "dGltZXN0YW1wGAQgASgLMhouZ29vZ2xlLnByb3RvYnVmLlRpbWVzdGFtcBIa", - "ChJpbnN0YW5jZV9ub3RfZm91bmQYBSABKAgi8AIKDFF1ZXJ5UmVxdWVzdBJO", - "Cg9ldmVudF9sb2dfcXVlcnkYASABKAsyMy5zY2FkYWJyaWRnZS5zaXRlY29t", - "bWFuZC52MS5FdmVudExvZ1F1ZXJ5UmVxdWVzdER0b0gAEk0KDmRlYnVnX3Nu", - "YXBzaG90GAIgASgLMjMuc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuRGVi", - "dWdTbmFwc2hvdFJlcXVlc3REdG9IABJYChRzdWJzY3JpYmVfZGVidWdfdmll", - "dxgDIAEoCzI4LnNjYWRhYnJpZGdlLnNpdGVjb21tYW5kLnYxLlN1YnNjcmli", - "ZURlYnVnVmlld1JlcXVlc3REdG9IABJcChZ1bnN1YnNjcmliZV9kZWJ1Z192", - "aWV3GAQgASgLMjouc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuVW5zdWJz", - "Y3JpYmVEZWJ1Z1ZpZXdSZXF1ZXN0RHRvSABCCQoHY29tbWFuZCKRAgoKUXVl", - "cnlSZXBseRJPCg9ldmVudF9sb2dfcXVlcnkYASABKAsyNC5zY2FkYWJyaWRn", - "ZS5zaXRlY29tbWFuZC52MS5FdmVudExvZ1F1ZXJ5UmVzcG9uc2VEdG9IABJP", - "ChNkZWJ1Z192aWV3X3NuYXBzaG90GAIgASgLMjAuc2NhZGFicmlkZ2Uuc2l0", - "ZWNvbW1hbmQudjEuRGVidWdWaWV3U25hcHNob3REdG9IABJYChZ1bnN1YnNj", - "cmliZV9kZWJ1Z192aWV3GAMgASgLMjYuc2NhZGFicmlkZ2Uuc2l0ZWNvbW1h", - "bmQudjEuVW5zdWJzY3JpYmVEZWJ1Z1ZpZXdBY2tEdG9IAEIHCgVyZXBseSKe", - "AQocUGFya2VkTWVzc2FnZVF1ZXJ5UmVxdWVzdER0bxIWCg5jb3JyZWxhdGlv", - "bl9pZBgBIAEoCRIPCgdzaXRlX2lkGAIgASgJEhMKC3BhZ2VfbnVtYmVyGAMg", - "ASgFEhEKCXBhZ2Vfc2l6ZRgEIAEoBRItCgl0aW1lc3RhbXAYBSABKAsyGi5n", - "b29nbGUucHJvdG9idWYuVGltZXN0YW1wIvICChVQYXJrZWRNZXNzYWdlRW50", - "cnlEdG8SEgoKbWVzc2FnZV9pZBgBIAEoCRIVCg10YXJnZXRfc3lzdGVtGAIg", - "ASgJEhMKC21ldGhvZF9uYW1lGAMgASgJEhUKDWVycm9yX21lc3NhZ2UYBCAB", - "KAkSFQoNYXR0ZW1wdF9jb3VudBgFIAEoBRI2ChJvcmlnaW5hbF90aW1lc3Rh", - "bXAYBiABKAsyGi5nb29nbGUucHJvdG9idWYuVGltZXN0YW1wEjoKFmxhc3Rf", - "YXR0ZW1wdF90aW1lc3RhbXAYByABKAsyGi5nb29nbGUucHJvdG9idWYuVGlt", - "ZXN0YW1wEhQKDG1heF9hdHRlbXB0cxgIIAEoBRJICghjYXRlZ29yeRgJIAEo", - "DjI2LnNjYWRhYnJpZGdlLnNpdGVjb21tYW5kLnYxLlN0b3JlQW5kRm9yd2Fy", - "ZENhdGVnb3J5RHRvEhcKD29yaWdpbl9pbnN0YW5jZRgKIAEoCSKhAgodUGFy", - "a2VkTWVzc2FnZVF1ZXJ5UmVzcG9uc2VEdG8SFgoOY29ycmVsYXRpb25faWQY", - "ASABKAkSDwoHc2l0ZV9pZBgCIAEoCRJDCghtZXNzYWdlcxgDIAMoCzIxLnNj", - "YWRhYnJpZGdlLnNpdGVjb21tYW5kLnYxLlBhcmtlZE1lc3NhZ2VFbnRyeUR0", - "bxITCgt0b3RhbF9jb3VudBgEIAEoBRITCgtwYWdlX251bWJlchgFIAEoBRIR", - "CglwYWdlX3NpemUYBiABKAUSDwoHc3VjY2VzcxgHIAEoCBIVCg1lcnJvcl9t", - "ZXNzYWdlGAggASgJEi0KCXRpbWVzdGFtcBgJIAEoCzIaLmdvb2dsZS5wcm90", - "b2J1Zi5UaW1lc3RhbXAiigEKHFBhcmtlZE1lc3NhZ2VSZXRyeVJlcXVlc3RE", - "dG8SFgoOY29ycmVsYXRpb25faWQYASABKAkSDwoHc2l0ZV9pZBgCIAEoCRIS", - "CgptZXNzYWdlX2lkGAMgASgJEi0KCXRpbWVzdGFtcBgEIAEoCzIaLmdvb2ds", - "ZS5wcm90b2J1Zi5UaW1lc3RhbXAiXwodUGFya2VkTWVzc2FnZVJldHJ5UmVz", - "cG9uc2VEdG8SFgoOY29ycmVsYXRpb25faWQYASABKAkSDwoHc3VjY2VzcxgC", - "IAEoCBIVCg1lcnJvcl9tZXNzYWdlGAMgASgJIowBCh5QYXJrZWRNZXNzYWdl", - "RGlzY2FyZFJlcXVlc3REdG8SFgoOY29ycmVsYXRpb25faWQYASABKAkSDwoH", - "c2l0ZV9pZBgCIAEoCRISCgptZXNzYWdlX2lkGAMgASgJEi0KCXRpbWVzdGFt", - "cBgEIAEoCzIaLmdvb2dsZS5wcm90b2J1Zi5UaW1lc3RhbXAiYQofUGFya2Vk", - "TWVzc2FnZURpc2NhcmRSZXNwb25zZUR0bxIWCg5jb3JyZWxhdGlvbl9pZBgB", - "IAEoCRIPCgdzdWNjZXNzGAIgASgIEhUKDWVycm9yX21lc3NhZ2UYAyABKAki", - "TwoXUmV0cnlQYXJrZWRPcGVyYXRpb25EdG8SFgoOY29ycmVsYXRpb25faWQY", - "ASABKAkSHAoUdHJhY2tlZF9vcGVyYXRpb25faWQYAiABKAkiUQoZRGlzY2Fy", - "ZFBhcmtlZE9wZXJhdGlvbkR0bxIWCg5jb3JyZWxhdGlvbl9pZBgBIAEoCRIc", - "ChR0cmFja2VkX29wZXJhdGlvbl9pZBgCIAEoCSJdChtQYXJrZWRPcGVyYXRp", - "b25BY3Rpb25BY2tEdG8SFgoOY29ycmVsYXRpb25faWQYASABKAkSDwoHYXBw", - "bGllZBgCIAEoCBIVCg1lcnJvcl9tZXNzYWdlGAMgASgJIt4DCg1QYXJrZWRS", - "ZXF1ZXN0ElgKFHBhcmtlZF9tZXNzYWdlX3F1ZXJ5GAEgASgLMjguc2NhZGFi", - "cmlkZ2Uuc2l0ZWNvbW1hbmQudjEuUGFya2VkTWVzc2FnZVF1ZXJ5UmVxdWVz", - "dER0b0gAElgKFHBhcmtlZF9tZXNzYWdlX3JldHJ5GAIgASgLMjguc2NhZGFi", - "cmlkZ2Uuc2l0ZWNvbW1hbmQudjEuUGFya2VkTWVzc2FnZVJldHJ5UmVxdWVz", - "dER0b0gAElwKFnBhcmtlZF9tZXNzYWdlX2Rpc2NhcmQYAyABKAsyOi5zY2Fk", - "YWJyaWRnZS5zaXRlY29tbWFuZC52MS5QYXJrZWRNZXNzYWdlRGlzY2FyZFJl", - "cXVlc3REdG9IABJVChZyZXRyeV9wYXJrZWRfb3BlcmF0aW9uGAQgASgLMjMu", - "c2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuUmV0cnlQYXJrZWRPcGVyYXRp", - "b25EdG9IABJZChhkaXNjYXJkX3BhcmtlZF9vcGVyYXRpb24YBSABKAsyNS5z", - "Y2FkYWJyaWRnZS5zaXRlY29tbWFuZC52MS5EaXNjYXJkUGFya2VkT3BlcmF0", - "aW9uRHRvSABCCQoHY29tbWFuZCKHAwoLUGFya2VkUmVwbHkSWQoUcGFya2Vk", - "X21lc3NhZ2VfcXVlcnkYASABKAsyOS5zY2FkYWJyaWRnZS5zaXRlY29tbWFu", - "ZC52MS5QYXJrZWRNZXNzYWdlUXVlcnlSZXNwb25zZUR0b0gAElkKFHBhcmtl", - "ZF9tZXNzYWdlX3JldHJ5GAIgASgLMjkuc2NhZGFicmlkZ2Uuc2l0ZWNvbW1h", - "bmQudjEuUGFya2VkTWVzc2FnZVJldHJ5UmVzcG9uc2VEdG9IABJdChZwYXJr", - "ZWRfbWVzc2FnZV9kaXNjYXJkGAMgASgLMjsuc2NhZGFicmlkZ2Uuc2l0ZWNv", - "bW1hbmQudjEuUGFya2VkTWVzc2FnZURpc2NhcmRSZXNwb25zZUR0b0gAEloK", - "F3BhcmtlZF9vcGVyYXRpb25fYWN0aW9uGAQgASgLMjcuc2NhZGFicmlkZ2Uu", - "c2l0ZWNvbW1hbmQudjEuUGFya2VkT3BlcmF0aW9uQWN0aW9uQWNrRHRvSABC", - "BwoFcmVwbHki7QEKFVJvdXRlVG9DYWxsUmVxdWVzdER0bxIWCg5jb3JyZWxh", - "dGlvbl9pZBgBIAEoCRIcChRpbnN0YW5jZV91bmlxdWVfbmFtZRgCIAEoCRIT", - "CgtzY3JpcHRfbmFtZRgDIAEoCRI9CgpwYXJhbWV0ZXJzGAQgASgLMikuc2Nh", - "ZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuTG9vc2VWYWx1ZU1hcBItCgl0aW1l", - "c3RhbXAYBSABKAsyGi5nb29nbGUucHJvdG9idWYuVGltZXN0YW1wEhsKE3Bh", - "cmVudF9leGVjdXRpb25faWQYBiABKAkixQEKFlJvdXRlVG9DYWxsUmVzcG9u", - "c2VEdG8SFgoOY29ycmVsYXRpb25faWQYASABKAkSDwoHc3VjY2VzcxgCIAEo", - "CBI8CgxyZXR1cm5fdmFsdWUYAyABKAsyJi5zY2FkYWJyaWRnZS5zaXRlY29t", - "bWFuZC52MS5Mb29zZVZhbHVlEhUKDWVycm9yX21lc3NhZ2UYBCABKAkSLQoJ", - "dGltZXN0YW1wGAUgASgLMhouZ29vZ2xlLnByb3RvYnVmLlRpbWVzdGFtcCK7", - "AQoeUm91dGVUb0dldEF0dHJpYnV0ZXNSZXF1ZXN0RHRvEhYKDmNvcnJlbGF0", - "aW9uX2lkGAEgASgJEhwKFGluc3RhbmNlX3VuaXF1ZV9uYW1lGAIgASgJEhcK", - "D2F0dHJpYnV0ZV9uYW1lcxgDIAMoCRItCgl0aW1lc3RhbXAYBCABKAsyGi5n", - "b29nbGUucHJvdG9idWYuVGltZXN0YW1wEhsKE3BhcmVudF9leGVjdXRpb25f", - "aWQYBSABKAkiywEKH1JvdXRlVG9HZXRBdHRyaWJ1dGVzUmVzcG9uc2VEdG8S", - "FgoOY29ycmVsYXRpb25faWQYASABKAkSOQoGdmFsdWVzGAIgASgLMikuc2Nh", - "ZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuTG9vc2VWYWx1ZU1hcBIPCgdzdWNj", - "ZXNzGAMgASgIEhUKDWVycm9yX21lc3NhZ2UYBCABKAkSLQoJdGltZXN0YW1w", - "GAUgASgLMhouZ29vZ2xlLnByb3RvYnVmLlRpbWVzdGFtcCLFAgoeUm91dGVU", - "b1NldEF0dHJpYnV0ZXNSZXF1ZXN0RHRvEhYKDmNvcnJlbGF0aW9uX2lkGAEg", - "ASgJEhwKFGluc3RhbmNlX3VuaXF1ZV9uYW1lGAIgASgJEmkKEGF0dHJpYnV0", - "ZV92YWx1ZXMYAyADKAsyTy5zY2FkYWJyaWRnZS5zaXRlY29tbWFuZC52MS5S", - "b3V0ZVRvU2V0QXR0cmlidXRlc1JlcXVlc3REdG8uQXR0cmlidXRlVmFsdWVz", - "RW50cnkSLQoJdGltZXN0YW1wGAQgASgLMhouZ29vZ2xlLnByb3RvYnVmLlRp", - "bWVzdGFtcBIbChNwYXJlbnRfZXhlY3V0aW9uX2lkGAUgASgJGjYKFEF0dHJp", - "YnV0ZVZhbHVlc0VudHJ5EgsKA2tleRgBIAEoCRINCgV2YWx1ZRgCIAEoCToC", - "OAEikAEKH1JvdXRlVG9TZXRBdHRyaWJ1dGVzUmVzcG9uc2VEdG8SFgoOY29y", - "cmVsYXRpb25faWQYASABKAkSDwoHc3VjY2VzcxgCIAEoCBIVCg1lcnJvcl9t", - "ZXNzYWdlGAMgASgJEi0KCXRpbWVzdGFtcBgEIAEoCzIaLmdvb2dsZS5wcm90", - "b2J1Zi5UaW1lc3RhbXAiwwIKIVJvdXRlVG9XYWl0Rm9yQXR0cmlidXRlUmVx", - "dWVzdER0bxIWCg5jb3JyZWxhdGlvbl9pZBgBIAEoCRIcChRpbnN0YW5jZV91", - "bmlxdWVfbmFtZRgCIAEoCRIWCg5hdHRyaWJ1dGVfbmFtZRgDIAEoCRI6ChR0", - "YXJnZXRfdmFsdWVfZW5jb2RlZBgEIAEoCzIcLmdvb2dsZS5wcm90b2J1Zi5T", - "dHJpbmdWYWx1ZRIqCgd0aW1lb3V0GAUgASgLMhkuZ29vZ2xlLnByb3RvYnVm", - "LkR1cmF0aW9uEi0KCXRpbWVzdGFtcBgGIAEoCzIaLmdvb2dsZS5wcm90b2J1", - "Zi5UaW1lc3RhbXASGwoTcGFyZW50X2V4ZWN1dGlvbl9pZBgHIAEoCRIcChRy", - "ZXF1aXJlX2dvb2RfcXVhbGl0eRgIIAEoCCL/AQoiUm91dGVUb1dhaXRGb3JB", - "dHRyaWJ1dGVSZXNwb25zZUR0bxIWCg5jb3JyZWxhdGlvbl9pZBgBIAEoCRIP", - "CgdtYXRjaGVkGAIgASgIEjUKBXZhbHVlGAMgASgLMiYuc2NhZGFicmlkZ2Uu", - "c2l0ZWNvbW1hbmQudjEuTG9vc2VWYWx1ZRIPCgdxdWFsaXR5GAQgASgJEhEK", - "CXRpbWVkX291dBgFIAEoCBIPCgdzdWNjZXNzGAYgASgIEhUKDWVycm9yX21l", - "c3NhZ2UYByABKAkSLQoJdGltZXN0YW1wGAggASgLMhouZ29vZ2xlLnByb3Rv", - "YnVmLlRpbWVzdGFtcCKJAwoMUm91dGVSZXF1ZXN0EkoKDXJvdXRlX3RvX2Nh", - "bGwYASABKAsyMS5zY2FkYWJyaWRnZS5zaXRlY29tbWFuZC52MS5Sb3V0ZVRv", - "Q2FsbFJlcXVlc3REdG9IABJdChdyb3V0ZV90b19nZXRfYXR0cmlidXRlcxgC", - "IAEoCzI6LnNjYWRhYnJpZGdlLnNpdGVjb21tYW5kLnYxLlJvdXRlVG9HZXRB", - "dHRyaWJ1dGVzUmVxdWVzdER0b0gAEl0KF3JvdXRlX3RvX3NldF9hdHRyaWJ1", - "dGVzGAMgASgLMjouc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuUm91dGVU", - "b1NldEF0dHJpYnV0ZXNSZXF1ZXN0RHRvSAASZAobcm91dGVfdG9fd2FpdF9m", - "b3JfYXR0cmlidXRlGAQgASgLMj0uc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQu", - "djEuUm91dGVUb1dhaXRGb3JBdHRyaWJ1dGVSZXF1ZXN0RHRvSABCCQoHY29t", - "bWFuZCKJAwoKUm91dGVSZXBseRJLCg1yb3V0ZV90b19jYWxsGAEgASgLMjIu", - "c2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuUm91dGVUb0NhbGxSZXNwb25z", - "ZUR0b0gAEl4KF3JvdXRlX3RvX2dldF9hdHRyaWJ1dGVzGAIgASgLMjsuc2Nh", - "ZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuUm91dGVUb0dldEF0dHJpYnV0ZXNS", - "ZXNwb25zZUR0b0gAEl4KF3JvdXRlX3RvX3NldF9hdHRyaWJ1dGVzGAMgASgL", - "Mjsuc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuUm91dGVUb1NldEF0dHJp", - "YnV0ZXNSZXNwb25zZUR0b0gAEmUKG3JvdXRlX3RvX3dhaXRfZm9yX2F0dHJp", - "YnV0ZRgEIAEoCzI+LnNjYWRhYnJpZGdlLnNpdGVjb21tYW5kLnYxLlJvdXRl", - "VG9XYWl0Rm9yQXR0cmlidXRlUmVzcG9uc2VEdG9IAEIHCgVyZXBseSJBChZU", - "cmlnZ2VyU2l0ZUZhaWxvdmVyRHRvEhYKDmNvcnJlbGF0aW9uX2lkGAEgASgJ", - "Eg8KB3NpdGVfaWQYAiABKAkibQoSU2l0ZUZhaWxvdmVyQWNrRHRvEhYKDmNv", - "cnJlbGF0aW9uX2lkGAEgASgJEhAKCGFjY2VwdGVkGAIgASgIEhYKDnRhcmdl", - "dF9hZGRyZXNzGAMgASgJEhUKDWVycm9yX21lc3NhZ2UYBCABKAkqywEKE0Rl", - "cGxveW1lbnRTdGF0dXNEdG8SJQohREVQTE9ZTUVOVF9TVEFUVVNfRFRPX1VO", - "U1BFQ0lGSUVEEAASIQodREVQTE9ZTUVOVF9TVEFUVVNfRFRPX1BFTkRJTkcQ", - "ARIlCiFERVBMT1lNRU5UX1NUQVRVU19EVE9fSU5fUFJPR1JFU1MQAhIhCh1E", - "RVBMT1lNRU5UX1NUQVRVU19EVE9fU1VDQ0VTUxADEiAKHERFUExPWU1FTlRf", - "U1RBVFVTX0RUT19GQUlMRUQQBCrEAQoSQnJvd3NlTm9kZUNsYXNzRHRvEiUK", - "IUJST1dTRV9OT0RFX0NMQVNTX0RUT19VTlNQRUNJRklFRBAAEiAKHEJST1dT", - "RV9OT0RFX0NMQVNTX0RUT19PQkpFQ1QQARIiCh5CUk9XU0VfTk9ERV9DTEFT", - "U19EVE9fVkFSSUFCTEUQAhIgChxCUk9XU0VfTk9ERV9DTEFTU19EVE9fTUVU", - "SE9EEAMSHwobQlJPV1NFX05PREVfQ0xBU1NfRFRPX09USEVSEAQqoQIKFEJy", - "b3dzZUZhaWx1cmVLaW5kRHRvEicKI0JST1dTRV9GQUlMVVJFX0tJTkRfRFRP", - "X1VOU1BFQ0lGSUVEEAASMAosQlJPV1NFX0ZBSUxVUkVfS0lORF9EVE9fQ09O", - "TkVDVElPTl9OT1RfRk9VTkQQARI0CjBCUk9XU0VfRkFJTFVSRV9LSU5EX0RU", - "T19DT05ORUNUSU9OX05PVF9DT05ORUNURUQQAhIpCiVCUk9XU0VfRkFJTFVS", - "RV9LSU5EX0RUT19OT1RfQlJPV1NBQkxFEAMSIwofQlJPV1NFX0ZBSUxVUkVf", - "S0lORF9EVE9fVElNRU9VVBAEEigKJEJST1dTRV9GQUlMVVJFX0tJTkRfRFRP", - "X1NFUlZFUl9FUlJPUhAFKqoCChtSZWFkVGFnVmFsdWVzRmFpbHVyZUtpbmRE", - "dG8SMAosUkVBRF9UQUdfVkFMVUVTX0ZBSUxVUkVfS0lORF9EVE9fVU5TUEVD", - "SUZJRUQQABI5CjVSRUFEX1RBR19WQUxVRVNfRkFJTFVSRV9LSU5EX0RUT19D", - "T05ORUNUSU9OX05PVF9GT1VORBABEj0KOVJFQURfVEFHX1ZBTFVFU19GQUlM", - "VVJFX0tJTkRfRFRPX0NPTk5FQ1RJT05fTk9UX0NPTk5FQ1RFRBACEiwKKFJF", - "QURfVEFHX1ZBTFVFU19GQUlMVVJFX0tJTkRfRFRPX1RJTUVPVVQQAxIxCi1S", - "RUFEX1RBR19WQUxVRVNfRkFJTFVSRV9LSU5EX0RUT19TRVJWRVJfRVJST1IQ", - "BCqTAgoUVmVyaWZ5RmFpbHVyZUtpbmREdG8SJwojVkVSSUZZX0ZBSUxVUkVf", - "S0lORF9EVE9fVU5TUEVDSUZJRUQQABInCiNWRVJJRllfRkFJTFVSRV9LSU5E", - "X0RUT19VTlJFQUNIQUJMRRABEicKI1ZFUklGWV9GQUlMVVJFX0tJTkRfRFRP", - "X0FVVEhfRkFJTEVEEAISMQotVkVSSUZZX0ZBSUxVUkVfS0lORF9EVE9fVU5U", - "UlVTVEVEX0NFUlRJRklDQVRFEAMSIwofVkVSSUZZX0ZBSUxVUkVfS0lORF9E", - "VE9fVElNRU9VVBAEEigKJFZFUklGWV9GQUlMVVJFX0tJTkRfRFRPX1NFUlZF", - "Ul9FUlJPUhAFKuUBChpTdG9yZUFuZEZvcndhcmRDYXRlZ29yeUR0bxIuCipT", - "VE9SRV9BTkRfRk9SV0FSRF9DQVRFR09SWV9EVE9fVU5TUEVDSUZJRUQQABIy", - "Ci5TVE9SRV9BTkRfRk9SV0FSRF9DQVRFR09SWV9EVE9fRVhURVJOQUxfU1lT", - "VEVNEAESLworU1RPUkVfQU5EX0ZPUldBUkRfQ0FURUdPUllfRFRPX05PVElG", - "SUNBVElPThACEjIKLlNUT1JFX0FORF9GT1JXQVJEX0NBVEVHT1JZX0RUT19D", - "QUNIRURfREJfV1JJVEUQAypoCg1BbGFybVN0YXRlRHRvEh8KG0FMQVJNX1NU", - "QVRFX0RUT19VTlNQRUNJRklFRBAAEhoKFkFMQVJNX1NUQVRFX0RUT19BQ1RJ", - "VkUQARIaChZBTEFSTV9TVEFURV9EVE9fTk9STUFMEAIquQEKDUFsYXJtTGV2", - "ZWxEdG8SHwobQUxBUk1fTEVWRUxfRFRPX1VOU1BFQ0lGSUVEEAASGAoUQUxB", - "Uk1fTEVWRUxfRFRPX05PTkUQARIXChNBTEFSTV9MRVZFTF9EVE9fTE9XEAIS", - "GwoXQUxBUk1fTEVWRUxfRFRPX0xPV19MT1cQAxIYChRBTEFSTV9MRVZFTF9E", - "VE9fSElHSBAEEh0KGUFMQVJNX0xFVkVMX0RUT19ISUdIX0hJR0gQBSqSAQoM", - "QWxhcm1LaW5kRHRvEh4KGkFMQVJNX0tJTkRfRFRPX1VOU1BFQ0lGSUVEEAAS", - "GwoXQUxBUk1fS0lORF9EVE9fQ09NUFVURUQQARIgChxBTEFSTV9LSU5EX0RU", - "T19OQVRJVkVfT1BDX1VBEAISIwofQUxBUk1fS0lORF9EVE9fTkFUSVZFX01Y", - "X0FDQ0VTUxADKugBChNBbGFybVNoZWx2ZVN0YXRlRHRvEiYKIkFMQVJNX1NI", - "RUxWRV9TVEFURV9EVE9fVU5TUEVDSUZJRUQQABIkCiBBTEFSTV9TSEVMVkVf", - "U1RBVEVfRFRPX1VOU0hFTFZFRBABEisKJ0FMQVJNX1NIRUxWRV9TVEFURV9E", - "VE9fT05FX1NIT1RfU0hFTFZFRBACEigKJEFMQVJNX1NIRUxWRV9TVEFURV9E", - "VE9fVElNRURfU0hFTFZFRBADEiwKKEFMQVJNX1NIRUxWRV9TVEFURV9EVE9f", - "UEVSTUFORU5UX1NIRUxWRUQQBDKEBQoSU2l0ZUNvbW1hbmRTZXJ2aWNlEmwK", - "EEV4ZWN1dGVMaWZlY3ljbGUSLC5zY2FkYWJyaWRnZS5zaXRlY29tbWFuZC52", - "MS5MaWZlY3ljbGVSZXF1ZXN0Giouc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQu", - "djEuTGlmZWN5Y2xlUmVwbHkSYAoMRXhlY3V0ZU9wY1VhEiguc2NhZGFicmlk", - "Z2Uuc2l0ZWNvbW1hbmQudjEuT3BjVWFSZXF1ZXN0GiYuc2NhZGFicmlkZ2Uu", - "c2l0ZWNvbW1hbmQudjEuT3BjVWFSZXBseRJgCgxFeGVjdXRlUXVlcnkSKC5z", - "Y2FkYWJyaWRnZS5zaXRlY29tbWFuZC52MS5RdWVyeVJlcXVlc3QaJi5zY2Fk", - "YWJyaWRnZS5zaXRlY29tbWFuZC52MS5RdWVyeVJlcGx5EmMKDUV4ZWN1dGVQ", - "YXJrZWQSKS5zY2FkYWJyaWRnZS5zaXRlY29tbWFuZC52MS5QYXJrZWRSZXF1", - "ZXN0Gicuc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuUGFya2VkUmVwbHkS", - "YAoMRXhlY3V0ZVJvdXRlEiguc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEu", - "Um91dGVSZXF1ZXN0GiYuc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuUm91", - "dGVSZXBseRJ1Cg9UcmlnZ2VyRmFpbG92ZXISMi5zY2FkYWJyaWRnZS5zaXRl", - "Y29tbWFuZC52MS5UcmlnZ2VyU2l0ZUZhaWxvdmVyRHRvGi4uc2NhZGFicmlk", - "Z2Uuc2l0ZWNvbW1hbmQudjEuU2l0ZUZhaWxvdmVyQWNrRHRvQiuqAihaQi5N", - "T00uV1cuU2NhZGFCcmlkZ2UuQ29tbXVuaWNhdGlvbi5HcnBjYgZwcm90bzM=")); + "KAkSFgoOY29ycmVsYXRpb25faWQYAiABKAkSEwoLYWxhcm1zX29ubHkYAyAB", + "KAgiVAocU3Vic2NyaWJlRGVidWdWaWV3UmVxdWVzdER0bxIcChRpbnN0YW5j", + "ZV91bmlxdWVfbmFtZRgBIAEoCRIWCg5jb3JyZWxhdGlvbl9pZBgCIAEoCSJW", + "Ch5VbnN1YnNjcmliZURlYnVnVmlld1JlcXVlc3REdG8SHAoUaW5zdGFuY2Vf", + "dW5pcXVlX25hbWUYASABKAkSFgoOY29ycmVsYXRpb25faWQYAiABKAkiHAoa", + "VW5zdWJzY3JpYmVEZWJ1Z1ZpZXdBY2tEdG8i1AEKFkFsYXJtQ29uZGl0aW9u", + "U3RhdGVEdG8SDgoGYWN0aXZlGAEgASgIEhQKDGFja25vd2xlZGdlZBgCIAEo", + "CBItCgljb25maXJtZWQYAyABKAsyGi5nb29nbGUucHJvdG9idWYuQm9vbFZh", + "bHVlEj8KBnNoZWx2ZRgEIAEoDjIvLnNjYWRhYnJpZGdlLnNpdGVjb21tYW5k", + "LnYxLkFsYXJtU2hlbHZlU3RhdGVEdG8SEgoKc3VwcHJlc3NlZBgFIAEoCBIQ", + "CghzZXZlcml0eRgGIAEoBSLdAQoWRGVidWdBdHRyaWJ1dGVWYWx1ZUR0bxIc", + "ChRpbnN0YW5jZV91bmlxdWVfbmFtZRgBIAEoCRIWCg5hdHRyaWJ1dGVfcGF0", + "aBgCIAEoCRIWCg5hdHRyaWJ1dGVfbmFtZRgDIAEoCRI1CgV2YWx1ZRgEIAEo", + "CzImLnNjYWRhYnJpZGdlLnNpdGVjb21tYW5kLnYxLkxvb3NlVmFsdWUSDwoH", + "cXVhbGl0eRgFIAEoCRItCgl0aW1lc3RhbXAYBiABKAsyGi5nb29nbGUucHJv", + "dG9idWYuVGltZXN0YW1wIq8FChJEZWJ1Z0FsYXJtU3RhdGVEdG8SHAoUaW5z", + "dGFuY2VfdW5pcXVlX25hbWUYASABKAkSEgoKYWxhcm1fbmFtZRgCIAEoCRI4", + "CgVzdGF0ZRgDIAEoDjIpLnNjYWRhYnJpZGdlLnNpdGVjb21tYW5kLnYxLkFs", + "YXJtU3RhdGVEdG8SEAoIcHJpb3JpdHkYBCABKAUSLQoJdGltZXN0YW1wGAUg", + "ASgLMhouZ29vZ2xlLnByb3RvYnVmLlRpbWVzdGFtcBI4CgVsZXZlbBgGIAEo", + "DjIpLnNjYWRhYnJpZGdlLnNpdGVjb21tYW5kLnYxLkFsYXJtTGV2ZWxEdG8S", + "DwoHbWVzc2FnZRgHIAEoCRI2CgRraW5kGAggASgOMiguc2NhZGFicmlkZ2Uu", + "c2l0ZWNvbW1hbmQudjEuQWxhcm1LaW5kRHRvEkUKCWNvbmRpdGlvbhgJIAEo", + "CzIyLnNjYWRhYnJpZGdlLnNpdGVjb21tYW5kLnYxLkFsYXJtQ29uZGl0aW9u", + "U3RhdGVEdG8SGAoQc291cmNlX3JlZmVyZW5jZRgKIAEoCRIXCg9hbGFybV90", + "eXBlX25hbWUYCyABKAkSEAoIY2F0ZWdvcnkYDCABKAkSFQoNb3BlcmF0b3Jf", + "dXNlchgNIAEoCRIYChBvcGVyYXRvcl9jb21tZW50GA4gASgJEjcKE29yaWdp", + "bmFsX3JhaXNlX3RpbWUYDyABKAsyGi5nb29nbGUucHJvdG9idWYuVGltZXN0", + "YW1wEhUKDWN1cnJlbnRfdmFsdWUYECABKAkSEwoLbGltaXRfdmFsdWUYESAB", + "KAkSJAocbmF0aXZlX3NvdXJjZV9jYW5vbmljYWxfbmFtZRgSIAEoCRIhChlp", + "c19jb25maWd1cmVkX3BsYWNlaG9sZGVyGBMgASgIIpwCChREZWJ1Z1ZpZXdT", + "bmFwc2hvdER0bxIcChRpbnN0YW5jZV91bmlxdWVfbmFtZRgBIAEoCRJMChBh", + "dHRyaWJ1dGVfdmFsdWVzGAIgAygLMjIuc2NhZGFicmlkZ2Uuc2l0ZWNvbW1h", + "bmQudjEuRGVidWdBdHRyaWJ1dGVWYWx1ZUR0bxJECgxhbGFybV9zdGF0ZXMY", + "AyADKAsyLi5zY2FkYWJyaWRnZS5zaXRlY29tbWFuZC52MS5EZWJ1Z0FsYXJt", + "U3RhdGVEdG8SNgoSc25hcHNob3RfdGltZXN0YW1wGAQgASgLMhouZ29vZ2xl", + "LnByb3RvYnVmLlRpbWVzdGFtcBIaChJpbnN0YW5jZV9ub3RfZm91bmQYBSAB", + "KAgi8AIKDFF1ZXJ5UmVxdWVzdBJOCg9ldmVudF9sb2dfcXVlcnkYASABKAsy", + "My5zY2FkYWJyaWRnZS5zaXRlY29tbWFuZC52MS5FdmVudExvZ1F1ZXJ5UmVx", + "dWVzdER0b0gAEk0KDmRlYnVnX3NuYXBzaG90GAIgASgLMjMuc2NhZGFicmlk", + "Z2Uuc2l0ZWNvbW1hbmQudjEuRGVidWdTbmFwc2hvdFJlcXVlc3REdG9IABJY", + "ChRzdWJzY3JpYmVfZGVidWdfdmlldxgDIAEoCzI4LnNjYWRhYnJpZGdlLnNp", + "dGVjb21tYW5kLnYxLlN1YnNjcmliZURlYnVnVmlld1JlcXVlc3REdG9IABJc", + "ChZ1bnN1YnNjcmliZV9kZWJ1Z192aWV3GAQgASgLMjouc2NhZGFicmlkZ2Uu", + "c2l0ZWNvbW1hbmQudjEuVW5zdWJzY3JpYmVEZWJ1Z1ZpZXdSZXF1ZXN0RHRv", + "SABCCQoHY29tbWFuZCKRAgoKUXVlcnlSZXBseRJPCg9ldmVudF9sb2dfcXVl", + "cnkYASABKAsyNC5zY2FkYWJyaWRnZS5zaXRlY29tbWFuZC52MS5FdmVudExv", + "Z1F1ZXJ5UmVzcG9uc2VEdG9IABJPChNkZWJ1Z192aWV3X3NuYXBzaG90GAIg", + "ASgLMjAuc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuRGVidWdWaWV3U25h", + "cHNob3REdG9IABJYChZ1bnN1YnNjcmliZV9kZWJ1Z192aWV3GAMgASgLMjYu", + "c2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuVW5zdWJzY3JpYmVEZWJ1Z1Zp", + "ZXdBY2tEdG9IAEIHCgVyZXBseSKeAQocUGFya2VkTWVzc2FnZVF1ZXJ5UmVx", + "dWVzdER0bxIWCg5jb3JyZWxhdGlvbl9pZBgBIAEoCRIPCgdzaXRlX2lkGAIg", + "ASgJEhMKC3BhZ2VfbnVtYmVyGAMgASgFEhEKCXBhZ2Vfc2l6ZRgEIAEoBRIt", + "Cgl0aW1lc3RhbXAYBSABKAsyGi5nb29nbGUucHJvdG9idWYuVGltZXN0YW1w", + "IvICChVQYXJrZWRNZXNzYWdlRW50cnlEdG8SEgoKbWVzc2FnZV9pZBgBIAEo", + "CRIVCg10YXJnZXRfc3lzdGVtGAIgASgJEhMKC21ldGhvZF9uYW1lGAMgASgJ", + "EhUKDWVycm9yX21lc3NhZ2UYBCABKAkSFQoNYXR0ZW1wdF9jb3VudBgFIAEo", + "BRI2ChJvcmlnaW5hbF90aW1lc3RhbXAYBiABKAsyGi5nb29nbGUucHJvdG9i", + "dWYuVGltZXN0YW1wEjoKFmxhc3RfYXR0ZW1wdF90aW1lc3RhbXAYByABKAsy", + "Gi5nb29nbGUucHJvdG9idWYuVGltZXN0YW1wEhQKDG1heF9hdHRlbXB0cxgI", + "IAEoBRJICghjYXRlZ29yeRgJIAEoDjI2LnNjYWRhYnJpZGdlLnNpdGVjb21t", + "YW5kLnYxLlN0b3JlQW5kRm9yd2FyZENhdGVnb3J5RHRvEhcKD29yaWdpbl9p", + "bnN0YW5jZRgKIAEoCSKhAgodUGFya2VkTWVzc2FnZVF1ZXJ5UmVzcG9uc2VE", + "dG8SFgoOY29ycmVsYXRpb25faWQYASABKAkSDwoHc2l0ZV9pZBgCIAEoCRJD", + "CghtZXNzYWdlcxgDIAMoCzIxLnNjYWRhYnJpZGdlLnNpdGVjb21tYW5kLnYx", + "LlBhcmtlZE1lc3NhZ2VFbnRyeUR0bxITCgt0b3RhbF9jb3VudBgEIAEoBRIT", + "CgtwYWdlX251bWJlchgFIAEoBRIRCglwYWdlX3NpemUYBiABKAUSDwoHc3Vj", + "Y2VzcxgHIAEoCBIVCg1lcnJvcl9tZXNzYWdlGAggASgJEi0KCXRpbWVzdGFt", + "cBgJIAEoCzIaLmdvb2dsZS5wcm90b2J1Zi5UaW1lc3RhbXAiigEKHFBhcmtl", + "ZE1lc3NhZ2VSZXRyeVJlcXVlc3REdG8SFgoOY29ycmVsYXRpb25faWQYASAB", + "KAkSDwoHc2l0ZV9pZBgCIAEoCRISCgptZXNzYWdlX2lkGAMgASgJEi0KCXRp", + "bWVzdGFtcBgEIAEoCzIaLmdvb2dsZS5wcm90b2J1Zi5UaW1lc3RhbXAiXwod", + "UGFya2VkTWVzc2FnZVJldHJ5UmVzcG9uc2VEdG8SFgoOY29ycmVsYXRpb25f", + "aWQYASABKAkSDwoHc3VjY2VzcxgCIAEoCBIVCg1lcnJvcl9tZXNzYWdlGAMg", + "ASgJIowBCh5QYXJrZWRNZXNzYWdlRGlzY2FyZFJlcXVlc3REdG8SFgoOY29y", + "cmVsYXRpb25faWQYASABKAkSDwoHc2l0ZV9pZBgCIAEoCRISCgptZXNzYWdl", + "X2lkGAMgASgJEi0KCXRpbWVzdGFtcBgEIAEoCzIaLmdvb2dsZS5wcm90b2J1", + "Zi5UaW1lc3RhbXAiYQofUGFya2VkTWVzc2FnZURpc2NhcmRSZXNwb25zZUR0", + "bxIWCg5jb3JyZWxhdGlvbl9pZBgBIAEoCRIPCgdzdWNjZXNzGAIgASgIEhUK", + "DWVycm9yX21lc3NhZ2UYAyABKAkiTwoXUmV0cnlQYXJrZWRPcGVyYXRpb25E", + "dG8SFgoOY29ycmVsYXRpb25faWQYASABKAkSHAoUdHJhY2tlZF9vcGVyYXRp", + "b25faWQYAiABKAkiUQoZRGlzY2FyZFBhcmtlZE9wZXJhdGlvbkR0bxIWCg5j", + "b3JyZWxhdGlvbl9pZBgBIAEoCRIcChR0cmFja2VkX29wZXJhdGlvbl9pZBgC", + "IAEoCSJdChtQYXJrZWRPcGVyYXRpb25BY3Rpb25BY2tEdG8SFgoOY29ycmVs", + "YXRpb25faWQYASABKAkSDwoHYXBwbGllZBgCIAEoCBIVCg1lcnJvcl9tZXNz", + "YWdlGAMgASgJIt4DCg1QYXJrZWRSZXF1ZXN0ElgKFHBhcmtlZF9tZXNzYWdl", + "X3F1ZXJ5GAEgASgLMjguc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuUGFy", + "a2VkTWVzc2FnZVF1ZXJ5UmVxdWVzdER0b0gAElgKFHBhcmtlZF9tZXNzYWdl", + "X3JldHJ5GAIgASgLMjguc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuUGFy", + "a2VkTWVzc2FnZVJldHJ5UmVxdWVzdER0b0gAElwKFnBhcmtlZF9tZXNzYWdl", + "X2Rpc2NhcmQYAyABKAsyOi5zY2FkYWJyaWRnZS5zaXRlY29tbWFuZC52MS5Q", + "YXJrZWRNZXNzYWdlRGlzY2FyZFJlcXVlc3REdG9IABJVChZyZXRyeV9wYXJr", + "ZWRfb3BlcmF0aW9uGAQgASgLMjMuc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQu", + "djEuUmV0cnlQYXJrZWRPcGVyYXRpb25EdG9IABJZChhkaXNjYXJkX3Bhcmtl", + "ZF9vcGVyYXRpb24YBSABKAsyNS5zY2FkYWJyaWRnZS5zaXRlY29tbWFuZC52", + "MS5EaXNjYXJkUGFya2VkT3BlcmF0aW9uRHRvSABCCQoHY29tbWFuZCKHAwoL", + "UGFya2VkUmVwbHkSWQoUcGFya2VkX21lc3NhZ2VfcXVlcnkYASABKAsyOS5z", + "Y2FkYWJyaWRnZS5zaXRlY29tbWFuZC52MS5QYXJrZWRNZXNzYWdlUXVlcnlS", + "ZXNwb25zZUR0b0gAElkKFHBhcmtlZF9tZXNzYWdlX3JldHJ5GAIgASgLMjku", + "c2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuUGFya2VkTWVzc2FnZVJldHJ5", + "UmVzcG9uc2VEdG9IABJdChZwYXJrZWRfbWVzc2FnZV9kaXNjYXJkGAMgASgL", + "Mjsuc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuUGFya2VkTWVzc2FnZURp", + "c2NhcmRSZXNwb25zZUR0b0gAEloKF3BhcmtlZF9vcGVyYXRpb25fYWN0aW9u", + "GAQgASgLMjcuc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuUGFya2VkT3Bl", + "cmF0aW9uQWN0aW9uQWNrRHRvSABCBwoFcmVwbHki7QEKFVJvdXRlVG9DYWxs", + "UmVxdWVzdER0bxIWCg5jb3JyZWxhdGlvbl9pZBgBIAEoCRIcChRpbnN0YW5j", + "ZV91bmlxdWVfbmFtZRgCIAEoCRITCgtzY3JpcHRfbmFtZRgDIAEoCRI9Cgpw", + "YXJhbWV0ZXJzGAQgASgLMikuc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEu", + "TG9vc2VWYWx1ZU1hcBItCgl0aW1lc3RhbXAYBSABKAsyGi5nb29nbGUucHJv", + "dG9idWYuVGltZXN0YW1wEhsKE3BhcmVudF9leGVjdXRpb25faWQYBiABKAki", + "xQEKFlJvdXRlVG9DYWxsUmVzcG9uc2VEdG8SFgoOY29ycmVsYXRpb25faWQY", + "ASABKAkSDwoHc3VjY2VzcxgCIAEoCBI8CgxyZXR1cm5fdmFsdWUYAyABKAsy", + "Ji5zY2FkYWJyaWRnZS5zaXRlY29tbWFuZC52MS5Mb29zZVZhbHVlEhUKDWVy", + "cm9yX21lc3NhZ2UYBCABKAkSLQoJdGltZXN0YW1wGAUgASgLMhouZ29vZ2xl", + "LnByb3RvYnVmLlRpbWVzdGFtcCK7AQoeUm91dGVUb0dldEF0dHJpYnV0ZXNS", + "ZXF1ZXN0RHRvEhYKDmNvcnJlbGF0aW9uX2lkGAEgASgJEhwKFGluc3RhbmNl", + "X3VuaXF1ZV9uYW1lGAIgASgJEhcKD2F0dHJpYnV0ZV9uYW1lcxgDIAMoCRIt", + "Cgl0aW1lc3RhbXAYBCABKAsyGi5nb29nbGUucHJvdG9idWYuVGltZXN0YW1w", + "EhsKE3BhcmVudF9leGVjdXRpb25faWQYBSABKAkiywEKH1JvdXRlVG9HZXRB", + "dHRyaWJ1dGVzUmVzcG9uc2VEdG8SFgoOY29ycmVsYXRpb25faWQYASABKAkS", + "OQoGdmFsdWVzGAIgASgLMikuc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEu", + "TG9vc2VWYWx1ZU1hcBIPCgdzdWNjZXNzGAMgASgIEhUKDWVycm9yX21lc3Nh", + "Z2UYBCABKAkSLQoJdGltZXN0YW1wGAUgASgLMhouZ29vZ2xlLnByb3RvYnVm", + "LlRpbWVzdGFtcCLFAgoeUm91dGVUb1NldEF0dHJpYnV0ZXNSZXF1ZXN0RHRv", + "EhYKDmNvcnJlbGF0aW9uX2lkGAEgASgJEhwKFGluc3RhbmNlX3VuaXF1ZV9u", + "YW1lGAIgASgJEmkKEGF0dHJpYnV0ZV92YWx1ZXMYAyADKAsyTy5zY2FkYWJy", + "aWRnZS5zaXRlY29tbWFuZC52MS5Sb3V0ZVRvU2V0QXR0cmlidXRlc1JlcXVl", + "c3REdG8uQXR0cmlidXRlVmFsdWVzRW50cnkSLQoJdGltZXN0YW1wGAQgASgL", + "MhouZ29vZ2xlLnByb3RvYnVmLlRpbWVzdGFtcBIbChNwYXJlbnRfZXhlY3V0", + "aW9uX2lkGAUgASgJGjYKFEF0dHJpYnV0ZVZhbHVlc0VudHJ5EgsKA2tleRgB", + "IAEoCRINCgV2YWx1ZRgCIAEoCToCOAEikAEKH1JvdXRlVG9TZXRBdHRyaWJ1", + "dGVzUmVzcG9uc2VEdG8SFgoOY29ycmVsYXRpb25faWQYASABKAkSDwoHc3Vj", + "Y2VzcxgCIAEoCBIVCg1lcnJvcl9tZXNzYWdlGAMgASgJEi0KCXRpbWVzdGFt", + "cBgEIAEoCzIaLmdvb2dsZS5wcm90b2J1Zi5UaW1lc3RhbXAiwwIKIVJvdXRl", + "VG9XYWl0Rm9yQXR0cmlidXRlUmVxdWVzdER0bxIWCg5jb3JyZWxhdGlvbl9p", + "ZBgBIAEoCRIcChRpbnN0YW5jZV91bmlxdWVfbmFtZRgCIAEoCRIWCg5hdHRy", + "aWJ1dGVfbmFtZRgDIAEoCRI6ChR0YXJnZXRfdmFsdWVfZW5jb2RlZBgEIAEo", + "CzIcLmdvb2dsZS5wcm90b2J1Zi5TdHJpbmdWYWx1ZRIqCgd0aW1lb3V0GAUg", + "ASgLMhkuZ29vZ2xlLnByb3RvYnVmLkR1cmF0aW9uEi0KCXRpbWVzdGFtcBgG", + "IAEoCzIaLmdvb2dsZS5wcm90b2J1Zi5UaW1lc3RhbXASGwoTcGFyZW50X2V4", + "ZWN1dGlvbl9pZBgHIAEoCRIcChRyZXF1aXJlX2dvb2RfcXVhbGl0eRgIIAEo", + "CCL/AQoiUm91dGVUb1dhaXRGb3JBdHRyaWJ1dGVSZXNwb25zZUR0bxIWCg5j", + "b3JyZWxhdGlvbl9pZBgBIAEoCRIPCgdtYXRjaGVkGAIgASgIEjUKBXZhbHVl", + "GAMgASgLMiYuc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuTG9vc2VWYWx1", + "ZRIPCgdxdWFsaXR5GAQgASgJEhEKCXRpbWVkX291dBgFIAEoCBIPCgdzdWNj", + "ZXNzGAYgASgIEhUKDWVycm9yX21lc3NhZ2UYByABKAkSLQoJdGltZXN0YW1w", + "GAggASgLMhouZ29vZ2xlLnByb3RvYnVmLlRpbWVzdGFtcCKJAwoMUm91dGVS", + "ZXF1ZXN0EkoKDXJvdXRlX3RvX2NhbGwYASABKAsyMS5zY2FkYWJyaWRnZS5z", + "aXRlY29tbWFuZC52MS5Sb3V0ZVRvQ2FsbFJlcXVlc3REdG9IABJdChdyb3V0", + "ZV90b19nZXRfYXR0cmlidXRlcxgCIAEoCzI6LnNjYWRhYnJpZGdlLnNpdGVj", + "b21tYW5kLnYxLlJvdXRlVG9HZXRBdHRyaWJ1dGVzUmVxdWVzdER0b0gAEl0K", + "F3JvdXRlX3RvX3NldF9hdHRyaWJ1dGVzGAMgASgLMjouc2NhZGFicmlkZ2Uu", + "c2l0ZWNvbW1hbmQudjEuUm91dGVUb1NldEF0dHJpYnV0ZXNSZXF1ZXN0RHRv", + "SAASZAobcm91dGVfdG9fd2FpdF9mb3JfYXR0cmlidXRlGAQgASgLMj0uc2Nh", + "ZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuUm91dGVUb1dhaXRGb3JBdHRyaWJ1", + "dGVSZXF1ZXN0RHRvSABCCQoHY29tbWFuZCKJAwoKUm91dGVSZXBseRJLCg1y", + "b3V0ZV90b19jYWxsGAEgASgLMjIuc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQu", + "djEuUm91dGVUb0NhbGxSZXNwb25zZUR0b0gAEl4KF3JvdXRlX3RvX2dldF9h", + "dHRyaWJ1dGVzGAIgASgLMjsuc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEu", + "Um91dGVUb0dldEF0dHJpYnV0ZXNSZXNwb25zZUR0b0gAEl4KF3JvdXRlX3Rv", + "X3NldF9hdHRyaWJ1dGVzGAMgASgLMjsuc2NhZGFicmlkZ2Uuc2l0ZWNvbW1h", + "bmQudjEuUm91dGVUb1NldEF0dHJpYnV0ZXNSZXNwb25zZUR0b0gAEmUKG3Jv", + "dXRlX3RvX3dhaXRfZm9yX2F0dHJpYnV0ZRgEIAEoCzI+LnNjYWRhYnJpZGdl", + "LnNpdGVjb21tYW5kLnYxLlJvdXRlVG9XYWl0Rm9yQXR0cmlidXRlUmVzcG9u", + "c2VEdG9IAEIHCgVyZXBseSJBChZUcmlnZ2VyU2l0ZUZhaWxvdmVyRHRvEhYK", + "DmNvcnJlbGF0aW9uX2lkGAEgASgJEg8KB3NpdGVfaWQYAiABKAkibQoSU2l0", + "ZUZhaWxvdmVyQWNrRHRvEhYKDmNvcnJlbGF0aW9uX2lkGAEgASgJEhAKCGFj", + "Y2VwdGVkGAIgASgIEhYKDnRhcmdldF9hZGRyZXNzGAMgASgJEhUKDWVycm9y", + "X21lc3NhZ2UYBCABKAkqywEKE0RlcGxveW1lbnRTdGF0dXNEdG8SJQohREVQ", + "TE9ZTUVOVF9TVEFUVVNfRFRPX1VOU1BFQ0lGSUVEEAASIQodREVQTE9ZTUVO", + "VF9TVEFUVVNfRFRPX1BFTkRJTkcQARIlCiFERVBMT1lNRU5UX1NUQVRVU19E", + "VE9fSU5fUFJPR1JFU1MQAhIhCh1ERVBMT1lNRU5UX1NUQVRVU19EVE9fU1VD", + "Q0VTUxADEiAKHERFUExPWU1FTlRfU1RBVFVTX0RUT19GQUlMRUQQBCrEAQoS", + "QnJvd3NlTm9kZUNsYXNzRHRvEiUKIUJST1dTRV9OT0RFX0NMQVNTX0RUT19V", + "TlNQRUNJRklFRBAAEiAKHEJST1dTRV9OT0RFX0NMQVNTX0RUT19PQkpFQ1QQ", + "ARIiCh5CUk9XU0VfTk9ERV9DTEFTU19EVE9fVkFSSUFCTEUQAhIgChxCUk9X", + "U0VfTk9ERV9DTEFTU19EVE9fTUVUSE9EEAMSHwobQlJPV1NFX05PREVfQ0xB", + "U1NfRFRPX09USEVSEAQqoQIKFEJyb3dzZUZhaWx1cmVLaW5kRHRvEicKI0JS", + "T1dTRV9GQUlMVVJFX0tJTkRfRFRPX1VOU1BFQ0lGSUVEEAASMAosQlJPV1NF", + "X0ZBSUxVUkVfS0lORF9EVE9fQ09OTkVDVElPTl9OT1RfRk9VTkQQARI0CjBC", + "Uk9XU0VfRkFJTFVSRV9LSU5EX0RUT19DT05ORUNUSU9OX05PVF9DT05ORUNU", + "RUQQAhIpCiVCUk9XU0VfRkFJTFVSRV9LSU5EX0RUT19OT1RfQlJPV1NBQkxF", + "EAMSIwofQlJPV1NFX0ZBSUxVUkVfS0lORF9EVE9fVElNRU9VVBAEEigKJEJS", + "T1dTRV9GQUlMVVJFX0tJTkRfRFRPX1NFUlZFUl9FUlJPUhAFKqoCChtSZWFk", + "VGFnVmFsdWVzRmFpbHVyZUtpbmREdG8SMAosUkVBRF9UQUdfVkFMVUVTX0ZB", + "SUxVUkVfS0lORF9EVE9fVU5TUEVDSUZJRUQQABI5CjVSRUFEX1RBR19WQUxV", + "RVNfRkFJTFVSRV9LSU5EX0RUT19DT05ORUNUSU9OX05PVF9GT1VORBABEj0K", + "OVJFQURfVEFHX1ZBTFVFU19GQUlMVVJFX0tJTkRfRFRPX0NPTk5FQ1RJT05f", + "Tk9UX0NPTk5FQ1RFRBACEiwKKFJFQURfVEFHX1ZBTFVFU19GQUlMVVJFX0tJ", + "TkRfRFRPX1RJTUVPVVQQAxIxCi1SRUFEX1RBR19WQUxVRVNfRkFJTFVSRV9L", + "SU5EX0RUT19TRVJWRVJfRVJST1IQBCqTAgoUVmVyaWZ5RmFpbHVyZUtpbmRE", + "dG8SJwojVkVSSUZZX0ZBSUxVUkVfS0lORF9EVE9fVU5TUEVDSUZJRUQQABIn", + "CiNWRVJJRllfRkFJTFVSRV9LSU5EX0RUT19VTlJFQUNIQUJMRRABEicKI1ZF", + "UklGWV9GQUlMVVJFX0tJTkRfRFRPX0FVVEhfRkFJTEVEEAISMQotVkVSSUZZ", + "X0ZBSUxVUkVfS0lORF9EVE9fVU5UUlVTVEVEX0NFUlRJRklDQVRFEAMSIwof", + "VkVSSUZZX0ZBSUxVUkVfS0lORF9EVE9fVElNRU9VVBAEEigKJFZFUklGWV9G", + "QUlMVVJFX0tJTkRfRFRPX1NFUlZFUl9FUlJPUhAFKuUBChpTdG9yZUFuZEZv", + "cndhcmRDYXRlZ29yeUR0bxIuCipTVE9SRV9BTkRfRk9SV0FSRF9DQVRFR09S", + "WV9EVE9fVU5TUEVDSUZJRUQQABIyCi5TVE9SRV9BTkRfRk9SV0FSRF9DQVRF", + "R09SWV9EVE9fRVhURVJOQUxfU1lTVEVNEAESLworU1RPUkVfQU5EX0ZPUldB", + "UkRfQ0FURUdPUllfRFRPX05PVElGSUNBVElPThACEjIKLlNUT1JFX0FORF9G", + "T1JXQVJEX0NBVEVHT1JZX0RUT19DQUNIRURfREJfV1JJVEUQAypoCg1BbGFy", + "bVN0YXRlRHRvEh8KG0FMQVJNX1NUQVRFX0RUT19VTlNQRUNJRklFRBAAEhoK", + "FkFMQVJNX1NUQVRFX0RUT19BQ1RJVkUQARIaChZBTEFSTV9TVEFURV9EVE9f", + "Tk9STUFMEAIquQEKDUFsYXJtTGV2ZWxEdG8SHwobQUxBUk1fTEVWRUxfRFRP", + "X1VOU1BFQ0lGSUVEEAASGAoUQUxBUk1fTEVWRUxfRFRPX05PTkUQARIXChNB", + "TEFSTV9MRVZFTF9EVE9fTE9XEAISGwoXQUxBUk1fTEVWRUxfRFRPX0xPV19M", + "T1cQAxIYChRBTEFSTV9MRVZFTF9EVE9fSElHSBAEEh0KGUFMQVJNX0xFVkVM", + "X0RUT19ISUdIX0hJR0gQBSqSAQoMQWxhcm1LaW5kRHRvEh4KGkFMQVJNX0tJ", + "TkRfRFRPX1VOU1BFQ0lGSUVEEAASGwoXQUxBUk1fS0lORF9EVE9fQ09NUFVU", + "RUQQARIgChxBTEFSTV9LSU5EX0RUT19OQVRJVkVfT1BDX1VBEAISIwofQUxB", + "Uk1fS0lORF9EVE9fTkFUSVZFX01YX0FDQ0VTUxADKugBChNBbGFybVNoZWx2", + "ZVN0YXRlRHRvEiYKIkFMQVJNX1NIRUxWRV9TVEFURV9EVE9fVU5TUEVDSUZJ", + "RUQQABIkCiBBTEFSTV9TSEVMVkVfU1RBVEVfRFRPX1VOU0hFTFZFRBABEisK", + "J0FMQVJNX1NIRUxWRV9TVEFURV9EVE9fT05FX1NIT1RfU0hFTFZFRBACEigK", + "JEFMQVJNX1NIRUxWRV9TVEFURV9EVE9fVElNRURfU0hFTFZFRBADEiwKKEFM", + "QVJNX1NIRUxWRV9TVEFURV9EVE9fUEVSTUFORU5UX1NIRUxWRUQQBDKEBQoS", + "U2l0ZUNvbW1hbmRTZXJ2aWNlEmwKEEV4ZWN1dGVMaWZlY3ljbGUSLC5zY2Fk", + "YWJyaWRnZS5zaXRlY29tbWFuZC52MS5MaWZlY3ljbGVSZXF1ZXN0Giouc2Nh", + "ZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuTGlmZWN5Y2xlUmVwbHkSYAoMRXhl", + "Y3V0ZU9wY1VhEiguc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuT3BjVWFS", + "ZXF1ZXN0GiYuc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuT3BjVWFSZXBs", + "eRJgCgxFeGVjdXRlUXVlcnkSKC5zY2FkYWJyaWRnZS5zaXRlY29tbWFuZC52", + "MS5RdWVyeVJlcXVlc3QaJi5zY2FkYWJyaWRnZS5zaXRlY29tbWFuZC52MS5R", + "dWVyeVJlcGx5EmMKDUV4ZWN1dGVQYXJrZWQSKS5zY2FkYWJyaWRnZS5zaXRl", + "Y29tbWFuZC52MS5QYXJrZWRSZXF1ZXN0Gicuc2NhZGFicmlkZ2Uuc2l0ZWNv", + "bW1hbmQudjEuUGFya2VkUmVwbHkSYAoMRXhlY3V0ZVJvdXRlEiguc2NhZGFi", + "cmlkZ2Uuc2l0ZWNvbW1hbmQudjEuUm91dGVSZXF1ZXN0GiYuc2NhZGFicmlk", + "Z2Uuc2l0ZWNvbW1hbmQudjEuUm91dGVSZXBseRJ1Cg9UcmlnZ2VyRmFpbG92", + "ZXISMi5zY2FkYWJyaWRnZS5zaXRlY29tbWFuZC52MS5UcmlnZ2VyU2l0ZUZh", + "aWxvdmVyRHRvGi4uc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuU2l0ZUZh", + "aWxvdmVyQWNrRHRvQiuqAihaQi5NT00uV1cuU2NhZGFCcmlkZ2UuQ29tbXVu", + "aWNhdGlvbi5HcnBjYgZwcm90bzM=")); descriptor = pbr::FileDescriptor.FromGeneratedCode(descriptorData, new pbr::FileDescriptor[] { global::Google.Protobuf.WellKnownTypes.TimestampReflection.Descriptor, global::Google.Protobuf.WellKnownTypes.DurationReflection.Descriptor, global::Google.Protobuf.WellKnownTypes.WrappersReflection.Descriptor, }, new pbr::GeneratedClrTypeInfo(new[] {typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.DeploymentStatusDto), typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.BrowseNodeClassDto), typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.BrowseFailureKindDto), typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ReadTagValuesFailureKindDto), typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.VerifyFailureKindDto), typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.StoreAndForwardCategoryDto), typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AlarmStateDto), typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AlarmLevelDto), typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AlarmKindDto), typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AlarmShelveStateDto), }, null, new pbr::GeneratedClrTypeInfo[] { @@ -554,7 +555,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.EventLogQueryRequestDto), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.EventLogQueryRequestDto.Parser, new[]{ "CorrelationId", "SiteId", "From", "To", "EventType", "Severity", "InstanceId", "KeywordFilter", "ContinuationToken", "PageSize", "Timestamp" }, null, null, null, null), new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.EventLogEntryDto), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.EventLogEntryDto.Parser, new[]{ "Id", "Timestamp", "EventType", "Severity", "InstanceId", "Source", "Message", "Details" }, null, null, null, null), new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.EventLogQueryResponseDto), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.EventLogQueryResponseDto.Parser, new[]{ "CorrelationId", "SiteId", "Entries", "ContinuationToken", "HasMore", "Success", "ErrorMessage", "Timestamp" }, null, null, null, null), - new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.DebugSnapshotRequestDto), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.DebugSnapshotRequestDto.Parser, new[]{ "InstanceUniqueName", "CorrelationId" }, null, null, null, null), + new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.DebugSnapshotRequestDto), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.DebugSnapshotRequestDto.Parser, new[]{ "InstanceUniqueName", "CorrelationId", "AlarmsOnly" }, null, null, null, null), new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SubscribeDebugViewRequestDto), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SubscribeDebugViewRequestDto.Parser, new[]{ "InstanceUniqueName", "CorrelationId" }, null, null, null, null), new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.UnsubscribeDebugViewRequestDto), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.UnsubscribeDebugViewRequestDto.Parser, new[]{ "InstanceUniqueName", "CorrelationId" }, null, null, null, null), new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.UnsubscribeDebugViewAckDto), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.UnsubscribeDebugViewAckDto.Parser, null, null, null, null, null), @@ -19509,6 +19510,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { public DebugSnapshotRequestDto(DebugSnapshotRequestDto other) : this() { instanceUniqueName_ = other.instanceUniqueName_; correlationId_ = other.correlationId_; + alarmsOnly_ = other.alarmsOnly_; _unknownFields = pb::UnknownFieldSet.Clone(other._unknownFields); } @@ -19542,6 +19544,27 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { } } + /// Field number for the "alarms_only" field. + public const int AlarmsOnlyFieldNumber = 3; + private bool alarmsOnly_; + /// + /// Alarms-only projection (WP2.3 wire efficiency): when true the site builds + /// and returns ONLY the alarm rows of the debug snapshot, leaving + /// attribute_values empty. Used by the central per-site live alarm cache, + /// whose seed/reconcile fan-out discards every attribute row anyway — a full + /// snapshot ships the whole attribute surface of every enabled instance once + /// per reconcile for nothing. proto3 defaults it to false, so an older + /// central that never sets it keeps the full-snapshot behaviour. Additive-only. + /// + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public bool AlarmsOnly { + get { return alarmsOnly_; } + set { + alarmsOnly_ = value; + } + } + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] public override bool Equals(object other) { @@ -19559,6 +19582,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { } if (InstanceUniqueName != other.InstanceUniqueName) return false; if (CorrelationId != other.CorrelationId) return false; + if (AlarmsOnly != other.AlarmsOnly) return false; return Equals(_unknownFields, other._unknownFields); } @@ -19568,6 +19592,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { int hash = 1; if (InstanceUniqueName.Length != 0) hash ^= InstanceUniqueName.GetHashCode(); if (CorrelationId.Length != 0) hash ^= CorrelationId.GetHashCode(); + if (AlarmsOnly != false) hash ^= AlarmsOnly.GetHashCode(); if (_unknownFields != null) { hash ^= _unknownFields.GetHashCode(); } @@ -19594,6 +19619,10 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { output.WriteRawTag(18); output.WriteString(CorrelationId); } + if (AlarmsOnly != false) { + output.WriteRawTag(24); + output.WriteBool(AlarmsOnly); + } if (_unknownFields != null) { _unknownFields.WriteTo(output); } @@ -19612,6 +19641,10 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { output.WriteRawTag(18); output.WriteString(CorrelationId); } + if (AlarmsOnly != false) { + output.WriteRawTag(24); + output.WriteBool(AlarmsOnly); + } if (_unknownFields != null) { _unknownFields.WriteTo(ref output); } @@ -19628,6 +19661,9 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { if (CorrelationId.Length != 0) { size += 1 + pb::CodedOutputStream.ComputeStringSize(CorrelationId); } + if (AlarmsOnly != false) { + size += 1 + 1; + } if (_unknownFields != null) { size += _unknownFields.CalculateSize(); } @@ -19646,6 +19682,9 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { if (other.CorrelationId.Length != 0) { CorrelationId = other.CorrelationId; } + if (other.AlarmsOnly != false) { + AlarmsOnly = other.AlarmsOnly; + } _unknownFields = pb::UnknownFieldSet.MergeFrom(_unknownFields, other._unknownFields); } @@ -19673,6 +19712,10 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { CorrelationId = input.ReadString(); break; } + case 24: { + AlarmsOnly = input.ReadBool(); + break; + } } } #endif @@ -19700,6 +19743,10 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { CorrelationId = input.ReadString(); break; } + case 24: { + AlarmsOnly = input.ReadBool(); + break; + } } } } diff --git a/src/ZB.MOM.WW.ScadaBridge.Communication/SiteStreamGrpc/Sitestream.cs b/src/ZB.MOM.WW.ScadaBridge.Communication/SiteStreamGrpc/Sitestream.cs index 450aacd3..40173172 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Communication/SiteStreamGrpc/Sitestream.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Communication/SiteStreamGrpc/Sitestream.cs @@ -81,36 +81,36 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { "dEV2ZW50RHRvEjcKC29wZXJhdGlvbmFsGAIgASgLMiIuc2l0ZXN0cmVhbS5T", "aXRlQ2FsbE9wZXJhdGlvbmFsRHRvIkoKFENhY2hlZFRlbGVtZXRyeUJhdGNo", "EjIKB3BhY2tldHMYASADKAsyIS5zaXRlc3RyZWFtLkNhY2hlZFRlbGVtZXRy", - "eVBhY2tldCJbChZQdWxsQXVkaXRFdmVudHNSZXF1ZXN0Ei0KCXNpbmNlX3V0", + "eVBhY2tldCJtChZQdWxsQXVkaXRFdmVudHNSZXF1ZXN0Ei0KCXNpbmNlX3V0", "YxgBIAEoCzIaLmdvb2dsZS5wcm90b2J1Zi5UaW1lc3RhbXASEgoKYmF0Y2hf", - "c2l6ZRgCIAEoBSJcChdQdWxsQXVkaXRFdmVudHNSZXNwb25zZRIpCgZldmVu", - "dHMYASADKAsyGS5zaXRlc3RyZWFtLkF1ZGl0RXZlbnREdG8SFgoObW9yZV9h", - "dmFpbGFibGUYAiABKAgiawoUUHVsbFNpdGVDYWxsc1JlcXVlc3QSLQoJc2lu", - "Y2VfdXRjGAEgASgLMhouZ29vZ2xlLnByb3RvYnVmLlRpbWVzdGFtcBISCgpi", - "YXRjaF9zaXplGAIgASgFEhAKCGFmdGVyX2lkGAMgASgJImkKFVB1bGxTaXRl", - "Q2FsbHNSZXNwb25zZRI4CgxvcGVyYXRpb25hbHMYASADKAsyIi5zaXRlc3Ry", - "ZWFtLlNpdGVDYWxsT3BlcmF0aW9uYWxEdG8SFgoObW9yZV9hdmFpbGFibGUY", - "AiABKAgqXAoHUXVhbGl0eRIXChNRVUFMSVRZX1VOU1BFQ0lGSUVEEAASEAoM", - "UVVBTElUWV9HT09EEAESFQoRUVVBTElUWV9VTkNFUlRBSU4QAhIPCgtRVUFM", - "SVRZX0JBRBADKl0KDkFsYXJtU3RhdGVFbnVtEhsKF0FMQVJNX1NUQVRFX1VO", - "U1BFQ0lGSUVEEAASFgoSQUxBUk1fU1RBVEVfTk9STUFMEAESFgoSQUxBUk1f", - "U1RBVEVfQUNUSVZFEAIqhQEKDkFsYXJtTGV2ZWxFbnVtEhQKEEFMQVJNX0xF", - "VkVMX05PTkUQABITCg9BTEFSTV9MRVZFTF9MT1cQARIXChNBTEFSTV9MRVZF", - "TF9MT1dfTE9XEAISFAoQQUxBUk1fTEVWRUxfSElHSBADEhkKFUFMQVJNX0xF", - "VkVMX0hJR0hfSElHSBAEMoYEChFTaXRlU3RyZWFtU2VydmljZRJVChFTdWJz", - "Y3JpYmVJbnN0YW5jZRIhLnNpdGVzdHJlYW0uSW5zdGFuY2VTdHJlYW1SZXF1", - "ZXN0Ghsuc2l0ZXN0cmVhbS5TaXRlU3RyZWFtRXZlbnQwARJNCg1TdWJzY3Jp", - "YmVTaXRlEh0uc2l0ZXN0cmVhbS5TaXRlU3RyZWFtUmVxdWVzdBobLnNpdGVz", - "dHJlYW0uU2l0ZVN0cmVhbUV2ZW50MAESRwoRSW5nZXN0QXVkaXRFdmVudHMS", - "Gy5zaXRlc3RyZWFtLkF1ZGl0RXZlbnRCYXRjaBoVLnNpdGVzdHJlYW0uSW5n", - "ZXN0QWNrElAKFUluZ2VzdENhY2hlZFRlbGVtZXRyeRIgLnNpdGVzdHJlYW0u", - "Q2FjaGVkVGVsZW1ldHJ5QmF0Y2gaFS5zaXRlc3RyZWFtLkluZ2VzdEFjaxJa", - "Cg9QdWxsQXVkaXRFdmVudHMSIi5zaXRlc3RyZWFtLlB1bGxBdWRpdEV2ZW50", - "c1JlcXVlc3QaIy5zaXRlc3RyZWFtLlB1bGxBdWRpdEV2ZW50c1Jlc3BvbnNl", - "ElQKDVB1bGxTaXRlQ2FsbHMSIC5zaXRlc3RyZWFtLlB1bGxTaXRlQ2FsbHNS", - "ZXF1ZXN0GiEuc2l0ZXN0cmVhbS5QdWxsU2l0ZUNhbGxzUmVzcG9uc2VCK6oC", - "KFpCLk1PTS5XVy5TY2FkYUJyaWRnZS5Db21tdW5pY2F0aW9uLkdycGNiBnBy", - "b3RvMw==")); + "c2l6ZRgCIAEoBRIQCghhZnRlcl9pZBgDIAEoCSJcChdQdWxsQXVkaXRFdmVu", + "dHNSZXNwb25zZRIpCgZldmVudHMYASADKAsyGS5zaXRlc3RyZWFtLkF1ZGl0", + "RXZlbnREdG8SFgoObW9yZV9hdmFpbGFibGUYAiABKAgiawoUUHVsbFNpdGVD", + "YWxsc1JlcXVlc3QSLQoJc2luY2VfdXRjGAEgASgLMhouZ29vZ2xlLnByb3Rv", + "YnVmLlRpbWVzdGFtcBISCgpiYXRjaF9zaXplGAIgASgFEhAKCGFmdGVyX2lk", + "GAMgASgJImkKFVB1bGxTaXRlQ2FsbHNSZXNwb25zZRI4CgxvcGVyYXRpb25h", + "bHMYASADKAsyIi5zaXRlc3RyZWFtLlNpdGVDYWxsT3BlcmF0aW9uYWxEdG8S", + "FgoObW9yZV9hdmFpbGFibGUYAiABKAgqXAoHUXVhbGl0eRIXChNRVUFMSVRZ", + "X1VOU1BFQ0lGSUVEEAASEAoMUVVBTElUWV9HT09EEAESFQoRUVVBTElUWV9V", + "TkNFUlRBSU4QAhIPCgtRVUFMSVRZX0JBRBADKl0KDkFsYXJtU3RhdGVFbnVt", + "EhsKF0FMQVJNX1NUQVRFX1VOU1BFQ0lGSUVEEAASFgoSQUxBUk1fU1RBVEVf", + "Tk9STUFMEAESFgoSQUxBUk1fU1RBVEVfQUNUSVZFEAIqhQEKDkFsYXJtTGV2", + "ZWxFbnVtEhQKEEFMQVJNX0xFVkVMX05PTkUQABITCg9BTEFSTV9MRVZFTF9M", + "T1cQARIXChNBTEFSTV9MRVZFTF9MT1dfTE9XEAISFAoQQUxBUk1fTEVWRUxf", + "SElHSBADEhkKFUFMQVJNX0xFVkVMX0hJR0hfSElHSBAEMoYEChFTaXRlU3Ry", + "ZWFtU2VydmljZRJVChFTdWJzY3JpYmVJbnN0YW5jZRIhLnNpdGVzdHJlYW0u", + "SW5zdGFuY2VTdHJlYW1SZXF1ZXN0Ghsuc2l0ZXN0cmVhbS5TaXRlU3RyZWFt", + "RXZlbnQwARJNCg1TdWJzY3JpYmVTaXRlEh0uc2l0ZXN0cmVhbS5TaXRlU3Ry", + "ZWFtUmVxdWVzdBobLnNpdGVzdHJlYW0uU2l0ZVN0cmVhbUV2ZW50MAESRwoR", + "SW5nZXN0QXVkaXRFdmVudHMSGy5zaXRlc3RyZWFtLkF1ZGl0RXZlbnRCYXRj", + "aBoVLnNpdGVzdHJlYW0uSW5nZXN0QWNrElAKFUluZ2VzdENhY2hlZFRlbGVt", + "ZXRyeRIgLnNpdGVzdHJlYW0uQ2FjaGVkVGVsZW1ldHJ5QmF0Y2gaFS5zaXRl", + "c3RyZWFtLkluZ2VzdEFjaxJaCg9QdWxsQXVkaXRFdmVudHMSIi5zaXRlc3Ry", + "ZWFtLlB1bGxBdWRpdEV2ZW50c1JlcXVlc3QaIy5zaXRlc3RyZWFtLlB1bGxB", + "dWRpdEV2ZW50c1Jlc3BvbnNlElQKDVB1bGxTaXRlQ2FsbHMSIC5zaXRlc3Ry", + "ZWFtLlB1bGxTaXRlQ2FsbHNSZXF1ZXN0GiEuc2l0ZXN0cmVhbS5QdWxsU2l0", + "ZUNhbGxzUmVzcG9uc2VCK6oCKFpCLk1PTS5XVy5TY2FkYUJyaWRnZS5Db21t", + "dW5pY2F0aW9uLkdycGNiBnByb3RvMw==")); descriptor = pbr::FileDescriptor.FromGeneratedCode(descriptorData, new pbr::FileDescriptor[] { global::Google.Protobuf.WellKnownTypes.TimestampReflection.Descriptor, global::Google.Protobuf.WellKnownTypes.WrappersReflection.Descriptor, }, new pbr::GeneratedClrTypeInfo(new[] {typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.Quality), typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AlarmStateEnum), typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AlarmLevelEnum), }, null, new pbr::GeneratedClrTypeInfo[] { @@ -125,7 +125,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteCallOperationalDto), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteCallOperationalDto.Parser, new[]{ "TrackedOperationId", "Channel", "Target", "SourceSite", "Status", "RetryCount", "LastError", "HttpStatus", "CreatedAtUtc", "UpdatedAtUtc", "TerminalAtUtc", "SourceNode" }, null, null, null, null), new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.CachedTelemetryPacket), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.CachedTelemetryPacket.Parser, new[]{ "AuditEvent", "Operational" }, null, null, null, null), new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.CachedTelemetryBatch), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.CachedTelemetryBatch.Parser, new[]{ "Packets" }, null, null, null, null), - new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.PullAuditEventsRequest), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.PullAuditEventsRequest.Parser, new[]{ "SinceUtc", "BatchSize" }, null, null, null, null), + new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.PullAuditEventsRequest), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.PullAuditEventsRequest.Parser, new[]{ "SinceUtc", "BatchSize", "AfterId" }, null, null, null, null), new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.PullAuditEventsResponse), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.PullAuditEventsResponse.Parser, new[]{ "Events", "MoreAvailable" }, null, null, null, null), new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.PullSiteCallsRequest), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.PullSiteCallsRequest.Parser, new[]{ "SinceUtc", "BatchSize", "AfterId" }, null, null, null, null), new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.PullSiteCallsResponse), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.PullSiteCallsResponse.Parser, new[]{ "Operationals", "MoreAvailable" }, null, null, null, null) @@ -4942,8 +4942,11 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { /// /// Audit Log (#23) M6 reconciliation pull: central→site request for any /// site-local AuditLog rows with OccurredAtUtc >= since_utc that have not yet - /// been ingested centrally (ForwardState in {Pending, Forwarded}). The site - /// flips returned rows to Reconciled after the response is on the wire. + /// been ingested centrally (ForwardState in {Pending, Forwarded}). Rows are NOT + /// flipped to Reconciled when they are served — only when a LATER pull's cursor + /// proves central consumed them (see after_id), so a fault between the response + /// leaving the site and central committing it re-serves the rows instead of + /// silently losing them (at-least-once). /// more_available signals batch_size was saturated so the caller knows to /// issue a follow-up pull with an advanced since_utc cursor. /// @@ -4984,6 +4987,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { public PullAuditEventsRequest(PullAuditEventsRequest other) : this() { sinceUtc_ = other.sinceUtc_ != null ? other.sinceUtc_.Clone() : null; batchSize_ = other.batchSize_; + afterId_ = other.afterId_; _unknownFields = pb::UnknownFieldSet.Clone(other._unknownFields); } @@ -5017,6 +5021,29 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { } } + /// Field number for the "after_id" field. + public const int AfterIdFieldNumber = 3; + private string afterId_ = ""; + /// + /// Composite-keyset cursor (WP2.3), mirroring PullSiteCallsRequest.after_id: + /// the EventId ("D" GUID form) of the last row central has already CONSUMED at + /// since_utc. When set, the site returns only rows strictly after the composite + /// (OccurredAtUtc, EventId) pair — un-pinning a batch that would otherwise stall + /// when more than batch_size rows share one since_utc instant — AND treats the + /// cursor as proof of receipt: everything at or before it is flipped to + /// Reconciled. Empty (the proto3 string default) preserves the legacy inclusive + /// >= behaviour, under which only rows strictly older than since_utc are proven + /// received. Additive-only. + /// + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public string AfterId { + get { return afterId_; } + set { + afterId_ = pb::ProtoPreconditions.CheckNotNull(value, "value"); + } + } + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] public override bool Equals(object other) { @@ -5034,6 +5061,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { } if (!object.Equals(SinceUtc, other.SinceUtc)) return false; if (BatchSize != other.BatchSize) return false; + if (AfterId != other.AfterId) return false; return Equals(_unknownFields, other._unknownFields); } @@ -5043,6 +5071,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { int hash = 1; if (sinceUtc_ != null) hash ^= SinceUtc.GetHashCode(); if (BatchSize != 0) hash ^= BatchSize.GetHashCode(); + if (AfterId.Length != 0) hash ^= AfterId.GetHashCode(); if (_unknownFields != null) { hash ^= _unknownFields.GetHashCode(); } @@ -5069,6 +5098,10 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { output.WriteRawTag(16); output.WriteInt32(BatchSize); } + if (AfterId.Length != 0) { + output.WriteRawTag(26); + output.WriteString(AfterId); + } if (_unknownFields != null) { _unknownFields.WriteTo(output); } @@ -5087,6 +5120,10 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { output.WriteRawTag(16); output.WriteInt32(BatchSize); } + if (AfterId.Length != 0) { + output.WriteRawTag(26); + output.WriteString(AfterId); + } if (_unknownFields != null) { _unknownFields.WriteTo(ref output); } @@ -5103,6 +5140,9 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { if (BatchSize != 0) { size += 1 + pb::CodedOutputStream.ComputeInt32Size(BatchSize); } + if (AfterId.Length != 0) { + size += 1 + pb::CodedOutputStream.ComputeStringSize(AfterId); + } if (_unknownFields != null) { size += _unknownFields.CalculateSize(); } @@ -5124,6 +5164,9 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { if (other.BatchSize != 0) { BatchSize = other.BatchSize; } + if (other.AfterId.Length != 0) { + AfterId = other.AfterId; + } _unknownFields = pb::UnknownFieldSet.MergeFrom(_unknownFields, other._unknownFields); } @@ -5154,6 +5197,10 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { BatchSize = input.ReadInt32(); break; } + case 26: { + AfterId = input.ReadString(); + break; + } } } #endif @@ -5184,6 +5231,10 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { BatchSize = input.ReadInt32(); break; } + case 26: { + AfterId = input.ReadString(); + break; + } } } } diff --git a/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Configurations/AuditLogEntityTypeConfiguration.cs b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Configurations/AuditLogEntityTypeConfiguration.cs index 9b86006e..ee94b2a7 100644 --- a/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Configurations/AuditLogEntityTypeConfiguration.cs +++ b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Configurations/AuditLogEntityTypeConfiguration.cs @@ -172,22 +172,30 @@ public class AuditLogEntityTypeConfiguration : IEntityTypeConfiguration new { e.EventId, e.OccurredAtUtc }); - builder.HasIndex(e => e.EventId) - .IsUnique() - .HasDatabaseName("UX_AuditLog_EventId"); - // Index names are locked for reconciliation/migration discoverability. The // column SETS migrate to the canonical/computed shape (alog.md §4 semantics // preserved): Channel→Category, Site/Node/Execution/ParentExecution now read // off the computed columns. + // INCLUDE(Status) makes the audit KPI window query (count/aggregate by Status + // over a recent OccurredAtUtc range) a covering index seek — the persisted + // computed Status column rides along in the leaf without a key/RID lookup back + // into the clustered index per matching row (arch-review WP1.4). builder.HasIndex(e => e.OccurredAtUtc) .IsDescending(true) - .HasDatabaseName("IX_AuditLog_OccurredAtUtc"); + .HasDatabaseName("IX_AuditLog_OccurredAtUtc") + .IncludeProperties(nameof(AuditLogRow.Status)); builder.HasIndex(e => new { e.SourceSiteId, e.OccurredAtUtc }) .IsDescending(false, true) diff --git a/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Configurations/NotificationOutboxConfiguration.cs b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Configurations/NotificationOutboxConfiguration.cs index f1f9490e..b346fbed 100644 --- a/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Configurations/NotificationOutboxConfiguration.cs +++ b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Configurations/NotificationOutboxConfiguration.cs @@ -70,5 +70,18 @@ public class NotificationOutboxConfiguration : IEntityTypeConfiguration new { n.Status, n.NextAttemptAt }); builder.HasIndex(n => new { n.SourceSiteId, n.CreatedAt }); + + // Covers ComputeKpisAsync's "DeliveredLastInterval" count + // (Status == Delivered && DeliveredAt >= deliveredSince): filtered to just the + // Delivered rows so the index stays small relative to the full table and the KPI + // query can seek it directly instead of scanning every status (arch-review WP1.4). + // Status is stored as its enum name (HasConversion() above), so the filter + // predicate matches on the literal 'Delivered', not the underlying int value. + // No N'' prefix: the value is ASCII-only (fine on SQL Server's nvarchar column) + // and the repository tests' SQLite provider rejects the T-SQL N'' national-string + // literal syntax outright when EnsureCreated() runs this filter as DDL. + builder.HasIndex(n => n.DeliveredAt) + .HasDatabaseName("IX_Notifications_Delivered") + .HasFilter("[Status] = 'Delivered'"); } } diff --git a/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/20260814235335_AddAuditLogAndNotificationCoveringIndexes.Designer.cs b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/20260814235335_AddAuditLogAndNotificationCoveringIndexes.Designer.cs new file mode 100644 index 00000000..5f9c7757 --- /dev/null +++ b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/20260814235335_AddAuditLogAndNotificationCoveringIndexes.Designer.cs @@ -0,0 +1,2100 @@ +// +using System; +using Microsoft.EntityFrameworkCore; +using Microsoft.EntityFrameworkCore.Infrastructure; +using Microsoft.EntityFrameworkCore.Metadata; +using Microsoft.EntityFrameworkCore.Migrations; +using Microsoft.EntityFrameworkCore.Storage.ValueConversion; +using ZB.MOM.WW.ScadaBridge.ConfigurationDatabase; + +#nullable disable + +namespace ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Migrations +{ + [DbContext(typeof(ScadaBridgeDbContext))] + [Migration("20260814235335_AddAuditLogAndNotificationCoveringIndexes")] + partial class AddAuditLogAndNotificationCoveringIndexes + { + /// + protected override void BuildTargetModel(ModelBuilder modelBuilder) + { +#pragma warning disable 612, 618 + modelBuilder + .HasAnnotation("ProductVersion", "10.0.7") + .HasAnnotation("Relational:MaxIdentifierLength", 128); + + SqlServerModelBuilderExtensions.UseIdentityColumns(modelBuilder); + + modelBuilder.Entity("Microsoft.AspNetCore.DataProtection.EntityFrameworkCore.DataProtectionKey", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("FriendlyName") + .HasColumnType("nvarchar(max)"); + + b.Property("Xml") + .HasColumnType("nvarchar(max)"); + + b.HasKey("Id"); + + b.ToTable("DataProtectionKeys"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Audit.AuditLogEntry", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("Action") + .IsRequired() + .HasMaxLength(100) + .HasColumnType("nvarchar(100)"); + + b.Property("AfterStateJson") + .HasColumnType("nvarchar(max)"); + + b.Property("BundleImportId") + .HasColumnType("uniqueidentifier"); + + b.Property("EntityId") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("EntityName") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("EntityType") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("Timestamp") + .HasColumnType("datetimeoffset"); + + b.Property("User") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.HasKey("Id"); + + b.HasIndex("Action"); + + b.HasIndex("BundleImportId") + .HasDatabaseName("IX_AuditLogEntries_BundleImportId"); + + b.HasIndex("EntityId"); + + b.HasIndex("EntityType"); + + b.HasIndex("Timestamp"); + + b.HasIndex("User"); + + b.ToTable("AuditLogEntries"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Audit.SiteCall", b => + { + b.Property("TrackedOperationId") + .HasMaxLength(36) + .IsUnicode(false) + .HasColumnType("varchar(36)"); + + b.Property("Channel") + .IsRequired() + .HasMaxLength(32) + .IsUnicode(false) + .HasColumnType("varchar(32)"); + + b.Property("CreatedAtUtc") + .HasColumnType("datetime2"); + + b.Property("HttpStatus") + .HasColumnType("int"); + + b.Property("IngestedAtUtc") + .HasColumnType("datetime2"); + + b.Property("LastError") + .HasMaxLength(1024) + .HasColumnType("nvarchar(1024)"); + + b.Property("RetryCount") + .HasColumnType("int"); + + b.Property("SourceNode") + .HasMaxLength(64) + .IsUnicode(false) + .HasColumnType("varchar(64)"); + + b.Property("SourceSite") + .IsRequired() + .HasMaxLength(64) + .IsUnicode(false) + .HasColumnType("varchar(64)"); + + b.Property("Status") + .IsRequired() + .HasMaxLength(32) + .IsUnicode(false) + .HasColumnType("varchar(32)"); + + b.Property("Target") + .IsRequired() + .HasMaxLength(256) + .IsUnicode(false) + .HasColumnType("varchar(256)"); + + b.Property("TerminalAtUtc") + .HasColumnType("datetime2"); + + b.Property("UpdatedAtUtc") + .HasColumnType("datetime2"); + + b.HasKey("TrackedOperationId"); + + b.HasIndex("CreatedAtUtc") + .HasDatabaseName("IX_SiteCalls_NonTerminal") + .HasFilter("[TerminalAtUtc] IS NULL"); + + SqlServerIndexBuilderExtensions.IncludeProperties(b.HasIndex("CreatedAtUtc"), new[] { "SourceSite", "SourceNode", "Status" }); + + b.HasIndex("TerminalAtUtc") + .HasDatabaseName("IX_SiteCalls_Terminal") + .HasFilter("[TerminalAtUtc] IS NOT NULL"); + + b.HasIndex("SourceSite", "CreatedAtUtc") + .IsDescending(false, true) + .HasDatabaseName("IX_SiteCalls_Source_Created"); + + b.HasIndex("Status", "UpdatedAtUtc") + .IsDescending(false, true) + .HasDatabaseName("IX_SiteCalls_Status_Updated"); + + b.ToTable("SiteCalls", (string)null); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Deployment.DeployedConfigSnapshot", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("ConfigurationJson") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("DeployedAt") + .HasColumnType("datetimeoffset"); + + b.Property("DeploymentId") + .IsRequired() + .HasMaxLength(100) + .HasColumnType("nvarchar(100)"); + + b.Property("InstanceId") + .HasColumnType("int"); + + b.Property("RevisionHash") + .IsRequired() + .HasMaxLength(100) + .HasColumnType("nvarchar(100)"); + + b.HasKey("Id"); + + b.HasIndex("DeploymentId"); + + b.HasIndex("InstanceId") + .IsUnique(); + + b.ToTable("DeployedConfigSnapshots"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Deployment.DeploymentRecord", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("CompletedAt") + .HasColumnType("datetimeoffset"); + + b.Property("DeployedAt") + .HasColumnType("datetimeoffset"); + + b.Property("DeployedBy") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("DeploymentId") + .IsRequired() + .HasMaxLength(100) + .HasColumnType("nvarchar(100)"); + + b.Property("ErrorMessage") + .HasColumnType("nvarchar(max)"); + + b.Property("InstanceId") + .HasColumnType("int"); + + b.Property("RevisionHash") + .HasMaxLength(100) + .HasColumnType("nvarchar(100)"); + + b.Property("RowVersion") + .IsConcurrencyToken() + .IsRequired() + .ValueGeneratedOnAddOrUpdate() + .HasColumnType("rowversion"); + + b.Property("Status") + .IsRequired() + .HasMaxLength(50) + .HasColumnType("nvarchar(50)"); + + b.HasKey("Id"); + + b.HasIndex("DeployedAt"); + + b.HasIndex("DeploymentId") + .IsUnique(); + + b.HasIndex("InstanceId"); + + b.ToTable("DeploymentRecords"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Deployment.PendingDeployment", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("ConfigurationJson") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("CreatedAtUtc") + .HasColumnType("datetimeoffset"); + + b.Property("DeploymentId") + .IsRequired() + .HasMaxLength(100) + .HasColumnType("nvarchar(100)"); + + b.Property("ExpiresAtUtc") + .HasColumnType("datetimeoffset"); + + b.Property("InstanceId") + .HasColumnType("int"); + + b.Property("RevisionHash") + .IsRequired() + .HasMaxLength(100) + .HasColumnType("nvarchar(100)"); + + b.Property("Token") + .IsRequired() + .HasMaxLength(128) + .HasColumnType("nvarchar(128)"); + + b.HasKey("Id"); + + b.HasIndex("DeploymentId") + .IsUnique(); + + b.HasIndex("ExpiresAtUtc"); + + b.HasIndex("InstanceId"); + + b.ToTable("PendingDeployments", (string)null); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Deployment.SystemArtifactDeploymentRecord", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("ArtifactType") + .IsRequired() + .HasMaxLength(100) + .HasColumnType("nvarchar(100)"); + + b.Property("DeployedAt") + .HasColumnType("datetimeoffset"); + + b.Property("DeployedBy") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("PerSiteStatus") + .HasMaxLength(4000) + .HasColumnType("nvarchar(4000)"); + + b.HasKey("Id"); + + b.HasIndex("DeployedAt"); + + b.ToTable("SystemArtifactDeploymentRecords"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.ExternalSystems.DatabaseConnectionDefinition", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("ConnectionString") + .IsRequired() + .HasMaxLength(8000) + .HasColumnType("nvarchar(max)"); + + b.Property("MaxRetries") + .HasColumnType("int"); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("RetryDelay") + .HasColumnType("time"); + + b.HasKey("Id"); + + b.HasIndex("Name") + .IsUnique(); + + b.ToTable("DatabaseConnectionDefinitions"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.ExternalSystems.ExternalSystemDefinition", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("AuthConfiguration") + .HasMaxLength(8000) + .HasColumnType("nvarchar(max)"); + + b.Property("AuthType") + .IsRequired() + .HasMaxLength(50) + .HasColumnType("nvarchar(50)"); + + b.Property("EndpointUrl") + .IsRequired() + .HasMaxLength(2000) + .HasColumnType("nvarchar(2000)"); + + b.Property("MaxRetries") + .HasColumnType("int"); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("RetryDelay") + .HasColumnType("time"); + + b.Property("TimeoutSeconds") + .ValueGeneratedOnAdd() + .HasColumnType("int") + .HasDefaultValue(0); + + b.HasKey("Id"); + + b.HasIndex("Name") + .IsUnique(); + + b.ToTable("ExternalSystemDefinitions"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.ExternalSystems.ExternalSystemMethod", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("ExternalSystemDefinitionId") + .HasColumnType("int"); + + b.Property("HttpMethod") + .IsRequired() + .HasMaxLength(10) + .HasColumnType("nvarchar(10)"); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("ParameterDefinitions") + .HasMaxLength(4000) + .HasColumnType("nvarchar(4000)"); + + b.Property("Path") + .IsRequired() + .HasMaxLength(2000) + .HasColumnType("nvarchar(2000)"); + + b.Property("ReturnDefinition") + .HasMaxLength(4000) + .HasColumnType("nvarchar(4000)"); + + b.HasKey("Id"); + + b.HasIndex("ExternalSystemDefinitionId", "Name") + .IsUnique(); + + b.ToTable("ExternalSystemMethods"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.InboundApi.ApiMethod", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("ParameterDefinitions") + .HasMaxLength(4000) + .HasColumnType("nvarchar(4000)"); + + b.Property("ReturnDefinition") + .HasMaxLength(4000) + .HasColumnType("nvarchar(4000)"); + + b.Property("Script") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("TimeoutSeconds") + .HasColumnType("int"); + + b.HasKey("Id"); + + b.HasIndex("Name") + .IsUnique(); + + b.ToTable("ApiMethods"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.Area", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("ParentAreaId") + .HasColumnType("int"); + + b.Property("SiteId") + .HasColumnType("int"); + + b.HasKey("Id"); + + b.HasIndex("ParentAreaId"); + + b.HasIndex("SiteId", "ParentAreaId", "Name") + .IsUnique() + .HasFilter("[ParentAreaId] IS NOT NULL"); + + b.ToTable("Areas"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.Instance", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("AreaId") + .HasColumnType("int"); + + b.Property("SiteId") + .HasColumnType("int"); + + b.Property("State") + .IsRequired() + .HasMaxLength(50) + .HasColumnType("nvarchar(50)"); + + b.Property("TemplateId") + .HasColumnType("int"); + + b.Property("UniqueName") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.HasKey("Id"); + + b.HasIndex("AreaId"); + + b.HasIndex("TemplateId"); + + b.HasIndex("SiteId", "UniqueName") + .IsUnique(); + + b.ToTable("Instances"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.InstanceAlarmOverride", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("AlarmCanonicalName") + .IsRequired() + .HasMaxLength(400) + .HasColumnType("nvarchar(400)"); + + b.Property("InstanceId") + .HasColumnType("int"); + + b.Property("PriorityLevelOverride") + .HasColumnType("int"); + + b.Property("TriggerConfigurationOverride") + .HasMaxLength(4000) + .HasColumnType("nvarchar(4000)"); + + b.HasKey("Id"); + + b.HasIndex("InstanceId", "AlarmCanonicalName") + .IsUnique(); + + b.ToTable("InstanceAlarmOverrides"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.InstanceAttributeOverride", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("AttributeName") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("ElementDataType") + .HasMaxLength(50) + .HasColumnType("nvarchar(50)"); + + b.Property("InstanceId") + .HasColumnType("int"); + + b.Property("OverrideValue") + .HasColumnType("nvarchar(max)"); + + b.HasKey("Id"); + + b.HasIndex("InstanceId", "AttributeName") + .IsUnique(); + + b.ToTable("InstanceAttributeOverrides"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.InstanceConnectionBinding", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("AttributeName") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("DataConnectionId") + .HasColumnType("int"); + + b.Property("DataSourceReferenceOverride") + .HasMaxLength(512) + .HasColumnType("nvarchar(512)"); + + b.Property("InstanceId") + .HasColumnType("int"); + + b.HasKey("Id"); + + b.HasIndex("DataConnectionId"); + + b.HasIndex("InstanceId", "AttributeName") + .IsUnique(); + + b.ToTable("InstanceConnectionBindings"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.InstanceNativeAlarmSourceOverride", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("ConditionFilterOverride") + .HasMaxLength(1000) + .HasColumnType("nvarchar(1000)"); + + b.Property("ConnectionNameOverride") + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("InstanceId") + .HasColumnType("int"); + + b.Property("SourceCanonicalName") + .IsRequired() + .HasMaxLength(400) + .HasColumnType("nvarchar(400)"); + + b.Property("SourceReferenceOverride") + .HasMaxLength(1000) + .HasColumnType("nvarchar(1000)"); + + b.HasKey("Id"); + + b.HasIndex("InstanceId", "SourceCanonicalName") + .IsUnique(); + + b.ToTable("InstanceNativeAlarmSourceOverrides"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Kpi.KpiRollupHourly", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("bigint"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("HourStartUtc") + .HasColumnType("datetime2"); + + b.Property("MaxValue") + .HasColumnType("float"); + + b.Property("Metric") + .IsRequired() + .HasMaxLength(64) + .IsUnicode(false) + .HasColumnType("varchar(64)"); + + b.Property("MinValue") + .HasColumnType("float"); + + b.Property("SampleCount") + .HasColumnType("int"); + + b.Property("Scope") + .IsRequired() + .HasMaxLength(16) + .IsUnicode(false) + .HasColumnType("varchar(16)"); + + b.Property("ScopeKey") + .HasMaxLength(64) + .IsUnicode(false) + .HasColumnType("varchar(64)"); + + b.Property("Source") + .IsRequired() + .HasMaxLength(64) + .IsUnicode(false) + .HasColumnType("varchar(64)"); + + b.Property("Value") + .HasColumnType("float"); + + b.HasKey("Id"); + + b.HasIndex("HourStartUtc") + .HasDatabaseName("IX_KpiRollupHourly_Hour"); + + b.HasIndex("Source", "Metric", "Scope", "ScopeKey", "HourStartUtc") + .IsUnique() + .HasDatabaseName("IX_KpiRollupHourly_Series"); + + b.ToTable("KpiRollupHourly", (string)null); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Kpi.KpiSample", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("bigint"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("CapturedAtUtc") + .HasColumnType("datetime2"); + + b.Property("Metric") + .IsRequired() + .HasMaxLength(64) + .IsUnicode(false) + .HasColumnType("varchar(64)"); + + b.Property("Scope") + .IsRequired() + .HasMaxLength(16) + .IsUnicode(false) + .HasColumnType("varchar(16)"); + + b.Property("ScopeKey") + .HasMaxLength(64) + .IsUnicode(false) + .HasColumnType("varchar(64)"); + + b.Property("Source") + .IsRequired() + .HasMaxLength(64) + .IsUnicode(false) + .HasColumnType("varchar(64)"); + + b.Property("Value") + .HasColumnType("float"); + + b.HasKey("Id"); + + b.HasIndex("CapturedAtUtc") + .HasDatabaseName("IX_KpiSample_Captured"); + + b.HasIndex("Source", "Metric", "Scope", "ScopeKey", "CapturedAtUtc") + .HasDatabaseName("IX_KpiSample_Series"); + + b.ToTable("KpiSample", (string)null); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Notifications.Notification", b => + { + b.Property("NotificationId") + .HasMaxLength(64) + .HasColumnType("nvarchar(64)"); + + b.Property("Body") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("CreatedAt") + .HasColumnType("datetimeoffset"); + + b.Property("DeliveredAt") + .HasColumnType("datetimeoffset"); + + b.Property("LastAttemptAt") + .HasColumnType("datetimeoffset"); + + b.Property("LastError") + .HasMaxLength(4000) + .HasColumnType("nvarchar(4000)"); + + b.Property("ListName") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("NextAttemptAt") + .HasColumnType("datetimeoffset"); + + b.Property("OriginExecutionId") + .HasColumnType("uniqueidentifier"); + + b.Property("OriginParentExecutionId") + .HasColumnType("uniqueidentifier"); + + b.Property("ResolvedTargets") + .HasColumnType("nvarchar(max)"); + + b.Property("RetryCount") + .HasColumnType("int"); + + b.Property("SiteEnqueuedAt") + .HasColumnType("datetimeoffset"); + + b.Property("SourceInstanceId") + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("SourceNode") + .HasMaxLength(64) + .IsUnicode(false) + .HasColumnType("varchar(64)"); + + b.Property("SourceScript") + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("SourceSiteId") + .IsRequired() + .HasMaxLength(100) + .HasColumnType("nvarchar(100)"); + + b.Property("Status") + .IsRequired() + .HasMaxLength(32) + .HasColumnType("nvarchar(32)"); + + b.Property("Subject") + .IsRequired() + .HasMaxLength(1000) + .HasColumnType("nvarchar(1000)"); + + b.Property("Type") + .IsRequired() + .HasMaxLength(32) + .HasColumnType("nvarchar(32)"); + + b.Property("TypeData") + .HasColumnType("nvarchar(max)"); + + b.HasKey("NotificationId"); + + b.HasIndex("DeliveredAt") + .HasDatabaseName("IX_Notifications_Delivered") + .HasFilter("[Status] = 'Delivered'"); + + b.HasIndex("SourceSiteId", "CreatedAt"); + + b.HasIndex("Status", "NextAttemptAt"); + + b.ToTable("Notifications"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Notifications.NotificationList", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("Type") + .IsRequired() + .HasMaxLength(32) + .HasColumnType("nvarchar(32)"); + + b.HasKey("Id"); + + b.HasIndex("Name") + .IsUnique(); + + b.ToTable("NotificationLists"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Notifications.NotificationRecipient", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("EmailAddress") + .HasMaxLength(500) + .HasColumnType("nvarchar(500)"); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("NotificationListId") + .HasColumnType("int"); + + b.Property("PhoneNumber") + .HasMaxLength(32) + .HasColumnType("nvarchar(32)"); + + b.HasKey("Id"); + + b.HasIndex("NotificationListId"); + + b.ToTable("NotificationRecipients"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Notifications.SmsConfiguration", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("AccountSid") + .IsRequired() + .HasMaxLength(100) + .HasColumnType("nvarchar(100)"); + + b.Property("ApiBaseUrl") + .HasMaxLength(500) + .HasColumnType("nvarchar(500)"); + + b.Property("AuthToken") + .HasMaxLength(8000) + .HasColumnType("nvarchar(max)"); + + b.Property("ConnectionTimeoutSeconds") + .HasColumnType("int"); + + b.Property("FromNumber") + .HasMaxLength(32) + .HasColumnType("nvarchar(32)"); + + b.Property("MaxRetries") + .HasColumnType("int"); + + b.Property("MessagingServiceSid") + .HasMaxLength(100) + .HasColumnType("nvarchar(100)"); + + b.Property("RetryDelay") + .HasColumnType("time"); + + b.HasKey("Id"); + + b.ToTable("SmsConfigurations"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Notifications.SmtpConfiguration", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("AuthType") + .IsRequired() + .HasMaxLength(50) + .HasColumnType("nvarchar(50)"); + + b.Property("ConnectionTimeoutSeconds") + .HasColumnType("int"); + + b.Property("Credentials") + .HasMaxLength(8000) + .HasColumnType("nvarchar(max)"); + + b.Property("FromAddress") + .IsRequired() + .HasMaxLength(500) + .HasColumnType("nvarchar(500)"); + + b.Property("Host") + .IsRequired() + .HasMaxLength(500) + .HasColumnType("nvarchar(500)"); + + b.Property("MaxConcurrentConnections") + .HasColumnType("int"); + + b.Property("MaxRetries") + .HasColumnType("int"); + + b.Property("OAuth2Authority") + .HasMaxLength(500) + .HasColumnType("nvarchar(500)"); + + b.Property("OAuth2Scope") + .HasMaxLength(500) + .HasColumnType("nvarchar(500)"); + + b.Property("Port") + .HasColumnType("int"); + + b.Property("RetryDelay") + .HasColumnType("time"); + + b.Property("TlsMode") + .HasMaxLength(50) + .HasColumnType("nvarchar(50)"); + + b.Property("Transport") + .HasMaxLength(50) + .HasColumnType("nvarchar(50)"); + + b.HasKey("Id"); + + b.ToTable("SmtpConfigurations"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Schemas.SharedSchema", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("SchemaJson") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("Scope") + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.HasKey("Id"); + + b.HasIndex("Name") + .IsUnique(); + + b.ToTable("SharedSchemas"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Scripts.SharedScript", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("Code") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("ParameterDefinitions") + .HasMaxLength(4000) + .HasColumnType("nvarchar(4000)"); + + b.Property("ReturnDefinition") + .HasMaxLength(4000) + .HasColumnType("nvarchar(4000)"); + + b.HasKey("Id"); + + b.HasIndex("Name") + .IsUnique(); + + b.ToTable("SharedScripts"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.SecuredWrites.PendingSecuredWrite", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("bigint"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("ConnectionName") + .IsRequired() + .HasMaxLength(128) + .IsUnicode(false) + .HasColumnType("varchar(128)"); + + b.Property("DecidedAtUtc") + .HasColumnType("datetime2"); + + b.Property("ExecutedAtUtc") + .HasColumnType("datetime2"); + + b.Property("ExecutionError") + .HasMaxLength(2048) + .IsUnicode(false) + .HasColumnType("varchar(2048)"); + + b.Property("OperatorComment") + .HasMaxLength(1024) + .IsUnicode(false) + .HasColumnType("varchar(1024)"); + + b.Property("OperatorUser") + .IsRequired() + .HasMaxLength(256) + .IsUnicode(false) + .HasColumnType("varchar(256)"); + + b.Property("SiteId") + .IsRequired() + .HasMaxLength(128) + .IsUnicode(false) + .HasColumnType("varchar(128)"); + + b.Property("Status") + .IsRequired() + .HasMaxLength(32) + .IsUnicode(false) + .HasColumnType("varchar(32)"); + + b.Property("SubmittedAtUtc") + .HasColumnType("datetime2"); + + b.Property("TagPath") + .IsRequired() + .HasMaxLength(512) + .IsUnicode(false) + .HasColumnType("varchar(512)"); + + b.Property("ValueJson") + .IsRequired() + .HasMaxLength(4000) + .IsUnicode(false) + .HasColumnType("varchar(4000)"); + + b.Property("ValueType") + .IsRequired() + .HasMaxLength(128) + .IsUnicode(false) + .HasColumnType("varchar(128)"); + + b.Property("VerifierComment") + .HasMaxLength(1024) + .IsUnicode(false) + .HasColumnType("varchar(1024)"); + + b.Property("VerifierUser") + .HasMaxLength(256) + .IsUnicode(false) + .HasColumnType("varchar(256)"); + + b.HasKey("Id"); + + b.HasIndex("SiteId") + .HasDatabaseName("IX_PendingSecuredWrites_Site"); + + b.HasIndex("Status", "SubmittedAtUtc") + .HasDatabaseName("IX_PendingSecuredWrites_Status_Submitted"); + + b.ToTable("PendingSecuredWrites", (string)null); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Security.LdapGroupMapping", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("LdapGroupName") + .IsRequired() + .HasMaxLength(500) + .HasColumnType("nvarchar(500)"); + + b.Property("Role") + .IsRequired() + .HasMaxLength(100) + .HasColumnType("nvarchar(100)"); + + b.HasKey("Id"); + + b.HasIndex("LdapGroupName") + .IsUnique(); + + b.ToTable("LdapGroupMappings"); + + b.HasData( + new + { + Id = 1, + LdapGroupName = "SCADA-Admins", + Role = "Administrator" + }, + new + { + Id = 2, + LdapGroupName = "SCADA-Designers", + Role = "Designer" + }, + new + { + Id = 3, + LdapGroupName = "SCADA-Deploy-All", + Role = "Deployer" + }, + new + { + Id = 4, + LdapGroupName = "SCADA-Deploy-SiteA", + Role = "Deployer" + }, + new + { + Id = 5, + LdapGroupName = "SCADA-Viewers", + Role = "Viewer" + }); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Security.SiteScopeRule", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("LdapGroupMappingId") + .HasColumnType("int"); + + b.Property("SiteId") + .HasColumnType("int"); + + b.HasKey("Id"); + + b.HasIndex("SiteId"); + + b.HasIndex("LdapGroupMappingId", "SiteId") + .IsUnique(); + + b.ToTable("SiteScopeRules"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites.DataConnection", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("BackupConfiguration") + .HasMaxLength(4000) + .HasColumnType("nvarchar(4000)"); + + b.Property("FailoverRetryCount") + .ValueGeneratedOnAdd() + .HasColumnType("int") + .HasDefaultValue(3); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("PrimaryConfiguration") + .HasMaxLength(4000) + .HasColumnType("nvarchar(4000)"); + + b.Property("Protocol") + .IsRequired() + .HasMaxLength(50) + .HasColumnType("nvarchar(50)"); + + b.Property("SiteId") + .HasColumnType("int"); + + b.HasKey("Id"); + + b.HasIndex("SiteId", "Name") + .IsUnique(); + + b.ToTable("DataConnections"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites.Site", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("Description") + .HasMaxLength(2000) + .HasColumnType("nvarchar(2000)"); + + b.Property("GrpcNodeAAddress") + .HasMaxLength(500) + .HasColumnType("nvarchar(500)"); + + b.Property("GrpcNodeBAddress") + .HasMaxLength(500) + .HasColumnType("nvarchar(500)"); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("NodeAAddress") + .HasMaxLength(500) + .HasColumnType("nvarchar(500)"); + + b.Property("NodeBAddress") + .HasMaxLength(500) + .HasColumnType("nvarchar(500)"); + + b.Property("SiteIdentifier") + .IsRequired() + .HasMaxLength(100) + .HasColumnType("nvarchar(100)"); + + b.HasKey("Id"); + + b.HasIndex("Name") + .IsUnique(); + + b.HasIndex("SiteIdentifier") + .IsUnique(); + + b.ToTable("Sites"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.Template", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("Description") + .HasMaxLength(2000) + .HasColumnType("nvarchar(2000)"); + + b.Property("FolderId") + .HasColumnType("int"); + + b.Property("IsDerived") + .HasColumnType("bit"); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("OwnerCompositionId") + .HasColumnType("int"); + + b.Property("ParentTemplateId") + .HasColumnType("int"); + + b.HasKey("Id"); + + b.HasIndex("FolderId"); + + b.HasIndex("Name") + .IsUnique() + .HasFilter("[IsDerived] = 0"); + + b.HasIndex("ParentTemplateId"); + + b.ToTable("Templates"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.TemplateAlarm", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("Description") + .HasMaxLength(2000) + .HasColumnType("nvarchar(2000)"); + + b.Property("IsInherited") + .HasColumnType("bit"); + + b.Property("IsLocked") + .HasColumnType("bit"); + + b.Property("LockedInDerived") + .HasColumnType("bit"); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("OnTriggerScriptId") + .HasColumnType("int"); + + b.Property("PriorityLevel") + .HasColumnType("int"); + + b.Property("TemplateId") + .HasColumnType("int"); + + b.Property("TriggerConfiguration") + .HasMaxLength(4000) + .HasColumnType("nvarchar(4000)"); + + b.Property("TriggerType") + .IsRequired() + .HasMaxLength(50) + .HasColumnType("nvarchar(50)"); + + b.HasKey("Id"); + + b.HasIndex("TemplateId", "Name") + .IsUnique(); + + b.ToTable("TemplateAlarms"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.TemplateAttribute", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("DataSourceReference") + .HasMaxLength(500) + .HasColumnType("nvarchar(500)"); + + b.Property("DataType") + .IsRequired() + .HasMaxLength(50) + .HasColumnType("nvarchar(50)"); + + b.Property("Description") + .HasMaxLength(2000) + .HasColumnType("nvarchar(2000)"); + + b.Property("ElementDataType") + .HasMaxLength(50) + .HasColumnType("nvarchar(50)"); + + b.Property("IsInherited") + .HasColumnType("bit"); + + b.Property("IsLocked") + .HasColumnType("bit"); + + b.Property("LockedInDerived") + .HasColumnType("bit"); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("TemplateId") + .HasColumnType("int"); + + b.Property("Value") + .HasColumnType("nvarchar(max)"); + + b.HasKey("Id"); + + b.HasIndex("TemplateId", "Name") + .IsUnique(); + + b.ToTable("TemplateAttributes"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.TemplateComposition", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("ComposedTemplateId") + .HasColumnType("int"); + + b.Property("InstanceName") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("TemplateId") + .HasColumnType("int"); + + b.HasKey("Id"); + + b.HasIndex("ComposedTemplateId"); + + b.HasIndex("TemplateId", "InstanceName") + .IsUnique(); + + b.ToTable("TemplateCompositions"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.TemplateFolder", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("ParentFolderId") + .HasColumnType("int"); + + b.Property("SortOrder") + .HasColumnType("int"); + + b.HasKey("Id"); + + b.HasIndex("ParentFolderId", "Name") + .IsUnique() + .HasFilter("[ParentFolderId] IS NOT NULL"); + + b.ToTable("TemplateFolders"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.TemplateNativeAlarmSource", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("ConditionFilter") + .HasMaxLength(1000) + .HasColumnType("nvarchar(1000)"); + + b.Property("ConnectionName") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("Description") + .HasMaxLength(2000) + .HasColumnType("nvarchar(2000)"); + + b.Property("IsInherited") + .HasColumnType("bit"); + + b.Property("IsLocked") + .HasColumnType("bit"); + + b.Property("LockedInDerived") + .HasColumnType("bit"); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("SourceReference") + .IsRequired() + .HasMaxLength(1000) + .HasColumnType("nvarchar(1000)"); + + b.Property("TemplateId") + .HasColumnType("int"); + + b.HasKey("Id"); + + b.HasIndex("TemplateId", "Name") + .IsUnique(); + + b.ToTable("TemplateNativeAlarmSources"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.TemplateScript", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("Code") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("ExecutionTimeoutSeconds") + .HasColumnType("int"); + + b.Property("IsInherited") + .HasColumnType("bit"); + + b.Property("IsLocked") + .HasColumnType("bit"); + + b.Property("LockedInDerived") + .HasColumnType("bit"); + + b.Property("MinTimeBetweenRuns") + .HasColumnType("time"); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("ParameterDefinitions") + .HasMaxLength(4000) + .HasColumnType("nvarchar(4000)"); + + b.Property("ReturnDefinition") + .HasMaxLength(4000) + .HasColumnType("nvarchar(4000)"); + + b.Property("TemplateId") + .HasColumnType("int"); + + b.Property("TriggerConfiguration") + .HasMaxLength(4000) + .HasColumnType("nvarchar(4000)"); + + b.Property("TriggerType") + .HasMaxLength(50) + .HasColumnType("nvarchar(50)"); + + b.HasKey("Id"); + + b.HasIndex("TemplateId", "Name") + .IsUnique(); + + b.ToTable("TemplateScripts"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Entities.AuditLogRow", b => + { + b.Property("EventId") + .HasColumnType("uniqueidentifier"); + + b.Property("OccurredAtUtc") + .HasColumnType("datetime2"); + + b.Property("Action") + .IsRequired() + .HasMaxLength(64) + .IsUnicode(false) + .HasColumnType("varchar(64)"); + + b.Property("Actor") + .HasMaxLength(256) + .HasColumnType("nvarchar(256)"); + + b.Property("Channel") + .IsRequired() + .HasMaxLength(32) + .IsUnicode(false) + .HasColumnType("varchar(32)") + .HasColumnName("Category"); + + b.Property("CorrelationId") + .HasColumnType("uniqueidentifier"); + + b.Property("DetailsJson") + .HasColumnType("nvarchar(max)"); + + b.Property("ExecutionId") + .ValueGeneratedOnAddOrUpdate() + .HasColumnType("uniqueidentifier") + .HasComputedColumnSql("CAST(JSON_VALUE(DetailsJson,'$.executionId') AS uniqueidentifier)", true); + + b.Property("IngestedAtUtc") + .ValueGeneratedOnAddOrUpdate() + .HasColumnType("datetime2(7)") + .HasComputedColumnSql("CAST(SWITCHOFFSET(CAST(JSON_VALUE(DetailsJson,'$.ingestedAtUtc') AS datetimeoffset), 0) AS datetime2(7))", false); + + b.Property("Kind") + .IsRequired() + .ValueGeneratedOnAddOrUpdate() + .HasMaxLength(32) + .IsUnicode(false) + .HasColumnType("varchar(32)") + .HasComputedColumnSql("JSON_VALUE(DetailsJson,'$.kind')", true); + + b.Property("Outcome") + .IsRequired() + .HasMaxLength(16) + .IsUnicode(false) + .HasColumnType("varchar(16)"); + + b.Property("ParentExecutionId") + .ValueGeneratedOnAddOrUpdate() + .HasColumnType("uniqueidentifier") + .HasComputedColumnSql("CAST(JSON_VALUE(DetailsJson,'$.parentExecutionId') AS uniqueidentifier)", true); + + b.Property("SourceNode") + .HasMaxLength(64) + .IsUnicode(false) + .HasColumnType("varchar(64)"); + + b.Property("SourceSiteId") + .ValueGeneratedOnAddOrUpdate() + .HasMaxLength(64) + .IsUnicode(false) + .HasColumnType("varchar(64)") + .HasComputedColumnSql("JSON_VALUE(DetailsJson,'$.sourceSiteId')", true); + + b.Property("Status") + .IsRequired() + .ValueGeneratedOnAddOrUpdate() + .HasMaxLength(32) + .IsUnicode(false) + .HasColumnType("varchar(32)") + .HasComputedColumnSql("JSON_VALUE(DetailsJson,'$.status')", true); + + b.Property("Target") + .HasMaxLength(256) + .HasColumnType("nvarchar(256)"); + + b.HasKey("EventId", "OccurredAtUtc"); + + b.HasIndex("CorrelationId") + .HasDatabaseName("IX_AuditLog_CorrelationId") + .HasFilter("[CorrelationId] IS NOT NULL"); + + b.HasIndex("EventId") + .IsUnique() + .HasDatabaseName("UX_AuditLog_EventId"); + + b.HasIndex("ExecutionId") + .HasDatabaseName("IX_AuditLog_Execution"); + + b.HasIndex("OccurredAtUtc") + .IsDescending() + .HasDatabaseName("IX_AuditLog_OccurredAtUtc"); + + SqlServerIndexBuilderExtensions.IncludeProperties(b.HasIndex("OccurredAtUtc"), new[] { "Status" }); + + b.HasIndex("ParentExecutionId") + .HasDatabaseName("IX_AuditLog_ParentExecution"); + + b.HasIndex("SourceNode", "OccurredAtUtc") + .HasDatabaseName("IX_AuditLog_Node_Occurred"); + + b.HasIndex("SourceSiteId", "OccurredAtUtc") + .IsDescending(false, true) + .HasDatabaseName("IX_AuditLog_Site_Occurred"); + + b.HasIndex("Target", "OccurredAtUtc") + .IsDescending(false, true) + .HasDatabaseName("IX_AuditLog_Target_Occurred") + .HasFilter("[Target] IS NOT NULL"); + + b.HasIndex("Channel", "Status", "OccurredAtUtc") + .IsDescending(false, false, true) + .HasDatabaseName("IX_AuditLog_Channel_Status_Occurred"); + + b.ToTable("AuditLog", (string)null); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Deployment.DeployedConfigSnapshot", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.Instance", null) + .WithMany() + .HasForeignKey("InstanceId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Deployment.DeploymentRecord", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.Instance", null) + .WithMany() + .HasForeignKey("InstanceId") + .OnDelete(DeleteBehavior.Restrict) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Deployment.PendingDeployment", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.Instance", null) + .WithMany() + .HasForeignKey("InstanceId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.ExternalSystems.ExternalSystemMethod", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.ExternalSystems.ExternalSystemDefinition", null) + .WithMany() + .HasForeignKey("ExternalSystemDefinitionId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.Area", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.Area", null) + .WithMany("Children") + .HasForeignKey("ParentAreaId") + .OnDelete(DeleteBehavior.Restrict); + + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites.Site", null) + .WithMany() + .HasForeignKey("SiteId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.Instance", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.Area", null) + .WithMany() + .HasForeignKey("AreaId") + .OnDelete(DeleteBehavior.SetNull); + + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites.Site", null) + .WithMany() + .HasForeignKey("SiteId") + .OnDelete(DeleteBehavior.Restrict) + .IsRequired(); + + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.Template", null) + .WithMany() + .HasForeignKey("TemplateId") + .OnDelete(DeleteBehavior.Restrict) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.InstanceAlarmOverride", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.Instance", null) + .WithMany("AlarmOverrides") + .HasForeignKey("InstanceId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.InstanceAttributeOverride", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.Instance", null) + .WithMany("AttributeOverrides") + .HasForeignKey("InstanceId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.InstanceConnectionBinding", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites.DataConnection", null) + .WithMany() + .HasForeignKey("DataConnectionId") + .OnDelete(DeleteBehavior.Restrict) + .IsRequired(); + + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.Instance", null) + .WithMany("ConnectionBindings") + .HasForeignKey("InstanceId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.InstanceNativeAlarmSourceOverride", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.Instance", null) + .WithMany("NativeAlarmSourceOverrides") + .HasForeignKey("InstanceId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Notifications.NotificationRecipient", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Notifications.NotificationList", null) + .WithMany("Recipients") + .HasForeignKey("NotificationListId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Security.SiteScopeRule", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Security.LdapGroupMapping", null) + .WithMany() + .HasForeignKey("LdapGroupMappingId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites.Site", null) + .WithMany() + .HasForeignKey("SiteId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites.DataConnection", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites.Site", null) + .WithMany() + .HasForeignKey("SiteId") + .OnDelete(DeleteBehavior.Restrict) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.Template", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.TemplateFolder", null) + .WithMany() + .HasForeignKey("FolderId") + .OnDelete(DeleteBehavior.Restrict); + + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.Template", null) + .WithMany() + .HasForeignKey("ParentTemplateId") + .OnDelete(DeleteBehavior.Restrict); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.TemplateAlarm", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.Template", null) + .WithMany("Alarms") + .HasForeignKey("TemplateId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.TemplateAttribute", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.Template", null) + .WithMany("Attributes") + .HasForeignKey("TemplateId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.TemplateComposition", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.Template", null) + .WithMany() + .HasForeignKey("ComposedTemplateId") + .OnDelete(DeleteBehavior.Restrict) + .IsRequired(); + + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.Template", null) + .WithMany("Compositions") + .HasForeignKey("TemplateId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.TemplateFolder", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.TemplateFolder", null) + .WithMany() + .HasForeignKey("ParentFolderId") + .OnDelete(DeleteBehavior.Restrict); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.TemplateNativeAlarmSource", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.Template", null) + .WithMany("NativeAlarmSources") + .HasForeignKey("TemplateId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.TemplateScript", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.Template", null) + .WithMany("Scripts") + .HasForeignKey("TemplateId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.Area", b => + { + b.Navigation("Children"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.Instance", b => + { + b.Navigation("AlarmOverrides"); + + b.Navigation("AttributeOverrides"); + + b.Navigation("ConnectionBindings"); + + b.Navigation("NativeAlarmSourceOverrides"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Notifications.NotificationList", b => + { + b.Navigation("Recipients"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.Template", b => + { + b.Navigation("Alarms"); + + b.Navigation("Attributes"); + + b.Navigation("Compositions"); + + b.Navigation("NativeAlarmSources"); + + b.Navigation("Scripts"); + }); +#pragma warning restore 612, 618 + } + } +} diff --git a/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/20260814235335_AddAuditLogAndNotificationCoveringIndexes.cs b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/20260814235335_AddAuditLogAndNotificationCoveringIndexes.cs new file mode 100644 index 00000000..b09088d6 --- /dev/null +++ b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/20260814235335_AddAuditLogAndNotificationCoveringIndexes.cs @@ -0,0 +1,65 @@ +using Microsoft.EntityFrameworkCore.Migrations; + +#nullable disable + +namespace ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Migrations +{ + /// + /// Two covering indexes (arch-review WP1.4): IX_AuditLog_OccurredAtUtc gains + /// INCLUDE (Status) so the audit KPI window query (count/aggregate by Status + /// over a recent OccurredAtUtc range) can seek without a lookup back into the + /// clustered index per row; IX_Notifications_Delivered is a new filtered index + /// backing the notification outbox KPI's delivered-last-interval count. + /// + /// + /// IX_AuditLog_OccurredAtUtc is partition-aligned + /// (ON ps_AuditLog_Month(OccurredAtUtc), see CollapseAuditLogToCanonical) + /// so the monthly partition-switch purge keeps touching a single partition. EF's + /// generic DropIndex/CreateIndex migration builders have no concept of + /// partition schemes and would silently rebuild the index off [PRIMARY] instead + /// of the scheme — this migration uses raw SQL for that index (mirroring + /// CollapseAuditLogToCanonical's DDL) to preserve the alignment. The scaffolded + /// EF model itself carries no partition annotation either way (pre-existing gap, same + /// as every other AuditLog index) — only the physical DDL below matters. + /// Notifications is not partitioned, so its index uses the ordinary EF builder. + /// The filter predicate is a plain 'Delivered' literal (no N'' prefix): + /// the value is ASCII-only, and the ConfigurationDatabase repository tests run this + /// same filter as DDL against the SQLite provider via EnsureCreated(), which + /// rejects the T-SQL national-string-literal syntax outright. + /// + public partial class AddAuditLogAndNotificationCoveringIndexes : Migration + { + /// + protected override void Up(MigrationBuilder migrationBuilder) + { + migrationBuilder.CreateIndex( + name: "IX_Notifications_Delivered", + table: "Notifications", + column: "DeliveredAt", + filter: "[Status] = 'Delivered'"); + + migrationBuilder.Sql(@" +DROP INDEX IX_AuditLog_OccurredAtUtc ON dbo.AuditLog; + +CREATE NONCLUSTERED INDEX IX_AuditLog_OccurredAtUtc +ON dbo.AuditLog (OccurredAtUtc DESC) +INCLUDE (Status) +ON ps_AuditLog_Month(OccurredAtUtc);"); + } + + /// + protected override void Down(MigrationBuilder migrationBuilder) + { + migrationBuilder.DropIndex( + name: "IX_Notifications_Delivered", + table: "Notifications"); + + migrationBuilder.Sql(@" +DROP INDEX IX_AuditLog_OccurredAtUtc ON dbo.AuditLog; + +CREATE NONCLUSTERED INDEX IX_AuditLog_OccurredAtUtc +ON dbo.AuditLog (OccurredAtUtc DESC) +ON ps_AuditLog_Month(OccurredAtUtc);"); + } + } +} diff --git a/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/20260815004957_AlignAuditLogEventIdUniqueness.Designer.cs b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/20260815004957_AlignAuditLogEventIdUniqueness.Designer.cs new file mode 100644 index 00000000..fc39c29b --- /dev/null +++ b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/20260815004957_AlignAuditLogEventIdUniqueness.Designer.cs @@ -0,0 +1,2090 @@ +// +using System; +using Microsoft.EntityFrameworkCore; +using Microsoft.EntityFrameworkCore.Infrastructure; +using Microsoft.EntityFrameworkCore.Metadata; +using Microsoft.EntityFrameworkCore.Migrations; +using Microsoft.EntityFrameworkCore.Storage.ValueConversion; +using ZB.MOM.WW.ScadaBridge.ConfigurationDatabase; + +#nullable disable + +namespace ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Migrations +{ + [DbContext(typeof(ScadaBridgeDbContext))] + [Migration("20260815004957_AlignAuditLogEventIdUniqueness")] + partial class AlignAuditLogEventIdUniqueness + { + /// + protected override void BuildTargetModel(ModelBuilder modelBuilder) + { +#pragma warning disable 612, 618 + modelBuilder + .HasAnnotation("ProductVersion", "10.0.7") + .HasAnnotation("Relational:MaxIdentifierLength", 128); + + SqlServerModelBuilderExtensions.UseIdentityColumns(modelBuilder); + + modelBuilder.Entity("Microsoft.AspNetCore.DataProtection.EntityFrameworkCore.DataProtectionKey", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("FriendlyName") + .HasColumnType("nvarchar(max)"); + + b.Property("Xml") + .HasColumnType("nvarchar(max)"); + + b.HasKey("Id"); + + b.ToTable("DataProtectionKeys"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Audit.AuditLogEntry", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("Action") + .IsRequired() + .HasMaxLength(100) + .HasColumnType("nvarchar(100)"); + + b.Property("AfterStateJson") + .HasColumnType("nvarchar(max)"); + + b.Property("BundleImportId") + .HasColumnType("uniqueidentifier"); + + b.Property("EntityId") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("EntityName") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("EntityType") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("Timestamp") + .HasColumnType("datetimeoffset"); + + b.Property("User") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.HasKey("Id"); + + b.HasIndex("Action"); + + b.HasIndex("BundleImportId") + .HasDatabaseName("IX_AuditLogEntries_BundleImportId"); + + b.HasIndex("EntityId"); + + b.HasIndex("EntityType"); + + b.HasIndex("Timestamp"); + + b.HasIndex("User"); + + b.ToTable("AuditLogEntries"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Audit.SiteCall", b => + { + b.Property("TrackedOperationId") + .HasMaxLength(36) + .IsUnicode(false) + .HasColumnType("varchar(36)"); + + b.Property("Channel") + .IsRequired() + .HasMaxLength(32) + .IsUnicode(false) + .HasColumnType("varchar(32)"); + + b.Property("CreatedAtUtc") + .HasColumnType("datetime2"); + + b.Property("HttpStatus") + .HasColumnType("int"); + + b.Property("IngestedAtUtc") + .HasColumnType("datetime2"); + + b.Property("LastError") + .HasMaxLength(1024) + .HasColumnType("nvarchar(1024)"); + + b.Property("RetryCount") + .HasColumnType("int"); + + b.Property("SourceNode") + .HasMaxLength(64) + .IsUnicode(false) + .HasColumnType("varchar(64)"); + + b.Property("SourceSite") + .IsRequired() + .HasMaxLength(64) + .IsUnicode(false) + .HasColumnType("varchar(64)"); + + b.Property("Status") + .IsRequired() + .HasMaxLength(32) + .IsUnicode(false) + .HasColumnType("varchar(32)"); + + b.Property("Target") + .IsRequired() + .HasMaxLength(256) + .IsUnicode(false) + .HasColumnType("varchar(256)"); + + b.Property("TerminalAtUtc") + .HasColumnType("datetime2"); + + b.Property("UpdatedAtUtc") + .HasColumnType("datetime2"); + + b.HasKey("TrackedOperationId"); + + b.HasIndex("CreatedAtUtc") + .HasDatabaseName("IX_SiteCalls_NonTerminal") + .HasFilter("[TerminalAtUtc] IS NULL"); + + SqlServerIndexBuilderExtensions.IncludeProperties(b.HasIndex("CreatedAtUtc"), new[] { "SourceSite", "SourceNode", "Status" }); + + b.HasIndex("TerminalAtUtc") + .HasDatabaseName("IX_SiteCalls_Terminal") + .HasFilter("[TerminalAtUtc] IS NOT NULL"); + + b.HasIndex("SourceSite", "CreatedAtUtc") + .IsDescending(false, true) + .HasDatabaseName("IX_SiteCalls_Source_Created"); + + b.HasIndex("Status", "UpdatedAtUtc") + .IsDescending(false, true) + .HasDatabaseName("IX_SiteCalls_Status_Updated"); + + b.ToTable("SiteCalls", (string)null); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Deployment.DeployedConfigSnapshot", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("ConfigurationJson") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("DeployedAt") + .HasColumnType("datetimeoffset"); + + b.Property("DeploymentId") + .IsRequired() + .HasMaxLength(100) + .HasColumnType("nvarchar(100)"); + + b.Property("InstanceId") + .HasColumnType("int"); + + b.Property("RevisionHash") + .IsRequired() + .HasMaxLength(100) + .HasColumnType("nvarchar(100)"); + + b.HasKey("Id"); + + b.HasIndex("DeploymentId"); + + b.HasIndex("InstanceId") + .IsUnique(); + + b.ToTable("DeployedConfigSnapshots"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Deployment.DeploymentRecord", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("CompletedAt") + .HasColumnType("datetimeoffset"); + + b.Property("DeployedAt") + .HasColumnType("datetimeoffset"); + + b.Property("DeployedBy") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("DeploymentId") + .IsRequired() + .HasMaxLength(100) + .HasColumnType("nvarchar(100)"); + + b.Property("ErrorMessage") + .HasColumnType("nvarchar(max)"); + + b.Property("InstanceId") + .HasColumnType("int"); + + b.Property("RevisionHash") + .HasMaxLength(100) + .HasColumnType("nvarchar(100)"); + + b.Property("RowVersion") + .IsConcurrencyToken() + .IsRequired() + .ValueGeneratedOnAddOrUpdate() + .HasColumnType("rowversion"); + + b.Property("Status") + .IsRequired() + .HasMaxLength(50) + .HasColumnType("nvarchar(50)"); + + b.HasKey("Id"); + + b.HasIndex("DeployedAt"); + + b.HasIndex("DeploymentId") + .IsUnique(); + + b.HasIndex("InstanceId"); + + b.ToTable("DeploymentRecords"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Deployment.PendingDeployment", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("ConfigurationJson") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("CreatedAtUtc") + .HasColumnType("datetimeoffset"); + + b.Property("DeploymentId") + .IsRequired() + .HasMaxLength(100) + .HasColumnType("nvarchar(100)"); + + b.Property("ExpiresAtUtc") + .HasColumnType("datetimeoffset"); + + b.Property("InstanceId") + .HasColumnType("int"); + + b.Property("RevisionHash") + .IsRequired() + .HasMaxLength(100) + .HasColumnType("nvarchar(100)"); + + b.Property("Token") + .IsRequired() + .HasMaxLength(128) + .HasColumnType("nvarchar(128)"); + + b.HasKey("Id"); + + b.HasIndex("DeploymentId") + .IsUnique(); + + b.HasIndex("ExpiresAtUtc"); + + b.HasIndex("InstanceId"); + + b.ToTable("PendingDeployments", (string)null); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Deployment.SystemArtifactDeploymentRecord", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("ArtifactType") + .IsRequired() + .HasMaxLength(100) + .HasColumnType("nvarchar(100)"); + + b.Property("DeployedAt") + .HasColumnType("datetimeoffset"); + + b.Property("DeployedBy") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("PerSiteStatus") + .HasMaxLength(4000) + .HasColumnType("nvarchar(4000)"); + + b.HasKey("Id"); + + b.HasIndex("DeployedAt"); + + b.ToTable("SystemArtifactDeploymentRecords"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.ExternalSystems.DatabaseConnectionDefinition", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("ConnectionString") + .IsRequired() + .HasMaxLength(8000) + .HasColumnType("nvarchar(max)"); + + b.Property("MaxRetries") + .HasColumnType("int"); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("RetryDelay") + .HasColumnType("time"); + + b.HasKey("Id"); + + b.HasIndex("Name") + .IsUnique(); + + b.ToTable("DatabaseConnectionDefinitions"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.ExternalSystems.ExternalSystemDefinition", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("AuthConfiguration") + .HasMaxLength(8000) + .HasColumnType("nvarchar(max)"); + + b.Property("AuthType") + .IsRequired() + .HasMaxLength(50) + .HasColumnType("nvarchar(50)"); + + b.Property("EndpointUrl") + .IsRequired() + .HasMaxLength(2000) + .HasColumnType("nvarchar(2000)"); + + b.Property("MaxRetries") + .HasColumnType("int"); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("RetryDelay") + .HasColumnType("time"); + + b.Property("TimeoutSeconds") + .ValueGeneratedOnAdd() + .HasColumnType("int") + .HasDefaultValue(0); + + b.HasKey("Id"); + + b.HasIndex("Name") + .IsUnique(); + + b.ToTable("ExternalSystemDefinitions"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.ExternalSystems.ExternalSystemMethod", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("ExternalSystemDefinitionId") + .HasColumnType("int"); + + b.Property("HttpMethod") + .IsRequired() + .HasMaxLength(10) + .HasColumnType("nvarchar(10)"); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("ParameterDefinitions") + .HasMaxLength(4000) + .HasColumnType("nvarchar(4000)"); + + b.Property("Path") + .IsRequired() + .HasMaxLength(2000) + .HasColumnType("nvarchar(2000)"); + + b.Property("ReturnDefinition") + .HasMaxLength(4000) + .HasColumnType("nvarchar(4000)"); + + b.HasKey("Id"); + + b.HasIndex("ExternalSystemDefinitionId", "Name") + .IsUnique(); + + b.ToTable("ExternalSystemMethods"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.InboundApi.ApiMethod", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("ParameterDefinitions") + .HasMaxLength(4000) + .HasColumnType("nvarchar(4000)"); + + b.Property("ReturnDefinition") + .HasMaxLength(4000) + .HasColumnType("nvarchar(4000)"); + + b.Property("Script") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("TimeoutSeconds") + .HasColumnType("int"); + + b.HasKey("Id"); + + b.HasIndex("Name") + .IsUnique(); + + b.ToTable("ApiMethods"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.Area", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("ParentAreaId") + .HasColumnType("int"); + + b.Property("SiteId") + .HasColumnType("int"); + + b.HasKey("Id"); + + b.HasIndex("ParentAreaId"); + + b.HasIndex("SiteId", "ParentAreaId", "Name") + .IsUnique() + .HasFilter("[ParentAreaId] IS NOT NULL"); + + b.ToTable("Areas"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.Instance", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("AreaId") + .HasColumnType("int"); + + b.Property("SiteId") + .HasColumnType("int"); + + b.Property("State") + .IsRequired() + .HasMaxLength(50) + .HasColumnType("nvarchar(50)"); + + b.Property("TemplateId") + .HasColumnType("int"); + + b.Property("UniqueName") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.HasKey("Id"); + + b.HasIndex("AreaId"); + + b.HasIndex("TemplateId"); + + b.HasIndex("SiteId", "UniqueName") + .IsUnique(); + + b.ToTable("Instances"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.InstanceAlarmOverride", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("AlarmCanonicalName") + .IsRequired() + .HasMaxLength(400) + .HasColumnType("nvarchar(400)"); + + b.Property("InstanceId") + .HasColumnType("int"); + + b.Property("PriorityLevelOverride") + .HasColumnType("int"); + + b.Property("TriggerConfigurationOverride") + .HasMaxLength(4000) + .HasColumnType("nvarchar(4000)"); + + b.HasKey("Id"); + + b.HasIndex("InstanceId", "AlarmCanonicalName") + .IsUnique(); + + b.ToTable("InstanceAlarmOverrides"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.InstanceAttributeOverride", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("AttributeName") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("ElementDataType") + .HasMaxLength(50) + .HasColumnType("nvarchar(50)"); + + b.Property("InstanceId") + .HasColumnType("int"); + + b.Property("OverrideValue") + .HasColumnType("nvarchar(max)"); + + b.HasKey("Id"); + + b.HasIndex("InstanceId", "AttributeName") + .IsUnique(); + + b.ToTable("InstanceAttributeOverrides"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.InstanceConnectionBinding", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("AttributeName") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("DataConnectionId") + .HasColumnType("int"); + + b.Property("DataSourceReferenceOverride") + .HasMaxLength(512) + .HasColumnType("nvarchar(512)"); + + b.Property("InstanceId") + .HasColumnType("int"); + + b.HasKey("Id"); + + b.HasIndex("DataConnectionId"); + + b.HasIndex("InstanceId", "AttributeName") + .IsUnique(); + + b.ToTable("InstanceConnectionBindings"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.InstanceNativeAlarmSourceOverride", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("ConditionFilterOverride") + .HasMaxLength(1000) + .HasColumnType("nvarchar(1000)"); + + b.Property("ConnectionNameOverride") + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("InstanceId") + .HasColumnType("int"); + + b.Property("SourceCanonicalName") + .IsRequired() + .HasMaxLength(400) + .HasColumnType("nvarchar(400)"); + + b.Property("SourceReferenceOverride") + .HasMaxLength(1000) + .HasColumnType("nvarchar(1000)"); + + b.HasKey("Id"); + + b.HasIndex("InstanceId", "SourceCanonicalName") + .IsUnique(); + + b.ToTable("InstanceNativeAlarmSourceOverrides"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Kpi.KpiRollupHourly", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("bigint"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("HourStartUtc") + .HasColumnType("datetime2"); + + b.Property("MaxValue") + .HasColumnType("float"); + + b.Property("Metric") + .IsRequired() + .HasMaxLength(64) + .IsUnicode(false) + .HasColumnType("varchar(64)"); + + b.Property("MinValue") + .HasColumnType("float"); + + b.Property("SampleCount") + .HasColumnType("int"); + + b.Property("Scope") + .IsRequired() + .HasMaxLength(16) + .IsUnicode(false) + .HasColumnType("varchar(16)"); + + b.Property("ScopeKey") + .HasMaxLength(64) + .IsUnicode(false) + .HasColumnType("varchar(64)"); + + b.Property("Source") + .IsRequired() + .HasMaxLength(64) + .IsUnicode(false) + .HasColumnType("varchar(64)"); + + b.Property("Value") + .HasColumnType("float"); + + b.HasKey("Id"); + + b.HasIndex("HourStartUtc") + .HasDatabaseName("IX_KpiRollupHourly_Hour"); + + b.HasIndex("Source", "Metric", "Scope", "ScopeKey", "HourStartUtc") + .IsUnique() + .HasDatabaseName("IX_KpiRollupHourly_Series"); + + b.ToTable("KpiRollupHourly", (string)null); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Kpi.KpiSample", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("bigint"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("CapturedAtUtc") + .HasColumnType("datetime2"); + + b.Property("Metric") + .IsRequired() + .HasMaxLength(64) + .IsUnicode(false) + .HasColumnType("varchar(64)"); + + b.Property("Scope") + .IsRequired() + .HasMaxLength(16) + .IsUnicode(false) + .HasColumnType("varchar(16)"); + + b.Property("ScopeKey") + .HasMaxLength(64) + .IsUnicode(false) + .HasColumnType("varchar(64)"); + + b.Property("Source") + .IsRequired() + .HasMaxLength(64) + .IsUnicode(false) + .HasColumnType("varchar(64)"); + + b.Property("Value") + .HasColumnType("float"); + + b.HasKey("Id"); + + b.HasIndex("CapturedAtUtc") + .HasDatabaseName("IX_KpiSample_Captured"); + + b.HasIndex("Source", "Metric", "Scope", "ScopeKey", "CapturedAtUtc") + .HasDatabaseName("IX_KpiSample_Series"); + + b.ToTable("KpiSample", (string)null); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Notifications.Notification", b => + { + b.Property("NotificationId") + .HasMaxLength(64) + .HasColumnType("nvarchar(64)"); + + b.Property("Body") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("CreatedAt") + .HasColumnType("datetimeoffset"); + + b.Property("DeliveredAt") + .HasColumnType("datetimeoffset"); + + b.Property("LastAttemptAt") + .HasColumnType("datetimeoffset"); + + b.Property("LastError") + .HasMaxLength(4000) + .HasColumnType("nvarchar(4000)"); + + b.Property("ListName") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("NextAttemptAt") + .HasColumnType("datetimeoffset"); + + b.Property("OriginExecutionId") + .HasColumnType("uniqueidentifier"); + + b.Property("OriginParentExecutionId") + .HasColumnType("uniqueidentifier"); + + b.Property("ResolvedTargets") + .HasColumnType("nvarchar(max)"); + + b.Property("RetryCount") + .HasColumnType("int"); + + b.Property("SiteEnqueuedAt") + .HasColumnType("datetimeoffset"); + + b.Property("SourceInstanceId") + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("SourceNode") + .HasMaxLength(64) + .IsUnicode(false) + .HasColumnType("varchar(64)"); + + b.Property("SourceScript") + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("SourceSiteId") + .IsRequired() + .HasMaxLength(100) + .HasColumnType("nvarchar(100)"); + + b.Property("Status") + .IsRequired() + .HasMaxLength(32) + .HasColumnType("nvarchar(32)"); + + b.Property("Subject") + .IsRequired() + .HasMaxLength(1000) + .HasColumnType("nvarchar(1000)"); + + b.Property("Type") + .IsRequired() + .HasMaxLength(32) + .HasColumnType("nvarchar(32)"); + + b.Property("TypeData") + .HasColumnType("nvarchar(max)"); + + b.HasKey("NotificationId"); + + b.HasIndex("SourceSiteId", "CreatedAt"); + + b.HasIndex("Status", "NextAttemptAt"); + + b.ToTable("Notifications"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Notifications.NotificationList", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("Type") + .IsRequired() + .HasMaxLength(32) + .HasColumnType("nvarchar(32)"); + + b.HasKey("Id"); + + b.HasIndex("Name") + .IsUnique(); + + b.ToTable("NotificationLists"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Notifications.NotificationRecipient", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("EmailAddress") + .HasMaxLength(500) + .HasColumnType("nvarchar(500)"); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("NotificationListId") + .HasColumnType("int"); + + b.Property("PhoneNumber") + .HasMaxLength(32) + .HasColumnType("nvarchar(32)"); + + b.HasKey("Id"); + + b.HasIndex("NotificationListId"); + + b.ToTable("NotificationRecipients"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Notifications.SmsConfiguration", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("AccountSid") + .IsRequired() + .HasMaxLength(100) + .HasColumnType("nvarchar(100)"); + + b.Property("ApiBaseUrl") + .HasMaxLength(500) + .HasColumnType("nvarchar(500)"); + + b.Property("AuthToken") + .HasMaxLength(8000) + .HasColumnType("nvarchar(max)"); + + b.Property("ConnectionTimeoutSeconds") + .HasColumnType("int"); + + b.Property("FromNumber") + .HasMaxLength(32) + .HasColumnType("nvarchar(32)"); + + b.Property("MaxRetries") + .HasColumnType("int"); + + b.Property("MessagingServiceSid") + .HasMaxLength(100) + .HasColumnType("nvarchar(100)"); + + b.Property("RetryDelay") + .HasColumnType("time"); + + b.HasKey("Id"); + + b.ToTable("SmsConfigurations"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Notifications.SmtpConfiguration", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("AuthType") + .IsRequired() + .HasMaxLength(50) + .HasColumnType("nvarchar(50)"); + + b.Property("ConnectionTimeoutSeconds") + .HasColumnType("int"); + + b.Property("Credentials") + .HasMaxLength(8000) + .HasColumnType("nvarchar(max)"); + + b.Property("FromAddress") + .IsRequired() + .HasMaxLength(500) + .HasColumnType("nvarchar(500)"); + + b.Property("Host") + .IsRequired() + .HasMaxLength(500) + .HasColumnType("nvarchar(500)"); + + b.Property("MaxConcurrentConnections") + .HasColumnType("int"); + + b.Property("MaxRetries") + .HasColumnType("int"); + + b.Property("OAuth2Authority") + .HasMaxLength(500) + .HasColumnType("nvarchar(500)"); + + b.Property("OAuth2Scope") + .HasMaxLength(500) + .HasColumnType("nvarchar(500)"); + + b.Property("Port") + .HasColumnType("int"); + + b.Property("RetryDelay") + .HasColumnType("time"); + + b.Property("TlsMode") + .HasMaxLength(50) + .HasColumnType("nvarchar(50)"); + + b.Property("Transport") + .HasMaxLength(50) + .HasColumnType("nvarchar(50)"); + + b.HasKey("Id"); + + b.ToTable("SmtpConfigurations"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Schemas.SharedSchema", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("SchemaJson") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("Scope") + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.HasKey("Id"); + + b.HasIndex("Name") + .IsUnique(); + + b.ToTable("SharedSchemas"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Scripts.SharedScript", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("Code") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("ParameterDefinitions") + .HasMaxLength(4000) + .HasColumnType("nvarchar(4000)"); + + b.Property("ReturnDefinition") + .HasMaxLength(4000) + .HasColumnType("nvarchar(4000)"); + + b.HasKey("Id"); + + b.HasIndex("Name") + .IsUnique(); + + b.ToTable("SharedScripts"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.SecuredWrites.PendingSecuredWrite", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("bigint"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("ConnectionName") + .IsRequired() + .HasMaxLength(128) + .IsUnicode(false) + .HasColumnType("varchar(128)"); + + b.Property("DecidedAtUtc") + .HasColumnType("datetime2"); + + b.Property("ExecutedAtUtc") + .HasColumnType("datetime2"); + + b.Property("ExecutionError") + .HasMaxLength(2048) + .IsUnicode(false) + .HasColumnType("varchar(2048)"); + + b.Property("OperatorComment") + .HasMaxLength(1024) + .IsUnicode(false) + .HasColumnType("varchar(1024)"); + + b.Property("OperatorUser") + .IsRequired() + .HasMaxLength(256) + .IsUnicode(false) + .HasColumnType("varchar(256)"); + + b.Property("SiteId") + .IsRequired() + .HasMaxLength(128) + .IsUnicode(false) + .HasColumnType("varchar(128)"); + + b.Property("Status") + .IsRequired() + .HasMaxLength(32) + .IsUnicode(false) + .HasColumnType("varchar(32)"); + + b.Property("SubmittedAtUtc") + .HasColumnType("datetime2"); + + b.Property("TagPath") + .IsRequired() + .HasMaxLength(512) + .IsUnicode(false) + .HasColumnType("varchar(512)"); + + b.Property("ValueJson") + .IsRequired() + .HasMaxLength(4000) + .IsUnicode(false) + .HasColumnType("varchar(4000)"); + + b.Property("ValueType") + .IsRequired() + .HasMaxLength(128) + .IsUnicode(false) + .HasColumnType("varchar(128)"); + + b.Property("VerifierComment") + .HasMaxLength(1024) + .IsUnicode(false) + .HasColumnType("varchar(1024)"); + + b.Property("VerifierUser") + .HasMaxLength(256) + .IsUnicode(false) + .HasColumnType("varchar(256)"); + + b.HasKey("Id"); + + b.HasIndex("SiteId") + .HasDatabaseName("IX_PendingSecuredWrites_Site"); + + b.HasIndex("Status", "SubmittedAtUtc") + .HasDatabaseName("IX_PendingSecuredWrites_Status_Submitted"); + + b.ToTable("PendingSecuredWrites", (string)null); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Security.LdapGroupMapping", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("LdapGroupName") + .IsRequired() + .HasMaxLength(500) + .HasColumnType("nvarchar(500)"); + + b.Property("Role") + .IsRequired() + .HasMaxLength(100) + .HasColumnType("nvarchar(100)"); + + b.HasKey("Id"); + + b.HasIndex("LdapGroupName") + .IsUnique(); + + b.ToTable("LdapGroupMappings"); + + b.HasData( + new + { + Id = 1, + LdapGroupName = "SCADA-Admins", + Role = "Administrator" + }, + new + { + Id = 2, + LdapGroupName = "SCADA-Designers", + Role = "Designer" + }, + new + { + Id = 3, + LdapGroupName = "SCADA-Deploy-All", + Role = "Deployer" + }, + new + { + Id = 4, + LdapGroupName = "SCADA-Deploy-SiteA", + Role = "Deployer" + }, + new + { + Id = 5, + LdapGroupName = "SCADA-Viewers", + Role = "Viewer" + }); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Security.SiteScopeRule", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("LdapGroupMappingId") + .HasColumnType("int"); + + b.Property("SiteId") + .HasColumnType("int"); + + b.HasKey("Id"); + + b.HasIndex("SiteId"); + + b.HasIndex("LdapGroupMappingId", "SiteId") + .IsUnique(); + + b.ToTable("SiteScopeRules"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites.DataConnection", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("BackupConfiguration") + .HasMaxLength(4000) + .HasColumnType("nvarchar(4000)"); + + b.Property("FailoverRetryCount") + .ValueGeneratedOnAdd() + .HasColumnType("int") + .HasDefaultValue(3); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("PrimaryConfiguration") + .HasMaxLength(4000) + .HasColumnType("nvarchar(4000)"); + + b.Property("Protocol") + .IsRequired() + .HasMaxLength(50) + .HasColumnType("nvarchar(50)"); + + b.Property("SiteId") + .HasColumnType("int"); + + b.HasKey("Id"); + + b.HasIndex("SiteId", "Name") + .IsUnique(); + + b.ToTable("DataConnections"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites.Site", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("Description") + .HasMaxLength(2000) + .HasColumnType("nvarchar(2000)"); + + b.Property("GrpcNodeAAddress") + .HasMaxLength(500) + .HasColumnType("nvarchar(500)"); + + b.Property("GrpcNodeBAddress") + .HasMaxLength(500) + .HasColumnType("nvarchar(500)"); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("NodeAAddress") + .HasMaxLength(500) + .HasColumnType("nvarchar(500)"); + + b.Property("NodeBAddress") + .HasMaxLength(500) + .HasColumnType("nvarchar(500)"); + + b.Property("SiteIdentifier") + .IsRequired() + .HasMaxLength(100) + .HasColumnType("nvarchar(100)"); + + b.HasKey("Id"); + + b.HasIndex("Name") + .IsUnique(); + + b.HasIndex("SiteIdentifier") + .IsUnique(); + + b.ToTable("Sites"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.Template", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("Description") + .HasMaxLength(2000) + .HasColumnType("nvarchar(2000)"); + + b.Property("FolderId") + .HasColumnType("int"); + + b.Property("IsDerived") + .HasColumnType("bit"); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("OwnerCompositionId") + .HasColumnType("int"); + + b.Property("ParentTemplateId") + .HasColumnType("int"); + + b.HasKey("Id"); + + b.HasIndex("FolderId"); + + b.HasIndex("Name") + .IsUnique() + .HasFilter("[IsDerived] = 0"); + + b.HasIndex("ParentTemplateId"); + + b.ToTable("Templates"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.TemplateAlarm", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("Description") + .HasMaxLength(2000) + .HasColumnType("nvarchar(2000)"); + + b.Property("IsInherited") + .HasColumnType("bit"); + + b.Property("IsLocked") + .HasColumnType("bit"); + + b.Property("LockedInDerived") + .HasColumnType("bit"); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("OnTriggerScriptId") + .HasColumnType("int"); + + b.Property("PriorityLevel") + .HasColumnType("int"); + + b.Property("TemplateId") + .HasColumnType("int"); + + b.Property("TriggerConfiguration") + .HasMaxLength(4000) + .HasColumnType("nvarchar(4000)"); + + b.Property("TriggerType") + .IsRequired() + .HasMaxLength(50) + .HasColumnType("nvarchar(50)"); + + b.HasKey("Id"); + + b.HasIndex("TemplateId", "Name") + .IsUnique(); + + b.ToTable("TemplateAlarms"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.TemplateAttribute", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("DataSourceReference") + .HasMaxLength(500) + .HasColumnType("nvarchar(500)"); + + b.Property("DataType") + .IsRequired() + .HasMaxLength(50) + .HasColumnType("nvarchar(50)"); + + b.Property("Description") + .HasMaxLength(2000) + .HasColumnType("nvarchar(2000)"); + + b.Property("ElementDataType") + .HasMaxLength(50) + .HasColumnType("nvarchar(50)"); + + b.Property("IsInherited") + .HasColumnType("bit"); + + b.Property("IsLocked") + .HasColumnType("bit"); + + b.Property("LockedInDerived") + .HasColumnType("bit"); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("TemplateId") + .HasColumnType("int"); + + b.Property("Value") + .HasColumnType("nvarchar(max)"); + + b.HasKey("Id"); + + b.HasIndex("TemplateId", "Name") + .IsUnique(); + + b.ToTable("TemplateAttributes"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.TemplateComposition", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("ComposedTemplateId") + .HasColumnType("int"); + + b.Property("InstanceName") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("TemplateId") + .HasColumnType("int"); + + b.HasKey("Id"); + + b.HasIndex("ComposedTemplateId"); + + b.HasIndex("TemplateId", "InstanceName") + .IsUnique(); + + b.ToTable("TemplateCompositions"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.TemplateFolder", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("ParentFolderId") + .HasColumnType("int"); + + b.Property("SortOrder") + .HasColumnType("int"); + + b.HasKey("Id"); + + b.HasIndex("ParentFolderId", "Name") + .IsUnique() + .HasFilter("[ParentFolderId] IS NOT NULL"); + + b.ToTable("TemplateFolders"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.TemplateNativeAlarmSource", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("ConditionFilter") + .HasMaxLength(1000) + .HasColumnType("nvarchar(1000)"); + + b.Property("ConnectionName") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("Description") + .HasMaxLength(2000) + .HasColumnType("nvarchar(2000)"); + + b.Property("IsInherited") + .HasColumnType("bit"); + + b.Property("IsLocked") + .HasColumnType("bit"); + + b.Property("LockedInDerived") + .HasColumnType("bit"); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("SourceReference") + .IsRequired() + .HasMaxLength(1000) + .HasColumnType("nvarchar(1000)"); + + b.Property("TemplateId") + .HasColumnType("int"); + + b.HasKey("Id"); + + b.HasIndex("TemplateId", "Name") + .IsUnique(); + + b.ToTable("TemplateNativeAlarmSources"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.TemplateScript", b => + { + b.Property("Id") + .ValueGeneratedOnAdd() + .HasColumnType("int"); + + SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property("Id")); + + b.Property("Code") + .IsRequired() + .HasColumnType("nvarchar(max)"); + + b.Property("ExecutionTimeoutSeconds") + .HasColumnType("int"); + + b.Property("IsInherited") + .HasColumnType("bit"); + + b.Property("IsLocked") + .HasColumnType("bit"); + + b.Property("LockedInDerived") + .HasColumnType("bit"); + + b.Property("MinTimeBetweenRuns") + .HasColumnType("time"); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("nvarchar(200)"); + + b.Property("ParameterDefinitions") + .HasMaxLength(4000) + .HasColumnType("nvarchar(4000)"); + + b.Property("ReturnDefinition") + .HasMaxLength(4000) + .HasColumnType("nvarchar(4000)"); + + b.Property("TemplateId") + .HasColumnType("int"); + + b.Property("TriggerConfiguration") + .HasMaxLength(4000) + .HasColumnType("nvarchar(4000)"); + + b.Property("TriggerType") + .HasMaxLength(50) + .HasColumnType("nvarchar(50)"); + + b.HasKey("Id"); + + b.HasIndex("TemplateId", "Name") + .IsUnique(); + + b.ToTable("TemplateScripts"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Entities.AuditLogRow", b => + { + b.Property("EventId") + .HasColumnType("uniqueidentifier"); + + b.Property("OccurredAtUtc") + .HasColumnType("datetime2"); + + b.Property("Action") + .IsRequired() + .HasMaxLength(64) + .IsUnicode(false) + .HasColumnType("varchar(64)"); + + b.Property("Actor") + .HasMaxLength(256) + .HasColumnType("nvarchar(256)"); + + b.Property("Channel") + .IsRequired() + .HasMaxLength(32) + .IsUnicode(false) + .HasColumnType("varchar(32)") + .HasColumnName("Category"); + + b.Property("CorrelationId") + .HasColumnType("uniqueidentifier"); + + b.Property("DetailsJson") + .HasColumnType("nvarchar(max)"); + + b.Property("ExecutionId") + .ValueGeneratedOnAddOrUpdate() + .HasColumnType("uniqueidentifier") + .HasComputedColumnSql("CAST(JSON_VALUE(DetailsJson,'$.executionId') AS uniqueidentifier)", true); + + b.Property("IngestedAtUtc") + .ValueGeneratedOnAddOrUpdate() + .HasColumnType("datetime2(7)") + .HasComputedColumnSql("CAST(SWITCHOFFSET(CAST(JSON_VALUE(DetailsJson,'$.ingestedAtUtc') AS datetimeoffset), 0) AS datetime2(7))", false); + + b.Property("Kind") + .IsRequired() + .ValueGeneratedOnAddOrUpdate() + .HasMaxLength(32) + .IsUnicode(false) + .HasColumnType("varchar(32)") + .HasComputedColumnSql("JSON_VALUE(DetailsJson,'$.kind')", true); + + b.Property("Outcome") + .IsRequired() + .HasMaxLength(16) + .IsUnicode(false) + .HasColumnType("varchar(16)"); + + b.Property("ParentExecutionId") + .ValueGeneratedOnAddOrUpdate() + .HasColumnType("uniqueidentifier") + .HasComputedColumnSql("CAST(JSON_VALUE(DetailsJson,'$.parentExecutionId') AS uniqueidentifier)", true); + + b.Property("SourceNode") + .HasMaxLength(64) + .IsUnicode(false) + .HasColumnType("varchar(64)"); + + b.Property("SourceSiteId") + .ValueGeneratedOnAddOrUpdate() + .HasMaxLength(64) + .IsUnicode(false) + .HasColumnType("varchar(64)") + .HasComputedColumnSql("JSON_VALUE(DetailsJson,'$.sourceSiteId')", true); + + b.Property("Status") + .IsRequired() + .ValueGeneratedOnAddOrUpdate() + .HasMaxLength(32) + .IsUnicode(false) + .HasColumnType("varchar(32)") + .HasComputedColumnSql("JSON_VALUE(DetailsJson,'$.status')", true); + + b.Property("Target") + .HasMaxLength(256) + .HasColumnType("nvarchar(256)"); + + b.HasKey("EventId", "OccurredAtUtc"); + + b.HasIndex("CorrelationId") + .HasDatabaseName("IX_AuditLog_CorrelationId") + .HasFilter("[CorrelationId] IS NOT NULL"); + + b.HasIndex("ExecutionId") + .HasDatabaseName("IX_AuditLog_Execution"); + + b.HasIndex("OccurredAtUtc") + .IsDescending() + .HasDatabaseName("IX_AuditLog_OccurredAtUtc"); + + b.HasIndex("ParentExecutionId") + .HasDatabaseName("IX_AuditLog_ParentExecution"); + + b.HasIndex("SourceNode", "OccurredAtUtc") + .HasDatabaseName("IX_AuditLog_Node_Occurred"); + + b.HasIndex("SourceSiteId", "OccurredAtUtc") + .IsDescending(false, true) + .HasDatabaseName("IX_AuditLog_Site_Occurred"); + + b.HasIndex("Target", "OccurredAtUtc") + .IsDescending(false, true) + .HasDatabaseName("IX_AuditLog_Target_Occurred") + .HasFilter("[Target] IS NOT NULL"); + + b.HasIndex("Channel", "Status", "OccurredAtUtc") + .IsDescending(false, false, true) + .HasDatabaseName("IX_AuditLog_Channel_Status_Occurred"); + + b.ToTable("AuditLog", (string)null); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Deployment.DeployedConfigSnapshot", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.Instance", null) + .WithMany() + .HasForeignKey("InstanceId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Deployment.DeploymentRecord", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.Instance", null) + .WithMany() + .HasForeignKey("InstanceId") + .OnDelete(DeleteBehavior.Restrict) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Deployment.PendingDeployment", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.Instance", null) + .WithMany() + .HasForeignKey("InstanceId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.ExternalSystems.ExternalSystemMethod", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.ExternalSystems.ExternalSystemDefinition", null) + .WithMany() + .HasForeignKey("ExternalSystemDefinitionId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.Area", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.Area", null) + .WithMany("Children") + .HasForeignKey("ParentAreaId") + .OnDelete(DeleteBehavior.Restrict); + + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites.Site", null) + .WithMany() + .HasForeignKey("SiteId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.Instance", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.Area", null) + .WithMany() + .HasForeignKey("AreaId") + .OnDelete(DeleteBehavior.SetNull); + + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites.Site", null) + .WithMany() + .HasForeignKey("SiteId") + .OnDelete(DeleteBehavior.Restrict) + .IsRequired(); + + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.Template", null) + .WithMany() + .HasForeignKey("TemplateId") + .OnDelete(DeleteBehavior.Restrict) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.InstanceAlarmOverride", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.Instance", null) + .WithMany("AlarmOverrides") + .HasForeignKey("InstanceId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.InstanceAttributeOverride", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.Instance", null) + .WithMany("AttributeOverrides") + .HasForeignKey("InstanceId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.InstanceConnectionBinding", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites.DataConnection", null) + .WithMany() + .HasForeignKey("DataConnectionId") + .OnDelete(DeleteBehavior.Restrict) + .IsRequired(); + + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.Instance", null) + .WithMany("ConnectionBindings") + .HasForeignKey("InstanceId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.InstanceNativeAlarmSourceOverride", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.Instance", null) + .WithMany("NativeAlarmSourceOverrides") + .HasForeignKey("InstanceId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Notifications.NotificationRecipient", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Notifications.NotificationList", null) + .WithMany("Recipients") + .HasForeignKey("NotificationListId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Security.SiteScopeRule", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Security.LdapGroupMapping", null) + .WithMany() + .HasForeignKey("LdapGroupMappingId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites.Site", null) + .WithMany() + .HasForeignKey("SiteId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites.DataConnection", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites.Site", null) + .WithMany() + .HasForeignKey("SiteId") + .OnDelete(DeleteBehavior.Restrict) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.Template", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.TemplateFolder", null) + .WithMany() + .HasForeignKey("FolderId") + .OnDelete(DeleteBehavior.Restrict); + + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.Template", null) + .WithMany() + .HasForeignKey("ParentTemplateId") + .OnDelete(DeleteBehavior.Restrict); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.TemplateAlarm", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.Template", null) + .WithMany("Alarms") + .HasForeignKey("TemplateId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.TemplateAttribute", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.Template", null) + .WithMany("Attributes") + .HasForeignKey("TemplateId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.TemplateComposition", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.Template", null) + .WithMany() + .HasForeignKey("ComposedTemplateId") + .OnDelete(DeleteBehavior.Restrict) + .IsRequired(); + + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.Template", null) + .WithMany("Compositions") + .HasForeignKey("TemplateId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.TemplateFolder", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.TemplateFolder", null) + .WithMany() + .HasForeignKey("ParentFolderId") + .OnDelete(DeleteBehavior.Restrict); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.TemplateNativeAlarmSource", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.Template", null) + .WithMany("NativeAlarmSources") + .HasForeignKey("TemplateId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.TemplateScript", b => + { + b.HasOne("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.Template", null) + .WithMany("Scripts") + .HasForeignKey("TemplateId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired(); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.Area", b => + { + b.Navigation("Children"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances.Instance", b => + { + b.Navigation("AlarmOverrides"); + + b.Navigation("AttributeOverrides"); + + b.Navigation("ConnectionBindings"); + + b.Navigation("NativeAlarmSourceOverrides"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Notifications.NotificationList", b => + { + b.Navigation("Recipients"); + }); + + modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates.Template", b => + { + b.Navigation("Alarms"); + + b.Navigation("Attributes"); + + b.Navigation("Compositions"); + + b.Navigation("NativeAlarmSources"); + + b.Navigation("Scripts"); + }); +#pragma warning restore 612, 618 + } + } +} diff --git a/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/20260815004957_AlignAuditLogEventIdUniqueness.cs b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/20260815004957_AlignAuditLogEventIdUniqueness.cs new file mode 100644 index 00000000..44495515 --- /dev/null +++ b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/20260815004957_AlignAuditLogEventIdUniqueness.cs @@ -0,0 +1,93 @@ +using Microsoft.EntityFrameworkCore.Migrations; + +#nullable disable + +namespace ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Migrations +{ + /// + /// Makes dbo.AuditLog's EventId uniqueness partition-aligned by + /// dropping the non-aligned UX_AuditLog_EventId and leaving the clustered + /// PK_AuditLog (EventId, OccurredAtUtc) — already aligned on + /// ps_AuditLog_Month(OccurredAtUtc) — as the sole enforcement. + /// + /// + /// + /// Why. ALTER TABLE … SWITCH PARTITION refuses to run while a + /// non-aligned index exists on the table, so the monthly retention purge + /// (AuditLogRepository.SwitchOutPartitionAsync) had to DROP + /// UX_AuditLog_EventId, switch, and then CREATE it again — an OFFLINE + /// whole-table unique-index build, inside the switch transaction, blocking every + /// audit writer for its duration. It also opened a window in which the index that + /// backs ingest idempotency did not exist at all, and a mid-dance failure could + /// leave the live table without it until a later tick's CATCH branch repaired it. + /// With alignment there is nothing to drop, so the switch is metadata-only and the + /// purge stops competing with ingest. + /// + /// + /// Why dropping it is safe — EventId is globally unique by construction. + /// The composite key enforces uniqueness of the PAIR, not of EventId alone, so in + /// principle the same EventId could now be stored twice under two different + /// OccurredAtUtc values (in two different partitions). That cannot happen + /// here: EventId is a GUID minted ONCE at the emitting site, in the same + /// operation that stamps OccurredAtUtc, and both travel together verbatim + /// through telemetry and reconciliation — nothing downstream re-stamps either + /// field. A given EventId therefore always arrives with the same OccurredAtUtc and + /// can only ever map to one partition, which makes pair-uniqueness equivalent to + /// EventId-uniqueness for every row this system produces. GUID collision across + /// partitions is not a real risk. + /// + /// + /// The idempotency probe still seeks. Both ingest forms test + /// WHERE EventId = @id, which is the LEADING column of the clustered PK, so + /// the probe remains an index seek. The cost changes shape rather than order: it + /// becomes one seek per partition (the partition column is not in the predicate, so + /// SQL Server cannot eliminate partitions) instead of a single seek on a + /// non-partitioned index. Against a monthly scheme that is a couple of dozen + /// shallow B-tree seeks — cheap, and paid on a path that now issues one statement + /// per telemetry packet rather than one per row. + /// + /// + /// Edition note. The alternative remedy — keeping the non-aligned index and + /// rebuilding it with ONLINE = ON outside the switch transaction — requires + /// Enterprise (or Azure SQL / Developer) edition; online index rebuild is not + /// available on Standard, which this deployment does not guarantee. Alignment + /// needs no edition-specific feature and removes the rebuild entirely, so it is + /// preferred regardless of edition. + /// + /// + /// Down is a faithful reverse and recreates the index on [PRIMARY] + /// exactly as CollapseAuditLogToCanonical created it. Reverting also + /// reinstates the SWITCH incompatibility, so the purge's guarded defensive + /// DROP INDEX (retained in SwitchOutPartitionAsync for databases + /// restored from pre-alignment backups) would remove it again on the next purge. + /// The partition function/scheme (pf_AuditLog_Month / + /// ps_AuditLog_Month) and every aligned index are untouched by both + /// directions. + /// + /// + public partial class AlignAuditLogEventIdUniqueness : Migration + { + /// + protected override void Up(MigrationBuilder migrationBuilder) + { + // Raw, existence-guarded SQL rather than the scaffolded DropIndex: the + // AuditLog table is raw-SQL managed (partition scheme, persisted computed + // columns, append-only role grants), so its migrations stay explicit and + // re-runnable. The guard also lets this apply cleanly to a database whose + // index was already removed by the purge path's defensive cleanup. + migrationBuilder.Sql(@" +IF EXISTS (SELECT 1 FROM sys.indexes + WHERE name = 'UX_AuditLog_EventId' AND object_id = OBJECT_ID('dbo.AuditLog')) + DROP INDEX UX_AuditLog_EventId ON dbo.AuditLog;"); + } + + /// + protected override void Down(MigrationBuilder migrationBuilder) + { + migrationBuilder.Sql(@" +IF NOT EXISTS (SELECT 1 FROM sys.indexes + WHERE name = 'UX_AuditLog_EventId' AND object_id = OBJECT_ID('dbo.AuditLog')) + CREATE UNIQUE NONCLUSTERED INDEX UX_AuditLog_EventId ON dbo.AuditLog (EventId) ON [PRIMARY];"); + } + } +} diff --git a/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/ScadaBridgeDbContextModelSnapshot.cs b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/ScadaBridgeDbContextModelSnapshot.cs index dccfa59e..4487dbc2 100644 --- a/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/ScadaBridgeDbContextModelSnapshot.cs +++ b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/ScadaBridgeDbContextModelSnapshot.cs @@ -903,6 +903,10 @@ namespace ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Migrations b.HasKey("NotificationId"); + b.HasIndex("DeliveredAt") + .HasDatabaseName("IX_Notifications_Delivered") + .HasFilter("[Status] = 'Delivered'"); + b.HasIndex("SourceSiteId", "CreatedAt"); b.HasIndex("Status", "NextAttemptAt"); @@ -1802,10 +1806,6 @@ namespace ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Migrations .HasDatabaseName("IX_AuditLog_CorrelationId") .HasFilter("[CorrelationId] IS NOT NULL"); - b.HasIndex("EventId") - .IsUnique() - .HasDatabaseName("UX_AuditLog_EventId"); - b.HasIndex("ExecutionId") .HasDatabaseName("IX_AuditLog_Execution"); @@ -1813,6 +1813,8 @@ namespace ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Migrations .IsDescending() .HasDatabaseName("IX_AuditLog_OccurredAtUtc"); + SqlServerIndexBuilderExtensions.IncludeProperties(b.HasIndex("OccurredAtUtc"), new[] { "Status" }); + b.HasIndex("ParentExecutionId") .HasDatabaseName("IX_AuditLog_ParentExecution"); diff --git a/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/AuditLogRepository.cs b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/AuditLogRepository.cs index a165319c..6bfba4ae 100644 --- a/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/AuditLogRepository.cs +++ b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/AuditLogRepository.cs @@ -1,5 +1,8 @@ +using System.Data; +using System.Text; using Microsoft.Data.SqlClient; using Microsoft.EntityFrameworkCore; +using Microsoft.EntityFrameworkCore.Storage; using Microsoft.Extensions.Logging; using Microsoft.Extensions.Logging.Abstractions; using ZB.MOM.WW.Audit; @@ -17,15 +20,32 @@ namespace ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Repositories; /// public class AuditLogRepository : IAuditLogRepository { - // SQL Server error numbers for duplicate-key violations on - // UX_AuditLog_EventId. 2601 is a unique-index violation; 2627 is a - // primary-key/unique-constraint violation. The IF NOT EXISTS … INSERT - // pattern has a check-then-act race window — two sessions can both pass - // the EXISTS check and then both attempt the INSERT — and the loser - // surfaces as one of these errors. Idempotency demands we swallow them. + // SQL Server error numbers for duplicate-key violations on the + // partition-aligned clustered PK_AuditLog (EventId, OccurredAtUtc). + // 2601 is a unique-index violation; 2627 is a primary-key/unique-constraint + // violation. The IF NOT EXISTS … INSERT pattern has a check-then-act race + // window — two sessions can both pass the EXISTS check and then both attempt + // the INSERT — and the loser surfaces as one of these errors. Idempotency + // demands we swallow them. private const int SqlErrorUniqueIndexViolation = 2601; private const int SqlErrorPrimaryKeyViolation = 2627; + // Rows per set-based ingest statement. Ten bound parameters per row against + // SQL Server's 2,100-parameter ceiling leaves ample headroom at 100 rows + // (1,000 parameters) while still collapsing a typical telemetry packet into + // a single round trip. Larger chunks buy little — the win is round-trip + // elimination, not statement size — and would push plan-cache churn up + // (one cached plan per distinct row count). + private const int IngestChunkRows = 100; + + // Ordinal-stable column list shared by the single-row and set-based inserts. + // The five persisted computed columns (Kind/Status/SourceSiteId/ExecutionId/ + // ParentExecutionId) plus the non-persisted IngestedAtUtc are derived + // server-side from DetailsJson and must NEVER appear here — writing a + // computed column is an error. + private const string CanonicalColumnList = + "EventId, OccurredAtUtc, Actor, Action, Outcome, Category, Target, SourceNode, CorrelationId, DetailsJson"; + private readonly ScadaBridgeDbContext _context; private readonly ILogger _logger; @@ -83,7 +103,7 @@ VALUES { // Two concurrent sessions both passed the IF NOT EXISTS check and // both attempted the INSERT — the loser raises 2601/2627 against - // UX_AuditLog_EventId. First-write-wins idempotency is already the + // the clustered PK. First-write-wins idempotency is already the // documented contract for this method, so the race outcome is // semantically a no-op. Swallow at Debug; other SqlExceptions // bubble. @@ -95,6 +115,218 @@ VALUES } } + /// + public async Task InsertManyIfNotExistsAsync( + IReadOnlyList events, + TimeSpan? commandTimeout = null, + CancellationToken ct = default) + { + ArgumentNullException.ThrowIfNull(events); + + if (events.Count == 0) + { + return 0; + } + + // De-duplicate WITHIN the packet before the statement is built. The + // set-based INSERT … SELECT … WHERE NOT EXISTS only tests rows that are + // ALREADY committed, so two copies of one EventId inside a single VALUES + // constructor would both pass the anti-semi-join and collide on the + // clustered PK — losing the whole statement to a duplicate-key fault. + // First-write-wins matches the single-row contract exactly (a later copy + // of the same EventId is by definition the same immutable append-only + // row), so keeping the first occurrence is not merely convenient, it is + // the documented semantics. + var seen = new HashSet(events.Count); + var distinct = new List(events.Count); + foreach (var evt in events) + { + if (evt is not null && seen.Add(evt.EventId)) + { + distinct.Add(evt); + } + } + + var inserted = 0; + for (var offset = 0; offset < distinct.Count; offset += IngestChunkRows) + { + var length = Math.Min(IngestChunkRows, distinct.Count - offset); + var chunk = distinct.GetRange(offset, length); + + try + { + inserted += await InsertChunkAsync(chunk, commandTimeout, ct).ConfigureAwait(false); + } + catch (SqlException ex) when ( + ex.Number == SqlErrorUniqueIndexViolation + || ex.Number == SqlErrorPrimaryKeyViolation) + { + // A CONCURRENT writer committed one of this chunk's EventIds + // between the anti-semi-join and the insert (the same + // check-then-act window the single-row path documents), and the + // whole set-based statement rolled back with it. Fall back to + // the per-row path so the rows that are genuinely new still land + // — the batch is a throughput optimisation, never a correctness + // dependency. Every row is retried, including the one that + // collided, because InsertIfNotExistsAsync swallows its own + // duplicate-key fault as a no-op. + _logger.LogDebug( + ex, + "Set-based audit ingest chunk of {Count} row(s) hit a duplicate-key violation (error {SqlErrorNumber}); falling back to per-row inserts.", + chunk.Count, + ex.Number); + + foreach (var evt in chunk) + { + await InsertIfNotExistsAsync(evt, ct).ConfigureAwait(false); + } + } + } + + return inserted; + } + + /// + /// Executes one set-based idempotent insert: a VALUES table constructor + /// anti-semi-joined against the committed rows, so a whole telemetry packet + /// costs ONE round trip instead of one IF NOT EXISTS … INSERT per row. + /// + /// + /// Raw ADO.NET (rather than ExecuteSqlInterpolated) because the + /// statement's parameter count varies with the chunk size and every parameter + /// needs an explicit : the VALUES constructor's column + /// types are inferred from the first row's parameters, so leaving a null + /// Target/SourceNode untyped would give the derived column the + /// wrong type and defeat the seek on the anti-semi-join. The command enlists + /// in the DbContext's ambient transaction when one is open — the cached + /// telemetry dual-write runs the audit insert and the SiteCalls upsert inside + /// a single transaction and both must commit or roll back together. + /// + private async Task InsertChunkAsync( + IReadOnlyList chunk, TimeSpan? commandTimeout, CancellationToken ct) + { + var sql = new StringBuilder(256 + (chunk.Count * 64)); + sql.Append("INSERT INTO dbo.AuditLog (").Append(CanonicalColumnList).Append(")\n"); + sql.Append("SELECT v.EventId, v.OccurredAtUtc, v.Actor, v.Action, v.Outcome, v.Category, "); + sql.Append("v.Target, v.SourceNode, v.CorrelationId, v.DetailsJson\n"); + sql.Append("FROM (VALUES\n"); + for (var i = 0; i < chunk.Count; i++) + { + if (i > 0) + { + sql.Append(",\n"); + } + + sql.Append(" (@e").Append(i) + .Append(",@t").Append(i) + .Append(",@a").Append(i) + .Append(",@n").Append(i) + .Append(",@o").Append(i) + .Append(",@c").Append(i) + .Append(",@g").Append(i) + .Append(",@s").Append(i) + .Append(",@r").Append(i) + .Append(",@d").Append(i) + .Append(')'); + } + + sql.Append("\n) AS v (").Append(CanonicalColumnList).Append(")\n"); + sql.Append("WHERE NOT EXISTS (SELECT 1 FROM dbo.AuditLog x WHERE x.EventId = v.EventId);"); + + var conn = _context.Database.GetDbConnection(); + var openedHere = false; + if (conn.State != ConnectionState.Open) + { + await conn.OpenAsync(ct).ConfigureAwait(false); + openedHere = true; + } + + try + { + await using var cmd = conn.CreateCommand(); + cmd.CommandText = sql.ToString(); + cmd.Transaction = _context.Database.CurrentTransaction?.GetDbTransaction(); + if (commandTimeout is { } timeout) + { + cmd.CommandTimeout = (int)timeout.TotalSeconds; + } + + for (var i = 0; i < chunk.Count; i++) + { + var evt = chunk[i]; + + // Same canonical projection as the single-row path: UTC-kind + // OccurredAtUtc, empty Actor collapses to NULL, Outcome/Category + // bound as their varchar storage form. + var occurred = DateTime.SpecifyKind(evt.OccurredAtUtc.UtcDateTime, DateTimeKind.Utc); + object actor = string.IsNullOrEmpty(evt.Actor) ? DBNull.Value : evt.Actor; + + // NO explicit Size on the string parameters — size 0 means "bind + // the value's own length" (see AddParameter). Declaring the + // COLUMN width here (Actor/Target 256, Action 64, Outcome 16, + // Category 32, SourceNode 64) made SqlClient TRUNCATE an + // over-long value client-side and commit the mutilated row, while + // the single-row path (ExecuteSqlInterpolated, no Size) and the + // reconciliation path sent the same value in full and let the + // server reject it with error 2628. An append-only audit store + // must never silently store a shortened row — there is no + // truncation flag on the record to say it happened — so the batch + // path now defers the length check to the server exactly like + // every other path: reject everywhere, truncate nowhere. + // The explicit SqlDbType stays (it is what fixes the VALUES + // constructor's derived column types, and DateTime2 precision). + AddParameter(cmd, "@e" + i, SqlDbType.UniqueIdentifier, size: 0, evt.EventId); + AddParameter(cmd, "@t" + i, SqlDbType.DateTime2, size: 0, occurred); + AddParameter(cmd, "@a" + i, SqlDbType.NVarChar, size: 0, actor); + AddParameter(cmd, "@n" + i, SqlDbType.VarChar, size: 0, evt.Action); + AddParameter(cmd, "@o" + i, SqlDbType.VarChar, size: 0, evt.Outcome.ToString()); + AddParameter(cmd, "@c" + i, SqlDbType.VarChar, size: 0, evt.Category); + AddParameter(cmd, "@g" + i, SqlDbType.NVarChar, size: 0, evt.Target); + AddParameter(cmd, "@s" + i, SqlDbType.VarChar, size: 0, evt.SourceNode); + AddParameter(cmd, "@r" + i, SqlDbType.UniqueIdentifier, size: 0, evt.CorrelationId); + AddParameter(cmd, "@d" + i, SqlDbType.NVarChar, size: -1, evt.DetailsJson); + } + + return await cmd.ExecuteNonQueryAsync(ct).ConfigureAwait(false); + } + finally + { + if (openedHere) + { + await conn.CloseAsync().ConfigureAwait(false); + } + } + } + + /// + /// Binds one explicitly-typed parameter. A null CLR value binds as + /// while KEEPING its declared , + /// which is what makes the VALUES constructor's derived column types stable + /// regardless of which rows happen to carry nulls. + /// + /// + /// semantics: 0 leaves + /// unset so SqlClient sizes the parameter from the VALUE — the only safe + /// choice for the variable-length audit columns, because an explicit Size + /// SMALLER than the value silently truncates it at bind time instead of + /// letting the server raise 2628. -1 is the explicit MAX marker + /// (nvarchar(max) DetailsJson). Never pass a column width here. + /// + private static void AddParameter( + System.Data.Common.DbCommand cmd, string name, SqlDbType type, int size, object? value) + { + var p = (SqlParameter)cmd.CreateParameter(); + p.ParameterName = name; + p.SqlDbType = type; + if (size != 0) + { + p.Size = size; + } + + p.Value = value ?? DBNull.Value; + cmd.Parameters.Add(p); + } + /// public async Task> QueryAsync( AuditLogQueryFilter filter, AuditLogPaging paging, CancellationToken ct = default) @@ -229,13 +461,23 @@ VALUES /// public async Task SwitchOutPartitionAsync(DateTime monthBoundary, TimeSpan? commandTimeout = null, CancellationToken ct = default) { - // The drop-and-rebuild batch below runs via + // The switch batch below runs via // ExecuteSqlRaw with NO EF user-transaction — it carries its own server-side // BEGIN TRANSACTION / TRY-CATCH / ROLLBACK — so the DbContext's retrying // execution strategy (EnableRetryOnFailure) MAY auto-replay the whole batch on // a transient fault. That replay is safe: every step is IF-EXISTS / IF-NOT-EXISTS // guarded and the staging table is GUID-suffixed, so a re-run is idempotent. // + // ALIGNED UNIQUENESS (WP2.2): there is no longer an index drop/rebuild around + // the SWITCH. EventId uniqueness is now enforced solely by the clustered + // PK_AuditLog (EventId, OccurredAtUtc), which is partition-aligned on + // ps_AuditLog_Month(OccurredAtUtc) — so ALTER TABLE … SWITCH PARTITION has no + // non-aligned index to object to. The former dance dropped UX_AuditLog_EventId, + // switched, then rebuilt it OFFLINE inside the same transaction: a whole-table + // unique-index build blocking every writer for the duration of the purge, and a + // window in which the idempotency-supporting index did not exist at all. Both + // are gone. See migration AlignAuditLogEventIdUniqueness for the reasoning. + // // Maintenance timeout in whole seconds (ADO.NET CommandTimeout unit). Null leaves the // provider default in place. See AuditLogPurgeOptions.MaintenanceCommandTimeoutMinutes / // arch-review 04 S2 for why the ~30s default is unsafe for the switch-out dance. @@ -270,8 +512,13 @@ VALUES BEGIN TRY BEGIN TRANSACTION; - -- 1. Drop the non-aligned unique index. ALTER TABLE SWITCH refuses - -- to run while it exists. + -- 1. Defensive cleanup for databases created before + -- AlignAuditLogEventIdUniqueness: the migration drops the + -- non-aligned UX_AuditLog_EventId, but a database restored from an + -- older backup could still carry it and SWITCH refuses to run while + -- a non-aligned unique index exists. Dropping it here is idempotent + -- and permanent — the aligned clustered PK is the only uniqueness + -- enforcement the ingest path needs, so there is nothing to rebuild. IF EXISTS (SELECT 1 FROM sys.indexes WHERE name = 'UX_AuditLog_EventId' AND object_id = OBJECT_ID('dbo.AuditLog')) DROP INDEX UX_AuditLog_EventId ON dbo.AuditLog; @@ -313,31 +560,19 @@ VALUES -- 4. Drop staging — the rows are discarded here. This is the purge. DROP TABLE dbo.[{stagingTableName}]; - -- 5. Rebuild the non-aligned unique index. Live traffic that hit the - -- table during steps 1-4 saw composite-PK uniqueness only; from - -- here on, single-column EventId uniqueness is restored. - CREATE UNIQUE NONCLUSTERED INDEX UX_AuditLog_EventId ON dbo.AuditLog (EventId) ON [PRIMARY]; - COMMIT TRANSACTION; END TRY BEGIN CATCH IF @@TRANCOUNT > 0 ROLLBACK TRANSACTION; - -- Best-effort staging cleanup. The DROP INDEX in step 1 is now - -- rolled back (so the index is back), but the staging table from - -- step 2 may or may not survive the rollback depending on the - -- failure point. Guard the DROP so a missing staging table doesn't - -- mask the original error. + -- Best-effort staging cleanup. The staging table from step 2 may or + -- may not survive the rollback depending on the failure point. Guard + -- the DROP so a missing staging table doesn't mask the original error. + -- Nothing else needs repairing: uniqueness lives on the clustered PK, + -- which the switch never touches, so a failed purge can no longer + -- leave the live table without its idempotency enforcement. IF OBJECT_ID('dbo.[{stagingTableName}]', 'U') IS NOT NULL DROP TABLE dbo.[{stagingTableName}]; - -- Idempotent index rebuild — covers the niche case where ROLLBACK - -- failed to restore UX_AuditLog_EventId (or the failure happened - -- AFTER the COMMIT, which shouldn't be possible inside this TRY - -- but is cheap insurance). Without this, a failed switch could - -- leave the live table without its idempotency-supporting index. - IF NOT EXISTS (SELECT 1 FROM sys.indexes WHERE name = 'UX_AuditLog_EventId' AND object_id = OBJECT_ID('dbo.AuditLog')) - CREATE UNIQUE NONCLUSTERED INDEX UX_AuditLog_EventId ON dbo.AuditLog (EventId) ON [PRIMARY]; - -- Surface the original error to the caller — the purge actor logs -- and continues with the next boundary. THROW; diff --git a/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/DeploymentManagerRepository.cs b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/DeploymentManagerRepository.cs index 1b5e5413..0383adcf 100644 --- a/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/DeploymentManagerRepository.cs +++ b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/DeploymentManagerRepository.cs @@ -277,6 +277,89 @@ public class DeploymentManagerRepository : IDeploymentManagerRepository return expired.Count; } + /// + public async Task> QueryDeploymentSummariesAsync( + int? instanceId, + DeploymentStatus? status, + IReadOnlyCollection? instanceIdScope, + int skip, + int? take, + CancellationToken cancellationToken = default) + { + var query = _dbContext.DeploymentRecords.AsNoTracking(); + + if (instanceId.HasValue) + query = query.Where(d => d.InstanceId == instanceId.Value); + + if (status.HasValue) + query = query.Where(d => d.Status == status.Value); + + if (instanceIdScope != null) + { + // Site scoping applied DB-side. An empty scope is a real, meaningful + // filter — a user permitted no in-scope instances sees nothing — so it + // must not be short-circuited into "no filter". + var scope = instanceIdScope as int[] ?? instanceIdScope.ToArray(); + query = query.Where(d => scope.Contains(d.InstanceId)); + } + + // ThenByDescending(Id) for the same reason GetCurrentDeploymentStatusAsync + // does it: DeployedAt ties on rapid redeploys, and an unstable sort key + // makes Skip/Take paging non-deterministic (rows repeat or vanish between + // pages). + var paged = query + .OrderByDescending(d => d.DeployedAt) + .ThenByDescending(d => d.Id) + .Skip(Math.Max(0, skip)); + + if (take is > 0) + paged = paged.Take(Math.Min(take.Value, DeploymentRecordSummary.MaxPageSize)); + + return await paged + .Select(d => new DeploymentRecordSummary( + d.Id, + d.DeploymentId, + d.InstanceId, + d.Status, + d.RevisionHash, + d.DeployedBy, + d.DeployedAt, + d.CompletedAt, + d.ErrorMessage)) + .ToListAsync(cancellationToken); + } + + /// + public async Task PurgeTerminalDeploymentRecordsAsync( + DateTimeOffset cutoffUtc, + int batchSize, + CancellationToken cancellationToken = default) + { + // Terminal statuses ONLY. An InProgress (or Pending) row is never purged + // regardless of age: it is precisely the row TryReconcileWithSiteAsync + // looks for when deciding whether a prior deploy actually landed, and + // deleting it would silently disable the query-before-redeploy idempotency + // guard for that instance. + // + // Bounded batch rather than one unbounded DELETE so the first purge on a + // long-lived system cannot take a table-scale lock on DeploymentRecords. + // The caller re-invokes until it returns fewer rows than the batch size. + var stale = await _dbContext.DeploymentRecords + .Where(d => d.CompletedAt != null + && d.CompletedAt < cutoffUtc + && (d.Status == DeploymentStatus.Success || d.Status == DeploymentStatus.Failed)) + .OrderBy(d => d.Id) + .Take(batchSize) + .ToListAsync(cancellationToken); + + if (stale.Count == 0) + return 0; + + _dbContext.DeploymentRecords.RemoveRange(stale); + await _dbContext.SaveChangesAsync(cancellationToken); + return stale.Count; + } + // --- Startup reconciliation --- /// diff --git a/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/KpiHistoryRepository.cs b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/KpiHistoryRepository.cs index b1298cfb..4f16d50c 100644 --- a/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/KpiHistoryRepository.cs +++ b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/KpiHistoryRepository.cs @@ -128,6 +128,37 @@ public sealed class KpiHistoryRepository : IKpiHistoryRepository var groups = samples.GroupBy(s => new SeriesHourKey( s.Source, s.Metric, s.Scope, s.ScopeKey, TruncateToHour(s.CapturedAtUtc))); + // Preload every rollup row already covering this window in ONE query and + // index it by series+hour (WP2.2). The predecessor issued a + // FirstOrDefaultAsync existence probe PER (series, hour) group — an N+1 + // that scaled with the metric catalogue times the lookback: a 3 h re-fold + // over ~40 series cost ~120 sequential round trips before a single row was + // written. The window is bounded by the caller's small trailing lookback + // and the rollup table holds exactly one row per series-hour, so the + // preload is a narrow range seek on IX_KpiRollupHourly_Series. + // + // Deliberately TRACKED (not a projection): the re-fold path mutates the + // existing entity in place and relies on the change tracker to emit the + // UPDATE. The dictionary's ScopeKey comparison is ordinal where the + // previous SQL predicate used the database collation; both sides are + // written from the same KpiSample.ScopeKey values, so they are + // byte-identical in practice and a residual mismatch degrades to the + // already-handled upsert-race path rather than a wrong aggregate. + var existingRollups = await _context.KpiRollupHourly + .Where(r => r.HourStartUtc >= from && r.HourStartUtc < to) + .ToListAsync(cancellationToken); + + var existingByKey = new Dictionary(existingRollups.Count); + foreach (var row in existingRollups) + { + existingByKey[new SeriesHourKey( + row.Source, + row.Metric, + row.Scope, + row.ScopeKey, + DateTime.SpecifyKind(row.HourStartUtc, DateTimeKind.Utc))] = row; + } + foreach (var group in groups) { var key = group.Key; @@ -143,18 +174,11 @@ public sealed class KpiHistoryRepository : IKpiHistoryRepository var maxValue = group.Max(s => s.Value); var sampleCount = group.Count(); - // Idempotent upsert on the unique series+hour key. The ScopeKey == key.ScopeKey - // comparison matches null against the Global-scope rows (IS NULL) exactly as the - // UNIQUE IX_KpiRollupHourly_Series index treats a null key as participating. - var existing = await _context.KpiRollupHourly.FirstOrDefaultAsync( - r => r.Source == key.Source - && r.Metric == key.Metric - && r.Scope == key.Scope - && r.ScopeKey == key.ScopeKey - && r.HourStartUtc == key.HourStartUtc, - cancellationToken); - - if (existing is null) + // Idempotent upsert on the unique series+hour key, resolved against the + // preloaded dictionary. A null ScopeKey keys the Global-scope rows exactly + // as the UNIQUE IX_KpiRollupHourly_Series index treats a null key as + // participating. + if (!existingByKey.TryGetValue(key, out var existing)) { _context.KpiRollupHourly.Add(new KpiRollupHourly { diff --git a/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/NotificationOutboxRepository.cs b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/NotificationOutboxRepository.cs index bc3690a2..2232a72a 100644 --- a/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/NotificationOutboxRepository.cs +++ b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/NotificationOutboxRepository.cs @@ -140,7 +140,14 @@ VALUES public async Task> GetDueAsync( DateTimeOffset now, int batchSize, CancellationToken cancellationToken = default) { + // AsNoTracking (WP2.2): the dispatcher mutates each row's delivery state in + // memory and persists it through UpdateAsync, which is now a targeted + // server-side ExecuteUpdate and needs no change tracker. Tracking a batch + // of notifications — each carrying an nvarchar(max) Body and TypeData — + // paid for a full snapshot copy per row plus a DetectChanges scan of the + // whole batch on every save. return await _context.Notifications + .AsNoTracking() .Where(n => n.Status == NotificationStatus.Pending || (n.Status == NotificationStatus.Retrying && n.NextAttemptAt != null @@ -151,21 +158,75 @@ VALUES } /// - public async Task GetByIdAsync(string notificationId, CancellationToken cancellationToken = default) - => await _context.Notifications.FindAsync(new object[] { notificationId }, cancellationToken); + public async Task UpdateAsync(Notification n, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(n); + + // Targeted server-side UPDATE of the seven mutable delivery-state columns + // (WP2.2). The predecessor called DbSet.Update(n) + SaveChanges, which + // marks EVERY property modified and rewrites all 21 columns — including + // the immutable nvarchar(max) Body/TypeData payloads — on every single + // delivery attempt. ExecuteUpdate also bypasses the change tracker + // entirely, so it composes with the untracked GetDueAsync read. + // + // Immutable-by-contract columns (NotificationId, Type, ListName, Subject, + // Body, TypeData, Source*, Origin*, SiteEnqueuedAt, CreatedAt) are + // deliberately absent — see the interface contract: nothing in the + // notification lifecycle ever changes them, and omitting them is what + // makes the write narrow. + var status = n.Status; + var retryCount = n.RetryCount; + var lastError = n.LastError; + var resolvedTargets = n.ResolvedTargets; + var lastAttemptAt = n.LastAttemptAt; + var nextAttemptAt = n.NextAttemptAt; + var deliveredAt = n.DeliveredAt; + + var notificationId = n.NotificationId; + + // The row count is the not-found signal and MUST NOT be discarded. + // DbSet.Update + SaveChanges used to throw DbUpdateConcurrencyException + // when the row had vanished (the daily retention purge deletes terminal + // rows); ExecuteUpdate just reports "0 rows" and returns, which made an + // operator Retry/Discard of a purged notification answer "success". + var rowsAffected = await _context.Notifications + .Where(row => row.NotificationId == notificationId) + .ExecuteUpdateAsync( + setters => setters + .SetProperty(row => row.Status, status) + .SetProperty(row => row.RetryCount, retryCount) + .SetProperty(row => row.LastError, lastError) + .SetProperty(row => row.ResolvedTargets, resolvedTargets) + .SetProperty(row => row.LastAttemptAt, lastAttemptAt) + .SetProperty(row => row.NextAttemptAt, nextAttemptAt) + .SetProperty(row => row.DeliveredAt, deliveredAt), + cancellationToken); + + return rowsAffected > 0; + } /// - public async Task UpdateAsync(Notification n, CancellationToken cancellationToken = default) + public async Task GetByIdAsync(string notificationId, CancellationToken cancellationToken = default) { - _context.Notifications.Update(n); - await _context.SaveChangesAsync(cancellationToken); + // AsNoTracking + a keyed predicate rather than FindAsync: the write that + // follows this read is ExecuteUpdate, which bypasses the change tracker + // entirely, so tracking the entity bought nothing but a snapshot copy of + // the nvarchar(max) Body/TypeData columns — and left a stale tracked + // instance in the context that a later read of the same id would return + // in preference to the database. FindAsync cannot be made no-tracking. + return await _context.Notifications + .AsNoTracking() + .FirstOrDefaultAsync(row => row.NotificationId == notificationId, cancellationToken); } /// public async Task<(IReadOnlyList Rows, int TotalCount)> QueryAsync( NotificationOutboxFilter filter, int pageNumber, int pageSize, CancellationToken cancellationToken = default) { - var query = _context.Notifications.AsQueryable(); + // AsNoTracking (WP2.2): this is the Central UI's read-only list page. The + // rows are projected to the wire and never saved, so tracking them cost a + // snapshot copy of every nvarchar(max) Body in the page for nothing. + var query = _context.Notifications.AsNoTracking().AsQueryable(); if (filter.Status is { } status) { @@ -218,8 +279,14 @@ VALUES var totalCount = await query.CountAsync(cancellationToken); + // NotificationId breaks CreatedAt ties so the OFFSET window is deterministic — + // without it two rows sharing a CreatedAt could appear on both pages or on + // neither. (This page keeps OFFSET paging rather than the sibling repos' + // keyset cursor because its contract surfaces a page number and a total + // count, neither of which a keyset cursor can express.) var rows = await query .OrderByDescending(n => n.CreatedAt) + .ThenByDescending(n => n.NotificationId) .Skip((pageNumber - 1) * pageSize) .Take(pageSize) .ToListAsync(cancellationToken); @@ -230,9 +297,30 @@ VALUES /// public async Task DeleteTerminalOlderThanAsync(DateTimeOffset cutoff, CancellationToken cancellationToken = default) { - return await _context.Notifications + // Time-sliced batches: each DELETE covers at most one hour of terminal rows, + // capping the lock/log footprint per statement — mirrors + // KpiHistoryRepository.PurgeOlderThanAsync (arch-review WP1.4; an unbounded + // single-statement ExecuteDeleteAsync would otherwise turn a long-idle catch-up + // of the default 365-day window into one giant transaction against + // dbo.Notifications). Pure EF LINQ (no raw SQL) so this stays portable across + // the SQL Server production provider and the SQLite provider the repository + // tests run against. + var total = 0; + var floor = await _context.Notifications .Where(n => TerminalStatuses.Contains(n.Status) && n.CreatedAt < cutoff) - .ExecuteDeleteAsync(cancellationToken); + .MinAsync(n => (DateTimeOffset?)n.CreatedAt, cancellationToken); + while (floor is not null && floor < cutoff) + { + var ceiling = floor.Value.AddHours(1) < cutoff ? floor.Value.AddHours(1) : cutoff; + total += await _context.Notifications + .Where(n => TerminalStatuses.Contains(n.Status) && n.CreatedAt < ceiling) + .ExecuteDeleteAsync(cancellationToken); + floor = await _context.Notifications + .Where(n => TerminalStatuses.Contains(n.Status) && n.CreatedAt < cutoff) + .MinAsync(n => (DateTimeOffset?)n.CreatedAt, cancellationToken); + } + + return total; } /// @@ -244,7 +332,24 @@ VALUES // One conditional-aggregation pass replaces four sequential COUNT round trips: // each metric is a COUNT(CASE WHEN THEN 1 END) over the same scan // (arch-review 04). GroupBy(_ => 1) yields a single group (no rows → no group). + // + // WP2.2 — the aggregation is now PREDICATE-RESTRICTED instead of scanning the + // whole table. Every KPI here is about the live queue (Pending/Retrying), the + // parked backlog, or the last delivery interval; the overwhelming bulk of the + // Notifications table is historical Delivered and Discarded rows that + // contribute to NONE of them. The pre-filter below is the union of the + // metric-contributing predicates, which lets the optimizer seek the status + // index for the live/parked legs and the filtered + // (DeliveredAt) WHERE Status='Delivered' index (WP1.4) for the interval leg, + // instead of paying a full scan whose cost grows with retained history. + // Mirrors the pre-filter ComputePerSiteKpisAsync/ComputePerNodeKpisAsync + // already use — the global snapshot was the odd one out. var counts = await _context.Notifications + .Where(n => n.Status == NotificationStatus.Pending + || n.Status == NotificationStatus.Retrying + || n.Status == NotificationStatus.Parked + || (n.Status == NotificationStatus.Delivered + && n.DeliveredAt != null && n.DeliveredAt >= deliveredSince)) .GroupBy(_ => 1) .Select(g => new { diff --git a/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/SiteCallAuditRepository.cs b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/SiteCallAuditRepository.cs index 29137a77..4d14a7a7 100644 --- a/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/SiteCallAuditRepository.cs +++ b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/SiteCallAuditRepository.cs @@ -1,3 +1,4 @@ +using System.Data; using Microsoft.Data.SqlClient; using Microsoft.EntityFrameworkCore; using Microsoft.Extensions.Logging; @@ -72,49 +73,54 @@ public class SiteCallAuditRepository : ISiteCallAuditRepository var idText = siteCall.TrackedOperationId.Value.ToString("D"); var incomingRank = GetRankOrThrow(siteCall.Status); - // Step 1: insert-if-not-exists. Like AuditLogRepository.InsertIfNotExistsAsync - // this is check-then-act so a duplicate-key violation may surface under - // concurrent inserts on the same id — caught + logged at Debug. + // ONE round trip, INSERT-first. Both statements ship in a single command + // text, so the round-trip saving of the WP2.2 rewrite is preserved — but + // the ORDER is back to insert-then-update, because UPDATE-first LOSES + // DATA under a concurrent first-write. // - // SourceNode-stamping: the column is included in the INSERT - // column list / VALUES so a fresh row carries the originating node - // name (node-a/node-b for site rows). A null SourceNode (legacy hosts - // / unstamped reconciled rows) writes NULL straight through. - try - { - await _context.Database.ExecuteSqlInterpolatedAsync( - $@"IF NOT EXISTS (SELECT 1 FROM dbo.SiteCalls WHERE TrackedOperationId = {idText}) -INSERT INTO dbo.SiteCalls - (TrackedOperationId, Channel, Target, SourceSite, SourceNode, Status, RetryCount, - LastError, HttpStatus, CreatedAtUtc, UpdatedAtUtc, TerminalAtUtc, IngestedAtUtc) -VALUES - ({idText}, {siteCall.Channel}, {siteCall.Target}, {siteCall.SourceSite}, {siteCall.SourceNode}, {siteCall.Status}, {siteCall.RetryCount}, - {siteCall.LastError}, {siteCall.HttpStatus}, {siteCall.CreatedAtUtc}, {siteCall.UpdatedAtUtc}, {siteCall.TerminalAtUtc}, {siteCall.IngestedAtUtc});", - ct); - } - catch (SqlException ex) when ( - ex.Number == SqlErrorUniqueIndexViolation - || ex.Number == SqlErrorPrimaryKeyViolation) - { - _logger.LogDebug( - ex, - "SiteCallAuditRepository.UpsertAsync swallowed duplicate-key violation (error {SqlErrorNumber}) for TrackedOperationId {TrackedOperationId}; falling through to monotonic update.", - ex.Number, - idText); - } - - // Step 2: monotonic update with a same-rank freshness tiebreaker. The - // CASE expression maps the stored Status string to the same rank table - // the caller uses. We mutate when EITHER the incoming rank is strictly - // greater, OR the incoming rank equals the stored rank AND that rank is - // non-terminal (< TerminalRank) AND the incoming UpdatedAtUtc is strictly - // newer than the stored one — so a retrying call's Attempted-phase - // RetryCount/LastError/HttpStatus stay live instead of freezing at the - // first Attempted packet. Terminal ranks are excluded from the - // tiebreaker, so a later terminal NEVER overwrites an earlier one; equal - // stamps are inert (idempotent replay) and a lower rank is always a no-op. + // The UPDATE-first shape was: UPDATE; SET @updated = @@ROWCOUNT; + // IF @updated = 0 AND NOT EXISTS(…) INSERT. Two writers racing the FIRST + // packet of one TrackedOperationId — the cached dual-write and the + // reconciliation pull, which routinely carry DIFFERENT lifecycle states — + // both find no row, so both UPDATEs match nothing. The loser then either + // fails its own NOT EXISTS re-check (READ COMMITTED, after the winner + // committed) and skips the INSERT, or attempts it and eats a 2627 in the + // catch below. EITHER WAY the loser's Status/RetryCount/HttpStatus/ + // TerminalAtUtc are silently dropped: it never ran an UPDATE against the + // winner's row. // - // SourceNode-stamping: SourceNode is updated via + // INSERT-first has no such hole. The loser's INSERT is skipped or faults, + // and the monotonic UPDATE that FOLLOWS it applies its state to whichever + // row won — so the newer lifecycle state survives regardless of + // interleaving, and a stale one is still rejected by the rank guard. + // + // Monotonic update semantics are unchanged: mutate when EITHER the + // incoming rank is strictly greater, OR the incoming rank equals the + // stored rank AND that rank is non-terminal (< TerminalRank) AND the + // incoming UpdatedAtUtc is strictly newer than the stored one — so a + // retrying call's Attempted-phase RetryCount/LastError/HttpStatus stay + // live instead of freezing at the first Attempted packet. Terminal ranks + // are excluded from the tiebreaker, so a later terminal NEVER overwrites + // an earlier one; equal stamps are inert (idempotent replay) and a lower + // rank is always a no-op. That inertness is what makes the UPDATE + // harmless immediately after this same call's own INSERT: equal rank, + // equal UpdatedAtUtc, zero rows changed. + // + // Raw SQL with explicitly-typed parameters (rather than + // ExecuteSqlInterpolated) so the monotonic UPDATE exists as ONE statement + // text shared by the combined batch and the duplicate-key retry below — + // the predicate is far too intricate to keep in two copies. Explicit + // SqlDbType is load-bearing: an untyped DateTime parameter binds as + // `datetime` (3.33 ms rounding), which would corrupt both the stored + // datetime2 stamps and the `UpdatedAtUtc <` freshness tiebreaker. Sizes + // are deliberately left at the value's own length so the server enforces + // the column widths (see AuditLogRepository.AddParameter). + // + // SourceNode-stamping: the column is included in the INSERT column list / + // VALUES so a fresh row carries the originating node name (node-a/node-b + // for site rows). A null SourceNode (legacy hosts / unstamped reconciled + // rows) writes NULL straight through. On the UPDATE leg SourceNode is + // written via // COALESCE(@SourceNode, SourceNode). The operator returns @SourceNode // when it is non-null, otherwise the stored value — so the column // behaves protectively: a later packet that carries a null @@ -128,44 +134,126 @@ VALUES // lifecycle every packet should carry the same SourceNode value (one // execution, one node) so the "overwrite" path is in practice // idempotent. - await _context.Database.ExecuteSqlInterpolatedAsync( - $@"UPDATE dbo.SiteCalls -SET Status = {siteCall.Status}, - RetryCount = {siteCall.RetryCount}, - LastError = {siteCall.LastError}, - HttpStatus = {siteCall.HttpStatus}, - UpdatedAtUtc = {siteCall.UpdatedAtUtc}, - TerminalAtUtc = {siteCall.TerminalAtUtc}, - IngestedAtUtc = {siteCall.IngestedAtUtc}, - SourceNode = COALESCE({siteCall.SourceNode}, SourceNode) -WHERE TrackedOperationId = {idText} - AND ( {incomingRank} > (CASE Status - WHEN 'Submitted' THEN 0 - WHEN 'Forwarded' THEN 1 - WHEN 'Attempted' THEN 2 - WHEN 'Skipped' THEN 2 - WHEN 'Delivered' THEN 3 - WHEN 'Failed' THEN 3 - WHEN 'Parked' THEN 3 - WHEN 'Discarded' THEN 3 - ELSE -1 - END) - OR ( {incomingRank} = (CASE Status - WHEN 'Submitted' THEN 0 - WHEN 'Forwarded' THEN 1 - WHEN 'Attempted' THEN 2 - WHEN 'Skipped' THEN 2 - WHEN 'Delivered' THEN 3 - WHEN 'Failed' THEN 3 - WHEN 'Parked' THEN 3 - WHEN 'Discarded' THEN 3 - ELSE -1 - END) - AND {incomingRank} < {TerminalRank} - AND UpdatedAtUtc < {siteCall.UpdatedAtUtc} ) );", - ct); + try + { + await _context.Database.ExecuteSqlRawAsync( + InsertIfAbsentSql + "\n\n" + MonotonicUpdateSql, + BuildUpsertParameters(siteCall, idText, incomingRank), + ct) + .ConfigureAwait(false); + } + catch (SqlException ex) when ( + ex.Number == SqlErrorUniqueIndexViolation + || ex.Number == SqlErrorPrimaryKeyViolation) + { + // Two concurrent sessions both found the row absent and both raced to + // INSERT; the loser raises 2601/2627 against the TrackedOperationId + // primary key. The winner's row IS the first-write, but it is NOT + // necessarily this packet's lifecycle state — the reconciliation pull + // and the cached dual-write feed this method with different states — + // so the loser MUST still apply its monotonic UPDATE against the + // winner's row. The batch aborted at the faulting INSERT, so re-run + // the UPDATE alone here; it is idempotent and rank-guarded, so + // re-running it is safe even if the batch did reach it. + _logger.LogDebug( + ex, + "SiteCallAuditRepository.UpsertAsync swallowed duplicate-key violation (error {SqlErrorNumber}) for TrackedOperationId {TrackedOperationId}; re-running the monotonic update against the winning row.", + ex.Number, + idText); + + await _context.Database.ExecuteSqlRawAsync( + MonotonicUpdateSql, + BuildUpsertParameters(siteCall, idText, incomingRank), + ct) + .ConfigureAwait(false); + } } + // Leg 1 of the upsert: create the row when it does not exist yet. Runs FIRST + // so a concurrent first-write loser still has a row to update (see UpsertAsync). + private const string InsertIfAbsentSql = @" +IF NOT EXISTS (SELECT 1 FROM dbo.SiteCalls WHERE TrackedOperationId = @Id) +INSERT INTO dbo.SiteCalls + (TrackedOperationId, Channel, Target, SourceSite, SourceNode, Status, RetryCount, + LastError, HttpStatus, CreatedAtUtc, UpdatedAtUtc, TerminalAtUtc, IngestedAtUtc) +VALUES + (@Id, @Channel, @Target, @SourceSite, @SourceNode, @Status, @RetryCount, + @LastError, @HttpStatus, @CreatedAtUtc, @UpdatedAtUtc, @TerminalAtUtc, @IngestedAtUtc);"; + + // Leg 2 of the upsert: the monotonic guard. Also re-run standalone from the + // duplicate-key catch, which is why it lives in its own constant. + private const string MonotonicUpdateSql = @" +UPDATE dbo.SiteCalls +SET Status = @Status, + RetryCount = @RetryCount, + LastError = @LastError, + HttpStatus = @HttpStatus, + UpdatedAtUtc = @UpdatedAtUtc, + TerminalAtUtc = @TerminalAtUtc, + IngestedAtUtc = @IngestedAtUtc, + SourceNode = COALESCE(@SourceNode, SourceNode) +WHERE TrackedOperationId = @Id + AND ( @Rank > (CASE Status + WHEN 'Submitted' THEN 0 + WHEN 'Forwarded' THEN 1 + WHEN 'Attempted' THEN 2 + WHEN 'Skipped' THEN 2 + WHEN 'Delivered' THEN 3 + WHEN 'Failed' THEN 3 + WHEN 'Parked' THEN 3 + WHEN 'Discarded' THEN 3 + ELSE -1 + END) + OR ( @Rank = (CASE Status + WHEN 'Submitted' THEN 0 + WHEN 'Forwarded' THEN 1 + WHEN 'Attempted' THEN 2 + WHEN 'Skipped' THEN 2 + WHEN 'Delivered' THEN 3 + WHEN 'Failed' THEN 3 + WHEN 'Parked' THEN 3 + WHEN 'Discarded' THEN 3 + ELSE -1 + END) + AND @Rank < @TerminalRank + AND UpdatedAtUtc < @UpdatedAtUtc ) );"; + + /// + /// Builds one FRESH parameter set for the upsert statements. Fresh per call + /// because a instance cannot be attached to two + /// commands, and the duplicate-key retry issues a second command. + /// + private static object[] BuildUpsertParameters(SiteCall siteCall, string idText, int incomingRank) + { + return + [ + Param("@Id", SqlDbType.VarChar, idText), + Param("@Channel", SqlDbType.VarChar, siteCall.Channel), + Param("@Target", SqlDbType.VarChar, siteCall.Target), + Param("@SourceSite", SqlDbType.VarChar, siteCall.SourceSite), + Param("@SourceNode", SqlDbType.VarChar, siteCall.SourceNode), + Param("@Status", SqlDbType.VarChar, siteCall.Status), + Param("@RetryCount", SqlDbType.Int, siteCall.RetryCount), + Param("@LastError", SqlDbType.NVarChar, siteCall.LastError), + Param("@HttpStatus", SqlDbType.Int, siteCall.HttpStatus), + Param("@CreatedAtUtc", SqlDbType.DateTime2, siteCall.CreatedAtUtc), + Param("@UpdatedAtUtc", SqlDbType.DateTime2, siteCall.UpdatedAtUtc), + Param("@TerminalAtUtc", SqlDbType.DateTime2, siteCall.TerminalAtUtc), + Param("@IngestedAtUtc", SqlDbType.DateTime2, siteCall.IngestedAtUtc), + Param("@Rank", SqlDbType.Int, incomingRank), + Param("@TerminalRank", SqlDbType.Int, TerminalRank), + ]; + } + + /// + /// Binds one explicitly-typed parameter, mapping a null CLR value to + /// while KEEPING the declared type. Size is never set — + /// SqlClient sizes from the value, so an over-long string is rejected by the + /// server rather than truncated client-side. + /// + private static SqlParameter Param(string name, SqlDbType type, object? value) => + new(name, type) { Value = value ?? DBNull.Value }; + /// public async Task GetAsync(TrackedOperationId id, CancellationToken ct = default) { diff --git a/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/SiteRepository.cs b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/SiteRepository.cs index b9bcfde9..10de8662 100644 --- a/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/SiteRepository.cs +++ b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/SiteRepository.cs @@ -2,6 +2,7 @@ using Microsoft.EntityFrameworkCore; using ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances; using ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites; using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories; +using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services; namespace ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Repositories; @@ -11,14 +12,23 @@ namespace ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Repositories; public class SiteRepository : ISiteRepository { private readonly ScadaBridgeDbContext _dbContext; + private readonly ITemplateGraphWatermark? _watermark; /// /// Initializes a new instance of the SiteRepository. /// /// The database context. - public SiteRepository(ScadaBridgeDbContext dbContext) + /// + /// Graph version watermark bumped when a committed change alters a flattening + /// input this repository owns (see ). Optional so + /// existing single-argument construction (tests, ad-hoc tooling) keeps + /// compiling; when absent the caches simply never get the invalidation signal + /// from this repository, which is the pre-existing behaviour. + /// + public SiteRepository(ScadaBridgeDbContext dbContext, ITemplateGraphWatermark? watermark = null) { _dbContext = dbContext ?? throw new ArgumentNullException(nameof(dbContext)); + _watermark = watermark; } // --- Sites --- @@ -166,8 +176,58 @@ public class SiteRepository : ISiteRepository } /// + /// + /// Bumps the after a successful commit + /// that touched a . + /// + /// + /// A data connection's Protocol, primary/backup configuration and + /// failover retry count are flattening inputs — FlatteningService + /// packages them into the flattened config's Connections map and + /// RevisionHashService folds them into the revision hash. Without a bump, + /// the process-wide StaleInstanceProbe memo and the flatten-session + /// caches keep reporting the pre-edit hash for every instance bound to the + /// connection, so an instance whose deployed config genuinely drifted reads as + /// up to date. + /// + /// + /// + /// The bump is rather than a + /// per-template one because a connection has no owning template: the affected + /// set is "every instance at the site bound to it", which is exactly what the + /// unattributed fallback is for. Applied only AFTER a successful commit, so a + /// failed save does not invalidate caches for a change that never landed. + /// Sites, areas and instances saved here are left alone — only the connection + /// rows feed the flattener. + /// + /// public async Task SaveChangesAsync(CancellationToken cancellationToken = default) { - return await _dbContext.SaveChangesAsync(cancellationToken); + // Inspect BEFORE committing: afterwards every entry is Unchanged and the + // attribution is lost. + var connectionsChanged = _watermark is not null && HasPendingDataConnectionChange(); + + var written = await _dbContext.SaveChangesAsync(cancellationToken); + + if (connectionsChanged) + _watermark!.BumpAll(); + + return written; + } + + /// + /// True when the change tracker holds an added, modified or deleted + /// . + /// + /// Whether a data-connection mutation is about to be committed. + private bool HasPendingDataConnectionChange() + { + foreach (var entry in _dbContext.ChangeTracker.Entries()) + { + if (entry.State is EntityState.Added or EntityState.Modified or EntityState.Deleted) + return true; + } + + return false; } } diff --git a/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/TemplateEngineRepository.cs b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/TemplateEngineRepository.cs index a0a1e632..99a70640 100644 --- a/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/TemplateEngineRepository.cs +++ b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/TemplateEngineRepository.cs @@ -3,20 +3,32 @@ using ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances; using ZB.MOM.WW.ScadaBridge.Commons.Entities.Scripts; using ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates; using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories; +using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services; +using ZB.MOM.WW.ScadaBridge.Commons.Types.Templates; namespace ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Repositories; public class TemplateEngineRepository : ITemplateEngineRepository { private readonly ScadaBridgeDbContext _context; + private readonly ITemplateGraphWatermark _watermark; /// /// Initializes a new instance of the TemplateEngineRepository class. /// /// The database context used to access template and instance data. - public TemplateEngineRepository(ScadaBridgeDbContext context) + /// + /// Process-wide template/instance version watermark. This repository is the + /// single unit-of-work through which EVERY template-graph mutation commits — + /// TemplateService, the ManagementActor native-alarm-source + /// handlers, and the Transport bundle importer all funnel through the same + /// — so bumping here (rather than in each + /// caller) is the only placement that cannot be bypassed. + /// + public TemplateEngineRepository(ScadaBridgeDbContext context, ITemplateGraphWatermark watermark) { _context = context ?? throw new ArgumentNullException(nameof(context)); + _watermark = watermark ?? throw new ArgumentNullException(nameof(watermark)); } // Template @@ -78,6 +90,95 @@ public class TemplateEngineRepository : ITemplateEngineRepository .ToListAsync(cancellationToken); } + /// + /// + /// Two deliberate departures from : + /// + /// + /// AsNoTracking — the analysis walks (cycle detection, collision + /// detection, canonical-name resolution) never write through the loaded + /// graph, so paying for a change-tracker snapshot of every attribute, + /// alarm, script and composition in the database is pure waste. + /// + /// + /// TemplateScript.Code is projected away (left empty). Script + /// bodies are by far the largest column in the graph and NONE of the + /// analysis consumers read them — CycleDetector reads only ids and + /// edges, CollisionDetector and TemplateResolver read only + /// member names and lock flags. + /// + /// + /// Callers that DO need bodies or tracked entities — the inheritance + /// reconciler and the flattener — must keep using + /// / . + /// + public async Task> GetAllTemplatesForAnalysisAsync(CancellationToken cancellationToken = default) + { + return await _context.Templates + .AsNoTracking() + .Select(t => new Template(t.Name) + { + Id = t.Id, + Description = t.Description, + ParentTemplateId = t.ParentTemplateId, + FolderId = t.FolderId, + IsDerived = t.IsDerived, + OwnerCompositionId = t.OwnerCompositionId, + Attributes = t.Attributes.ToList(), + Alarms = t.Alarms.ToList(), + NativeAlarmSources = t.NativeAlarmSources.ToList(), + Compositions = t.Compositions.ToList(), + Scripts = t.Scripts + .Select(s => new TemplateScript(s.Name, string.Empty) + { + Id = s.Id, + TemplateId = s.TemplateId, + IsLocked = s.IsLocked, + TriggerType = s.TriggerType, + TriggerConfiguration = s.TriggerConfiguration, + ParameterDefinitions = s.ParameterDefinitions, + ReturnDefinition = s.ReturnDefinition, + MinTimeBetweenRuns = s.MinTimeBetweenRuns, + ExecutionTimeoutSeconds = s.ExecutionTimeoutSeconds, + IsInherited = s.IsInherited, + LockedInDerived = s.LockedInDerived + }) + .ToList() + }) + .ToListAsync(cancellationToken); + } + + /// + public async Task> GetTemplateSummariesAsync( + int skip, + int? take, + CancellationToken cancellationToken = default) + { + var query = _context.Templates + .AsNoTracking() + .OrderBy(t => t.Id) + .Skip(Math.Max(0, skip)); + + if (take is > 0) + query = query.Take(Math.Min(take.Value, TemplateSummary.MaxPageSize)); + + return await query + .Select(t => new TemplateSummary( + t.Id, + t.Name, + t.Description, + t.ParentTemplateId, + t.FolderId, + t.IsDerived, + t.OwnerCompositionId, + t.Attributes.Count, + t.Alarms.Count, + t.Scripts.Count, + t.Compositions.Count, + t.NativeAlarmSources.Count)) + .ToListAsync(cancellationToken); + } + /// public async Task> GetTemplatesComposingAsync(int composedTemplateId, CancellationToken cancellationToken = default) { @@ -659,8 +760,140 @@ public class TemplateEngineRepository : ITemplateEngineRepository } /// + /// + /// Inspects the change tracker BEFORE committing (afterwards every entry is + /// and the attribution is lost) and bumps + /// the for each affected template / + /// instance. The bump is applied only AFTER a successful commit, so a failed + /// save does not invalidate caches for a change that never landed. + /// public async Task SaveChangesAsync(CancellationToken cancellationToken = default) { - return await _context.SaveChangesAsync(cancellationToken); + var pending = CollectWatermarkBumps(); + var written = await _context.SaveChangesAsync(cancellationToken); + ApplyWatermarkBumps(pending); + return written; + } + + /// + /// Snapshot of the watermark bumps implied by the current change-tracker + /// contents. Captured pre-commit, applied post-commit. + /// + private readonly record struct WatermarkBumps( + HashSet Templates, + HashSet StructuralTemplates, + HashSet Instances, + bool Unattributed); + + /// + /// Walks the change tracker and maps every added/modified/deleted + /// template-graph or instance-graph entry back to the template (or instance) + /// whose flattened output it can affect. + /// + /// + /// A change is treated as STRUCTURAL — invalidating cached chain membership, + /// not just chain contents — when it adds or removes a + /// row, changes a , or touches a + /// row. Those are exactly the edges the + /// flattener walks to build a chain. + /// + /// + /// + /// An entry whose owning id cannot be resolved (a child row deleted before + /// its FK was materialised, or an entity type added later that this switch + /// does not know) sets Unattributed, which conservatively invalidates + /// every cached chain rather than silently letting a stale entry survive. + /// + /// + private WatermarkBumps CollectWatermarkBumps() + { + var templates = new HashSet(); + var structural = new HashSet(); + var instances = new HashSet(); + var unattributed = false; + + foreach (var entry in _context.ChangeTracker.Entries()) + { + if (entry.State is not (EntityState.Added or EntityState.Modified or EntityState.Deleted)) + continue; + + switch (entry.Entity) + { + case Template t: + templates.Add(t.Id); + // Add/Delete changes chain membership; a re-parent changes it too. + if (entry.State != EntityState.Modified + || entry.Property(nameof(Template.ParentTemplateId)).IsModified) + { + structural.Add(t.Id); + } + break; + case TemplateAttribute a: + templates.Add(a.TemplateId); + break; + case TemplateAlarm al: + templates.Add(al.TemplateId); + break; + case TemplateScript s: + templates.Add(s.TemplateId); + break; + case TemplateNativeAlarmSource ns: + templates.Add(ns.TemplateId); + break; + case TemplateComposition c: + templates.Add(c.TemplateId); + structural.Add(c.TemplateId); + break; + case Instance i: + instances.Add(i.Id); + break; + case InstanceAttributeOverride ao: + instances.Add(ao.InstanceId); + break; + case InstanceAlarmOverride alo: + instances.Add(alo.InstanceId); + break; + case InstanceNativeAlarmSourceOverride nso: + instances.Add(nso.InstanceId); + break; + case InstanceConnectionBinding cb: + instances.Add(cb.InstanceId); + break; + case SharedScript: + // Shared scripts are a session-global input to validation, not + // owned by any one template — invalidate everything. + unattributed = true; + break; + default: + // Anything this switch does not recognise is invalidated + // conservatively, as the doc comment above promises. The + // previous "areas, folders and anything else do not feed the + // flattener" no-op was wrong in at least one shipped case: + // DataConnection rows land here, and their Protocol / + // Primary+Backup configuration / FailoverRetryCount ARE hash + // inputs (FlatteningService packages them as ConnectionConfig, + // RevisionHashService hashes them as HashableConnection). A + // silent no-op on an unrecognised entity is the one failure + // mode this watermark must not have: it produces STALE work, + // whereas an over-broad bump only ever produces extra work. + unattributed = true; + break; + } + } + + return new WatermarkBumps(templates, structural, instances, unattributed); + } + + /// Applies a previously collected snapshot. + private void ApplyWatermarkBumps(WatermarkBumps bumps) + { + if (bumps.Unattributed) + _watermark.BumpAll(); + + foreach (var templateId in bumps.Templates) + _watermark.BumpTemplate(templateId, bumps.StructuralTemplates.Contains(templateId)); + + foreach (var instanceId in bumps.Instances) + _watermark.BumpInstance(instanceId); } } diff --git a/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/ServiceCollectionExtensions.cs b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/ServiceCollectionExtensions.cs index 1752e17e..1d2636f0 100644 --- a/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/ServiceCollectionExtensions.cs +++ b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/ServiceCollectionExtensions.cs @@ -1,10 +1,12 @@ using Microsoft.AspNetCore.DataProtection; using Microsoft.EntityFrameworkCore; using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.DependencyInjection.Extensions; using ZB.MOM.WW.ScadaBridge.Commons.Interfaces; using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories; using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services; using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Transport; +using ZB.MOM.WW.ScadaBridge.Commons.Types.Templates; using ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Maintenance; using ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Repositories; using ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Services; @@ -26,6 +28,22 @@ public static class ServiceCollectionExtensions // registers IDataProtectionProvider as a singleton; resolving it here does not recurse // because key-ring loading is lazy (first Protect/Unprotect), not triggered by // CreateProtector during model building. + // + // POOLING IS DELIBERATELY NOT USED (WP2.2 — verified, not overlooked). + // AddDbContextPool requires a context with a SINGLE public constructor taking + // only DbContextOptions; EF Core constructs pooled instances through + // its own activator and cannot supply anything else. ScadaBridgeDbContext has + // two public constructors and the runtime one takes IDataProtectionProvider, + // because the encrypting value converter for secret-bearing columns is built + // during OnModelCreating from that provider. Worse, the model itself DIFFERS + // between the two constructors (no provider ⇒ no encrypting converter), so a + // pooled activator would silently produce a context that reads secret columns + // as ciphertext. Making this poolable means moving the protector out of the + // constructor and into a DbContextOptions extension — a change to the + // secrets-at-rest path, which is not a performance refactor. The registration + // below (a scoped factory overriding AddDbContext's activator) is what makes + // the provider reach the context at all, and it also bypasses pooling by + // construction. Revisit only alongside a deliberate secrets-plumbing change. services.AddDbContext((serviceProvider, options) => { options.UseSqlServer( @@ -49,6 +67,13 @@ public static class ServiceCollectionExtensions return new ScadaBridgeDbContext(options, protectionProvider); }); + // Process-wide template/instance version watermark. Singleton because the + // writer (TemplateEngineRepository.SaveChangesAsync, scoped) and the readers + // (the flatten-session cache and the staleness fast path, also scoped) must + // observe the same counters across DI scopes. TryAdd so the DeploymentManager + // registration of the same interface is idempotent. + services.TryAddSingleton(); + services.AddScoped(); services.AddScoped(); services.AddScoped(); diff --git a/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Actors/DataConnectionActor.cs b/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Actors/DataConnectionActor.cs index a390a481..9dfff812 100644 --- a/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Actors/DataConnectionActor.cs +++ b/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Actors/DataConnectionActor.cs @@ -140,6 +140,38 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers private int _tagsUncertainQuality; private readonly Dictionary _lastTagQuality = new(); + /// + /// Set when a genuine quality TRANSITION has moved the counters since the last push to + /// the health collector. The push itself is coalesced onto a single-shot + /// quality-flush timer () + /// instead of running on every received value: at 37,500 tags the per-message push was + /// pure overhead against a collector that is only read every 30s. Disconnect, + /// unsubscribe and the reconnect reset still flush SYNCHRONOUSLY — their correctness + /// depends on the collector being current at that instant. + /// + private bool _qualityDirty; + + /// + /// Current tag-resolution retry interval. Starts at + /// , doubles after a + /// round in which no tag resolved, and is capped at + /// . Reset to the floor + /// whenever a tag resolves or the connection reconnects, so a dead device backs off + /// while a booting one is still picked up quickly. + /// + private TimeSpan _tagResolutionInterval; + + // ── Alarm subscriber prefix index ── + // HandleAlarmTransitionReceived used to scan EVERY subscribed source (two StartsWith + // per source) on every transition. Sources are bucketed by the FIRST path segment of + // their reference; a transition is matched only against the bucket for its own first + // segment, plus the residue list of sources that carry no separator at all (a prefix + // shorter than one segment, which can match transitions in other buckets). The + // per-source StartsWith + condition-type gate inside the bucket is unchanged, so + // routing decisions are identical to the linear scan. + private readonly Dictionary> _alarmSourcesByFirstSegment = new(StringComparer.Ordinal); + private readonly HashSet _alarmSourcesWithoutSeparator = new(StringComparer.Ordinal); + private IDictionary _connectionDetails; private readonly IDictionary _primaryConfig; private readonly IDictionary? _backupConfig; @@ -200,6 +232,7 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers _failoverRetryCount = failoverRetryCount; _siteEventLogger = siteEventLogger; _connectionDetails = _primaryConfig; + _tagResolutionInterval = _options.TagResolutionRetryInterval; } /// @@ -284,6 +317,17 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers case AlarmTransitionReceived: // No live feed yet in Connecting; ignore (snapshot replays on subscribe). break; + case BatchSubscribeCompleted: + // Re-subscribe / resolution-probe results from a previous connection — + // ReSubscribeAll re-issues everything once the link is back up. + break; + case RetryTagResolution: + // No session yet — a probe would fail for every tag. ReSubscribeAll's chunk + // completions re-arm the timer once the link is up. + break; + case QualityFlushTick: + FlushQualityCountersIfDirty(); + break; case BrowseNodeCommand browse: // Browse is an interactive design-time query; never stash. The // adapter has no session yet in this state, so reply with a @@ -366,11 +410,11 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers case TagValueReceived tvr: HandleTagValueReceived(tvr); break; - case TagResolutionSucceeded trs: - HandleTagResolutionSucceeded(trs); + case BatchSubscribeCompleted bsc: + HandleBatchSubscribeCompleted(bsc); break; - case TagResolutionFailed trf: - HandleTagResolutionFailed(trf); + case QualityFlushTick: + FlushQualityCountersIfDirty(); break; case AdapterDisconnected: HandleDisconnect(); @@ -510,10 +554,16 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers case AlarmTransitionReceived: // Ignore — stale alarm callback from previous connection; ReSubscribeAll re-seeds. break; - case TagResolutionSucceeded: - case TagResolutionFailed: + case BatchSubscribeCompleted: // Ignore — stale results from previous connection; ReSubscribeAll runs after reconnect break; + case RetryTagResolution: + // Ignore — the adapter has no live session; ReSubscribeAll's chunk + // completions re-arm the retry timer after reconnect. + break; + case QualityFlushTick: + FlushQualityCountersIfDirty(); + break; case SubscribeCompleted sc: // A subscribe started while Connected can complete after a transition; // apply it so its state survives into the next ReSubscribeAll. @@ -728,6 +778,13 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers } } + // ONE batch subscribe per chunk instead of one adapter round trip per tag. The + // Deployment Manager already staggers instance startup (SiteRuntimeOptions + // StartupBatchSize/Delay), so requests arrive pre-spaced on this path and the DCL + // deliberately adds NO extra delay between chunks here — double-staggering would + // only slow failover. A typical request is ~75 tags, i.e. a single chunk. + var batchSize = Math.Max(1, _options.SubscribeBatchSize); + Task.Run(async () => { var results = new List(request.TagPaths.Count); @@ -738,27 +795,19 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers tagsToSeed.Add(r.TagPath); } - foreach (var tagPath in tagsToSubscribe) + for (var offset = 0; offset < tagsToSubscribe.Count; offset += batchSize) { - try + var chunk = tagsToSubscribe.GetRange( + offset, Math.Min(batchSize, tagsToSubscribe.Count - offset)); + var chunkResults = await SubscribeTagsAsync(adapter, chunk, (path, value) => { - var subId = await adapter.SubscribeAsync(tagPath, (path, value) => - { - self.Tell(new TagValueReceived(path, value, generation)); - }); - results.Add(new SubscribeTagResult(tagPath, AlreadySubscribed: false, Success: true, subId, null)); - tagsToSeed.Add(tagPath); - } - catch (Exception ex) + self.Tell(new TagValueReceived(path, value, generation)); + }); + results.AddRange(chunkResults); + foreach (var r in chunkResults) { - // Distinguish a connection-level fault - // (adapter not connected / transport down) from a genuine - // node-not-found. Connection-level faults must drive the - // reconnection state machine, not be retried as unresolved tags. - var connectionLevel = IsConnectionLevelFailure(ex); - results.Add(new SubscribeTagResult( - tagPath, AlreadySubscribed: false, Success: false, null, ex.Message, - ConnectionLevelFailure: connectionLevel)); + if (r.Success) + tagsToSeed.Add(r.TagPath); } } @@ -782,18 +831,129 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers }).PipeTo(self); } + /// + /// Subscribes a CHUNK of tags against and returns one row + /// per requested tag. An adapter that advertises + /// does it in one round trip; any other + /// adapter falls back to the historical per-tag loop, so the capability is purely + /// additive. + /// + /// + /// Partial-failure contract: a per-tag fault is a result row with + /// Success:false. A THROWN exception means the whole chunk failed, and is + /// classified with exactly as the per-tag path + /// does — connection-level faults drive the reconnect state machine, everything else + /// is treated as a tag-resolution failure and retried on the backoff timer. + /// + /// + /// Runs on a background task: it touches no actor state. + /// + private static async Task> SubscribeTagsAsync( + IDataConnection adapter, IReadOnlyList tags, SubscriptionCallback callback) + { + var results = new List(tags.Count); + if (tags.Count == 0) + return results; + + if (adapter is IBatchSubscribableConnection batchAdapter) + { + IReadOnlyList rows; + try + { + rows = await batchAdapter.SubscribeBatchAsync(tags, callback); + } + catch (Exception ex) + { + var connectionLevel = IsConnectionLevelFailure(ex); + foreach (var tagPath in tags) + { + results.Add(new SubscribeTagResult( + tagPath, AlreadySubscribed: false, Success: false, null, ex.Message, + ConnectionLevelFailure: connectionLevel)); + } + return results; + } + + var byTag = new Dictionary(rows.Count, StringComparer.Ordinal); + foreach (var row in rows) + byTag[row.TagPath] = row; + + foreach (var tagPath in tags) + { + results.Add(byTag.TryGetValue(tagPath, out var row) + ? new SubscribeTagResult( + tagPath, AlreadySubscribed: false, row.Success, row.SubscriptionId, row.ErrorMessage) + : new SubscribeTagResult( + tagPath, AlreadySubscribed: false, Success: false, null, + "Adapter returned no subscribe result for this tag.")); + } + + return results; + } + + foreach (var tagPath in tags) + { + try + { + var subId = await adapter.SubscribeAsync(tagPath, callback); + results.Add(new SubscribeTagResult(tagPath, AlreadySubscribed: false, Success: true, subId, null)); + } + catch (Exception ex) + { + // Distinguish a connection-level fault + // (adapter not connected / transport down) from a genuine + // node-not-found. Connection-level faults must drive the + // reconnection state machine, not be retried as unresolved tags. + var connectionLevel = IsConnectionLevelFailure(ex); + results.Add(new SubscribeTagResult( + tagPath, AlreadySubscribed: false, Success: false, null, ex.Message, + ConnectionLevelFailure: connectionLevel)); + } + } + + return results; + } + + /// + /// Releases adapter subscription handles, in one round trip where the adapter supports + /// it. Fire-and-forget at every call site (the actor never awaits a release), so + /// faults are swallowed the same way the per-tag UnsubscribeAsync calls were. + /// + private static Task UnsubscribeIdsAsync(IDataConnection adapter, IReadOnlyList subscriptionIds) + { + if (subscriptionIds.Count == 0) + return Task.CompletedTask; + + if (adapter is IBatchSubscribableConnection batchAdapter) + return batchAdapter.UnsubscribeBatchAsync(subscriptionIds); + + return Task.WhenAll(subscriptionIds.Select(id => adapter.UnsubscribeAsync(id))); + } + /// /// Reads the current value of each tag so the Instance Actor /// has an initial value, retrying the still-empty subset a bounded number of times. A /// STATIC tag (one that emits no further OnDataChange after the advise) depends /// entirely on this seed; on a cold/fresh advise the read can race the just-created /// subscription and return an empty/failed result, which pre-fix was swallowed — - /// leaving the attribute Uncertain forever even though the source reads Good. Per-tag - /// (not a single bulk read) is deliberate: - /// some gateways time out on a large batch. The retry delay is applied once per round - /// across the whole pending subset, so total added latency is bounded to - /// (attempts - 1) × SeedReadRetryDelay regardless of tag count. Tags still - /// empty after the budget are logged (named) and left to heal from a future change. + /// leaving the attribute Uncertain forever even though the source reads Good. + /// + /// + /// Reads are issued as CHUNKED bulk reads ( + /// tags per chunk, chunks in + /// flight) against a batch-capable adapter, and per tag otherwise. + /// now bounds a CHUNK rather than a + /// single tag — chunking plus that per-chunk bound answers the "some gateways time out + /// on a large batch" caveat that originally motivated per-tag reads — and the whole + /// seed (all chunks, all retry rounds) is bounded by + /// , replacing the old + /// 30s-per-tag serial worst case. + /// + /// + /// The retry delay is applied once per round across the whole pending subset, so total + /// added latency is bounded to (attempts - 1) × SeedReadRetryDelay regardless of + /// tag count. Tags still empty after the budget (or when the deadline expires) are + /// logged (named) and left to heal from a future change. /// Runs on a background task: it reads only the supplied /// and returns the seeds — all actor-state mutation/delivery stays on the actor thread. /// @@ -805,35 +965,60 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers var pending = new HashSet(tags); var attempts = Math.Max(1, _options.SeedReadMaxAttempts); - for (var attempt = 1; attempt <= attempts && pending.Count > 0; attempt++) + var chunkSize = Math.Max(1, _options.SeedReadBatchSize); + var parallelism = Math.Max(1, _options.SeedReadMaxParallelism); + var useBatch = adapter is IBatchSubscribableConnection; + + // ONE deadline across every chunk and every retry round. + using var overall = new CancellationTokenSource(_options.SeedOverallTimeout); + var deadlineHit = false; + + for (var attempt = 1; attempt <= attempts && pending.Count > 0 && !overall.IsCancellationRequested; attempt++) { - foreach (var tagPath in pending.ToList()) + var round = pending.ToList(); + var chunks = new List>(); + for (var offset = 0; offset < round.Count; offset += chunkSize) + chunks.Add(round.GetRange(offset, Math.Min(chunkSize, round.Count - offset))); + + var gate = new SemaphoreSlim(parallelism); + var chunkTasks = chunks.Select(async chunk => { - // Bound each per-tag read with SeedReadTimeout so a - // hung device read cannot delay SubscribeCompleted/the ack indefinitely. - // On timeout the catch block treats it identically to any other failed seed - // read — the tag stays in pending, is retried up to SeedReadMaxAttempts, and - // left Uncertain if still empty after the budget. Same CancellationTokenSource - // mechanism used by HandleWrite for WriteTimeout. - using var cts = new CancellationTokenSource(_options.SeedReadTimeout); + await gate.WaitAsync(CancellationToken.None); try { - var readResult = await adapter.ReadAsync(tagPath, cts.Token); - if (readResult.Success && readResult.Value is { Value: not null } value) - { - seedValues.Add(new SeededValue(tagPath, value)); - pending.Remove(tagPath); - } + return await ReadSeedChunkAsync(adapter, chunk, useBatch, overall.Token); } - catch + finally { - // Best-effort read — retried below, or logged once the budget is spent. - // Includes OperationCanceledException on SeedReadTimeout expiry. + gate.Release(); } + }).ToList(); + + var chunkResults = await Task.WhenAll(chunkTasks); + foreach (var seeded in chunkResults.SelectMany(r => r)) + { + if (pending.Remove(seeded.TagPath)) + seedValues.Add(seeded); + } + + if (overall.IsCancellationRequested) + { + deadlineHit = true; + break; } if (pending.Count > 0 && attempt < attempts) - await Task.Delay(_options.SeedReadRetryDelay); + { + try + { + await Task.Delay(_options.SeedReadRetryDelay, overall.Token); + } + catch (OperationCanceledException) + { + deadlineHit = true; + break; + } + } } if (pending.Count > 0) @@ -843,13 +1028,68 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers if (pending.Count > maxNamed) named += $", … (+{pending.Count - maxNamed} more)"; _log.Warning( - "[{0}] Seed read returned no value for {1} tag(s) after {2} attempt(s); they stay Uncertain until a change notification arrives: {3}", - _connectionName, pending.Count, attempts, named); + "[{0}] Seed read returned no value for {1} tag(s) after {2} attempt(s){3}; they stay Uncertain until a change notification arrives: {4}", + _connectionName, pending.Count, attempts, + deadlineHit ? $" (overall {_options.SeedOverallTimeout.TotalSeconds:F0}s seed deadline expired)" : string.Empty, + named); } return seedValues; } + /// + /// Reads one seed chunk, bounded by + /// and by the caller's overall seed deadline. Every failure mode (bad read, chunk + /// timeout, deadline) yields "no seed for these tags", which the caller retries or + /// leaves Uncertain — identical to the historical per-tag behaviour. + /// + private async Task> ReadSeedChunkAsync( + IDataConnection adapter, IReadOnlyList chunk, bool useBatch, CancellationToken overallToken) + { + var seeded = new List(chunk.Count); + using var cts = CancellationTokenSource.CreateLinkedTokenSource(overallToken); + cts.CancelAfter(_options.SeedReadTimeout); + + if (useBatch) + { + try + { + var results = await adapter.ReadBatchAsync(chunk, cts.Token); + foreach (var tagPath in chunk) + { + if (results.TryGetValue(tagPath, out var readResult) + && readResult.Success && readResult.Value is { Value: not null } value) + { + seeded.Add(new SeededValue(tagPath, value)); + } + } + } + catch + { + // Best-effort read — retried by the caller, or logged once the budget is + // spent. Includes OperationCanceledException on the chunk/overall deadline. + } + + return seeded; + } + + foreach (var tagPath in chunk) + { + try + { + var readResult = await adapter.ReadAsync(tagPath, cts.Token); + if (readResult.Success && readResult.Value is { Value: not null } value) + seeded.Add(new SeededValue(tagPath, value)); + } + catch + { + // Best-effort read — see above. + } + } + + return seeded; + } + /// /// Applies the result of an asynchronous subscribe on the actor thread. ALL mutation /// of subscription state and counters happens here — never on the background task — @@ -880,6 +1120,7 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers "{2} adapter handle(s) and discarding state mutations.", _connectionName, instanceName, msg.Results.Count(r => r.Success && !r.AlreadySubscribed)); + var orphanedIds = new List(); foreach (var result in msg.Results) { // Clear in-flight markers we placed in HandleSubscribe. @@ -891,10 +1132,12 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers // owns the adapter handle and unsubscribing it would break them. if (result is { Success: true, AlreadySubscribed: false, SubscriptionId: not null }) { - _ = _adapter.UnsubscribeAsync(result.SubscriptionId); + orphanedIds.Add(result.SubscriptionId); } } + _ = UnsubscribeIdsAsync(_adapter, orphanedIds); + // The original sender is already gone (unsubscribed). Telling a dead // ref produces a dead letter, which is the harmless and observable // outcome — but skipping the reply altogether keeps dead-letter noise @@ -909,6 +1152,10 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers // request be re-stashed/retried after reconnect via ReSubscribeAll. var connectionLevelFailure = msg.Results.Any(r => !r.Success && r.ConnectionLevelFailure); + // Handles this request created that turn out to be redundant; released in ONE + // round trip below, symmetric with HandleBatchSubscribeCompleted. + var redundantIds = new List(); + foreach (var result in msg.Results) { // A result with AlreadySubscribed: false means @@ -932,9 +1179,26 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers // Re-check against current state: another subscribe may have resolved the // same tag while this request's I/O was in flight. - if (result.AlreadySubscribed || _subscriptionIds.ContainsKey(result.TagPath)) + // AlreadySubscribed rows are partitioned out in HandleSubscribe and carry NO + // SubscriptionId of their own (another caller owns the adapter handle), so + // there is nothing to release for them. + if (result.AlreadySubscribed) continue; + if (_subscriptionIds.ContainsKey(result.TagPath)) + { + // Another path (a resolution probe, a reconnect re-subscribe or a + // concurrent request for a different instance) stored a handle for this + // tag while this request's I/O was in flight. This request issued its OWN + // SubscribeAsync, so dropping the row without releasing its handle leaks a + // monitored item forever — _subscriptionIds holds a single id per tag, so + // no later unsubscribe can ever reference this one. Release it, exactly as + // HandleBatchSubscribeCompleted does for its duplicate rows. + if (result is { Success: true, SubscriptionId: not null }) + redundantIds.Add(result.SubscriptionId); + continue; + } + if (result.Success) { _subscriptionIds[result.TagPath] = result.SubscriptionId!; @@ -996,6 +1260,9 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers } } + // Fire-and-forget release of every handle this request created redundantly. + _ = UnsubscribeIdsAsync(_adapter, redundantIds); + // Now that every tag is registered in // _subscriptionsByInstance, deliver the values captured by the initial read. // Re-entering via Self reuses HandleTagValueReceived's generation guard, fan-out @@ -1012,20 +1279,7 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers } // Start the tag-resolution retry timer if any tags are unresolved. - // StartPeriodicTimer with an existing key CANCELS - // and replaces the prior timer, so a fan-out of SubscribeTagsRequests - // arriving faster than TagResolutionRetryInterval would keep resetting - // the timer and starve the retry indefinitely. Gating on IsTimerActive - // means the first failure starts the timer and subsequent failures - // simply pile onto _unresolvedTags without restarting the clock. - if (_unresolvedTags.Count > 0 && !Timers.IsTimerActive("tag-resolution-retry")) - { - Timers.StartPeriodicTimer( - "tag-resolution-retry", - new RetryTagResolution(), - _options.TagResolutionRetryInterval, - _options.TagResolutionRetryInterval); - } + ScheduleTagResolutionRetry(); // The response must match the actor's own assessment. // When a connection-level failure is driving the actor into Reconnecting, the @@ -1070,6 +1324,10 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers if (!_subscriptionsByInstance.TryGetValue(request.InstanceUniqueName, out var tags)) return; + // Released adapter handles are collected and released in ONE round trip below + // (batch-capable adapters) instead of one call per tag. + var idsToRelease = new List(); + // Cleanup on Instance Actor stop foreach (var tagPath in tags) { @@ -1095,7 +1353,7 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers // an unconditional Remove was always-true dead logic). if (_subscriptionIds.TryGetValue(tagPath, out var subId)) { - _ = _adapter.UnsubscribeAsync(subId); + idsToRelease.Add(subId); _subscriptionIds.Remove(tagPath); _resolutionInFlight.Remove(tagPath); _totalSubscribed--; @@ -1130,9 +1388,13 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers _subscriptionsByInstance.Remove(request.InstanceUniqueName); _subscribers.Remove(request.InstanceUniqueName); + // Fire-and-forget release of every handle this unsubscribe orphaned. + _ = UnsubscribeIdsAsync(_adapter, idsToRelease); + // Keep the reported quality counters in sync after the - // unsubscribed tags' buckets were decremented above. - _healthCollector.UpdateTagQuality(_connectionName, _tagsGoodQuality, _tagsBadQuality, _tagsUncertainQuality); + // unsubscribed tags' buckets were decremented above. SYNCHRONOUS flush: the + // coalescing timer must never delay a count that just dropped. + FlushQualityCounters(); _healthCollector.UpdateTagResolution(_connectionName, _totalSubscribed, _resolvedTags); } @@ -1544,7 +1806,7 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers var self = Self; // Only dispatch retries for tags that do not already - // have an attempt in flight. A slow SubscribeAsync overlapping the next tick + // have an attempt in flight. A slow subscribe overlapping the next tick // would otherwise produce duplicate concurrent subscribes for the same tag. var toResolve = _unresolvedTags.Where(t => !_resolutionInFlight.Contains(t)).ToList(); @@ -1555,22 +1817,200 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers return; } - _log.Debug("[{0}] Retrying resolution for {1} unresolved tags", _connectionName, toResolve.Count); + _log.Debug("[{0}] Retrying resolution for {1} unresolved tags (interval {2:F0}s)", + _connectionName, toResolve.Count, _tagResolutionInterval.TotalSeconds); var generation = _adapterGeneration; + var adapter = _adapter; + var batchSize = Math.Max(1, _options.SubscribeBatchSize); + foreach (var tagPath in toResolve) - { _resolutionInFlight.Add(tagPath); - _adapter.SubscribeAsync(tagPath, (path, value) => + + // One probe ROUND = chunked batch subscribes over the whole not-in-flight + // unresolved set, reported back as a single completion. Rescheduling happens on + // that completion (never on a periodic tick), which is what makes the backoff + // possible AND preserves the anti-starvation property the old IsTimerActive gate + // defended: a fan-out of failures can never reset the clock. + Task.Run(async () => + { + var rows = new List(toResolve.Count); + for (var offset = 0; offset < toResolve.Count; offset += batchSize) { - self.Tell(new TagValueReceived(path, value, generation)); - }).ContinueWith(t => - { - if (t.IsCompletedSuccessfully) - return new TagResolutionSucceeded(tagPath, t.Result) as object; - return new TagResolutionFailed(tagPath, t.Exception?.GetBaseException().Message ?? "Unknown error"); - }).PipeTo(self); + var chunk = toResolve.GetRange(offset, Math.Min(batchSize, toResolve.Count - offset)); + rows.AddRange(await SubscribeTagsAsync(adapter, chunk, (path, value) => + { + self.Tell(new TagValueReceived(path, value, generation)); + })); + } + + return new BatchSubscribeCompleted(rows, generation, BatchSubscribeSource.ResolutionProbe); + }).PipeTo(self); + } + + /// + /// Arms the single-shot tag-resolution retry timer at the current backoff interval + /// when unresolved tags remain, and cancels it when none do. Single-shot (rescheduled + /// on probe completion) rather than periodic — a periodic timer cannot back off, and + /// re-arming a periodic timer on every failure is what starved the retry before + /// (DataConnectionLayer-022). Gating on IsTimerActive keeps a burst of failed + /// subscribes from pushing the already-running deadline out. + /// + private void ScheduleTagResolutionRetry() + { + if (_unresolvedTags.Count == 0) + { + Timers.Cancel("tag-resolution-retry"); + return; } + + if (Timers.IsTimerActive("tag-resolution-retry")) + return; + + Timers.StartSingleTimer("tag-resolution-retry", new RetryTagResolution(), _tagResolutionInterval); + } + + /// Resets the tag-resolution backoff to its floor (a tag resolved, or the connection reconnected). + private void ResetTagResolutionBackoff() => _tagResolutionInterval = _options.TagResolutionRetryInterval; + + /// + /// Doubles the tag-resolution retry interval, capped at + /// . Called only after + /// a probe round in which NOTHING resolved, so a device that is merely slow to boot is + /// still picked up at the floor interval. + /// + private void BackOffTagResolution() => + _tagResolutionInterval = NextTagResolutionInterval( + _tagResolutionInterval, _options.TagResolutionRetryInterval, _options.TagResolutionRetryMaxInterval); + + /// + /// Pure backoff step: the interval after a fully-failed probe round — double the + /// current one, never above (itself never below + /// , so a misconfigured ceiling degrades to a fixed interval + /// rather than shrinking below the floor). + /// + /// Interval that has just been used. + /// Configured floor interval. + /// Configured ceiling interval. + /// The next retry interval. + internal static TimeSpan NextTagResolutionInterval(TimeSpan current, TimeSpan floor, TimeSpan max) + { + var ceiling = max < floor ? floor : max; + if (current < floor) + current = floor; + + var doubled = current + current; + return doubled > ceiling ? ceiling : doubled; + } + + /// + /// Applies one chunk of batch-subscribe results on the actor thread — the shared tail + /// of the reconnect re-subscribe and the tag-resolution probe. Both paths subscribe + /// tags that are ALREADY counted in (they come from + /// ), so neither touches that counter. + /// + private void HandleBatchSubscribeCompleted(BatchSubscribeCompleted msg) + { + // Results produced by a disposed adapter (post-failover) are dropped, mirroring + // the TagValueReceived generation guard. + if (msg.Generation != _adapterGeneration) + { + _log.Debug("[{0}] Dropping {1} stale batch-subscribe result(s) from adapter generation {2} (current {3})", + _connectionName, msg.Results.Count, msg.Generation, _adapterGeneration); + return; + } + + var anyResolved = false; + var idsToRelease = new List(); + + foreach (var row in msg.Results) + { + _resolutionInFlight.Remove(row.TagPath); + + if (row is { Success: true, SubscriptionId: not null }) + { + var wasUnresolved = _unresolvedTags.Remove(row.TagPath); + + // The tag lost its last subscriber while this batch was in flight — an + // instance unsubscribe (disable/undeploy/redeploy) raced the resolution + // probe or the reconnect re-subscribe. _instancesByTag is the authority: + // it is the inverse of _subscriptionsByInstance and, like it, is preserved + // across reconnect, so an empty entry means nobody wants this tag any more. + // Applying the row anyway would store a handle no future unsubscribe can + // ever release (HandleUnsubscribe already ran for this tag) AND push + // _resolvedTags above the _totalSubscribed that same unsubscribe just + // decremented — permanently corrupting the health counters and driving + // _totalSubscribed negative on the next redeploy round trip. So discard + // every state mutation for this row and release the adapter handle the + // batch just created. This restores the gate the pre-merge per-tag handler + // had (it applied only when the tag was still in _unresolvedTags) and + // additionally releases the handle that path leaked. + if (!_instancesByTag.ContainsKey(row.TagPath)) + { + _log.Debug( + "[{0}] Discarding batch-subscribe result for {1} — the tag was " + + "unsubscribed while the subscribe was in flight; releasing its handle.", + _connectionName, row.TagPath); + idsToRelease.Add(row.SubscriptionId); + continue; + } + + if (_subscriptionIds.ContainsKey(row.TagPath)) + { + // Another path already stored a handle for this tag while this one was + // in flight — release the redundant handle instead of leaking it + // (mirrors the duplicate-alarm-feed guard). + idsToRelease.Add(row.SubscriptionId); + } + else + { + _subscriptionIds[row.TagPath] = row.SubscriptionId; + _resolvedTags++; + anyResolved = true; + if (wasUnresolved) + _log.Info("[{0}] Tag resolved: {1}", _connectionName, row.TagPath); + } + } + else if (row.ConnectionLevelFailure) + { + // Connection-level fault: not a tag-resolution problem. The reconnect + // cycle re-issues everything from _subscriptionsByInstance. + _log.Warning("[{0}] Batch subscribe for {1} failed at connection level: {2}", + _connectionName, row.TagPath, row.Error); + } + else + { + _log.Debug("[{0}] Tag resolution still failing for {1}: {2}", + _connectionName, row.TagPath, row.Error); + + // Same in-flight-unsubscribe race as the success branch above: the tag was + // already dropped from _unresolvedTags AND from _totalSubscribed by + // HandleUnsubscribe, so re-adding it here would probe a tag nobody + // subscribes to forever, at a retry count TotalSubscribedTags no longer + // accounts for. + if (_instancesByTag.ContainsKey(row.TagPath)) + _unresolvedTags.Add(row.TagPath); + } + } + + _ = UnsubscribeIdsAsync(_adapter, idsToRelease); + _healthCollector.UpdateTagResolution(_connectionName, _totalSubscribed, _resolvedTags); + + // Backoff bookkeeping belongs to the probe round only: a reconnect re-subscribe + // chunk is not a "retry round" and must not double the interval. + if (msg.Source == BatchSubscribeSource.ResolutionProbe) + { + if (anyResolved) + ResetTagResolutionBackoff(); + else + BackOffTagResolution(); + } + else if (anyResolved) + { + ResetTagResolutionBackoff(); + } + + ScheduleTagResolutionRetry(); } // ── Bad Quality Push ── @@ -1592,6 +2032,44 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers _tagsBadQuality = _lastTagQuality.Count; foreach (var key in _lastTagQuality.Keys.ToList()) _lastTagQuality[key] = QualityCode.Bad; + // SYNCHRONOUS — "immediate bad quality on disconnect" must never be deferred to + // the quality-flush coalescing timer. + FlushQualityCounters(); + } + + // ── Quality counter flush ── + + /// + /// Marks the quality counters dirty and arms the coalescing flush timer. Called only + /// on a genuine quality TRANSITION — an unchanged quality moves no counter, so there + /// is nothing to push. + /// + private void MarkQualityDirty() + { + _qualityDirty = true; + if (!Timers.IsTimerActive("quality-flush")) + Timers.StartSingleTimer("quality-flush", new QualityFlushTick(), _options.QualityFlushInterval); + } + + /// + /// Coalescing-timer tick: pushes only when a transition is still pending. A tick that + /// races a synchronous flush (disconnect / unsubscribe) does nothing. + /// + private void FlushQualityCountersIfDirty() + { + if (_qualityDirty) + FlushQualityCounters(); + } + + /// + /// Pushes the current quality counters to the health collector and disarms the + /// coalescing timer. Used both by the timer tick and by the paths whose correctness + /// depends on an immediate push (disconnect, unsubscribe, reconnect reset). + /// + private void FlushQualityCounters() + { + _qualityDirty = false; + Timers.Cancel("quality-flush"); _healthCollector.UpdateTagQuality(_connectionName, _tagsGoodQuality, _tagsBadQuality, _tagsUncertainQuality); } @@ -1632,25 +2110,51 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers _tagsGoodQuality = 0; _tagsBadQuality = 0; _tagsUncertainQuality = 0; - _healthCollector.UpdateTagQuality(_connectionName, _tagsGoodQuality, _tagsBadQuality, _tagsUncertainQuality); + // SYNCHRONOUS reset push — never deferred to the coalescing timer. + FlushQualityCounters(); + + // A reconnect starts from the retry floor: the device just answered, so any + // backoff accumulated against the previous outage is stale. + ResetTagResolutionBackoff(); var generation = _adapterGeneration; - // Capture the adapter up front (S7), symmetric with reseedAdapter below. This loop - // runs on the actor thread so a mid-loop swap can't happen, but reading the local - // keeps every adapter access in this method tied to one generation. + // Capture the adapter up front (S7), symmetric with reseedAdapter below, so every + // chunk runs against the adapter that started the re-subscribe even if a later + // failover swaps the _adapter field mid-flight. var subscribeAdapter = _adapter; - foreach (var tagPath in allTags) + var batchSize = Math.Max(1, _options.SubscribeBatchSize); + var chunkDelay = _options.SubscribeBatchDelay; + + // Bounded, sequential chunks in ONE background task — replacing the previous + // "one fire-and-forget task per tag" storm (37,500 concurrent subscribes at the + // sizing target). Sequential because subscription creation on one session is + // serialized by the adapter's apply lock anyway, and sequencing bounds device load; + // the inter-chunk delay paces it further. Each chunk reports its per-tag results + // back to the actor as its own message, so progress is applied incrementally. + Task.Run(async () => { - subscribeAdapter.SubscribeAsync(tagPath, (path, value) => + for (var offset = 0; offset < allTags.Count; offset += batchSize) { - self.Tell(new TagValueReceived(path, value, generation)); - }).ContinueWith(t => - { - if (t.IsCompletedSuccessfully) - return new TagResolutionSucceeded(tagPath, t.Result) as object; - return new TagResolutionFailed(tagPath, t.Exception?.GetBaseException().Message ?? "Unknown error"); - }).PipeTo(self); - } + if (offset > 0 && chunkDelay > TimeSpan.Zero) + await Task.Delay(chunkDelay); + + var chunk = allTags.GetRange(offset, Math.Min(batchSize, allTags.Count - offset)); + try + { + var rows = await SubscribeTagsAsync(subscribeAdapter, chunk, (path, value) => + { + self.Tell(new TagValueReceived(path, value, generation)); + }); + self.Tell(new BatchSubscribeCompleted(rows, generation, BatchSubscribeSource.Resubscribe)); + } + catch (Exception ex) + { + // SubscribeTagsAsync already converts faults into rows; reaching here is + // unexpected, so guarantee a trace rather than an unobserved task fault. + _log.Warning("[{0}] Reconnect re-subscribe chunk faulted: {1}", _connectionName, ex.Message); + } + } + }); // Re-advising alone does NOT restore a STATIC tag's value. // PushBadQualityForAllTags flipped every tag Bad on disconnect, and a tag that fires @@ -1697,47 +2201,6 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers // ── Internal message handlers for piped async results ── - private void HandleTagResolutionSucceeded(TagResolutionSucceeded msg) - { - // The retry attempt for this tag has completed. - _resolutionInFlight.Remove(msg.TagPath); - - if (_unresolvedTags.Remove(msg.TagPath)) - { - _subscriptionIds[msg.TagPath] = msg.SubscriptionId; - _resolvedTags++; - _healthCollector.UpdateTagResolution(_connectionName, _totalSubscribed, _resolvedTags); - _log.Info("[{0}] Tag resolved: {1}", _connectionName, msg.TagPath); - } - - if (_unresolvedTags.Count == 0) - { - Timers.Cancel("tag-resolution-retry"); - } - } - - private void HandleTagResolutionFailed(TagResolutionFailed msg) - { - _log.Debug("[{0}] Tag resolution still failing for {1}: {2}", - _connectionName, msg.TagPath, msg.Error); - - // The retry attempt for this tag has completed — - // it is eligible for the next retry tick again. - _resolutionInFlight.Remove(msg.TagPath); - - // Track as unresolved so periodic retry picks it up. Gate on - // IsTimerActive so a stream of TagResolutionFailed events doesn't keep - // cancelling and re-starting the timer faster than its own interval. - if (_unresolvedTags.Add(msg.TagPath) && !Timers.IsTimerActive("tag-resolution-retry")) - { - Timers.StartPeriodicTimer( - "tag-resolution-retry", - new RetryTagResolution(), - _options.TagResolutionRetryInterval, - _options.TagResolutionRetryInterval); - } - } - /// /// Adds to the reverse index for . /// Call at every site that adds a tag to an instance's set. @@ -1789,8 +2252,17 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers } } - // Track quality transitions - if (_lastTagQuality.TryGetValue(msg.TagPath, out var prevQuality)) + // Track quality transitions. A value whose quality is UNCHANGED moves no bucket — + // the old code still ran the decrement/increment pair and pushed the identical + // counters to the health collector on every single value. At 37,500 tags that push + // was the dominant cost of the value hot path, against a collector only read every + // 30s. Now: unchanged quality is a no-op; a genuine transition updates the buckets + // and arms the coalescing flush timer. + var hadPrevious = _lastTagQuality.TryGetValue(msg.TagPath, out var prevQuality); + if (hadPrevious && prevQuality == msg.Value.Quality) + return; + + if (hadPrevious) { // Decrement old quality bucket switch (prevQuality) @@ -1808,7 +2280,7 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers case QualityCode.Uncertain: _tagsUncertainQuality++; break; } _lastTagQuality[msg.TagPath] = msg.Value.Quality; - _healthCollector.UpdateTagQuality(_connectionName, _tagsGoodQuality, _tagsBadQuality, _tagsUncertainQuality); + MarkQualityDirty(); } // ── Native alarm subscriptions ── @@ -1832,6 +2304,7 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers { subs = new HashSet(); _alarmSourceSubscribers[request.SourceReference] = subs; + IndexAlarmSource(request.SourceReference); } subs.Add(subscriber); // S10: the adapter feed carries a single shared condition filter per source. @@ -1971,8 +2444,11 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers return; } - foreach (var (sourceRef, subs) in _alarmSourceSubscribers) + foreach (var sourceRef in CandidateAlarmSources(transition)) { + if (!_alarmSourceSubscribers.TryGetValue(sourceRef, out var subs)) + continue; + // A subscriber bound to source S receives a transition whose source // object (or full reference) falls under S. var match = transition.SourceObjectReference.StartsWith(sourceRef, StringComparison.Ordinal) @@ -1996,6 +2472,86 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers } } + /// + /// First path segment of a source reference (everything before the first .), or + /// the whole reference when it carries no separator. + /// + private static string FirstSegment(string reference) + { + var separator = reference.IndexOf('.', StringComparison.Ordinal); + return separator < 0 ? reference : reference[..separator]; + } + + /// + /// Adds a source reference to the routing index. A reference containing a separator + /// is bucketed by its first segment; one WITHOUT a separator is a prefix shorter than + /// a full segment (it can match transitions whose first segment merely starts with it) + /// and goes to the small residue list that every transition is checked against. + /// + private void IndexAlarmSource(string sourceReference) + { + if (sourceReference.IndexOf('.', StringComparison.Ordinal) < 0) + { + _alarmSourcesWithoutSeparator.Add(sourceReference); + return; + } + + var segment = FirstSegment(sourceReference); + if (!_alarmSourcesByFirstSegment.TryGetValue(segment, out var bucket)) + { + bucket = new HashSet(StringComparer.Ordinal); + _alarmSourcesByFirstSegment[segment] = bucket; + } + bucket.Add(sourceReference); + } + + /// Removes a source reference from the routing index, dropping an emptied bucket. + private void UnindexAlarmSource(string sourceReference) + { + if (sourceReference.IndexOf('.', StringComparison.Ordinal) < 0) + { + _alarmSourcesWithoutSeparator.Remove(sourceReference); + return; + } + + var segment = FirstSegment(sourceReference); + if (_alarmSourcesByFirstSegment.TryGetValue(segment, out var bucket) + && bucket.Remove(sourceReference) && bucket.Count == 0) + { + _alarmSourcesByFirstSegment.Remove(segment); + } + } + + /// + /// Source references that could possibly match this transition: the buckets for the + /// first segment of its source-object and full references, plus the separator-less + /// residue. Sound because a source reference containing a separator can only be a + /// prefix of a reference sharing its ENTIRE first segment — everything before the + /// first . must match character for character. The caller still applies the + /// full StartsWith test and the condition-type gate, so routing is identical to the + /// previous linear scan. + /// + private IEnumerable CandidateAlarmSources(NativeAlarmTransition transition) + { + var objectSegment = FirstSegment(transition.SourceObjectReference); + if (_alarmSourcesByFirstSegment.TryGetValue(objectSegment, out var objectBucket)) + { + foreach (var sourceRef in objectBucket) + yield return sourceRef; + } + + var referenceSegment = FirstSegment(transition.SourceReference); + if (!string.Equals(referenceSegment, objectSegment, StringComparison.Ordinal) + && _alarmSourcesByFirstSegment.TryGetValue(referenceSegment, out var referenceBucket)) + { + foreach (var sourceRef in referenceBucket) + yield return sourceRef; + } + + foreach (var sourceRef in _alarmSourcesWithoutSeparator) + yield return sourceRef; + } + private void HandleUnsubscribeAlarms(UnsubscribeAlarmsRequest request) { if (!_alarmSourceSubscribers.TryGetValue(request.SourceReference, out var subs)) @@ -2007,6 +2563,7 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers // No subscribers remain for this source — tear down the adapter feed. _alarmSourceSubscribers.Remove(request.SourceReference); + UnindexAlarmSource(request.SourceReference); _alarmSourceFilter.Remove(request.SourceReference); _alarmSourceFilterPredicate.Remove(request.SourceReference); // Clear the in-flight marker so that if an adapter @@ -2064,9 +2621,27 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers internal record ConnectResult(bool Success, string? Error); internal record AdapterDisconnected; internal record TagValueReceived(string TagPath, TagValue Value, int AdapterGeneration); - internal record TagResolutionFailed(string TagPath, string Error); - internal record TagResolutionSucceeded(string TagPath, string SubscriptionId); internal record RetryTagResolution; + + /// Coalescing-timer tick for the tag-quality counter push. + internal record QualityFlushTick; + + /// Which path produced a . + internal enum BatchSubscribeSource + { + /// One chunk of the post-reconnect transparent re-subscribe. + Resubscribe, + + /// A whole tag-resolution retry round (all chunks), which drives the backoff. + ResolutionProbe + } + + /// + /// Per-tag results of a batch subscribe issued off the actor thread. Carries the + /// adapter generation it ran against so post-failover results are dropped. + /// + internal record BatchSubscribeCompleted( + IReadOnlyList Results, int Generation, BatchSubscribeSource Source); internal record SubscribeTagResult( string TagPath, bool AlreadySubscribed, bool Success, string? SubscriptionId, string? Error, bool ConnectionLevelFailure = false); diff --git a/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/AlarmFilterPrefix.cs b/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/AlarmFilterPrefix.cs new file mode 100644 index 00000000..e5e2e368 --- /dev/null +++ b/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/AlarmFilterPrefix.cs @@ -0,0 +1,63 @@ +namespace ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Adapters; + +/// +/// Computes the single source-reference prefix an MxAccess Gateway alarm stream can be +/// opened with while still covering every subscribed source. StreamAlarms carries +/// ONE alarm_filter_prefix, so the only pushable union of N source references is +/// their longest common prefix. +/// +/// +/// The prefix is a BANDWIDTH optimisation, never correctness: the DataConnectionActor's +/// per-source match plus condition-type gate stays authoritative, mirroring the OPC UA +/// server-side WhereClause stance. A too-broad prefix (including the empty, +/// gateway-wide one) is therefore always safe. +/// +/// +internal static class AlarmFilterPrefix +{ + /// + /// Longest common prefix of the supplied source references, or the empty string when + /// they share none (or when any source is empty — an empty source means "everything"). + /// + /// Currently subscribed source references. + /// The prefix to open the gateway alarm stream with; empty = gateway-wide. + public static string LongestCommonPrefix(IEnumerable sourceReferences) + { + string? prefix = null; + foreach (var source in sourceReferences) + { + if (string.IsNullOrEmpty(source)) + return string.Empty; + + if (prefix is null) + { + prefix = source; + continue; + } + + var max = Math.Min(prefix.Length, source.Length); + var common = 0; + while (common < max && prefix[common] == source[common]) + common++; + + if (common == 0) + return string.Empty; + prefix = prefix[..common]; + } + + return prefix ?? string.Empty; + } + + /// + /// Whether an existing stream opened with already + /// covers . Only a source falling OUTSIDE the + /// current prefix justifies restarting the stream; an unsubscribe never does, + /// because the prefix it leaves behind is at worst too broad — which costs + /// bandwidth, not correctness. + /// + /// Prefix the live stream was opened with. + /// Newly subscribed source reference. + /// true when the live stream already carries this source. + public static bool Covers(string currentPrefix, string sourceReference) => + currentPrefix.Length == 0 || sourceReference.StartsWith(currentPrefix, StringComparison.Ordinal); +} diff --git a/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/AsyncSerialGate.cs b/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/AsyncSerialGate.cs new file mode 100644 index 00000000..88f44cac --- /dev/null +++ b/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/AsyncSerialGate.cs @@ -0,0 +1,55 @@ +namespace ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Adapters; + +/// +/// Serializes async critical sections — one caller inside at a time, FIFO. Used by +/// so every structural mutation of an OPC UA Subscription +/// (AddItem / RemoveItem / ApplyChanges / Create / Delete) and every shard-list mutation +/// runs alone: the OPC Foundation Subscription object is not safe under concurrent +/// structural mutation, and nothing else orders the subscribe background task, the +/// tag-resolution probe and an alarm subscribe against each other. +/// +/// +/// A named type rather than a bare field so the "exactly one +/// gate, released on every path" discipline is visible at each call site and testable on +/// its own. +/// +/// +internal sealed class AsyncSerialGate +{ + private readonly SemaphoreSlim _gate = new(1, 1); + + /// Runs with no other gated section in flight. + /// The critical section. + /// Cancellation token observed while waiting to enter. + /// A task that completes when the section has run and the gate is released. + public async Task RunAsync(Func action, CancellationToken cancellationToken = default) + { + await _gate.WaitAsync(cancellationToken).ConfigureAwait(false); + try + { + await action().ConfigureAwait(false); + } + finally + { + _gate.Release(); + } + } + + /// Value-returning counterpart of . + /// Result type of the critical section. + /// The critical section. + /// Cancellation token observed while waiting to enter. + /// The section's result. + public async Task RunAsync(Func> action, CancellationToken cancellationToken = default) + { + await _gate.WaitAsync(cancellationToken).ConfigureAwait(false); + try + { + return await action().ConfigureAwait(false); + } + finally + { + _gate.Release(); + } + } +} diff --git a/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/BulkPipeline.cs b/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/BulkPipeline.cs new file mode 100644 index 00000000..3ad42328 --- /dev/null +++ b/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/BulkPipeline.cs @@ -0,0 +1,79 @@ +namespace ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Adapters; + +/// +/// Bounded-parallel execution of a per-item operation. Used where a protocol offers no +/// bulk form of a command and the only alternative is N serial round trips — today the +/// MxAccess Gateway's supervisory advise, which exists per item only (the worker's +/// SubscribeBulk issues plain advises). Pipelining 37,500 tags at a window of 16 +/// replaces 37,500 serial round trips with ~2,344 windows. +/// +/// +/// Factored out of the gateway client so the concurrency bound is unit-testable without +/// a live MXAccess session. +/// +/// +internal static class BulkPipeline +{ + /// + /// Runs for every item with at most + /// in flight, preserving result order. A per-item + /// fault is captured into that item's result slot via + /// rather than aborting the pipeline; cancellation propagates. + /// + /// Input item type. + /// Per-item result type. + /// Items to process. + /// Maximum operations in flight; values < 1 are treated as 1. + /// The per-item operation. + /// Maps an item plus its exception onto a result row. + /// Cancellation token. + /// One result per item, in input order. + public static async Task RunAsync( + IReadOnlyList items, + int maxParallelism, + Func> operation, + Func onError, + CancellationToken ct = default) + { + var results = new TResult[items.Count]; + if (items.Count == 0) + return results; + + // Deliberately NOT disposed: on cancellation Task.WhenAll rethrows while sibling + // operations may still be unwinding, and a disposed SemaphoreSlim would turn that + // into an ObjectDisposedException. SemaphoreSlim without AvailableWaitHandle holds + // no unmanaged resource, so letting it be collected is safe. + var gate = new SemaphoreSlim(Math.Max(1, maxParallelism)); + var tasks = new Task[items.Count]; + for (var i = 0; i < items.Count; i++) + { + var index = i; + var item = items[i]; + tasks[i] = RunOneAsync(item, index); + } + + await Task.WhenAll(tasks).ConfigureAwait(false); + return results; + + async Task RunOneAsync(TItem item, int index) + { + await gate.WaitAsync(ct).ConfigureAwait(false); + try + { + results[index] = await operation(item, ct).ConfigureAwait(false); + } + catch (OperationCanceledException) + { + throw; + } + catch (Exception ex) + { + results[index] = onError(item, ex); + } + finally + { + gate.Release(); + } + } + } +} diff --git a/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/IMxGatewayClient.cs b/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/IMxGatewayClient.cs index edb2f9f9..e91f0bae 100644 --- a/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/IMxGatewayClient.cs +++ b/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/IMxGatewayClient.cs @@ -6,7 +6,11 @@ namespace ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Adapters; /// Connection parameters resolved from the flat config dict. public record MxGatewayConnectionOptions( string Endpoint, string ApiKey, string ClientName, int WriteUserId, - bool UseTls, string? CaFile, string? ServerName, int ReadTimeoutMs); + bool UseTls, string? CaFile, string? ServerName, int ReadTimeoutMs, + // Maximum in-flight supervisory advise commands on the bulk-subscribe path when + // WriteUserId == 0 (the gateway worker has no BULK supervisory advise). Sourced from + // DataConnectionOptions.MxSupervisoryAdviseParallelism. + int SupervisoryAdviseParallelism = 16); /// One advised-tag value change pushed from the gateway event stream. public record MxValueUpdate(string TagPath, object? Value, QualityCode Quality, DateTimeOffset Timestamp); @@ -17,6 +21,13 @@ public record MxReadOutcome(string TagPath, bool Success, object? Value, Quality /// Per-tag write outcome. public record MxWriteOutcome(string TagPath, bool Success, string? Error); +/// Per-tag outcome of a bulk subscribe (AddItem + Advise in one gateway command). +/// The requested tag address. +/// Whether the item was added and advised. +/// Gateway item handle (as a string) when successful. +/// Per-tag failure reason when not successful. +public record MxSubscribeOutcome(string TagPath, bool Success, string? SubscriptionId, string? Error); + /// One node in a Galaxy browse level. public record MxBrowseChild(string NodeId, string DisplayName, BrowseNodeClass NodeClass, bool HasChildren, string? DataType = null); @@ -51,6 +62,25 @@ public interface IMxGatewayClient : IAsyncDisposable /// A task that represents the asynchronous operation. Task UnsubscribeAsync(string subscriptionId, CancellationToken ct = default); + /// + /// Adds and advises MANY tags in as few gateway commands as the worker allows — + /// ONE SubscribeBulk round trip in plain-advise mode, or one + /// AddItemBulk plus bounded-parallel supervisory advises when the connection + /// has no write-user context (the worker has no bulk supervisory advise). + /// Replaces the historical 2-RPC-per-tag AddItem + Advise pair. + /// + /// Tag addresses to subscribe. + /// Cancellation token. + /// One outcome per requested tag path, in request order. + Task> SubscribeBulkAsync( + IReadOnlyList tagPaths, CancellationToken ct = default); + + /// UnAdvise + RemoveItem for many subscription ids in one gateway command. + /// Subscription ids previously returned by a subscribe call. + /// Cancellation token. + /// A task that represents the asynchronous operation. + Task UnsubscribeBulkAsync(IReadOnlyList subscriptionIds, CancellationToken ct = default); + /// Snapshot read of one or more tags (ReadBulk). /// Tag addresses to read. /// Cancellation token. diff --git a/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/IOpcUaClient.cs b/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/IOpcUaClient.cs index 8430d58a..82c1d67d 100644 --- a/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/IOpcUaClient.cs +++ b/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/IOpcUaClient.cs @@ -26,7 +26,11 @@ public record OpcUaConnectionOptions( string SubscriptionDisplayName = "ScadaBridge", string TimestampsToReturn = "Source", OpcUaDeadbandOptions? Deadband = null, - OpcUaUserIdentityOptions? UserIdentity = null); + OpcUaUserIdentityOptions? UserIdentity = null, + // Monitored items above this ceiling are sharded onto an additional + // subscription on the same session (see RealOpcUaClient). Per endpoint, because + // item-count limits are a property of the target server. + int MaxMonitoredItemsPerSubscription = 5000); public record OpcUaDeadbandOptions(string Type, double Value); @@ -37,6 +41,32 @@ public record OpcUaUserIdentityOptions( string CertificatePath, string CertificatePassword); +/// +/// Per-node outcome of a batch monitored-item create. A node that fails to resolve or +/// whose monitored item comes back with a bad create status is reported as a failed row +/// — never by aborting the whole batch (a connection-level fault still throws). +/// +/// The requested node id, verbatim. +/// Whether the monitored item was created. +/// Handle for when successful. +/// Failure reason when not successful. +public record OpcUaSubscribeOutcome(string NodeId, bool Success, string? SubscriptionHandle, string? Error); + +/// Per-node outcome of a batch read. +/// The requested node id, verbatim. +/// The value read, when the read produced one. +/// Source timestamp reported by the server. +/// OPC UA status code for this node's read. +/// Set when the node could not be read at all (e.g. an unresolvable node id). +public record OpcUaReadOutcome( + string NodeId, object? Value, DateTime SourceTimestamp, uint StatusCode, string? Error); + +/// Per-node outcome of a batch write. +/// The requested node id, verbatim. +/// OPC UA status code for this node's write (0 = Good). +/// Set when the node could not be written at all (e.g. an unresolvable node id). +public record OpcUaWriteOutcome(string NodeId, uint StatusCode, string? Error); + /// /// Abstraction over OPC UA client library for testability. /// The real implementation would wrap an OPC UA SDK (e.g., OPC Foundation .NET Standard Library). @@ -74,6 +104,22 @@ public interface IOpcUaClient : IAsyncDisposable Action onValueChanged, CancellationToken cancellationToken = default); + /// + /// Creates monitored items for MANY nodes with ONE ApplyChanges per touched + /// subscription — the batch counterpart of and + /// the reason the DCL no longer issues one ApplyChanges per tag. All nodes share the + /// single callback (its first argument is the node + /// id, so per-node delegates would carry nothing extra). + /// + /// The node ids to monitor. + /// Callback invoked with (nodeId, value, sourceTimestamp, statusCode) on each change. + /// A cancellation token that can be used to cancel the operation. + /// One outcome per requested node id, in request order. + Task> CreateSubscriptionsAsync( + IReadOnlyList nodeIds, + Action onValueChanged, + CancellationToken cancellationToken = default); + /// /// Removes a monitored item subscription by handle. /// @@ -82,6 +128,16 @@ public interface IOpcUaClient : IAsyncDisposable /// A task representing the asynchronous operation. Task RemoveSubscriptionAsync(string subscriptionHandle, CancellationToken cancellationToken = default); + /// + /// Removes many monitored items, grouping ONE ApplyChanges per touched subscription. + /// Unknown handles are ignored. + /// + /// Handles previously returned by a create call. + /// A cancellation token that can be used to cancel the operation. + /// A task representing the asynchronous operation. + Task RemoveSubscriptionsAsync( + IReadOnlyList subscriptionHandles, CancellationToken cancellationToken = default); + /// /// Subscribes to OPC UA Alarms & Conditions events under /// (or the Server object when null). On @@ -124,6 +180,26 @@ public interface IOpcUaClient : IAsyncDisposable /// A task that completes with the OPC UA status code of the write operation. Task WriteValueAsync(string nodeId, object? value, CancellationToken cancellationToken = default); + /// + /// Reads many nodes in ONE OPC UA Read service call (chunked internally against the + /// server's operation limits). + /// + /// The node ids to read. + /// A cancellation token that can be used to cancel the operation. + /// One outcome per requested node id, in request order. + Task> ReadValuesAsync( + IReadOnlyList nodeIds, CancellationToken cancellationToken = default); + + /// + /// Writes many nodes in ONE OPC UA Write service call (chunked internally against the + /// server's operation limits). + /// + /// The node id / value pairs to write. + /// A cancellation token that can be used to cancel the operation. + /// One outcome per requested node id, in request order. + Task> WriteValuesAsync( + IReadOnlyList<(string NodeId, object? Value)> values, CancellationToken cancellationToken = default); + /// /// Raised when the OPC UA session detects a keep-alive failure or the server /// becomes unreachable. The adapter layer uses this to trigger reconnection. @@ -320,12 +396,29 @@ internal class StubOpcUaClient : IOpcUaClient return Task.FromResult(Guid.NewGuid().ToString()); } + /// + public Task> CreateSubscriptionsAsync( + IReadOnlyList nodeIds, + Action onValueChanged, + CancellationToken cancellationToken = default) + { + IReadOnlyList outcomes = nodeIds + .Select(n => new OpcUaSubscribeOutcome(n, true, Guid.NewGuid().ToString(), null)) + .ToList(); + return Task.FromResult(outcomes); + } + /// public Task RemoveSubscriptionAsync(string subscriptionHandle, CancellationToken cancellationToken = default) { return Task.CompletedTask; } + /// + public Task RemoveSubscriptionsAsync( + IReadOnlyList subscriptionHandles, CancellationToken cancellationToken = default) + => Task.CompletedTask; + /// public Task CreateAlarmSubscriptionAsync( string? sourceNodeId, string? conditionFilter, @@ -352,6 +445,26 @@ internal class StubOpcUaClient : IOpcUaClient return Task.FromResult(0); // Good status } + /// + public Task> ReadValuesAsync( + IReadOnlyList nodeIds, CancellationToken cancellationToken = default) + { + IReadOnlyList outcomes = nodeIds + .Select(n => new OpcUaReadOutcome(n, null, DateTime.UtcNow, 0u, null)) + .ToList(); + return Task.FromResult(outcomes); + } + + /// + public Task> WriteValuesAsync( + IReadOnlyList<(string NodeId, object? Value)> values, CancellationToken cancellationToken = default) + { + IReadOnlyList outcomes = values + .Select(v => new OpcUaWriteOutcome(v.NodeId, 0u, null)) + .ToList(); + return Task.FromResult(outcomes); + } + /// public Task BrowseChildrenAsync( string? parentNodeId, string? continuationToken = null, CancellationToken cancellationToken = default) diff --git a/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/MonitoredItemShardPlanner.cs b/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/MonitoredItemShardPlanner.cs new file mode 100644 index 00000000..ad0e53d0 --- /dev/null +++ b/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/MonitoredItemShardPlanner.cs @@ -0,0 +1,63 @@ +namespace ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Adapters; + +/// +/// Pure placement policy for monitored items across OPC UA subscription shards: first +/// shard with free capacity, otherwise a new shard. Factored out of +/// so the budget arithmetic is testable without a live +/// server (the SDK's Session/Subscription types cannot be faked). +/// +internal static class MonitoredItemShardPlanner +{ + /// + /// Plans where new items go given the current per-shard + /// item counts. Returns one shard index per item, in order; an index equal to or above + /// .Count means "a shard that must be created first". + /// + /// Item count of each existing shard, in shard order. + /// Maximum items per shard; values <= 0 fall back to 1. + /// Number of items to place. + /// Shard index per item, in request order. + public static IReadOnlyList Plan(IReadOnlyList currentCounts, int budget, int itemCount) + { + var effectiveBudget = budget > 0 ? budget : 1; + var counts = currentCounts.ToList(); + var placement = new List(itemCount); + + for (var i = 0; i < itemCount; i++) + { + var target = -1; + for (var shard = 0; shard < counts.Count; shard++) + { + if (counts[shard] < effectiveBudget) + { + target = shard; + break; + } + } + + if (target < 0) + { + counts.Add(0); + target = counts.Count - 1; + } + + counts[target]++; + placement.Add(target); + } + + return placement; + } + + /// + /// Number of shards needed to hold items at + /// items per shard, starting from nothing. + /// + /// Total monitored items. + /// Maximum items per shard; values <= 0 fall back to 1. + /// The shard count. + public static int ShardCountFor(int itemCount, int budget) + { + var effectiveBudget = budget > 0 ? budget : 1; + return (itemCount + effectiveBudget - 1) / effectiveBudget; + } +} diff --git a/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/MxGatewayDataConnection.cs b/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/MxGatewayDataConnection.cs index 3b29c7fe..20c22f61 100644 --- a/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/MxGatewayDataConnection.cs +++ b/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/MxGatewayDataConnection.cs @@ -21,11 +21,13 @@ namespace ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Adapters; /// , the actor disposes this adapter, creates a fresh one, /// reconnects and re-subscribes all tags. /// -public class MxGatewayDataConnection : IDataConnection, IBrowsableDataConnection, IAlarmSubscribableConnection +public class MxGatewayDataConnection + : IDataConnection, IBrowsableDataConnection, IAlarmSubscribableConnection, IBatchSubscribableConnection { private readonly IMxGatewayClientFactory _clientFactory; private readonly ILogger _logger; private readonly ISecretResolver? _secretResolver; + private readonly int _supervisoryAdviseParallelism; private IMxGatewayClient? _client; private ConnectionHealth _status = ConnectionHealth.Disconnected; private CancellationTokenSource? _eventLoopCts; @@ -39,6 +41,13 @@ public class MxGatewayDataConnection : IDataConnection, IBrowsableDataConnection private int _alarmSubCount; private readonly object _alarmLock = new(); + // Source references currently subscribed (alarm subscription id → source reference), + // and the prefix the live stream was opened with. StreamAlarms carries ONE prefix, so + // the pushable union is the longest common prefix of the active sources: the stream is + // restarted only when a NEW source falls outside it. Guarded by _alarmLock. + private readonly Dictionary _alarmSubSources = new(StringComparer.Ordinal); + private string _alarmStreamPrefix = string.Empty; + // subscriptionId → (tagPath, callback) so the event loop can route updates by tag, // plus tagPath → subscriptionId for reverse lookup. Concurrent because the event // loop reads from a background thread while Subscribe/Unsubscribe mutate. @@ -58,14 +67,20 @@ public class MxGatewayDataConnection : IDataConnection, IBrowsableDataConnection /// connect time. When null, a literal ApiKey still works, but a secret: reference /// fails closed (a connection is never established with an unresolved key). /// + /// + /// Maximum in-flight supervisory advise commands on the bulk-subscribe path when the + /// endpoint has no write-user context (DataConnectionOptions.MxSupervisoryAdviseParallelism). + /// public MxGatewayDataConnection( IMxGatewayClientFactory clientFactory, ILogger logger, - ISecretResolver? secretResolver = null) + ISecretResolver? secretResolver = null, + int supervisoryAdviseParallelism = 16) { _clientFactory = clientFactory; _logger = logger; _secretResolver = secretResolver; + _supervisoryAdviseParallelism = Math.Max(1, supervisoryAdviseParallelism); } /// @@ -129,7 +144,8 @@ public class MxGatewayDataConnection : IDataConnection, IBrowsableDataConnection cfg.UseTls, string.IsNullOrWhiteSpace(cfg.CaFile) ? null : cfg.CaFile, string.IsNullOrWhiteSpace(cfg.ServerName) ? null : cfg.ServerName, - cfg.ReadTimeoutMs), cancellationToken); + cfg.ReadTimeoutMs, + _supervisoryAdviseParallelism), cancellationToken); _status = ConnectionHealth.Connected; @@ -192,6 +208,8 @@ public class MxGatewayDataConnection : IDataConnection, IBrowsableDataConnection _alarmCts?.Dispose(); _alarmCts = null; _alarmSubCount = 0; + _alarmSubSources.Clear(); + _alarmStreamPrefix = string.Empty; } if (_client is not null) await _client.DisconnectAsync(cancellationToken); @@ -215,28 +233,79 @@ public class MxGatewayDataConnection : IDataConnection, IBrowsableDataConnection await _client!.UnsubscribeAsync(subscriptionId, cancellationToken); } + /// + public async Task> SubscribeBatchAsync( + IReadOnlyList tagPaths, SubscriptionCallback callback, CancellationToken cancellationToken = default) + { + // ONE gateway round trip per batch (SubscribeBulk = AddItem + Advise per tag in a + // single worker pass) instead of the historical 2 RPCs per tag. + var outcomes = await _client!.SubscribeBulkAsync(tagPaths, cancellationToken); + var results = new List(outcomes.Count); + foreach (var outcome in outcomes) + { + if (outcome is { Success: true, SubscriptionId: not null }) + { + _subs[outcome.SubscriptionId] = (outcome.TagPath, callback); + _tagToSub[outcome.TagPath] = outcome.SubscriptionId; + } + + results.Add(new TagSubscribeResult( + outcome.TagPath, outcome.Success, outcome.SubscriptionId, outcome.Error)); + } + + return results; + } + + /// + public async Task UnsubscribeBatchAsync( + IReadOnlyList subscriptionIds, CancellationToken cancellationToken = default) + { + foreach (var id in subscriptionIds) + { + if (_subs.TryRemove(id, out var s)) + _tagToSub.TryRemove(s.TagPath, out _); + } + + await _client!.UnsubscribeBulkAsync(subscriptionIds, cancellationToken); + } + /// public Task SubscribeAlarmsAsync( string sourceReference, string? conditionFilter, AlarmTransitionCallback callback, CancellationToken cancellationToken = default) { + var subscriptionId = Guid.NewGuid().ToString(); lock (_alarmLock) { _alarmSubCount++; - if (_alarmCts == null) + _alarmSubSources[subscriptionId] = sourceReference; + + // Push the UNION of the subscribed source references to the gateway as the + // stream's single filter prefix. It is a bandwidth optimisation only — the + // MxGateway has no server-side CONDITION filter either, so conditionFilter is + // still not forwarded: the DataConnectionActor's per-source + per-condition-type + // gate stays authoritative, uniform with the OPC UA WhereClause stance. + if (_alarmCts != null && AlarmFilterPrefix.Covers(_alarmStreamPrefix, sourceReference)) + return Task.FromResult(subscriptionId); + + // A new source outside the live prefix (or the first subscriber): (re)open the + // stream on the recomputed union. The source replays a fresh snapshot, which + // NativeAlarmActor already handles. + if (_alarmCts != null) { - _alarmCts = new CancellationTokenSource(); - var token = _alarmCts.Token; - var client = _client!; - // Gateway-wide feed (null prefix). The MxGateway has no server-side - // condition filter, so conditionFilter is intentionally NOT forwarded - // here: the DataConnectionActor applies it as the authoritative - // client-side gate per source reference AND per condition type - // (AlarmConditionFilter), uniform with the OPC UA path. - _ = Task.Run(() => client.RunAlarmStreamAsync(null, t => callback(t), token), token); + _alarmCts.Cancel(); + _alarmCts.Dispose(); } + + _alarmStreamPrefix = AlarmFilterPrefix.LongestCommonPrefix(_alarmSubSources.Values); + _alarmCts = new CancellationTokenSource(); + var token = _alarmCts.Token; + var client = _client!; + var prefix = _alarmStreamPrefix.Length == 0 ? null : _alarmStreamPrefix; + _ = Task.Run(() => client.RunAlarmStreamAsync(prefix, t => callback(t), token), token); } - return Task.FromResult(Guid.NewGuid().ToString()); + + return Task.FromResult(subscriptionId); } /// @@ -244,13 +313,17 @@ public class MxGatewayDataConnection : IDataConnection, IBrowsableDataConnection { lock (_alarmLock) { + _alarmSubSources.Remove(subscriptionId); if (_alarmSubCount > 0) _alarmSubCount--; + // Deliberately no stream restart here: dropping a source can only leave the + // prefix too BROAD, which costs bandwidth, never correctness. if (_alarmSubCount == 0) { _alarmCts?.Cancel(); _alarmCts?.Dispose(); _alarmCts = null; + _alarmStreamPrefix = string.Empty; } } return Task.CompletedTask; @@ -354,6 +427,8 @@ public class MxGatewayDataConnection : IDataConnection, IBrowsableDataConnection _alarmCts?.Dispose(); _alarmCts = null; _alarmSubCount = 0; + _alarmSubSources.Clear(); + _alarmStreamPrefix = string.Empty; } if (_client is not null) await _client.DisposeAsync(); diff --git a/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/OpcUaDataConnection.cs b/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/OpcUaDataConnection.cs index 7176492c..4cabfc02 100644 --- a/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/OpcUaDataConnection.cs +++ b/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/OpcUaDataConnection.cs @@ -18,7 +18,9 @@ namespace ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Adapters; /// - Read/Write → Read/Write service calls /// - Quality → OPC UA StatusCode mapping /// -public class OpcUaDataConnection : IDataConnection, IBrowsableDataConnection, IAlarmSubscribableConnection, IAddressSpaceSearchable +public class OpcUaDataConnection + : IDataConnection, IBrowsableDataConnection, IAlarmSubscribableConnection, IAddressSpaceSearchable, + IBatchSubscribableConnection { private readonly IOpcUaClientFactory _clientFactory; private readonly ILogger _logger; @@ -93,7 +95,8 @@ public class OpcUaDataConnection : IDataConnection, IBrowsableDataConnection, IA ? new OpcUaUserIdentityOptions( ui.TokenType.ToString(), ui.Username, ui.Password, ui.CertificatePath, ui.CertificatePassword) - : null); + : null, + MaxMonitoredItemsPerSubscription: config.MaxMonitoredItemsPerSubscription); _status = ConnectionHealth.Connecting; @@ -193,6 +196,36 @@ public class OpcUaDataConnection : IDataConnection, IBrowsableDataConnection, IA cancellationToken); } + /// + public async Task> SubscribeBatchAsync( + IReadOnlyList tagPaths, SubscriptionCallback callback, CancellationToken cancellationToken = default) + { + EnsureConnected(); + + // ONE shared callback for the whole batch — the client hands back the node id + // (== tag path for OPC UA), so no per-tag closure is needed. + var outcomes = await _client!.CreateSubscriptionsAsync( + tagPaths, + (nodeId, value, timestamp, statusCode) => + { + var quality = MapStatusCode(statusCode); + callback(nodeId, new TagValue(value, quality, new DateTimeOffset(timestamp, TimeSpan.Zero))); + }, + cancellationToken); + + return outcomes + .Select(o => new TagSubscribeResult(o.NodeId, o.Success, o.SubscriptionHandle, o.Error)) + .ToList(); + } + + /// + public async Task UnsubscribeBatchAsync( + IReadOnlyList subscriptionIds, CancellationToken cancellationToken = default) + { + if (_client != null) + await _client.RemoveSubscriptionsAsync(subscriptionIds, cancellationToken); + } + /// public async Task SubscribeAlarmsAsync( string sourceReference, string? conditionFilter, @@ -249,27 +282,59 @@ public class OpcUaDataConnection : IDataConnection, IBrowsableDataConnection, IA /// public async Task> ReadBatchAsync(IEnumerable tagPaths, CancellationToken cancellationToken = default) { - // A single failing tag must not abort the whole batch. - // ReadAsync re-throws non-cancellation exceptions; catch them per tag and record - // a failed ReadResult so the caller receives a complete result map for every - // requested tag (the ReadResult shape already carries per-tag Success/error). - var results = new Dictionary(); - foreach (var tagPath in tagPaths) + EnsureConnected(); + + // TRUE bulk: one OPC UA Read service call (chunked inside the client against the + // server's operation limits), not a loop over ReadAsync. A single failing tag + // still comes back as a failed ReadResult row rather than aborting the batch; + // OperationCanceledException still aborts the whole batch. + var requested = tagPaths as IReadOnlyList ?? tagPaths.ToList(); + var results = new Dictionary(requested.Count); + if (requested.Count == 0) + return results; + + try { - try + var outcomes = await _client!.ReadValuesAsync(requested, cancellationToken); + foreach (var outcome in outcomes) { - results[tagPath] = await ReadAsync(tagPath, cancellationToken); - } - catch (OperationCanceledException) - { - // Cancellation aborts the whole batch — propagate it. - throw; - } - catch (Exception ex) - { - results[tagPath] = new ReadResult(false, null, ex.Message); + if (outcome.Error != null) + { + results[outcome.NodeId] = new ReadResult(false, null, outcome.Error); + continue; + } + + var quality = MapStatusCode(outcome.StatusCode); + results[outcome.NodeId] = quality == QualityCode.Bad + ? new ReadResult(false, null, $"OPC UA read returned bad status: 0x{outcome.StatusCode:X8}") + : new ReadResult(true, + new TagValue(outcome.Value, quality, new DateTimeOffset(outcome.SourceTimestamp, TimeSpan.Zero)), + null); } } + catch (OperationCanceledException) + { + // Cancellation aborts the whole batch — propagate it. + throw; + } + catch (Exception ex) + { + // A batch-level fault (session dropped mid-call) mirrors ReadAsync: signal the + // disconnect, then report every requested tag as failed so the caller still + // receives a complete map instead of an exception. + _logger.LogWarning(ex, "OPC UA batch read failed — connection may be lost"); + RaiseDisconnected(); + foreach (var tagPath in requested) + results[tagPath] = new ReadResult(false, null, ex.Message); + } + + // Defensive: a non-conformant client/server that omits a requested tag. + foreach (var tagPath in requested) + { + if (!results.ContainsKey(tagPath)) + results[tagPath] = new ReadResult(false, null, "Tag missing from OPC UA read result."); + } + return results; } @@ -288,29 +353,46 @@ public class OpcUaDataConnection : IDataConnection, IBrowsableDataConnection, IA /// public async Task> WriteBatchAsync(IDictionary values, CancellationToken cancellationToken = default) { - // A mid-batch fault must not abort the whole batch. - // WriteAsync calls EnsureConnected(), which throws InvalidOperationException when - // the connection drops partway through; catch per-tag exceptions and record a - // failed WriteResult so the caller (including WriteBatchAndWaitAsync) receives a - // complete result map. OperationCanceledException is still propagated so a - // cancelled batch aborts as a whole — mirrors the ReadBatchAsync fix. - var results = new Dictionary(); - foreach (var (tagPath, value) in values) + EnsureConnected(); + + // TRUE bulk: one OPC UA Write service call (chunked inside the client). A mid-batch + // fault must not abort the batch — every requested tag gets a WriteResult row — + // while OperationCanceledException still aborts as a whole. + var requested = values.ToList(); + var results = new Dictionary(requested.Count); + if (requested.Count == 0) + return results; + + try { - try + var outcomes = await _client!.WriteValuesAsync( + requested.Select(kv => (kv.Key, kv.Value)).ToList(), cancellationToken); + foreach (var outcome in outcomes) { - results[tagPath] = await WriteAsync(tagPath, value, cancellationToken); - } - catch (OperationCanceledException) - { - // Cancellation aborts the whole batch — propagate it. - throw; - } - catch (Exception ex) - { - results[tagPath] = new WriteResult(false, ex.Message); + results[outcome.NodeId] = outcome.Error != null + ? new WriteResult(false, outcome.Error) + : outcome.StatusCode != 0 + ? new WriteResult(false, $"OPC UA write failed with status: 0x{outcome.StatusCode:X8}") + : new WriteResult(true, null); } } + catch (OperationCanceledException) + { + // Cancellation aborts the whole batch — propagate it. + throw; + } + catch (Exception ex) + { + foreach (var (tagPath, _) in requested) + results[tagPath] = new WriteResult(false, ex.Message); + } + + foreach (var (tagPath, _) in requested) + { + if (!results.ContainsKey(tagPath)) + results[tagPath] = new WriteResult(false, "Tag missing from OPC UA write result."); + } + return results; } diff --git a/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/RealMxGatewayClient.cs b/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/RealMxGatewayClient.cs index 5a5d2c4e..5684fd22 100644 --- a/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/RealMxGatewayClient.cs +++ b/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/RealMxGatewayClient.cs @@ -26,6 +26,7 @@ public sealed class RealMxGatewayClient : IMxGatewayClient private int _serverHandle; private int _writeUserId; private int _readTimeoutMs; + private int _supervisoryAdviseParallelism = 16; private ulong _lastSeq; // tag ↔ MXAccess item handle, maintained across subscribe/write. @@ -65,6 +66,7 @@ public sealed class RealMxGatewayClient : IMxGatewayClient { _writeUserId = options.WriteUserId; _readTimeoutMs = options.ReadTimeoutMs; + _supervisoryAdviseParallelism = Math.Max(1, options.SupervisoryAdviseParallelism); var clientOptions = new MxGatewayClientOptions { @@ -102,6 +104,122 @@ public sealed class RealMxGatewayClient : IMxGatewayClient return handle.ToString(CultureInfo.InvariantCulture); } + /// + public async Task> SubscribeBulkAsync( + IReadOnlyList tagPaths, CancellationToken ct = default) + { + if (tagPaths.Count == 0) + return []; + + // Plain-advise mode (a configured write user): the worker's SubscribeBulk does + // AddItem + Advise for every tag in ONE gateway command, collapsing the historical + // 2 RPCs per tag to 1 RPC per chunk. + if (!UseSupervisoryAdvise) + { + var replies = await _session!.SubscribeBulkAsync(_serverHandle, tagPaths, ct).ConfigureAwait(false); + return MapSubscribeResults(tagPaths, replies); + } + + // Supervisory mode (WriteUserId == 0): the worker has no BULK supervisory advise + // (SubscribeBulk issues plain Advise), so bulk-add the items in one command and + // pipeline the per-item AdviseSupervisory commands with a bounded window instead + // of issuing them serially. Cross-repo follow-up: an additive `supervisory` flag on + // SubscribeBulkCommand in mxaccessgw would collapse this to one RPC as well. + var added = await _session!.AddItemBulkAsync(_serverHandle, tagPaths, ct).ConfigureAwait(false); + var outcomes = MapSubscribeResults(tagPaths, added); + + var advisable = outcomes + .Select((o, i) => (Outcome: o, Index: i)) + .Where(x => x.Outcome.Success) + .ToList(); + + var advised = await BulkPipeline.RunAsync( + advisable, + _supervisoryAdviseParallelism, + async (x, token) => + { + var handle = int.Parse(x.Outcome.SubscriptionId!, NumberStyles.Integer, CultureInfo.InvariantCulture); + await EnsureSupervisoryAdvisedAsync(handle, token).ConfigureAwait(false); + return x.Outcome; + }, + (x, ex) => x.Outcome with { Success = false, SubscriptionId = null, Error = ex.Message }, + ct).ConfigureAwait(false); + + var result = outcomes.ToArray(); + for (var i = 0; i < advisable.Count; i++) + result[advisable[i].Index] = advised[i]; + + // A failed advise leaves an added-but-unadvised item; drop our handle mapping so a + // later retry re-adds cleanly rather than reusing a half-subscribed handle. + foreach (var failed in result.Where(r => !r.Success)) + { + if (_tagToHandle.TryRemove(failed.TagPath, out var staleHandle)) + _handleToTag.TryRemove(staleHandle, out _); + } + + return result; + } + + /// + /// Maps the gateway's per-item bulk results back onto the requested tag paths. Results + /// are returned in request order; the tag address on the reply is preferred when the + /// gateway echoes it. Successful rows populate the tag ↔ item-handle maps that the + /// event loop and the write path key off. + /// + private MxSubscribeOutcome[] MapSubscribeResults( + IReadOnlyList tagPaths, IReadOnlyList replies) + { + var outcomes = new MxSubscribeOutcome[tagPaths.Count]; + for (var i = 0; i < tagPaths.Count; i++) + { + var tag = tagPaths[i]; + if (i >= replies.Count) + { + outcomes[i] = new MxSubscribeOutcome(tag, false, null, "Gateway returned no result for this tag."); + continue; + } + + var reply = replies[i]; + if (!reply.WasSuccessful) + { + outcomes[i] = new MxSubscribeOutcome(tag, false, null, + string.IsNullOrEmpty(reply.ErrorMessage) ? "Bulk subscribe failed." : reply.ErrorMessage); + continue; + } + + _tagToHandle[tag] = reply.ItemHandle; + _handleToTag[reply.ItemHandle] = tag; + outcomes[i] = new MxSubscribeOutcome( + tag, true, reply.ItemHandle.ToString(CultureInfo.InvariantCulture), null); + } + + return outcomes; + } + + /// + public async Task UnsubscribeBulkAsync(IReadOnlyList subscriptionIds, CancellationToken ct = default) + { + var handles = new List(subscriptionIds.Count); + foreach (var id in subscriptionIds) + { + if (int.TryParse(id, NumberStyles.Integer, CultureInfo.InvariantCulture, out var handle)) + handles.Add(handle); + } + + if (handles.Count == 0) + return; + + await _session!.UnsubscribeBulkAsync(_serverHandle, handles, ct).ConfigureAwait(false); + + foreach (var handle in handles) + { + // UnsubscribeBulk unadvises AND removes the item, releasing every advice kind. + _supervisoryAdvised.TryRemove(handle, out _); + if (_handleToTag.TryRemove(handle, out var tag)) + _tagToHandle.TryRemove(tag, out _); + } + } + /// public async Task UnsubscribeAsync(string subscriptionId, CancellationToken ct = default) { diff --git a/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/RealOpcUaClient.cs b/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/RealOpcUaClient.cs index 0fdf8db6..26dd75bf 100644 --- a/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/RealOpcUaClient.cs +++ b/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/RealOpcUaClient.cs @@ -19,7 +19,42 @@ namespace ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Adapters; public class RealOpcUaClient : IOpcUaClient { private ISession? _session; - private Subscription? _subscription; + + /// + /// One data shard = one OPC UA Subscription holding at most + /// monitored + /// items. The count is tracked here rather than read back off the SDK object so + /// placement decisions are made from state this class owns (and mutates only under + /// ). + /// + private sealed class DataShard(Subscription subscription) + { + public Subscription Subscription { get; } = subscription; + public int ItemCount { get; set; } + } + + // Data-item shards, in creation order. A monitored item is placed on the first shard + // with free capacity; when all are full a new shard is created, and a shard is + // deleted once it empties. Mutated ONLY under _applyGate. + private readonly List _dataShards = new(); + + // Alarms & Conditions event items live on their own shard: TriggerConditionRefresh + // needs a single subscription id to target, and QueueSize:1000 event items would + // otherwise consume data-shard capacity. Created lazily, torn down on disconnect. + private Subscription? _eventShard; + + // subscription handle → owning shard's Subscription, so a removal can group ONE + // ApplyChanges per touched shard. + private readonly ConcurrentDictionary _itemShard = new(); + + // The OPC Foundation Subscription object is not safe under concurrent structural + // mutation, and nothing else orders HandleSubscribe's background task, the + // tag-resolution probe and an alarm subscribe against each other. Every + // AddItem/RemoveItem/ApplyChanges/Create/Delete — and every shard-list mutation — + // serializes behind this ONE per-client gate. Batching makes contention rare; a + // single gate (rather than per-shard) also serializes shard creation and is the + // simplest thing that is correct. + private readonly AsyncSerialGate _applyGate = new(); // These maps are read from the OPC Foundation SDK's // internal publish threads (the MonitoredItem.Notification handler reads @@ -146,20 +181,68 @@ public class RealOpcUaClient : IOpcUaClient // Store options for monitored item creation _options = opts; - // Create a default subscription for all monitored items - _subscription = new Subscription(_session.DefaultSubscription) + // Create the first data shard up front so a session is immediately usable; + // further shards are added on demand as the item budget fills. + await _applyGate.RunAsync(async () => { - DisplayName = opts.SubscriptionDisplayName, - Priority = opts.SubscriptionPriority, + _dataShards.Clear(); + _itemShard.Clear(); + _eventShard = null; + await CreateDataShardAsync(cancellationToken); + }, cancellationToken); + } + + /// Item budget per subscription, guarding against a non-positive configured value. + private int ItemsPerShard => _options.MaxMonitoredItemsPerSubscription > 0 + ? _options.MaxMonitoredItemsPerSubscription + : 5000; + + /// + /// Builds a Subscription from the current connection options. Callers must hold + /// . + /// + private Subscription NewSubscription(string displayName) => + new(_session!.DefaultSubscription) + { + DisplayName = displayName, + Priority = _options.SubscriptionPriority, PublishingEnabled = true, - PublishingInterval = opts.PublishingIntervalMs, - KeepAliveCount = (uint)opts.KeepAliveCount, - LifetimeCount = (uint)opts.LifetimeCount, - MaxNotificationsPerPublish = (uint)opts.MaxNotificationsPerPublish + PublishingInterval = _options.PublishingIntervalMs, + KeepAliveCount = (uint)_options.KeepAliveCount, + LifetimeCount = (uint)_options.LifetimeCount, + MaxNotificationsPerPublish = (uint)_options.MaxNotificationsPerPublish }; - _session.AddSubscription(_subscription); - await _subscription.CreateAsync(cancellationToken); + /// + /// Adds and creates a new data shard. Callers must hold . + /// + private async Task CreateDataShardAsync(CancellationToken cancellationToken) + { + var suffix = _dataShards.Count == 0 ? string.Empty : $"-{_dataShards.Count + 1}"; + var subscription = NewSubscription(_options.SubscriptionDisplayName + suffix); + _session!.AddSubscription(subscription); + await subscription.CreateAsync(cancellationToken); + var shard = new DataShard(subscription); + _dataShards.Add(shard); + _logger.LogDebug("OPC UA data shard {Shard} created (budget {Budget} items)", + subscription.DisplayName, ItemsPerShard); + return shard; + } + + /// + /// Returns the dedicated Alarms & Conditions event shard, creating it on first + /// use. Callers must hold . + /// + private async Task GetOrCreateEventShardAsync(CancellationToken cancellationToken) + { + if (_eventShard != null) + return _eventShard; + + var subscription = NewSubscription(_options.SubscriptionDisplayName + "-events"); + _session!.AddSubscription(subscription); + await subscription.CreateAsync(cancellationToken); + _eventShard = subscription; + return subscription; } /// @@ -439,11 +522,19 @@ public class RealOpcUaClient : IOpcUaClient /// public async Task DisconnectAsync(CancellationToken cancellationToken = default) { - if (_subscription != null) + await _applyGate.RunAsync(async () => { - await _subscription.DeleteAsync(true); - _subscription = null; - } + foreach (var shard in _dataShards) + await shard.Subscription.DeleteAsync(true); + _dataShards.Clear(); + _itemShard.Clear(); + if (_eventShard != null) + { + await _eventShard.DeleteAsync(true); + _eventShard = null; + } + }, cancellationToken); + if (_session != null) { _session.KeepAlive -= OnSessionKeepAlive; @@ -459,55 +550,230 @@ public class RealOpcUaClient : IOpcUaClient string nodeId, Action onValueChanged, CancellationToken cancellationToken = default) { - if (_subscription == null || _session == null) - throw new InvalidOperationException("Not connected."); + // Batch-of-one delegation: the single-node entry point is kept (heartbeat monitor, + // interactive paths) but shares the batch implementation so there is exactly one + // placement / ApplyChanges code path. A per-item failure throws here — preserving + // the caller's historical "create throws on failure" contract AND the exception + // TYPE, so DataConnectionActor.IsConnectionLevelFailure still classifies a + // resolution failure as a tag-resolution failure rather than a connection fault. + var (outcomes, errors) = await CreateSubscriptionsCoreAsync( + [nodeId], onValueChanged, cancellationToken); - var handle = Guid.NewGuid().ToString(); - var monitoredItem = new MonitoredItem(_subscription.DefaultItem) - { - DisplayName = nodeId, - StartNodeId = OpcUaNodeReference.Resolve(nodeId, _session.NamespaceUris), - AttributeId = Attributes.Value, - SamplingInterval = _options.SamplingIntervalMs, - QueueSize = (uint)_options.QueueSize, - DiscardOldest = _options.DiscardOldest, - Filter = BuildDataChangeFilter(_options.Deadband) - }; + var outcome = outcomes[0]; + if (outcome.Success) + return outcome.SubscriptionHandle!; - _callbacks[handle] = onValueChanged; - - monitoredItem.Notification += (item, e) => - { - if (e.NotificationValue is MonitoredItemNotification notification) - { - var value = notification.Value?.Value; - var timestamp = notification.Value?.SourceTimestamp ?? DateTime.UtcNow; - var statusCode = notification.Value?.StatusCode.Code ?? 0; - - if (_callbacks.TryGetValue(handle, out var cb)) - { - cb(nodeId, value, timestamp, statusCode); - } - } - }; - - _subscription.AddItem(monitoredItem); - await _subscription.ApplyChangesAsync(cancellationToken); - - _monitoredItems[handle] = monitoredItem; - return handle; + if (errors[0] is { } captured) + System.Runtime.ExceptionServices.ExceptionDispatchInfo.Capture(captured).Throw(); + throw new ServiceResultException(StatusCodes.BadNodeIdUnknown, + outcome.Error ?? $"Monitored item for '{nodeId}' could not be created."); } /// - public async Task RemoveSubscriptionAsync(string subscriptionHandle, CancellationToken cancellationToken = default) + public async Task> CreateSubscriptionsAsync( + IReadOnlyList nodeIds, + Action onValueChanged, + CancellationToken cancellationToken = default) { - if (_subscription != null && _monitoredItems.TryGetValue(subscriptionHandle, out var item)) + var (outcomes, _) = await CreateSubscriptionsCoreAsync(nodeIds, onValueChanged, cancellationToken); + return outcomes; + } + + /// + /// Creates monitored items for every requested node with ONE ApplyChanges per touched + /// shard — the core of the batch subscribe seam. Node-id resolution runs outside the + /// apply lock (it is pure over the session's namespace table); placement, AddItem and + /// ApplyChanges run under it. Returns per-node outcomes plus the captured exception + /// (when any) per node, so the single-node entry point can rethrow the original. + /// + private async Task<(OpcUaSubscribeOutcome[] Outcomes, Exception?[] Errors)> CreateSubscriptionsCoreAsync( + IReadOnlyList nodeIds, + Action onValueChanged, + CancellationToken cancellationToken) + { + var session = _session; + if (session == null) + throw new InvalidOperationException("Not connected."); + + var outcomes = new OpcUaSubscribeOutcome[nodeIds.Count]; + var errors = new Exception?[nodeIds.Count]; + + // Resolve first: a bad node id is a per-tag fault and must not abort the batch. + var resolved = new List<(int Index, string NodeId, NodeId Resolved)>(nodeIds.Count); + for (var i = 0; i < nodeIds.Count; i++) { - _subscription.RemoveItem(item); - await _subscription.ApplyChangesAsync(cancellationToken); - _monitoredItems.TryRemove(subscriptionHandle, out _); - _callbacks.TryRemove(subscriptionHandle, out _); + try + { + resolved.Add((i, nodeIds[i], OpcUaNodeReference.Resolve(nodeIds[i], session.NamespaceUris))); + } + catch (Exception ex) + { + outcomes[i] = new OpcUaSubscribeOutcome(nodeIds[i], false, null, ex.Message); + errors[i] = ex; + } } + + if (resolved.Count == 0) + return (outcomes, errors); + + return await _applyGate.RunAsync(async () => + { + var created = new List<(int Index, string NodeId, string Handle, MonitoredItem Item)>(resolved.Count); + var touched = new List(); + + foreach (var (index, nodeId, startNode) in resolved) + { + var shard = await PlaceOnDataShardAsync(cancellationToken); + var handle = Guid.NewGuid().ToString(); + var monitoredItem = new MonitoredItem(shard.Subscription.DefaultItem) + { + DisplayName = nodeId, + StartNodeId = startNode, + AttributeId = Attributes.Value, + SamplingInterval = _options.SamplingIntervalMs, + QueueSize = (uint)_options.QueueSize, + DiscardOldest = _options.DiscardOldest, + Filter = BuildDataChangeFilter(_options.Deadband) + }; + + _callbacks[handle] = onValueChanged; + monitoredItem.Notification += (_, e) => + { + if (e.NotificationValue is MonitoredItemNotification notification) + { + var value = notification.Value?.Value; + var timestamp = notification.Value?.SourceTimestamp ?? DateTime.UtcNow; + var statusCode = notification.Value?.StatusCode.Code ?? 0; + + if (_callbacks.TryGetValue(handle, out var cb)) + { + cb(nodeId, value, timestamp, statusCode); + } + } + }; + + shard.Subscription.AddItem(monitoredItem); + shard.ItemCount++; + _itemShard[handle] = shard.Subscription; + created.Add((index, nodeId, handle, monitoredItem)); + if (!touched.Contains(shard)) + touched.Add(shard); + } + + // ONE ApplyChanges per touched shard — the whole point of the batch seam. + foreach (var shard in touched) + await shard.Subscription.ApplyChangesAsync(cancellationToken); + + foreach (var (index, nodeId, handle, item) in created) + { + var error = item.Status?.Error; + if (error != null && ServiceResult.IsBad(error)) + { + // Per-item create failure (unknown node, access denied): release the + // half-created item so the shard budget stays honest, and report a + // failed row. The actor treats it as an unresolved tag and retries it + // on the backoff timer. + ReleaseFailedItem(handle, item); + outcomes[index] = new OpcUaSubscribeOutcome(nodeId, false, null, error.ToString()); + } + else + { + _monitoredItems[handle] = item; + outcomes[index] = new OpcUaSubscribeOutcome(nodeId, true, handle, null); + } + } + + return (outcomes, errors); + }, cancellationToken); + } + + /// + /// Returns the shard this item belongs on per + /// — first shard with free capacity, else a + /// freshly created one. Callers must hold . + /// + private async Task PlaceOnDataShardAsync(CancellationToken cancellationToken) + { + var index = MonitoredItemShardPlanner.Plan( + _dataShards.Select(s => s.ItemCount).ToList(), ItemsPerShard, 1)[0]; + return index < _dataShards.Count + ? _dataShards[index] + : await CreateDataShardAsync(cancellationToken); + } + + /// + /// Drops a monitored item that failed to create from the shard it was placed on. + /// Callers must hold . No ApplyChanges is issued: the server + /// never created the item, so removing it locally keeps the budget honest. + /// + private void ReleaseFailedItem(string handle, MonitoredItem item) + { + _callbacks.TryRemove(handle, out _); + if (!_itemShard.TryRemove(handle, out var subscription)) + return; + + subscription.RemoveItem(item); + var shard = _dataShards.FirstOrDefault(s => ReferenceEquals(s.Subscription, subscription)); + if (shard is { ItemCount: > 0 }) + shard.ItemCount--; + } + + /// + public Task RemoveSubscriptionAsync(string subscriptionHandle, CancellationToken cancellationToken = default) + => RemoveSubscriptionsAsync([subscriptionHandle], cancellationToken); + + /// + public async Task RemoveSubscriptionsAsync( + IReadOnlyList subscriptionHandles, CancellationToken cancellationToken = default) + { + if (subscriptionHandles.Count == 0) + return; + + await _applyGate.RunAsync(async () => + { + var touched = new List(); + foreach (var handle in subscriptionHandles) + { + if (!_monitoredItems.TryGetValue(handle, out var item)) + continue; + if (!_itemShard.TryGetValue(handle, out var subscription)) + continue; + + subscription.RemoveItem(item); + _monitoredItems.TryRemove(handle, out _); + _callbacks.TryRemove(handle, out _); + _itemShard.TryRemove(handle, out _); + + var shard = _dataShards.FirstOrDefault(s => ReferenceEquals(s.Subscription, subscription)); + if (shard == null) + continue; + if (shard.ItemCount > 0) + shard.ItemCount--; + if (!touched.Contains(shard)) + touched.Add(shard); + } + + // ONE ApplyChanges per touched shard, then drop the shards that emptied. + foreach (var shard in touched) + { + await shard.Subscription.ApplyChangesAsync(cancellationToken); + if (shard.ItemCount != 0) + continue; + + _dataShards.Remove(shard); + try + { + await shard.Subscription.DeleteAsync(true); + } + catch (Exception ex) + { + // Best-effort: an empty shard the server already dropped must not fail + // the unsubscribe path. + _logger.LogDebug(ex, "Deleting emptied OPC UA data shard {Shard} failed (ignored).", + shard.Subscription.DisplayName); + } + } + }, cancellationToken); } // ── Native alarm (Alarms & Conditions) subscription ── @@ -523,7 +789,7 @@ public class RealOpcUaClient : IOpcUaClient string? sourceNodeId, string? conditionFilter, Action onTransition, CancellationToken cancellationToken = default) { - if (_subscription == null || _session == null) + if (_session == null) throw new InvalidOperationException("Not connected."); var handle = Guid.NewGuid().ToString(); @@ -533,7 +799,14 @@ public class RealOpcUaClient : IOpcUaClient var startNode = string.IsNullOrEmpty(sourceNodeId) ? ObjectIds.Server : OpcUaNodeReference.Resolve(sourceNodeId, _session.NamespaceUris); - var item = new MonitoredItem(_subscription.DefaultItem) + + // A&C event items live on their own shard (see _eventShard): ConditionRefresh + // needs one subscription id to target, and QueueSize:1000 event items must not + // occupy data-shard capacity. + var eventShard = await _applyGate.RunAsync( + () => GetOrCreateEventShardAsync(cancellationToken), cancellationToken); + + var item = new MonitoredItem(eventShard.DefaultItem) { DisplayName = $"alarm:{sourceNodeId ?? "Server"}", StartNodeId = startNode, @@ -554,22 +827,31 @@ public class RealOpcUaClient : IOpcUaClient HandleAlarmEvent(handle, sourceNodeId, efl, onTransition); }; - _subscription.AddItem(item); - await _subscription.ApplyChangesAsync(cancellationToken); + await _applyGate.RunAsync(async () => + { + eventShard.AddItem(item); + await eventShard.ApplyChangesAsync(cancellationToken); + }, cancellationToken); _alarmItems[handle] = item; // Replay currently-active conditions as a Snapshot…SnapshotComplete sequence. - await TriggerConditionRefreshAsync(handle, cancellationToken); + await TriggerConditionRefreshAsync(handle, eventShard, cancellationToken); return handle; } /// public async Task RemoveAlarmSubscriptionAsync(string subscriptionHandle, CancellationToken cancellationToken = default) { - if (_subscription != null && _alarmItems.TryRemove(subscriptionHandle, out var item)) + if (_alarmItems.TryRemove(subscriptionHandle, out var item)) { - _subscription.RemoveItem(item); - await _subscription.ApplyChangesAsync(cancellationToken); + await _applyGate.RunAsync(async () => + { + if (_eventShard != null) + { + _eventShard.RemoveItem(item); + await _eventShard.ApplyChangesAsync(cancellationToken); + } + }, cancellationToken); } _alarmInRefresh.TryRemove(subscriptionHandle, out _); _alarmLastState.TryRemove(subscriptionHandle, out _); @@ -775,15 +1057,17 @@ public class RealOpcUaClient : IOpcUaClient }; } - private async Task TriggerConditionRefreshAsync(string handle, CancellationToken cancellationToken) + private async Task TriggerConditionRefreshAsync( + string handle, Subscription eventShard, CancellationToken cancellationToken) { try { // ConditionRefresh replays active conditions; RefreshStart/End events - // bracket the replay so HandleAlarmEvent can mark them Snapshot. + // bracket the replay so HandleAlarmEvent can mark them Snapshot. It targets + // the EVENT shard's subscription id — the shard every A&C item is pinned to. await _session!.CallAsync( ObjectTypeIds.ConditionType, MethodIds.ConditionType_ConditionRefresh, - cancellationToken, _subscription!.Id); + cancellationToken, eventShard.Id); } catch (Exception ex) { @@ -906,42 +1190,143 @@ public class RealOpcUaClient : IOpcUaClient new Commons.Types.Alarms.AlarmConditionState(false, true, null, AlarmShelveState.Unshelved, false, 0), "", "", "", "", "", null, DateTimeOffset.UtcNow, "", ""); + /// + /// Nodes per OPC UA Read/Write service call. Stays under the + /// MaxNodesPerRead / MaxNodesPerWrite operation limits typical servers + /// advertise, so a 37,500-tag re-seed never builds one oversized request. + /// + private const int MaxNodesPerServiceCall = 1000; + /// public async Task<(object? Value, DateTime SourceTimestamp, uint StatusCode)> ReadValueAsync( string nodeId, CancellationToken cancellationToken = default) { - if (_session == null) throw new InvalidOperationException("Not connected."); + // Batch-of-one delegation; a resolution failure still surfaces as the thrown + // exception the single-node callers expect. + var outcomes = await ReadValuesAsync([nodeId], cancellationToken); + var outcome = outcomes[0]; + if (outcome.Error != null) + throw new ServiceResultException(StatusCodes.BadNodeIdUnknown, outcome.Error); + return (outcome.Value, outcome.SourceTimestamp, outcome.StatusCode); + } - var readValue = new ReadValueId + /// + public async Task> ReadValuesAsync( + IReadOnlyList nodeIds, CancellationToken cancellationToken = default) + { + var session = _session; + if (session == null) throw new InvalidOperationException("Not connected."); + + var outcomes = new OpcUaReadOutcome[nodeIds.Count]; + + // Resolve first — an unresolvable node id is a per-node fault, not a batch abort. + var resolvable = new List<(int Index, string NodeId, ReadValueId Read)>(nodeIds.Count); + for (var i = 0; i < nodeIds.Count; i++) { - NodeId = OpcUaNodeReference.Resolve(nodeId, _session.NamespaceUris), - AttributeId = Attributes.Value - }; + try + { + resolvable.Add((i, nodeIds[i], new ReadValueId + { + NodeId = OpcUaNodeReference.Resolve(nodeIds[i], session.NamespaceUris), + AttributeId = Attributes.Value + })); + } + catch (Exception ex) + { + outcomes[i] = new OpcUaReadOutcome(nodeIds[i], null, DateTime.UtcNow, StatusCodes.BadNodeIdUnknown, ex.Message); + } + } - var response = await _session.ReadAsync( - null, 0, MapTimestampsToReturn(_options.TimestampsToReturn), - new ReadValueIdCollection { readValue }, cancellationToken); + for (var offset = 0; offset < resolvable.Count; offset += MaxNodesPerServiceCall) + { + var chunk = resolvable.GetRange(offset, Math.Min(MaxNodesPerServiceCall, resolvable.Count - offset)); + var collection = new ReadValueIdCollection(chunk.Select(c => c.Read)); + var response = await session.ReadAsync( + null, 0, MapTimestampsToReturn(_options.TimestampsToReturn), collection, cancellationToken); - var result = response.Results[0]; - return (result.Value, result.SourceTimestamp, result.StatusCode.Code); + for (var i = 0; i < chunk.Count; i++) + { + var (index, nodeId, _) = chunk[i]; + if (response.Results is { Count: > 0 } results && i < results.Count) + { + var result = results[i]; + outcomes[index] = new OpcUaReadOutcome( + nodeId, result.Value, result.SourceTimestamp, result.StatusCode.Code, null); + } + else + { + // Non-conformant server: fewer results than requested nodes. + outcomes[index] = new OpcUaReadOutcome( + nodeId, null, DateTime.UtcNow, StatusCodes.BadUnexpectedError, + "OPC UA read returned no result for this node."); + } + } + } + + return outcomes; } /// public async Task WriteValueAsync(string nodeId, object? value, CancellationToken cancellationToken = default) { - if (_session == null) throw new InvalidOperationException("Not connected."); + // Batch-of-one delegation, mirroring ReadValueAsync. + var outcomes = await WriteValuesAsync([(nodeId, value)], cancellationToken); + var outcome = outcomes[0]; + if (outcome.Error != null) + throw new ServiceResultException(StatusCodes.BadNodeIdUnknown, outcome.Error); + return outcome.StatusCode; + } - var writeValue = new WriteValue + /// + public async Task> WriteValuesAsync( + IReadOnlyList<(string NodeId, object? Value)> values, CancellationToken cancellationToken = default) + { + var session = _session; + if (session == null) throw new InvalidOperationException("Not connected."); + + var outcomes = new OpcUaWriteOutcome[values.Count]; + + var resolvable = new List<(int Index, string NodeId, WriteValue Write)>(values.Count); + for (var i = 0; i < values.Count; i++) { - NodeId = OpcUaNodeReference.Resolve(nodeId, _session.NamespaceUris), - AttributeId = Attributes.Value, - Value = new DataValue(new Variant(value)) - }; + try + { + resolvable.Add((i, values[i].NodeId, new WriteValue + { + NodeId = OpcUaNodeReference.Resolve(values[i].NodeId, session.NamespaceUris), + AttributeId = Attributes.Value, + Value = new DataValue(new Variant(values[i].Value)) + })); + } + catch (Exception ex) + { + outcomes[i] = new OpcUaWriteOutcome(values[i].NodeId, StatusCodes.BadNodeIdUnknown, ex.Message); + } + } - var response = await _session.WriteAsync( - null, new WriteValueCollection { writeValue }, cancellationToken); + for (var offset = 0; offset < resolvable.Count; offset += MaxNodesPerServiceCall) + { + var chunk = resolvable.GetRange(offset, Math.Min(MaxNodesPerServiceCall, resolvable.Count - offset)); + var collection = new WriteValueCollection(chunk.Select(c => c.Write)); + var response = await session.WriteAsync(null, collection, cancellationToken); - return response.Results[0].Code; + for (var i = 0; i < chunk.Count; i++) + { + var (index, nodeId, _) = chunk[i]; + if (response.Results is { Count: > 0 } results && i < results.Count) + { + outcomes[index] = new OpcUaWriteOutcome(nodeId, results[i].Code, null); + } + else + { + outcomes[index] = new OpcUaWriteOutcome( + nodeId, StatusCodes.BadUnexpectedError, + "OPC UA write returned no result for this node."); + } + } + } + + return outcomes; } /// diff --git a/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/DataConnectionFactory.cs b/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/DataConnectionFactory.cs index 81b1f53f..759233d2 100644 --- a/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/DataConnectionFactory.cs +++ b/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/DataConnectionFactory.cs @@ -37,11 +37,16 @@ public class DataConnectionFactory : IDataConnectionFactory public DataConnectionFactory( ILoggerFactory loggerFactory, IOptions opcUaGlobalOptions, - ISecretResolver? secretResolver = null) + ISecretResolver? secretResolver = null, + IOptions? dataConnectionOptions = null) { _loggerFactory = loggerFactory; _secretResolver = secretResolver; var globalOptions = opcUaGlobalOptions.Value; + // Only the MxGateway supervisory-advise window is adapter-level today; the rest of + // DataConnectionOptions is consumed by the actor, which is constructed elsewhere. + var supervisoryAdviseParallelism = + (dataConnectionOptions?.Value ?? new DataConnectionOptions()).MxSupervisoryAdviseParallelism; // Register built-in protocols. // Pass the ILoggerFactory into RealOpcUaClientFactory so @@ -57,7 +62,8 @@ public class DataConnectionFactory : IDataConnectionFactory RegisterAdapter("MxGateway", details => new MxGatewayDataConnection( new RealMxGatewayClientFactory(_loggerFactory), _loggerFactory.CreateLogger(), - _secretResolver)); + _secretResolver, + supervisoryAdviseParallelism)); } /// diff --git a/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/DataConnectionOptions.cs b/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/DataConnectionOptions.cs index aefe2f67..bbc6bec2 100644 --- a/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/DataConnectionOptions.cs +++ b/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/DataConnectionOptions.cs @@ -8,9 +8,70 @@ public class DataConnectionOptions /// Fixed interval between reconnect attempts after disconnect. public TimeSpan ReconnectInterval { get; set; } = TimeSpan.FromSeconds(5); - /// Interval for retrying failed tag path resolution. + /// + /// Floor interval for retrying failed tag path resolution. The retry backs off + /// exponentially (doubling per fully-failed round) up to + /// , and resets to this floor as soon as + /// any tag resolves or the connection reconnects. + /// public TimeSpan TagResolutionRetryInterval { get; set; } = TimeSpan.FromSeconds(10); + /// + /// Ceiling for the exponential tag-resolution retry backoff. A dead device with + /// thousands of unresolved tags would otherwise probe forever at full width every + /// . + /// + public TimeSpan TagResolutionRetryMaxInterval { get; set; } = TimeSpan.FromMinutes(5); + + /// + /// Number of tags per adapter subscribe round trip on the batch subscribe path + /// (initial subscribe, reconnect re-subscribe, tag-resolution probes). + /// + public int SubscribeBatchSize { get; set; } = 500; + + /// + /// Delay inserted BETWEEN reconnect re-subscribe chunks so a site-wide reconnect + /// paces the device instead of issuing one task per tag in a tight loop. Not applied + /// on the instance-driven subscribe path — the Deployment Manager already staggers + /// instance startup there, and double-staggering would only slow failover. + /// + public TimeSpan SubscribeBatchDelay { get; set; } = TimeSpan.FromMilliseconds(50); + + /// + /// Number of tags per seed-read chunk. Moderate on purpose: chunking plus the + /// per-chunk answers the "some gateways time out on a + /// large batch" caveat that originally motivated per-tag seed reads. + /// + public int SeedReadBatchSize { get; set; } = 250; + + /// Maximum seed-read chunks in flight concurrently. + public int SeedReadMaxParallelism { get; set; } = 4; + + /// + /// Wall-clock deadline for the WHOLE seed (all chunks and all retry rounds). On + /// expiry the remaining tags are logged and stay Uncertain until a change + /// notification arrives — the same outcome as exhausting + /// . + /// + public TimeSpan SeedOverallTimeout { get; set; } = TimeSpan.FromSeconds(120); + + /// + /// Coalescing window for pushing tag-quality counters to the health collector. + /// Counter math stays per message; only the collector push is deferred, and only + /// when a genuine quality transition occurred. Health reports poll at 30s, so a + /// short window loses nothing. Disconnect / unsubscribe / reconnect-reset flushes + /// stay synchronous. + /// + public TimeSpan QualityFlushInterval { get; set; } = TimeSpan.FromSeconds(1); + + /// + /// Maximum in-flight supervisory advise commands when the MxGateway adapter has no + /// write-user context. The gateway worker has no BULK supervisory advise, so the + /// client pipelines per-item advises after one bulk AddItem instead of issuing them + /// serially. + /// + public int MxSupervisoryAdviseParallelism { get; set; } = 16; + /// Timeout for synchronous write operations to devices. public TimeSpan WriteTimeout { get; set; } = TimeSpan.FromSeconds(30); diff --git a/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/DataConnectionOptionsValidator.cs b/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/DataConnectionOptionsValidator.cs index e80dce81..c74e4ed8 100644 --- a/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/DataConnectionOptionsValidator.cs +++ b/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/DataConnectionOptionsValidator.cs @@ -35,5 +35,30 @@ public sealed class DataConnectionOptionsValidator : OptionsValidatorBase 0, $"ScadaBridge:DataConnection:SeedReadMaxAttempts must be positive (was {options.SeedReadMaxAttempts})."); + + builder.RequireThat(options.TagResolutionRetryMaxInterval >= options.TagResolutionRetryInterval, + "ScadaBridge:DataConnection:TagResolutionRetryMaxInterval must be at least " + + $"TagResolutionRetryInterval (was {options.TagResolutionRetryMaxInterval} vs {options.TagResolutionRetryInterval})."); + + builder.RequireThat(options.SubscribeBatchSize > 0, + $"ScadaBridge:DataConnection:SubscribeBatchSize must be positive (was {options.SubscribeBatchSize})."); + + builder.RequireThat(options.SubscribeBatchDelay >= TimeSpan.Zero, + $"ScadaBridge:DataConnection:SubscribeBatchDelay must not be negative (was {options.SubscribeBatchDelay})."); + + builder.RequireThat(options.SeedReadBatchSize > 0, + $"ScadaBridge:DataConnection:SeedReadBatchSize must be positive (was {options.SeedReadBatchSize})."); + + builder.RequireThat(options.SeedReadMaxParallelism > 0, + $"ScadaBridge:DataConnection:SeedReadMaxParallelism must be positive (was {options.SeedReadMaxParallelism})."); + + builder.RequireThat(options.SeedOverallTimeout > TimeSpan.Zero, + $"ScadaBridge:DataConnection:SeedOverallTimeout must be a positive duration (was {options.SeedOverallTimeout})."); + + builder.RequireThat(options.QualityFlushInterval > TimeSpan.Zero, + $"ScadaBridge:DataConnection:QualityFlushInterval must be a positive duration (was {options.QualityFlushInterval})."); + + builder.RequireThat(options.MxSupervisoryAdviseParallelism > 0, + $"ScadaBridge:DataConnection:MxSupervisoryAdviseParallelism must be positive (was {options.MxSupervisoryAdviseParallelism})."); } } diff --git a/src/ZB.MOM.WW.ScadaBridge.DeploymentManager/DeploymentManagerOptions.cs b/src/ZB.MOM.WW.ScadaBridge.DeploymentManager/DeploymentManagerOptions.cs index 7fc4dabf..46f905c7 100644 --- a/src/ZB.MOM.WW.ScadaBridge.DeploymentManager/DeploymentManagerOptions.cs +++ b/src/ZB.MOM.WW.ScadaBridge.DeploymentManager/DeploymentManagerOptions.cs @@ -17,4 +17,43 @@ public class DeploymentManagerOptions /// Timeout for acquiring an operation lock on an instance. public TimeSpan OperationLockTimeout { get; set; } = TimeSpan.FromSeconds(5); + + /// + /// Maximum number of instance deployments whose SITE ROUND-TRIP runs + /// concurrently during a bulk DeploySiteAsync. The flatten/validate and + /// persistence phases stay serial regardless (they share one non-thread-safe + /// DbContext); only the network wait is fanned out. + /// + /// + /// Bounded rather than unbounded so a large site cannot open one in-flight + /// gRPC command per instance against a two-node site pair. Four mirrors the + /// conservative end of the site's own apply concurrency; raise it only with + /// evidence from the target-scale load test. + /// + /// + public int SiteDeploymentMaxParallelism { get; set; } = 4; + + /// + /// Per-instance deadline applied to the site round-trip inside a bulk + /// DeploySiteAsync. One unreachable or wedged instance is recorded as a + /// failed row and the rest of the batch continues — the bulk operation never + /// inherits a single instance's hang. + /// + public TimeSpan SiteDeploymentTimeoutPerInstance { get; set; } = TimeSpan.FromSeconds(120); + + /// + /// Retention window for TERMINAL deployment records (Success, + /// Failed). Deployments are insert-only — one row per attempt, forever — + /// so without a window the table grows without bound and every unfiltered + /// deployment query gets slower with the age of the system. + /// + /// + /// Deliberately generous (one year) because deployment history is + /// operator-facing forensics, and deliberately terminal-only: an + /// InProgress or Pending row is never purged regardless of age, + /// because it is exactly the row the query-before-redeploy reconciliation + /// path needs to find. Set to to disable purging. + /// + /// + public TimeSpan TerminalDeploymentRecordRetention { get; set; } = TimeSpan.FromDays(365); } diff --git a/src/ZB.MOM.WW.ScadaBridge.DeploymentManager/DeploymentManagerOptionsValidator.cs b/src/ZB.MOM.WW.ScadaBridge.DeploymentManager/DeploymentManagerOptionsValidator.cs index d041088e..a4431212 100644 --- a/src/ZB.MOM.WW.ScadaBridge.DeploymentManager/DeploymentManagerOptionsValidator.cs +++ b/src/ZB.MOM.WW.ScadaBridge.DeploymentManager/DeploymentManagerOptionsValidator.cs @@ -27,5 +27,20 @@ public sealed class DeploymentManagerOptionsValidator : OptionsValidatorBase TimeSpan.Zero, $"ScadaBridge:DeploymentManager:OperationLockTimeout must be a positive duration " + $"(was {options.OperationLockTimeout}); it bounds acquiring an instance operation lock."); + + builder.RequireThat(options.SiteDeploymentMaxParallelism > 0, + $"ScadaBridge:DeploymentManager:SiteDeploymentMaxParallelism must be at least 1 " + + $"(was {options.SiteDeploymentMaxParallelism}); it is the SemaphoreSlim bound on a bulk " + + $"site deployment's site round-trips, and a non-positive value would deadlock the fan-out."); + + builder.RequireThat(options.SiteDeploymentTimeoutPerInstance > TimeSpan.Zero, + $"ScadaBridge:DeploymentManager:SiteDeploymentTimeoutPerInstance must be a positive duration " + + $"(was {options.SiteDeploymentTimeoutPerInstance}); it bounds each instance's site round-trip " + + $"inside a bulk site deployment."); + + builder.RequireThat(options.TerminalDeploymentRecordRetention >= TimeSpan.Zero, + $"ScadaBridge:DeploymentManager:TerminalDeploymentRecordRetention must not be negative " + + $"(was {options.TerminalDeploymentRecordRetention}); use TimeSpan.Zero to disable purging of " + + $"terminal deployment records."); } } diff --git a/src/ZB.MOM.WW.ScadaBridge.DeploymentManager/DeploymentService.cs b/src/ZB.MOM.WW.ScadaBridge.DeploymentManager/DeploymentService.cs index 321d6fe6..f88a6111 100644 --- a/src/ZB.MOM.WW.ScadaBridge.DeploymentManager/DeploymentService.cs +++ b/src/ZB.MOM.WW.ScadaBridge.DeploymentManager/DeploymentService.cs @@ -57,6 +57,24 @@ public class DeploymentService /// private const string TimeoutFailurePrefix = "Communication failure:"; + /// Rows deleted per terminal-record purge batch. + private const int TerminalPurgeBatchSize = 500; + + /// Batches deleted per opportunistic sweep, bounding one caller's latency cost. + private const int TerminalPurgeBatchesPerSweep = 4; + + /// Minimum interval between opportunistic terminal-record sweeps. + private static readonly TimeSpan TerminalPurgeMinInterval = TimeSpan.FromHours(6); + + /// + /// Ticks of the last opportunistic terminal-record sweep, process-wide. + /// is scoped, so this must be static to + /// rate-limit across requests; makes the + /// claim-the-sweep step a compare-and-swap so concurrent readers do not all + /// sweep at once. + /// + private static long _lastTerminalPurgeTicks; + /// /// Initializes a new instance of with all required dependencies. /// @@ -161,28 +179,123 @@ public class DeploymentService int instanceId, string user, CancellationToken cancellationToken = default) + { + // Single-instance deploy = the same three phases the bulk path runs, with a + // batch of one and a private flatten session. Composing both entry points + // out of the same phase helpers is what keeps deployment identity, + // idempotency, lock coverage and optimistic concurrency from drifting apart + // between them. + var prepared = await PrepareDeploymentAsync( + instanceId, user, session: null, cancellationToken); + + if (prepared.EarlyResult is { } early) + return early; + + using (prepared.LockHandle) + { + var outcome = await StageAndSendDeploymentAsync( + prepared, user, dbGate: null, cancellationToken); + return await FinalizeDeploymentAsync(prepared, outcome, user, cancellationToken); + } + } + + /// + /// Phase 1 of a deployment (serial, database-bound): validate the state + /// transition, take the per-instance operation lock, mint the deployment id, + /// flatten + validate, run query-before-redeploy reconciliation and insert the + /// InProgress . + /// + /// + /// Note what is NOT here: the PendingDeployment row. Its fetch token + /// expires after it is + /// STAGED, so staging in phase 1 started the clock on every instance in a bulk + /// deploy at once while phase 2 reached them one batch at a time — with enough + /// instances the tail's tokens expired before their command was ever sent, and + /// the site's fetch 404'd. Staging therefore moved to + /// , immediately before each send. + /// + /// + /// + /// Everything here touches the scoped, non-thread-safe DbContext, so it + /// must run serially — a bulk site deploy loops this phase before fanning out + /// the phase-2 network waits. On success the returned + /// owns the operation lock; the CALLER is + /// responsible for disposing it once phase 3 has completed. + /// + /// + /// Instance being deployed. + /// User attributed on the deployment record and audit rows. + /// + /// Shared flatten session for a batch, or for a private + /// one. Sharing it across a batch is what collapses N identical template-chain + /// walks into one. + /// + /// Cancellation token. + /// + /// A prepared deployment ready to send, or one carrying + /// when the deploy resolved before + /// any site round-trip (validation failure, or a reconciled prior deployment). + /// + private async Task PrepareDeploymentAsync( + int instanceId, + string user, + FlattenSession? session, + CancellationToken cancellationToken) { // Load instance var instance = await _repository.GetInstanceByIdAsync(instanceId, cancellationToken); if (instance == null) - return Result.Failure($"Instance with ID {instanceId} not found."); + return PreparedDeployment.Resolved(Result.Failure($"Instance with ID {instanceId} not found.")); // Validate state transition var transitionError = StateTransitionValidator.ValidateTransition(instance.State, "deploy"); if (transitionError != null) - return Result.Failure(transitionError); + return PreparedDeployment.Resolved(Result.Failure(transitionError)); // Acquire per-instance operation lock - using var lockHandle = await _lockManager.AcquireAsync( + var lockHandle = await _lockManager.AcquireAsync( instance.UniqueName, _options.OperationLockTimeout, cancellationToken); + try + { + return await PrepareUnderLockAsync(instance, user, session, lockHandle, cancellationToken); + } + catch + { + // The lock is handed to the caller only on the success path; any fault + // during preparation must release it here or a failed prepare would + // wedge the instance until the process restarts. + lockHandle.Dispose(); + throw; + } + } + + /// + /// The body of that runs while the + /// per-instance operation lock is held. Split out so the lock can be released + /// on every fault path with one try/catch rather than a nested one per step. + /// + private async Task PrepareUnderLockAsync( + Instance instance, + string user, + FlattenSession? session, + IDisposable lockHandle, + CancellationToken cancellationToken) + { + var instanceId = instance.Id; + // Generate unique deployment ID var deploymentId = Guid.NewGuid().ToString("N"); // Flatten configuration (captures template state at this point in time) - var flattenResult = await _flatteningPipeline.FlattenAndValidateAsync(instanceId, cancellationToken); + var flattenResult = await _flatteningPipeline.FlattenAndValidateAsync( + instanceId, cancellationToken, validateScripts: true, session); if (flattenResult.IsFailure) - return Result.Failure($"Validation failed: {flattenResult.Error}"); + { + lockHandle.Dispose(); + return PreparedDeployment.Resolved( + Result.Failure($"Validation failed: {flattenResult.Error}")); + } var flattenedConfig = flattenResult.Value.Configuration; var revisionHash = flattenResult.Value.RevisionHash; @@ -200,8 +313,9 @@ public class DeploymentService validationResult.Errors.Count, string.Join("; ", validationResult.Errors.Select(e => e.Message))); - return Result.Failure( - $"Pre-deployment validation failed: {validationResult.SummarizeErrors()}"); + lockHandle.Dispose(); + return PreparedDeployment.Resolved(Result.Failure( + $"Pre-deployment validation failed: {validationResult.SummarizeErrors()}")); } // Serialize for transmission (also the payload stored in the deployed @@ -218,7 +332,10 @@ public class DeploymentService var reconciled = await TryReconcileWithSiteAsync( instance, revisionHash, configJson, user, cancellationToken); if (reconciled != null) - return Result.Success(reconciled); + { + lockHandle.Dispose(); + return PreparedDeployment.Resolved(Result.Success(reconciled)); + } // Notify-and-fetch: the site fetches the staged config from // CentralFetchBaseUrl, so a deploy is impossible without it. Fail fast @@ -227,8 +344,11 @@ public class DeploymentService // confusing downstream site-fetch failure (and no InProgress record is // stranded). if (string.IsNullOrEmpty(_commOptions.CentralFetchBaseUrl)) - return Result.Failure( - "CentralFetchBaseUrl is not configured — required for deployment (notify-and-fetch)."); + { + lockHandle.Dispose(); + return PreparedDeployment.Resolved(Result.Failure( + "CentralFetchBaseUrl is not configured — required for deployment (notify-and-fetch).")); + } // Create the deployment record directly in InProgress. // @@ -257,39 +377,205 @@ public class DeploymentService try { - // Notify-and-fetch: instead of shipping the (potentially oversized, - // silently-dropped >128 KB) flattened config inline in a - // DeployInstanceCommand, stage it in a PendingDeployment row and send - // a small RefreshDeploymentCommand. The site fetches the config from - // CentralFetchBaseUrl over HTTP using the per-deployment fetch token. + // Site routing is resolved here (a repository read) so phase 2 needs + // nothing but the staging write itself. var siteId = await ResolveSiteIdentifierAsync(instance.SiteId, cancellationToken); - var token = DeploymentFetchToken.Generate(); - var stagedAt = DateTimeOffset.UtcNow; - await _repository.AddPendingDeploymentAsync(new PendingDeployment( - deploymentId, instanceId, revisionHash, configJson, token, - stagedAt, stagedAt + _commOptions.PendingDeploymentTtl), cancellationToken); - await _repository.SaveChangesAsync(cancellationToken); + return new PreparedDeployment( + instance, deploymentId, record, revisionHash, configJson, + siteId, lockHandle, EarlyResult: null); + } + catch (Exception ex) + { + // Preparation failed before anything was staged or sent. Record the + // failure exactly as the post-send path does (never leave the record + // InProgress) and release the lock — there is no phase 2 or 3 to run. + await MarkDeploymentFailedAsync(record, instance, deploymentId, user, ex); + lockHandle.Dispose(); + return PreparedDeployment.Resolved(FailureResultFor(ex)); + } + } - var command = new RefreshDeploymentCommand( - deploymentId, instance.UniqueName, revisionHash, user, stagedAt, - _commOptions.CentralFetchBaseUrl, token); + /// + /// Stages the deployment's PendingDeployment row and immediately sends + /// its RefreshDeploymentCommand — phase 2. + /// + /// + /// Notify-and-fetch: instead of shipping the (potentially oversized, + /// silently-dropped >128 KB) flattened config inline in a + /// DeployInstanceCommand, the config is staged in a + /// PendingDeployment row and a small RefreshDeploymentCommand is + /// sent; the site then fetches the config from CentralFetchBaseUrl over + /// HTTP using the per-deployment fetch token. + /// + /// + /// + /// Staging is immediately before the send, on purpose. The row's + /// ExpiresAt is stagedAt + PendingDeploymentTtl (5 min by + /// default) and the fetch endpoint enforces it, so the gap between staging and + /// sending is dead time burned off the token's life. Staging the whole batch up + /// front in phase 1 made that gap grow with batch size — for a large site the + /// last instances' tokens could expire before their command was even sent. + /// + /// + /// + /// Serialisation. Staging is the ONE repository touch in this otherwise + /// network-only phase, and the scoped DbContext is not thread-safe, so a + /// bulk deploy passes — a 1-permit semaphore that lets + /// exactly one in-flight send do its staging write at a time. The sends + /// themselves stay fully concurrent (they touch no repository), so this costs no + /// parallelism, and the single-instance path passes + /// because there is nothing to serialise against. + /// + /// + /// + /// A staging failure is returned as the phase-2 fault rather than thrown, so + /// phase 3 marks the record Failed on exactly the same path as a send fault — + /// the record must never be left InProgress. + /// + /// + /// The prepared deployment to stage and send. + /// User attributed on the refresh command. + /// Serialises the staging write across a concurrent batch; for a batch of one. + /// Cancellation token, already carrying any per-instance deadline. + /// The site's response, or the exception that prevented one. + private async Task StageAndSendDeploymentAsync( + PreparedDeployment prepared, + string user, + SemaphoreSlim? dbGate, + CancellationToken cancellationToken) + { + RefreshDeploymentCommand command; + try + { + if (dbGate is not null) + await dbGate.WaitAsync(cancellationToken); + try + { + command = await StagePendingDeploymentAsync(prepared, user, cancellationToken); + } + finally + { + dbGate?.Release(); + } + } + catch (Exception ex) + { + return new SendOutcome(null, ex); + } + return await SendDeploymentAsync(prepared, command, cancellationToken); + } + + /// + /// Writes the PendingDeployment row for a prepared deployment and builds + /// the matching RefreshDeploymentCommand, stamping stagedAt at the + /// moment of the write. + /// + /// + /// Cleanup of the staged row is TTL-based ONLY — it is deliberately NOT deleted + /// on success or on failure. On a central-side Ask timeout the site may have + /// applied AND told the standby node to fetch; deleting now would 404 that + /// in-flight standby fetch and break failover. Supersession bounds pending rows + /// to ≤1 per instance and the fetch endpoint enforces the TTL, so leaving rows + /// for TTL purge is safe. Expired rows are swept by the central + /// PendingDeploymentPurgeActor singleton on its + /// cadence. + /// + /// + /// The prepared deployment being staged. + /// User attributed on the refresh command. + /// Cancellation token. + /// The refresh command to send to the site. + private async Task StagePendingDeploymentAsync( + PreparedDeployment prepared, + string user, + CancellationToken cancellationToken) + { + var token = DeploymentFetchToken.Generate(); + var stagedAt = DateTimeOffset.UtcNow; + + await _repository.AddPendingDeploymentAsync(new PendingDeployment( + prepared.DeploymentId, prepared.Instance.Id, prepared.RevisionHash, + prepared.ConfigJson, token, + stagedAt, stagedAt + _commOptions.PendingDeploymentTtl), cancellationToken); + await _repository.SaveChangesAsync(cancellationToken); + + return new RefreshDeploymentCommand( + prepared.DeploymentId, prepared.Instance.UniqueName, prepared.RevisionHash, + user, stagedAt, _commOptions.CentralFetchBaseUrl, token); + } + + /// + /// Phase 2 of a deployment (parallelisable, network-bound): the + /// RefreshDeploymentCommand round-trip to the site. Touches no + /// repository and mutates no shared state, which is what makes it safe to fan + /// out across a batch while phases 1 and 3 stay serial. + /// + /// The prepared deployment to send. + /// The refresh command built by the staging step immediately before this call. + /// Cancellation token, already carrying any per-instance deadline. + /// The site's response, or the exception that prevented one. + private async Task SendDeploymentAsync( + PreparedDeployment prepared, + RefreshDeploymentCommand command, + CancellationToken cancellationToken) + { + try + { _logger.LogInformation( "Sending deployment {DeploymentId} for instance {Instance} to site {SiteId} (notify-and-fetch)", - deploymentId, instance.UniqueName, siteId); + prepared.DeploymentId, prepared.Instance.UniqueName, prepared.SiteIdentifier); - // Cleanup of the staged PendingDeployment is TTL-based ONLY — the row - // is deliberately NOT deleted on success or in the catch. On a - // central-side Ask timeout the site may have applied AND told the - // standby node to fetch; deleting now would 404 that in-flight - // standby fetch and break failover. Supersession bounds pending rows - // to ≤1 per instance and the fetch endpoint enforces the TTL, so - // leaving rows for TTL purge is safe. Expired rows are swept by the - // central PendingDeploymentPurgeActor singleton on its - // CommunicationOptions.PendingDeploymentPurgeInterval cadence. - var response = await _communicationService.RefreshDeploymentAsync(siteId, command, cancellationToken); + var response = await _communicationService.RefreshDeploymentAsync( + prepared.SiteIdentifier, command, cancellationToken); + return new SendOutcome(response, null); + } + catch (Exception ex) + { + return new SendOutcome(null, ex); + } + } + /// + /// Phase 3 of a deployment (serial, database-bound): commit the terminal + /// status, apply post-success side effects, and write the audit row. + /// + /// The prepared deployment whose send just completed. + /// The site response or the fault from phase 2. + /// User attributed on the audit rows. + /// Cancellation token. + /// The deployment result for the caller. + private async Task> FinalizeDeploymentAsync( + PreparedDeployment prepared, + SendOutcome outcome, + string user, + CancellationToken cancellationToken) + { + var instance = prepared.Instance; + var record = prepared.Record; + var deploymentId = prepared.DeploymentId; + var instanceId = instance.Id; + + if (outcome.Error is { } sendError) + { + // Any fault out of the site round-trip (timeout, cancellation, + // transport, serialization) must leave the deployment record as + // Failed -- the design requires an interrupted deployment to be + // treated as failed, never stuck in InProgress. + await MarkDeploymentFailedAsync(record, instance, deploymentId, user, sendError); + + _logger.LogError(sendError, + "Deployment {DeploymentId} for instance {Instance} failed", + deploymentId, instance.UniqueName); + + return FailureResultFor(sendError); + } + + var response = outcome.Response!; + + try + { // Update status based on site response. record.Status = response.Status; record.ErrorMessage = response.ErrorMessage; @@ -322,7 +608,7 @@ public class DeploymentService // logged loudly for operator reconciliation but must not flip // the already-committed Success record back to Failed. await ApplyPostSuccessSideEffectsAsync( - instance, deploymentId, revisionHash, configJson, + instance, deploymentId, prepared.RevisionHash, prepared.ConfigJson, forceEnabledState: true, cancellationToken); } @@ -345,61 +631,479 @@ public class DeploymentService } catch (Exception ex) { - // Any exception out of the try (timeout, - // cancellation, transport, serialization, DB) must leave the - // deployment record as Failed -- the design requires an interrupted - // deployment to be treated as failed, never stuck in InProgress. - // - // The failure-status write must NOT use the - // operation's cancellation token. If the operation was cancelled or - // timed out, that token is already cancelled and the cleanup writes - // would themselves throw before the Failed status is persisted. - // Use CancellationToken.None so the failure is durably recorded. - var isTimeout = ex is TimeoutException or OperationCanceledException or Akka.Actor.AskTimeoutException; - - record.Status = DeploymentStatus.Failed; - record.ErrorMessage = isTimeout - ? $"{TimeoutFailurePrefix} {ex.Message}" - : $"Deployment error: {ex.Message}"; - record.CompletedAt = DateTimeOffset.UtcNow; - - try - { - await _repository.UpdateDeploymentRecordAsync(record, CancellationToken.None); - // Note: if the staging SaveChangesAsync above was interrupted, an - // Added PendingDeployment may still be tracked and will be - // committed by this cleanup save. That row is orphaned (no - // RefreshDeploymentCommand was sent, so no site holds its token) - // and is removed by TTL purge / superseded by the next deploy -- - // harmless. - await _repository.SaveChangesAsync(CancellationToken.None); - NotifyStatusChange(record); - - await _auditService.LogAsync(user, "DeployFailed", "Instance", instanceId.ToString(), - instance.UniqueName, new { DeploymentId = deploymentId, Error = ex.Message }, - CancellationToken.None); - } - catch (Exception cleanupEx) - { - // The deployment already failed; a failed cleanup write must not - // mask the original error. Log loudly so an operator can reconcile. - _logger.LogError(cleanupEx, - "Failed to persist Failed status for deployment {DeploymentId} of instance {Instance} " + - "after deployment error: {Error}", - deploymentId, instance.UniqueName, ex.Message); - } + // A fault in the post-send persistence path is handled identically to + // a send fault: the record must never be left InProgress. + await MarkDeploymentFailedAsync(record, instance, deploymentId, user, ex); _logger.LogError(ex, "Deployment {DeploymentId} for instance {Instance} failed", deploymentId, instance.UniqueName); - return Result.Failure( - isTimeout - ? $"Deployment timed out: {ex.Message}" - : $"Deployment failed: {ex.Message}"); + return FailureResultFor(ex); } } + /// + /// Writes the terminal status (plus the + /// failure audit row) for a deployment that faulted, on any phase. + /// + /// + /// The failure-status write deliberately uses : + /// if the operation was cancelled or timed out, the operation's own token is + /// already cancelled and these cleanup writes would themselves throw before + /// the Failed status was persisted, leaving the record stuck InProgress. + /// + /// + /// + /// A fault DURING cleanup is logged loudly and swallowed — it must not mask + /// the original error the caller is about to report. + /// + /// + private async Task MarkDeploymentFailedAsync( + DeploymentRecord record, + Instance instance, + string deploymentId, + string user, + Exception cause) + { + record.Status = DeploymentStatus.Failed; + record.ErrorMessage = IsTimeoutFault(cause) + ? $"{TimeoutFailurePrefix} {cause.Message}" + : $"Deployment error: {cause.Message}"; + record.CompletedAt = DateTimeOffset.UtcNow; + + try + { + await _repository.UpdateDeploymentRecordAsync(record, CancellationToken.None); + // Note: if the staging SaveChangesAsync was interrupted, an Added + // PendingDeployment may still be tracked and will be committed by this + // cleanup save. That row is orphaned (no RefreshDeploymentCommand was + // sent, so no site holds its token) and is removed by TTL purge / + // superseded by the next deploy -- harmless. + await _repository.SaveChangesAsync(CancellationToken.None); + NotifyStatusChange(record); + + await _auditService.LogAsync(user, "DeployFailed", "Instance", instance.Id.ToString(), + instance.UniqueName, new { DeploymentId = deploymentId, Error = cause.Message }, + CancellationToken.None); + } + catch (Exception cleanupEx) + { + _logger.LogError(cleanupEx, + "Failed to persist Failed status for deployment {DeploymentId} of instance {Instance} " + + "after deployment error: {Error}", + deploymentId, instance.UniqueName, cause.Message); + } + } + + /// + /// True when a deployment fault is a timeout/cancellation rather than a hard + /// error. Drives both the marker (which the + /// query-before-redeploy trigger reads) and the caller-facing wording. + /// + private static bool IsTimeoutFault(Exception ex) => + ex is TimeoutException or OperationCanceledException or Akka.Actor.AskTimeoutException; + + /// Maps a deployment fault to the caller-facing failure result. + private static Result FailureResultFor(Exception ex) => + Result.Failure( + IsTimeoutFault(ex) + ? $"Deployment timed out: {ex.Message}" + : $"Deployment failed: {ex.Message}"); + + /// + /// A deployment that has cleared phase 1 and is ready for its site round-trip, + /// OR one that resolved during phase 1 (carrying ). + /// + /// + /// When is this record OWNS the + /// per-instance operation lock in , and the caller must + /// dispose it after phase 3. When is set the lock has + /// already been released and every other member is a placeholder. + /// + /// + private sealed record PreparedDeployment( + Instance Instance, + string DeploymentId, + DeploymentRecord Record, + string RevisionHash, + string ConfigJson, + string SiteIdentifier, + IDisposable? LockHandle, + Result? EarlyResult) + { + /// Builds a phase-1-resolved deployment carrying the given result and holding no lock. + /// The result to hand back to the caller. + /// A prepared deployment whose is set. + public static PreparedDeployment Resolved(Result result) => + new(null!, string.Empty, null!, string.Empty, string.Empty, string.Empty, null, result); + } + + /// + /// Result of phase 2: either the site's response or the exception that + /// prevented one. Exactly one is non-null. + /// + private readonly record struct SendOutcome(DeploymentStatusResponse? Response, Exception? Error); + + /// + /// Deploy every deployable instance at a site in one operation. + /// + /// + /// Structured as the three deployment phases with the middle one fanned out: + /// + /// + /// + /// Prepare (serial). Every instance is flattened, validated and given + /// an InProgress record on the caller's single scoped + /// DbContext. All instances share ONE , + /// so a template chain common to N instances is walked once, and the + /// session-global queries (shared scripts, schema library, the site's data + /// connections) run once for the whole batch instead of once per instance. + /// + /// + /// Stage + send (bounded parallel). Site round-trips run concurrently + /// up to , + /// each under its own + /// + /// deadline, so one wedged instance cannot stall the batch. Each instance's + /// PendingDeployment row is staged here, immediately before its own + /// send, so a fetch token's TTL is not burned waiting for the batch ahead of + /// it; that single repository touch is serialised behind a 1-permit + /// semaphore, since the DbContext is not thread-safe. + /// + /// + /// Finalize (serial). Terminal statuses, post-success side effects + /// and audit rows are committed one instance at a time, then each + /// instance's operation lock is released. + /// + /// + /// + /// + /// Semantics per instance are byte-for-byte those of + /// — its own deployment id and revision hash, + /// its own operation lock held across the whole operation, its own + /// optimistically-concurrent status record, and the same query-before-redeploy + /// idempotency check. Instances that cannot be deployed (wrong state, failed + /// validation, lock already held) are reported as failed rows; the batch is + /// NOT all-or-nothing, mirroring the artifact deployment path. + /// + /// + /// The database ID of the site whose instances are deployed. + /// The username initiating the deployment, recorded on every record and audit row. + /// Cancellation token for the operation. + /// A per-instance result matrix, or a failure result when the site itself cannot be resolved. + public async Task> DeploySiteAsync( + int siteId, + string user, + CancellationToken cancellationToken = default) + { + var site = await _siteRepository.GetSiteByIdAsync(siteId, cancellationToken); + if (site == null) + return Result.Failure($"Site with ID {siteId} not found."); + + var instances = await _siteRepository.GetInstancesBySiteIdAsync(siteId, cancellationToken); + if (instances.Count == 0) + return Result.Success(new SiteDeploymentSummary(site.SiteIdentifier, [], 0, 0)); + + // ---- Phase 1: prepare, serially, on ONE shared flatten session. ---- + var session = _flatteningPipeline.CreateSession(); + var prepared = new List(instances.Count); + var results = new List(instances.Count); + + // Index of the first prepared deployment phase 3 has NOT yet finalised. + // Everything from here to the end of `prepared` still owns an operation + // lock and an InProgress record, so the cleanup handler below knows exactly + // what it must unwind if the operation escapes early. + var finalizedUpTo = 0; + try + { + foreach (var instance in instances) + { + cancellationToken.ThrowIfCancellationRequested(); + + PreparedDeployment step; + try + { + step = await PrepareDeploymentAsync(instance.Id, user, session, cancellationToken); + } + catch (Exception ex) + { + // A prepare fault (most commonly a TimeoutException from the + // operation lock because another operation holds this instance) + // fails only this instance. Recording it and moving on is what + // makes a bulk deploy usable while individual instances are busy. + _logger.LogWarning(ex, + "Preparing instance {Instance} for bulk deployment of site {SiteId} failed", + instance.UniqueName, site.SiteIdentifier); + results.Add(new InstanceDeploymentResult( + instance.Id, instance.UniqueName, null, false, ex.Message)); + continue; + } + + if (step.EarlyResult is { } early) + { + results.Add(ToInstanceResult(instance.Id, instance.UniqueName, early)); + continue; + } + + prepared.Add(step); + } + + // ---- Phase 2: stage + bounded-parallel site round-trips. ---- + // Never throws: faults (cancellation included) come back as per-instance + // outcomes so phase 3 still runs for every prepared deployment. + var outcomes = await SendPreparedAsync(prepared, user, cancellationToken); + + // ---- Phase 3: finalize, serially, releasing each lock as we go. ---- + for (; finalizedUpTo < prepared.Count; finalizedUpTo++) + { + var step = prepared[finalizedUpTo]; + try + { + var result = await FinalizeDeploymentAsync(step, outcomes[finalizedUpTo], user, cancellationToken); + results.Add(ToInstanceResult(step.Instance.Id, step.Instance.UniqueName, result, step.DeploymentId)); + } + finally + { + step.LockHandle?.Dispose(); + } + } + } + catch (Exception ex) + { + // Nothing above may escape while a prepared deployment still holds its + // per-instance operation lock: OperationLockManager hands out real + // semaphores, so an undisposed handle wedges that instance against + // EVERY future mutating command for the life of the process — a + // permanent, restart-only outage for one instance. The two live escape + // routes are phase 1's ThrowIfCancellationRequested and a fault from + // phase 3's own persistence; both land here. + // + // Records are finalised as Failed for the same reason the send/persist + // paths do it: a deployment must never be left InProgress. Uses + // CancellationToken.None inside (see MarkDeploymentFailedAsync) so the + // already-cancelled token cannot abort the cleanup writes themselves. + await ReleasePreparedAsync(prepared, finalizedUpTo, user, ex); + throw; + } + + var summary = new SiteDeploymentSummary( + site.SiteIdentifier, + results, + results.Count(r => r.Success), + results.Count(r => !r.Success)); + + _logger.LogInformation( + "Bulk deployment of site {SiteId} complete: {SuccessCount} succeeded, {FailureCount} failed " + + "({ChainLoads} template-chain load(s) across {InstanceCount} instance(s))", + site.SiteIdentifier, summary.SuccessCount, summary.FailureCount, + session.ChainLoads, instances.Count); + + return Result.Success(summary); + } + + /// + /// Unwinds the prepared deployments a bulk deploy never finalised, after the + /// operation escaped early (cancellation, or a fault in phase 3's own + /// persistence). For each one it writes the terminal Failed status and releases + /// the per-instance operation lock. + /// + /// + /// Best-effort throughout: a fault while unwinding one entry must not stop the + /// remaining locks from being released, and must not mask the original + /// exception the caller is about to rethrow. The lock release in particular is + /// in a finally, because leaking it is the unrecoverable outcome — the + /// Failed status can be reconciled by an operator, a wedged semaphore cannot. + /// + /// + /// The batch's prepared deployments. + /// Index of the first entry phase 3 did not finalise. + /// User attributed on the failure audit rows. + /// The exception that aborted the operation. + /// A task that completes once every outstanding entry has been unwound. + private async Task ReleasePreparedAsync( + IReadOnlyList prepared, + int from, + string user, + Exception cause) + { + for (var i = from; i < prepared.Count; i++) + { + var step = prepared[i]; + try + { + await MarkDeploymentFailedAsync( + step.Record, step.Instance, step.DeploymentId, user, cause); + } + catch (Exception ex) + { + _logger.LogError(ex, + "Failed to finalize deployment {DeploymentId} for instance {Instance} while unwinding " + + "an aborted bulk deployment", + step.DeploymentId, step.Instance.UniqueName); + } + finally + { + step.LockHandle?.Dispose(); + } + } + } + + /// + /// Runs phase 2 for a whole batch: every prepared deployment's staging write + /// plus site round-trip, concurrent up to + /// and each + /// bounded by + /// . + /// Outcomes are returned positionally so phase 3 can pair them back with their + /// prepared deployment. + /// + /// + /// Never throws on cancellation: the whole point of returning outcomes + /// positionally is that phase 3 must run for EVERY prepared deployment, whatever + /// happened in phase 2 — otherwise records stay InProgress and their operation + /// locks stay held. A cancelled instance is recorded as its own + /// fault and finalised as Failed like any other. + /// + /// + /// The prepared deployments to stage and send. + /// User attributed on each refresh command. + /// Cancellation token for the batch. + /// One outcome per prepared deployment, positionally aligned. + private async Task SendPreparedAsync( + IReadOnlyList prepared, + string user, + CancellationToken cancellationToken) + { + var outcomes = new SendOutcome[prepared.Count]; + if (prepared.Count == 0) + return outcomes; + + using var gate = new SemaphoreSlim(_options.SiteDeploymentMaxParallelism); + // Serialises the ONE repository touch left in this phase (the staging + // write) across the concurrent sends — the scoped DbContext is not + // thread-safe. The sends themselves stay fully concurrent. + using var dbGate = new SemaphoreSlim(1, 1); + + var sends = prepared.Select(async (step, index) => + { + // Cancellation must not escape as an exception: it would abandon phase 3 + // for every instance, leaking their operation locks (a wedged + // per-instance semaphore is permanent for the life of the process) and + // stranding their records InProgress. + try + { + await gate.WaitAsync(cancellationToken); + } + catch (OperationCanceledException ex) + { + outcomes[index] = new SendOutcome(null, ex); + return; + } + + try + { + using var cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + cts.CancelAfter(_options.SiteDeploymentTimeoutPerInstance); + outcomes[index] = await StageAndSendDeploymentAsync(step, user, dbGate, cts.Token); + } + catch (Exception ex) + { + // StageAndSendDeploymentAsync already converts its own faults into + // outcomes; this is the belt-and-braces net so no send task can + // fault Task.WhenAll and skip phase 3. + outcomes[index] = new SendOutcome(null, ex); + } + finally + { + gate.Release(); + } + }); + + await Task.WhenAll(sends); + return outcomes; + } + + /// Maps a per-instance deployment result into the bulk summary row shape. + private static InstanceDeploymentResult ToInstanceResult( + int instanceId, + string uniqueName, + Result result, + string? deploymentId = null) => + result.IsSuccess + ? new InstanceDeploymentResult( + instanceId, uniqueName, result.Value.DeploymentId, true, null) + : new InstanceDeploymentResult( + instanceId, uniqueName, deploymentId, false, result.Error); + + /// + /// Opportunistic retention sweep for TERMINAL deployment records, rate-limited + /// to once every per process and bounded + /// to batches per call. + /// + /// + /// Swept on read (the deployment-list path) rather than from a timer for the + /// same reason expired secured writes are: the component owns no scheduling + /// infrastructure of its own, and the read path is exactly the operation that + /// suffers when the table is allowed to grow without bound. Both central nodes + /// may sweep; the delete is idempotent, so a duplicate pass is a no-op. + /// + /// + /// + /// Best-effort: a failed sweep is logged and swallowed. Retention maintenance + /// must never fail the operator's actual query. + /// + /// + /// Cancellation token. + /// The number of records purged by this call (0 when the sweep was skipped). + public async Task TryPurgeTerminalDeploymentRecordsAsync(CancellationToken cancellationToken = default) + { + var retention = _options.TerminalDeploymentRecordRetention; + if (retention <= TimeSpan.Zero) + return 0; + + var now = DateTimeOffset.UtcNow; + var last = Interlocked.Read(ref _lastTerminalPurgeTicks); + if (last != 0 && now.UtcTicks - last < TerminalPurgeMinInterval.Ticks) + return 0; + + // Claim the sweep. A loser of this CAS returns immediately rather than + // running a redundant second pass. + if (Interlocked.CompareExchange(ref _lastTerminalPurgeTicks, now.UtcTicks, last) != last) + return 0; + + var cutoff = now - retention; + var purged = 0; + + try + { + for (var batch = 0; batch < TerminalPurgeBatchesPerSweep; batch++) + { + var deleted = await _repository.PurgeTerminalDeploymentRecordsAsync( + cutoff, TerminalPurgeBatchSize, cancellationToken); + purged += deleted; + if (deleted < TerminalPurgeBatchSize) + break; + } + + if (purged > 0) + { + _logger.LogInformation( + "Purged {Count} terminal deployment record(s) completed before {Cutoff} " + + "(retention {Retention}).", + purged, cutoff, retention); + } + } + catch (Exception ex) + { + _logger.LogWarning(ex, + "Terminal deployment-record retention sweep failed; retrying after {Interval}.", + TerminalPurgeMinInterval); + } + + return purged; + } + /// /// Disable an instance. Stops Instance Actor, retains config, S&F drains. /// @@ -1178,3 +1882,37 @@ public record DeploymentComparisonResult( bool IsStale, DateTimeOffset DeployedAt, ConfigurationDiff? Diff = null); + +/// +/// One instance's outcome inside a bulk site deployment. +/// +/// Database id of the instance. +/// The instance's unique name (also its operation-lock key). +/// +/// The deployment id minted for this instance, or when the +/// instance never got that far (state-transition rejection, lock contention). +/// +/// Whether the site confirmed the apply. +/// Failure detail; on success. +public record InstanceDeploymentResult( + int InstanceId, + string UniqueName, + string? DeploymentId, + bool Success, + string? ErrorMessage); + +/// +/// Result matrix of a bulk site deployment. Mirrors the shape of +/// ArtifactDeploymentSummary: successes are not rolled back when other +/// instances fail, and each failed instance is individually retryable through the +/// ordinary single-instance deploy. +/// +/// The site's string identifier (e.g. site-a). +/// Per-instance outcomes. +/// Number of instances the site confirmed. +/// Number of instances that failed for any reason. +public record SiteDeploymentSummary( + string SiteIdentifier, + IReadOnlyList InstanceResults, + int SuccessCount, + int FailureCount); diff --git a/src/ZB.MOM.WW.ScadaBridge.DeploymentManager/FlattenSession.cs b/src/ZB.MOM.WW.ScadaBridge.DeploymentManager/FlattenSession.cs new file mode 100644 index 00000000..6c0fe23c --- /dev/null +++ b/src/ZB.MOM.WW.ScadaBridge.DeploymentManager/FlattenSession.cs @@ -0,0 +1,257 @@ +using ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites; +using ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates; +using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services; +using ZB.MOM.WW.ScadaBridge.Commons.Types.Flattening; + +namespace ZB.MOM.WW.ScadaBridge.DeploymentManager; + +/// +/// Memoises the repository reads that performs, +/// for the lifetime of ONE flatten/validate session. +/// +/// +/// Without a session every instance flatten re-walks its whole template +/// inheritance chain (one query per link), re-loads the compositions of every +/// template in that chain and of every composed chain it reaches, and re-issues +/// the three session-global queries — shared scripts, the shared-schema library, +/// and the target site's data connections. For a bulk deploy of N instances off +/// the same template that is N× the same work; even a single instance re-loads a +/// composed chain once per composing template. +/// +/// +/// +/// Cache key. Templates are keyed on (id, ITemplateGraphWatermark +/// version) and chain MEMBERSHIP additionally on the watermark's +/// , so a template edited +/// mid-session is re-read rather than served from the memo. The session is also +/// short-lived by construction — it is created per deploy/validate operation and +/// discarded with it — so a cached graph can never outlive the operation that +/// captured it. That is the design's "template state is captured at the time of +/// flatten" guarantee, unchanged: the session narrows the capture window, it does +/// not widen it. +/// +/// +/// +/// Threading. A session is NOT thread-safe and must not be shared across +/// threads. It caches entities materialised by one scoped DbContext, which +/// is itself single-threaded, and it hands out tracked entities that the +/// in-flight bundle importer relies on observing. DeploySiteAsync honours +/// this by running the whole prepare (flatten) phase serially on one scope and +/// parallelising only the site round-trips, which touch no repository at all. +/// +/// +public sealed class FlattenSession +{ + private readonly ITemplateGraphWatermark _watermark; + + private readonly Dictionary _templates = new(); + private readonly Dictionary _chains = new(); + private readonly Dictionary Compositions)> _compositions = new(); + private readonly Dictionary> _dataConnections = new(); + + private IReadOnlyList? _sharedScripts; + private IReadOnlyDictionary? _schemaLibrary; + + /// Initializes a session bound to the process-wide graph watermark. + /// Watermark supplying the per-template versions used as cache keys. + public FlattenSession(ITemplateGraphWatermark watermark) + { + _watermark = watermark ?? throw new ArgumentNullException(nameof(watermark)); + } + + /// + /// Number of template-chain walks that actually hit the repository. Reset + /// never; used by tests to assert that N instances sharing a template load + /// that chain exactly once. + /// + public int ChainLoads { get; private set; } + + /// Number of single-template repository reads that missed the memo. + public int TemplateLoads { get; private set; } + + /// Number of per-template composition queries that missed the memo. + public int CompositionLoads { get; private set; } + + /// Number of times the session-global queries (shared scripts, schema library) ran. + public int GlobalLoads { get; private set; } + + /// Number of per-site data-connection queries that missed the memo. + public int DataConnectionLoads { get; private set; } + + /// + /// Every template id this session resolved — the union of all inheritance and + /// composed chains it walked. The staleness fast path records the watermark + /// version of exactly this set, so a change to ANY template that fed a + /// flatten invalidates the memoised revision hash. + /// + public IReadOnlyCollection VisitedTemplateIds => _templates.Keys; + + /// + /// Returns the inheritance chain for (the + /// template itself, then each ancestor), loading it through + /// on a miss. + /// + /// Root of the chain to build. + /// Repository read for one template with its children. + /// The chain, ordered leaf-first. + public async Task> GetChainAsync( + int templateId, + Func> loadTemplate) + { + var structure = _watermark.StructureVersion; + if (_chains.TryGetValue(templateId, out var cached) && IsChainCurrent(cached, structure)) + return cached.Chain; + + var chain = new List