Merge branch 'arch-review-remediation'
This commit is contained in:
Joseph Doherty
@@ -64,7 +64,7 @@ spec for each is `docs/requirements/Component-<Name>.md`, and `README.md` carrie
|
||||
### Architecture & Runtime
|
||||
- Instance modeled as Akka actor (Instance Actor) — single source of truth for runtime state.
|
||||
- Site Runtime actor hierarchy: Deployment Manager singleton → Instance Actors → Script Actors + Alarm Actors.
|
||||
- Script Actors spawn short-lived Script Execution Actors on a dedicated blocking I/O dispatcher.
|
||||
- Script/Alarm Actors launch script runs directly onto a dedicated blocking-I/O thread pool (`ScriptExecutionScheduler`) — the short-lived `ScriptExecutionActor`/`AlarmExecutionActor` children were **deleted** by WP3.1 (they had no `Receive` handler, no `PostStop`, and their `IActorRef` was never a message target; `ScriptRunLauncher` carries the run body verbatim). Same pool, same telemetry, same audit `ExecutionId`/`ParentExecutionId` threading, same "a stop does NOT cancel an in-flight run" semantics — minus a per-run actor cell. WP3.1 also: **trigger-expression evaluation left that pool** for the shared .NET thread pool behind `TriggerEvalGate` (arch-review finding #4 — blocked script bodies used to stall every Expression trigger on the node indefinitely); script and eval deadlines are armed **at enqueue** so queue wait burns the budget (a run dequeuing past its deadline is shed unrun); the pool is instance-scaled grow-only `clamp(max(ScriptExecutionThreadCount, ceil(instances/8)), 1, ScriptExecutionMaxThreadCount)`; the stuck-script watchdog now **detaches and replaces** a wedged worker (capped at pool size, gauge `DetachedScriptThreads`); `MaxConcurrentRunsPerScript` (4) sheds the newest run with a counter + rate-limited site event + an explicit error reply to Ask callers; `SiteScriptCompileCache` evicts an oldest-⅛ batch by approximate LRU instead of clearing wholesale; and deploy/startup compiles are warmed off-thread before the (ordering-critical, now cache-hit-only) synchronous deploy gate. Design memo: `docs/plans/2026-08-15-script-pool-split-design.md`.
|
||||
- Alarm Actors are separate peer subsystem from scripts (not inside Script Engine).
|
||||
- Shared scripts execute inline as compiled code (no separate actors).
|
||||
- Site-wide Akka stream for attribute value and alarm state changes with per-subscriber buffering.
|
||||
@@ -79,7 +79,7 @@ spec for each is `docs/requirements/Component-<Name>.md`, and `README.md` carrie
|
||||
- DCL write failures returned synchronously to calling script.
|
||||
- Tag path resolution retried periodically for devices still booting.
|
||||
- Static attribute writes persisted to local SQLite (survive restart/failover, reset on redeployment).
|
||||
- **Consolidated site database (LocalDb Phase 1 + 2, complete 2026-07-20).** Ten tables now live in ONE `ZB.MOM.WW.LocalDb`-managed SQLite file (enumerate them with `grep -r RegisterReplicated src/`), configured by the **required** `LocalDb:Path` (`/app/data/site-localdb.db` on the rig; validated with `ValidateOnStart`, so a site config missing it fails to boot). Consequences worth knowing:
|
||||
- **Consolidated site database (LocalDb Phase 1 + 2, complete 2026-07-20).** Ten tables now live in ONE `ZB.MOM.WW.LocalDb`-managed SQLite file (the `SiteLocalDbSetup.ReplicatedTables` list), configured by the **required** `LocalDb:Path` (`/app/data/site-localdb.db` on the rig; validated with `ValidateOnStart`, so a site config missing it fails to boot). Consequences worth knowing:
|
||||
- `site_events.id` changed from autoincrement INTEGER to an application-minted **GUID**. Last-writer-wins keys on the primary key, so two nodes independently minting `id=1,2,3…` would destroy each other's events rather than merge them. The event-log read path uses a composite `(timestamp, id)` keyset cursor with an **opaque string** continuation token; `EventLogEntry.Id` and both `ContinuationToken`s are `string`/`string?` on the site↔central Akka DTOs.
|
||||
- `ScadaBridge:OperationTracking:ConnectionString`, `ScadaBridge:SiteEventLog:DatabasePath` and — as of Phase 2 — `ScadaBridge:StoreAndForward:SqliteDbPath` + `ScadaBridge:Database:SiteDbPath` are all **migration-only** — nothing reads them but `SiteLocalDbLegacyMigrator`, which copies a pre-Phase-1 file in once (deterministic `mig-{NodeName}-{legacyId}` event ids, `INSERT OR IGNORE`, runs AFTER `RegisterReplicated` so migrated rows replicate) and renames it `.migrated`. Delete the keys once a node has migrated.
|
||||
- This incidentally fixes a data-loss bug: both legacy databases defaulted to CWD-relative paths **outside** the mounted volume and were discarded on every container recreate.
|
||||
@@ -89,14 +89,15 @@ spec for each is `docs/requirements/Component-<Name>.md`, and `README.md` carrie
|
||||
- CDC replication does all three jobs now: config deploys reach the standby as ordinary row changes — **the standby makes no fetch at all** during a deploy (`SiteReconciliationActor`'s node-STARTUP fetch when central reports gaps is a different, surviving path) — and buffer mutations replicate via triggers on `sf_messages`. `ReplaceAllAsync` was a destructive delete-all-then-insert-all resync and is **unsafe to reintroduce**: a mass DELETE on a replicated table would be captured and shipped to the peer. LocalDb's snapshot resync merges per row under LWW and never deletes, which is also why the old N1 directional-authority guard is gone — there is no wipe left to gate.
|
||||
- **`notification_lists` and `smtp_configurations` are created but deliberately NOT registered.** They are permanently empty on a site (no writer since 2026-07-10, the migrator skips them, the active-node purge keeps them empty), and registering them would open a standing replication channel whose only historical payload was plaintext SMTP passwords. Pinned by a security-named test, and verified live: those two tables have **no CDC triggers** on either rig node.
|
||||
- **Operational constraints (read before upgrading a site pair):** stop and start both nodes TOGETHER — rolling one at a time is no longer supported, since the legacy `SfBufferSnapshot` compatibility handler went with the replicator. And a node offline longer than `LocalDb:Replication:TombstoneRetention` (default 7 days) can resurrect deleted rows on rejoin. See `docs/deployment/topology-guide.md`.
|
||||
- `LocalDb:Replication:MaxBatchSize` batches by ROW COUNT, not bytes, against a 4 MB gRPC cap — the rig pins it to **16** (~70 KB worst-case `config_json` x 16 ~= 1.1 MB). The 500 default would allow ~35 MB.
|
||||
- **Batching is by BYTE BUDGET as of LocalDb 0.2.0** — `LocalDb:Replication:MaxBatchBytes` (default **2 MB**, sized under the 4 MB gRPC cap) bounds a delta/snapshot message by summed serialized size via a per-message split in `SyncSession.PumpLoopAsync`, with `MaxBatchSize` demoted to a secondary row cap; a single row over budget is sent alone rather than stalling the stream. **The rig's old `MaxBatchSize = 16` pin (a hand-computed byte-budget proxy: ~70 KB worst-case `config_json` x the 500 default is ~35 MB) is retired, but `MaxBatchSize` is NOT fully redundant with `MaxBatchBytes`** — it also bounds the separate DB READ page in `OplogStore.ReadBatchAboveAsync`/`SnapshotStreamer`, which materializes the whole page into memory *before* the byte-budget split runs, so an unset (500-default) `MaxBatchSize` still lets a reconnect drain transiently allocate ~35 MB per read even though every wire message stays under `MaxBatchBytes` (arch-review adversarial finding F2). Both site-a nodes on `docker/` therefore pin an explicit `"MaxBatchSize": 64` to bound that transient allocation, while `MaxBatchBytes` stays unset (its 2 MB default) to bound the wire message; site-b/site-c stay unreplicated so the key doesn't apply there.
|
||||
- **CDC registration is conditional, and both directions self-heal at boot** (arch-review WP1.3 + WP3.3, `Host/SiteLocalDbSetup.cs`). Capture triggers are installed only when this node has replication configured — `PeerAddress` **or** `ApiKey`, an OR because only the dialling half sets `PeerAddress` while the passive half carries the key alone. An unreplicated node calls **`DeregisterReplicated`** on all ten tables at boot, dropping triggers an earlier build left behind and pruning their oplog/row-version rows (idempotent; logs once at Information when something was actually cleaned). A replicating node registers with **`baselineExistingRows: true`**, which seeds `__localdb_row_version` for pre-existing rows at the LWW floor (HLC `0`, this node's id) and flags a snapshot resync — so turning replication ON for a site that has been running without it now converges on the rows already in the file instead of only on writes made after the restart. Deregistration must be symmetric (the handshake compares registered-table digests fail-closed), so replication is a both-nodes-together change in either direction. LocalDb 0.2.0 also makes backlog depth O(1) and drops the unused `__localdb_oplog_hlc` index; the on-disk bookkeeping schema goes to v2, upgraded in place on open, with no wire change (0.1.x peers still sync). **LocalDb 0.2.1** (two adversarial-review findings against 0.2.0, wire-compatible, no schema change) fixes: (1) `DeregisterReplicated` was deleting the HLC clock's only durable crash anchor along with the table rows — an ungraceful exit (crash, SIGKILL, power loss) on a fully-deregistered file left nothing to recover from, so the reopened clock could restart from wall-clock time and re-issue stamps below HLCs peers already hold (silently discarded by LWW); the current clock is now flushed inside the deregistration transaction, before the prune. (2) The sync inbox was an **unbounded** channel, so inbound memory was a function of what the peer sends — a mutual full snapshot (both sides of a pair baseline unconditionally when replication is enabled late) buffered whole, in memory, on both nodes at once; it is now **bounded at 64 messages** with wait-mode backpressure (unrelated to, and not to be confused with, the rig's separate `MaxBatchSize: 64` DB-read-page pin above), which required moving the snapshot to run alongside the receive loop (not inside the pre-loop handshake) so the inbox is drained for the whole session and can't deadlock itself.
|
||||
- All timestamps are UTC throughout the system.
|
||||
- Inter-cluster communication uses **three** transports, not two — **all cross-cluster command/control and data now rides gRPC** after the ClusterClient→gRPC migration's Phase 4 (`docs/plans/2026-07-22-clusterclient-to-grpc-plan.md`) deleted Akka `ClusterClient`/`ClusterClientReceptionist`: (1) **gRPC command/control** — site→central over the central-hosted `CentralControlService` (`GrpcCentralTransport`, sticky central-a→central-b channel pair; deployments/notifications/health/heartbeat/audit-ingest/reconcile), and central→site over the site-hosted `SiteCommandService` (`GrpcSiteTransport`, per-site NodeA→NodeB channel pair; the 28 lifecycle/OPC-UA/query/parked/route/failover commands); (2) **gRPC** server-streaming for real-time data (attribute values, alarm states, `SiteStreamService`); and (3) **plain token-gated HTTP** for the deployment config itself — notify-and-fetch, the site pulls the config from `DeploymentConfigEndpoints` (`ManagementService/DeploymentConfigEndpoints.cs`) with an `X-Deployment-Token` header, `AllowAnonymous` with the per-deployment token as the entire security boundary. The gRPC boundary is per-site PSK-authenticated (`ControlPlaneAuthInterceptor`, unchanged). There is **no receptionist registration** — discovery is by dialling configured endpoints; central builds one `SitePairChannelProvider` per site (addresses from `Site.GrpcNodeAAddress`/`GrpcNodeBAddress`, refreshed from the DB every 60s and on admin changes), sites dial `ScadaBridge:Communication:CentralGrpcEndpoints` (both central nodes, h2c on `CentralGrpcPort` 8083, **NOT** via Traefik). **Discovery is asymmetric by design:** central discovers site gRPC addresses from the *database* (refreshable at runtime), sites discover central from *appsettings* (`CentralGrpcEndpoints`, static — restart required; `StartupValidator` requires a Site node to list at least one). `Akka.Cluster.Tools` stays for ClusterSingleton; only the ClusterClient part is gone. **Central never buffers for an unreachable site** — the send fails with the caller's Ask/deadline timing out; a `ConnectionStateChanged` mechanism built for this was deleted as dead code.
|
||||
- **All clusters share ONE ActorSystem name**, `"scadabridge"` — hardcoded in `AkkaHostedService` at the `ActorSystem.Create` call. Central and each site are separate clusters *only* by seed-node partitioning. The constraint originated with ClusterClient (Akka.Remote address matching meant it could not reach a differently-named system); whether it is still load-bearing after the gRPC migration has **not** been re-verified, so treat the name as fixed until someone checks.
|
||||
- **`ActiveNodeEvaluator.SelfIsOldestUp` is THE single definition of "active node"** (`Communication/ClusterState/ActiveNodeEvaluator.cs`) — the **oldest Up member** in a role scope, and explicitly **never `cluster.State.Leader`**: leadership (lowest address) is an Akka-internal concept that diverges from singleton placement permanently once the original first node restarts and rejoins, and both sides claim it during a partition. The equivalence *oldest-Up == where `ClusterSingletonManager` places singletons* **is** the design. `ClusterActivityEvaluator.SelfIsOldest`, the S&F delivery gate, `/health/active` and the heartbeat `IsActive` stamp all delegate here.
|
||||
- Site nodes carry **two Akka roles**: the base `Site` plus a site-specific `site-{SiteId}` (`AkkaHostedService.BuildRoles`). Singletons scope to the **site-specific** role.
|
||||
- **The gRPC boundary is authenticated (PSK) as of 2026-07-22; Akka remoting still is not, and nothing is encrypted.** Akka remoting sets no `enable-ssl`, no secure cookie, no `trusted-selection-paths` — so intra-cluster Akka remoting remains open to anyone who can reach the remoting port, and that boundary still assumes a trusted network. The gRPC listener stays **h2c**, but `SiteStreamService` is no longer open: `ControlPlaneAuthInterceptor` (`Host/ControlPlaneAuthInterceptor.cs`) gates `/sitestream.SiteStreamService/` — including the `PullAuditEvents`/`PullSiteCalls` RPCs that return audit rows — against a **per-site preshared key**, fail-closed, constant-time compared, alongside the separate `LocalDbSyncAuthInterceptor` on `/localdb_sync.v1.LocalDbSync/` with its own separate key. **Site side:** `ScadaBridge:Communication:GrpcPsk`, in production `${secret:SB-GRPC-PSK-<siteId>}`, and **`StartupValidator` refuses to boot a site node without it** (an unset key would leave the node healthy-looking but serving nothing). **Central side:** `SitePskProvider` resolves `SB-GRPC-PSK-{siteId}` from the secrets store at channel-build time (sites are added at runtime, so no boot-time expansion is possible), with `ScadaBridge:Communication:SitePsks:{siteId}` as an override for hosts running without a master key — the docker rig uses the latter. One key per site, never fleet-wide. A bearer token over h2c is readable and replayable on-path; TLS is the follow-on hardening and needs no change to this design. Introduced by Phase 0 of the ClusterClient→gRPC migration (`docs/plans/2026-07-22-clusterclient-to-grpc-plan.md`).
|
||||
- gRPC streaming channel — **note the direction is inverted from the data flow**: data moves site→central, but each **site node hosts the gRPC server** (`SiteStreamGrpcServer`, Kestrel h2c, port 8083, mapped **only in the Site branch** of `Program.cs`) and **central is the client**, dialling in. Central creates per-site `SiteStreamGrpcClient` via `SiteStreamGrpcClientFactory`, keyed **`(siteId, endpoint)`** — the key was widened from site-only to fix an arch-review High where one session's NodeA→NodeB flip disposed a channel another session was still using. Proto evolution is **additive only** and field numbers are never reused (`AlarmStateUpdate` grew 7→23 fields for the native-alarm mirror). Generated C# is **vendored** under `Communication/SiteStreamGrpc/` with the `<Protobuf>` include commented out — regeneration is a manual toggle-build-copy-untoggle.
|
||||
- gRPC streaming channel — **note the direction is inverted from the data flow**: data moves site→central, but each **site node hosts the gRPC server** (`SiteStreamGrpcServer`, Kestrel h2c, port 8083, mapped **only in the Site branch** of `Program.cs`) and **central is the client**, dialling in. Central creates per-site `SiteStreamGrpcClient` via `SiteStreamGrpcClientFactory`, keyed **`(siteId, endpoint)`** — the key was widened from site-only to fix an arch-review High where one session's NodeA→NodeB flip disposed a channel another session was still using. Proto evolution is **additive only** and field numbers are never reused (`AlarmStateUpdate` grew 7→23 fields for the native-alarm mirror). Generated C# is **vendored** under `Communication/SiteStreamGrpc/` with the `<Protobuf>` include commented out — regeneration is a manual toggle-build-copy-untoggle. **The client reconnects on graceful (OK-status) stream completion, not just on fault (arch-review remediation, commit `34a3f4bb`)** — Kestrel's `MaxStreamLifetime`/`Grpc.AspNetCore.Server` max-connection-age periodically ends a healthy stream with a normal completion, which the client used to treat as terminal (no reconnect attempt), silently killing a site's live feed until the next process restart (observed up to ~4h on the rig); live-probed at a 2-minute forced lifetime, reconnect lands within one reconcile tick and `IsLive` reflects the gap in between.
|
||||
- Native alarms are a **read-only** mirror of OPC UA Alarms & Conditions and MxAccess Gateway alarms — **no ack-back, no central tables**; state lives in the site's `native_alarm_state`, survives failover, and is cleared on redeploy/undeploy (mirrors static overrides). Central's per-site live alarm cache (`ISiteAlarmLiveCache`) is **transient in-memory only** — there is deliberately no persisted central alarm store, so the 15s poll remains the NotReporting authority behind the live stream. See `Component-DataConnectionLayer.md` / `Component-CentralUI.md` for the model and the authoring surface.
|
||||
- **`AckTime` mirror enrichment + the `Alarms` script accessor (MES alarm-status API Phase 1, 2026-08-01).** `AlarmStateChanged` carries an additive `AckTime` (`DateTimeOffset?`), mirrored on the vendored `AlarmStateUpdate` proto as **field 24** and persisted inside `native_alarm_state`'s `metadata_json` — deliberately NOT a new column, because that table is `RegisterReplicated` and LocalDb builds its CDC triggers from the column list at registration time. Set only while a condition is active AND acknowledged (so it is null while unacked and cleared on re-raise); the DCL stamps the source's own ack instant for OPC UA (new SelectClause **index 18** = `AckedState/TransitionTime`) and its observation time of the ack transition for MxGateway, which supplies none. Site `Call` scripts read alarms via the new **`Alarms.CurrentAsync()`** accessor (`ScriptRuntimeContext` + `ScriptGlobals`, local Ask on `GetAlarmSnapshotRequest`, returns `Commons.Types.Scripts.ScriptAlarm`), mirrored on `ScriptCompileSurface` AND the Central UI `SandboxScriptHost` editor surface. The trust model needed no change — it is a deny-list over API roots, not an allow-list of context members. Plan: `docs/plans/2026-06-30-mes-alarm-status-api.md` (Phases 2–4 are deployed config, not repo).
|
||||
- OPC UA cert trust is **site-local and not persisted centrally** (follow-up): the verify-endpoint probe captures an untrusted server cert but **NEVER trusts it**, and DeploymentManager broadcasts `TrustServerCertCommand`/`RemoveServerCertCommand` to **BOTH** site nodes — `CertStoreActor` runs on every site node, not as a singleton, so PKI stores stay consistent across failover.
|
||||
@@ -121,6 +122,7 @@ spec for each is `docs/requirements/Component-<Name>.md`, and `README.md` carrie
|
||||
- Flattened configs include a revision hash for staleness detection.
|
||||
- Deployment identity: unique deployment ID + revision hash for idempotency.
|
||||
- Per-instance operation lock covers all mutating commands (deploy, disable, enable, delete).
|
||||
- **Bulk site deployment (`DeploymentService.DeploySiteAsync`, arch-review WP2.5)** redeploys every out-of-date instance at a site through one flatten session (a template chain shared by N instances is walked once) in three phases — serial Prepare, bounded-parallel Send (`SiteDeploymentMaxParallelism`, default 4), serial Finalize — still **not all-or-nothing**: any one instance's failure doesn't block the rest. The `PendingDeployment` staging row (5-minute TTL) is written per-instance immediately before that instance's round-trip in the Send phase, not upfront in Prepare, so a large batch's tail instances don't have their TTL clock already half-spent by the time they're reached.
|
||||
- Site-side apply is all-or-nothing per instance.
|
||||
- System-wide artifact version skew across sites is supported.
|
||||
- Last-write-wins for concurrent template editing (no optimistic concurrency on templates).
|
||||
@@ -152,7 +154,7 @@ spec for each is `docs/requirements/Component-<Name>.md`, and `README.md` carrie
|
||||
- One row per lifecycle event; cached calls produce 4+ rows per operation (`Submitted`, `Forwarded`, `Attempted`, `Delivered`/`Parked`/`Discarded`).
|
||||
- `ExecutionId` (`uniqueidentifier NULL`) is the universal per-run correlation value — every audit row emitted by one script execution / inbound request shares it; `CorrelationId` remains the per-operation lifecycle id (NULL for sync one-shots).
|
||||
- `ParentExecutionId` (`uniqueidentifier NULL`) is the cross-execution spawn pointer — every row of a spawned run carries the spawner's `ExecutionId`; bridges inbound API → routed-site-script, alarm-triggered on-trigger scripts, and nested `CallScript`/`CallShared` invocations; `IX_AuditLog_ParentExecution` backs the filter + the recursive execution-tree walk. **Tag-cascade (alarm leg) is populated, not just plumbed** — M5.4 T4 threaded the `parentExecutionId` parameter but every `AlarmActor.SpawnAlarmExecution` call site passed null, so alarm runs were silently always roots; the id now rides site-locally as `SetStaticAttributeCommand.SourceExecutionId` → `AttributeValueChanged.SourceExecutionId` → `SpawnAlarmExecution` (all additive/nullable, no wire/proto/schema change; `Expression` triggers capture the writer *with* the evaluated snapshot since the eval completes off-dispatcher). Sources are `ScriptRuntimeContext.SetAttribute` (the run's own `ExecutionId`) and `Route.To(...).SetAttributes(...)` (the inbound request's). **Still roots by design, not omission:** alarms fired by DCL data (external values have no spawning execution — including the device echo of a script write to a *data-sourced* attribute, so only **static** writes cascade), and `ScriptActor` value-change/conditional/expression/timer trigger runs (a timer tick has no spawner; a `WhileTrue`/interval run has no single identifiable write).
|
||||
- Site SQLite hot-path first, then gRPC telemetry to central; ingest is idempotent on `EventId`; periodic reconciliation pull as fallback when telemetry is lost.
|
||||
- Site SQLite hot-path first, then gRPC telemetry to central; ingest is idempotent on `EventId`; periodic reconciliation pull as fallback when telemetry is lost. **The site audit DB is a separate, unreplicated SQLite file — NOT one of LocalDb's ten consolidated tables** — at the **required** `AuditLog:SiteWriter:DatabasePath` (arch-review WP1.2, finding #2 High), `/app/data/auditlog.db` on the rig; `StartupValidator` refuses to boot a Site node without it. `DatabasePath` used to default to CWD-relative `auditlog.db`, which on the docker rig landed on the container's ephemeral overlay, not the mounted volume — the same data-loss pattern the pre-LocalDb legacy S&F/tracking databases hit (see the LocalDb bullet above), independently rediscovered here because this file sits outside LocalDb's management. The writer loop now also honors `FlushIntervalMs` (was validated but never read — one fsync'ing commit per event even at trickle rate) and sets `PRAGMA synchronous = NORMAL` (audit is best-effort by design, so the narrower power-loss window is an acceptable trade for far fewer fsyncs; WAL still guarantees no corruption).
|
||||
- Cached operations: site emits a single additively-extended `CachedCallTelemetry` packet carrying both audit events and operational state; central writes `AuditLog` + `SiteCalls` in one transaction.
|
||||
- Payload cap 8 KB by default / 64 KB on error rows; auth headers redacted by default; SQL parameter values captured by default; per-target redaction opt-in. Inbound API: full verbatim capture up to `InboundMaxBytes` (default 1 MiB); request headers stored in `Extra.requestHeaders` (post-redaction); per-method `SkipBodyCapture` flag suppresses bodies while still recording headers + metadata; `AuditInboundCeilingHits` counter surfaced on health snapshot. (M5.3 T7)
|
||||
- Audit-write failure NEVER aborts the user-facing action — audit is best-effort, the action's own success/failure path is authoritative.
|
||||
@@ -213,6 +215,7 @@ spec for each is `docs/requirements/Component-<Name>.md`, and `README.md` carrie
|
||||
## Tool Usage
|
||||
|
||||
- When a task requires setting up or controlling system state (sites, templates, instances, data connections, deployments, security, etc.) and the Central UI is not needed, prefer the ScadaBridge CLI over manual DB edits or UI navigation. See [`src/ZB.MOM.WW.ScadaBridge.CLI/README.md`](src/ZB.MOM.WW.ScadaBridge.CLI/README.md) for the full command reference.
|
||||
- **CLI HTTP timeouts are honest (arch-review remediation, finding #1 High).** `ManagementHttpClient` used to cap every call at `min(30s, caller timeout)`, silently truncating `deploy site`'s 5-minute `BulkDeployTimeout` and the 5-minute bundle export/preview/import calls — the CLI printed a fake `504 Request timed out` while the server kept working to completion. `HttpClient.Timeout` is now `Timeout.InfiniteTimeSpan`; the per-call `CancellationTokenSource` is the single overall deadline (connect phase included), with connect bounded separately via `SocketsHttpHandler.ConnectTimeout` (env override renamed `SCADABRIDGE_HTTP_CONNECT_TIMEOUT_SECONDS` to match its narrower new meaning).
|
||||
|
||||
### CLI Quick Reference (Docker / OrbStack)
|
||||
|
||||
|
||||
@@ -109,9 +109,9 @@
|
||||
<PackageVersion Include="ZB.MOM.WW.Secrets.Ui" Version="0.6.2" />
|
||||
<PackageVersion Include="ZB.MOM.WW.Secrets.Replicator.SqlServer" Version="0.6.2" />
|
||||
<PackageVersion Include="ZB.MOM.WW.Secrets.Replicator.Grpc" Version="0.6.2" />
|
||||
<PackageVersion Include="ZB.MOM.WW.LocalDb" Version="0.1.3" />
|
||||
<PackageVersion Include="ZB.MOM.WW.LocalDb.Replication" Version="0.1.3" />
|
||||
<PackageVersion Include="ZB.MOM.WW.LocalDb.Contracts" Version="0.1.3" />
|
||||
<PackageVersion Include="ZB.MOM.WW.LocalDb" Version="0.2.1" />
|
||||
<PackageVersion Include="ZB.MOM.WW.LocalDb.Replication" Version="0.2.1" />
|
||||
<PackageVersion Include="ZB.MOM.WW.LocalDb.Contracts" Version="0.2.1" />
|
||||
</ItemGroup>
|
||||
|
||||
<!--
|
||||
|
||||
@@ -78,5 +78,16 @@
|
||||
// Replication is opt-in and configured separately; absent = local-only.
|
||||
"LocalDb": {
|
||||
"Path": "/app/data/site-localdb.db"
|
||||
},
|
||||
// arch-review remediation WP1.2: the site hot-path audit writer's SQLite file has no
|
||||
// default path (mirrors LocalDb:Path above) and StartupValidator now requires it for
|
||||
// Site nodes — an unset value used to fall back to a bare "auditlog.db" resolved
|
||||
// relative to CWD, i.e. the container's ephemeral overlayfs, so the pending audit
|
||||
// backlog was silently discarded on every recreate. On the mounted /app/data volume,
|
||||
// same as LocalDb:Path, so it survives container recreate.
|
||||
"AuditLog": {
|
||||
"SiteWriter": {
|
||||
"DatabasePath": "/app/data/auditlog.db"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -78,5 +78,16 @@
|
||||
// Replication is opt-in and configured separately; absent = local-only.
|
||||
"LocalDb": {
|
||||
"Path": "/app/data/site-localdb.db"
|
||||
},
|
||||
// arch-review remediation WP1.2: the site hot-path audit writer's SQLite file has no
|
||||
// default path (mirrors LocalDb:Path above) and StartupValidator now requires it for
|
||||
// Site nodes — an unset value used to fall back to a bare "auditlog.db" resolved
|
||||
// relative to CWD, i.e. the container's ephemeral overlayfs, so the pending audit
|
||||
// backlog was silently discarded on every recreate. On the mounted /app/data volume,
|
||||
// same as LocalDb:Path, so it survives container recreate.
|
||||
"AuditLog": {
|
||||
"SiteWriter": {
|
||||
"DatabasePath": "/app/data/auditlog.db"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -104,12 +104,19 @@
|
||||
"ApiKey": "dev-site-a-localdb-sync-key",
|
||||
// ---- Phase 2 sizing, from the Task 1 rig soak (not from the defaults) ----
|
||||
//
|
||||
// MaxBatchSize (default 500) is a ROW count, not a byte budget, so the batch
|
||||
// size in bytes is set by the widest replicated column. That is
|
||||
// deployed_configurations.config_json: ~721 B on this rig, but up to ~60-70 KB
|
||||
// in production (measured, Task 1) - and 70 KB x 500 is ~35 MB against gRPC's
|
||||
// 4 MB default receive limit. 16 keeps a worst-case batch near 1.1 MB.
|
||||
"MaxBatchSize": 16,
|
||||
// The old MaxBatchSize = 16 pin (a hand-computed byte-budget proxy) was retired
|
||||
// when LocalDb 0.2.0 added MaxBatchBytes (arch-review WP3.3) - but MaxBatchSize
|
||||
// is NOT purely redundant with it. MaxBatchBytes bounds the WIRE message via a
|
||||
// per-message split in SyncSession.PumpLoopAsync; MaxBatchSize separately bounds
|
||||
// the DB READ page in OplogStore.ReadBatchAboveAsync / SnapshotStreamer, which
|
||||
// materializes the whole page into memory BEFORE the byte-budget split runs. At
|
||||
// the unset 500 default, a reconnect drain of 70 KB worst-case config_json rows
|
||||
// can transiently allocate ~35 MB per read even though every wire message stays
|
||||
// under the 2 MB MaxBatchBytes default (arch-review adversarial finding F2). Pin
|
||||
// it explicitly here to bound that transient allocation; MaxBatchBytes is left
|
||||
// unset (its 2 MB default) to bound the wire message.
|
||||
"MaxBatchSize": 64,
|
||||
//
|
||||
// Backlog caps bound the oplog while the peer is offline. Exceeding them is
|
||||
// NOT data loss: the oplog is pruned to the ceiling and needs_snapshot is set,
|
||||
// so the peer catches up by snapshot resync instead of incrementally. That
|
||||
@@ -120,5 +127,16 @@
|
||||
"MaxOplogRows": 250000,
|
||||
"MaxOplogAge": "2.00:00:00"
|
||||
}
|
||||
},
|
||||
// arch-review remediation WP1.2: the site hot-path audit writer's SQLite file has no
|
||||
// default path (mirrors LocalDb:Path above) and StartupValidator now requires it for
|
||||
// Site nodes — an unset value used to fall back to a bare "auditlog.db" resolved
|
||||
// relative to CWD, i.e. the container's ephemeral overlayfs, so the pending audit
|
||||
// backlog was silently discarded on every recreate. On the mounted /app/data volume,
|
||||
// same as LocalDb:Path, so it survives container recreate.
|
||||
"AuditLog": {
|
||||
"SiteWriter": {
|
||||
"DatabasePath": "/app/data/auditlog.db"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -97,12 +97,19 @@
|
||||
"ApiKey": "dev-site-a-localdb-sync-key",
|
||||
// ---- Phase 2 sizing, from the Task 1 rig soak (not from the defaults) ----
|
||||
//
|
||||
// MaxBatchSize (default 500) is a ROW count, not a byte budget, so the batch
|
||||
// size in bytes is set by the widest replicated column. That is
|
||||
// deployed_configurations.config_json: ~721 B on this rig, but up to ~60-70 KB
|
||||
// in production (measured, Task 1) - and 70 KB x 500 is ~35 MB against gRPC's
|
||||
// 4 MB default receive limit. 16 keeps a worst-case batch near 1.1 MB.
|
||||
"MaxBatchSize": 16,
|
||||
// The old MaxBatchSize = 16 pin (a hand-computed byte-budget proxy) was retired
|
||||
// when LocalDb 0.2.0 added MaxBatchBytes (arch-review WP3.3) - but MaxBatchSize
|
||||
// is NOT purely redundant with it. MaxBatchBytes bounds the WIRE message via a
|
||||
// per-message split in SyncSession.PumpLoopAsync; MaxBatchSize separately bounds
|
||||
// the DB READ page in OplogStore.ReadBatchAboveAsync / SnapshotStreamer, which
|
||||
// materializes the whole page into memory BEFORE the byte-budget split runs. At
|
||||
// the unset 500 default, a reconnect drain of 70 KB worst-case config_json rows
|
||||
// can transiently allocate ~35 MB per read even though every wire message stays
|
||||
// under the 2 MB MaxBatchBytes default (arch-review adversarial finding F2). Pin
|
||||
// it explicitly here to bound that transient allocation; MaxBatchBytes is left
|
||||
// unset (its 2 MB default) to bound the wire message.
|
||||
"MaxBatchSize": 64,
|
||||
//
|
||||
// Backlog caps bound the oplog while the peer is offline. Exceeding them is
|
||||
// NOT data loss: the oplog is pruned to the ceiling and needs_snapshot is set,
|
||||
// so the peer catches up by snapshot resync instead of incrementally. That
|
||||
@@ -113,5 +120,16 @@
|
||||
"MaxOplogRows": 250000,
|
||||
"MaxOplogAge": "2.00:00:00"
|
||||
}
|
||||
},
|
||||
// arch-review remediation WP1.2: the site hot-path audit writer's SQLite file has no
|
||||
// default path (mirrors LocalDb:Path above) and StartupValidator now requires it for
|
||||
// Site nodes — an unset value used to fall back to a bare "auditlog.db" resolved
|
||||
// relative to CWD, i.e. the container's ephemeral overlayfs, so the pending audit
|
||||
// backlog was silently discarded on every recreate. On the mounted /app/data volume,
|
||||
// same as LocalDb:Path, so it survives container recreate.
|
||||
"AuditLog": {
|
||||
"SiteWriter": {
|
||||
"DatabasePath": "/app/data/auditlog.db"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -86,5 +86,16 @@
|
||||
// Replication is opt-in and configured separately; absent = local-only.
|
||||
"LocalDb": {
|
||||
"Path": "/app/data/site-localdb.db"
|
||||
},
|
||||
// arch-review remediation WP1.2: the site hot-path audit writer's SQLite file has no
|
||||
// default path (mirrors LocalDb:Path above) and StartupValidator now requires it for
|
||||
// Site nodes — an unset value used to fall back to a bare "auditlog.db" resolved
|
||||
// relative to CWD, i.e. the container's ephemeral overlayfs, so the pending audit
|
||||
// backlog was silently discarded on every recreate. On the mounted /app/data volume,
|
||||
// same as LocalDb:Path, so it survives container recreate.
|
||||
"AuditLog": {
|
||||
"SiteWriter": {
|
||||
"DatabasePath": "/app/data/auditlog.db"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -86,5 +86,16 @@
|
||||
// Replication is opt-in and configured separately; absent = local-only.
|
||||
"LocalDb": {
|
||||
"Path": "/app/data/site-localdb.db"
|
||||
},
|
||||
// arch-review remediation WP1.2: the site hot-path audit writer's SQLite file has no
|
||||
// default path (mirrors LocalDb:Path above) and StartupValidator now requires it for
|
||||
// Site nodes — an unset value used to fall back to a bare "auditlog.db" resolved
|
||||
// relative to CWD, i.e. the container's ephemeral overlayfs, so the pending audit
|
||||
// backlog was silently discarded on every recreate. On the mounted /app/data volume,
|
||||
// same as LocalDb:Path, so it survives container recreate.
|
||||
"AuditLog": {
|
||||
"SiteWriter": {
|
||||
"DatabasePath": "/app/data/auditlog.db"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -86,5 +86,16 @@
|
||||
// Replication is opt-in and configured separately; absent = local-only.
|
||||
"LocalDb": {
|
||||
"Path": "/app/data/site-localdb.db"
|
||||
},
|
||||
// arch-review remediation WP1.2: the site hot-path audit writer's SQLite file has no
|
||||
// default path (mirrors LocalDb:Path above) and StartupValidator now requires it for
|
||||
// Site nodes — an unset value used to fall back to a bare "auditlog.db" resolved
|
||||
// relative to CWD, i.e. the container's ephemeral overlayfs, so the pending audit
|
||||
// backlog was silently discarded on every recreate. On the mounted /app/data volume,
|
||||
// same as LocalDb:Path, so it survives container recreate.
|
||||
"AuditLog": {
|
||||
"SiteWriter": {
|
||||
"DatabasePath": "/app/data/auditlog.db"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -86,5 +86,16 @@
|
||||
// Replication is opt-in and configured separately; absent = local-only.
|
||||
"LocalDb": {
|
||||
"Path": "/app/data/site-localdb.db"
|
||||
},
|
||||
// arch-review remediation WP1.2: the site hot-path audit writer's SQLite file has no
|
||||
// default path (mirrors LocalDb:Path above) and StartupValidator now requires it for
|
||||
// Site nodes — an unset value used to fall back to a bare "auditlog.db" resolved
|
||||
// relative to CWD, i.e. the container's ephemeral overlayfs, so the pending audit
|
||||
// backlog was silently discarded on every recreate. On the mounted /app/data volume,
|
||||
// same as LocalDb:Path, so it survives container recreate.
|
||||
"AuditLog": {
|
||||
"SiteWriter": {
|
||||
"DatabasePath": "/app/data/auditlog.db"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -78,13 +78,15 @@ builder.Property(e => e.ExecutionId)
|
||||
"CAST(JSON_VALUE(DetailsJson,'$.executionId') AS uniqueidentifier)", stored: true)
|
||||
.ValueGeneratedOnAddOrUpdate();
|
||||
|
||||
// Composite PK includes OccurredAtUtc for partition alignment
|
||||
// Composite PK includes OccurredAtUtc for partition alignment — and is the ONLY
|
||||
// EventId uniqueness enforcement (WP2.2 / AlignAuditLogEventIdUniqueness). EventId
|
||||
// is a GUID minted once at the site alongside OccurredAtUtc, so a given EventId
|
||||
// always lands in one partition and pair-uniqueness is EventId-uniqueness.
|
||||
builder.HasKey(e => new { e.EventId, e.OccurredAtUtc });
|
||||
|
||||
builder.HasIndex(e => e.EventId).IsUnique()
|
||||
.HasDatabaseName("UX_AuditLog_EventId");
|
||||
```
|
||||
|
||||
The predecessor `UX_AuditLog_EventId` — a single-column unique index on `[PRIMARY]`, deliberately NOT partition-aligned — was dropped: a non-aligned index blocks `ALTER TABLE … SWITCH PARTITION`, so the retention purge had to drop it, switch, and rebuild it offline inside the switch transaction on every run.
|
||||
|
||||
**`TemplateConfiguration`** (representative of the domain-area configs) sets up the self-referencing parent FK, folder FK, cascade-delete relationships to attributes/alarms/scripts/compositions/native alarm sources, and the filtered unique index that enforces name uniqueness only on non-derived (base) templates.
|
||||
|
||||
**`SiteCallEntityTypeConfiguration`** maps `SiteCall` to `dbo.SiteCalls` with a `TrackedOperationId` PK stored as `varchar(36)` (GUID in `"D"` format) so the column shape matches the wire format and the site SQLite store — one consistent format for operational debugging.
|
||||
@@ -108,11 +110,13 @@ VALUES
|
||||
|
||||
`FormattableString` interpolation parameterises every value so there is no injection surface. SQL error numbers `2601` and `2627` (unique-index violation) are swallowed as no-ops because the IF NOT EXISTS check has a race window; both the check-loser and the retrying telemetry path are semantically correct duplicates.
|
||||
|
||||
`InsertManyIfNotExistsAsync` is the set-based form the ingest hot path uses: one `INSERT … SELECT … FROM (VALUES …) WHERE NOT EXISTS` per chunk of 100 rows (ten bound parameters per row against SQL Server's 2,100-parameter ceiling), so a telemetry packet costs one round trip instead of one per event. Because the anti-semi-join only sees committed rows, the packet is de-duplicated by `EventId` in C# first (first-write-wins, matching the single-row contract); a duplicate-key fault from a concurrent writer falls back to the per-row path so the batch is never a correctness dependency. It uses raw ADO.NET with explicitly typed `SqlParameter`s — the `VALUES` constructor derives its column types from the first row's parameters, so an untyped null would give the derived column the wrong type — and enlists in the DbContext's ambient transaction when one is open (the cached-telemetry dual-write).
|
||||
|
||||
`QueryAsync` builds LINQ predicates over `AuditLogRow` using `AsNoTracking()`, translating filter dimensions (`Channels`, `Kinds`, `Statuses`, `SourceSiteIds`, `SourceNodes`, `ExecutionId`, `ParentExecutionId`, time range) to server-side SQL IN/equality predicates and using keyset pagination on `(OccurredAtUtc DESC, EventId DESC)`.
|
||||
|
||||
`GetExecutionTreeAsync` walks the `ParentExecutionId` graph in two phases: a loop climbs to the root (bounded at 32 levels), then a recursive CTE descends the full tree and LEFT JOINs back to `AuditLog` so stub nodes (purged or row-less executions) still appear with `RowCount = 0`.
|
||||
|
||||
`SwitchOutPartitionAsync` executes a drop-and-rebuild dance — dropping `UX_AuditLog_EventId`, creating a byte-identical staging table (including the computed-column definitions), switching the target partition to staging, dropping staging, and rebuilding the unique index — all inside a single `BEGIN TRY / BEGIN CATCH` block that guarantees the index is present whether the switch succeeds or rolls back.
|
||||
`SwitchOutPartitionAsync` creates a byte-identical staging table (including the computed-column definitions), switches the target partition to staging, and drops staging, inside a single `BEGIN TRY / BEGIN CATCH` block whose CATCH cleans up the staging table on any failure. No index is dropped or rebuilt: uniqueness rides the partition-aligned clustered PK, so the switch is metadata-only. A guarded defensive `DROP INDEX UX_AuditLog_EventId` remains at the head of the batch purely so a database restored from a pre-alignment backup still purges.
|
||||
|
||||
### IAuditService — config-change audit
|
||||
|
||||
@@ -241,7 +245,7 @@ The host is running in production mode and `GetPendingMigrationsAsync` found una
|
||||
|
||||
### AuditLog partition switch fails mid-operation
|
||||
|
||||
`SwitchOutPartitionAsync` wraps the drop-and-rebuild dance in `BEGIN TRY / BEGIN CATCH`. On failure the CATCH block drops the staging table if it exists and rebuilds `UX_AuditLog_EventId` if it was dropped before the failure. The original exception is re-thrown so the Audit Log purge actor logs it and retries on the next daily tick. Verify that the `scadabridge_audit_purger` role still holds `ALTER ON SCHEMA::dbo` if the operation fails with a permissions error.
|
||||
`SwitchOutPartitionAsync` wraps the staging/switch batch in `BEGIN TRY / BEGIN CATCH`. On failure the CATCH block drops the staging table if it exists, so no orphaned `AuditLog_Staging_*` object is left behind. There is no index to repair — uniqueness lives on the clustered PK, which the switch never touches. The original exception is re-thrown so the Audit Log purge actor logs it and retries on the next daily tick. Verify that the `scadabridge_audit_purger` role still holds `ALTER ON SCHEMA::dbo` if the operation fails with a permissions error.
|
||||
|
||||
### Design-time `dotnet ef` tooling cannot find a connection string
|
||||
|
||||
|
||||
@@ -241,7 +241,7 @@ The inbound body-capture cap for audit is configured separately under `AuditLog:
|
||||
|
||||
- [Commons (#16)](./Commons.md) — owns `ApiMethod`, `ParameterDefinition`, `ScriptParameters`, `ScriptParameterException`, the `RouteToCall*` / `RouteToGetAttributes*` / `RouteToSetAttributes*` message records, `IInboundApiRepository`, and `IInstanceLocator`. Also owns `ICentralAuditWriter` (via `ZB.MOM.WW.Audit`), `AuditChannel`, `AuditKind`, `AuditStatus`, and `ScadaBridgeAuditEventFactory`.
|
||||
- [Configuration Database (#17)](./ConfigurationDatabase.md) — provides the `IInboundApiRepository` implementation (`GetMethodByNameAsync`, `GetAllApiMethodsAsync`, CRUD). Method definitions persist in the central MS SQL configuration database.
|
||||
- [Central–Site Communication (#5)](./Communication.md) — `CommunicationServiceInstanceRouter` delegates every `Route.To()` operation to `CommunicationService`. The routed call travels from the central `CentralCommunicationActor` to the target site via `ClusterClient`, reaches the target `InstanceActor`, and a `ScriptExecutionActor` executes the named script. The return value flows back synchronously.
|
||||
- [Central–Site Communication (#5)](./Communication.md) — `CommunicationServiceInstanceRouter` delegates every `Route.To()` operation to `CommunicationService`. The routed call travels from the central `CentralCommunicationActor` to the target site over gRPC (`CentralControlService`/`SiteCommandService` — `ClusterClient` was removed in the ClusterClient→gRPC migration), reaches the target `InstanceActor`, and the named script runs via `ScriptRunLauncher` (no per-run child actor since WP3.1). The return value flows back synchronously.
|
||||
- [Audit Log (#23)](./AuditLog.md) — `AuditWriteMiddleware` resolves `ICentralAuditWriter` to emit the `ApiInbound` row via the central direct-write path. The inbound request is the parent execution for any site script it spawns: the middleware's `ExecutionId` becomes `RouteToCallRequest.ParentExecutionId` on every routed `Call`. Cross-link: `AuditWriteMiddleware.InboundExecutionIdItemKey` / `AuditWriteMiddleware.AuditActorItemKey` are the `HttpContext.Items` keys that tie the endpoint handler and middleware together.
|
||||
- [Security (#10)](./Security.md) — API key verification (`IApiKeyVerifier`, `AddZbApiKeyAuth`) is registered by the Host. The inbound API uses a dedicated key scheme independent of LDAP/AD session auth.
|
||||
- [Cluster Infrastructure (#13)](./ClusterInfrastructure.md) — `IActiveNodeGate` (interface in this project; implementation in the Host) gates the endpoint to the active central node. A standby returns `503` without running any script logic.
|
||||
|
||||
@@ -157,7 +157,7 @@ public class EventLogHandlerActor : ReceiveActor
|
||||
Callers resolve `ISiteEventLogger` from DI. Because the write is non-blocking and best-effort, site actors discard the returned `Task` with `_ =` rather than awaiting it on the hot path:
|
||||
|
||||
```csharp
|
||||
// ScriptExecutionActor — reporting a script failure
|
||||
// ScriptRunLauncher (via ScriptActor) — reporting a script failure
|
||||
_ = siteEventLogger?.LogEventAsync(
|
||||
"script", "Error", instanceName, $"ScriptActor:{scriptName}", errorMsg, ex.ToString());
|
||||
|
||||
@@ -196,7 +196,7 @@ The docker cluster appsettings (`ScadaBridge:SiteEventLog`) sets `RetentionDays:
|
||||
|
||||
- [Commons (#16)](./Commons.md) — defines the `EventLogQueryRequest` / `EventLogQueryResponse` / `EventLogEntry` message contracts in `ZB.MOM.WW.ScadaBridge.Commons.Messages.RemoteQuery`, shared across the site query path and the central dispatch path (`QueryEventLogsCommand`).
|
||||
- [Central–Site Communication (#5)](./Communication.md) — the `SiteCommunicationActor` dispatches inbound `EventLogQueryRequest` messages to `EventLogHandlerActor` and carries the `EventLogQueryResponse` back to central. The query timeout is 30 s.
|
||||
- [Site Runtime (#3)](./SiteRuntime.md) — `ScriptActor` and `ScriptExecutionActor` log `script`-type events: trigger expression failures, script execution errors, and timeouts. `ISiteEventLogger` is resolved from DI inside execution actors.
|
||||
- [Site Runtime (#3)](./SiteRuntime.md) — `ScriptActor` and `AlarmActor` log `script`-type events (via `ScriptRunLauncher`, no per-run child actor since WP3.1): trigger expression failures, script execution errors, and timeouts. `ISiteEventLogger` is resolved from DI for the run.
|
||||
- [Data Connection Layer (#4)](./DataConnectionLayer.md) — `DataConnectionActor` logs `connection`-type events: connection loss, reconnection, and endpoint failover. `DataConnectionManagerActor` may also log connection-category events.
|
||||
- [Store-and-Forward Engine (#6)](./StoreAndForward.md) — logs `store_and_forward`-type events on the site→central notification forward path (forward failures, long-buffered notifications). Routine enqueue and forward-success events are not logged; central's `Notifications` table is the authoritative record.
|
||||
- [Host (#15)](./Host.md) — `SiteServiceRegistration` calls `AddSiteEventLogging` and binds `SiteEventLogOptions`. `AkkaHostedService` wires `EventLogHandlerActor` as a cluster singleton scoped to `"site-{SiteId}"`. The `SiteEventLogActiveNodeCheck` delegate is an optional seam defined in `SiteEventLogging` for the Host to register when it wants to gate the purge to the active node only; the Host does not currently register it, so the purge defaults to always-active and runs on every node.
|
||||
|
||||
@@ -4,12 +4,12 @@ The Site Runtime component runs the site-side actor hierarchy that executes depl
|
||||
|
||||
## Overview
|
||||
|
||||
Site Runtime (#3) operates exclusively on site clusters. Its entry point is the `DeploymentManagerActor` cluster singleton, which re-creates the full actor hierarchy on every site startup or failover. Each deployed enabled instance gets an `InstanceActor` child; each `InstanceActor` spawns `ScriptActor` and `AlarmActor` coordinator children, plus a `NativeAlarmActor` peer for every configured native alarm source. Script invocations spawn short-lived `ScriptExecutionActor` children; alarm on-trigger invocations spawn short-lived `AlarmExecutionActor` children.
|
||||
Site Runtime (#3) operates exclusively on site clusters. Its entry point is the `DeploymentManagerActor` cluster singleton, which re-creates the full actor hierarchy on every site startup or failover. Each deployed enabled instance gets an `InstanceActor` child; each `InstanceActor` spawns `ScriptActor` and `AlarmActor` coordinator children, plus a `NativeAlarmActor` peer for every configured native alarm source. **As of WP3.1 (arch-review remediation, 2026-08-15) a script or alarm on-trigger invocation is no longer a per-run child actor** — `ScriptExecutionActor`/`AlarmExecutionActor` were deleted (they were already inert shells: no `Receive` handler, no `PostStop`, never a message target). `ScriptActor`/`AlarmActor` instead launch the run body directly onto the shared script-execution pool via `ScriptRunLauncher`, keyed by instance+script so `MaxConcurrentRunsPerScript` and the deadline-at-enqueue timeout still apply per script.
|
||||
|
||||
The component code lives in `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/`:
|
||||
|
||||
- `Actors/` — `DeploymentManagerActor`, `InstanceActor`, `ScriptActor`, `ScriptExecutionActor`, `AlarmActor`, `AlarmExecutionActor`, `NativeAlarmActor`.
|
||||
- `Scripts/` — `ScriptCompilationService`, `ScriptExecutionScheduler`, `SharedScriptLibrary`, `ScriptRuntimeContext`, `ScopeAccessors`, `TriggerExpressionGlobals`.
|
||||
- `Actors/` — `DeploymentManagerActor`, `InstanceActor`, `ScriptActor`, `AlarmActor`, `NativeAlarmActor`. (`ScriptExecutionActor`/`AlarmExecutionActor` were removed by WP3.1 — see Overview.)
|
||||
- `Scripts/` — `ScriptCompilationService`, `ScriptExecutionScheduler`, `ScriptRunLauncher`, `TriggerEvalGate`, `ScriptRunSummaryRecorder`, `SharedScriptLibrary`, `ScriptRuntimeContext`, `ScopeAccessors`, `TriggerExpressionGlobals`.
|
||||
- `Streaming/` — `SiteStreamManager` (the site-wide Akka broadcast stream).
|
||||
- `Persistence/` — `SiteStorageService` (raw SQLite via `Microsoft.Data.Sqlite`), `SiteStorageInitializer`.
|
||||
- `Repositories/` — `SiteExternalSystemRepository`. (The `SiteNotificationRepository` variant was removed 2026-07-10, arch-review 08 §1.3/#23, because notification config is central-only and never lives on a site.)
|
||||
@@ -36,14 +36,12 @@ The singleton reads all deployed configurations from SQLite in `PreStart`, compi
|
||||
| `ScriptActor` | long-lived coordinator | `OneForOneStrategy` | Stop execution child, keep self |
|
||||
| `AlarmActor` | long-lived coordinator | `OneForOneStrategy` | Stop execution child, keep self |
|
||||
| `NativeAlarmActor` | long-lived coordinator | — | Supervised by Instance Actor (Resume) |
|
||||
| `ScriptExecutionActor` | short-lived per invocation | — | Stops itself; parent logs failure |
|
||||
| `AlarmExecutionActor` | short-lived per invocation | — | Stops itself; parent logs failure |
|
||||
|
||||
Coordinator actors resume on exception because their in-memory state (trigger timers, last execution time, alarm level) must survive child crashes. Short-lived execution actors stop themselves on completion or exception — the coordinator remains available for the next trigger.
|
||||
Coordinator actors resume on exception because their in-memory state (trigger timers, last execution time, alarm level) must survive child crashes. **Since WP3.1 there is no per-run child actor** — `ScriptActor`/`AlarmActor` launch the run body directly via `ScriptRunLauncher` onto the shared pool and observe its completion `Task`; a run failure is caught and reported by the launcher/coordinator, not by an actor `PostStop`.
|
||||
|
||||
### Dedicated script-execution dispatcher
|
||||
|
||||
Script and alarm on-trigger bodies run on the `ScriptExecutionScheduler` (`SiteRuntime-009`): a custom `TaskScheduler` backed by a bounded set of dedicated threads (default 8, `ScriptExecutionThreadCount`). The script body is submitted to this scheduler via `Task.Factory.StartNew(..., scheduler)` inside `ScriptExecutionActor` and `AlarmExecutionActor`. Scripts that block on I/O (database connections, synchronous external system calls) block only the scheduler's threads, leaving the shared .NET thread pool and all Akka dispatchers unaffected.
|
||||
Script and alarm on-trigger bodies run on the `ScriptExecutionScheduler` (`SiteRuntime-009`): a custom `TaskScheduler` backed by a bounded, **instance-scaled, grow-only** set of dedicated threads (`clamp(max(ScriptExecutionThreadCount, ceil(enabledInstances/8)), 1, ScriptExecutionMaxThreadCount)`; floor default 8, ceiling default 32 — WP3.1). The script body is submitted to this scheduler via `Task.Factory.StartNew(..., scheduler)` from `ScriptRunLauncher`, called directly by `ScriptActor`/`AlarmActor` — no per-run actor is spawned. Scripts that block on I/O (database connections, synchronous external system calls) block only the scheduler's threads, leaving the shared .NET thread pool and all Akka dispatchers unaffected. **Expression-trigger evaluation runs on a separate path** — the shared .NET thread pool behind `TriggerEvalGate`, not this scheduler (WP3.1, arch-review finding #4) — so scripts blocked in synchronous I/O can no longer stall every Expression trigger on the node.
|
||||
|
||||
### Tell vs. Ask
|
||||
|
||||
@@ -52,7 +50,7 @@ Script and alarm on-trigger bodies run on the `ScriptExecutionScheduler` (`SiteR
|
||||
|
||||
### Attribute serialization through the Instance Actor
|
||||
|
||||
All in-memory state mutations (attribute values, qualities, alarm states) run inside `InstanceActor`'s mailbox. Multiple `ScriptExecutionActor` instances may run concurrently but all `SetAttribute` calls serialize through the `InstanceActor` mailbox, preventing race conditions. Concurrent script executions may interleave external side effects (HTTP calls, database writes, notifications); those are independent and intentionally not serialized.
|
||||
All in-memory state mutations (attribute values, qualities, alarm states) run inside `InstanceActor`'s mailbox. Multiple script/alarm runs may execute concurrently on the shared pool but all `SetAttribute` calls serialize through the `InstanceActor` mailbox, preventing race conditions. Concurrent script executions may interleave external side effects (HTTP calls, database writes, notifications); those are independent and intentionally not serialized.
|
||||
|
||||
## Architecture
|
||||
|
||||
@@ -61,16 +59,15 @@ All in-memory state mutations (attribute values, qualities, alarm states) run in
|
||||
```text
|
||||
DeploymentManagerActor (Akka.NET cluster singleton)
|
||||
└── InstanceActor "MachineA-001"
|
||||
├── ScriptActor "MonitorSpeed" (coordinator)
|
||||
│ └── ScriptExecutionActor (short-lived, per invocation)
|
||||
├── ScriptActor "MonitorSpeed" (coordinator — launches runs via ScriptRunLauncher, no per-run child)
|
||||
├── ScriptActor "CalculateOEE" (coordinator)
|
||||
│ └── ScriptExecutionActor (short-lived)
|
||||
├── AlarmActor "OverTemp" (coordinator, computed)
|
||||
│ └── AlarmExecutionActor (short-lived, on-trigger)
|
||||
├── AlarmActor "OverTemp" (coordinator, computed — on-trigger runs also via ScriptRunLauncher)
|
||||
├── AlarmActor "LowPressure" (coordinator, computed)
|
||||
└── NativeAlarmActor "OpcUaServer1" (read-only mirror, peer to AlarmActor)
|
||||
```
|
||||
|
||||
Prior to WP3.1 (2026-08-15) each invocation spawned a short-lived `ScriptExecutionActor`/`AlarmExecutionActor` child; those actor classes were deleted (see Overview) and every run now executes on the shared `ScriptExecutionScheduler` pool directly, tracked by `ScriptRunLauncher` rather than by an actor cell.
|
||||
|
||||
`NativeAlarmActor` is a sibling of `AlarmActor` — a peer under the same `InstanceActor` parent. It is not a child of `AlarmActor` and has no relationship to the script engine.
|
||||
|
||||
### Deployment flow
|
||||
@@ -234,17 +231,17 @@ Central sends commands to the site `DeploymentManagerActor` singleton over the C
|
||||
|
||||
### Script API surface
|
||||
|
||||
Scripts run inside `ScriptExecutionActor` with a `ScriptGlobals` object as the Roslyn host object. The `Instance` global is a `ScriptRuntimeContext`. Convenience top-level aliases (`ExternalSystem`, `Database`, `Notify`, `Scripts`, `Attributes`, `Children`, `Parent`) delegate to context methods. Key calls:
|
||||
Scripts run on the shared script-execution pool (via `ScriptRunLauncher`, launched directly by `ScriptActor` — no per-run child actor since WP3.1) with a `ScriptGlobals` object as the Roslyn host object. The `Instance` global is a `ScriptRuntimeContext`. Convenience top-level aliases (`ExternalSystem`, `Database`, `Notify`, `Scripts`, `Attributes`, `Children`, `Parent`) delegate to context methods. Key calls:
|
||||
|
||||
- `Instance.GetAttribute("name")` / `Instance.SetAttribute("name", value)` — Ask to `InstanceActor` for write, in-process for read.
|
||||
- `Instance.CallScript("scriptName", params)` — Ask from `ScriptExecutionActor` to sibling `ScriptActor`, which spawns a new `ScriptExecutionActor`.
|
||||
- `Instance.CallScript("scriptName", params)` — Ask from the running script (via `ScriptRuntimeContext.CallScript`) to sibling `ScriptActor`, which launches a new run via `ScriptRunLauncher`. Nested self-recursion is exempt from `MaxConcurrentRunsPerScript` (bounded instead by `MaxScriptCallDepth`), since the calling run is already holding one of the cap's slots while it awaits the callee.
|
||||
- `Scripts.CallShared("name", params)` — `SharedScriptLibrary.ExecuteAsync`, inline on the current scheduler thread.
|
||||
- `ExternalSystem.Call(...)` — synchronous HTTP call through `IExternalSystemClient`.
|
||||
- `ExternalSystem.CachedCall(...)` / `Database.CachedWrite(...)` — store-and-forwarded; returns a `TrackedOperationId`.
|
||||
- `Tracking.Status(id)` — reads the site-local `OperationTrackingStore` synchronously.
|
||||
- `Notify.To("list").Send(...)` — enqueues a notification in the Store-and-Forward Engine for delivery to central.
|
||||
|
||||
Alarm on-trigger scripts run in `AlarmExecutionActor` with a **restricted** context: they receive an `Alarm` global (`AlarmContext` carrying `Name`, `Level`, `Priority`, `Message`) and have access to the instance/shared-script surface (`Instance.*`, `Scripts.CallShared`, `Instance.CallScript`), but **not** the external-system, database, notification, or audit integration APIs. `AlarmExecutionActor` builds its `ScriptRuntimeContext` without a `serviceProvider`, so `ExternalSystem`, `Database`, `Notify`, and audit writes are unavailable to alarm on-trigger scripts — those APIs are only resolved inside `ScriptExecutionActor` (instance scripts).
|
||||
Alarm on-trigger scripts run via the same `ScriptRunLauncher` path (launched by `AlarmActor`) with a **restricted** context: they receive an `Alarm` global (`AlarmContext` carrying `Name`, `Level`, `Priority`, `Message`) and have access to the instance/shared-script surface (`Instance.*`, `Scripts.CallShared`, `Instance.CallScript`), but **not** the external-system, database, notification, or audit integration APIs. The alarm run's `ScriptRuntimeContext` is built without a `serviceProvider`, so `ExternalSystem`, `Database`, `Notify`, and audit writes are unavailable to alarm on-trigger scripts — those APIs are only resolved for instance-script runs.
|
||||
|
||||
### Debug view
|
||||
|
||||
@@ -261,7 +258,9 @@ All options live in the `ScadaBridge:SiteRuntime` section, bound to `SiteRuntime
|
||||
| `MaxScriptCallDepth` | `10` | Maximum `Instance.CallScript` / `Scripts.CallShared` recursion depth |
|
||||
| `ScriptExecutionTimeoutSeconds` | `30` | Per-script body execution timeout; exceeding it cancels and logs an error |
|
||||
| `StreamBufferSize` | `1000` | Per-subscriber drop-oldest buffer size for the Akka broadcast stream |
|
||||
| `ScriptExecutionThreadCount` | `8` | Dedicated threads in the `ScriptExecutionScheduler` (covers both scripts and alarm on-trigger bodies) |
|
||||
| `ScriptExecutionThreadCount` | `8` | Floor thread count in the `ScriptExecutionScheduler` (covers both scripts and alarm on-trigger bodies) — the pool grows above this with `enabledInstances`, see Dedicated script-execution dispatcher |
|
||||
| `ScriptExecutionMaxThreadCount` | `32` | Ceiling on the instance-scaled pool growth (WP3.1) |
|
||||
| `MaxConcurrentRunsPerScript` | `4` | Runs in flight (queued or executing) per script/alarm on-trigger script before the newest is shed (WP3.1) |
|
||||
| `MirroredAlarmCapPerSource` | `1000` | Maximum mirrored conditions per `NativeAlarmActor` source binding before oldest is dropped and logged |
|
||||
| `NativeAlarmRetryIntervalMs` | `5000` | Milliseconds before retrying a failed native alarm subscription |
|
||||
|
||||
@@ -271,11 +270,11 @@ The SQLite connection string is passed directly to `AddSiteRuntime(connectionStr
|
||||
|
||||
- [Data Connection Layer (#4)](./DataConnectionLayer.md) — supplies `TagValueUpdate` and `ConnectionQualityChanged` messages to `InstanceActor`; receives `SubscribeTagsRequest` and `WriteTagRequest`. Also supplies `NativeAlarmTransitionUpdate` and `NativeAlarmSourceUnavailable` to `NativeAlarmActor` via `SubscribeAlarmsRequest` (connections implementing `IAlarmSubscribableConnection`).
|
||||
- [Central–Site Communication (#5)](./Communication.md) — routes `DeployInstanceCommand`, `DisableInstanceCommand`, `EnableInstanceCommand`, `DeleteInstanceCommand`, `DeployArtifactsCommand`, debug view requests, and Inbound API `RouteToCallRequest` / `RouteToGetAttributesRequest` / `RouteToSetAttributesRequest` to the singleton; receives `DeploymentStatusResponse` and `ArtifactDeploymentResponse` back. The `SiteStreamManager` implements `ISiteStreamSubscriber` so the Communication Layer's `SiteStreamGrpcServer` can subscribe `StreamRelayActor` instances to the broadcast hub.
|
||||
- [Store-and-Forward Engine (#6)](./StoreAndForward.md) — `ScriptRuntimeContext` passes `StoreAndForwardService` (resolved from DI inside `ScriptExecutionActor`) for `ExternalSystem.CachedCall`, `Database.CachedWrite`, and `Notify.To().Send()`. Owns the site-local operation tracking table that `Tracking.Status(id)` reads.
|
||||
- [Store-and-Forward Engine (#6)](./StoreAndForward.md) — `ScriptRuntimeContext` passes `StoreAndForwardService` (resolved from DI for the run, via `ScriptRunLauncher`) for `ExternalSystem.CachedCall`, `Database.CachedWrite`, and `Notify.To().Send()`. Owns the site-local operation tracking table that `Tracking.Status(id)` reads.
|
||||
- [External System Gateway (#7)](./ExternalSystemGateway.md) — `IExternalSystemClient` called by `ScriptRuntimeContext.ExternalSystemHelper` for synchronous and cached external system calls.
|
||||
- [Site Event Logging (#12)](./SiteEventLogging.md) — `ISiteEventLogger` (resolved from DI inside execution actors) receives script error, alarm error, and script execution events.
|
||||
- [Site Event Logging (#12)](./SiteEventLogging.md) — `ISiteEventLogger` (resolved from DI for the run) receives script error, alarm error, and script execution events.
|
||||
- [Health Monitoring (#11)](./HealthMonitoring.md) — `ISiteHealthCollector` (injected into `DeploymentManagerActor`, `InstanceActor`, `ScriptActor`, `AlarmActor`) tracks instance counts (`SetInstanceCounts`), script errors (`IncrementScriptError`), and alarm errors (`IncrementAlarmError`); sets `SetActiveNode` in `DeploymentManagerActor.PreStart`/`PostStop` so the health report reflects which node holds the singleton.
|
||||
- [Audit Log (#23)](./AuditLog.md) — `IAuditWriter` (resolved from DI inside `ScriptExecutionActor`) receives one row per script-trust-boundary call; audit writes are best-effort and never abort the calling script.
|
||||
- [Audit Log (#23)](./AuditLog.md) — `IAuditWriter` (resolved from DI for the run) receives one row per script-trust-boundary call; audit writes are best-effort and never abort the calling script.
|
||||
- [Commons (#16)](./Commons.md) — owns all message contracts (`DeployInstanceCommand`, `AttributeValueChanged`, `AlarmStateChanged`, `ScriptCallRequest`, `NativeAlarmTransitionUpdate`, etc.), the `FlattenedConfiguration` / `ResolvedScript` / `ResolvedAlarm` / `ResolvedNativeAlarmSource` types, and the `AlarmKind` / `AlarmState` / `AlarmLevel` / `AlarmConditionState` / `AlarmTransitionKind` enums.
|
||||
- Local SQLite — `SiteStorageService` owns the site database. Peer SQLite stores (Store-and-Forward buffer, AuditLog, operation tracking, site event log) are owned by their respective components but share the same SQLite file path convention.
|
||||
- Design spec: [Component-SiteRuntime.md](../requirements/Component-SiteRuntime.md).
|
||||
|
||||
@@ -157,6 +157,46 @@ Each site has its own two-node cluster:
|
||||
- SQLite persistence: each node owns its own consolidated LocalDb database, kept in step by
|
||||
asynchronous CDC replication over a gRPC sync stream (LocalDb Phase 1 + 2). The nodes do NOT
|
||||
share a SQLite file.
|
||||
- CDC capture triggers are installed **only on a node that has replication configured** —
|
||||
`LocalDb:Replication:PeerAddress` *or* `LocalDb:Replication:ApiKey`. Either key counts, because
|
||||
only the initiating half of a pair sets `PeerAddress` (one bidirectional stream, dialled by one
|
||||
side); the passive half carries the key alone. A deliberately unreplicated node — site-b and
|
||||
site-c on the rig — runs with no triggers at all and stops paying the per-write capture cost.
|
||||
- **Stale-trigger cleanup is automatic** (LocalDb 0.2.0). A node with no replication configured does
|
||||
not merely skip registration — at boot it calls `DeregisterReplicated` on all ten tables, dropping
|
||||
any capture triggers an earlier build installed and pruning those tables' oplog and row-version
|
||||
rows. It is idempotent, so a file that was never registered reports nothing to clean; when
|
||||
something *was* cleaned the node logs it once at Information. Recreating the data volume is no
|
||||
longer required to stop an in-place-upgraded node from capturing.
|
||||
|
||||
#### Turning replication ON for a site that has been running without it
|
||||
|
||||
**Supported as of LocalDb 0.2.0.** Set the keys on **both** nodes and restart both (see the
|
||||
stop-and-start-together rule below — this is a pair-wide change, not a rolling one). Existing rows
|
||||
are carried across:
|
||||
|
||||
- **Pre-existing rows are baselined automatically.** ScadaBridge registers every replicated table
|
||||
with `baselineExistingRows: true`. Capture is change-data-capture, so rows written while the node
|
||||
had no triggers appear in neither the oplog nor `__localdb_row_version` — and LocalDb's snapshot
|
||||
resync streams from that ledger. Baselining seeds the ledger for those rows at the LWW floor
|
||||
(HLC `0`, stamped with the node's own id) and flags a snapshot resync, so the peer actually
|
||||
receives them. Copying one node's database onto the other beforehand is no longer necessary.
|
||||
- **What the floor means for conflicts.** Every genuine HLC is a UTC millisecond shifted left 16
|
||||
bits, so it is strictly greater than `0`: a baselined row loses to any real remote write of the
|
||||
same key and wins only where the peer holds no version of that key at all. The one ambiguous case
|
||||
is **both** nodes baselining the same key (e.g. both were restored from the same legacy file) —
|
||||
both hold HLC `0` and the node-id tie-break decides. That is convergent but arbitrary as to which
|
||||
content survives, so if the two files may disagree on a key, start both nodes from one node's
|
||||
database.
|
||||
- **Seeding is idempotent** (`ON CONFLICT DO NOTHING`), so a row that already has a genuine version
|
||||
keeps it and no snapshot is flagged. Booting with baselining on every start is free after the
|
||||
first.
|
||||
|
||||
Turning replication back **OFF** is likewise a both-nodes change: deregistration must be symmetric,
|
||||
because the sync handshake compares the two nodes' registered-table digests fail-closed — a node
|
||||
that drops a table its peer still replicates stops syncing with a schema-mismatch error rather than
|
||||
diverging silently. Turning it on again later re-baselines, which is what makes the ledger prune on
|
||||
deregistration safe.
|
||||
|
||||
### Site Pair Upgrades — stop and start BOTH nodes together
|
||||
|
||||
|
||||
@@ -22,11 +22,28 @@ The central→site hop is unchanged — it still sends a small `RefreshDeploymen
|
||||
still fetches over HTTP, so that half of the original fix stands.
|
||||
|
||||
**The successor ceiling is different in kind.** The gRPC sync stream has a 4 MB default receive
|
||||
limit, and LocalDb batches by ROW COUNT (`LocalDb:Replication:MaxBatchSize`, default 500), not by
|
||||
bytes. A ~70 KB `config_json` — the largest measured in production — times 500 rows is ~35 MB,
|
||||
which would exceed the limit. The rig therefore pins `MaxBatchSize` to **16** (~1.1 MB worst case).
|
||||
Any deployment replicating wide rows must size that key deliberately; see the Phase 2 plan (D6) and
|
||||
`docs/plans/2026-07-19-localdb-phase2-live-gate.md`.
|
||||
limit. Until LocalDb 0.2.0 the library batched by ROW COUNT only (`LocalDb:Replication:MaxBatchSize`,
|
||||
default 500): a ~70 KB `config_json` — the largest measured in production — times 500 rows is
|
||||
~35 MB, which would exceed the limit, so the rig pinned `MaxBatchSize` to **16** (~1.1 MB worst
|
||||
case). See the Phase 2 plan (D6) and `docs/plans/2026-07-19-localdb-phase2-live-gate.md`.
|
||||
|
||||
**Closed by LocalDb 0.2.0** (arch-review WP3.3): batching is now bounded by
|
||||
`LocalDb:Replication:MaxBatchBytes` — summed serialized bytes, default **2 MB**, sized under the
|
||||
4 MB limit — with the row count demoted to a secondary cap, and a single row above the budget sent
|
||||
alone rather than stalling the stream. The rig's old `MaxBatchSize = 16` pin (sized against the
|
||||
4 MB *wire* limit) is retired.
|
||||
|
||||
**Amended (2026-08-14) — `MaxBatchSize` is back, pinned to a different value for a different
|
||||
reason.** `MaxBatchBytes` bounds the wire message, but it doesn't bound the DB **read page**:
|
||||
`OplogStore.ReadBatchAboveAsync`/`SnapshotStreamer` materializes a whole `MaxBatchSize`-row page
|
||||
into memory *before* the byte-budget split runs, so an unset (500-default) `MaxBatchSize` still
|
||||
lets a reconnect drain transiently allocate ~35 MB per read even though every wire message stays
|
||||
under `MaxBatchBytes` (arch-review adversarial finding F2). Both site-a nodes on `docker/` now pin
|
||||
an explicit `"MaxBatchSize": 64` to bound that transient read-side allocation, while
|
||||
`MaxBatchBytes` is left at its 2 MB default to bound the wire message; site-b/site-c stay
|
||||
unreplicated so the key doesn't apply there. Not to be confused with LocalDb 0.2.1's unrelated
|
||||
sync-inbox bound (also 64, but message count on the *receive* side, hardcoded in the library, not
|
||||
a config key) — see CLAUDE.md's LocalDb bullet.
|
||||
|
||||
Note the failure mode differs from the one documented below: an oversized gRPC message is
|
||||
**rejected**, not silently dropped.
|
||||
|
||||
@@ -0,0 +1,533 @@
|
||||
# Architectural Review Remediation Plan
|
||||
|
||||
**Date:** 2026-08-14 · **Status:** PROPOSED · **Source:** 2026-08-14 performance/architecture review
|
||||
(eight-pass, all 27 projects + `ZB.MOM.WW.LocalDb`; report artifact
|
||||
`https://claude.ai/code/artifact/8986cd89-6ab6-48aa-973f-119fddd5cafb`).
|
||||
|
||||
This plan resolves **every** finding from the review — the ten ranked findings plus all
|
||||
section-level Med/Low items — as an automated, multi-phase program executed by subagents,
|
||||
parallel where file-ownership makes it safe. It is written so that each work package (WP) is a
|
||||
self-contained subagent assignment: an executor agent should be able to complete a WP from its
|
||||
spec here plus the code, without further orchestrator input.
|
||||
|
||||
---
|
||||
|
||||
## 1. Execution model
|
||||
|
||||
**Orchestrator:** the interactive Claude session (Fable) in this repo. It spawns one subagent per
|
||||
work package, merges results, runs phase gates, and commits.
|
||||
|
||||
**Isolation:** every code-writing package runs in its **own git worktree**
|
||||
(`Agent` tool, `isolation: "worktree"`). Packages within a phase touch disjoint file sets (see
|
||||
the conflict matrix, §7), so merges are clean; the orchestrator resolves any residual conflict
|
||||
itself. Worktrees also make concurrent `dotnet build`/`test` safe (no shared `obj/`).
|
||||
|
||||
**Per-package protocol.** Each executor agent must:
|
||||
1. Read this plan's WP section and the referenced source files first; verify the finding still
|
||||
reproduces in code (line numbers may have drifted — the *description* is authoritative).
|
||||
2. Implement per the spec. Repo rules apply: edit in place, no backup copies; design doc + code +
|
||||
tests + docker config travel together.
|
||||
3. Build the touched projects and run the listed test suites in the worktree
|
||||
(`dotnet build ZB.MOM.WW.ScadaBridge.slnx`, `dotnet test` filtered to relevant projects).
|
||||
4. Commit on the worktree branch with the message given in the WP spec.
|
||||
5. Report back: commit SHA, test summary (counts, any failures), and any spec deviations with
|
||||
rationale.
|
||||
|
||||
**Phase gates (orchestrator, serial).** After merging a phase's branches into the remediation
|
||||
branch: full `dotnet build ZB.MOM.WW.ScadaBridge.slnx` + full `dotnet test` must be green; for
|
||||
phases touching cluster runtime (2 and 3), rebuild the rig (`bash docker/deploy.sh`) and run the
|
||||
phase's live probes (§6). A red gate stops the program — the orchestrator fixes or reverts the
|
||||
offending package before proceeding.
|
||||
|
||||
**Branching.** All work lands on `arch-review-remediation` (branched from `main` in Phase 0).
|
||||
One commit per package (the worktree commit, merged `--no-ff`), plus one docs commit in Phase 4.
|
||||
Cross-repo work in `~/Desktop/scadaproj` gets its own branch there (WP3.3). Nothing is pushed
|
||||
without the user.
|
||||
|
||||
### Model matrix
|
||||
|
||||
| Model | Used for | Rationale |
|
||||
|---|---|---|
|
||||
| **fable** | Orchestration; design docs for structural seams (WP2.1a, WP3.1a, WP3.2a); final review synthesis | Hardest open design decisions and cross-package judgment |
|
||||
| **opus** | Concurrency-sensitive implementation (streams, DCL, actors, ingest, UI coalescing, LocalDb library) | Subtle correctness work; failure modes are silent |
|
||||
| **sonnet** | Mechanical/localized fixes, index migrations, schema tweaks, docs updates | Well-specified, verifiable by tests |
|
||||
| **opus** (code-reviewer agents) | Phase 4 adversarial diff review | Independent verification pass |
|
||||
|
||||
Approximate agent budget: Phase 1 = 7 executors; Phase 2 = 6 executors + 1 design; Phase 3 =
|
||||
2 design + 3 executors; Phase 4 = 6 reviewers + 1 docs agent. ~26 agents total.
|
||||
|
||||
---
|
||||
|
||||
## 2. Phase 0 — Preflight (orchestrator, serial, ~minutes)
|
||||
|
||||
1. `git checkout -b arch-review-remediation` (current `main`; note the pre-existing dirty
|
||||
`docs/requirements/Component-SiteRuntime.md` — stash or leave untouched, do not absorb it).
|
||||
2. Baseline: full build + full test run; record pass/fail counts as the reference state.
|
||||
3. Confirm the rig is deployable (`docker/deploy.sh` last known good) but do **not** redeploy yet.
|
||||
|
||||
Gate: baseline green. If baseline has pre-existing failures, record them — packages are judged
|
||||
against the baseline, not absolute green.
|
||||
|
||||
---
|
||||
|
||||
## 3. Phase 1 — Quick wins (7 packages, ALL PARALLEL)
|
||||
|
||||
Highest leverage per line changed. All packages are independent; the two silent-failure defects
|
||||
(WP1.1, WP1.2) lead.
|
||||
|
||||
### WP1.1 — Stream OK-completion reconnect `[opus]`
|
||||
**Findings:** #1 (High) — streams silently die at the 4-hour max lifetime; OK completion is
|
||||
invisible to reconnect logic.
|
||||
**Files:** `Communication/Grpc/SiteStreamGrpcClient.cs`, `Communication/Actors/SiteAlarmAggregatorActor.cs`,
|
||||
`Communication/Actors/DebugStreamBridgeActor.cs` (+ tests).
|
||||
**Changes:**
|
||||
- Add an `onCompleted` callback to the client subscribe surface (preferred over an error
|
||||
sentinel — callers can distinguish graceful end from fault). Invoke it when the
|
||||
`await foreach` ends without exception, including server OK completion at max lifetime and
|
||||
graceful site shutdown.
|
||||
- `SiteAlarmAggregatorActor`: on completion, `Tell` self a new `GrpcAlarmStreamCompleted(generation)`
|
||||
message → set `_streamDown = true`, mark `IsLive = false`, and let the reconcile tick reopen
|
||||
**without** consuming error-retry/backoff budget (completion is not a fault). Also observe the
|
||||
`Task.Run` subscription task itself (`ContinueWith` faulted → route into the existing error
|
||||
path) so no completion or fault is ever unobserved again.
|
||||
- Same treatment for the debug stream path in `DebugStreamBridgeActor`.
|
||||
- Generation-fence everything (the fencing pattern already exists in the actor — reuse it).
|
||||
**Tests:** unit test on the client (fake stream that completes OK → `onCompleted` fires, `onError`
|
||||
does not); actor test (stream completes OK → `IsLive` false → reopen on next reconcile tick →
|
||||
backoff counter unchanged). Regression: error path still burns retry budget.
|
||||
**Commit:** `fix(comms): reconnect on graceful stream completion — kills the 4h silent stream death`
|
||||
|
||||
### WP1.2 — Site audit DB placement + flush `[sonnet]`
|
||||
**Findings:** #2 (High) — `auditlog.db` on container overlayfs; dead `FlushIntervalMs`; fsync per event.
|
||||
**Files:** `AuditLog/Site/SqliteAuditWriterOptions.cs`, `AuditLog/Site/SqliteAuditWriter.cs`,
|
||||
all site-node `appsettings` under `docker/` and `docker-env2/` (+ tests).
|
||||
**Changes:**
|
||||
- Make `DatabasePath` **required on Site nodes** (drop the CWD-relative default; `ValidateOnStart`),
|
||||
mirroring the `LocalDb:Path` precedent — a healthy-looking node silently writing to overlayfs is
|
||||
the failure mode being closed. Update all rig site-node configs to `/app/data/auditlog.db`.
|
||||
Update any tests that relied on the implicit default to pass an explicit temp path.
|
||||
- Implement `FlushIntervalMs` as designed: the writer loop groups commits within the interval
|
||||
(batch already drains up to 256 — add the time-based commit coalescing). Additionally set
|
||||
`PRAGMA synchronous=NORMAL` on the audit connection — audit is best-effort by design
|
||||
(CLAUDE.md: "audit-write failure NEVER aborts the user-facing action"), so NORMAL's
|
||||
power-loss window is acceptable; say so in a code comment.
|
||||
- One-time migration note in the commit message: the existing container-local file is abandoned;
|
||||
already-forwarded rows are safe centrally, pending rows on the old path are lost once — accepted,
|
||||
this is the bug being fixed. (Cross-reference `docs/known-issues/2026-07-20-cached-telemetry-drain-hot-loop.md`,
|
||||
which this placement bug caused.)
|
||||
**Tests:** options validation test (site role boots ⇒ path required); writer test asserting
|
||||
multi-event single-transaction commit under trickle load.
|
||||
**Commit:** `fix(auditlog): site audit DB onto the data volume; required path + soft flush`
|
||||
|
||||
### WP1.3 — Conditional CDC registration `[opus]`
|
||||
**Findings:** #5 (High), partial — non-replicating nodes pay full CDC capture forever. (The
|
||||
library half — trigger cleanup API, O(1) backlog — is WP3.3.)
|
||||
**Files:** `Host/SiteLocalDbSetup.cs` (+ tests). Read `~/Desktop/scadaproj/ZB.MOM.WW.LocalDb/`
|
||||
and the site-a rig configs first to confirm the exact replication-config shape on both the active
|
||||
and passive peer.
|
||||
**Changes:**
|
||||
- Skip all ten `RegisterReplicated` calls when the node has **no replication configured**
|
||||
(neither `LocalDb:Replication:PeerAddress` nor `ApiKey` — verify the passive-side shape on the
|
||||
rig before choosing the predicate; the guard must be true on BOTH members of a replicated pair
|
||||
and false on site-b/site-c).
|
||||
- Preserve the load-bearing ordering documented in the file: when registration is skipped, the
|
||||
legacy migrator still runs (its rows simply don't replicate — consistent with an unreplicated node).
|
||||
- **Known residual, documented in-file:** a database file that was registered under an older build
|
||||
keeps its stale triggers until WP3.3 ships the library cleanup API; on the docker rig this is
|
||||
moot (volumes are recreated on redeploy of a schema change) — state this in a comment and in the
|
||||
commit message.
|
||||
**Tests:** setup test with and without replication config, asserting trigger presence/absence via
|
||||
`sqlite_master` (`name LIKE '%localdb%'` — confirm the library's trigger naming first).
|
||||
**Commit:** `perf(host): install CDC capture only when replication is configured`
|
||||
|
||||
### WP1.4 — SQL quick indexes + sliced purge `[sonnet]`
|
||||
**Findings:** S&F sweep O(N) scan (High); audit KPI window key-lookup; Notification KPI
|
||||
delivered-interval scan; unbounded terminal purge (all part of #7/#8 support).
|
||||
**Files:** `StoreAndForward/StoreAndForwardSchema.cs` (+ its versioned-upgrade path),
|
||||
`ConfigurationDatabase/Migrations/` (+ manual SQL script for production per repo convention),
|
||||
`ConfigurationDatabase/Repositories/NotificationOutboxRepository.cs`.
|
||||
**Changes:**
|
||||
- `sf_messages`: add `(status, created_at)` index so the ordered due-scan terminates at `LIMIT`;
|
||||
follow the schema's existing versioned-upgrade mechanism so existing site files get it.
|
||||
- `IX_AuditLog_OccurredAtUtc` → add `INCLUDE (Status)` (EF migration; remember the repo gotcha —
|
||||
**build first**, never `--no-build`, delete empty migrations if scaffolded).
|
||||
- Notifications: filtered index `(DeliveredAt) WHERE Status = 'Delivered'` (or the enum's stored
|
||||
value — check the column type) covering the delivered-last-interval KPI count.
|
||||
- `DeleteTerminalOlderThanAsync`: slice into batched deletes — copy the slicing pattern used twice
|
||||
elsewhere in the same layer (per-channel audit retention delete is one).
|
||||
**Tests:** migration applies cleanly (dev auto-apply path); a query-shape test is not required —
|
||||
Phase 4 live probes verify plans via `EXPLAIN`/showplan.
|
||||
**Commit:** `perf(sql): sweep/KPI covering indexes + sliced notification terminal purge`
|
||||
|
||||
### WP1.5 — Mechanical hot-path fixes `[sonnet]`
|
||||
**Findings:** InstanceActor nits (Med); per-call `JsonSerializerOptions` ×4 (Med).
|
||||
**Files:** `SiteRuntime/Actors/InstanceActor.cs`, `ExternalSystemGateway/ExternalSystemClient.cs`,
|
||||
`SiteRuntime/Repositories/SiteExternalSystemRepository.cs` (options instance only — the query
|
||||
redesign is WP2.6), `ManagementService/ManagementEndpoints.cs`, `CentralUI/.../ScriptAnalysisService.cs`.
|
||||
**Changes:**
|
||||
- `HandleSetStaticAttribute`: use the existing `_resolvedAttributeByName` `TryGetValue` instead of
|
||||
the LINQ scan (InstanceActor.cs:373).
|
||||
- `HandleTagValueUpdate`: precompute the parsed `DataType` into the resolved-attribute index entry
|
||||
at config-apply time; stop re-`Enum.TryParse`-ing per update. Intern/cache the quality strings
|
||||
(`Quality.ToString()` per update → static readonly map).
|
||||
- Static-override writes: coalesce per instance on a short timer, reusing the batched-flush pattern
|
||||
already present for native alarms in the same codebase (copy that shape, don't invent).
|
||||
- One `static readonly JsonSerializerOptions` per assembly at the four verified sites.
|
||||
**Tests:** existing InstanceActor suite must stay green; add a test pinning single-flush
|
||||
coalescing of N rapid static writes.
|
||||
**Commit:** `perf(runtime): O(1) attribute resolution, precomputed types, coalesced static writes, shared JSON options`
|
||||
|
||||
### WP1.6 — Transport fail-fast `[sonnet]`
|
||||
**Findings:** known-failure sends burn full Ask timeouts (Med, first half — the timeout stagger
|
||||
and double-hop are WP2.2).
|
||||
**Files:** `Communication/Grpc/GrpcSiteTransport.cs` (+ tests).
|
||||
**Changes:** a send to a site with no configured channel replies `Status.Failure` (with a
|
||||
descriptive exception) to the asker immediately instead of warn-and-drop. Audit every
|
||||
warn-and-drop in the file for the same pattern.
|
||||
**Tests:** Ask against an unconfigured site faults in <1s, not at the 30s timeout.
|
||||
**Commit:** `fix(comms): fail known-dead sends immediately instead of burning Ask timeouts`
|
||||
|
||||
### WP1.7 — Event-log writer batching + sliced retention `[sonnet]`
|
||||
**Findings:** per-event transactions in `SiteEventLogger` (part of the site_events High — the
|
||||
volume *policy* is WP3.2); unbatched `site_events` retention DELETE (Med).
|
||||
**Files:** `SiteEventLogging/SiteEventLogger.cs`, `SiteEventLogging/EventLogPurgeService.cs` (+ tests).
|
||||
**Changes:**
|
||||
- Writer loop: drain the existing batching channel into **one transaction per drain** (up to 256),
|
||||
exactly the `SqliteAuditWriter` shape — cite it in a comment.
|
||||
- Retention purge: batch the DELETE at 1000 rows/iteration like the cap purge next to it.
|
||||
**Tests:** writer test (N queued events → 1 transaction); purge test over >1000 expired rows.
|
||||
**Commit:** `perf(sitelog): batched event-log commits and sliced retention purge`
|
||||
|
||||
**Phase 1 gate:** merge all seven, full build + test, then rebuild the rig (`docker/deploy.sh` —
|
||||
WP1.2 changed site configs) and run the Phase 1 live probes (§6).
|
||||
|
||||
---
|
||||
|
||||
## 4. Phase 2 — Seam rework (6 packages, PARALLEL after WP2.1a)
|
||||
|
||||
### WP2.1 — DCL batch seam `[design: fable → implement: opus]` *(largest package)*
|
||||
**Findings:** #3 (High, four collapsed findings) + MxGateway 2-RPC subscribe / no-backoff retry
|
||||
(Med) + quality-counter and alarm-fanout costs (Med).
|
||||
**Stage a — design memo** (fable, ~1 page, committed as
|
||||
`docs/plans/2026-08-XX-dcl-batch-seam-design.md`): the batch API shape on `IDataConnection`
|
||||
(`SubscribeBatchAsync(IReadOnlyList<TagSubscription>)` returning per-tag results; true bulk
|
||||
`ReadBatchAsync`/`WriteBatchAsync`), per-subscription monitored-item budget + sharding scheme,
|
||||
reconnect chunking/stagger parameters (configurable; defaults sized for 37,500 tags), re-seed
|
||||
deadline semantics, and what the MxGateway adapter can do **without** a cross-repo mxaccessgw
|
||||
change (fallback: client-side chunked pipelining; note the cross-repo bulk-RPC follow-up if the
|
||||
proto lacks one — check `~/Desktop/MxAccessGateway` first).
|
||||
**Stage b — implementation** (opus):
|
||||
- `RealOpcUaClient`: N×`AddItem` + one `ApplyChangesAsync` per batch; serialize all
|
||||
`ApplyChangesAsync` on the shared Subscription behind a `SemaphoreSlim` (batching makes
|
||||
contention rare; the lock makes it safe); shard monitored items across subscriptions above the
|
||||
per-subscription budget (default ~5,000, configurable).
|
||||
- `OpcUaDataConnection`: implement the batch interface; `ReadBatchAsync`/`WriteBatchAsync` become
|
||||
single OPC UA service calls (the MxGateway adapter's `ReadBulk`/`WriteBulk` is the in-repo
|
||||
reference).
|
||||
- `DataConnectionActor`: `HandleSubscribe`/`ReSubscribeAll` pass whole tag lists; reconnect
|
||||
becomes bounded chunked batches honoring the existing failover stagger; `SeedTagsAsync` becomes
|
||||
chunked batch reads with bounded parallelism and an **overall** deadline (no more 30s-per-tag
|
||||
serial worst case). Tag-resolution retry gains exponential backoff with a ceiling and batched
|
||||
probes. Quality counters flush on transitions or a short timer instead of per-change; alarm
|
||||
subscriber matching gets a source-prefix index and pushes the union filter to the gateway feed.
|
||||
**Tests:** the existing DCL suite (Become/Stash lifecycle, generation fencing) must stay green;
|
||||
new tests with a fake client: batch subscribe issues one apply per chunk; concurrent applies
|
||||
serialize; re-seed respects the deadline; backoff caps.
|
||||
**Commit:** `perf(dcl): batch subscribe/read/write seam, bounded reconnect, sharded subscriptions`
|
||||
|
||||
### WP2.2 — Central ingest & SQL set-based rework `[opus]`
|
||||
**Findings:** #6 (High, partition purge), #7 (High, row-at-a-time ingest), #8 (High, KPI scans —
|
||||
query-shape half), KPI rollup N+1 / read-path hygiene / no pooling (Med).
|
||||
**Files:** `AuditLog/.../AuditLogIngestActor.cs`, `ConfigurationDatabase/Repositories/AuditLogRepository.cs`,
|
||||
`SiteCallAudit/.../SiteCallAuditActor.cs` + `SiteCallAuditRepository.cs`,
|
||||
`ConfigurationDatabase/Repositories/NotificationOutboxRepository.cs`, `KpiHistoryRepository.cs`,
|
||||
`ConfigurationDatabase/ServiceCollectionExtensions.cs`, gRPC ingest entry in Communication (+ migration + manual SQL script).
|
||||
**Changes:**
|
||||
- **Ingest batching:** one TVP (or multi-row `INSERT … SELECT … WHERE NOT EXISTS` batch) per
|
||||
telemetry packet with per-row fallback on constraint violation; cached-telemetry entry =
|
||||
one transaction, not ~5 round trips.
|
||||
- **SiteCalls upsert:** single statement (`UPDATE … ; IF @@ROWCOUNT = 0 INSERT …` guarded for the
|
||||
newer-status predicate, or `MERGE` with the usual HOLDLOCK hygiene).
|
||||
- **Reconciliation off-mailbox:** the drain moves out of `ReceiveAsync` to the
|
||||
`PipeTo` + in-flight-guard shape — `NotificationOutboxActor` next door is the in-repo reference;
|
||||
ingest/queries/KPI asks must not queue behind post-outage catch-up.
|
||||
- **Timeout stagger + hop removal:** inner budgets strictly less than outer (site-ask 25s →
|
||||
ingest 20s → SQL 15s, or similar); the gRPC handler Asks the ingest singleton proxy directly,
|
||||
removing the redundant intermediate actor hop.
|
||||
- **Partition purge:** prefer making EventId uniqueness **partition-aligned**
|
||||
(`UNIQUE (EventId, OccurredAtUtc)`) so `SWITCH` needs no index drop at all; EventId is globally
|
||||
unique by construction (GUID minted at source), so the idempotency probe still seeks correctly —
|
||||
document this reasoning in the migration. If alignment proves incompatible with the probe shape,
|
||||
fall back to `ONLINE = ON` rebuild outside the switch transaction (Standard-edition caveat: check
|
||||
target SQL edition; if ONLINE is unavailable, alignment is the only acceptable option).
|
||||
- **KPI query shapes:** split the live-queue seeks from the delivered-interval count (filtered
|
||||
index from WP1.4); rollup fold preloads existing rows per window into a dictionary (kill the
|
||||
per-(series,hour) existence SELECT).
|
||||
- **Hygiene:** `AsNoTracking` + keyset paging on the Notifications read path (mirror the two
|
||||
sibling repos); targeted `ExecuteUpdate` for delivery-attempt status writes instead of detached
|
||||
full-row `Update()`; `AddDbContextPool` — **first verify** the context registers no scoped
|
||||
state/injected services that break pooling; if it does, fix or document why pooling is skipped.
|
||||
**Tests:** ingest idempotency under batch (duplicate EventIds in one packet + across packets);
|
||||
upsert newer-status semantics; reconciliation running while an ingest Ask completes promptly.
|
||||
**Commit:** `perf(central): set-based ingest, aligned partition purge, KPI query shapes, EF hygiene`
|
||||
|
||||
### WP2.3 — Wire efficiency `[opus]`
|
||||
**Findings:** #10 (Med, alarm seed waste + per-attempt re-fan-out), unbounded pre-snapshot
|
||||
buffers / lossy shared channel (Med), `PullAuditEvents` at-most-once (Low).
|
||||
**Files:** `Communication/Actors/SiteAlarmAggregatorActor.cs`, `SiteAlarmLiveCacheService.cs`,
|
||||
`SiteCommandDtoMapper.cs`, `DebugStreamBridgeActor.cs`, `Communication/Grpc/SiteStreamGrpcServer.cs`,
|
||||
site audit queue (`ISiteAuditQueue` impl) (+ tests).
|
||||
**Changes:**
|
||||
- **Alarms-only snapshot:** the seed travels the `SiteCommandService` query surface
|
||||
(`QueryReply.DebugViewSnapshot`), so add an additive alarms-only flag on the query DTO — verify
|
||||
whether the vendored proto is involved at all; if it is, follow the documented manual
|
||||
toggle-build-copy-untoggle regeneration, additive field numbers only.
|
||||
- Seed **once per successful (re)connect** plus the existing reconcile as the diff/backstop —
|
||||
not a full unconditional snapshot every 60s. Jitter the per-site aggregator timers so 10 sites
|
||||
don't fan out in lockstep; give the seed-retry leg backoff.
|
||||
- Cap the pre-snapshot buffers (drop-oldest + a dropped-count surfaced on the health report);
|
||||
hard timer on snapshot arrival that fails the session rather than buffering forever; stream
|
||||
events must not reset the receive-timeout that ends a stuck session.
|
||||
- `SubscribeSite` gets its own channel (larger, or wait-mode) instead of sharing the debug view's
|
||||
1000-slot DropOldest — an alarm burst during a WAN stall must not silently drop transitions.
|
||||
Batch multiple events per proto message where the contract allows (additive).
|
||||
- `PullAuditEvents`: mirror `PullSiteCalls`' composite `(timestamp, id)` keyset cursor; flip rows
|
||||
to `Reconciled` only when the **next** pull's cursor proves receipt (at-least-once restored).
|
||||
**Tests:** cursor-proof flip semantics (fault between response and next pull → rows re-served);
|
||||
seed-once behavior across a simulated reconnect; buffer cap + counter.
|
||||
**Commit:** `perf(comms): alarms-only seed, capped buffers, at-least-once audit pull`
|
||||
|
||||
### WP2.4 — Central UI performance `[opus]`
|
||||
**Findings:** #8 (High, per-circuit KPI polling half), AlarmSummary/Health/DebugView per-circuit
|
||||
costs (High), push-triggered full reloads / serial Asks / no virtualization (Med), sandbox
|
||||
thread-parking (Low).
|
||||
**Files:** `CentralUI` — `Health.razor`, `AlarmSummary.razor`, `DebugView.razor`,
|
||||
`Deployments.razor`, a new process-level KPI snapshot service, sandbox host (+ tests where the
|
||||
project has them).
|
||||
**Changes:**
|
||||
- **Process-level memoized KPI snapshot service** (singleton, 5–10s TTL, single-flight so N
|
||||
circuits share one SQL round per interval) consumed by Health + Notification Outbox + Site
|
||||
Calls pages. This plus WP2.2's query shapes closes finding #8 from both ends.
|
||||
- AlarmSummary derives not-reporting from the alarm aggregator's own reconcile state instead of
|
||||
re-running the per-instance snapshot fan-out per circuit per 15s.
|
||||
- DebugView: coalesce renders at ~250ms; rebuild the trees behind a version stamp instead of per
|
||||
event; marshal off the gRPC thread once per coalesce window.
|
||||
- Deployments: apply the pushed delta (or debounce reloads ~500ms) instead of reloading all
|
||||
records + instances per push; server-side paging.
|
||||
- Health tick: `Task.WhenAll` with short per-tile timeouts so one hung singleton degrades one
|
||||
tile, not the tick. `Virtualize` on the flat alarm table.
|
||||
- Sandbox Test Runs: bound with a semaphore instead of parking thread-pool threads on sync accessors.
|
||||
- Preserve the existing visual design exactly — these are behavioral changes only, no restyling,
|
||||
no new component frameworks (repo rule).
|
||||
**Tests:** KPI cache single-flight test; render-coalescing unit test if the code shape allows.
|
||||
**Commit:** `perf(ui): shared KPI cache, live-cache-backed alarm summary, coalesced debug renders`
|
||||
|
||||
### WP2.5 — Deployment & authoring pipeline `[opus]`
|
||||
**Findings:** #9 (High, both halves), verdict-cache Clear() leak + unpaged management queries (Med).
|
||||
**Files:** `TemplateEngine/.../FlatteningPipeline.cs`, `TemplateEngineRepository.cs`,
|
||||
`TemplateService.cs`, `DeploymentManager/.../DeploymentService.cs`,
|
||||
`ManagementService/ManagementActor.cs`, `ScriptAnalysis/.../ScriptCompileVerdictCache.cs`,
|
||||
CLI (+ tests).
|
||||
**Changes:**
|
||||
- **Flatten-session cache:** within one deploy/validation session, memoize template chain and
|
||||
composition loads keyed by template id + version; hoist the three global queries (shared
|
||||
scripts, schemas, connections) once per session — `ArtifactDeploymentService` is the in-repo
|
||||
reference shape.
|
||||
- **`DeploySiteAsync`:** bulk orchestration over a site's instances with bounded parallel fan-out
|
||||
and per-instance timeouts, reusing the per-instance operation lock; surfaced via ManagementActor
|
||||
command + CLI (`instance deploy-site <site>` or consistent naming — follow the CLI's existing
|
||||
verb conventions); UI wiring optional, defer if it inflates scope.
|
||||
- **Staleness watermark:** per-template monotonic version bumped on any template-graph mutation;
|
||||
staleness detection compares watermarks before paying a full flatten.
|
||||
- **Authoring:** slim projections for acyclicity/collision checks (no script bodies);
|
||||
`AsNoTracking` on read-only walks; `ReconcileDescendantsAsync` receives the already-loaded
|
||||
graph instead of re-calling `GetAllTemplatesAsync`.
|
||||
- **Management queries:** `QueryDeployments` pages DB-side with a summary projection; `ListTemplates`
|
||||
pages DB-side without full script bodies; batch override apply becomes one read + one commit.
|
||||
- **Verdict cache:** LRU/segmented eviction instead of wholesale `Clear()` (the Clear re-opens the
|
||||
non-collectible `InteractiveAssemblyLoader` leak the cache exists to bound); add a retention
|
||||
policy for terminal deployment records (config, default generous).
|
||||
**Tests:** flatten count test (N instances, same template ⇒ chain loaded once); DeploySiteAsync
|
||||
fan-out with an injected slow site (bounded, others complete); verdict-cache eviction keeps hot
|
||||
entries.
|
||||
**Commit:** `perf(deploy): flatten-session caching, bulk DeploySiteAsync, paged management queries`
|
||||
|
||||
### WP2.6 — Cross-cutting misc `[sonnet]`
|
||||
**Findings:** site external-system resolution (Med), Inbound API per-request SQL (Low), unbounded
|
||||
S&F observer queue (Low), CLI timeout (Low), failback-probe heartbeat pollution (Low), site stream
|
||||
alarm-vs-attribute shared buffer (Med).
|
||||
**Files:** `SiteRuntime/Repositories/SiteExternalSystemRepository.cs`, InboundAPI method-lookup
|
||||
path, `StoreAndForward` observer queue, `SiteRuntime/Streaming/SiteStreamManager.cs`, CLI
|
||||
`HttpClient` setup, failback probe (+ tests).
|
||||
**Changes:**
|
||||
- External-system resolution: name-keyed lookup (the ID reverse-map scan goes); cache the parsed
|
||||
method list per system, invalidated on redeploy — honoring the documented "indexed query" contract.
|
||||
- Inbound API: short-TTL ApiMethod cache invalidated by the existing `ScriptArtifactChangeSubscriber`
|
||||
bus (the invalidation plumbing already exists — lean on it).
|
||||
- S&F observer queue: bound it DropOldest like its siblings (it is the one unbounded channel in
|
||||
the system) + a dropped counter.
|
||||
- `SiteStreamManager`: separate alarm publish source (or priority path) so attribute storms can't
|
||||
evict alarm transitions; drop counter on the site health report; skip publish at zero subscribers.
|
||||
- CLI `HttpClient`: explicit timeout (30s, config-overridable).
|
||||
- Failback probe: mark the probe heartbeat synthetic (flag or reserved SiteId) so it stops
|
||||
polluting liveness data — check what consumes heartbeats before choosing the mechanism.
|
||||
**Commit:** `perf(misc): cached hot-path lookups, bounded observer queue, alarm-priority stream path`
|
||||
|
||||
**Phase 2 gate:** merge order WP2.2 → WP2.3 → WP2.1 → WP2.4 → WP2.5 → WP2.6 (Communication-heavy
|
||||
first — likeliest residual conflicts surface early). Full build + test; rig rebuild; Phase 2 live
|
||||
probes (§6).
|
||||
|
||||
---
|
||||
|
||||
## 5. Phase 3 — Structural (design-first; 3 tracks)
|
||||
|
||||
Design stages run in parallel (fable); implementations start as each design lands. Defaults are
|
||||
pre-decided below so automation never blocks — the design memos may refine but not stall.
|
||||
|
||||
### WP3.1 — Script execution pool split `[design: fable → implement: opus]`
|
||||
**Findings:** #4 (High) + actor-per-execution overhead / compile-cache Clear (Med).
|
||||
**Stage a — design doc** (`docs/plans/2026-08-XX-script-pool-split-design.md`), deciding:
|
||||
- Trigger-expression evaluation leaves the blocking pool — default decision: evals are
|
||||
non-blocking by construction, so run them as plain async on the default dispatcher with a
|
||||
concurrency gate, no second dedicated pool.
|
||||
- Blocking-script pool sizing scales with deployed instance count (default: `max(8, instances/8)`
|
||||
capped at a config ceiling); stuck-script watchdog **replaces** lost threads.
|
||||
- Timeout budget includes queue wait (deadline captured at enqueue).
|
||||
- Per-script in-flight cap — default policy: bounded queue per script (cap 4), overflow **sheds
|
||||
newest with a site event + counter** (an alarm-triggered run that can't start within its
|
||||
deadline is stale anyway); design memo must justify or amend.
|
||||
- `ScriptExecutionActor` elimination: run the guarded task directly under `ScriptActor`,
|
||||
preserving supervision semantics (Stop-on-failure equivalence), telemetry, and the audit
|
||||
`ExecutionId`/`ParentExecutionId` threading — this is the riskiest part; the design memo maps
|
||||
every behavior the actor currently provides to its replacement.
|
||||
- `SiteScriptCompileCache`: LRU eviction; cache-miss compiles move off-dispatcher (the
|
||||
DeploymentManager shared-script path is the in-repo reference).
|
||||
**Stage b — implement + tests:** starvation regression test (8 blocked scripts ⇒ alarm eval still
|
||||
completes <2s); queue-time-inclusive timeout test; watchdog replacement test; supervision-parity
|
||||
tests for the actor removal.
|
||||
**Commit:** `perf(runtime): split trigger evals from the blocking pool; bounded, deadline-aware execution`
|
||||
|
||||
### WP3.2 — site_events volume policy `[design: fable → implement: sonnet]`
|
||||
**Findings:** #High site_events (policy half; mechanics done in WP1.7).
|
||||
**Stage a — design memo** deciding: per-run Started/Completed Info events become **sampled/opt-in**
|
||||
(default: off per-run, aggregate counters per interval instead; per-script opt-in flag for
|
||||
debugging); whether `site_events` should replicate at all (default recommendation: keep replicated
|
||||
— central event-log queries hit either node — but the memo must check what actually reads
|
||||
`site_events` on the peer before confirming; if nothing does, deregister it and use clock-based
|
||||
local purges that never enter the oplog, coordinating with WP3.3's dereg API).
|
||||
**Stage b:** implement per memo; update `Component-SiteEventLogging` doc (Phase 4 collects).
|
||||
**Commit:** `perf(sitelog): sampled per-run events; site_events replication policy`
|
||||
|
||||
### WP3.3 — LocalDb library (CROSS-REPO: `~/Desktop/scadaproj`) `[opus]`
|
||||
**Findings:** #5 (library half), replication row-count batching (Med), library-level costs (Low).
|
||||
**Files:** `~/Desktop/scadaproj/ZB.MOM.WW.LocalDb/` on its own branch; then the consuming update here.
|
||||
**Changes (library):**
|
||||
- `DeregisterReplicated(table)` / cleanup API: idempotently drops capture triggers (and optionally
|
||||
prunes that table's oplog rows) — consumed by `SiteLocalDbSetup` when replication is
|
||||
unconfigured, closing WP1.3's stale-trigger residual.
|
||||
- Backlog depth becomes O(1) (`MAX(seq) − last_acked_seq` or a maintained counter) — it currently
|
||||
runs an O(backlog) `COUNT(*)` per minute and per metrics scrape.
|
||||
- Replication batching by **summed bytes** (budget under the 4 MB gRPC cap, row-count as a
|
||||
secondary cap) + ack coalescing; the rig's `MaxBatchSize=16` pin can then be retired.
|
||||
- `LwwApplier` prepare-once-rebind per table; drop the unused `__localdb_oplog_hlc` index (handle
|
||||
existing files in the schema-version upgrade).
|
||||
**Changes (this repo, after the library lands):** bump the package/project reference; call the
|
||||
cleanup API from `SiteLocalDbSetup` when unregistered; relax the rig `MaxBatchSize` pin to the
|
||||
byte budget; update the CLAUDE.md LocalDb bullet and the **umbrella `../scadaproj/CLAUDE.md`
|
||||
ScadaBridge entry in the same change** (repo rule).
|
||||
**Tests:** library suite for dereg idempotency, O(1) depth, byte-budget batching; ScadaBridge
|
||||
integration: site-b boots with zero triggers and zero oplog growth.
|
||||
**Commits:** library commit in scadaproj + `chore(deps): LocalDb <version> — dereg API, byte-budget replication` here.
|
||||
|
||||
**Phase 3 gate:** full build + test in BOTH repos; rig rebuild; live probes.
|
||||
|
||||
---
|
||||
|
||||
## 6. Phase 4 — Verification, review, docs (mixed parallel)
|
||||
|
||||
### Live probes (orchestrator + rig, accumulate per phase gate)
|
||||
- **Stream lifetime:** set max stream lifetime to ~2 min on the rig; verify the alarm stream
|
||||
reconnects within one reconcile tick of OK completion and `IsLive` reflects the gap (WP1.1).
|
||||
- **CDC:** on site-b/site-c nodes, `sqlite_master` shows no capture triggers; oplog table empty
|
||||
and not growing (WP1.3/WP3.3). Site-a pair still converges (replication regression).
|
||||
- **Audit path:** site `auditlog.db` present under `/app/data`; survives `docker compose
|
||||
rm`+recreate of one site container (WP1.2).
|
||||
- **Query plans:** `EXPLAIN QUERY PLAN` on the S&F due-sweep (index-terminated); SQL Server
|
||||
showplan on the notification KPI and audit KPI queries (seeks, no full scans) (WP1.4/WP2.2).
|
||||
- **Deploy timing:** bulk-redeploy a site on the rig before/after; record wall clock (WP2.5).
|
||||
- **Failover drill:** `docker/failover-drill.sh` still passes (~25s takeover) — protects against
|
||||
regressions from the actor/timeout changes.
|
||||
- **Load validation:** the deferred target-scale load test (deferred-work register #25) is the
|
||||
final proof the ceilings moved — schedule as its own follow-on; this plan's exit criterion is
|
||||
the probes above, not #25.
|
||||
|
||||
### Adversarial review `[6 parallel code-reviewer agents, opus; fable synthesis]`
|
||||
One reviewer per area over the full remediation diff (site runtime, DCL, comms, site persistence,
|
||||
central SQL, UI/deploy), each instructed to try to **refute** the fixes (regressions, changed
|
||||
semantics, missed call sites, test gaps). Confirmed findings are fixed by targeted follow-up
|
||||
agents before the docs commit.
|
||||
|
||||
### Documentation propagation `[sonnet]`
|
||||
- Component docs touched: `Component-DataConnectionLayer`, `Component-SiteRuntime`,
|
||||
`Component-CentralSiteCommunication`, `Component-AuditLog`, `Component-SiteEventLogging`,
|
||||
`Component-StoreAndForward`, `Component-NotificationOutbox`, `Component-SiteCallAudit`,
|
||||
`Component-TemplateEngine`, `Component-DeploymentManager`, `Component-CentralUI`,
|
||||
`Component-ClusterInfrastructure` (only where behavior/config changed — no cosmetic edits).
|
||||
- CLAUDE.md Key Design Decisions: stream-completion reconnect, conditional CDC, required site
|
||||
audit DB path, DCL batch seam, `DeploySiteAsync`, script-pool split, LocalDb byte-budget
|
||||
replication (retire the MaxBatchSize=16 note).
|
||||
- Umbrella `../scadaproj/CLAUDE.md`: LocalDb entry (done with WP3.3).
|
||||
- Close/annotate any related `docs/known-issues/` entries; README component table needs no change
|
||||
(no components added/removed).
|
||||
- Final `git diff` review by the orchestrator, then the docs commit. **Push only on user request.**
|
||||
|
||||
---
|
||||
|
||||
## 7. Parallelism & conflict matrix
|
||||
|
||||
| Phase | Package | Projects touched | Conflicts within phase |
|
||||
|---|---|---|---|
|
||||
| 1 | WP1.1 | Communication (client+actors) | WP1.6 same project, different files — merge-safe |
|
||||
| 1 | WP1.2 | AuditLog(site), docker configs | none |
|
||||
| 1 | WP1.3 | Host | none |
|
||||
| 1 | WP1.4 | ConfigurationDatabase, StoreAndForward | none |
|
||||
| 1 | WP1.5 | SiteRuntime, ESG, ManagementService, CentralUI | single-line edits, none |
|
||||
| 1 | WP1.6 | Communication (transport) | see WP1.1 |
|
||||
| 1 | WP1.7 | SiteEventLogging | none |
|
||||
| 2 | WP2.1 | DataConnectionLayer | none |
|
||||
| 2 | WP2.2 | AuditLog(central), SiteCallAudit, ConfigurationDatabase, NotificationOutbox | none (WP2.3 owns Communication) |
|
||||
| 2 | WP2.3 | Communication, site audit queue | none |
|
||||
| 2 | WP2.4 | CentralUI | none |
|
||||
| 2 | WP2.5 | TemplateEngine, DeploymentManager, ManagementService, ScriptAnalysis, CLI | none |
|
||||
| 2 | WP2.6 | SiteRuntime(streaming+repos), InboundAPI, S&F, CLI(http) | CLI touched by WP2.5 (commands) and WP2.6 (HttpClient) — different files; merge WP2.5 first |
|
||||
| 3 | WP3.1 | SiteRuntime(scripts) | none |
|
||||
| 3 | WP3.2 | SiteEventLogging, SiteRuntime(one call site) | coordinate with WP3.1 merge order (3.1 first) |
|
||||
| 3 | WP3.3 | scadaproj (separate repo) + Host/config here | in-repo half lands after 3.1/3.2 |
|
||||
|
||||
## 8. Risks & mitigations
|
||||
|
||||
- **Vendored proto drift** (WP2.3): additive fields only, never reuse numbers; follow the
|
||||
documented manual regeneration toggle. Mitigation: the executor verifies whether the change is
|
||||
DTO-only before touching proto at all.
|
||||
- **Partition-aligned uniqueness** (WP2.2) changes the enforcement scope of EventId uniqueness
|
||||
from global to per-partition. GUID collision across partitions is not a real risk, but the
|
||||
ingest probe's semantics must be re-tested for duplicate delivery across a month boundary.
|
||||
- **Pooled DbContext** (WP2.2): pooling breaks contexts with per-scope injected state — verify
|
||||
before switching; skipping with a documented reason is an acceptable outcome.
|
||||
- **ScriptExecutionActor removal** (WP3.1) is the highest-semantic-risk change: supervision,
|
||||
telemetry, and audit-correlation parity are explicit design-memo obligations, and it ships in
|
||||
its own commit so it can be reverted independently.
|
||||
- **Rolling-upgrade caveat** (WP3.3): site pairs already require stop/start together (repo rule);
|
||||
the LocalDb schema-version bump rides that existing constraint — call it out in the topology guide.
|
||||
- **EF migration gotcha:** always build before `dotnet ef migrations add`; delete empty scaffolds.
|
||||
- **Merge conflicts:** worktree isolation + the matrix above; residual conflicts are resolved by
|
||||
the orchestrator, never by re-running an executor blindly.
|
||||
|
||||
## 9. Exit criteria
|
||||
|
||||
1. Full solution build + tests green in both repos; baseline test count not reduced.
|
||||
2. All §6 live probes pass on the rig; failover drill unchanged (~25s).
|
||||
3. Adversarial review pass complete with all CONFIRMED findings resolved.
|
||||
4. Docs propagated (component docs, both CLAUDE.mds, known-issues) with no stale cross-references.
|
||||
5. One reviewed commit per package on `arch-review-remediation`; branch ready for the user to
|
||||
merge/push; target-scale load test (#25) scheduled as the follow-on validation.
|
||||
@@ -0,0 +1,103 @@
|
||||
# DCL Batch Seam Design — WP2.1a
|
||||
|
||||
**Date:** 2026-08-14 · **Scope:** design contract for WP2.1b (implementation) · **Findings:** arch-review #3 (High) + MxGateway 2-RPC subscribe / no-backoff retry (Med) + quality-counter and alarm-fanout costs (Med) · **Sized for:** 37,500 tags/site (500 instances × 75 tags, `docs/deployment/topology-guide.md:255`).
|
||||
|
||||
## 1. Batch API on `IDataConnection`
|
||||
|
||||
```csharp
|
||||
public record TagSubscribeResult(string TagPath, bool Success, string? SubscriptionId, string? ErrorMessage);
|
||||
|
||||
Task<IReadOnlyList<TagSubscribeResult>> SubscribeBatchAsync(
|
||||
IReadOnlyList<string> tagPaths, SubscriptionCallback callback, CancellationToken ct = default);
|
||||
Task UnsubscribeBatchAsync(IReadOnlyList<string> subscriptionIds, CancellationToken ct = default);
|
||||
```
|
||||
|
||||
- **One shared callback, plain tag list** — deliberately simpler than the plan's sketched `IReadOnlyList<TagSubscription>` input record. The only caller passes an identical closure for every tag (`DataConnectionActor.HandleSubscribe`, `DataConnectionActor.cs:745-748`: `(path, value) => self.Tell(new TagValueReceived(path, value, generation))`), and `SubscriptionCallback` already delivers the tag path. A per-tag record would carry nothing.
|
||||
- **Partial-failure contract:** per-tag faults (bad node id, resolution failure) come back as result rows with `Success:false` and never throw — mirroring the gateway's `SubscribeResult`/`BulkSubscribeReply` (mxaccess_gateway.proto:632) and OPC UA per-item create status. A **thrown** exception means the whole batch failed at connection level; the actor classifies it with the existing `IsConnectionLevelFailure` (`DataConnectionActor.cs:1056`) and drives Reconnecting, exactly as the per-tag path does today (`SubscribeTagResult.ConnectionLevelFailure`, line 758-761).
|
||||
- **`ReadBatchAsync`/`WriteBatchAsync` keep their existing signatures** (`IDataConnection.cs:47,58` — dictionary-of-results, already right) but the contract is strengthened from "may loop over singles" to **true bulk**: `OpcUaDataConnection` currently loops per tag (`OpcUaDataConnection.cs:250-274, 289-315`); it becomes one `ReadValueIdCollection`/`WriteValueCollection` service call (the MxGateway adapter is the in-repo reference — `MxGatewayDataConnection.cs:267-290` already delegates to `ReadBulkAsync`/`WriteBulkAsync`). Adapters chunk internally at 1,000 items per service call to stay under typical OPC UA `MaxNodesPerRead/Write` operation limits. Preserved semantics: per-tag failure rows, `OperationCanceledException` aborts the whole batch (`OpcUaDataConnection.cs:263-267`).
|
||||
- **Single-tag methods are KEPT**, implemented as batch-of-one delegations. They have real callers outside the hot path: the OPC UA heartbeat monitor (`OpcUaDataConnection.cs:145`), `WriteBatchAndWaitAsync`'s poll loop (line 355), `HandleWrite`/`HandleReadTagValues`. Removing them churns every test fake for zero gain; the actor's hot paths simply stop using them.
|
||||
|
||||
## 2. OPC UA: sharding + apply serialization
|
||||
|
||||
- **Budget option:** `OpcUaEndpointConfig.MaxMonitoredItemsPerSubscription`, default **5000**, flowing into `OpcUaConnectionOptions` like the other subscription knobs (`OpcUaDataConnection.cs:74-96`). Per-endpoint, not global — item-count ceilings are a property of the target server. 37,500 tags → 8 shards.
|
||||
- `RealOpcUaClient` replaces the single `_subscription` (`RealOpcUaClient.cs:22,150-162`) with a shard list. Placement: first shard with free capacity, else create a new shard from the same options; a shard is deleted when it empties. A `handle → shard` map serves removal and keeps `RemoveSubscriptionsAsync` grouping one `ApplyChangesAsync` per touched shard.
|
||||
- **Alarm (A&C) monitored items get a dedicated event shard.** `TriggerConditionRefreshAsync` passes a subscription id (`RealOpcUaClient.cs:784-786`); pinning event items to one shard keeps that bookkeeping trivial and keeps `QueueSize:1000` event items from occupying data-shard capacity.
|
||||
- **Batch subscribe = N×`AddItem` + one `ApplyChangesAsync` per touched shard per chunk** (today it is one Apply per tag — `RealOpcUaClient.cs:494-495`, the core of finding #3). New `IOpcUaClient` members: `CreateSubscriptionsAsync`, `RemoveSubscriptionsAsync`, `ReadValuesAsync`, `WriteValuesAsync`, all with per-item results.
|
||||
- **All `ApplyChangesAsync`/`CreateAsync`/`DeleteAsync` calls serialize behind ONE `SemaphoreSlim(1,1)` per client.** The SDK Subscription is not safe under concurrent structural mutation, and today nothing prevents `HandleSubscribe`'s background task, the resolution-retry tick, and an alarm subscribe from applying concurrently. Batching makes contention rare; the single lock (rather than per-shard) also serializes shard creation and is the simplest thing that is correct.
|
||||
|
||||
## 3. Reconnect orchestration & re-seed
|
||||
|
||||
Two distinct paths, deliberately staggered differently:
|
||||
|
||||
- **Instance-driven subscribes** (site failover): DeploymentManager already staggers instance creation (`SiteRuntimeOptions.StartupBatchSize` 20 / `StartupBatchDelayMs` 100, `SiteRuntimeOptions.cs:13-19`), so `SubscribeTagsRequest`s arrive pre-spaced at ~75 tags each. Each request becomes ONE `SubscribeBatchAsync` (a single chunk). **The DCL adds no extra delay on this path** — double-staggering would only slow failover.
|
||||
- **Adapter-level reconnect** (`ReSubscribeAll`, `DataConnectionActor.cs:1600-1684`): today it fires 37,500 independent `SubscribeAsync` tasks in a tight loop (lines 1642-1653). It becomes sequential chunks of **`SubscribeBatchSize` = 500** with **`SubscribeBatchDelay` = 50 ms** between chunks, all inside one background task carrying the captured adapter + generation; each chunk pipes one completion message (per-tag results inside) back to the actor. 75 chunks ≈ 15-25 s at realistic RTTs — versus today's unbounded task storm. Sequential (not parallel) chunks: subscription creation on one session is serialized by the apply lock anyway, and sequencing bounds device load.
|
||||
- **`SeedTagsAsync`** (`DataConnectionActor.cs:800-851`) currently reads **per tag, serially**, each with a 30 s timeout — the 30 s-per-tag worst case the plan calls out. It becomes chunked `ReadBatchAsync` calls: **`SeedReadBatchSize` = 250**, **`SeedReadMaxParallelism` = 4** concurrent chunks, `SeedReadTimeout` (existing, 30 s) now applying **per chunk**, under an **overall deadline `SeedOverallTimeout` = 120 s** enforced by one linked CTS across all chunks and retry rounds. The existing retry semantics survive unchanged in shape: still-empty tags re-read up to `SeedReadMaxAttempts` (3) with `SeedReadRetryDelay` (250 ms) between rounds; on deadline expiry the remaining tags are logged and stay Uncertain until a change notification — the exact behavior documented at `DataConnectionOptions.cs:17-25`. Chunk size stays moderate on purpose: the "some gateways time out on a large batch" caveat that motivated per-tag reads (`DataConnectionActor.cs:792-793`) is answered by chunking + per-chunk timeout, not by giving up bulk. Sizing: 150 chunks / 4-way ≈ 38 waves; ~20-40 s typical, so 120 s has honest headroom.
|
||||
- All new knobs live on `DataConnectionOptions` (validated in `DataConnectionOptionsValidator`): `SubscribeBatchSize`, `SubscribeBatchDelay`, `SeedReadBatchSize`, `SeedReadMaxParallelism`, `SeedOverallTimeout`.
|
||||
|
||||
## 4. Tag-resolution retry backoff
|
||||
|
||||
Today: fixed 10 s periodic timer, one `SubscribeAsync` task per unresolved tag per tick (`DataConnectionActor.cs:1536-1574`). A dead device with thousands of unresolved tags probes forever at full width.
|
||||
|
||||
- **Exponential backoff:** interval starts at `TagResolutionRetryInterval` (existing, 10 s), doubles per fully-failed round, capped at **`TagResolutionRetryMaxInterval` = 5 min** (new option). Reset to the floor when any tag resolves or on reconnect (`ReSubscribeAll` already clears `_unresolvedTags`, line 1617).
|
||||
- **Batched probes:** each tick issues `SubscribeBatchAsync` over the not-in-flight unresolved set, chunked at `SubscribeBatchSize` — not N single subscribes. `_resolutionInFlight` dedup survives at chunk granularity.
|
||||
- **Single-shot timer, rescheduled on probe completion** — a periodic timer cannot back off. This inherently preserves the anti-starvation property the current code defends with its `IsTimerActive` gate (`DataConnectionActor.cs:1015-1028`): the next tick is scheduled only when the previous round's completion message arrives, so a fan-out of failures can never reset the clock.
|
||||
|
||||
## 5. MxGateway strategy — no cross-repo change needed for the main path
|
||||
|
||||
Checked `~/Desktop/MxAccessGateway`: the proto **already has** the bulk family — `SubscribeBulkCommand` (AddItem + Advise per tag in one worker STA pass, one gRPC round trip; `mxaccess_gateway.proto:325`, worker `MxAccessSession.SubscribeBulk`), `UnsubscribeBulkCommand`, `AddItemBulkCommand`, `ReadBulkCommand`, `WriteBulkCommand`, all with per-item results. The **shipped `ZB.MOM.WW.MxGateway.Client` 0.2.0** — the exact version this repo pins (`Directory.Packages.props:98`) — exposes `SubscribeBulkAsync`/`AddItemBulkAsync`/`UnsubscribeBulkAsync` (verified in the packaged DLL, `dist/ZB.MOM.WW.MxGateway.Client.0.2.0.nupkg`). So:
|
||||
|
||||
- **Plain-advise mode** (a configured write-user): `SubscribeBatchAsync` = chunked `SubscribeBulkAsync` — the 2-RPC-per-tag subscribe (`RealMxGatewayClient.cs:93-103`: `AddItemAsync` + `AdviseAsync`) collapses to one RPC per 500 tags. Unsubscribe = `UnsubscribeBulkAsync`.
|
||||
- **Supervisory mode** (`WriteUserId == 0`, `RealMxGatewayClient.cs:52`): the worker has **no bulk supervisory advise** (`SubscribeBulk` issues plain `Advise`; only single `AdviseSupervisory` exists). Fallback per plan: one `AddItemBulkAsync` for the chunk, then client-side **pipelined** per-item `AdviseSupervisory` commands with bounded parallelism **`MxSupervisoryAdviseParallelism` = 16** (new option) — 37,500 tags ≈ 2,344 in-flight windows instead of 75,000 serial round trips. The existing `Lazy<Task>` advise-once map (`RealMxGatewayClient.cs:47`) is pre-populated from the bulk path so a concurrent write still awaits the same advise.
|
||||
- **Cross-repo follow-up (note, not a blocker):** add an additive `bool supervisory` field to `SubscribeBulkCommand` (or an `AdviseSupervisoryBulkCommand`) in mxaccessgw so supervisory sites also get one-RPC subscribe. File against mxaccessgw; ScadaBridge picks it up on the next client bump with a capability/`Unimplemented` fallback to the pipelined path.
|
||||
- Read/Write are already true bulk in this adapter — no change.
|
||||
|
||||
## 6. Quality-counter flush
|
||||
|
||||
`HandleTagValueReceived` calls `_healthCollector.UpdateTagQuality` on **every** value received, even when quality is unchanged (`DataConnectionActor.cs:1811`). New policy: in-actor bucket math stays per-message (cheap), but the collector push is **transition-gated + coalesced** — skip entirely when `prevQuality == newQuality`; on a genuine transition set a dirty flag and flush on a **`QualityFlushInterval` = 1 s** single-shot timer. Health reports poll at 30 s, so 1 s coalescing loses nothing. **Immediate synchronous flushes are retained** where correctness depends on them: `PushBadQualityForAllTags` on disconnect (line 1595), unsubscribe (line 1135), and the `ReSubscribeAll` reset (line 1635).
|
||||
|
||||
## 7. Alarm subscriber prefix index + gateway union filter
|
||||
|
||||
- **Index:** `HandleAlarmTransitionReceived` scans every subscribed source and does two `StartsWith` per source per transition (`DataConnectionActor.cs:1974-1979`). Replace with buckets keyed by the source reference's **first path segment** (split on `.`): lookup extracts the first segment of `SourceObjectReference`/`SourceReference` and tests only that bucket. Sources whose reference contains no separator (a prefix shorter than one full segment) go to a small linear residue list — exact semantics preserved, including the per-source condition-type gate and the `SnapshotComplete` broadcast-to-all (lines 1965-1972), which bypasses the index entirely.
|
||||
- **Gateway union filter:** the MxGateway alarm stream is opened **gateway-wide** today (`alarmFilterPrefix: null`, `MxGatewayDataConnection.cs:231-236`). `StreamAlarmsRequest` carries a single `alarm_filter_prefix` (`mxaccess_gateway.proto:986`), so the pushable union is the **longest common prefix** of the active source references. The adapter recomputes the LCP on subscribe; the stream restarts (cancel + reopen — the source replays a fresh snapshot, which `NativeAlarmActor` already handles) **only when a new source falls outside the current prefix**. Unsubscribes never restart — a too-broad prefix is only bandwidth. The prefix stays a bandwidth optimisation, never correctness: the actor's per-source + condition-type gate remains authoritative, mirroring the OPC UA WhereClause stance (`RealOpcUaClient.cs:659-662`).
|
||||
|
||||
## 8. Behaviors that MUST be preserved (stage-b checklist)
|
||||
|
||||
1. **Become/Stash lifecycle** — Connecting/Connected/Reconnecting states, stash rules per state (`DataConnectionActor.cs:249-550`) unchanged.
|
||||
2. **Generation fencing** — batch callbacks capture `_adapterGeneration` exactly as the per-tag closures do (`:697,1637`); `TagValueReceived`/`AlarmTransitionReceived` still drop stale generations (`:1771,1949`). Chunked reconnect runs against the adapter captured at chunk-dispatch time (S7 discipline, `:698-704`).
|
||||
3. **Immediate bad quality on disconnect** — `PushBadQualityForAllTags` stays synchronous in `BecomeReconnecting` (`:473`), never deferred to the quality-flush timer.
|
||||
4. **Transparent re-subscribe** — derived from `_subscriptionsByInstance` (durable truth, `:1602-1607`), `_instancesByTag` NOT cleared (`:1612-1615`), reconnect re-seed included (`:1655-1683`).
|
||||
5. **Write failures synchronous to the calling script** — `HandleWrite`/`HandleWriteBatch` `PipeTo` with `WriteTimeout` translation unchanged (`:1141-1242`).
|
||||
6. **Seed-after-registration** — seeds still ride `SubscribeCompleted` and are delivered only after `_subscriptionsByInstance` registration (the static-tag fix, `:765-782,999-1012`); only resolved tags are seeded.
|
||||
7. **In-flight orphan guards** — unsubscribe-during-subscribe releases adapter handles (`:860-904`), duplicate alarm feeds released not overwritten (`:1908-1932`); batch completions apply per-tag through the same guard logic.
|
||||
8. **Unresolved-tag semantics** — Bad-quality `TagValueUpdate` signal (`:991-995`), `_subscribesInFlight`/`AlreadySubscribed` dedup (`:706-729`), counter promotion rules (`:938-996`).
|
||||
9. **Alarm filter last-writer-wins warning** (`:1841-1848`) and the client-side gate as sole authority.
|
||||
|
||||
## 9. Stage-b test plan (fake-client)
|
||||
|
||||
Existing DCL suite (`tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/` — actor lifecycle, generation fencing, alarm routing) must stay green. New pins:
|
||||
|
||||
- **One apply per chunk:** fake `IOpcUaClient` counts `ApplyChangesAsync` calls — N tags through `SubscribeBatchAsync` → `ceil(N/SubscribeBatchSize)` applies, not N.
|
||||
- **Sharding:** 12,001 tags with budget 5,000 → 3 shards; removal empties → shard deleted; alarm item lands on the event shard and `ConditionRefresh` targets it.
|
||||
- **Apply serialization:** fake apply with an injected delay + concurrency flag — concurrent batch subscribe / unsubscribe / alarm subscribe never overlap an apply.
|
||||
- **Re-seed deadline:** fake `ReadBatchAsync` that hangs → seed returns at `SeedOverallTimeout`, remaining tags pending/Uncertain, `SubscribeCompleted` still delivered and acked.
|
||||
- **Backoff caps:** all-fail probe rounds produce intervals 10 s, 20 s, 40 s … capped at `TagResolutionRetryMaxInterval` (compressed values in-test); any success resets to the floor; probes are batched (one client call per chunk, not per tag).
|
||||
- **MxGateway modes:** plain mode → exactly one `SubscribeBulkAsync` per chunk; supervisory mode → one `AddItemBulkAsync` + per-item advises with in-flight count ≤ `MxSupervisoryAdviseParallelism`.
|
||||
- **Partial failure surface:** a per-tag failure row → `_unresolvedTags` + Bad-quality push, response `Success:true`; a thrown batch → all-tags connection-level → `SubscribeTagsResponse(Success:false)` + Reconnecting.
|
||||
- **Quality flush:** M updates at unchanged quality → zero collector calls; one transition → one flush within the interval; disconnect flush still immediate.
|
||||
- **Alarm index equivalence:** routing decisions identical to the linear scan over a sample set including shared-prefix sources, sub-segment prefixes (residue list), and the empty-source `SnapshotComplete` broadcast. LCP stream restarts only when a new source escapes the current prefix.
|
||||
|
||||
## 10. Defaults summary
|
||||
|
||||
| Option | Home | Default |
|
||||
|---|---|---|
|
||||
| `MaxMonitoredItemsPerSubscription` | `OpcUaEndpointConfig` | 5000 |
|
||||
| `SubscribeBatchSize` | `DataConnectionOptions` | 500 |
|
||||
| `SubscribeBatchDelay` | `DataConnectionOptions` | 50 ms |
|
||||
| `SeedReadBatchSize` | `DataConnectionOptions` | 250 |
|
||||
| `SeedReadMaxParallelism` | `DataConnectionOptions` | 4 |
|
||||
| `SeedOverallTimeout` | `DataConnectionOptions` | 120 s |
|
||||
| `TagResolutionRetryMaxInterval` | `DataConnectionOptions` | 5 min |
|
||||
| `QualityFlushInterval` | `DataConnectionOptions` | 1 s |
|
||||
| `MxSupervisoryAdviseParallelism` | `DataConnectionOptions` | 16 |
|
||||
|
||||
Existing options keep their meanings; `SeedReadTimeout` (30 s) becomes per-chunk rather than per-tag.
|
||||
@@ -0,0 +1,135 @@
|
||||
# Arch-review remediation — execution log
|
||||
|
||||
**Ran:** 2026-08-14/15 · **Plan:** `docs/plans/2026-08-14-arch-review-remediation-plan.md` ·
|
||||
**Branch:** `arch-review-remediation` (off `main`) · **Status:** code-complete, docs-propagated,
|
||||
not yet pushed/merged.
|
||||
|
||||
Source: the 2026-08-14 eight-pass performance/architecture review (all 27 components +
|
||||
`ZB.MOM.WW.LocalDb`). This log is the compact record of what shipped, how it was gated, and
|
||||
what's left. Full package specs live in the plan; this is the after-action summary.
|
||||
|
||||
## Phases and package → commit map
|
||||
|
||||
Every package ran as an isolated-worktree subagent, merged `--no-ff` onto
|
||||
`arch-review-remediation`; commit hashes below are the substantive commit on the branch (merge
|
||||
commits omitted).
|
||||
|
||||
**Phase 0 — Preflight.** `0b201e41` docs(plans): the remediation plan itself.
|
||||
|
||||
**Phase 1 — Quick wins (7 packages, parallel).**
|
||||
| WP | Commit | What |
|
||||
|---|---|---|
|
||||
| 1.1 | `34a3f4bb` | Reconnect on graceful (OK-status) stream completion — closes the 4h silent stream death |
|
||||
| 1.2 | `2e4e41a8` | Site audit DB onto the mounted data volume; required path + soft flush |
|
||||
| 1.3 | `7ebdcd37` | CDC capture installed only when replication is configured |
|
||||
| 1.4 | `600659d5` | Sweep/KPI covering indexes + sliced notification terminal purge |
|
||||
| 1.5 | `125055d9` | O(1) attribute resolution, precomputed types, coalesced static writes, shared JSON options |
|
||||
| 1.6 | `c5e66ed4` | Fail known-dead sends immediately instead of burning Ask timeouts |
|
||||
| 1.7 | `2cfcd890` | Batched event-log commits + sliced retention purge |
|
||||
|
||||
**Phase 2 — Seam rework (6 packages, parallel after the WP2.1a design doc `1040dc0f`).**
|
||||
| WP | Commit | What |
|
||||
|---|---|---|
|
||||
| 2.1 | `d15c5f02` | DCL batch subscribe/read/write seam, bounded reconnect, sharded subscriptions |
|
||||
| 2.2 | `5db2a810` | Central set-based ingest, aligned partition purge, KPI query shapes, EF hygiene |
|
||||
| 2.3 | `2ce0ad7e` | Alarms-only stream seed, capped buffers, at-least-once audit pull |
|
||||
| 2.4 | `8c0b36b2` | Shared KPI cache, live-cache-backed alarm summary, coalesced Debug View renders |
|
||||
| 2.5 | `48b3c40a` | Flatten-session caching, bulk `DeploySiteAsync`, paged management queries |
|
||||
| 2.6 | `a2122831` | Cached hot-path lookups, bounded observer queue, alarm-priority stream path |
|
||||
|
||||
Plus `a5882753` closing Phase-2-gate residuals (direct ingest path, monotonic timeouts, synthetic
|
||||
probe, not-reporting set, cursor-exact audit pull) found while gating.
|
||||
|
||||
**Phase 3 — Structural (design-first).**
|
||||
| WP | Design | Commit | What |
|
||||
|---|---|---|---|
|
||||
| 3.1 | `312216ff` | `c4fc1f8e` | Script execution pool split — trigger evals off the blocking pool, bounded/deadline-aware execution |
|
||||
| 3.2 | `6cfb2dd8` | `c254d074` | `site_events` volume policy — sampled per-run events, interval summaries, replication pinned |
|
||||
| 3.3 | (scadaproj) | `cca7f178` + scadaproj `9377fa1` | LocalDb 0.2.0 — dereg cleanup, late-opt-in baselining, byte-budget replication |
|
||||
|
||||
**Phase 4 — Verification, adversarial review, docs.** Six parallel `code-reviewer` passes over
|
||||
the full diff, one per area (site runtime, DCL, comms, site persistence, central SQL, UI/deploy),
|
||||
each instructed to try to refute the fixes. Confirmed findings landed as targeted follow-ups:
|
||||
|
||||
| Area | Commit | What |
|
||||
|---|---|---|
|
||||
| Site runtime | `950c54c5` | Recursion-exempt run cap (nested `CallScript` no longer double-gated), atomic detach counter, summary edge cases, per-row event-log fallback |
|
||||
| DCL | `37f13e2e` | Discard in-flight subscribe results for tags unsubscribed mid-flight; release the orphaned handle |
|
||||
| Central SQL | `5d075f13` | No client-side audit truncation, insert-first upsert (Site Call Audit), QI-safe filtered-index scripts, honest operator-not-found replies |
|
||||
| Comms | `fd5e023d` | Consumer-based debug-stream orphan net, foreign-cancel triad, honest `onConnected`, served-row-exact retirement, full-rate reconcile |
|
||||
| UI/deploy | `e0e4b246` | Honest CLI HTTP timeouts, watermark-complete staleness (3 missed bump sites), phase-2 `PendingDeployment` staging, lock-safe cancellation |
|
||||
| Site persistence | `56c99c92` + `f689f495` | Required audit DB path on wonder; explicit `MaxBatchSize:64` LocalDb read-page cap; rate-limited observer drop logging; LocalDb 0.2.1 (HLC anchor flush on dereg, bounded 64-message sync inbox) |
|
||||
|
||||
## Gate results
|
||||
|
||||
- **Baseline:** 7587 tests green at Phase 0 entry; test count never reduced across any phase gate.
|
||||
- **Live probes (rig):**
|
||||
- Stream lifetime forced to 2 minutes — the alarm stream reconnected in **8.5s**, within one
|
||||
reconcile tick of the OK completion, `IsLive` correctly reflected the gap (WP1.1).
|
||||
- CDC conditional-registration: site-b booted clean on LocalDb 0.2.0 with **30 stale triggers
|
||||
dropped** at startup (WP1.3/WP3.3) — confirms the self-heal path fires, not just the steady
|
||||
state.
|
||||
- S&F due-sweep: `EXPLAIN QUERY PLAN` confirmed index-terminated (WP1.4).
|
||||
- Failover drill (`docker/failover-drill.sh`) unaffected by the actor/timeout changes.
|
||||
- **Adversarial review tally:** ~25 confirmed findings across the six areas, **4 High**, all
|
||||
fixed in the Phase 4 commits above. Zero findings deferred as won't-fix.
|
||||
- **Test-flake root causes (3, all test-side, not production bugs):**
|
||||
1. `cfa6acbf` — an assertion on `MarkForwarded` ran before the push it depended on was
|
||||
guaranteed to have landed; reordered behind the push.
|
||||
2. `c4caebe9` — two dispatcher audit-safety tests asserted an attempt count without
|
||||
synchronizing on the async write that produced it; the unsynchronized assertion was removed.
|
||||
3. `950c54c5` (embedded) — `ScriptDeadlineAtEnqueueTests`' "no started event" assertion went
|
||||
vacuous once WP3.2 flipped `PerRunScriptEvents` to off-by-default (fixed by opting the test
|
||||
back in); `ScriptRunLauncherParityTests` widened an `ExpectMsg` window that would have passed
|
||||
for any deadline from 1s to 300s, not just the intended one (sharpened to assert the reported
|
||||
timeout value AND a tight wall-clock range).
|
||||
- **Follow-up recommended, not done here:** a suite-wide sweep for the same
|
||||
`AwaitAssert(...)`-then-bare-`Assert` pattern — an `AwaitAssert` that only proves "eventually
|
||||
true," followed by a plain assertion that silently inherits its timing slack, is the shape
|
||||
behind all three; worth a grep-and-review pass rather than fixing on-demand as flakes surface.
|
||||
|
||||
## Residuals register
|
||||
|
||||
Deliberately not fixed in this program — each has a stated reason, not an oversight:
|
||||
|
||||
1. **DCL unsubscribe-during-reconnect count staleness.** The `37f13e2e` fix discards orphaned
|
||||
in-flight results but a per-connection counter can still drift under rapid
|
||||
subscribe/unsubscribe churn during a reconnect; needs a per-tag counted set. Low severity,
|
||||
cosmetic (a health-report number), deferred.
|
||||
2. **Per-table `needs_snapshot` in LocalDb.** Baselining one table currently re-streams every
|
||||
registered table in both directions. Narrowing it needs an on-disk schema change LocalDb 0.2.1
|
||||
deliberately avoided (wire/schema compatibility). Documented as a follow-up in the library's
|
||||
own README and `RegisterReplicated` remarks.
|
||||
3. **Event batching per proto message.** Individual `AttributeValueChanged`/`AlarmStateChanged`
|
||||
events still ride one gRPC message each; batching them is a new wire shape (proto + both
|
||||
client/server), deferred rather than folded into this program's additive-only changes.
|
||||
4. **Deployments page server-side paging + status counts.** Still client-materializes the full
|
||||
list; out of scope for this pass (WP2.5 touched the deploy pipeline, not this specific UI
|
||||
surface).
|
||||
5. **OtOpcUa still pins LocalDb 0.1.3.** A supported skew — 0.1.x peers sync with 0.2.x under the
|
||||
library's wire-compatibility guarantee — not a blocker for this program.
|
||||
6. **Fragile `SandboxTests` timing pin.** Pre-existing, unrelated to this remediation's changes;
|
||||
noted so it isn't mistaken for a regression if it flakes later.
|
||||
7. **Target-scale load test (deferred-work register #25).** This program's exit criterion is the
|
||||
live probes above, not #25 — #25 remains the follow-on validation that the moved ceilings hold
|
||||
under real load; schedule separately.
|
||||
8. **Playwright 14 pre-existing env failures.** Present on `main` too, rig-state related, not
|
||||
introduced by this branch.
|
||||
9. **`site_events` retention purge still oplog-visible.** WP3.2's sliced retention DELETE is a row
|
||||
change like any other and is captured by CDC on a replicated site (site-a) — correct per the
|
||||
"CDC does all three jobs" design (no separate resync path to gate), but means a purge burst is
|
||||
visible in the oplog/backlog metrics; not a correctness issue, just a metrics-reading note for
|
||||
operators watching `LocalDbOplogBacklog` during a purge window.
|
||||
|
||||
## Docs propagated
|
||||
|
||||
Component docs (`DataConnectionLayer`, `SiteRuntime`, `Communication`, `AuditLog`,
|
||||
`SiteEventLogging`, `StoreAndForward`, `NotificationOutbox`, `SiteCallAudit`, `TemplateEngine`,
|
||||
`DeploymentManager`, `CentralUI`, `ConfigurationDatabase`), CLAUDE.md Key Design Decisions
|
||||
(stream-completion reconnect, required site audit DB path, CLI HTTP timeout honesty, bulk
|
||||
`DeploySiteAsync`, LocalDb 0.2.1), and `docs/known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md`
|
||||
(amended for the `MaxBatchSize:64` read-page pin superseding the "retired, left at defaults" note).
|
||||
No components added/removed; README component table unchanged. Cross-reference sweep found no
|
||||
stale live references to `ScriptExecutionActor`/`AlarmExecutionActor` or `UX_AuditLog_EventId`
|
||||
outside historical plan/known-issue records, after correcting four residual mentions in
|
||||
`docs/requirements/Component-SiteRuntime.md` left over from the WP3.1 doc pass.
|
||||
@@ -0,0 +1,412 @@
|
||||
# Script Pool Split — Design Memo (WP3.1 stage a)
|
||||
|
||||
**Date:** 2026-08-15 · **Status:** APPROVED FOR IMPLEMENTATION (WP3.1 stage b)
|
||||
**Source:** `docs/plans/2026-08-14-arch-review-remediation-plan.md` §5 WP3.1 — findings #4 (High:
|
||||
trigger evals starve behind blocking scripts on one fixed 8-thread pool) + actor-per-execution
|
||||
overhead and the `SiteScriptCompileCache` wholesale-Clear cliff (Med).
|
||||
**Scope:** design only. Implementation, tests, and the `Component-SiteRuntime.md` /
|
||||
CLAUDE.md doc updates land in stage (b).
|
||||
|
||||
---
|
||||
|
||||
## 0. Current machinery (evidence baseline)
|
||||
|
||||
All line numbers verified 2026-08-15 on `arch-review-remediation` (post-Phase-2, tip `a5882753`).
|
||||
|
||||
| Mechanism | Where |
|
||||
|---|---|
|
||||
| One fixed pool for everything: `ScriptExecutionScheduler`, N dedicated threads, unbounded FIFO `BlockingCollection` | `SiteRuntime/Scripts/ScriptExecutionScheduler.cs:21-84` |
|
||||
| Pool size: `SiteRuntimeOptions.ScriptExecutionThreadCount`, static default 8, never scales | `SiteRuntime/SiteRuntimeOptions.cs:46` |
|
||||
| Script bodies run on the pool via short-lived `ScriptExecutionActor` (spawn-execute-die, ctor kicks the task) | `Actors/ScriptExecutionActor.cs:79-87,130,339` |
|
||||
| Alarm on-trigger bodies: same pattern, `AlarmExecutionActor` | `Actors/AlarmExecutionActor.cs:100-165` |
|
||||
| ScriptActor Expression-trigger eval: `Task.Factory.StartNew(..., scheduler)` on the SAME pool, hardcoded 2 s CTS created **inside** the body (timeout excludes queue wait) | `Actors/ScriptActor.cs:297-331` (2 s at :314) |
|
||||
| AlarmActor Expression-trigger eval: identical, plus `SourceExecutionId` snapshot capture | `Actors/AlarmActor.cs:578-619` (2 s at :600, capture at :591) |
|
||||
| Per-trigger coalescing (one in flight + one pending) already bounds eval fan-out per actor | `ScriptActor.cs:75-76,300`; `AlarmActor.cs:102-103,581` |
|
||||
| Script timeout CTS created inside the pooled body ⇒ the 30 s script budget also starts at **dequeue**, not enqueue | `ScriptExecutionActor.cs:136`; `AlarmExecutionActor.cs:102` |
|
||||
| Stuck-script watchdog: names the script, logs + site event — **observability only, the thread stays lost** | `ScriptExecutionActor.cs:147-161`; grace `SiteRuntimeOptions.cs:79` |
|
||||
| Pool gauges (queue depth / busy / oldest-busy age) → health report | `ScriptExecutionScheduler.cs:94-127`; `Scripts/ScriptSchedulerStatsReporter.cs:75-91` |
|
||||
| No per-script concurrency bound: every trigger spawns another child actor | `ScriptActor.cs:463-501`; `AlarmActor.cs:725-758` |
|
||||
| Compile cache: 1024 entries, **wholesale `Clear()` on overflow** | `Scripts/SiteScriptCompileCache.cs:37,71-72` |
|
||||
| Compile call sites on actor threads: deploy gate (deliberately synchronous, S3) and InstanceActor start | `Actors/DeploymentManagerActor.cs:522-556` (Validate at :540); `Actors/InstanceActor.cs:1631,1682,1795` |
|
||||
| In-repo off-dispatcher-compile reference: shared-script load = `Task.Run` + `PipeTo(Self)` | `DeploymentManagerActor.cs:1540-1563` |
|
||||
| Akka HOCON has no script dispatcher — the pool lives entirely outside Akka (only `audit-telemetry-dispatcher` is custom) | `Host/Actors/AkkaHostedService.cs:296-302` |
|
||||
|
||||
The failure mode (finding #4): 8 scripts blocked in synchronous I/O ⇒ every Expression trigger
|
||||
eval on the node (scripts **and alarms**) queues behind them. An alarm that should raise in
|
||||
milliseconds waits until a thread frees — unbounded. The 2 s eval CTS doesn't even start until
|
||||
the eval is dequeued, so the operator sees neither a timeout nor a raise.
|
||||
|
||||
---
|
||||
|
||||
## 1. (a) Trigger/expression evals leave the blocking pool
|
||||
|
||||
**Decision (confirms the plan default): evals run as plain async on the .NET thread pool
|
||||
(`Task.Run`), gated by a process-wide `SemaphoreSlim`. No second dedicated pool.**
|
||||
|
||||
Justification: trigger expressions are non-blocking **by construction** —
|
||||
`TriggerExpressionGlobals` exposes only reads over an in-memory snapshot dictionary ("no I/O,
|
||||
no actor Ask, no side-effecting APIs", `Scripts/TriggerExpressionGlobals.cs:5-13,42-50`), and
|
||||
the script trust gate has already denied I/O/network/threading before the expression ever
|
||||
deploys. They are short CPU-bound work — exactly what the shared thread pool is for. A second
|
||||
dedicated pool would add threads, gauges, and a second starvation surface for no isolation gain;
|
||||
the gate alone prevents an eval storm from swamping the pool.
|
||||
|
||||
Mechanics (both `ScriptActor.StartExpressionEvaluation` and `AlarmActor.StartExpressionEvaluation`):
|
||||
|
||||
```csharp
|
||||
var cts = new CancellationTokenSource(evalTimeout); // clock starts AT ENQUEUE
|
||||
Task.Run(async () =>
|
||||
{
|
||||
try
|
||||
{
|
||||
await gate.WaitAsync(cts.Token); // queue wait burns the same budget
|
||||
try
|
||||
{
|
||||
var state = await expression.RunAsync(new TriggerExpressionGlobals(snapshot), cts.Token);
|
||||
return state.ReturnValue is bool b && b;
|
||||
}
|
||||
finally { gate.Release(); }
|
||||
}
|
||||
catch (Exception ex) { /* existing treated-as-false path, incl. OCE */ }
|
||||
finally { cts.Dispose(); }
|
||||
}).PipeTo(self, success: r => new ExpressionEvalResult(r, ...), failure: ex => new ExpressionEvalFailed(ex, ...));
|
||||
```
|
||||
|
||||
- **Queue-inclusive timeout falls out of CTS placement**: today the CTS is constructed inside
|
||||
the pooled body (`ScriptActor.cs:314`, `AlarmActor.cs:600`), so its 2 s starts at dequeue.
|
||||
Moving construction to the actor thread before `Task.Run` makes gate-wait time part of the
|
||||
budget; a cancellation during `WaitAsync` flows through the existing catch → `false` → the
|
||||
existing error/edge handling (`HandleExpressionEvalResult` / `HandleExpressionEvalFailed`,
|
||||
which already survive faults per N2). No new message types.
|
||||
- **Gate**: one process-wide `SemaphoreSlim` (mirroring the `ScriptExecutionScheduler.Shared`
|
||||
lazy-singleton + injectable-seam pattern, `ScriptExecutionScheduler.cs:48-60` / the #18 ctor
|
||||
seams). Size: **`TriggerEvalMaxConcurrency`, default `max(2, Environment.ProcessorCount)`**.
|
||||
Per-trigger coalescing (unchanged) already caps waiters at ≤ one per Expression trigger, so
|
||||
the gate queue is bounded by trigger count.
|
||||
- **Timeout becomes configurable**: `TriggerEvalTimeoutSeconds`, default 2 (today hardcoded in
|
||||
two places). Validated > 0.
|
||||
- The tag-cascade capture is untouched: `sourceExecutionId` is still captured on the actor
|
||||
thread with the snapshot (`AlarmActor.cs:584-591`) and echoed on the result records
|
||||
(`AlarmActor.cs:888,899`) — only the execution context under the lambda changes.
|
||||
- `ScriptExecutionScheduler` keeps its `TryExecuteTaskInline` name-prefix guard and thread
|
||||
naming; nothing about the blocking pool's identity changes in this section.
|
||||
|
||||
Net effect: the starvation scenario is dead by construction — evals never share a queue or a
|
||||
thread with blocking script bodies.
|
||||
|
||||
---
|
||||
|
||||
## 2. (b) Blocking-pool sizing, config, watchdog thread replacement
|
||||
|
||||
### Sizing
|
||||
|
||||
**Decision (confirms plan default): `threads = clamp(max(ScriptExecutionThreadCount, ceil(enabledInstances / 8)), 1, ScriptExecutionMaxThreadCount)`.**
|
||||
|
||||
- `ScriptExecutionThreadCount` (existing, default 8, `SiteRuntimeOptions.cs:46`) keeps its name
|
||||
and becomes the **floor** — existing configs keep exactly today's behavior at ≤ 64 instances.
|
||||
- New `ScriptExecutionMaxThreadCount`, default **32** — a 1 MB-stack thread is cheap; 32 covers
|
||||
256 instances at the /8 ratio, and beyond that the per-script cap (§3) is the real regulator.
|
||||
Validated ≥ `ScriptExecutionThreadCount`.
|
||||
- The `/8` divisor is a named constant (`InstancesPerScriptThread = 8`), not another option —
|
||||
no evidence any deployment needs to tune the ratio independently of floor and ceiling.
|
||||
- **Grow-only.** `ScriptExecutionScheduler` gains `EnsureCapacity(int target)`: starts
|
||||
additional workers up to `target` (idempotent, lock-guarded, widens `_busySinceTicks`
|
||||
bookkeeping); it never shrinks — undeploying instances leaves idle threads, which cost
|
||||
nothing measurable and avoid drain/steal complexity.
|
||||
- **Call site**: `DeploymentManagerActor.UpdateInstanceCounts()` (`DeploymentManagerActor.cs:2095`,
|
||||
already invoked on every deploy/undeploy/enable/disable and per startup batch at :439, :485,
|
||||
:578, :645, :666, and `:485` per staggered batch) additionally calls
|
||||
`scheduler.EnsureCapacity(ComputeTargetThreads(enabled, options))`. `ComputeTargetThreads`
|
||||
is a pure static — unit-testable without threads.
|
||||
|
||||
### Watchdog replaces lost threads
|
||||
|
||||
Today the watchdog (CTS-register + grace delay, `ScriptExecutionActor.cs:147-161`) only *names*
|
||||
the wedged script; the pool permanently loses the thread. New mechanics:
|
||||
|
||||
1. **Slot capture at body start.** `WorkerLoop` (`ScriptExecutionScheduler.cs:129-150`) exposes
|
||||
`[ThreadStatic] internal static int? CurrentWorkerSlot`. The first synchronous segment of the
|
||||
script body records `(slot, busySinceTicks)` into the run's state. A truly wedged (blocking)
|
||||
script never leaves that thread, so the pair identifies its worker exactly; an async script
|
||||
that hopped threads will fail the guard below and cause no detach (correct — it holds no
|
||||
worker).
|
||||
2. **Detach on watchdog fire.** When the existing watchdog declares the script stuck (timeout +
|
||||
`StuckScriptGraceMs`), it additionally calls
|
||||
`scheduler.TryDetachWorker(slot, observedBusySince)`: atomically, if the slot's current
|
||||
`_busySinceTicks` still equals the recorded value (same task still running), mark the slot
|
||||
**detached** and start one replacement worker thread. The detached worker, on eventually
|
||||
finishing its task, sees its flag and exits instead of pulling more work — capacity never
|
||||
silently doubles-and-drains.
|
||||
3. **Bound.** Live detached threads are capped at the current pool size (worst case the process
|
||||
briefly holds 2× threads, all but N wedged). At the cap: no replacement, log Error + site
|
||||
event `script`/`Error` ("script-execution pool has N detached stuck threads; not replacing")
|
||||
— bounded starvation is preferable to unbounded thread growth when scripts wedge en masse.
|
||||
4. **Observability.** New gauge `DetachedScriptThreads` alongside the existing
|
||||
queue/busy/oldest-age stats through `ISiteHealthCollector.SetScriptSchedulerStats`
|
||||
(`ScriptSchedulerStatsReporter.cs:80-83` extends by one argument; additive on the health
|
||||
report DTO).
|
||||
|
||||
The watchdog itself (grace, message text, site event) is otherwise unchanged, and it keeps
|
||||
running on the shared thread pool, never on the possibly-saturated script pool
|
||||
(`ScriptExecutionActor.cs:146` comment holds).
|
||||
|
||||
### Script-body deadline captured at enqueue
|
||||
|
||||
Same CTS relocation as §1: the run's deadline CTS (30 s default / per-script override,
|
||||
`ScriptExecutionActor.cs:113-116,136`) is created **before** the body is queued to the
|
||||
scheduler, so queue wait consumes the script's own budget. Additionally, the launcher records
|
||||
`deadline = enqueueUtc + timeout`; if the body dequeues with the deadline already passed it
|
||||
**skips execution entirely** and takes the existing timeout path (site event + `IncrementScriptError`
|
||||
+ error reply + `ScriptExecutionCompleted(false)`) — a saturated pool now sheds stale work at
|
||||
dequeue instead of running it late. The watchdog registers on the same token, so its grace
|
||||
window is deadline-anchored too; a run cancelled while still queued flips `completed` before the
|
||||
grace check and is correctly not reported as a stuck thread.
|
||||
|
||||
---
|
||||
|
||||
## 3. (c) Per-script in-flight cap + shed policy
|
||||
|
||||
**Decision (confirms plan default): cap = 4 concurrent runs per script; overflow sheds the
|
||||
NEWEST (the incoming trigger), with a counter always and a rate-limited site event.**
|
||||
Justification stands as the plan gave it: a trigger-driven run that cannot even start before
|
||||
its own deadline is stale; keeping the 4 oldest (already queued/running, closest to their
|
||||
deadlines… and already charged against them) and refusing the newcomer is the policy that never
|
||||
reorders runs and needs no queue at all — the "bounded queue" in the plan collapses to a
|
||||
counter, because the scheduler's own FIFO *is* the queue and in-flight = enqueued-or-running.
|
||||
|
||||
Mechanics (identical in `ScriptActor` and `AlarmActor`):
|
||||
|
||||
- **Counter**: private `_runsInFlight` on the owning actor, incremented at launch
|
||||
(`SpawnExecution` / `SpawnAlarmExecution`), decremented on the **existing completion
|
||||
messages** — `ScriptActor.ScriptExecutionCompleted` (`ScriptActor.cs:648`, handled at :503)
|
||||
and `AlarmActor.AlarmExecutionCompleted` (`AlarmActor.cs:878`, handled at :221). Both are
|
||||
emitted on every path (success/timeout/failure) today and remain so under §4; the launch-path
|
||||
try/catch (§4, behavior 2) guarantees a completion message even when the launch itself
|
||||
throws, so the counter cannot leak.
|
||||
- **Config**: `MaxConcurrentRunsPerScript`, default 4, validated ≥ 1. One knob for both script
|
||||
and alarm on-trigger runs.
|
||||
- **Shed action** when `_runsInFlight >= cap`:
|
||||
- increment new `ISiteHealthCollector.IncrementScriptRunShed()` — surfaced on the site health
|
||||
report as a raw per-interval count like the existing script/alarm error counters
|
||||
(`ISiteHealthCollector.cs:16,21` pattern);
|
||||
- site event: channel `script`, severity `Warning`, source `ScriptActor:<name>` /
|
||||
`AlarmActor:<name>`, message naming script, instance, and the in-flight count — **rate-limited
|
||||
to one event per script per minute** (actor-local last-emit timestamp; the counter still
|
||||
counts every shed) so a hot trigger cannot flood `site_events` (WP3.2's concern);
|
||||
- **Ask callers are never silently dropped**: a shed `ScriptCallRequest`
|
||||
(`ScriptActor.cs:223-242`) replies
|
||||
`ScriptCallResult(correlationId, false, null, "shed: N runs already in flight")` — a nested
|
||||
`CallScript` or inbound-API route fails fast instead of hanging to Ask-timeout;
|
||||
- trigger-driven spawns (interval/value-change/conditional/expression, alarm on-trigger) are
|
||||
simply not launched.
|
||||
- `MinTimeBetweenRuns` (`ScriptActor.cs:443-453`) remains the first gate; the cap is the second.
|
||||
|
||||
---
|
||||
|
||||
## 4. (d) `ScriptExecutionActor` / `AlarmExecutionActor` elimination
|
||||
|
||||
**Decision: ELIMINATE both actors.** The pre-work for this memo shows the risk is materially
|
||||
lower than the plan feared, because the actors are already almost inert shells:
|
||||
|
||||
- **Neither actor has a single `Receive` handler** — they execute from the constructor
|
||||
(`ScriptExecutionActor.cs:79-87`; `AlarmExecutionActor.cs:61-68`) and the entire lifecycle
|
||||
(timeout, DI scope, telemetry, replies) lives inside a detached `Task` the actor does not
|
||||
observe.
|
||||
- **The execution actor's `IActorRef` is never a message target.** `ScriptRuntimeContext._self`
|
||||
is stored (`ScriptRuntimeContext.cs:43,243`) and passed to child shared-script contexts
|
||||
(`:305`) but never used in a `Tell`/`Ask` anywhere in the class. Asks to the instance actor
|
||||
use temporary ask-actors. Nothing routes through the shell.
|
||||
- **No `PostStop`, no state, no stash** — cleanup is the task's own `finally`
|
||||
(`ScriptExecutionActor.cs:329-338`), which merely poisons the shell.
|
||||
|
||||
What remains is per-run actor-cell/mailbox/name-registration overhead plus a per-spawn
|
||||
expression-tree `Props.Create` (`ScriptActor.cs:476-498`) — pure cost, no semantics.
|
||||
|
||||
### Replacement shape
|
||||
|
||||
New `Scripts/ScriptRunLauncher` (static class, same project): the current
|
||||
`ScriptExecutionActor.ExecuteScript` and `AlarmExecutionActor.ExecuteAlarmScript` bodies move
|
||||
there essentially verbatim (they are already `static`), unified behind two entry points
|
||||
(`LaunchScript`, `LaunchAlarmScript`) that take `completionTarget` (the owning coordinator's
|
||||
`Self`) instead of `self`/`parent`. `ScriptActor.SpawnExecution` and
|
||||
`AlarmActor.SpawnAlarmExecution` call it directly — no `Props`, no child. This also deletes
|
||||
~150 lines of duplicated body between the two actors.
|
||||
|
||||
### Behavior map — every current behavior → its replacement
|
||||
|
||||
| # | Current behavior (provider) | Replacement under direct launch |
|
||||
|---|---|---|
|
||||
| 1 | Execution starts immediately on spawn (ctor, `ScriptExecutionActor.cs:79-87`) | `ScriptRunLauncher.LaunchScript(...)` called inline in `SpawnExecution` — same thread, same ordering |
|
||||
| 2 | **Supervision Stop-on-failure** (`ScriptActor.cs:205-217`, `AlarmActor.cs:242-254`). Reachable today ONLY via ctor throw → `ActorInitializationException` (e.g. `StartNew` on a disposed scheduler; the body's exceptions never escape its try/catch). Effect: warn-log, child stopped, coordinator unaffected; the Ask caller **hangs** (no reply is sent on this path today). | `try/catch` around the launch call: warn-log with the same message shape, `replyTo.Tell(ScriptCallResult(false, error))` (an intentional improvement — the caller no longer hangs), `Self.Tell(ScriptExecutionCompleted(false, error))` for the §3 counter. The now-childless `SupervisorStrategy()` overrides are **deleted**; ScriptActor/AlarmActor remain supervised by InstanceActor (Resume) exactly as before — a launch-path throw must not reach InstanceActor, and the catch guarantees it. |
|
||||
| 3 | Script-body exception/timeout containment (in-lambda try/catch, `ScriptExecutionActor.cs:295-328`) | Moves verbatim into the launcher — unchanged |
|
||||
| 4 | **DI scope per run**: `CreateScope` at `:198`, disposed in `finally :335` | Unchanged — scope creation/disposal stays inside the launched task, one scope per run, disposed on every path |
|
||||
| 5 | **Telemetry**: `IncrementScriptError` (`:297,:314`), started/completed/error site events (`:275-320`), stuck-script watchdog (`:147-161`) | Unchanged — all live inside the launcher body. (Started/Completed Info events become sampled under WP3.2; not this WP's concern.) |
|
||||
| 6 | **ExecutionId / ParentExecutionId**: `ScriptRuntimeContext` mints its own `ExecutionId`; `parentExecutionId` threaded at `:247` (`AlarmExecutionActor.cs:121`); shared-script child contexts chain via `CreateChildContextForSharedScript` (`ScriptRuntimeContext.cs:302-330`) | Unchanged — the launcher builds the identical context. The dead `self` ctor argument is replaced with `ActorRefs.Nobody` in stage (b) and the parameter deleted (it is stored-and-passed but never messaged). |
|
||||
| 7 | Ask-caller reply from the task (`:284,:307,:324`) — `IActorRef.Tell` off-thread | Unchanged (`Tell` is thread-safe) |
|
||||
| 8 | Completion notification `parent.Tell(ScriptExecutionCompleted/AlarmExecutionCompleted)` (`:293,:310,:327`; `AlarmExecutionActor.cs:144,151,159`) | `completionTarget.Tell(...)` with the coordinator's captured `Self` — identical delivery, and now doubles as the §3 in-flight decrement |
|
||||
| 9 | `PoisonPill` self-stop in `finally` (`:337`) | Nothing to stop — dropped. The per-run actor teardown (cell, mailbox, name unregistration) simply disappears. |
|
||||
| 10 | Per-run actor name `{script}-exec-{n}` (`ScriptActor.cs:470,500`) — visible in Akka logs | `_executionCounter` survives as a run-id inside the launcher's log messages; the actor *path* is gone. Accepted cosmetic loss; log message content is preserved. |
|
||||
| 11 | **Dead letters / stop-during-redeploy**: `Context.Stop` on the instance subtree stops the shells but does NOT cancel the detached task; its later `parent.Tell` dead-letters (dead letters are a health metric). | Parity by construction: an in-flight run still runs to completion after its ScriptActor stops, and its `completionTarget.Tell` dead-letters identically. **Explicit decision: ScriptActor/AlarmActor `PostStop` does NOT cancel in-flight CTSs** — redeploy/undeploy semantics (let running scripts finish) are unchanged. Cancelling-on-stop is a separate behavior change; if wanted later it gets its own decision record. |
|
||||
| 12 | Exec-actor mailbox on the default dispatcher | Never received a message — no replacement needed |
|
||||
| 13 | `Terminated`-based redeploy sequencing (`DeploymentManagerActor.cs:508-556` waits for the instance subtree to fully stop) | Unaffected: exec shells were stopped instantly anyway (no children of their own); subtree termination time is dominated by Instance/Script/Alarm actors, which all remain |
|
||||
|
||||
### Parity pins stage (b) MUST implement as tests
|
||||
|
||||
1. Throwing script body ⇒ ScriptActor stays alive (no restart — assert no `PreRestart`), error
|
||||
site event + `IncrementScriptError`, Ask caller gets `ScriptCallResult(false)`,
|
||||
`ScriptExecutionCompleted(false)` arrives, in-flight counter returns to 0.
|
||||
2. Launch-path throw (inject a disposed `ScriptExecutionScheduler` via the #18 seam) ⇒
|
||||
ScriptActor neither dies nor restarts, caller gets an error reply (improved over today's
|
||||
hang — pin the reply), counter returns to 0.
|
||||
3. DI scope disposed exactly once per run on success, failure, AND timeout paths (spy provider).
|
||||
4. Audit threading: existing tag-cascade tests (`LastOnTriggerParentExecutionId`,
|
||||
`AlarmActor.cs:135`; `SeedAttributesReference` isolation contracts) pass unmodified; a
|
||||
routed `ScriptCallRequest.ParentExecutionId` still reaches the context.
|
||||
5. Timeout parity: per-script override and global fallback (`perScript ?? global`, ≤ 0 ⇒
|
||||
global) produce the same OperationCanceled path, site event, and watchdog behavior.
|
||||
6. Stop-during-run parity: stop the ScriptActor mid-run; the run completes, no crash, its
|
||||
completion message dead-letters (assert via dead-letter subscription), CTS not cancelled.
|
||||
7. Alarm side: on-trigger run gets `Alarm` globals (name/level/priority/message) and
|
||||
`AlarmExecutionCompleted` still reaches the AlarmActor — existing `ExecutionActorTests`
|
||||
reworked to launcher-level tests rather than deleted.
|
||||
|
||||
Fallback stance: if stage (b) uncovers a real consumer of the exec-actor `IActorRef` this memo
|
||||
missed, the cheap variant is pooled Props reuse — but the `_self` grep evidence above makes
|
||||
that contingency unlikely, and no design effort is spent on it here.
|
||||
|
||||
---
|
||||
|
||||
## 5. (e) `SiteScriptCompileCache` LRU + off-dispatcher cache-miss compiles
|
||||
|
||||
### LRU eviction (kills the overflow cliff)
|
||||
|
||||
The wholesale `Cache.Clear()` at 1024 entries (`SiteScriptCompileCache.cs:71-72`) is a latency
|
||||
cliff — the uncommitted `Component-SiteRuntime.md` WIP documents it precisely (instance scripts
|
||||
*and* trigger expressions share the cache; crossing the bound triggers a full recompile storm on
|
||||
actor threads). **This design resolves it; the WIP doc paragraph gets its "remains one" clause
|
||||
rewritten in stage (b).**
|
||||
|
||||
**Decision: approximate LRU via access-stamped entries + batch eviction; no wholesale clear.**
|
||||
|
||||
- Entry becomes `(ScriptCompilationResult Result, long LastAccess)` where `LastAccess` is a
|
||||
`Volatile`-written stamp from a process-wide `Interlocked` access sequence (not a clock —
|
||||
deterministic for tests, immune to clock steps).
|
||||
- Hits update the stamp lock-free; no lock, no linked list on the hot path.
|
||||
- On insert with `Count >= MaxEntries`: evict the oldest **⅛ (128) entries** by one O(n) scan
|
||||
under a small eviction lock (double-checked so concurrent inserts don't stampede). Overflow
|
||||
now costs one 1024-element scan instead of 1023 future recompiles.
|
||||
- `MaxEntries` stays 1024; `Hits`/`Count`/`Clear()` keep their signatures (tests use them,
|
||||
`SiteScriptCompileCacheTests.cs`).
|
||||
|
||||
### Cache-miss compiles move off the actor thread
|
||||
|
||||
Two on-actor-thread compile sites (`Task.Run`+`PipeTo` shared-script loader at
|
||||
`DeploymentManagerActor.cs:1540-1563` is the pattern reference):
|
||||
|
||||
1. **Deploy gate** (`HandleDeploy`, `DeploymentManagerActor.cs:522-556`). The S3 comment's
|
||||
ordering constraint (mailbox-FIFO between deploy/delete/disable) is real — so the design is
|
||||
**warm-then-gate**: `HandleDeploy` first runs `_deployCompileValidator.Validate` in a
|
||||
`Task.Run`, piping back `DeployCompileWarmed(command, replyTo)`; on receipt it re-runs the
|
||||
existing synchronous gate (now all cache hits — memoised at `ScriptCompilationService.cs:148`)
|
||||
and proceeds down the unchanged `ProceedWithDeploy` path. Ordering is preserved with a
|
||||
**per-instance in-flight guard**: while a warm is in flight for instance X, other mutating
|
||||
commands for X are buffered last-write-wins with a `Failed`-superseded reply to the displaced
|
||||
sender — the exact machinery the redeploy path already uses
|
||||
(`_pendingRedeploys`/`_terminatingActorsByName`, `DeploymentManagerActor.cs:569-601`).
|
||||
Commands for other instances flow freely (an improvement over today's whole-mailbox stall
|
||||
during a long compile, and safe: cross-instance ordering was never guaranteed to callers).
|
||||
2. **Staggered startup / InstanceActor PreStart** (`InstanceActor.cs:1631,1682,1795`). Each
|
||||
startup batch (`HandleStartNextBatch`, `DeploymentManagerActor.cs:467-500`) gains a pre-warm
|
||||
step: before creating the batch's Instance Actors, compile the batch's distinct script bodies
|
||||
and trigger expressions in a `Task.Run`, then pipe `BatchCompileWarmed` and create the actors
|
||||
— PreStart compiles become cache hits. This closes the long-deferred "per-instance
|
||||
compilation during staggered startup" item the WIP `Component-SiteRuntime.md` carries
|
||||
(failover time-to-recover), at the cost of one extra message per batch.
|
||||
InstanceActor's own compile calls stay where they are (they are the correctness backstop and
|
||||
are hits in every warmed path).
|
||||
|
||||
No change to `ScriptCompilationService` semantics: results (success AND failure) stay memoised,
|
||||
error text stays name-free, the process-static `ScriptOptions` + `CachingScriptMetadataResolver`
|
||||
invariants (`ScriptCompilationService.cs:104-111`) are untouched.
|
||||
|
||||
---
|
||||
|
||||
## 6. (f) Test plan (stage b)
|
||||
|
||||
All in `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests` unless noted; the injectable scheduler
|
||||
seam (#18) and injectable eval gate make every scenario deterministic without the process-wide
|
||||
singletons.
|
||||
|
||||
1. **Starvation regression (the finding-#4 pin):** injected 8-thread scheduler; 8 scripts
|
||||
blocked on a `ManualResetEventSlim`; fire an Expression-trigger alarm eval ⇒
|
||||
`AlarmStateChanged` (or `ExpressionEvalResult`) observed **< 2 s** while all 8 threads stay
|
||||
blocked. Companion negative: same setup on the *old* wiring is the currently-failing shape
|
||||
(documented in the test comment, not committed failing).
|
||||
2. **Queue-inclusive eval timeout:** eval gate of 1, occupy it > `TriggerEvalTimeoutSeconds`
|
||||
with a slow eval; second eval resolves as false within ~the timeout **measured from
|
||||
enqueue** (assert elapsed-from-StartExpressionEvaluation, not from gate acquisition), and
|
||||
the trigger is not parked (`_evalPending` drains).
|
||||
3. **Queue-inclusive script deadline:** 1-thread scheduler, first script blocks past the second
|
||||
script's full timeout; second script's failure path fires without its body ever executing
|
||||
(side-effect flag never set), error reply carries the timeout message, watchdog does NOT
|
||||
flag it as a stuck thread.
|
||||
4. **Watchdog thread replacement:** 1-thread scheduler, wedge a script past timeout+grace ⇒
|
||||
`TryDetachWorker` fires, a subsequent script executes (replacement thread live),
|
||||
`DetachedScriptThreads` gauge = 1; unwedge ⇒ detached worker exits, gauge returns to 0.
|
||||
Cap test: detached-at-cap ⇒ no new thread, Error site event emitted.
|
||||
5. **Pool sizing:** `ComputeTargetThreads` pure-function table (floor 8, /8 growth, ceiling,
|
||||
override precedence); `EnsureCapacity` grows once, is idempotent, never shrinks; gauges
|
||||
reflect the widened pool.
|
||||
6. **Shed policy:** cap 4 via options; block 4 runs; 5th trigger ⇒ shed counter +1, Warning
|
||||
site event (and second shed within the minute ⇒ counter +1, NO second event), Ask caller
|
||||
gets error reply; one completion ⇒ next trigger launches. Alarm on-trigger variant.
|
||||
7. **Supervision-parity set:** the seven pins of §4, replacing/reworking
|
||||
`Actors/ExecutionActorTests.cs`.
|
||||
8. **Compile cache LRU:** fill to `MaxEntries`, touch entry #1, insert one more ⇒ entry #1
|
||||
survives, an untouched old entry is evicted, `Count ≤ MaxEntries`, no wholesale clear
|
||||
(`Hits` preserved); concurrent GetOrAdd storm keeps the bound.
|
||||
9. **Warm-then-gate ordering:** Deploy(X) immediately followed by Delete(X) ⇒ delete is
|
||||
buffered, deploy applies first, terminal states correct; Deploy(X) then Deploy(X') ⇒ first
|
||||
sender gets superseded reply; Deploy(X) + Deploy(Y) interleave freely. Batch pre-warm:
|
||||
staggered startup with a compile-heavy config ⇒ InstanceActor PreStart compiles are cache
|
||||
hits (assert via `SiteScriptCompileCache.Hits`).
|
||||
|
||||
Suite gate: full `dotnet test ZB.MOM.WW.ScadaBridge.slnx` green; rig rebuild + failover drill
|
||||
per the Phase-3 gate (the drill guards the timeout/actor changes, plan §6).
|
||||
|
||||
---
|
||||
|
||||
## 7. Configuration summary (all under `ScadaBridge:SiteRuntime`, validated in `SiteRuntimeOptionsValidator`)
|
||||
|
||||
| Option | Default | New? | Meaning |
|
||||
|---|---|---|---|
|
||||
| `ScriptExecutionThreadCount` | 8 | existing | Blocking-pool floor (was: fixed size) |
|
||||
| `ScriptExecutionMaxThreadCount` | 32 | new | Blocking-pool ceiling for instance-scaled growth |
|
||||
| `TriggerEvalMaxConcurrency` | `max(2, ProcessorCount)` | new | Thread-pool gate for Expression evals |
|
||||
| `TriggerEvalTimeoutSeconds` | 2 | new | Was hardcoded; now queue-inclusive |
|
||||
| `MaxConcurrentRunsPerScript` | 4 | new | Per-script/per-alarm in-flight cap; overflow sheds newest |
|
||||
| `ScriptExecutionTimeoutSeconds` / per-script override | 30 / null | existing | Unchanged semantics; deadline now anchored at enqueue |
|
||||
| `StuckScriptGraceMs` | 30000 | existing | Unchanged; now also triggers thread detach/replace |
|
||||
|
||||
---
|
||||
|
||||
## 8. Notes for WP3.1 stage (b) — scope & risk deltas vs. the plan
|
||||
|
||||
- **Risk DOWN on actor elimination:** the shells have no Receive handlers, no PostStop, and
|
||||
their `IActorRef` is provably unused as a message target (`ScriptRuntimeContext.cs:43,243,305`
|
||||
are the only `_self` references). Elimination is a refactor of dead structure, not a
|
||||
semantics change — the parity surface is the launcher body, which moves verbatim.
|
||||
- **Scope +:** `AlarmExecutionActor` is eliminated together with `ScriptExecutionActor`
|
||||
(same pattern, shared launcher); the plan text named only the latter.
|
||||
- **Scope +:** batch pre-warm of startup compiles closes the deferred "per-instance compilation
|
||||
during staggered startup" item; the uncommitted `Component-SiteRuntime.md` WIP (compile-cache
|
||||
overflow-cliff paragraph) should be finalized in stage (b) to record the cliff as resolved by
|
||||
LRU eviction. Do not touch that file in stage (a) — it is user WIP.
|
||||
- **Scope note:** one intentional behavior improvement is folded in (launch-path failure now
|
||||
replies to the Ask caller instead of letting it hang, §4 map row 2) — pinned by test 2 of the
|
||||
parity set so it is explicit, not accidental.
|
||||
- **Ordering with WP3.2:** the shed site event is deliberately rate-limited (1/script/minute)
|
||||
so this WP does not add a new `site_events` volume source while WP3.2 is deciding volume
|
||||
policy; merge order (3.1 before 3.2) per the plan's conflict matrix is unaffected.
|
||||
- **Not in scope:** cancelling in-flight runs on actor stop (explicitly kept at parity, §4 row
|
||||
11); eval-side dedicated pool (rejected, §1); pool shrink (rejected, §2).
|
||||
@@ -0,0 +1,210 @@
|
||||
# site_events Volume Policy — WP3.2 Stage (a) Design Memo
|
||||
|
||||
**Date:** 2026-08-15 · **Author:** fable (design) → sonnet (stage b implement)
|
||||
**Parent:** `docs/plans/2026-08-14-arch-review-remediation-plan.md` §5, WP3.2
|
||||
**Finding:** #High site_events (policy half; writer mechanics already fixed in WP1.7)
|
||||
|
||||
---
|
||||
|
||||
## 1. The problem, quantified
|
||||
|
||||
Every script run emits **two Info rows** into `site_events` — "started" and "completed" —
|
||||
from `ScriptExecutionActor` (`src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/ScriptExecutionActor.cs`,
|
||||
lines ~275 and ~288). `site_events` is a CDC-replicated table (`SiteLocalDbSetup.OnReady`,
|
||||
`db.RegisterReplicated("site_events")`), so each row does not cost one write — the capture
|
||||
triggers (`TriggerSqlGenerator` in `~/Desktop/scadaproj/ZB.MOM.WW.LocalDb`) add **one
|
||||
`__localdb_oplog` row + one `__localdb_row_version` row per INSERT**, the batch ships to the
|
||||
peer and is applied there, and 30 days later the retention purge DELETEs the row — which the
|
||||
same triggers capture as **an oplog tombstone + a row_version tombstone**, also shipped.
|
||||
|
||||
Net: one script run ≈ **12 physical row-writes across the pair** plus two replication round
|
||||
trips (insert-time and purge-time), for two rows nobody looks at unless something went wrong.
|
||||
A modest site running 10 script executions/second writes ~1.7M `site_events` rows/day plus the
|
||||
same again in oplog traffic. WP1.7 batched the commits and sliced the purge — the *mechanics*
|
||||
are fine; this memo fixes the *volume*.
|
||||
|
||||
Every other `site_events` writer is per-occurrence, not per-run: DCL connect/disconnect
|
||||
(`DataConnectionActor`), alarm state transitions (`AlarmActor`, `NativeAlarmActor`), S&F
|
||||
parking (`StoreAndForwardService`), script **errors** (timeout / failure / stuck-watchdog /
|
||||
recursion-limit). Those stay untouched — they are the rows the log exists for.
|
||||
|
||||
---
|
||||
|
||||
## 2. Decision (a) — per-run events off by default; interval summaries instead
|
||||
|
||||
### Policy
|
||||
|
||||
1. **Per-run Started/Completed Info events are OFF by default.** The Error emissions
|
||||
(timeout, failure, stuck-watchdog in the CTS callback, recursion-limit in
|
||||
`ScriptRuntimeContext`) are unchanged and unconditional.
|
||||
2. **An interval aggregate replaces them:** a new `ScriptRunSummaryRecorder` (SiteRuntime)
|
||||
accumulates per-`(instanceName, scriptName)` counters — started, completed, failed,
|
||||
timedOut, plus total/max duration ms — incremented from the same call sites that today
|
||||
emit the two Info events. A timer flush (hosted service, Site nodes only) emits **one**
|
||||
site event per interval: `event_type "script"`, severity `Info`, `instance_id` NULL,
|
||||
source `"ScriptRunSummary"`, message = headline totals (e.g. "1,243 runs: 1,240 completed,
|
||||
2 failed, 1 timed out across 17 scripts"), details = per-script JSON breakdown, capped at
|
||||
the top 50 scripts by run count with an "others" rollup so a pathological deployment
|
||||
cannot mint oversized rows. **Zero activity ⇒ no row** (the standby therefore emits
|
||||
nothing naturally — only the active node runs scripts). Counters reset on flush.
|
||||
3. **Per-script opt-in for debugging** restores the per-run rows for named scripts without a
|
||||
redeploy (mechanism below).
|
||||
|
||||
At the default interval this is ≤288 summary rows/day instead of 2 rows/run — the CDC tax on
|
||||
the residual table content (errors + lifecycle + summaries) is negligible, which is what makes
|
||||
Decision (b) cheap to settle on merit rather than on cost.
|
||||
|
||||
### Config keys (`ScadaBridge:SiteRuntime`, on `SiteRuntimeOptions`)
|
||||
|
||||
| Key | Default | Meaning |
|
||||
|---|---|---|
|
||||
| `PerRunScriptEvents` | `false` | Global switch back to legacy per-run Started/Completed rows. |
|
||||
| `PerRunScriptEventScripts` | `[]` | Per-script opt-in list; entries `"{InstanceName}/{ScriptName}"`, `"{InstanceName}/*"` supported. Effective when the global switch is off. |
|
||||
| `ScriptRunSummaryIntervalSeconds` | `300` | Summary flush interval; `0` disables summaries. Validated ≥ 0. |
|
||||
|
||||
**Hot toggle, no new plumbing:** the execution path already threads `serviceProvider`
|
||||
(it resolves `ISiteEventLogger` at line ~135); it additionally resolves
|
||||
`IOptionsMonitor<SiteRuntimeOptions>` and reads `CurrentValue` per run, so with
|
||||
`reloadOnChange` on the mounted appsettings an operator can flip per-run events for one
|
||||
script live, debug, and flip back — no restart, no redeploy, no template/schema change.
|
||||
Fallback to the constructor-passed `SiteRuntimeOptions` when the provider is null (tests).
|
||||
This keeps stage (b)'s blast radius exactly what the plan's dependency table promises:
|
||||
**SiteEventLogging + one SiteRuntime call site** — the opt-in deliberately does NOT ride the
|
||||
script config entity, because that would drag Template Engine, flattened config, Commons
|
||||
DTOs and the deploy pipeline into a volume-policy fix.
|
||||
|
||||
### What the Site Event Log page loses and what replaces it
|
||||
|
||||
- **Loses:** the two Info rows per run. Operators who used them as a liveness signal get the
|
||||
`ScriptRunSummary` rows instead (findable via the existing keyword search — no UI change
|
||||
required), the unchanged Error rows for anything abnormal, and the existing health-report
|
||||
script-error counters. Call this out in the release note: the page's texture changes.
|
||||
- **Gains:** per-interval activity summaries with per-script counts and durations — strictly
|
||||
more diagnostic signal per row than "started"/"completed" pairs.
|
||||
- Optional follow-up (not required for stage b): surface run counts on the health snapshot
|
||||
via a new default-interface `ISiteHealthCollector` member. If anyone instead reaches for a
|
||||
Prometheus counter, remember `ZbTelemetryOptions.Meters` is an allowlist — an unlisted
|
||||
meter exports nothing, silently.
|
||||
|
||||
**WP3.1 coordination:** WP3.1 deletes `ScriptExecutionActor` and runs the guarded task under
|
||||
`ScriptActor`. Per the plan's dependency table, **3.1 merges first**; stage (b) lands this
|
||||
policy at wherever the two emissions live after that move. The policy is about the emission
|
||||
call site, not the actor, so the design is unaffected by the move.
|
||||
|
||||
---
|
||||
|
||||
## 3. Decision (b) — keep `site_events` replicated; fix the *reason* on record
|
||||
|
||||
### Verdict: KEEP registered. But the plan's presumed justification is false, and the real one is different.
|
||||
|
||||
The plan's default recommendation said "keep replicated — central event-log queries hit
|
||||
either node." **That presumption does not survive contact with the code:**
|
||||
|
||||
- The event-log query handler is a **ClusterSingleton** scoped to the site role:
|
||||
`AkkaHostedService.cs` (~905–917) starts `EventLogHandlerActor` via `SingletonRegistrar`
|
||||
("event-log-handler", `role: siteRole`) and registers the **singleton proxy** as the
|
||||
handler.
|
||||
- Both central→site transports route through the same table:
|
||||
`SiteCommandDispatcher.ResolveRoute` sends `EventLogQueryRequest` to `_eventLogHandler` —
|
||||
the singleton proxy (`SiteCommandDispatcher.cs` ~165). The dispatcher is the single routing
|
||||
truth for the Akka actor path AND `SiteCommandGrpcService`, so even when central's
|
||||
NodeA→NodeB channel flip lands the gRPC call on the **standby**, the query is forwarded to
|
||||
the **active** node's handler, whose `EventLogQueryService` reads the active node's local
|
||||
SQLite. **No code path ever reads the standby's `site_events` replica while it is standby.**
|
||||
|
||||
So by the plan's own test ("if nothing reads it on the peer, deregister it"), dereg would
|
||||
follow — except one consumer remains, and it is the one Phase 1 built the replication for:
|
||||
|
||||
- **Post-failover history continuity.** When the active node dies, the singleton restarts on
|
||||
the survivor and its query service reads the survivor's local copy — which has content
|
||||
*only because* it was replicated. Without registration, a failover starts an empty event
|
||||
log at precisely the moment an operator is investigating an incident. That is not
|
||||
hypothetical: it was the shipped pre-Phase-1 behavior (the stale comment at
|
||||
`AkkaHostedService.cs` ~905 still describes it), and Phase 1 explicitly fixed it —
|
||||
`SiteEventLogger`'s doc header and `SiteServiceRegistration.cs` ~79 ("so the pair stops
|
||||
losing them on failover") record the intent.
|
||||
|
||||
With Decision (a) removing the dominant per-run volume, the residual replication cost is
|
||||
small, and the failover-continuity benefit lands exactly when the log is most valuable.
|
||||
**Keep `site_events` in `RegisterReplicated`.** Rig posture note: site-b/site-c run
|
||||
unreplicated by deliberate choice; there the log stays node-local and failover starts fresh —
|
||||
that is the documented trade of not configuring a peer, unchanged by this memo.
|
||||
|
||||
Stage (b) must also **fix the stale comment** at `AkkaHostedService.cs` ~905–908 ("The event
|
||||
log is node-local SQLite and is not replicated") — the singleton rationale it states is still
|
||||
correct, the replication claim is two phases out of date. The correct rationale to write:
|
||||
singleton = queries always read the actively-written copy; replication = the standby's copy
|
||||
exists so the singleton has history after it moves.
|
||||
|
||||
### Purge traffic — coordinate with WP3.3, do not block on it
|
||||
|
||||
Retention/cap purge DELETEs are trigger-captured today: every event pays the CDC tax **twice**
|
||||
(insert + tombstone), and tombstones linger for `TombstoneRetention` (7 days). WP3.3's menu
|
||||
includes clock-based local purges that never enter the oplog. When that API lands:
|
||||
|
||||
- switch `EventLogPurgeService` to the oplog-bypassing delete,
|
||||
- run the **retention** purge on **both** nodes (drop the active-node gate for it): the age
|
||||
cutoff is deterministic, so both nodes converge independently without shipping tombstones.
|
||||
(The 1GB **cap** purge stays per-node local hygiene; minor divergence is harmless.)
|
||||
- Accepted edge: with no tombstones, a lagging peer's late oplog INSERT can resurrect an
|
||||
already-expired row; the next purge tick removes it. Harmless for an operational log.
|
||||
|
||||
If WP3.3's API is not merged when stage (b) lands, keep the current WP1.7 purge shape and
|
||||
leave the switch as a recorded follow-up — Decision (a) already removed the bulk of the
|
||||
tombstone traffic.
|
||||
|
||||
---
|
||||
|
||||
## 4. Migration / compat
|
||||
|
||||
- **No schema change, no wire change.** `EventLogQueryRequest`/`EventLogQueryResponse`,
|
||||
`EventLogEntry`, and the opaque continuation token are untouched — central↔site version
|
||||
skew is a non-issue.
|
||||
- **Existing rows:** nothing to migrate; historical per-run rows age out via the 30-day
|
||||
retention window.
|
||||
- **Pair upgrade:** site pairs already stop/start together (topology guide), so both nodes
|
||||
adopt the new emission policy in the same instant — no mixed-emission pair exists. If the
|
||||
WP3.3 local-purge lands later, its first tick on the formerly-gated standby simply purges
|
||||
rows the active had already deleted; converges in one tick.
|
||||
- **Config:** absent keys = new defaults (per-run OFF, summaries every 300s). Operators who
|
||||
want the legacy behavior set `PerRunScriptEvents: true` — byte-for-byte the old rows.
|
||||
- **Release note:** Site Event Log page loses per-run Started/Completed rows by default;
|
||||
`ScriptRunSummary` rows replace them; per-script opt-in documented.
|
||||
|
||||
---
|
||||
|
||||
## 5. Stage (b) test plan
|
||||
|
||||
1. **Default-off:** successful run emits no Info events; timeout/failure/stuck-watchdog
|
||||
Error paths still emit (site-runtime tests at the post-WP3.1 call site).
|
||||
2. **Global opt-in:** `PerRunScriptEvents=true` → started+completed emitted with the legacy
|
||||
message shape.
|
||||
3. **Per-script opt-in:** exact `"Instance/Script"` match emits; `"Instance/*"` wildcard
|
||||
emits; non-matching script stays silent.
|
||||
4. **Hot toggle:** fake `IOptionsMonitor` flipped mid-test changes behavior on the next run
|
||||
without actor restart.
|
||||
5. **Summary recorder:** N runs with mixed outcomes across scripts → exactly one flush event
|
||||
with correct counts and capped details JSON (>50 scripts → top-50 + "others" rollup);
|
||||
zero-activity interval → no row; counters reset after flush; concurrent increments are
|
||||
race-free.
|
||||
6. **Replication stance pinned:** test asserting `site_events` remains in
|
||||
`SiteLocalDbSetup`'s `RegisterReplicated` set (guards Decision (b) against drive-by
|
||||
"optimization").
|
||||
7. **If the WP3.3 purge switch lands in the same window:** purge deletes produce zero
|
||||
`__localdb_oplog` rows; both-node retention convergence test.
|
||||
8. **Live probes (Phase 4 gate):** rig site-a `site_events` growth rate before/after under
|
||||
the demo script load; failover drill confirms the new active node serves pre-failover
|
||||
event history (replication continuity, the reason (b) kept it).
|
||||
|
||||
---
|
||||
|
||||
## 6. Affected documents (Phase 4 collects)
|
||||
|
||||
- `docs/requirements/Component-SiteEventLogging.md` — volume policy, summary events, the
|
||||
replication rationale (failover continuity, NOT either-node queries), purge/oplog note.
|
||||
- `docs/requirements/Component-SiteRuntime.md` — per-run emission policy + the three config
|
||||
keys. **This file currently carries unrelated user WIP — stage (b) must merge around it,
|
||||
and this memo's commit deliberately excludes it.**
|
||||
- Code comment fix (stage b, not a doc): `AkkaHostedService.cs` ~905 stale "not replicated"
|
||||
rationale.
|
||||
- `Component-HealthMonitoring.md` only if the optional run-count snapshot member is taken.
|
||||
@@ -0,0 +1,45 @@
|
||||
-- Run with QUOTED_IDENTIFIER ON. SiteCalls / Notifications / AuditLog all carry
|
||||
-- filtered indexes (and AuditLog persisted computed columns), and SQL Server
|
||||
-- refuses BOTH the filtered-index DDL below and any later DML on those tables
|
||||
-- with error 1934 when the session has QUOTED_IDENTIFIER OFF -- which is exactly
|
||||
-- what `sqlcmd` gives you by default (pass -I, or keep this header). The SETs sit
|
||||
-- in their own batch so they are in force when the statements after the GO parse.
|
||||
SET QUOTED_IDENTIFIER ON;
|
||||
SET ANSI_NULLS ON;
|
||||
GO
|
||||
|
||||
BEGIN TRANSACTION;
|
||||
IF NOT EXISTS (
|
||||
SELECT * FROM [__EFMigrationsHistory]
|
||||
WHERE [MigrationId] = N'20260814235335_AddAuditLogAndNotificationCoveringIndexes'
|
||||
)
|
||||
BEGIN
|
||||
EXEC(N'CREATE INDEX [IX_Notifications_Delivered] ON [Notifications] ([DeliveredAt]) WHERE [Status] = ''Delivered''');
|
||||
END;
|
||||
|
||||
IF NOT EXISTS (
|
||||
SELECT * FROM [__EFMigrationsHistory]
|
||||
WHERE [MigrationId] = N'20260814235335_AddAuditLogAndNotificationCoveringIndexes'
|
||||
)
|
||||
BEGIN
|
||||
|
||||
DROP INDEX IX_AuditLog_OccurredAtUtc ON dbo.AuditLog;
|
||||
|
||||
CREATE NONCLUSTERED INDEX IX_AuditLog_OccurredAtUtc
|
||||
ON dbo.AuditLog (OccurredAtUtc DESC)
|
||||
INCLUDE (Status)
|
||||
ON ps_AuditLog_Month(OccurredAtUtc);
|
||||
END;
|
||||
|
||||
IF NOT EXISTS (
|
||||
SELECT * FROM [__EFMigrationsHistory]
|
||||
WHERE [MigrationId] = N'20260814235335_AddAuditLogAndNotificationCoveringIndexes'
|
||||
)
|
||||
BEGIN
|
||||
INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion])
|
||||
VALUES (N'20260814235335_AddAuditLogAndNotificationCoveringIndexes', N'10.0.7');
|
||||
END;
|
||||
|
||||
COMMIT;
|
||||
GO
|
||||
|
||||
@@ -1,4 +1,14 @@
|
||||
BEGIN TRANSACTION;
|
||||
-- Run with QUOTED_IDENTIFIER ON. SiteCalls / Notifications / AuditLog all carry
|
||||
-- filtered indexes (and AuditLog persisted computed columns), and SQL Server
|
||||
-- refuses BOTH the filtered-index DDL below and any later DML on those tables
|
||||
-- with error 1934 when the session has QUOTED_IDENTIFIER OFF -- which is exactly
|
||||
-- what `sqlcmd` gives you by default (pass -I, or keep this header). The SETs sit
|
||||
-- in their own batch so they are in force when the statements after the GO parse.
|
||||
SET QUOTED_IDENTIFIER ON;
|
||||
SET ANSI_NULLS ON;
|
||||
GO
|
||||
|
||||
BEGIN TRANSACTION;
|
||||
IF NOT EXISTS (
|
||||
SELECT * FROM [__EFMigrationsHistory]
|
||||
WHERE [MigrationId] = N'20260709110614_AddSiteCallsNonTerminalIndex'
|
||||
|
||||
@@ -1,4 +1,14 @@
|
||||
BEGIN TRANSACTION;
|
||||
-- Run with QUOTED_IDENTIFIER ON. SiteCalls / Notifications / AuditLog all carry
|
||||
-- filtered indexes (and AuditLog persisted computed columns), and SQL Server
|
||||
-- refuses BOTH the filtered-index DDL below and any later DML on those tables
|
||||
-- with error 1934 when the session has QUOTED_IDENTIFIER OFF -- which is exactly
|
||||
-- what `sqlcmd` gives you by default (pass -I, or keep this header). The SETs sit
|
||||
-- in their own batch so they are in force when the statements after the GO parse.
|
||||
SET QUOTED_IDENTIFIER ON;
|
||||
SET ANSI_NULLS ON;
|
||||
GO
|
||||
|
||||
BEGIN TRANSACTION;
|
||||
IF NOT EXISTS (
|
||||
SELECT * FROM [__EFMigrationsHistory]
|
||||
WHERE [MigrationId] = N'20260713142234_AddSiteCallsTerminalIndex'
|
||||
|
||||
@@ -0,0 +1,34 @@
|
||||
-- Run with QUOTED_IDENTIFIER ON. SiteCalls / Notifications / AuditLog all carry
|
||||
-- filtered indexes (and AuditLog persisted computed columns), and SQL Server
|
||||
-- refuses BOTH the filtered-index DDL below and any later DML on those tables
|
||||
-- with error 1934 when the session has QUOTED_IDENTIFIER OFF -- which is exactly
|
||||
-- what `sqlcmd` gives you by default (pass -I, or keep this header). The SETs sit
|
||||
-- in their own batch so they are in force when the statements after the GO parse.
|
||||
SET QUOTED_IDENTIFIER ON;
|
||||
SET ANSI_NULLS ON;
|
||||
GO
|
||||
|
||||
BEGIN TRANSACTION;
|
||||
IF NOT EXISTS (
|
||||
SELECT * FROM [__EFMigrationsHistory]
|
||||
WHERE [MigrationId] = N'20260815004957_AlignAuditLogEventIdUniqueness'
|
||||
)
|
||||
BEGIN
|
||||
|
||||
IF EXISTS (SELECT 1 FROM sys.indexes
|
||||
WHERE name = 'UX_AuditLog_EventId' AND object_id = OBJECT_ID('dbo.AuditLog'))
|
||||
DROP INDEX UX_AuditLog_EventId ON dbo.AuditLog;
|
||||
END;
|
||||
|
||||
IF NOT EXISTS (
|
||||
SELECT * FROM [__EFMigrationsHistory]
|
||||
WHERE [MigrationId] = N'20260815004957_AlignAuditLogEventIdUniqueness'
|
||||
)
|
||||
BEGIN
|
||||
INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion])
|
||||
VALUES (N'20260815004957_AlignAuditLogEventIdUniqueness', N'10.0.7');
|
||||
END;
|
||||
|
||||
COMMIT;
|
||||
GO
|
||||
|
||||
@@ -196,7 +196,9 @@ known spawn points:
|
||||
`ExecutionId` as its `ParentExecutionId`, while the inbound `InboundRequest` row
|
||||
is top-level (`ParentExecutionId` NULL).
|
||||
- **Alarm-triggered on-trigger script** — when a write trips an alarm and its
|
||||
on-trigger script runs (via `AlarmActor → AlarmExecutionActor`), the run
|
||||
on-trigger script runs (via `AlarmActor → ScriptRunLauncher.LaunchAlarmScript`,
|
||||
which builds the run's `ScriptRuntimeContext` directly on the script-execution
|
||||
pool — the per-run `AlarmExecutionActor` was removed by WP3.1), the run
|
||||
records the **writing execution's** `ExecutionId` as its `ParentExecutionId`.
|
||||
The write's originating execution rides site-locally from
|
||||
`ScriptRuntimeContext.SetAttribute` (or the inbound API's
|
||||
@@ -261,7 +263,15 @@ room is a compliance violation, not a self-healing behavior. To bound that
|
||||
growth in practice, the site emits a `SiteAuditBacklog` health metric (pending
|
||||
row count, oldest pending age, bytes on disk); crossing operator-configured
|
||||
thresholds surfaces a warning on the relevant site tile in the Health
|
||||
dashboard, mirroring the Store-and-Forward Engine's backlog metric.
|
||||
dashboard, mirroring the Store-and-Forward Engine's backlog metric, and
|
||||
`SiteAuditBacklogReporter` additionally logs a rate-limited warning once the
|
||||
oldest pending row passes 24 h.
|
||||
|
||||
The same invariant carries a liveness dependency worth stating explicitly: the
|
||||
`Pending` floor clears only when central acknowledges the rows — a telemetry ack
|
||||
or a reconciliation cursor that covers them — never on age. See *Reconciliation
|
||||
pull* below for the served-row retirement rule that decides which rows can be
|
||||
acknowledged by a cursor at all.
|
||||
|
||||
Central is the durable home. Site SQLite is a write-buffer with a forwarding
|
||||
guarantee.
|
||||
@@ -291,14 +301,63 @@ stay `Pending` for the next sweep. Cadence is short (default 5 s) when
|
||||
non-empty, longer (default 30 s) when idle; telemetry runs on a dedicated
|
||||
dispatcher.
|
||||
|
||||
**Central-side ingest is set-based.** `AuditLogIngestActor` writes a whole
|
||||
packet with ONE `InsertManyIfNotExistsAsync` statement rather than one
|
||||
`IF NOT EXISTS … INSERT` round trip per event; the cached-telemetry dual-write
|
||||
similarly runs the whole packet in ONE transaction (set-based audit insert plus
|
||||
one single-statement `SiteCalls` upsert per entry) instead of a transaction per
|
||||
entry. Idempotency is unchanged: duplicates that repeat *within* a packet
|
||||
collapse first-write-wins before the statement is built, duplicates *across*
|
||||
packets are eliminated by the anti-semi-join, and any failure falls back to the
|
||||
per-row / per-entry path — so the documented invariant that one bad row cannot
|
||||
sink the rest of the batch still holds, it is simply no longer paid for on the
|
||||
healthy path.
|
||||
|
||||
The set-based path binds its string parameters at the VALUE's own length, never
|
||||
at the column width. Declaring the width makes the client TRUNCATE an over-long
|
||||
value at bind time and commit the shortened row — silent mutilation in an
|
||||
append-only store, with no `PayloadTruncated` flag to admit it, and inconsistent
|
||||
with the per-row and reconciliation paths, which send the value in full and let
|
||||
the server reject it. Length enforcement belongs to the server on every path:
|
||||
reject everywhere, truncate nowhere. (Deliberate, policy-driven truncation of
|
||||
`RequestSummary`/`ResponseSummary` under the payload caps is a different thing
|
||||
entirely — it happens before the write and always sets `PayloadTruncated`.)
|
||||
|
||||
The per-row fallback runs on its OWN short cancellation budget rather than the
|
||||
batch's. Sharing it meant that a batch failing *because* the ingest budget
|
||||
expired handed every fallback insert an already-cancelled token: N instant
|
||||
failures, N counter bumps, nothing accepted — the fallback defeated at exactly
|
||||
the moment it was needed. A blown budget is also counted ONCE for the batch
|
||||
rather than once per row, so the health counter reads as one timeout instead of
|
||||
a burst of write failures.
|
||||
|
||||
**Timeout ladder.** The ingest budget is deliberately the smallest on the path:
|
||||
the site's Ask and the central gRPC handler's Ask are both 30 s, the actor's
|
||||
own database budget is 20 s and the per-statement SQL timeout is 15 s. Before
|
||||
this the three were identical, so they expired at the same instant and the
|
||||
caller learned nothing but "it took 30 s" — no partial ack, no way to tell a
|
||||
slow database from a wedged singleton. With the inner budgets strictly smaller,
|
||||
a slow batch is abandoned by the actor first and the accepted-so-far ids are
|
||||
still replied while the outer Asks are still waiting.
|
||||
|
||||
### Reconciliation pull (self-healing for missed telemetry)
|
||||
|
||||
A central `SiteAuditReconciliationActor` periodically (default 5 min per site)
|
||||
asks each site for its oldest `Pending` row and pending count; if backlog is
|
||||
non-draining (e.g., telemetry actor wedged), central issues a
|
||||
`PullAuditEvents(sinceUtc, batchSize)` and inserts-if-not-exists. Accepted rows
|
||||
are flipped to `ForwardState = 'Reconciled'` site-side. Same self-healing
|
||||
pattern as Site Call Audit's reconciliation of `SiteCalls`.
|
||||
`PullAuditEvents(sinceUtc, batchSize[, afterId])` and inserts-if-not-exists.
|
||||
Same self-healing pattern as Site Call Audit's reconciliation of `SiteCalls`.
|
||||
|
||||
**The pull is at-least-once (WP2.3).** Serving a row is not proof that central
|
||||
received it, so rows are **not** flipped when they are served. The site flips
|
||||
`ForwardState = 'Reconciled'` for everything at or before the cursor carried by
|
||||
the **next** pull — the only evidence central actually consumed them
|
||||
(`ISiteAuditQueue.MarkReconciledUpToAsync`, run before the read so retired rows
|
||||
do not consume the batch budget). A fault between the response leaving the site
|
||||
and central committing it therefore re-serves the batch instead of losing it;
|
||||
central dedups on `EventId`, so a re-ship is a no-op. Previously the site flipped
|
||||
each served batch immediately, and a central-side fault in that window dropped
|
||||
those rows permanently — `ReadPendingSinceAsync` would never return them again.
|
||||
|
||||
**Endpoint resolution & NodeB failover.** Each pull dials the site's `NodeA`
|
||||
gRPC address first; if `NodeA` is blank the site's `NodeB` address becomes the
|
||||
@@ -312,13 +371,61 @@ mapping/unexpected fault collapses to empty without a second dial (the other
|
||||
node would hit the same fault). The same resolution + failover applies to Site
|
||||
Call Audit's `PullSiteCalls`.
|
||||
|
||||
> **Cursor keyset (tracked follow-up).** The `PullAuditEvents` cursor is still a
|
||||
> single `sinceUtc` timestamp. Site Call Audit's pull now uses a composite
|
||||
> `(UpdatedAtUtc, TrackedOperationId)` keyset to avoid a single-timestamp pin
|
||||
> (see Component-SiteCallAudit.md → Reconciliation). The same keyset should be
|
||||
> applied here, but it is lower urgency because the audit cursor already re-pulls
|
||||
> idempotently on `EventId` — a saturated single-timestamp window re-inserts
|
||||
> harmless no-ops rather than losing rows.
|
||||
**Cursor keyset.** The composite `(OccurredAtUtc, EventId)` keyset is wired end to
|
||||
end. `PullAuditEventsRequest.after_id` (additive field 3) mirrors
|
||||
`PullSiteCallsRequest.after_id`; `ISiteAuditQueue.ReadPendingSinceAsync` switches
|
||||
from the inclusive `OccurredAtUtc >= since` to a strict composite comparison when it
|
||||
is set, so a burst sharing one instant drains via the id tiebreak instead of pinning
|
||||
the cursor; and `IPullAuditEventsClient` /`GrpcPullAuditEventsClient` /
|
||||
`SiteAuditReconciliationActor` populate it — the actor's per-site watermark is the
|
||||
`(timestamp, id)` pair of the highest row ingested, threaded back into the next pull,
|
||||
mirroring `SiteCallAuditActor`'s `PullSiteCalls` cursor exactly.
|
||||
|
||||
That makes retirement **exact**: `MarkReconciledUpToAsync` is handed the id half, so
|
||||
the rows **at** the cursor instant are proven received and retired, where a bare
|
||||
timestamp could only prove receipt of rows strictly older than itself and left the
|
||||
boundary rows servable until a newer row moved the cursor. `after_id` is null only on
|
||||
the first pull for a site (and against a site that predates the field), which keeps
|
||||
the legacy inclusive `>=` read as the compatible fallback. The cursor stays in-memory
|
||||
and resets on singleton restart; idempotent `InsertIfNotExistsAsync` still absorbs any
|
||||
re-pulled duplicates, so exactness is an optimization, not a correctness dependency.
|
||||
|
||||
Unlike Site Call Audit, this actor issues ONE pull per tick rather than paging within
|
||||
it — a lagging drain is meant to surface as the stalled signal. The id tiebreak is
|
||||
what makes that safe against a same-instant burst larger than one batch: the cursor
|
||||
advances on every tick even when the timestamp cannot.
|
||||
|
||||
**Only rows that were actually served may retire.** `OccurredAtUtc` is stamped by the
|
||||
caller, so a row can be *inserted* after a batch was served yet carry a timestamp
|
||||
*below* central's (by then advanced) cursor — a back-dated stamp, a clock nudge, a
|
||||
write flushed late. A cursor flip that keyed on the timestamp alone retired exactly
|
||||
those rows: never served, never servable again (the keyset read has moved past them),
|
||||
and, being `Reconciled`, purged on age. That is silent audit loss, and it is a failure
|
||||
mode the pre-WP2.3 explicit id-set flip could not produce, so `MarkReconciledUpToAsync`
|
||||
carries a second, insertion-order bound:
|
||||
|
||||
> A `Pending` row retires only if its insertion order is at or below the high-water
|
||||
> mark of rows this site node has actually served from `ReadPendingSinceAsync`
|
||||
> (SQLite `rowid`). A `Forwarded` row is exempt — central ACKED it over the telemetry
|
||||
> push path, which is proof independent of the pull.
|
||||
|
||||
The bound is per-process, so after a site-node restart the first pull retires only
|
||||
`Forwarded` rows and the pull after it resumes normal retirement — conservative in the
|
||||
safe direction (retirement is delayed, never a row lost). The gRPC handler's ordering
|
||||
(retire, *then* read) is what keeps the bound from ever vouching for the batch it is
|
||||
about to serve.
|
||||
|
||||
**Liveness note (accepted, documented).** A row that was served but never covered by a
|
||||
later cursor — central reconciliation stopped for good, or the bound reset over a
|
||||
restart before the next cursor arrived — stays `Pending` indefinitely, and the site
|
||||
retention purge never purges `Pending`. That is the hard `ForwardState` invariant
|
||||
working as intended (an unacknowledged row is not droppable), but it means the site
|
||||
store's floor depends on reconciliation actually running, not merely on the retention
|
||||
window elapsing. It is observable rather than silent: `GetBacklogStatsAsync` reports
|
||||
the pending count and oldest-pending instant on every site health report
|
||||
(`SiteAuditBacklog`), and `SiteAuditBacklogReporter` logs a rate-limited warning once
|
||||
the oldest pending row exceeds `StalePendingThreshold` (24 h) naming the drain and the
|
||||
reconciliation pull as the things to check.
|
||||
|
||||
### Central direct-write (central-originated events)
|
||||
|
||||
@@ -490,12 +597,19 @@ MS SQL for direct-write events). Unredacted secrets never persist.
|
||||
silently failing retention job would otherwise be invisible until the table grew
|
||||
unbounded. Per-boundary/per-channel error isolation is unchanged — a single failure
|
||||
still never abandons the rest of the tick.
|
||||
- **Maintenance command timeout:** the switch-out drop-and-rebuild dance and each
|
||||
- **Maintenance command timeout:** the switch-out staging batch and each
|
||||
per-channel `DELETE TOP` batch run with an explicit command timeout
|
||||
(`AuditLog:Purge:MaintenanceCommandTimeoutMinutes`, default 30, floor 1 min) rather
|
||||
than the ~30 s ADO.NET default, which could abort the metadata-only SWITCH mid-dance
|
||||
on a large or contended partition and leave the live table without
|
||||
`UX_AuditLog_EventId` until a later tick's CATCH branch rebuilt it.
|
||||
than the ~30 s ADO.NET default, which could abort the metadata-only SWITCH mid-batch
|
||||
on a large or contended partition and leave an orphaned staging table for the next
|
||||
tick's CATCH branch to clean up.
|
||||
- **Partition-aligned uniqueness:** the switch no longer drops and rebuilds an index.
|
||||
`EventId` uniqueness rides the clustered `PK_AuditLog (EventId, OccurredAtUtc)`,
|
||||
which is aligned on `ps_AuditLog_Month(OccurredAtUtc)`, so `SWITCH PARTITION` has no
|
||||
non-aligned unique index to object to. The predecessor `UX_AuditLog_EventId` forced
|
||||
an offline whole-table index build inside the switch transaction — blocking every
|
||||
audit writer for its duration — and left a window in which the idempotency-supporting
|
||||
index did not exist at all. See migration `AlignAuditLogEventIdUniqueness`.
|
||||
- **Per-channel retention overrides (M5.5 T3):** `AuditLog:PerChannelRetentionDays`
|
||||
is a dictionary keyed by canonical channel name (`ApiOutbound`, `DbOutbound`,
|
||||
`Notification`, `ApiInbound`, `SecuredWrite` — all five `AuditChannel` values are
|
||||
|
||||
@@ -139,6 +139,8 @@ Central cluster only. Sites have no user interface.
|
||||
- View diff between deployed and current template-derived configuration.
|
||||
- Deploy updated configuration to individual instances. **Pre-deployment validation** runs automatically before any deployment is sent — validation errors are displayed and block deployment.
|
||||
- Track deployment status (pending, in-progress, success, failed).
|
||||
- **Push-reload coalescing (arch-review WP2.4).** The page reloads its whole table (every deployment record + every instance) on each `DeploymentStatusChange` push, but the notifier fires per status *write* — a site-wide bulk deploy of N instances previously drove 2N+ back-to-back full reloads on the same circuit. Pushes are now leading-edge debounced (500ms): the first push after an idle gap reloads immediately (a single deployment stays as responsive as before), and every push inside the window collapses into one trailing reload.
|
||||
- **Known residual — no server-side paging.** The table still loads and filters every deployment record client-side; server-side paging plus precomputed status counts (deferred-work register item) is the follow-on for large fleets, not shipped in this remediation.
|
||||
|
||||
### System-Wide Artifact Deployment (Deployment Role)
|
||||
- Explicitly deploy shared scripts, external system definitions, database connection definitions, and data connection definitions to all sites or to an individual site. (Notification lists and SMTP configuration are central-only and are not deployed.)
|
||||
@@ -153,6 +155,7 @@ Central cluster only. Sites have no user interface.
|
||||
- The `DebugStreamService` creates a `DebugStreamBridgeActor` on the central side. The bridge actor opens a **gRPC server-streaming subscription** to the site's `SiteStreamGrpcServer` for the selected instance, then requests an initial `DebugViewSnapshot` over the central→site gRPC command channel (`SiteCommandService`).
|
||||
- Ongoing events (`AttributeValueChanged`, `AlarmStateChanged`) flow via the gRPC data stream directly to the bridge actor — they do not travel on the command channel.
|
||||
- Events are delivered to the Blazor component via callbacks, which call `InvokeAsync(StateHasChanged)` to push UI updates through the built-in SignalR circuit.
|
||||
- **Render coalescing (arch-review WP2.4).** Streamed events no longer trigger an individual dispatcher marshal + `StateHasChanged()` each — a chatty instance used to drive one full render (and two full tree rebuilds) per value change. Events now land in a thread-safe pending map keyed by attribute/alarm name (repeated updates to the same tag inside one window collapse to the latest), and exactly one dispatcher marshal per **250ms coalesce window** drains the map, bumps a version stamp, and renders once. This also fixes a latent thread-safety issue: the render dictionaries were plain (non-concurrent) `Dictionary`s enumerated by the render thread while written from the Akka/gRPC callback thread.
|
||||
- A pulsing "Live" indicator replaces the static "Connected" badge when streaming is active.
|
||||
- Subscribe-on-demand — stream starts when opened, stops when closed.
|
||||
- Read-only per-instance view (one instance per connection); no alarm acknowledgement is available from Debug View.
|
||||
@@ -192,8 +195,10 @@ Per-leaf alarm rendering (leaf nodes are individual conditions for native alarms
|
||||
- **Data path** — no new site-side code and no central alarm store. The page selects a site, queries its deployed instances, then fans out the existing per-instance `DebugViewSnapshot` Ask **concurrently** (capped with a `SemaphoreSlim`) and aggregates the returned `AlarmStates` client-side. The fan-out is **partial-results tolerant**: instances that time out are listed as "not reporting" while the rest still render. This snapshot fan-out now doubles as the **seed** for a near-real-time live feed (see **Live updates** below) rather than the sole refresh mechanism.
|
||||
- **Live updates** — the page is driven by a **transient, per-site central live alarm cache** (`ISiteAlarmLiveCache`, owned by the Communication component; see [Component-Communication](Component-Communication.md)). On site select the page subscribes to the cache; the cache runs one shared, reference-counted per-site aggregator that **seeds** from the snapshot fan-out and then stays warm on a single **site-wide, alarm-only** `SubscribeSite` gRPC stream (seed-then-stream, dedup by `(InstanceUniqueName, AlarmName, SourceReference)`). Applied deltas raise an in-process change event (mirroring `IDeploymentStatusNotifier`) that the Blazor circuit pushes to the browser via `StateHasChanged()` — no new SignalR hub. `AlarmSummaryService.BuildFromLiveAlarms` rebuilds the roll-up + rows from the cache's current alarm set. The cache is **purely in-memory on the active central node** — there is still **no persisted central alarm store**; on a NodeA↔NodeB failover the new active node re-seeds from scratch.
|
||||
- **View** — roll-up tiles (total active, worst severity, unacked count, per-`AlarmKind` counts) plus a flat, sortable, filterable table. Filters cover instance, `AlarmKind` (Computed / NativeOpcUa / NativeMxAccess), state, acked/unacked, severity threshold, and name search.
|
||||
- **Row virtualization (arch-review WP2.4).** Above `VirtualizeThreshold` (150) visible rows the table switches from a plain `foreach` to Blazor `Virtualize` (`ItemSize=37`, `<tr>` spacers), so a large-site alarm burst renders only the on-screen rows instead of every row in the DOM. Below the threshold the plain `foreach` stays — cheaper than the virtualization machinery and needs no JS interop. Both paths share one row-template, so the switch is invisible to styling/behavior.
|
||||
- **Read-only** — there are no ack / shelve / suppress controls (native alarms remain read-only by design).
|
||||
- **Refresh** — manual refresh button plus the 15s poll timer (mirroring the Health dashboard), now retained as a **fallback + `NotReporting` authority** behind the live cache: when the cache reports `IsLive`, the page renders live-cache state; when a stream is unhealthy, **the aggregator has died (deathwatch resets `IsLive`)**, or a site has not yet seeded, the poll keeps the page fresh so a stream failure never blanks it. When live, the poll updates only the `NotReporting` list and leaves the row set to the delta path, so a slow fan-out can never momentarily revert a fresher live delta (R2 N5). (Aggregated live stream **delivered 2026-07-10** — see `docs/plans/2026-07-10-aggregated-live-alarm-stream-plan.md`.)
|
||||
- **Refresh** — manual refresh button plus the 15s poll timer (mirroring the Health dashboard), now retained as a **fallback + `NotReporting` authority** behind the live cache: when the cache reports `IsLive`, the page renders live-cache state; when a stream is unhealthy, **the aggregator has died (deathwatch resets `IsLive`)**, or a site has not yet seeded, the poll keeps the page fresh so a stream failure never blanks it. Since WP2.3 `IsLive` also tracks the *stream* itself: a site-wide stream that faults or ends gracefully drops `IsLive` on the spot, so the page falls back to polling for the reopen window instead of rendering a snapshot that has quietly stopped updating. When live, the poll updates only the `NotReporting` list and leaves the row set to the delta path, so a slow fan-out can never momentarily revert a fresher live delta (R2 N5). (Aggregated live stream **delivered 2026-07-10** — see `docs/plans/2026-07-10-aggregated-live-alarm-stream-plan.md`.)
|
||||
- **Poll fan-out is deduplicated too (arch-review WP2.4).** The cold-cache/fallback poll runs through `SharedAlarmSummaryService`, a process-level memoizing façade over `AlarmSummaryService`: one memo slot per site, single-flight, just-under-15s window, so N operators watching the same site's fallback poll cost the node ONE per-instance debug-snapshot fan-out per window, not N. While the live cache is serving a site there is no poll fan-out at all — the façade instead answers straight from `ISiteAlarmLiveCache` (same `BuildFromLiveAlarmsCore` path the live subscription uses), so the aggregator's own seed/reconcile fan-out is the only one running.
|
||||
- **Reuse** — the alarm badge/formatter markup is factored out of Debug View into a shared `AlarmStateBadges` component consumed by both Debug View and this page.
|
||||
|
||||
### Parked Message Management (Deployment Role)
|
||||
@@ -232,6 +237,7 @@ Per-leaf alarm rendering (leaf nodes are individual conditions for native alarms
|
||||
- Headline **Notification Outbox KPI tiles** — queue depth, stuck count, and parked count. These are central-computed by the Notification Outbox from the central `Notifications` table (not part of any site health report). The full outbox view is on the dedicated Notification Outbox page.
|
||||
- Headline **Site Call Audit KPI tiles** — buffered count, parked count, and failed-last-interval. These are central-computed by the Site Call Audit component from the central `SiteCalls` table (not part of any site health report). The full cached-call view is on the dedicated Site Calls page.
|
||||
- Headline **Audit KPI tiles** — three tiles in a new "Audit" KPI group: **Audit volume**, **Audit error rate**, and **Audit backlog**. These are sourced from the Audit Log component (#23) and Health Monitoring per the metric definitions in Component-HealthMonitoring.md; the dashboard simply surfaces them. The full audit query view is on the dedicated Audit Log page.
|
||||
- **Shared KPI cache (arch-review WP2.4).** All four KPI families above — Notification Outbox, Site Call Audit, Audit, and their per-site/per-node breakdowns — are read through a process-level `IKpiSnapshotCache`, not queried per page load. Each accessor is independently memoized (8s TTL) with single-flight production, so N Blazor circuits polling the same KPI inside one window (e.g. ten operators with the Health dashboard open) cost the node ONE aggregate SQL round trip per KPI per window, not N. A failed query is never memoized — the next caller re-attempts it, and the calling page's existing per-tile "unavailable" degradation is unchanged. Every KPI-tile page (Health, Notification Outbox, Site Calls, Audit Log) shares the same cache instance.
|
||||
|
||||
### Site Event Log Viewer (Deployment Role)
|
||||
- Query site event logs remotely.
|
||||
|
||||
@@ -66,18 +66,25 @@ Both central and site clusters. Each side has communication actors that handle m
|
||||
- Central sends an unsubscribe request over `SiteCommandService` (gRPC command/control) when the debug session ends. The gRPC stream is cancelled. The site's `StreamRelayActor` is stopped and the SiteStreamManager subscription is removed.
|
||||
- The stream is session-based and temporary.
|
||||
- **Accepted limitation (central-side session locality):** debug sessions are process-local state on the central node hosting the `DebugStreamBridgeActor`. A central **restart or failover drops all active sessions** with no `OnStreamTerminated` signal to the operator — the engineer simply re-establishes the debug session from the UI. Debug streaming is an interactive, transient diagnostic aid, so this is an accepted trade-off rather than a durability gap (arch review 02, U7).
|
||||
- **Backpressure is lossy-by-design and now observable:** the per-session `Channel<SiteStreamEvent>` (bounded 1000, `DropOldest`) silently evicts the oldest event when a slow consumer falls behind. The eviction is counted via the channel's `itemDropped` callback and logged at Warning (first eviction, then every 500th) so real event loss on a debug stream is visible instead of silent.
|
||||
- **Backpressure is lossy-by-design and now observable:** the per-session `Channel<SiteStreamEvent>` (bounded `GrpcInstanceStreamChannelCapacity`, default 1000, `DropOldest`) silently evicts the oldest event when a slow consumer falls behind. The eviction is counted via the channel's `itemDropped` callback, logged at Warning (first eviction, then every 500th) and exported as `scadabridge.site.stream.events_dropped` (tagged `stream=instance`), so real event loss on a debug stream is visible instead of silent. The **site-wide alarm feed has its own, larger channel** — see §6.1.
|
||||
- **Bridge-session hardening (WP2.3):**
|
||||
- The pre-snapshot buffer is **bounded (20 000 events, drop-oldest)** and its evictions counted (`scadabridge.central.debug_view.presnapshot_dropped`). It was previously unbounded, so a session whose snapshot never arrived grew without limit on the CENTRAL node. Dropping the oldest is correct here: the snapshot that ends the buffering phase is authoritative for anything that old.
|
||||
- A **hard snapshot deadline** (`DebugStreamBridgeActor.SnapshotTimeout`, 60 s) fails the session if no `DebugViewSnapshot` arrives. Nothing else ended a session wedged in the buffering phase — a lost site reply raises no gRPC error.
|
||||
- **The orphan net measures the CONSUMER, not the mailbox.** A session self-terminates after `DebugStreamBridgeActor.ConsumerIdleTimeout` (5 min) without a sign of life from its consumer, checked by a periodic self-tick (`ConsumerLivenessCheckInterval`, 30 s) against a consumer-last-seen stamp. The stamp is renewed only by `DebugStreamConsumerAlive`, which `DebugStreamService` Tells to every session still in its registry on a shared 30 s timer — holding a session there IS what "a consumer is attached" means, since both consumers (the Blazor debug view and the SignalR hub) release it on dispose/disconnect.
|
||||
This replaced an Akka `SetReceiveTimeout(5 min)`. That version measured the MAILBOX, which conflates site chatter with consumer liveness: a busy site kept an abandoned session alive forever. Marking stream events `INotInfluenceReceiveTimeout` fixed that but left the timeout with nothing recurring to reset it — the snapshot lands once and the stability tick once — so **every healthy session self-terminated ~6 min in and the operator was told "Site disconnected"**. Both failure directions are pinned by tests: streaming events with keepalives survive many windows; an orphaned session terminates while events keep arriving.
|
||||
|
||||
### 6.1 Aggregated Live Alarm Stream (Site → Central)
|
||||
|
||||
Delivered 2026-07-10 (`docs/plans/2026-07-10-aggregated-live-alarm-stream-plan.md`) for the operator Alarm Summary page (Central UI #9). Where §6 streams **one instance** for the interactive Debug View, this feeds a **transient, per-site central live alarm cache** so the Alarm Summary reflects alarm transitions in near-real-time instead of re-polling the whole site every 15s.
|
||||
|
||||
- **Site-wide, alarm-only stream**: a single `SubscribeSite` gRPC stream per site carries `AlarmStateChanged` events for **all** instances (attribute events are dropped — the summary never shows them, and attributes are far higher-volume). On the site this reuses the existing `SiteStreamManager` broadcast hub via `SiteStreamManager.SubscribeSiteAlarms(...)` (same wiring as the per-instance `Subscribe`, minus the `InstanceUniqueName` filter and filtered to alarms). `StreamRelayActor` is reused unchanged, so placeholder rows (`is_configured_placeholder`) are still dropped at the relay.
|
||||
- **Central live cache** (`ISiteAlarmLiveCache`, singleton `SiteAlarmLiveCacheService`): a DI singleton on the active central node. For each site with ≥1 active viewer it runs ONE shared, **reference-counted** per-site aggregator (`SiteAlarmAggregatorActor`); the first `Subscribe(siteId, onChanged)` starts it, the last subscriber leaving stops it after a short **linger** to avoid re-seed thrash. `GetCurrentAlarms(siteId)` returns the current immutable snapshot; `IsLive(siteId)` reports whether the aggregator has seeded and published at least once.
|
||||
- **Central live cache** (`ISiteAlarmLiveCache`, singleton `SiteAlarmLiveCacheService`): a DI singleton on the active central node. For each site with ≥1 active viewer it runs ONE shared, **reference-counted** per-site aggregator (`SiteAlarmAggregatorActor`); the first `Subscribe(siteId, onChanged)` starts it, the last subscriber leaving stops it after a short **linger** to avoid re-seed thrash. `GetCurrentAlarms(siteId)` returns the current immutable snapshot; `IsLive(siteId)` reports whether the aggregator has seeded **and its site-wide stream is currently up**. Liveness rides every publish from the aggregator, so a stream that faults or ends gracefully drops `IsLive` immediately rather than leaving the page grafting a freezing snapshot over fresh poll data until the next reconcile (WP2.3).
|
||||
- **Seed-then-stream** (copied from `DebugStreamBridgeActor` ordering): open the `SubscribeSite` stream first (buffer live deltas), run the snapshot fan-out once via the existing `DebugViewSnapshot` path (bounded by `LiveAlarmCacheSeedConcurrency`), flush the buffer with **dedup by `(InstanceUniqueName, AlarmName, SourceReference)`**, then live pass-through. Placeholders are seeded from the snapshot and never expected on the live stream.
|
||||
- **Failover & drift**: NodeA↔NodeB reconnect (client via `SiteStreamGrpcClient.SubscribeSiteAsync`, factory endpoint failover) re-seeds rather than serving stale; a periodic **reconcile snapshot** (default 60s) corrects any missed enable/disable so the cache can't drift indefinitely. `[PERM]` (`docs/plans/2026-05-29-native-alarms-design.md`): the cache is **purely in-memory** — no EF entity/table/migration, no persisted central alarm store — so a new active node simply re-seeds from scratch.
|
||||
- **Options** (`Communication` section, `CommunicationOptions`; eagerly validated by `CommunicationOptionsValidator` / `ValidateOnStart`): `LiveAlarmCacheLinger` (default 30s), `LiveAlarmCacheReconcileInterval` (default 60s), `LiveAlarmCacheSeedConcurrency` (default 8), `LiveAlarmCacheMaxSubscribersPerSite` (default 200), `LiveAlarmCachePublishCoalesce` (default 250ms; `0` = publish per delta — legacy — batches an alarm storm into one snapshot copy + one viewer fan-out per window; arch review 02 round 2, N6).
|
||||
- **Telemetry** (`ScadaBridgeTelemetry` meter): observable gauge `scadabridge.site.alarm_cache.aggregators.active` (running per-site aggregators) and counter `scadabridge.site.alarm_cache.reconnects` (site-wide stream reconnects — a NodeA↔NodeB flip or reconcile-driven reopen; a sustained climb signals a flapping site link).
|
||||
- **Alarms-only seed (WP2.3)**: the seed/reconcile fan-out sets `DebugSnapshotRequest.AlarmsOnly` (wire: `DebugSnapshotRequestDto.alarms_only`, field 3, additive), so the site builds and ships only the alarm half of the snapshot. The fan-out discarded every attribute row anyway, and an instance's attribute surface dwarfs its alarm set. A pre-WP2.3 site ignores the flag and returns the full snapshot, which reads identically.
|
||||
- **Failover & drift**: a re-seed runs **once per successful (re)connect**, not once per reconnect ATTEMPT — the connect signal is `SiteStreamGrpcClient.SubscribeSiteAsync`'s `onConnected` callback, raised when the site's response headers arrive (the site flushes them as soon as its relay is attached, so no event can be missed after it) or, for a peer that defers its headers, when the FIRST EVENT arrives — whichever comes first, and at most once. A header **timeout** is deliberately not a connect signal: an unreachable or wedged site produces exactly that shape, and reporting it as connected cleared `_streamDown`, consumed the pending re-seed and fanned a whole-site snapshot out at a site that never answered. Fanning a whole-site snapshot out per retry meant N snapshots against a site that was, by definition of the retry, unreachable. A periodic **reconcile snapshot** (default 60s, **jittered** by `LiveAlarmCacheReconcileJitterFraction` so aggregators started together do not stampede one boundary) remains the drift backstop, but it is **skipped when a CONNECT- OR FAILOVER-DRIVEN seed already ran in that window** and **publishes only when the snapshot actually changed the cache** (a diff, not an unconditional viewer fan-out). Staleness stays bounded at two intervals: the skip consumes its flag, so the next tick always fans out. A **tick's own fan-out never arms that skip** — when it did, steady state alternated fan-out/skip and the effective reconcile rate was one per TWO intervals (120 s by default), halving both the not-reporting refresh and the alarm reconcile backstop for no benefit: with no reconnect in play there is nothing duplicated to suppress. A fan-out that fails as a whole now retries on its own **backoff** timer (`reconnectDelay` doubling, capped at 8× the reconcile interval) instead of waiting a full interval. `[PERM]` (`docs/plans/2026-05-29-native-alarms-design.md`): the cache is **purely in-memory** — no EF entity/table/migration, no persisted central alarm store — so a new active node simply re-seeds from scratch.
|
||||
- **Stream terminations are a triad, and every ending hits exactly one leg.** `SiteStreamGrpcClient.ConsumeStreamAsync` classifies a stream's end as a fault (`onError`), a graceful server-side end (`onCompleted`), or our own teardown (neither). Only a cancellation **we asked for** takes the silent leg — the `RpcException(Cancelled)` filter is guarded by `cts.IsCancellationRequested`. A **foreign** `Cancelled` (the peer cancelled, or the channel was disposed underneath us) falls through to `onError`; unguarded, it fired none of the three, so the aggregator kept `_streamDown = false` — `IsLive` stuck true and the reconcile tick's reopen, which only runs on a stream known to be down, never fired.
|
||||
- **Options** (`Communication` section, `CommunicationOptions`; eagerly validated by `CommunicationOptionsValidator` / `ValidateOnStart`): `LiveAlarmCacheLinger` (default 30s), `LiveAlarmCacheReconcileInterval` (default 60s), `LiveAlarmCacheSeedConcurrency` (default 8), `LiveAlarmCacheMaxSubscribersPerSite` (default 200), `LiveAlarmCachePublishCoalesce` (default 250ms; `0` = publish per delta — legacy — batches an alarm storm into one snapshot copy + one viewer fan-out per window; arch review 02 round 2, N6), `LiveAlarmCacheReconcileJitterFraction` (default 0.2 = up to +20% per tick; `0` disables). Stream channel sizing lives alongside the other gRPC limits: `GrpcInstanceStreamChannelCapacity` (default 1000) and `GrpcSiteAlarmStreamChannelCapacity` (default 20 000).
|
||||
- **Telemetry** (`ScadaBridgeTelemetry` meter): observable gauge `scadabridge.site.alarm_cache.aggregators.active` (running per-site aggregators) and counter `scadabridge.site.alarm_cache.reconnects` (site-wide stream reconnects — a NodeA↔NodeB flip or reconcile-driven reopen; a sustained climb signals a flapping site link), and counter `scadabridge.site.alarm_cache.buffer_dropped` (deltas evicted from the aggregator's **bounded** 20 000-entry pre-seed buffer, drop-oldest — non-zero means a fan-out ran long enough for the delta storm behind it to exceed the cap; the fan-out's snapshot is authoritative for the evicted rows).
|
||||
- **Accepted limitations (arch review 02 round 2, N8):**
|
||||
- **Standby-node aggregators**: the live cache is per-node and `SetActorSystem` is wired on every central node, so browsing the standby node directly (diagnostic ports, e.g. 9002) starts a second, fully-functional read-only aggregator + `SubscribeSite` stream there. Accepted — not gated: the aggregator is read-only, bounded (one stream/site, viewer-capped), and torn down by the viewer linger; gating `SetActorSystem` behind the active check would break the diagnostic-browse path and buy nothing. The `[PERM]` "lives only on the active central node" claim is corrected to "per-node; in routine operation only the active node hosts viewers".
|
||||
- **Site deleted while an Alarm Summary viewer is open**: the viewer's aggregator reconciles to an empty snapshot (the deleted site's instances vanish from the fan-out) until its viewers leave, then the linger stop reaps it; the site's cached gRPC channels are disposed at deletion via `SiteStreamGrpcClientFactory.RemoveSiteAsync` (see `ManagementActor.HandleDeleteSite`, arch review 02 round 2, N8).
|
||||
@@ -101,12 +108,12 @@ The streaming protocol is defined in `sitestream.proto` (`src/ZB.MOM.WW.ScadaBri
|
||||
|
||||
- **Service**: `SiteStreamService` — hosted on each site node by `SiteStreamGrpcServer` — exposes six RPCs. Two are real-time **server-streaming** subscriptions (`SubscribeInstance`, `SubscribeSite`); the other four are **unary request/response** calls added by the Audit Log (#23) and Site Call Audit (#22) components. These are distinct from the command/control gRPC services (`CentralControlService` on central, `SiteCommandService` on the site) — `SiteStreamService` is the streaming + audit-pull surface, not the command/control channel:
|
||||
- `SubscribeInstance(InstanceStreamRequest) returns (stream SiteStreamEvent)` — the per-instance real-time debug stream (§6).
|
||||
- `SubscribeSite(SiteStreamRequest) returns (stream SiteStreamEvent)` — the **site-wide, alarm-only** aggregated stream (§6.1) added for the operator Alarm Summary live cache. Additive (new RPC + new `SiteStreamRequest { correlation_id }` message; no field renumbering). The server handler mirrors `SubscribeInstance` — same bounded `Channel(1000, DropOldest)`, `GrpcMaxConcurrentStreams`/`GrpcMaxStreamLifetime` limits, `SiteConnectionOpened/Closed` telemetry, and `StreamRelayActor` mapping — but subscribes via `SiteStreamManager.SubscribeSiteAlarms` (all instances, `AlarmStateChanged` only; attribute events dropped, no per-instance filter).
|
||||
- `SubscribeSite(SiteStreamRequest) returns (stream SiteStreamEvent)` — the **site-wide, alarm-only** aggregated stream (§6.1) added for the operator Alarm Summary live cache. Additive (new RPC + new `SiteStreamRequest { correlation_id }` message; no field renumbering). The server handler mirrors `SubscribeInstance` — but with **its own, larger send channel** (`GrpcSiteAlarmStreamChannelCapacity`, default 20 000, `DropOldest`; WP2.3): sharing the Debug View's 1000-slot channel meant an alarm burst during a WAN stall silently evicted operator-visible transitions to make room for diagnostics traffic. Drops are tagged `stream=site-alarms` on `scadabridge.site.stream.events_dropped`. Otherwise identical — same `GrpcMaxConcurrentStreams`/`GrpcMaxStreamLifetime` limits, `SiteConnectionOpened/Closed` telemetry, and `StreamRelayActor` mapping — but subscribes via `SiteStreamManager.SubscribeSiteAlarms` (all instances, `AlarmStateChanged` only; attribute events dropped, no per-instance filter).
|
||||
- `IngestAuditEvents(AuditEventBatch) returns (IngestAck)` — a legacy central-side ingest surface on the *site*-hosted service; it is **dead in the shipped topology** because no site ever dials a site for ingest. The production audit-telemetry *push* path is site→central gRPC to `CentralControlService`, which routes the batch to the central `AuditLogIngestActor` proxy and returns the accepted `EventId`s.
|
||||
- `IngestCachedTelemetry(CachedTelemetryBatch) returns (IngestAck)` — ingest receiving surface for the combined cached-call telemetry packet (audit row + `SiteCalls` operational upsert written in one transaction).
|
||||
- `PullAuditEvents(PullAuditEventsRequest) returns (PullAuditEventsResponse)` — central→site **reconciliation pull** for the Audit Log self-heal feed; the site serves `Pending`/`Forwarded` rows from its `ISiteAuditQueue`.
|
||||
- `PullAuditEvents(PullAuditEventsRequest) returns (PullAuditEventsResponse)` — central→site **reconciliation pull** for the Audit Log self-heal feed; the site serves `Pending`/`Forwarded` rows from its `ISiteAuditQueue`. **At-least-once (WP2.3):** rows are NOT retired when served. The site retires (`Reconciled`) everything at or before the cursor of the **next** pull, because that cursor is central's only proof of receipt — a fault between the response leaving the site and central committing it re-serves the rows instead of losing them (central dedups on `EventId`). The request carries an additive composite-keyset `after_id` (field 3) mirroring `PullSiteCallsRequest`; with it the read is a strict `(OccurredAtUtc, EventId)` keyset and the retirement is exact, without it the legacy inclusive `>=` read applies and only rows strictly older than the cursor instant are provably received.
|
||||
- `PullSiteCalls(PullSiteCallsRequest) returns (PullSiteCallsResponse)` — central→site reconciliation pull for the Site Call Audit (#22) self-heal feed; the site serves operation-tracking rows changed since a cursor from its `IOperationTrackingStore`. A separate RPC from `PullAuditEvents` because the tracking store is the operational source of truth, distinct from the site audit queue.
|
||||
- **Messages**: `InstanceStreamRequest` (correlation_id, instance_unique_name), `SiteStreamRequest` (correlation_id only — no instance name; drives the site-wide `SubscribeSite` stream), `SiteStreamEvent` (correlation_id, oneof event: `AttributeValueUpdate`, `AlarmStateUpdate`); `AuditEventDto`/`AuditEventBatch`/`IngestAck` for ingest; `CachedTelemetryPacket`/`CachedTelemetryBatch` (each packet pairing an `AuditEventDto` with a `SiteCallOperationalDto`); `PullAuditEventsRequest`/`PullAuditEventsResponse` and `PullSiteCallsRequest`/`PullSiteCallsResponse` (each request carries `since_utc` + `batch_size`; each response carries `more_available` to signal a saturated batch).
|
||||
- **Messages**: `InstanceStreamRequest` (correlation_id, instance_unique_name), `SiteStreamRequest` (correlation_id only — no instance name; drives the site-wide `SubscribeSite` stream), `SiteStreamEvent` (correlation_id, oneof event: `AttributeValueUpdate`, `AlarmStateUpdate`); `AuditEventDto`/`AuditEventBatch`/`IngestAck` for ingest; `CachedTelemetryPacket`/`CachedTelemetryBatch` (each packet pairing an `AuditEventDto` with a `SiteCallOperationalDto`); `PullAuditEventsRequest`/`PullAuditEventsResponse` and `PullSiteCallsRequest`/`PullSiteCallsResponse` (each request carries `since_utc` + `batch_size` + an optional composite-keyset `after_id`; each response carries `more_available` to signal a saturated batch).
|
||||
- The `oneof event` pattern is extensible — future event types (health metrics, connection state changes) are added as new fields without breaking existing consumers.
|
||||
- Proto field numbers are never reused; new RPCs and message fields are appended additively. Old clients ignore unknown `oneof` variants.
|
||||
|
||||
@@ -162,7 +169,7 @@ read the traffic". TLS is follow-on hardening and does not change this design.
|
||||
- **Placeholder rows are dropped at the relay**: `is_configured_placeholder` (field 23) is a **Debug View snapshot-only** concept emitted by `InstanceActor.BuildAlarmStatesSnapshot` for quiet bindings — it is never a real alarm transition (its timestamp may be `DateTimeOffset.MinValue`, the Protobuf `Timestamp` lower boundary). `StreamRelayActor.HandleAlarmStateChanged` therefore returns early — **never relaying a placeholder row to the live gRPC stream** — so field 23 is always `false` on the live stream and only ever carries `true` in the snapshot path.
|
||||
- **Client-side mapping (`SiteStreamGrpcClient.ConvertToDomainEvent`)**: reconstructs the domain `AlarmStateChanged` from the proto — `Kind` is parsed via `ParseAlarmKind`, the `Condition` is rebuilt with `severity` taken from the existing wire `priority`, and native metadata is repopulated from fields 8–23 (`native_source_canonical_name` → `NativeSourceCanonicalName`, `is_configured_placeholder` → `IsConfiguredPlaceholder`) — so central-side consumers receive the same domain event the site emitted.
|
||||
|
||||
> **Regeneration is manual (macOS-only).** `sitestream.proto` is **not** auto-compiled: the `<Protobuf>` include is commented out in the `.csproj`, and the generated C# is **vendored** under `SiteStreamGrpc/`. To regenerate after editing the proto: toggle the `<Protobuf>` include on, build so `Grpc.Tools` regenerates the C#, copy the generated files into `SiteStreamGrpc/`, then re-comment the include. Adding `AlarmStateUpdate` fields 8–23 and the four unary RPCs (`IngestAuditEvents`, `IngestCachedTelemetry`, `PullAuditEvents`, `PullSiteCalls`) plus their message types followed this process.
|
||||
> **Regeneration is manual (macOS-only).** `sitestream.proto` is **not** auto-compiled: the `<Protobuf>` include is commented out in the `.csproj`, and the generated C# is **vendored** under `SiteStreamGrpc/`. To regenerate after editing the proto: toggle the `<Protobuf>` include on, build so `Grpc.Tools` regenerates the C#, copy the generated files into `SiteStreamGrpc/`, then re-comment the include. Adding `AlarmStateUpdate` fields 8–23 and the four unary RPCs (`IngestAuditEvents`, `IngestCachedTelemetry`, `PullAuditEvents`, `PullSiteCalls`) plus their message types followed this process, as did WP2.3's two additive fields — `PullAuditEventsRequest.after_id` (field 3, `sitestream.proto`) and `DebugSnapshotRequestDto.alarms_only` (field 3, `site_command.proto`). The same applies to `site_command.proto`/`SiteCommandGrpc/` and `central_control.proto`/`CentralControlGrpc/`; `docker/regen-proto.sh [sitestream|centralcontrol|sitecommand|all]` automates the toggle-build-copy-untoggle and always restores the csproj.
|
||||
|
||||
#### gRPC Connection Keepalive
|
||||
|
||||
@@ -343,6 +350,7 @@ Disconnect is detected at the **transport layer**, never via an application-leve
|
||||
|
||||
- **In-flight command/control messages (gRPC call + deadline)**: When a connection drops while a request is in flight (e.g., a deployment sent but no response received), the gRPC call fails or hits its deadline and the caller receives a failure. There is **no automatic retry or buffering at central** — the engineer sees the failure in the UI and re-initiates the action. This is consistent with the design principle that central does not buffer messages. An in-progress deployment whose round-trip exceeds the timeout (default 120 s at `CommunicationService.DeployInstanceAsync`) surfaces as `DeploymentStatus.Failed` to the caller.
|
||||
- **Debug streams (gRPC)**: Any gRPC stream interruption is detected by the HTTP/2 keepalive PING (~25 s) and triggers reconnection logic in the `DebugStreamBridgeActor`. The bridge actor attempts to reconnect to the other site node endpoint (NodeB if NodeA failed, or vice versa), with up to 3 retries and 5-second backoff. If all retries fail, the consumer is notified via `OnStreamTerminated` and the bridge actor is stopped. Events during the reconnection gap are lost (acceptable for real-time debug view). On successful reconnection, the consumer can request a fresh snapshot to re-sync state.
|
||||
- **Graceful end of stream is NOT a fault, but it IS a reconnect trigger.** When the site's `GrpcMaxStreamLifetime` (4 h) elapses — and on a graceful site shutdown — the server ends the RPC with status **OK**, so the client's read loop simply finishes with no exception. `SiteStreamGrpcClient` reports that through a dedicated `onCompleted` callback, distinct from `onError`, and both consuming actors treat it as a reconnect trigger that **does not spend (or refund) the error-retry budget** and does **not** flip the node — the peer that closed cleanly is healthy. `SiteAlarmAggregatorActor` marks the stream down and lets the periodic reconcile tick reopen it (which also re-seeds); `DebugStreamBridgeActor` reopens on the same endpoint after `ReconnectDelay`. Both stamp the stream generation on the callback so a completion racing out of an already-replaced stream is ignored, and both observe the subscription `Task` itself so a fault escaping the subscribe call can never go unobserved. Without the `onCompleted` leg an OK completion was invisible to the reconnect logic and the stream stayed silently dead until the central process restarted.
|
||||
|
||||
## Failover Behavior
|
||||
|
||||
|
||||
@@ -68,7 +68,7 @@ The configuration database stores all central system data, organized by domain a
|
||||
- **New tables use UTC `DateTime` + the UTC value converter**, not `DateTimeOffset`. The `Notification` entity's `DateTimeOffset` timestamps are the **documented legacy exception** — on the SQLite test harness the `DateTimeOffset`→ISO-8601-string converter forced in-memory reductions in a few KPI paths (now pushed server-side where translatable). Prefer `datetime2` UTC columns for any new table.
|
||||
- **New repositories write MSSQL-only SQL with MSSQL-backed test fixtures** (`SkippableFact` gated on the live SQL Server container). The `NotificationOutboxRepository` dual SQLite/T-SQL dialect is the **documented legacy exception** (finding S8) — it predates the MSSQL-fixture convention; do not copy that pattern.
|
||||
- **AuditLog clustered key.** The `dbo.AuditLog` primary/clustered key is `(EventId, OccurredAtUtc)` — `OccurredAtUtc` is required by the monthly partition scheme (`ps_AuditLog_Month`), and `EventId` carries a separate unique index for `InsertIfNotExists` idempotency. **Tracked follow-up (P3):** benchmark whether flipping the clustered key to `(OccurredAtUtc, EventId)` improves range-scan locality once real volume exists.
|
||||
- **Tracked follow-ups (P4/P6):** the per-node KPI sampling cadence and the AuditLog reconciliation-pull keyset paging (the SiteCalls reconciliation pull already uses a composite keyset cursor; `PullAuditEvents` remains inclusive-`>=`, idempotent on `EventId`) are deferred, lower-urgency hardening items.
|
||||
- **Tracked follow-ups (P4/P6):** the per-node KPI sampling cadence and the AuditLog reconciliation-pull keyset paging (the SiteCalls reconciliation pull already uses a composite keyset cursor; `PullAuditEvents` gained the site-side `(OccurredAtUtc, EventId)` keyset + `after_id` field in WP2.3, but central still sends a bare `sinceUtc`, so the effective read stays inclusive-`>=`, idempotent on `EventId`) are deferred, lower-urgency hardening items.
|
||||
|
||||
### Audit Log
|
||||
- **AuditLog**: The central, append-only audit table owned by the Audit Log component — one row per script-trust-boundary lifecycle event across all channels (outbound API calls, outbound DB writes/reads, notifications, and inbound API requests). Sibling of the `Notifications` and `SiteCalls` tables but distinct: `AuditLog` is the immutable history that observes the other subsystems, not an operational state store.
|
||||
@@ -95,9 +95,11 @@ The configuration database stores all central system data, organized by domain a
|
||||
- `ParentExecutionId` — `CAST(JSON_VALUE(DetailsJson,'$.parentExecutionId') AS uniqueidentifier)` PERSISTED — spawner's `ExecutionId`; null for top-level runs.
|
||||
- `IngestedAtUtc` — `CAST(SWITCHOFFSET(CAST(JSON_VALUE(DetailsJson,'$.ingestedAtUtc') AS datetimeoffset), 0) AS datetime2(7))` — central ingest timestamp; **not** persisted (SQL Server rejects PERSISTED on the non-deterministic `SWITCHOFFSET` expression).
|
||||
|
||||
*Clustered primary key:* `(EventId, OccurredAtUtc)` — composite so the key is partition-aligned. `UX_AuditLog_EventId` (unique, non-aligned on `[PRIMARY]`) enforces global `EventId` uniqueness for `InsertIfNotExistsAsync` idempotency.
|
||||
*Clustered primary key:* `(EventId, OccurredAtUtc)` — composite so the key is partition-aligned, and the **sole** `EventId` uniqueness enforcement backing `InsertIfNotExistsAsync` idempotency. `EventId` is a GUID minted once at the emitting site in the same operation that stamps `OccurredAtUtc`, and neither field is ever re-stamped downstream, so a given `EventId` always arrives with the same `OccurredAtUtc` and can only map to one partition — pair-uniqueness is `EventId`-uniqueness for every row the system produces. The idempotency probe (`WHERE EventId = @id`) still seeks the clustered key's leading column, at the cost of one seek per partition rather than one seek overall.
|
||||
|
||||
*Indexes* (all non-clustered, partition-aligned on `ps_AuditLog_Month(OccurredAtUtc)` except `UX_AuditLog_EventId`):
|
||||
A non-aligned single-column `UX_AuditLog_EventId` on `[PRIMARY]` used to carry that uniqueness; it was dropped by the `AlignAuditLogEventIdUniqueness` migration because a non-aligned index blocks `ALTER TABLE … SWITCH PARTITION`, forcing the retention purge to drop it, switch, and rebuild it **offline inside the switch transaction** on every run.
|
||||
|
||||
*Indexes* (all non-clustered, partition-aligned on `ps_AuditLog_Month(OccurredAtUtc)`):
|
||||
- `IX_AuditLog_OccurredAtUtc` (primary time-range index for global scans)
|
||||
- `IX_AuditLog_Site_Occurred (SourceSiteId, OccurredAtUtc)` (per-site filters)
|
||||
- `IX_AuditLog_CorrelationId (CorrelationId) WHERE CorrelationId IS NOT NULL` (drilldown from a single operation)
|
||||
@@ -377,6 +379,22 @@ dotnet ef migrations script FromMigration ToMigration --output migration.sql
|
||||
|
||||
Generated scripts are idempotent — they can be safely re-run without causing errors or duplicate changes.
|
||||
|
||||
### Running a Script: QUOTED_IDENTIFIER
|
||||
|
||||
`SiteCalls`, `Notifications` and `AuditLog` all carry **filtered indexes**, and `AuditLog` additionally carries persisted computed columns. SQL Server refuses both the filtered-index DDL and any subsequent `INSERT`/`UPDATE`/`DELETE` on such a table with **error 1934** unless the session has `QUOTED_IDENTIFIER ON`.
|
||||
|
||||
`sqlcmd` defaults that option **OFF** — including the `docker exec … /opt/mssql-tools18/bin/sqlcmd` invocation used throughout the test-infra docs — so a script that applies cleanly in SSMS (which defaults ON) fails there. Two mitigations, both in place:
|
||||
|
||||
- Every checked-in script under `docs/plans/sql/` begins with a `SET QUOTED_IDENTIFIER ON; SET ANSI_NULLS ON; GO` header, in its own batch so the setting is in force when the following batch parses.
|
||||
- Run `sqlcmd` with **`-I`** (enable quoted identifiers) regardless — it is the correct default for every script this repo generates, and it protects hand-written one-off statements that carry no header:
|
||||
|
||||
```bash
|
||||
docker exec -i scadabridge-mssql /opt/mssql-tools18/bin/sqlcmd \
|
||||
-S localhost -U sa -P 'ScadaBridge_Dev1#' -C -I -d ScadaBridgeConfig -i /path/to/script.sql
|
||||
```
|
||||
|
||||
The application itself is unaffected: SqlClient sets `QUOTED_IDENTIFIER ON` on every connection, so EF Core migrations and all runtime DML already run with it on.
|
||||
|
||||
---
|
||||
|
||||
## Seed Data
|
||||
@@ -414,6 +432,7 @@ The `AuditLog` table is append-only and grows by every script-trust-boundary eve
|
||||
- Connection strings are provided via the Host's `DatabaseConfiguration` options (bound from `appsettings.json`).
|
||||
- EF Core manages connection pooling via the underlying ADO.NET SQL Server provider.
|
||||
- The DbContext is registered as a **scoped** service in the DI container, ensuring each request/operation gets its own instance.
|
||||
- **`DbContext` pooling (`AddDbContextPool`) is deliberately NOT used (arch-review WP2.2 — verified, not overlooked).** Pooling requires a context with a single public constructor taking only `DbContextOptions<TContext>`; `ScadaBridgeDbContext` has a second, runtime constructor taking `IDataProtectionProvider`, because the encrypting value converter for secret-bearing columns is built during `OnModelCreating` from that provider — and the model itself *differs* between the two constructors (no provider ⇒ no encrypting converter), so a pooled activator could silently produce a context that reads secret columns as ciphertext. Adopting pooling would mean moving the protector out of the constructor into a `DbContextOptions` extension — a change to the secrets-at-rest path, not a performance refactor — so it is deferred rather than folded into this pass.
|
||||
- No connection management for the Machine Data Database — that is handled separately by consumers (Inbound API scripts, external system gateway).
|
||||
|
||||
---
|
||||
|
||||
@@ -38,6 +38,23 @@ IDataConnection : IAsyncDisposable
|
||||
|
||||
The `Disconnected` event is raised by an adapter when it detects an unexpected connection loss (server offline, network failure, keep-alive timeout). The `DataConnectionActor` subscribes to this event to trigger the reconnection state machine. Additional protocols can be added by implementing this interface.
|
||||
|
||||
### Batch Capability Seam
|
||||
|
||||
A protocol whose wire form can subscribe MANY tags in one round trip additionally implements the optional capability interface `IBatchSubscribableConnection` (same pattern as `IBrowsableDataConnection` / `IAlarmSubscribableConnection`):
|
||||
|
||||
```
|
||||
IBatchSubscribableConnection
|
||||
├── SubscribeBatchAsync(tagPaths, callback) → IReadOnlyList<TagSubscribeResult>
|
||||
└── UnsubscribeBatchAsync(subscriptionIds) → void
|
||||
```
|
||||
|
||||
- **One shared callback, plain tag list.** Every tag in a batch shares one callback; `SubscriptionCallback` already carries the tag path, so per-tag delegates would carry nothing.
|
||||
- **Partial-failure contract.** A per-tag fault (bad node id, unresolvable path) is a result row with `Success:false` and never aborts the batch — mirroring the gateway's `SubscribeResult` and OPC UA's per-monitored-item create status. A **thrown** exception means the whole batch failed; the actor classifies it exactly as the per-tag path does and drives the Reconnecting state machine on a connection-level fault.
|
||||
- Implementing the interface also asserts that `ReadBatchAsync` / `WriteBatchAsync` are **true bulk** service calls rather than loops over the single-tag methods. Both shipped adapters (`OpcUaDataConnection`, `MxGatewayDataConnection`) implement it; an adapter that does not keeps the historical per-tag behaviour, so the capability is purely additive.
|
||||
- Single-tag `SubscribeAsync` / `ReadAsync` / `WriteAsync` are **kept** (heartbeat monitor, interactive read/write paths) and delegate to the batch form as a batch of one.
|
||||
|
||||
Sizing: at the 37,500-tag site target (`docs/deployment/topology-guide.md`), the pre-batch path issued one adapter round trip — and, on OPC UA, one `ApplyChanges` — per tag on every subscribe and every reconnect.
|
||||
|
||||
### Common Value Type
|
||||
|
||||
All protocols produce the same value tuple consumed by Instance Actors. Before the first value update arrives from the DCL, data-sourced attributes are held at **uncertain** quality by the Instance Actor (see Site Runtime — Initialization):
|
||||
@@ -147,6 +164,7 @@ All settings are parsed from the data connection's configuration JSON dictionari
|
||||
| `MaxNotificationsPerPublish` | int | `100` | Max notifications batched per publish cycle |
|
||||
| `SamplingIntervalMs` | int | `1000` | Per-item server sampling rate in milliseconds |
|
||||
| `QueueSize` | int | `10` | Per-item notification buffer size |
|
||||
| `MaxMonitoredItemsPerSubscription` | int | `5000` | Monitored-item budget per OPC UA subscription. Above it the adapter **shards** onto an additional subscription on the same session (37,500 tags → 8 shards). Per endpoint, because item-count ceilings are a property of the target server. |
|
||||
| `SecurityMode` | string | `None` | Preferred endpoint security: `None`, `Sign`, or `SignAndEncrypt` |
|
||||
| `AutoAcceptUntrustedCerts` | bool | `true` | Accept untrusted server certificates |
|
||||
|
||||
@@ -172,9 +190,17 @@ These are configured via `DataConnectionOptions` in `appsettings.json`, not per-
|
||||
| Setting | Default | Description |
|
||||
|---------|---------|-------------|
|
||||
| `ReconnectInterval` | 5s | Fixed interval between reconnection attempts |
|
||||
| `TagResolutionRetryInterval` | 10s | Retry interval for unresolved tag paths |
|
||||
| `TagResolutionRetryInterval` | 10s | **Floor** interval for the unresolved-tag retry; the retry backs off from here |
|
||||
| `TagResolutionRetryMaxInterval` | 5m | Ceiling for the exponential tag-resolution backoff |
|
||||
| `WriteTimeout` | 30s | Timeout for write operations |
|
||||
| `SeedReadTimeout` | 30s | Per-tag timeout for seed reads on the initial-subscribe (and reconnect re-seed) path. A hung device read is treated as a failed seed: retried up to `SeedReadMaxAttempts`, then the tag stays Uncertain until a change notification arrives. |
|
||||
| `SeedReadTimeout` | 30s | **Per-chunk** timeout for seed reads on the initial-subscribe (and reconnect re-seed) path. A hung device read is treated as a failed seed: retried up to `SeedReadMaxAttempts`, then the tag stays Uncertain until a change notification arrives. |
|
||||
| `SeedOverallTimeout` | 120s | Wall-clock deadline for the WHOLE seed (all chunks, all retry rounds). Replaces the old 30s-per-tag serial worst case; remaining tags are logged and stay Uncertain. |
|
||||
| `SeedReadBatchSize` | 250 | Tags per seed-read chunk |
|
||||
| `SeedReadMaxParallelism` | 4 | Seed-read chunks in flight concurrently |
|
||||
| `SubscribeBatchSize` | 500 | Tags per adapter subscribe round trip (initial subscribe, reconnect re-subscribe, resolution probes) |
|
||||
| `SubscribeBatchDelay` | 50ms | Delay BETWEEN reconnect re-subscribe chunks. Not applied on the instance-driven subscribe path — the Deployment Manager already staggers instance startup there. |
|
||||
| `QualityFlushInterval` | 1s | Coalescing window for pushing tag-quality counters to the health collector |
|
||||
| `MxSupervisoryAdviseParallelism` | 16 | Max in-flight supervisory advise commands on the MxGateway bulk-subscribe path when the endpoint has no write-user context |
|
||||
|
||||
## Subscription Management
|
||||
|
||||
@@ -184,6 +210,22 @@ These are configured via `DataConnectionOptions` in `appsettings.json`, not per-
|
||||
- When an Instance Actor is stopped (due to disable, delete, or redeployment), the DCL cleans up the associated subscriptions.
|
||||
- When a new Instance Actor is created for a redeployment, subscriptions are established fresh based on the new configuration.
|
||||
|
||||
### Batching, chunking and pacing
|
||||
|
||||
Three subscribe paths, deliberately paced differently:
|
||||
|
||||
| Path | Shape |
|
||||
|---|---|
|
||||
| **Instance-driven subscribe** (`SubscribeTagsRequest`, i.e. site failover / deploy) | One `SubscribeBatchAsync` per `SubscribeBatchSize` chunk with **no** added delay. The Deployment Manager already staggers instance startup (`SiteRuntimeOptions.StartupBatchSize`/`StartupBatchDelayMs`), so requests arrive pre-spaced at ~75 tags each — double-staggering would only slow failover. |
|
||||
| **Reconnect re-subscribe** (`ReSubscribeAll`) | **Sequential** chunks of `SubscribeBatchSize` with `SubscribeBatchDelay` between them, all inside ONE background task; each chunk reports its per-tag results back to the actor as its own message. Replaces a tight loop that fired one fire-and-forget task per tag. |
|
||||
| **Tag-resolution probe** | The whole not-in-flight unresolved set, chunked at `SubscribeBatchSize`, as one probe round per timer tick. |
|
||||
|
||||
Seeding (the initial-value read that keeps a STATIC tag from staying Uncertain) issues chunked bulk reads — `SeedReadBatchSize` per chunk, `SeedReadMaxParallelism` chunks in flight, `SeedReadTimeout` per chunk — under a single `SeedOverallTimeout` deadline covering every chunk and retry round.
|
||||
|
||||
**OPC UA subscription sharding.** `RealOpcUaClient` places monitored items on the first subscription shard with free capacity (`MaxMonitoredItemsPerSubscription`), creating a shard when all are full and deleting one once it empties. A batch is N × `AddItem` plus **one** `ApplyChanges` per touched shard. Alarms & Conditions event items are pinned to a **dedicated event shard**, so `ConditionRefresh` has a single subscription id to target and `QueueSize:1000` event items never consume data-shard capacity. Every structural mutation (AddItem / RemoveItem / ApplyChanges / Create / Delete) and every shard-list mutation serializes behind ONE per-client gate — the SDK `Subscription` is not safe under concurrent structural mutation, and the subscribe task, resolution probe and alarm subscribe are otherwise unordered.
|
||||
|
||||
**MxGateway.** A batch is one `SubscribeBulk` gateway command (AddItem + Advise per tag in a single worker pass), replacing the historical 2 RPCs per tag. When the endpoint has no write-user context (`WriteUserId == 0`) the worker offers no BULK supervisory advise, so the client issues one `AddItemBulk` and then pipelines the per-item supervisory advises with a bounded window (`MxSupervisoryAdviseParallelism`). A cross-repo follow-up (an additive `supervisory` flag on `SubscribeBulkCommand` in mxaccessgw) would collapse that to one RPC as well.
|
||||
|
||||
## Write-Back Support
|
||||
|
||||
- When a script calls `Instance.SetAttribute` for an attribute with a data source reference, the Instance Actor sends a write request to the DCL.
|
||||
@@ -263,6 +305,8 @@ The registered `SourceReference` is a **NodeId** for OPC UA (the picker, CSV and
|
||||
- **Capability check**: if `_adapter is not IAlarmSubscribableConnection`, the actor replies `SubscribeAlarmsResponse(Success = false, ...)`.
|
||||
- **Reconnect handling**: on entering **Reconnecting**, the actor pushes a `NativeAlarmSourceUnavailable` to every alarm subscriber (consumers mark mirrored alarms uncertain rather than clearing them). On successful reconnection it re-subscribes the feed; the adapter re-emits a snapshot, reconciling state.
|
||||
- **Shared condition filter (last-subscriber-wins)**: because the feed is opened once per source, it carries a single condition filter. A second subscriber that registers a *different* filter for the same source overwrites it (last writer wins) and the actor logs a warning — co-subscribers to one source are expected to agree on the filter.
|
||||
- **Routing index**: subscribed sources are bucketed by the **first path segment** of their reference (everything before the first `.`), and a transition is matched only against the bucket for its own first segment, plus a residue list of sources that carry no separator at all (a prefix shorter than one segment, which can match other buckets). This is sound because a source reference containing a separator can only prefix a reference sharing its entire first segment. The per-source `StartsWith` test and the condition-type gate inside the bucket are unchanged, so routing decisions are identical to the previous linear scan over every subscribed source; the `SnapshotComplete` sentinel still bypasses the index entirely and is broadcast to every subscriber.
|
||||
- **Gateway union filter (MxGateway)**: `StreamAlarms` carries a single `alarm_filter_prefix`, so the adapter opens the stream on the **longest common prefix** of the currently subscribed source references and restarts it only when a NEW source falls outside that prefix (the source then replays a fresh snapshot, which `NativeAlarmActor` already handles). An unsubscribe never restarts the stream — the prefix it leaves behind is at worst too broad, which costs bandwidth, not correctness. The prefix is a bandwidth optimisation only; the actor's per-source + condition-type gate remains authoritative, mirroring the OPC UA server-side WhereClause stance.
|
||||
|
||||
### Protocol-Neutral Types & Messages
|
||||
|
||||
@@ -359,8 +403,9 @@ When the DCL subscribes to a tag path from the flattened configuration but the p
|
||||
|
||||
1. The failure is **logged to Site Event Logging**.
|
||||
2. The attribute is marked with quality `bad`.
|
||||
3. The DCL **periodically retries resolution** at a configurable interval, accommodating devices that come online in stages or load modules after startup.
|
||||
4. On successful resolution, the subscription activates normally and quality reflects the live value from the device.
|
||||
3. The DCL **retries resolution with exponential backoff**, accommodating devices that come online in stages or load modules after startup: the interval starts at `TagResolutionRetryInterval` (10s), doubles after a round in which nothing resolved, and is capped at `TagResolutionRetryMaxInterval` (5m). It resets to the floor as soon as **any** tag resolves or the connection reconnects, so a device that is merely slow to boot is still picked up quickly while a dead one is not probed at full width forever. Each round probes the whole unresolved set in `SubscribeBatchSize` chunks, not one call per tag.
|
||||
4. The retry timer is **single-shot, rescheduled on probe completion** — a periodic timer cannot back off, and rescheduling only on completion is what keeps a fan-out of failures from resetting the clock (the anti-starvation property previously defended by an `IsTimerActive` gate, which is retained).
|
||||
5. On successful resolution, the subscription activates normally and quality reflects the live value from the device.
|
||||
|
||||
Note: Pre-deployment validation at central does **not** verify that tag paths resolve to real tags on physical devices — that is a runtime concern handled here.
|
||||
|
||||
@@ -370,6 +415,7 @@ The DCL reports the following metrics to the Health Monitoring component via the
|
||||
|
||||
- **Connection status**: `connected`, `disconnected`, or `reconnecting` per data connection.
|
||||
- **Tag resolution counts**: Per connection, the number of total subscribed tags vs. successfully resolved tags. This gives operators visibility into misconfigured templates without needing to open the debug view for individual instances.
|
||||
- **Tag quality counters** are pushed on a genuine quality **transition** only, coalesced onto a `QualityFlushInterval` (1s) single-shot timer. Counter arithmetic still runs per message; only the collector push is deferred, and a value whose quality is unchanged moves no counter at all. Health reports poll at 30s, so the coalescing loses nothing. Three paths flush **synchronously** because their correctness depends on it: the bad-quality push on disconnect, unsubscribe, and the reconnect counter reset.
|
||||
|
||||
## Dependencies
|
||||
|
||||
|
||||
@@ -137,6 +137,71 @@ When deploying artifacts (shared scripts, external system definitions, etc.) to
|
||||
- Deployment is performed at the **individual instance level**.
|
||||
- The UI may provide convenience operations (e.g., "deploy all out-of-date instances at Site A"), but these decompose into individual instance deployments.
|
||||
|
||||
### Bulk site deployment (`DeploySiteAsync`, WP2.5)
|
||||
|
||||
`DeploySiteAsync(siteId, user)` deploys every instance at one site in a single
|
||||
operation, surfaced as the `MgmtDeploySite` management command and the CLI
|
||||
`deploy site --site-id`. It is the "deploy all at Site A" convenience above, made
|
||||
first-class — it still decomposes into individual instance deployments, and each
|
||||
instance keeps its own deployment id, revision hash, operation lock, and
|
||||
optimistically-concurrent status record.
|
||||
|
||||
It runs the ordinary deployment pipeline in three phases, of which only the middle
|
||||
one is parallel:
|
||||
|
||||
1. **Prepare (serial).** Validate transition, take the operation lock, flatten +
|
||||
validate, run query-before-redeploy reconciliation, insert the `InProgress`
|
||||
record. Every step here touches the scoped, non-thread-safe `DbContext`, so the
|
||||
phase is strictly serial. All instances share ONE `FlattenSession`, so a
|
||||
template chain common to N instances is walked once and the session-global
|
||||
queries (shared scripts, schema library, the site's data connections) run once
|
||||
for the batch.
|
||||
2. **Send (bounded parallel).** Concurrently up to `SiteDeploymentMaxParallelism`
|
||||
(default 4), each under a `SiteDeploymentTimeoutPerInstance` deadline
|
||||
(default 120 s): stage the `PendingDeployment` row **immediately before** that
|
||||
instance's `RefreshDeploymentCommand` round-trip, not upfront in phase 1 —
|
||||
`PendingDeployment` carries a 5-minute TTL, and staging every instance in the
|
||||
batch serially in phase 1 would burn a chunk of that TTL for the tail
|
||||
instances before their round-trip even starts (review fix, commit
|
||||
`e0e4b246`). Staging touches the repository but is scoped per-instance, so it
|
||||
does not reintroduce serialization. Shape mirrors
|
||||
`ArtifactDeploymentService.DeployCoreAsync`.
|
||||
3. **Finalize (serial).** Commit terminal statuses, apply post-success side
|
||||
effects, write audit rows, release each operation lock.
|
||||
|
||||
**Not all-or-nothing.** An instance that fails for any reason (wrong state, failed
|
||||
validation, lock already held by another operation, site round-trip timed out) is
|
||||
reported as a failed row while the rest proceed, and is individually retryable via
|
||||
the ordinary single-instance deploy. This matches the artifact-deployment policy:
|
||||
successful targets are never rolled back because another target failed.
|
||||
|
||||
**Cancellation is lock-safe (review fix, commit `e0e4b246`).** A cancelled bulk
|
||||
deploy no longer leaks the per-instance operation lock of any instance that had
|
||||
already taken one — a leaked lock is a wedged semaphore that never releases for
|
||||
the life of the process. Phase 2 does not throw on cancellation; each in-flight
|
||||
instance is instead recorded with a `Failed` outcome so phase 3 (finalize) still
|
||||
runs for it. An escape from phase 1 or phase 3 unwinds every not-yet-finalised
|
||||
entry the same way: `Failed` status plus lock release, so no instance touched by
|
||||
a cancelled bulk deploy can be left holding its lock.
|
||||
|
||||
`DeployInstanceAsync` is composed from the same three phase helpers with a batch of
|
||||
one, so the two entry points cannot drift on deployment identity, idempotency, lock
|
||||
coverage, or optimistic concurrency.
|
||||
|
||||
### Terminal deployment-record retention
|
||||
|
||||
Deployment records are insert-only — one row per deploy attempt — so without a
|
||||
window the table grows for the life of the system and every unfiltered deployment
|
||||
query degrades with age rather than with page size.
|
||||
`TerminalDeploymentRecordRetention` (default **365 days**, deliberately generous
|
||||
because deployment history is operator forensics) bounds it, swept opportunistically
|
||||
and rate-limited on the deployment-list read path.
|
||||
|
||||
Only **terminal** rows (`Success` / `Failed`) are eligible. An `InProgress` row is
|
||||
never purged regardless of age: it is precisely the row the query-before-redeploy
|
||||
reconciliation reads to decide whether a prior deploy actually landed at the site,
|
||||
and expiring it by age would silently disable that idempotency guard.
|
||||
|
||||
## Diff View
|
||||
|
||||
Before deploying, the Deployment Manager can request a diff from the Template Engine showing:
|
||||
|
||||
@@ -87,7 +87,7 @@ Both endpoints honour any site-scope rules attached to the caller's audit role b
|
||||
|
||||
### Templates
|
||||
|
||||
- **ListTemplates** / **GetTemplate**: Query template definitions. `ListTemplates` supports additive offset paging via `Skip` / `Take` (arch-review P2): `Take = null` (the default) preserves the historical unpaged behaviour (unlimited), a set `Take` is clamped to `1..1000` and `Skip` floors at `0`.
|
||||
- **ListTemplates** / **GetTemplate**: Query template definitions. `ListTemplates` supports additive offset paging via `Skip` / `Take` (arch-review P2): `Take = null` (the default) preserves the historical unpaged behaviour (unlimited), a set `Take` is clamped to `1..1000` and `Skip` floors at `0`. **Paging and projection are DB-side (WP2.5):** the handler reads `TemplateSummary` rows (child collections reduced to counts, no script bodies) via `GetTemplateSummariesAsync`, instead of materialising every template's full five-Include child graph and then discarding all but one page in memory. A caller that needs a template's members already fetches it with `GetTemplate`.
|
||||
- **CreateTemplate** / **UpdateTemplate** / **DeleteTemplate**: Manage templates.
|
||||
- **ValidateTemplate**: Run on-demand pre-deployment validation (flattening, naming collisions, script compilation).
|
||||
- **GetTemplateDiff**: Compare deployed vs. template-derived configuration for an instance.
|
||||
@@ -115,9 +115,10 @@ The whole folder surface is reachable from the CLI as `template folder list|crea
|
||||
|
||||
### Instances
|
||||
|
||||
- **ListInstances** / **GetInstance**: Query instances, with filtering by site and area. `ListInstances` supports additive offset paging via `Skip` / `Take` (arch-review P2), applied **after** the in-memory site-scope filter so a site-scoped user never sees an out-of-scope instance surface into their page window; same `Take = null` unlimited default and `1..1000` clamp as `ListTemplates`. (`QueryDeployments` and `ExportBundle` still load the full table internally — a logged scale follow-up, arch-review P2 deferred.)
|
||||
- **ListInstances** / **GetInstance**: Query instances, with filtering by site and area. `ListInstances` supports additive offset paging via `Skip` / `Take` (arch-review P2), applied **after** the in-memory site-scope filter so a site-scoped user never sees an out-of-scope instance surface into their page window; same `Take = null` unlimited default and `1..1000` clamp as `ListTemplates`. (`ExportBundle` still loads the full table internally — a logged scale follow-up, arch-review P2 deferred. `QueryDeployments` was closed by WP2.5, below.)
|
||||
- **CreateInstance**: Create a new instance from a template.
|
||||
- **UpdateInstanceOverrides**: Set attribute overrides on an instance.
|
||||
- **UpdateInstanceOverrides** (`SetInstanceOverrides`): Set attribute overrides on an instance. Every requested attribute is validated against the instance's template up front (unknown or locked ⇒ the whole batch is rejected before any write), then the batch is applied as **one bulk read of the existing override rows plus one commit** (WP2.5). It previously ran K independent read-modify-commit cycles — each re-reading the instance, the template's attributes and the existing overrides, then committing and writing its own audit row — so the all-or-nothing promise was only as good as the pre-validation. It is now a real single-transaction apply, with one audit row summarising the batch.
|
||||
- **MgmtDeploySite** (`SiteId`, WP2.5): Bulk-deploy every deployable instance at one site. Site scope is enforced on the SITE (the command's target); every instance the batch touches belongs to that site by construction. Same `Deployer` authority as `MgmtDeployInstance` — it is N of those, not a new class of privilege — and registered as a long-running command so the caller does not time out mid-batch. Semantics, phasing and failure policy: see `Component-DeploymentManager.md` → Bulk site deployment.
|
||||
- **SetInstanceAlarmOverride** / **DeleteInstanceAlarmOverride** / **ListInstanceAlarmOverrides**: Manage per-instance computed-alarm overrides.
|
||||
- **SetInstanceNativeAlarmSourceOverride** / **DeleteInstanceNativeAlarmSourceOverride**: Retarget or clear a per-instance native alarm source binding, keyed by `SourceCanonicalName` — `ConnectionNameOverride` / `SourceReferenceOverride` / `ConditionFilterOverride` each apply only when non-null. Gated to the **Deployment** role.
|
||||
- **ListInstanceNativeAlarmSourceOverrides** (`InstanceId`): List an instance's native alarm source overrides (read-only).
|
||||
@@ -144,7 +145,9 @@ The whole folder surface is reachable from the CLI as `template folder list|crea
|
||||
|
||||
- **DeployInstance**: Deploy configuration to a specific instance (includes pre-deployment validation).
|
||||
- **DeployArtifacts**: Deploy system-wide artifacts (shared scripts, external system definitions, DB connections, data connections) to all sites or a specific site. The command's `SiteId` is honored (arch-review C2): a value routes to the single-site deploy path (`ArtifactDeploymentService.DeployToSiteAsync`), while `null` deploys fleet-wide (`DeployToAllSitesAsync`). Site scope is enforced — a site-scoped (non-Administrator) Deployer may only target a site within its `PermittedSiteIds`, and may **not** deploy fleet-wide (`SiteId is null` is rejected with `SiteScopeViolationException` → `ManagementUnauthorized`, since `EnforceSiteScope(null)` is a deliberate no-op); fleet-wide deployment requires a system-wide Deployer or Administrator.
|
||||
- **DeploySite** (`MgmtDeploySite`, WP2.5): Bulk-deploy every deployable instance at one site — see the Instances section above and `Component-DeploymentManager.md`.
|
||||
- **GetDeploymentStatus**: Query deployment status.
|
||||
- **QueryDeployments** (`InstanceId?`, `Status?`, `Page`, `PageSize`): List deployment records. **All four arguments are honoured DB-side (WP2.5)** and the result is a `DeploymentRecordSummary` projection. The handler previously ignored `Status`, `Page` and `PageSize` entirely (the CLI had been sending them all along) and loaded the whole insert-only `DeploymentRecords` table; for a site-scoped user it additionally loaded every instance and intersected the two sets in memory. Site scope is now resolved to the set of in-scope instance ids and pushed into the query as an id filter — an **empty** scope stays a real filter (a user permitted no in-scope instances sees nothing), never a no-op. This read path also drives the opportunistic, rate-limited terminal-record retention sweep described in `Component-DeploymentManager.md`.
|
||||
|
||||
### Secured Writes (MxGateway, two-person)
|
||||
|
||||
@@ -259,7 +262,7 @@ The ManagementActor receives the following services and repositories via DI (inj
|
||||
| Section | Options Class | Contents |
|
||||
|---------|--------------|----------|
|
||||
| `ScadaBridge:ManagementService` | `ManagementServiceOptions` | `CommandTimeout` (`TimeSpan`, default 30 s) — Ask timeout the HTTP endpoint applies when forwarding to the `ManagementActor`. A non-positive configured value falls back to the 30 s default. |
|
||||
| | | `LongRunningCommandTimeout` (`TimeSpan`, default 5 min) — Ask timeout applied to long-running commands (`ImportBundle`, `PreviewBundle`, `ExportBundle`, `MgmtDeployArtifacts`, `MgmtDeployInstance`); all other commands use `CommandTimeout`. A non-positive configured value falls back to the 5 min default. |
|
||||
| | | `LongRunningCommandTimeout` (`TimeSpan`, default 5 min) — Ask timeout applied to long-running commands (`ImportBundle`, `PreviewBundle`, `ExportBundle`, `MgmtDeployArtifacts`, `MgmtDeployInstance`, `MgmtDeploySite`); all other commands use `CommandTimeout`. A non-positive configured value falls back to the 5 min default. |
|
||||
| | | `SecuredWritePendingTtl` (`TimeSpan`, default 24 h) — age after which a `Pending` secured write is transitioned to `Expired` and can no longer be approved/executed; enforced at approve/reject and swept opportunistically on list. A non-positive value disables expiry (arch-review S2). |
|
||||
|
||||
## Dependencies
|
||||
|
||||
@@ -167,6 +167,8 @@ A notification is **stuck** if it is `Pending` or `Retrying` and older than a co
|
||||
- **Health Monitoring dashboard** — headline KPI tiles: queue depth, stuck count, parked count. These are central-computed and are not part of the site health report. The site S&F notification backlog remains a separate site health metric covering the site→central leg.
|
||||
- **Central UI "Notification Outbox" page** — KPI tiles plus a queryable notification list: filter by status, type, source site, list, and time range; a stuck-only toggle; keyword search on subject. Parked notifications offer **Retry** (→ `Pending`, reset `RetryCount` / `NextAttemptAt`) and **Discard** (→ `Discarded`) actions. Stuck rows are badged.
|
||||
|
||||
Both operator actions are **read-then-write against a row the daily retention purge may delete in between**, so the write reports whether it matched a row and the actor answers `Success: false` / `"notification not found"` when it did not — never a success against a notification that no longer exists (and no audit row is emitted for the action that did not happen). The dispatcher's own delivery-state write takes the opposite stance on the same signal: the delivery has already happened, there is nothing left to retry, so a vanished row is logged as a warning and the audit rows stand as the durable record.
|
||||
|
||||
## Configuration
|
||||
|
||||
The component is configured via `NotificationOutboxOptions`, bound from an `appsettings.json` section on the central host (Options pattern):
|
||||
|
||||
@@ -158,6 +158,19 @@ Mirrors `TriggerExpressionGlobals` in the same way. Used by `ValidationService.C
|
||||
|
||||
`CheckExpressionSyntax` memoises its verdict in the Template Engine's process-wide `ScriptCompileVerdictCache`, whose key is the pair **(globals surface, SHA-256 of the code)** — not the code alone. The surface discriminator is load-bearing: a trigger expression vetted against `TriggerCompileSurface` is **not** interchangeable with a `ScriptCompileSurface` script-body verdict (different globals resolve different identifiers), so a code-only key could return a stale "clean" for code never compiled against the caller's surface. Keying by surface makes that cross-surface verdict reuse structurally impossible.
|
||||
|
||||
**Eviction is segmented, never wholesale (WP2.5).** The cache is bounded at two
|
||||
generations of 2048 entries. New verdicts land in the *hot* generation; a hit in
|
||||
*cold* is promoted back into hot; on overflow hot becomes the new cold and only the
|
||||
old cold — the half nothing has touched for a full generation — is dropped.
|
||||
|
||||
The bound must never be enforced by clearing the cache outright. That reads as
|
||||
harmless ("a verdict is cheap to recompute") but is not: a recompute is a fresh
|
||||
Roslyn compile, and every script compile loads an assembly through a
|
||||
**non-collectible `InteractiveAssemblyLoader`**. Bounding that leak is the entire
|
||||
reason this cache exists, so dropping every hot entry at the 4096th distinct script
|
||||
would re-open it for the whole working set at once. Segmenting retains anything in
|
||||
active use across an eviction while keeping the same ceiling.
|
||||
|
||||
#### Parity guard
|
||||
|
||||
A reflection-based parity test in `SiteRuntime.Tests` compares the public member names on `ScriptCompileSurface` against `ScriptGlobals` (and `TriggerCompileSurface` against `TriggerExpressionGlobals`). Any drift between the stub and the real globals causes this test to fail, ensuring the stubs cannot silently fall out of sync.
|
||||
|
||||
@@ -135,6 +135,40 @@ Pinned)` on the EventStream (transition-only, mirroring
|
||||
health-observable condition rather than a silent log line, and the latch clears
|
||||
with `Pinned=false` once a later tick makes progress.
|
||||
|
||||
**The drain runs off the mailbox.** A reconciliation pass is unbounded work —
|
||||
every site, up to a page ceiling of network pulls each, one upsert per row — so
|
||||
it runs as a background task with a `PipeTo`-delivered completion message and a
|
||||
single-flight guard, not as an actor message handler. A handler occupies the
|
||||
actor for its whole duration, so a post-outage catch-up used to park telemetry
|
||||
ingest, UI queries and KPI Asks behind it; those callers time out rather than
|
||||
queue, which made a slow site look like a dead central. The single-flight guard
|
||||
is raised and lowered ON the actor thread, so the per-site cursor and pinned-latch
|
||||
dictionaries the pass mutates are still only ever touched by one task at a time,
|
||||
and the mailbox supplies the memory barrier between consecutive passes. The
|
||||
daily terminal-row purge uses the same shape. (`NotificationOutboxActor`'s
|
||||
dispatch sweep is the in-repo reference.)
|
||||
|
||||
**The central upsert is one round trip, INSERT-first.**
|
||||
`SiteCallAuditRepository.UpsertAsync` ships both statements in a single command
|
||||
text — `IF NOT EXISTS … INSERT;` then the monotonic `UPDATE` — so a packet costs
|
||||
one round trip, not two. The insert leg is gated on the row's existence *alone*,
|
||||
never on "the update matched nothing": a zero row count also means "the monotonic
|
||||
guard rejected this packet", and inserting there would fork the mirror with a
|
||||
second row for an id that already exists.
|
||||
|
||||
The ORDER is load-bearing, and running the UPDATE first is a data-loss bug. The
|
||||
two writers — the cached dual-write and the reconciliation pull — routinely carry
|
||||
*different* lifecycle states for the same `TrackedOperationId`, and both can race
|
||||
its very first packet. Under UPDATE-first both find no row, so both updates match
|
||||
nothing; the loser then either fails its existence re-check and skips its insert,
|
||||
or attempts it and takes a duplicate-key fault — either way its
|
||||
`Status`/`RetryCount`/`HttpStatus`/`TerminalAtUtc` are dropped, because it never
|
||||
ran an update against the winner's row. Under INSERT-first the loser's insert is
|
||||
skipped or faults and its monotonic update still applies to whichever row won, so
|
||||
the newer state survives every interleaving while a stale one is still rejected by
|
||||
the rank guard. The duplicate-key catch re-runs the monotonic update for the same
|
||||
reason (it is idempotent and rank-guarded, so a redundant re-run is inert).
|
||||
|
||||
## Retry / Discard Relay
|
||||
|
||||
Parked cached calls live in the owning site's S&F buffer. Operator Retry/Discard
|
||||
|
||||
@@ -19,7 +19,8 @@ Site clusters (event recording and storage). Central cluster (remote query acces
|
||||
|
||||
| Category | Events |
|
||||
|----------|--------|
|
||||
| Script Executions | Script started, completed, failed (with error details), recursion limit exceeded |
|
||||
| Script Executions | Script started, completed *(sampled/opt-in — see Volume Policy below)*; failed, timed out, recursion limit exceeded (with error details; always logged) |
|
||||
| Script Run Summary | Interval aggregate replacing the per-run rows by default — one Info row per flush interval summarizing every instance script run since the last flush (see Volume Policy) |
|
||||
| Alarm Events | Alarm activated, alarm cleared (which alarm, which instance), alarm evaluation error |
|
||||
| Deployment Events | Configuration received from central, scripts compiled, applied successfully, apply failed |
|
||||
| Data Connection Status | Connected, disconnected, reconnected (per connection) |
|
||||
@@ -27,6 +28,41 @@ Site clusters (event recording and storage). Central cluster (remote query acces
|
||||
| Instance Lifecycle | Instance enabled, disabled, deleted |
|
||||
| Notification | Site→central forward failure, long-buffered notification (still in the site buffer past a threshold) |
|
||||
|
||||
## Volume Policy — Script Run Events (WP3.2, 2026-08-15)
|
||||
|
||||
Per-run instance-script Started/Completed Info events were, before this policy, the dominant
|
||||
`site_events` writer: two rows per run, translating to roughly a dozen physical row-writes
|
||||
across a replicated site pair once CDC oplog/row-version rows and their eventual purge
|
||||
tombstones are counted — for content nobody looks at unless something went wrong. The policy
|
||||
below (design memo: `docs/plans/2026-08-15-site-events-policy-design.md`) keeps the signal
|
||||
while removing the volume.
|
||||
|
||||
- **Off by default.** `ScadaBridge:SiteRuntime:PerRunScriptEvents` (default `false`) is the
|
||||
global switch back to the legacy per-run Started/Completed rows for every script.
|
||||
- **Interval aggregate instead.** `ScriptRunSummaryRecorder` accumulates per-`(instance,
|
||||
script)` counters (started, completed, failed, timed out, total/max duration) from the same
|
||||
call sites that used to emit the per-run rows. A hosted flush service emits **one** "script"
|
||||
Info row per interval — `ScadaBridge:SiteRuntime:ScriptRunSummaryIntervalSeconds` (default
|
||||
`300`, i.e. 5 minutes; `0` disables the flush) — with headline totals in the message (e.g.
|
||||
"1,243 runs: 1,240 completed, 2 failed, 1 timed out across 17 scripts") and a per-script
|
||||
breakdown in `Details`, capped at the **top 50 scripts by run activity** with an "others"
|
||||
rollup entry so a pathological deployment cannot mint an oversized row. **An interval with
|
||||
zero activity emits no row** — the standby, which never runs scripts, therefore never
|
||||
produces a summary row naturally. Source is `"ScriptRunSummary"`; Instance ID is `NULL`
|
||||
(the row spans every instance).
|
||||
- **Per-script opt-in for debugging.** `ScadaBridge:SiteRuntime:PerRunScriptEventScripts`
|
||||
(default empty) restores the legacy per-run rows for named scripts only, without paying the
|
||||
volume cost for the rest of the deployment. Entries are `"{InstanceName}/{ScriptName}"` for
|
||||
an exact match or `"{InstanceName}/*"` for every script on that instance. Effective only
|
||||
while the global switch is off.
|
||||
- **Hot-togglable.** Both keys are read from the live options snapshot at the top of every
|
||||
run (`IOptionsMonitor<SiteRuntimeOptions>.CurrentValue`), so an operator can flip either one
|
||||
— including narrowing an opt-in to a single misbehaving script — without a restart or a
|
||||
redeploy.
|
||||
- **Error-level events are unconditional.** Timeout, failure, stuck-watchdog, and
|
||||
recursion-limit Error rows are unaffected by any of the above — they always fire, regardless
|
||||
of `PerRunScriptEvents` or the opt-in list.
|
||||
|
||||
## Event Entry Schema
|
||||
|
||||
Each event entry contains:
|
||||
@@ -40,9 +76,23 @@ Each event entry contains:
|
||||
|
||||
## Storage
|
||||
|
||||
- Events are stored in **local SQLite** on each site node.
|
||||
- Each node maintains its own event log. Only the **active node** generates and stores events. Event logs are **not replicated** to the standby node. On failover, the new active node starts logging to its own SQLite database; historical events from the previous active node are no longer queryable via central until that node comes back online. This is acceptable because event logs are diagnostic, not transactional.
|
||||
- **Retention**: 30 days. A **daily background job** runs on the active node and deletes all events older than 30 days. Hard delete — no archival.
|
||||
- Events are stored in the **consolidated site LocalDb** SQLite file (`site_events` table;
|
||||
see CLAUDE.md → Consolidated site database). Only the **active node** generates and stores
|
||||
events (a standby runs no scripts, subsystems, or deployments to log).
|
||||
- **`site_events` IS replicated** (`SiteLocalDbSetup.ReplicatedTables`) on a site pair with
|
||||
replication configured — this reverses the component's pre-LocalDb-Phase-1 behavior, and the
|
||||
reason is **failover history continuity, not either-node queries**: the query handler
|
||||
(`EventLogHandlerActor`) is a **cluster singleton**, so a central query always routes to the
|
||||
actively-written copy regardless of which node answers the gRPC call — replication is never
|
||||
needed to make queries *work*. What replication buys is **content to read after a failover**:
|
||||
when the active node dies, the singleton restarts on the survivor, and its query service reads
|
||||
the survivor's *local* copy — which has history *only because it was replicated*. Without
|
||||
replication, a failover would start the event log empty at precisely the moment an operator
|
||||
is investigating an incident (WP3.2 design memo, Decision (b): `docs/plans/2026-08-15-site-events-policy-design.md`).
|
||||
On a site pair running **without** replication configured (by deliberate choice, e.g. the
|
||||
rig's site-b/site-c), the log stays node-local and a failover does start it fresh — the
|
||||
documented trade of not configuring a peer, unchanged by this policy.
|
||||
- **Retention**: 30 days. A **daily background job** runs on the active node and deletes all events older than 30 days. Hard delete — no archival. (Today, retention/cap deletes on a replicated node are captured by CDC like any other write — see the Volume Policy section above for why this is a small residual cost now that per-run rows are off by default.)
|
||||
- **Storage cap**: A configurable maximum database size (default: 1 GB) is enforced. If the storage cap is reached before the 30-day retention window, the oldest events are purged first. This prevents disk exhaustion from alarm storms, script failure loops, or connection flapping.
|
||||
|
||||
## Central Access
|
||||
|
||||
@@ -41,9 +41,9 @@ flowchart TD
|
||||
AA2["Alarm Actor ('LowPressure')<br/>— coordinator (computed)"]
|
||||
NAA1["Native Alarm Actor ('OpcUaServer1')<br/>— read-only mirror, peer to Alarm Actor"]
|
||||
|
||||
SEA1["Script Execution Actor<br/>— short-lived, per invocation"]
|
||||
SEA2["Script Execution Actor<br/>— short-lived, per invocation"]
|
||||
AEA1["Alarm Execution Actor<br/>— short-lived, per on-trigger invocation"]
|
||||
SEA1["script run<br/>— launched task, not an actor"]
|
||||
SEA2["script run<br/>— launched task, not an actor"]
|
||||
AEA1["alarm on-trigger run<br/>— launched task, not an actor"]
|
||||
|
||||
IA2CHILD["… (Script / Alarm Actors)"]
|
||||
|
||||
@@ -57,9 +57,9 @@ flowchart TD
|
||||
IA1 --> AA2
|
||||
IA1 --> NAA1
|
||||
|
||||
SA1 --> SEA1
|
||||
SA2 --> SEA2
|
||||
AA1 --> AEA1
|
||||
SA1 -.-> SEA1
|
||||
SA2 -.-> SEA2
|
||||
AA1 -.-> AEA1
|
||||
|
||||
IA2 -.-> IA2CHILD
|
||||
|
||||
@@ -96,9 +96,11 @@ flowchart TD
|
||||
|
||||
> **Startup config load is retried (S5)**: step 1's deployed-config read is best-effort against local SQLite, which can transiently fail (locked/busy). A failed load must **not** leave the site as a silent zero-instance node — it is re-attempted on a fixed interval until it succeeds, then the retry self-cancels.
|
||||
|
||||
> **Per-instance compilation during staggered startup (P6, follow-up)**: the Roslyn compile of each instance's scripts still runs inside Instance Actor start during staggered startup; moving it off-thread is a deferred optimization (it affects failover time-to-recover only, not correctness). As of the round-2 hardening, a process-wide compile cache dedupes identical script bodies within a node's process lifetime (the deploy gate's compile is reused by Instance Actor start), shrinking the recompile cost; the *first* compile after process start still runs inside Instance Actor start, so the deferral stands.
|
||||
> **Per-instance compilation during staggered startup — CLOSED (WP3.1)**: this was a deferred optimization (P6) affecting failover time-to-recover; each Instance Actor Roslyn-compiled its own scripts inside its own start, serialising a site's whole recovery behind compilation. Each staggered startup batch now runs a **pre-warm step**: before the batch's Instance Actors are created, the Deployment Manager compiles that batch's distinct script bodies and trigger expressions on a background task and pipes `BatchCompileWarmed` back, so every `PreStart` compile is a cache hit. Instance Actor start still calls the compilation service — it remains the correctness backstop — but in every warmed path those calls are memoised lookups. Cost: one extra message per batch.
|
||||
|
||||
> **The Roslyn `ScriptOptions` are process-static, and must stay that way**: the reference set backing every site-side compile is built once per process, never per compile. `ScriptOptions.WithReferences(Assembly[])` resolves each assembly through `MetadataReference.CreateFromFile`, which does **not** cache — each call mints a fresh `MetadataReference` owning an `AssemblyMetadata` → `PEReader` → `NativeHeapMemoryBlock`, an unmanaged copy of the assembly metadata that nothing disposes. Building the options per compile leaks native memory permanently: no GC reclaims it, and it is invisible to gcdump and to the managed allocation counters, so the node's working set grows without the GC heap growing. This was a live defect (a Site node at 2,885 MB working set with 150 MB of live GC heap, ~2,469 MB of it on the default process heap across ~6,700 undisposed `AssemblyMetadata` instances). Note this is orthogonal to the compile cache above — the cache dedupes *identical* script bodies, so it bounds nothing when the bodies differ, and it clears wholesale on overflow, after which every script recompiles. Pinned by a reference-equality regression test rather than by watching memory.
|
||||
> **The Roslyn `ScriptOptions` are process-static, and must stay that way**: the reference set backing every site-side compile is built once per process, never per compile. `ScriptOptions.WithReferences(Assembly[])` resolves each assembly through `MetadataReference.CreateFromFile`, which does **not** cache — each call mints a fresh `MetadataReference` owning an `AssemblyMetadata` → `PEReader` → `NativeHeapMemoryBlock`, an unmanaged copy of the assembly metadata that nothing disposes. Building the options per compile leaks native memory permanently: no GC reclaims it, and it is invisible to gcdump and to the managed allocation counters, so the node's working set grows without the GC heap growing. This was a live defect (a Site node at 2,885 MB working set with 150 MB of live GC heap, ~2,469 MB of it on the default process heap across ~6,700 undisposed `AssemblyMetadata` instances). Note this is orthogonal to the compile cache above — the cache dedupes *identical* script bodies, so it bounds nothing when the bodies differ. (It no longer clears wholesale on overflow: WP3.1 replaced that latency cliff with approximate-LRU batch eviction, see **Compile cache eviction** below.) Pinned by a reference-equality regression test rather than by watching memory.
|
||||
|
||||
> **Compile cache eviction is approximate LRU, not a wholesale clear (WP3.1)**: `SiteScriptCompileCache` is bounded at 1024 entries — deliberately smaller than the TemplateEngine verdict cache's 4096, because entries pin compiled assemblies rather than verdict strings. It used to `Clear()` **everything** on overflow. Instance scripts *and* trigger expressions share this cache, so on a site large enough to cross the bound every overflow discarded up to 1023 live compiled scripts and the next deploy or Instance-Actor start paid a full recompile storm **on actor threads**. Overflow now evicts only the oldest ⅛ (128 entries) by last-access stamp: one 1024-element scan instead of 1023 future recompiles, and hot entries survive. Recency is an `Interlocked` access sequence rather than a clock — deterministic for tests and immune to clock steps; hits update it lock-free, and only the (rare) eviction sweep takes a lock, double-checked so concurrent inserts do not stampede it. The sweep snapshots through `ConcurrentDictionary.ToArray()` and **not** LINQ over the dictionary: LINQ's `ToArray` picks the `ICollection.CopyTo` fast path, which throws when a concurrent insert lands between its Count read and its copy, and inserts are deliberately lock-free.
|
||||
|
||||
> **Static options were only half the fix — the compile must also use the shared caching metadata resolver**: the static `ScriptOptions` carry only the *direct* API-surface references (5 assemblies here), and each `script.Compile()` still binds their **transitive closure** (~105 assemblies on a site node). Every transitively-referenced assembly is resolved through the options' `MetadataReferenceResolver`, and Roslyn's default one has no cross-compilation cache — it calls `MetadataReference.CreateFromFile` afresh on **every compile**, minting the same undisposed `AssemblyMetadata` → `PEReader` → `NativeHeapMemoryBlock` triple per assembly per compiled script, pinned for the process lifetime by the compile cache. That per-compile *re-resolution* — not the per-compile options construction — was the dominant term (measured live: 21 site scripts holding 6,640 metadata objects, counts bit-identical before and after the static-options fix). `ScriptCompilationService` therefore also attaches the process-wide `CachingScriptMetadataResolver.Instance` (owned by the Script Analysis component, see `Component-ScriptAnalysis.md`) via `ScriptOptions.WithMetadataResolver`, so each distinct assembly is materialized once per process. The decorator returns exactly what the undecorated resolver returned — only object identity is de-duplicated — so compile results and the trust gate are unchanged.
|
||||
|
||||
@@ -171,7 +173,7 @@ flowchart TD
|
||||
### Debug View Support
|
||||
- On request from central (via Communication Layer), the Instance Actor provides a **snapshot** of all current attribute values and alarm states.
|
||||
- Subsequent changes are delivered via the **SiteStreamManager** → **SiteStreamGrpcServer** → gRPC stream to central. The Instance Actor publishes attribute value and alarm state changes to the SiteStreamManager; it does not forward events directly to the Communication Layer.
|
||||
- The Instance Actor also handles one-shot `DebugSnapshotRequest` messages: it builds the same snapshot (attribute values and alarm states) and replies directly to the sender. Unlike `SubscribeDebugViewRequest`, no subscriber is registered and no stream is established.
|
||||
- The Instance Actor also handles one-shot `DebugSnapshotRequest` messages: it builds the same snapshot (attribute values and alarm states) and replies directly to the sender. Unlike `SubscribeDebugViewRequest`, no subscriber is registered and no stream is established. When the request sets `AlarmsOnly` (WP2.3 — the central live alarm cache's seed/reconcile fan-out, which discards attribute rows), only the alarm half is built and `AttributeValues` comes back empty; the flag defaults to `false`, so the Debug View path is unchanged.
|
||||
|
||||
### Supervision Strategy
|
||||
|
||||
@@ -179,10 +181,10 @@ The Instance Actor supervises all child Script and Alarm Actors with explicit st
|
||||
|
||||
| Child Actor | Exception Type | Strategy | Rationale |
|
||||
|-------------|---------------|----------|-----------|
|
||||
| Script Actor | Any exception | Resume | Script Actor is a coordinator — its state (trigger timers, last execution time) should survive child failures. Script Execution Actor failures are isolated. |
|
||||
| Script Actor | Any exception | Resume | Script Actor is a coordinator — its state (trigger timers, last execution time) should survive run failures. |
|
||||
| Alarm Actor | Any exception | Resume | Alarm Actor holds alarm state. Resume preserves state and continues evaluation on next value update. |
|
||||
| Script Execution Actor | Unhandled exception | Stop | Short-lived, per-invocation. Failure is logged; the Script Actor coordinator remains active for future triggers. |
|
||||
| Alarm Execution Actor | Unhandled exception | Stop | Short-lived, per on-trigger invocation. Same as Script Execution Actor. |
|
||||
|
||||
Script Actor and Alarm Actor have **no children of their own** since WP3.1 removed the per-run execution actors, so neither declares a supervision strategy any more. A run's own exceptions were always contained inside the run's try/catch (they never reached supervision); a failure of the *launch* is caught by the coordinator, which answers the caller and releases the run slot rather than letting the throw escalate to the Instance Actor.
|
||||
|
||||
The Deployment Manager singleton supervises Instance Actors with a **OneForOneStrategy** — one Instance Actor's failure does not affect other instances.
|
||||
|
||||
@@ -196,11 +198,11 @@ When the Instance Actor is stopped (due to disable, delete, or redeployment), Ak
|
||||
- **Coordinator** for a single script definition on an instance.
|
||||
- Holds the compiled script code and trigger configuration.
|
||||
- Manages trigger evaluation (interval timer, value change detection, conditional evaluation).
|
||||
- Spawns short-lived Script Execution Actors for each invocation.
|
||||
- Launches script runs directly for each invocation (no per-run child actor since WP3.1).
|
||||
|
||||
### Trigger Management
|
||||
- **Interval**: The Script Actor manages an internal timer. When the timer fires, it spawns a Script Execution Actor.
|
||||
- **Value Change**: The Script Actor subscribes to attribute change notifications from its parent Instance Actor for the specific monitored attribute. When the attribute changes, it spawns a Script Execution Actor.
|
||||
- **Interval**: The Script Actor manages an internal timer. When the timer fires, it launches a run.
|
||||
- **Value Change**: The Script Actor subscribes to attribute change notifications from its parent Instance Actor for the specific monitored attribute. When the attribute changes, it launches a run.
|
||||
- **Conditional**: The Script Actor subscribes to attribute change notifications for the monitored attribute. On each update, it evaluates the condition (compares the attribute against a threshold). Firing depends on the **fire mode** (see below).
|
||||
- **Expression**: The Script Actor evaluates a compiled boolean expression against an attribute snapshot on each attribute change. Firing depends on the **fire mode** (see below).
|
||||
- **Fire mode (Conditional + Expression)**:
|
||||
@@ -211,20 +213,24 @@ When the Instance Actor is stopped (due to disable, delete, or redeployment), Ak
|
||||
> **Per-attribute change routing (P2).** Rather than broadcasting every attribute change to all child Script/Alarm Actors, the Instance Actor routes each change only to the children that monitor that attribute (built from their trigger configs at spawn time); **Expression-trigger children, which read an attribute snapshot, receive all changes**. Each child still keeps its own trigger gate as defense-in-depth — the routing is an optimization, not the authority.
|
||||
|
||||
### Concurrent Execution
|
||||
- Each invocation spawns a **new Script Execution Actor** as a child.
|
||||
- Multiple Script Execution Actors can run concurrently (e.g., a trigger fires while a previous `Instance.CallScript` invocation is still running).
|
||||
- The Script Actor coordinates but does not block on child completion.
|
||||
- Each invocation launches a **new run** on the script-execution scheduler.
|
||||
- Multiple runs can execute concurrently (e.g., a trigger fires while a previous `Instance.CallScript` invocation is still running), up to `MaxConcurrentRunsPerScript` — beyond which the newest is shed (see below).
|
||||
- The Script Actor coordinates but does not block on run completion.
|
||||
|
||||
### Script Execution Actor
|
||||
- **Short-lived** child actor created per invocation.
|
||||
- Receives: compiled script code, input parameters, reference to the parent Instance Actor, current call depth.
|
||||
- Executes the script in the Akka actor context.
|
||||
- Has access to the full Script Runtime API (see below).
|
||||
- Returns the script's return value (if defined) to the caller, then stops.
|
||||
- The script body itself runs on the **dedicated `ScriptExecutionScheduler`** (a bounded set of dedicated threads), not the shared .NET thread pool, so blocking script I/O cannot starve the global pool or stall Akka dispatchers. The scheduler is **process-wide by default** (one pool per host, sized from `SiteRuntimeOptions.ScriptExecutionThreadCount`), but each script/alarm actor takes it through an **optional injection seam** rather than reaching for the static directly: the Host injects nothing and gets the shared pool, while tests (or a future multi-site host) can hand an actor its own instance. The shared accessor also recreates a disposed pool rather than returning it, so a disposed scheduler can never silently poison later executions.
|
||||
### Script Run Launch (WP3.1 — replaces the Script Execution Actor)
|
||||
- A run is launched **directly by the Script Actor** through `ScriptRunLauncher`. There is no per-run child actor.
|
||||
- The former `ScriptExecutionActor` / `AlarmExecutionActor` were already inert shells: neither declared a `Receive` handler (they executed from their constructor), neither had a `PostStop`, state, or stash, and neither's `IActorRef` was ever a message target — the whole lifecycle lived inside a detached task the actor never observed. What they cost was an actor cell, mailbox, and name registration **per run**, plus a per-spawn expression-tree `Props.Create`. Removing them changed no semantics; the run body moved verbatim into the shared launcher.
|
||||
- Everything the shells provided is preserved: exception and timeout containment, one DI scope per run disposed on every path, the site-event/health telemetry, the Ask reply, the completion notification (now the coordinator's own `Self`), and the audit `ExecutionId` / `ParentExecutionId` threading. A run still in flight when its Script Actor is stopped runs to completion and its completion message dead-letters, exactly as before — **stopping does NOT cancel in-flight runs**; redeploy/undeploy semantics are unchanged.
|
||||
- **Per-run Started/Completed site events are sampled, not unconditional (WP3.2).** The two Info rows an instance script run used to emit unconditionally were the dominant `site_events` writer under load (design memo: `docs/plans/2026-08-15-site-events-policy-design.md`). They are now **off by default** (`ScadaBridge:SiteRuntime:PerRunScriptEvents`, default `false`); a `ScriptRunSummaryRecorder` accumulates per-`(instance, script)` run counters from the same call sites and a site-only hosted flush service emits **one aggregate "script" Info row per interval** (`ScriptRunSummaryIntervalSeconds`, default 300s) with a top-50-scripts breakdown, capped so a pathological deployment cannot mint an oversized row — an idle interval emits nothing. `ScadaBridge:SiteRuntime:PerRunScriptEventScripts` restores the legacy per-run rows for named scripts (`"Instance/Script"` exact or `"Instance/*"` wildcard) for live debugging. All three keys are read from `IOptionsMonitor<SiteRuntimeOptions>` per run, so they are hot-togglable without a restart. **Error-level rows (timeout, failure, stuck-watchdog, recursion-limit) are unaffected — always emitted, regardless of this policy.** See Component-SiteEventLogging.md → Volume Policy for the full design.
|
||||
- One deliberate improvement: a failure of the *launch itself* (e.g. queueing onto a disposed scheduler) is caught by the Script Actor, which replies to the Ask caller and releases the run slot. The old per-run child's constructor throw was handled by a Stop supervision directive that sent no reply, leaving the caller to hang to its Ask timeout.
|
||||
- The script body runs on the **dedicated `ScriptExecutionScheduler`** (a bounded set of dedicated threads), not the shared .NET thread pool, so blocking script I/O cannot starve the global pool or stall Akka dispatchers. The scheduler is **process-wide by default** (one pool per host), but each script/alarm actor takes it through an **optional injection seam** rather than reaching for the static directly: the Host injects nothing and gets the shared pool, while tests (or a future multi-site host) can hand an actor its own instance. The shared accessor also recreates a disposed pool rather than returning it, so a disposed scheduler can never silently poison later executions.
|
||||
- **Pool sizing is instance-scaled and grow-only (WP3.1).** The pool was a fixed 8 threads regardless of load. It is now `clamp(max(ScriptExecutionThreadCount, ceil(enabledInstances / 8)), 1, ScriptExecutionMaxThreadCount)` — the existing `ScriptExecutionThreadCount` (default 8) becomes the **floor**, so configurations at or below 64 instances behave exactly as before, and the new `ScriptExecutionMaxThreadCount` (default 32) is the ceiling. Beyond the ceiling the per-script cap below is the real regulator. `DeploymentManagerActor.UpdateInstanceCounts` calls `EnsureCapacity` on every deploy / undeploy / enable / disable and once per staggered startup batch. Growth only: undeploying leaves idle threads, which cost nothing measurable and avoid drain/steal complexity.
|
||||
- **The deadline is armed at enqueue, not at dequeue (WP3.1).** The run's timeout `CancellationTokenSource` is created on the actor thread *before* the body is queued, so queue wait consumes the script's own budget. A body that dequeues past its deadline **skips execution entirely** and takes the existing timeout path (site event, script-error counter, error reply, completion message): a saturated pool sheds stale work instead of running it late with a fresh full budget.
|
||||
- **Concurrent runs per script are capped (WP3.1).** `MaxConcurrentRunsPerScript` (default 4) bounds runs in flight — queued or executing — for any one script or alarm on-trigger script. Over the cap the **newest** run is shed: the four already in flight are closest to their own deadlines and already charged against them, so nothing is ever reordered and no extra queue is needed (the scheduler's FIFO already is the queue). A shed increments `ISiteHealthCollector.IncrementScriptRunShed` (surfaced as `ScriptRunShedCount`), emits a `script`/`Warning` site event **rate-limited to one per script per minute** so a hot trigger cannot flood `site_events`, and — for an Ask-based `CallScript` — replies with an explicit error so a nested call or inbound-API route fails fast instead of hanging.
|
||||
- **The cap gates only depth-0 launches (review fix).** A trigger fire or a depth-0 `CallScript`/inbound-API route counts against `MaxConcurrentRunsPerScript`; a *nested* `Instance.CallScript` self-recursion (`callDepth > 0`) is exempt — it is already bounded by `MaxScriptCallDepth` instead, and the calling run is holding one of the four cap slots itself while it awaits the callee, so counting the nested launch too would spuriously shed legitimate self-recursion.
|
||||
|
||||
### Handling `Instance.CallScript`
|
||||
- When an external caller (another Script Execution Actor, an Alarm Execution Actor, or a routed call from the Inbound API) sends a `CallScript` message to the Script Actor, it spawns a Script Execution Actor to handle the call.
|
||||
- When an external caller (another script run, an alarm on-trigger run, or a routed call from the Inbound API) sends a `CallScript` message to the Script Actor, it launches a run to handle the call.
|
||||
- The caller uses the **Akka ask pattern** and receives the return value when the execution completes.
|
||||
|
||||
---
|
||||
@@ -244,10 +250,10 @@ When the Instance Actor is stopped (due to disable, delete, or redeployment), Ak
|
||||
- **Range Violation**: Value is outside the allowed min/max range.
|
||||
- **Rate of Change**: Value change rate exceeds the defined threshold over a configurable time window. Direction filter (rising / falling / either) restricts which side of the rate triggers.
|
||||
- **HiLo**: Multi-setpoint level alarm with up to four configurable setpoints (LoLo, Lo, Hi, HiHi). Any subset may be configured. Each setpoint may carry its own priority that overrides the alarm-level priority for that band.
|
||||
- **Expression** trigger evaluation runs on the bounded script-execution thread pool shared with script bodies (and Expression *script* triggers): scheduler saturation — e.g. stuck scripts occupying all threads — delays Expression alarm transitions site-wide until a thread frees. This is a deliberate trade-off (evaluation no longer blocks dispatcher threads; per-actor coalescing bounds queue growth) and the scheduler gauges + stuck-script watchdog make the saturated state visible.
|
||||
- **Expression** trigger evaluation runs on the shared .NET thread pool behind a process-wide concurrency gate (`TriggerEvalGate`, sized by `TriggerEvalMaxConcurrency`, default `max(2, ProcessorCount)`) — **not** on the bounded script-execution pool. This is the WP3.1 fix for arch-review finding #4 (High): when evaluation shared that pool, N script bodies blocked in synchronous I/O stalled *every* Expression trigger on the node — scripts and alarms alike — for an unbounded time, and the evaluation's own 2 s timeout was constructed inside the queued body, so it did not start ticking until dequeue. An alarm that should have raised in milliseconds simply never raised, with neither a raise nor a timeout visible to the operator. Trigger expressions are non-blocking **by construction** (`TriggerExpressionGlobals` exposes only reads over an in-memory snapshot, and the script trust gate has already denied I/O, network, threading, and reflection), so the shared pool is where they belong; a second dedicated pool was considered and rejected as adding threads, gauges, and a second starvation surface for no isolation gain. The evaluation deadline (`TriggerEvalTimeoutSeconds`, default 2, previously hardcoded) is now armed **at enqueue**, so gate-wait time burns the same budget and a saturated gate yields a timely `false` rather than an unbounded stall. Per-actor coalescing (one evaluation in flight, one pending) is unchanged and caps waiters at one per Expression trigger, so the gate queue is bounded by trigger count.
|
||||
- For binary trigger types (ValueMatch / RangeViolation / RateOfChange), when the condition is met and the alarm is currently in **normal** state, the alarm transitions to **active**:
|
||||
- Updates the alarm state on the parent Instance Actor (which publishes to the Akka stream).
|
||||
- If an on-trigger script is defined, spawns an Alarm Execution Actor to execute it.
|
||||
- If an on-trigger script is defined, launches its run via `ScriptRunLauncher` (see Alarm On-Trigger Run below) — no per-run child actor.
|
||||
- When the condition clears and the alarm is in **active** state, the alarm transitions to **normal**.
|
||||
- For HiLo triggers, the actor tracks the current `AlarmLevel` (None / Low / LowLow / High / HighHigh). Each level transition emits a fresh `AlarmStateChanged` with the new level and its priority; level escalations (e.g., High → HighHigh) and de-escalations (HighHigh → High) both produce events. The on-trigger script fires only on the Normal → non-None edge, not on escalations between alarm bands.
|
||||
- No script execution on clear in any trigger type.
|
||||
@@ -256,9 +262,8 @@ When the Instance Actor is stopped (due to disable, delete, or redeployment), Ak
|
||||
- Held **in memory** only — not persisted to SQLite. State comprises `AlarmState` (Active / Normal) and `AlarmLevel` (None for binary triggers; the active band for HiLo).
|
||||
- On restart (or failover), alarm states are re-evaluated from incoming values. All alarms start in normal state with level None and transition based on incoming values.
|
||||
|
||||
### Alarm Execution Actor
|
||||
- **Short-lived** child actor created when an on-trigger script needs to execute.
|
||||
- Same pattern as Script Execution Actor — receives compiled code, executes, returns, and stops.
|
||||
### Alarm On-Trigger Run
|
||||
- Launched directly by the Alarm Actor when an on-trigger script needs to execute — the same `ScriptRunLauncher` path as an instance script run, with the firing alarm's name/level/priority/message exposed through the `Alarm` global. Bounded by the same `MaxConcurrentRunsPerScript` cap.
|
||||
- Has access to the Instance Actor for `GetAttribute`/`SetAttribute`.
|
||||
- **Can** call instance scripts via `Instance.CallScript()` — sends an ask message to the appropriate sibling Script Actor.
|
||||
- Instance scripts **cannot** call alarm on-trigger scripts — the call direction is one-way.
|
||||
@@ -275,7 +280,7 @@ change:
|
||||
|---|---|
|
||||
| `ScriptRuntimeContext.SetAttribute` (this run's `ExecutionId`) / `RouteToSetAttributesRequest.ParentExecutionId` (the inbound request's) | → `SetStaticAttributeCommand.SourceExecutionId` |
|
||||
| Instance Actor static-write path (`HandleSetStaticAttributeCore`) | → `AttributeValueChanged.SourceExecutionId` |
|
||||
| Alarm Actor trigger evaluation → `SpawnAlarmExecution` | → `AlarmExecutionActor` → `ScriptRuntimeContext.ParentExecutionId` |
|
||||
| Alarm Actor trigger evaluation → `SpawnAlarmExecution` | → `ScriptRunLauncher.LaunchAlarmScript` → `ScriptRuntimeContext.ParentExecutionId` |
|
||||
|
||||
All four computed trigger types participate. `Expression` triggers evaluate a
|
||||
whole attribute snapshot **off the dispatcher**, so the firing change is no
|
||||
@@ -382,7 +387,7 @@ The enriched message flows Instance Actor → site-wide Akka stream → `SiteStr
|
||||
|
||||
- Shared scripts are compiled at the site when received from central.
|
||||
- Compiled code is stored in memory and made available to all Script Actors.
|
||||
- When a Script Execution Actor calls `Scripts.CallShared("scriptName", params)`, the shared script code executes **inline** in the Script Execution Actor's context — it is a direct method invocation, not an actor message.
|
||||
- When a script run calls `Scripts.CallShared("scriptName", params)`, the shared script code executes **inline** in that run's context — it is a direct method invocation, not an actor message.
|
||||
- This avoids serialization bottlenecks since there is no shared script actor to contend for.
|
||||
- Shared scripts have access to the same runtime API as instance scripts (GetAttribute, SetAttribute, external systems, notifications, databases).
|
||||
- **Shared scripts always execute with Root scope (C5)**: `Attributes[...]` inside a shared script addresses **root** attributes even when the shared script is invoked from a composed-module script. Callers needing module-scoped access must pass the values in as parameters.
|
||||
@@ -417,14 +422,14 @@ All script types can be updated without restarting the cluster, but the mechanis
|
||||
|
||||
## Script Runtime API
|
||||
|
||||
Available to all Script Execution Actors and Alarm Execution Actors:
|
||||
Available to all script runs and alarm on-trigger runs:
|
||||
|
||||
### Instance Attributes
|
||||
- `Instance.GetAttribute("name")` — Read an attribute value from the parent Instance Actor.
|
||||
- `Instance.SetAttribute("name", value)` — Write an attribute value. For data-connected attributes, writes to the DCL; for static attributes, updates in-memory and persists to local SQLite (survives restart/failover, reset on redeployment).
|
||||
|
||||
### Other Scripts
|
||||
- `Instance.CallScript("scriptName", parameters)` — Send an ask message to a sibling Script Actor. The target Script Actor spawns a Script Execution Actor, executes, and returns the result. The call includes the current recursion depth.
|
||||
- `Instance.CallScript("scriptName", parameters)` — Send an ask message to a sibling Script Actor. The target Script Actor launches a run, executes, and returns the result (or an explicit shed error if that script is already at its concurrent-run cap). The call includes the current recursion depth.
|
||||
- `Scripts.CallShared("scriptName", parameters)` — Execute shared script code inline (direct method invocation). The call includes the current recursion depth.
|
||||
|
||||
### External Systems
|
||||
@@ -474,7 +479,7 @@ Scripts execute **in-process** with constrained access. The following restrictio
|
||||
|
||||
- **Allowed**: Access to the Script Runtime API (GetAttribute, SetAttribute, CallScript, CallShared, ExternalSystem, Notify, Database, Tracking, Alarms), standard C# language features, basic .NET types (collections, string manipulation, math, date/time). `System.Diagnostics.Stopwatch`, `Debug`, and `Activity` are permitted.
|
||||
- **Forbidden**: File system access (`System.IO`), process spawning (`System.Diagnostics.Process`), threading (`System.Threading` — except `Tasks`, `CancellationToken`, and `CancellationTokenSource`), reflection (`System.Reflection`), all raw network access (`System.Net` — must use `ExternalSystem.Call`), native interop (`System.Runtime.InteropServices`, `Microsoft.Win32`), assembly loading, unsafe code, `dynamic`, `Activator`.
|
||||
- **Execution timeout**: Configurable per-script maximum execution time. Exceeding the timeout cancels the script **cooperatively** (S2/UA5): a script blocked in synchronous I/O or a tight CPU loop does not observe cancellation and continues to occupy its dedicated script-execution thread. A **watchdog logs the script by name** once its thread has not returned within a grace period after the timeout, and the script-execution scheduler's **queue depth / busy-thread / oldest-busy-age gauges** are reported through Health Monitoring so a saturated or stuck scheduler is visible.
|
||||
- **Execution timeout**: Configurable per-script maximum execution time, **measured from enqueue** (WP3.1) — a run that spends its whole budget queued behind blocked scripts is shed at dequeue without executing, rather than being granted a fresh full budget for work whose triggering condition is already stale. Exceeding the timeout cancels the script **cooperatively** (S2/UA5): a script blocked in synchronous I/O or a tight CPU loop does not observe cancellation and continues to occupy its dedicated script-execution thread. A **watchdog names the script** once its thread has not returned within a grace period after the timeout — and, as of WP3.1, **detaches and replaces that worker thread**, so a wedged script costs the pool a thread only until the watchdog fires rather than for the rest of the process. The detached worker exits (instead of pulling more work) when its body finally returns, so capacity never silently doubles. Live detached workers are capped at the pool size — at the cap no replacement is started and an Error site event says so, because bounded starvation is preferable to unbounded thread growth when scripts wedge en masse. The scheduler's **queue depth / busy-thread / oldest-busy-age / detached-thread gauges** and the **per-interval shed count** are reported through Health Monitoring so a saturated, stuck, or shedding scheduler is visible.
|
||||
- **Memory**: Scripts share the host process memory. No per-script memory limit, but the execution timeout prevents runaway allocations.
|
||||
|
||||
The forbidden-API policy is defined authoritatively in `ScriptTrustPolicy` (Script Analysis component, #25). `ScriptCompilationService.ValidateTrustModel` delegates to `ScriptTrustValidator.FindViolations` for the trust verdict; the Site Runtime also performs a `CSharpScript.Compile` against the real `ScriptGlobals` for execution. This is defence-in-depth static enforcement, not a true runtime sandbox.
|
||||
@@ -503,14 +508,14 @@ Per Akka.NET best practices, internal actor communication uses **Tell** (fire-an
|
||||
|
||||
**Ask** is reserved for system boundaries where a synchronous response is needed:
|
||||
|
||||
- **`Instance.CallScript()`**: Ask pattern from Script Execution Actor to sibling Script Actor. The caller needs the return value. Acceptable because script calls are infrequent relative to tag updates.
|
||||
- **`Instance.CallScript()`**: Ask pattern from the running script (via `ScriptRuntimeContext.CallScript`) to sibling Script Actor. The caller needs the return value. Acceptable because script calls are infrequent relative to tag updates.
|
||||
- **`Route.To().Call()`**: Ask from Inbound API to site Instance Actor via Communication Layer. External caller needs a response.
|
||||
- **Debug view snapshot**: Ask from Communication Layer to Instance Actor for initial state.
|
||||
|
||||
## Concurrency & Serialization
|
||||
|
||||
- The Instance Actor processes messages **sequentially** (standard Akka actor model). This means `SetAttribute` calls from concurrent Script Execution Actors are serialized at the Instance Actor, preventing race conditions on attribute state.
|
||||
- Script Execution Actors may run concurrently, but all state mutations (attribute reads/writes, alarm state updates) are mediated through the parent Instance Actor's message queue.
|
||||
- The Instance Actor processes messages **sequentially** (standard Akka actor model). This means `SetAttribute` calls from concurrent script runs are serialized at the Instance Actor, preventing race conditions on attribute state.
|
||||
- Script runs (launched via `ScriptRunLauncher`, no per-run child actor) may run concurrently, but all state mutations (attribute reads/writes, alarm state updates) are mediated through the parent Instance Actor's message queue.
|
||||
- External side effects (external system calls, notifications, database writes) are not serialized — concurrent scripts may produce interleaved side effects. This is acceptable because each side effect is independent.
|
||||
|
||||
## SiteStreamManager and gRPC Integration
|
||||
@@ -528,7 +533,7 @@ Per Akka.NET best practices, internal actor communication uses **Tell** (fire-an
|
||||
## Error Handling
|
||||
|
||||
### Script Errors
|
||||
- Unhandled exceptions and timeouts in Script Execution Actors are **logged locally** to the site event log.
|
||||
- Unhandled exceptions and timeouts in a script run are **logged locally** to the site event log — unconditionally, regardless of the WP3.2 per-run Started/Completed sampling policy (see Script Run Launch above).
|
||||
- The Script Actor (coordinator) is **not affected** — it remains active for future trigger events.
|
||||
- Script failures are **not reported to central** (except as aggregated error rate metrics via Health Monitoring).
|
||||
|
||||
@@ -540,7 +545,7 @@ Per Akka.NET best practices, internal actor communication uses **Tell** (fire-an
|
||||
### Script Compilation Errors
|
||||
- If script compilation fails when a deployment is received, the entire deployment for that instance is **rejected**. No partial state is applied.
|
||||
- The failure is reported back to central as a failed deployment.
|
||||
- **Enforced by a site-side pre-compile validation gate (S3)**: before the Instance Actor is created or the config persisted, the Deployment Manager compiles **all** of the instance's scripts, alarm on-trigger scripts, and trigger expressions **synchronously, as a pure prefix step of the deploy handler on the singleton's actor thread** — deliberately not off-thread, because piping the verdict back to self would reorder concurrent deploys against delete/disable and break the redeploy-supersede mailbox-FIFO ordering (see the in-code rationale in `DeploymentManagerActor.HandleDeploy`); unchanged script bodies hit the process-wide compile cache, so the gate recompiles only genuinely new code. Any compile failure rejects the deployment with the collected compile errors. This makes the deployment result honest even if central's pre-deployment validation was bypassed or skew let a bad artifact through.
|
||||
- **Enforced by a site-side pre-compile validation gate (S3)**: before the Instance Actor is created or the config persisted, the Deployment Manager compiles **all** of the instance's scripts, alarm on-trigger scripts, and trigger expressions **synchronously, as a pure prefix step of the deploy application on the singleton's actor thread** — the verdict is never derived off-thread, because piping a verdict back to self would reorder concurrent deploys against delete/disable and break the redeploy-supersede mailbox-FIFO ordering. WP3.1 made it **warm-then-gate**: the same compile is first run off-thread purely to populate the process-wide compile cache (`DeployCompileWarmed` pipes back), so the authoritative synchronous gate is then all cache hits and no longer holds the singleton for a Roslyn compile. Ordering is preserved by a **per-instance in-flight guard** — while a warm is in flight for instance X, further mutating commands for X (deploy / delete / disable / enable) are queued and re-dispatched in arrival order once the deploy applies; a second deploy with nothing queued behind it supersedes the pending one last-write-wins and its displaced deployer is answered `Failed`-superseded rather than left to Ask-timeout. Commands for **other** instances flow freely, which is the point, and is safe because cross-instance ordering was never guaranteed to callers. Unchanged script bodies hit the compile cache, so the gate recompiles only genuinely new code. Any compile failure rejects the deployment with the collected compile errors. This makes the deployment result honest even if central's pre-deployment validation was bypassed or skew let a bad artifact through.
|
||||
- Note: Pre-deployment validation at central should still catch compilation errors before they reach the site; the site gate is the authoritative backstop.
|
||||
|
||||
---
|
||||
|
||||
@@ -108,6 +108,12 @@ Notifications are unaffected: they have no tracking table. Their `NotificationId
|
||||
|
||||
On every tracking-table status transition, the site emits a `CachedCallTelemetry` message to the central Site Call Audit component over the site→central channel. Emission is best-effort, at-least-once, and idempotent on `TrackedOperationId`. Because telemetry is best-effort, the site also responds to `CachedCallReconcileRequest` reconciliation pulls — cursor-based per-site reads of tracking rows changed since a cursor — so any missed telemetry self-heals. The site never depends on central; central converges to the site.
|
||||
|
||||
The telemetry-emitting `ICachedCallLifecycleObserver` hook is dispatched through a bounded, single-reader **observer queue** (`StoreAndForwardOptions.ObserverQueueCapacity`, default 10,000, `DropOldest`) — the one unbounded `Channel<T>` left in the system before this bound was added (arch-review WP2.6c). A slow or stuck observer (e.g. a SQLite audit write wedged behind disk contention) can no longer grow this queue without limit; it instead sheds the oldest unprocessed notification, incrementing an `ObserverQueueDroppedCount` counter that counts every drop. **Logging is separately rate-limited (adversarial review finding F3):** a Warning fires for the first drop of an episode, then at most one rollup Warning per minute while drops keep happening — never one Warning-per-dropped-item, which would itself have been a log-flood risk under the exact sustained-drop condition the bound exists to survive. This bounds memory, not delivery: telemetry loss here is covered by the reconciliation pull above, same as any other missed telemetry.
|
||||
|
||||
### Retry Sweep Indexing
|
||||
|
||||
`GetMessagesForRetryAsync` orders candidates `ORDER BY created_at ASC` within the due `status`. The existing `idx_sf_messages_status_due (status, last_attempt_at_ms)` matches the status filter and due-time predicate but not this ordering, forcing a sort/scan on a large sweep. A second covering index, `idx_sf_messages_status_created (status, created_at)`, lets the sweep walk matching rows already in `created_at` order and stop at the batch limit without re-sorting or touching non-pending rows (arch-review WP1.4). Both indexes are retained — `idx_sf_messages_status_due` still backs status+due-time lookups that don't order by `created_at`.
|
||||
|
||||
## Parked Message Management
|
||||
|
||||
- Parked messages remain stored at the site in SQLite.
|
||||
|
||||
@@ -137,6 +137,72 @@ When an instance is deployed, the Template Engine resolves the full configuratio
|
||||
5. Resolve data connection bindings — replace connection name references with concrete connection details from the site.
|
||||
6. Output a flat structure: list of attributes with resolved values and data source addresses, list of alarms with resolved trigger definitions, list of scripts with resolved code and triggers.
|
||||
|
||||
### Flatten-session caching and the graph watermark (WP2.5)
|
||||
|
||||
The flatten above is driven per instance, so an unmemoised implementation re-walks
|
||||
the same template chain — one query per link, plus the compositions of every
|
||||
template it reaches and of every composed chain those reach — once per instance,
|
||||
and re-issues the three session-global queries (shared scripts, the shared-schema
|
||||
library, the target site's data connections) each time too.
|
||||
|
||||
A **`FlattenSession`** memoises all of that for the lifetime of ONE
|
||||
flatten/validate operation. Callers that flatten a batch (`DeploySiteAsync`) pass
|
||||
one shared session; a caller that passes none gets a private single-use session,
|
||||
which still collapses the repeated composed-chain loads inside a single instance's
|
||||
flatten.
|
||||
|
||||
Cache validity is decided by **`ITemplateGraphWatermark`**, a process-wide set of
|
||||
monotonic counters. `TemplateService` and the `ManagementActor` native-alarm-source
|
||||
handlers bump it via `TemplateEngineRepository.SaveChangesAsync`'s change-tracker
|
||||
inspection, which attributes each pending change to its owning template or
|
||||
instance pre-commit. The Transport bundle importer (`BundleImporter`) commits
|
||||
through the raw `DbContext` instead — bypassing that repository — so it calls
|
||||
`ITemplateGraphWatermark.BumpAll()` itself, once per import: after commit on
|
||||
success, after rollback on failure (a bundle import can touch templates,
|
||||
instances, and connections in one transaction, so a scoped bump isn't
|
||||
worthwhile). A data connection edit (`SiteRepository.SaveChangesAsync`) also bumps — via `BumpAll()`,
|
||||
since a connection has no owning template — because `Protocol`/`PrimaryConfiguration`/
|
||||
`BackupConfiguration`/`FailoverRetryCount` are revision-hash inputs (review fix, commit
|
||||
`e0e4b246`, F2c). All three writers converge on the same watermark, so a cached flatten
|
||||
still invalidates correctly regardless of which one changed the graph:
|
||||
|
||||
- a memoised **template** is keyed on `(id, template version)`;
|
||||
- a memoised **chain** is valid only while BOTH the graph's `StructureVersion`
|
||||
(bumped on add/remove, re-parent, or any composition change — i.e. anything that
|
||||
can alter chain MEMBERSHIP) and every member's own version are unchanged. Both
|
||||
halves are load-bearing: structure alone misses an ordinary member edit, member
|
||||
versions alone miss a re-parent.
|
||||
|
||||
Sessions are short-lived by construction, so the design's "template state is
|
||||
captured at the time of flatten" guarantee is unchanged — the session narrows the
|
||||
capture window, it never widens it.
|
||||
|
||||
The watermark is **in-memory and process-local**, deliberately not persisted and
|
||||
not replicated between central nodes. A restart, a failover, or any mutation simply
|
||||
misses and falls back to the authoritative full flatten, so the watermark can only
|
||||
ever cause EXTRA work, never stale work.
|
||||
|
||||
The same watermark backs a **staleness fast path** in `StaleInstanceProbe`: a
|
||||
previously computed revision hash is reused when the instance's version, the
|
||||
structure version, and the version of every template that flatten walked are all
|
||||
unchanged. This matters most where the probe is called per-instance across a whole
|
||||
bundle import or a fleet-wide staleness sweep.
|
||||
|
||||
### Design-time analysis reads
|
||||
|
||||
The design-time checks — acyclicity (`CycleDetector`), naming collisions
|
||||
(`CollisionDetector`) and canonical-name resolution (`TemplateResolver`) — read the
|
||||
whole template graph but write through none of it and read no script bodies. They
|
||||
use `GetAllTemplatesForAnalysisAsync`: `AsNoTracking`, with `TemplateScript.Code`
|
||||
projected away.
|
||||
|
||||
The tracked, body-bearing `GetAllTemplatesAsync` remains the read for the two paths
|
||||
that genuinely need it — the inheritance reconciler (which compares and copies
|
||||
script bodies and writes through the loaded entities) and the flattener (which must
|
||||
observe rows an in-flight bundle import has staged on the shared change tracker).
|
||||
`ReconcileDescendantsAsync` additionally accepts an already-loaded tracked graph so
|
||||
a caller holding one does not force a second load.
|
||||
|
||||
### Native Alarm Source Resolution
|
||||
|
||||
The `FlatteningService` resolves native alarm sources alongside alarms, emitting a `ResolvedNativeAlarmSource` (CanonicalName, ConnectionName, SourceReference, ConditionFilter *(optional)*, and `Source` ∈ `Template` | `Inherited` | `Composed` | `Override`) for each. The resolved set is attached to `FlattenedConfiguration.NativeAlarmSources`.
|
||||
@@ -167,6 +233,18 @@ A derived template stores `IsInherited` **placeholder rows** mirroring every mem
|
||||
|
||||
Both delegate to one **order-independent reconcile** that compares a template's stored inherited rows against the inheritance resolver's effective set (the same precedence + HiLo merge the editor preview and deploy use) and only ever touches `IsInherited` placeholder rows — never an authored override (`IsInherited == false`). Because the resolver ignores placeholder rows when picking winners, reconciling one template never changes what another resolves, so the operation needs no particular ordering. The effective value mirrored into each placeholder matches the staleness comparison key per member type, so after a reconcile the "base changed" banner clears. Reconcile is best-effort housekeeping for the *stored authoring rows*: a deploy always re-resolves the chain fresh regardless, so a not-yet-resynced template still deploys correctly.
|
||||
|
||||
### Slim List Projections (WP2.5)
|
||||
|
||||
The `ListTemplates` management command (backing the CLI `template list` and the Central UI
|
||||
template tree) previously materialised every template's full child graph — attributes, alarms,
|
||||
script bodies, compositions, native alarm sources — to page it in memory and discard all but one
|
||||
page. `ITemplateEngineRepository.GetTemplateSummariesAsync(skip, take)` instead pages **in the
|
||||
database** against `TemplateSummary`, a row-shaped projection (id, name, description, parent/folder
|
||||
ids, derived flag) with child collections reduced to **counts** — the only thing a list surface
|
||||
renders. `TemplateSummary.MaxPageSize` (1000) mirrors the existing in-memory page clamp so DB-side
|
||||
paging cannot be used to pull an unbounded result set. Additive-only message-contract evolution,
|
||||
same rule as other Commons message types.
|
||||
|
||||
## Diff Calculation
|
||||
|
||||
The Template Engine can compare:
|
||||
|
||||
@@ -18,8 +18,9 @@ namespace ZB.MOM.WW.ScadaBridge.AuditLog.Central;
|
||||
/// ingest timestamp into DetailsJson (there is no promoted IngestedAtUtc
|
||||
/// column — the value is a DetailsJson field set via
|
||||
/// <see cref="AuditRowProjection.WithIngestedAtUtc"/>) and inserted idempotently
|
||||
/// via <see cref="IAuditLogRepository.InsertIfNotExistsAsync"/> — duplicates are
|
||||
/// silently swallowed (first-write-wins).
|
||||
/// via <see cref="IAuditLogRepository.InsertManyIfNotExistsAsync"/> — duplicates
|
||||
/// are silently swallowed (first-write-wins), whether they repeat inside one
|
||||
/// packet or across packets.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
@@ -28,9 +29,11 @@ namespace ZB.MOM.WW.ScadaBridge.AuditLog.Central;
|
||||
/// consistent and the site is free to flip its local row to <c>Forwarded</c>.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Audit-write failures must NEVER abort the user-facing action. The actor
|
||||
/// wraps each repository call in its own try/catch so a single bad row cannot
|
||||
/// cause the rest of the batch to be lost, and it guards scope/repository
|
||||
/// Audit-write failures must NEVER abort the user-facing action. Each message
|
||||
/// is written as ONE set-based statement (or one transaction, for the cached
|
||||
/// dual-write); if that fails the actor retries the same work row-by-row inside
|
||||
/// per-row try/catch, so a single bad row still cannot cause the rest of the
|
||||
/// batch to be lost. It also guards scope/repository
|
||||
/// resolution so a transient DI fault cannot restart the singleton — those
|
||||
/// catches are what keep this actor alive across handler throws, not the
|
||||
/// supervisor strategy. The <see cref="SupervisorStrategy"/> override returns
|
||||
@@ -50,6 +53,53 @@ namespace ZB.MOM.WW.ScadaBridge.AuditLog.Central;
|
||||
/// </remarks>
|
||||
public class AuditLogIngestActor : ReceiveActor
|
||||
{
|
||||
/// <summary>
|
||||
/// Overall budget for one ingest message's database work, deliberately
|
||||
/// SHORTER than the gRPC Ask that wraps it.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// The path used to stack three identical 30 s budgets — the site's Ask, the
|
||||
/// central gRPC handler's Ask (<c>SiteStreamGrpcServer.AuditIngestAskTimeout</c>)
|
||||
/// and the ADO.NET command default — so they all expired at the same instant. The
|
||||
/// caller therefore learned nothing except "it took 30 s": no partial ack, no
|
||||
/// distinction between a slow database and a wedged singleton. Making the
|
||||
/// innermost budget strictly smallest means a slow batch is abandoned by the
|
||||
/// actor FIRST, with the accepted-so-far ids still replied, while the outer
|
||||
/// Asks are still waiting.
|
||||
/// <para>
|
||||
/// The full ladder is now strictly monotonic end to end:
|
||||
/// <c>CommunicationOptions.AuditForwardTimeout</c> (35 s, the site-side forward Ask) >
|
||||
/// <c>SiteStreamGrpcServer.AuditIngestAskTimeout</c> (30 s, the gRPC deadline AND central's
|
||||
/// Ask of this singleton) > <see cref="IngestBudget"/> (20 s) >
|
||||
/// <see cref="IngestSqlCommandTimeout"/> (15 s).
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
internal static readonly TimeSpan IngestBudget = TimeSpan.FromSeconds(20);
|
||||
|
||||
/// <summary>
|
||||
/// Per-statement SQL timeout for the ingest write, strictly inside
|
||||
/// <see cref="IngestBudget"/> so a single wedged statement surfaces as a SQL
|
||||
/// timeout the per-batch catch can log, rather than consuming the whole
|
||||
/// actor-level budget and starving the rest of the batch.
|
||||
/// </summary>
|
||||
internal static readonly TimeSpan IngestSqlCommandTimeout = TimeSpan.FromSeconds(15);
|
||||
|
||||
/// <summary>
|
||||
/// Budget for the per-row fallback loop, granted as a FRESH
|
||||
/// <see cref="CancellationTokenSource"/> rather than reusing the batch's.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// The fallback used to run on the batch's own token. When the batch failed
|
||||
/// BECAUSE that token expired, every fallback insert was handed an
|
||||
/// already-cancelled token and failed instantly: N logged errors, N counter
|
||||
/// bumps, zero rows accepted, and the site retried the whole packet anyway.
|
||||
/// A fresh, deliberately SHORT budget (shorter than <see cref="IngestBudget"/>,
|
||||
/// so the reply still beats the caller's Ask even when the batch consumed its
|
||||
/// whole allowance) gives the poison-row isolation the fallback exists for a
|
||||
/// real chance to land the good rows.
|
||||
/// </remarks>
|
||||
internal static readonly TimeSpan IngestFallbackBudget = TimeSpan.FromSeconds(5);
|
||||
|
||||
private readonly IServiceProvider? _serviceProvider;
|
||||
private readonly IAuditLogRepository? _injectedRepository;
|
||||
private readonly ILogger<AuditLogIngestActor> _logger;
|
||||
@@ -185,39 +235,101 @@ public class AuditLogIngestActor : ReceiveActor
|
||||
DateTime nowUtc,
|
||||
List<Guid> accepted)
|
||||
{
|
||||
// Stamp the ingest timestamp here, not at the site. Redact BEFORE the
|
||||
// IngestedAtUtc stamp so the redacted copy carries the central-side
|
||||
// ingest timestamp. The redactor is contract-bound to never throw; a null
|
||||
// redactor (test composition root, no IAuditRedactor registered) falls
|
||||
// back to the SafeDefault rather than pass-through, so HTTP header
|
||||
// redaction always runs. IngestedAtUtc is a DetailsJson field on the
|
||||
// canonical record, so stamp it via the projection helper.
|
||||
var safeRedactor = redactor ?? SafeDefaultAuditRedactor.Instance;
|
||||
var projected = new List<AuditEvent>(cmd.Events.Count);
|
||||
foreach (var evt in cmd.Events)
|
||||
{
|
||||
try
|
||||
var filtered = safeRedactor.Apply(evt);
|
||||
projected.Add(AuditRowProjection.WithIngestedAtUtc(filtered, nowUtc));
|
||||
}
|
||||
|
||||
// ONE set-based statement for the whole packet (WP2.2) instead of one
|
||||
// IF NOT EXISTS … INSERT round trip per event. A telemetry packet of 200
|
||||
// rows used to cost 200 sequential round trips on the central singleton's
|
||||
// dispatcher; it now costs two. Idempotency is unchanged — duplicates
|
||||
// within the packet collapse first-write-wins, duplicates against
|
||||
// committed rows are eliminated by the anti-semi-join, and a concurrent
|
||||
// writer degrades to the repository's per-row fallback.
|
||||
using var budget = new CancellationTokenSource(IngestBudget);
|
||||
try
|
||||
{
|
||||
await repository
|
||||
.InsertManyIfNotExistsAsync(projected, IngestSqlCommandTimeout, budget.Token)
|
||||
.ConfigureAwait(false);
|
||||
|
||||
// A batch that returned without throwing means every row is now
|
||||
// present — inserted here or already committed by an earlier delivery.
|
||||
// Both count as accepted: the storage state is consistent and the site
|
||||
// is free to flip its local rows to Forwarded.
|
||||
foreach (var evt in cmd.Events)
|
||||
{
|
||||
// Stamp the ingest timestamp here, not at the site. The
|
||||
// repository's duplicate-key hardening already swallows
|
||||
// duplicate-key races, so the same id arriving twice (site
|
||||
// retry, reconciliation) is a silent no-op.
|
||||
// Redact BEFORE the IngestedAtUtc stamp so the redacted
|
||||
// copy carries the central-side ingest timestamp. The redactor
|
||||
// is contract-bound to never throw. A null
|
||||
// redactor (test composition root, no IAuditRedactor
|
||||
// registered) now falls back to the SafeDefault rather than
|
||||
// pass-through, so HTTP header redaction always runs.
|
||||
// IngestedAtUtc is a DetailsJson field on
|
||||
// the canonical record, so stamp it via the projection helper.
|
||||
var safeRedactor = redactor ?? SafeDefaultAuditRedactor.Instance;
|
||||
var filtered = safeRedactor.Apply(evt);
|
||||
var ingested = AuditRowProjection.WithIngestedAtUtc(filtered, nowUtc);
|
||||
await repository.InsertIfNotExistsAsync(ingested).ConfigureAwait(false);
|
||||
accepted.Add(evt.EventId);
|
||||
}
|
||||
catch (Exception ex)
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
// The batch is a throughput optimisation, NOT a change to the
|
||||
// failure grain. The documented invariant — "a single bad row cannot
|
||||
// cause the rest of the batch to be lost" — is preserved by falling
|
||||
// back to the per-row path here, so a poison row still costs only
|
||||
// itself. Re-running rows the failed batch may already have committed
|
||||
// is safe: every insert is idempotent on EventId.
|
||||
//
|
||||
// The fallback gets its OWN token. Reusing the batch's meant that a
|
||||
// batch which failed because the budget EXPIRED handed the loop an
|
||||
// already-cancelled token, so every row failed instantly and the
|
||||
// fallback did nothing but multiply the log and counter noise by the
|
||||
// row count.
|
||||
var budgetExpired = budget.IsCancellationRequested;
|
||||
using var fallbackBudget = new CancellationTokenSource(IngestFallbackBudget);
|
||||
|
||||
_logger.LogWarning(ex,
|
||||
"Set-based ingest of {Count} audit event(s) failed{BudgetNote}; falling back to per-row inserts (fresh {FallbackSeconds}s budget) so one bad row does not sink the batch.",
|
||||
cmd.Events.Count,
|
||||
budgetExpired ? " after exhausting the ingest budget" : string.Empty,
|
||||
IngestFallbackBudget.TotalSeconds);
|
||||
|
||||
if (budgetExpired)
|
||||
{
|
||||
// Per-row catch — one bad row never sinks the whole batch.
|
||||
// The row stays Pending at the site; the next drain retries.
|
||||
// Bump the central health counter so a
|
||||
// sustained insert-throw failure surfaces on the dashboard.
|
||||
// A blown budget is ONE failure event for the batch, not one per
|
||||
// row — bump the health counter once here and suppress the
|
||||
// per-row bumps below so the dashboard sees a timeout as a
|
||||
// timeout rather than as N independent write failures.
|
||||
try { failureCounter?.Increment(); }
|
||||
catch { /* counter must never throw — defence in depth */ }
|
||||
_logger.LogError(ex,
|
||||
"Failed to persist audit event {EventId} during batch ingest; row will be retried by the site.",
|
||||
evt.EventId);
|
||||
}
|
||||
|
||||
for (var i = 0; i < projected.Count; i++)
|
||||
{
|
||||
try
|
||||
{
|
||||
await repository.InsertIfNotExistsAsync(projected[i], fallbackBudget.Token).ConfigureAwait(false);
|
||||
accepted.Add(cmd.Events[i].EventId);
|
||||
}
|
||||
catch (Exception rowEx)
|
||||
{
|
||||
// Per-row catch — one bad row never sinks the whole batch.
|
||||
// The row stays Pending at the site; the next drain retries.
|
||||
// Bump the central health counter so a sustained insert-throw
|
||||
// failure surfaces on the dashboard — unless the batch already
|
||||
// counted itself as a single budget-exhaustion failure.
|
||||
if (!budgetExpired)
|
||||
{
|
||||
try { failureCounter?.Increment(); }
|
||||
catch { /* counter must never throw — defence in depth */ }
|
||||
}
|
||||
|
||||
_logger.LogError(rowEx,
|
||||
"Failed to persist audit event {EventId} during batch ingest; row will be retried by the site.",
|
||||
cmd.Events[i].EventId);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -262,6 +374,24 @@ public class AuditLogIngestActor : ReceiveActor
|
||||
|
||||
var strategy = dbContext.Database.CreateExecutionStrategy();
|
||||
|
||||
// Fast path (WP2.2): the WHOLE packet in ONE transaction — one
|
||||
// set-based audit insert plus one single-statement upsert per entry.
|
||||
// The predecessor opened a transaction PER entry and issued three
|
||||
// statements inside it (IF NOT EXISTS insert, then the two-statement
|
||||
// SiteCalls upsert), so a 50-entry packet cost ~200 round trips and 50
|
||||
// commits. If anything faults, the per-entry loop below re-runs the
|
||||
// packet with its original per-entry isolation, so the documented
|
||||
// "one entry's failure does not abort the others" invariant survives —
|
||||
// it is simply no longer paid for on the healthy path.
|
||||
using var budget = new CancellationTokenSource(IngestBudget);
|
||||
if (await TryIngestCachedBatchAsync(
|
||||
strategy, dbContext, auditRepo, siteCallRepo, redactor, cmd, accepted, budget.Token)
|
||||
.ConfigureAwait(false))
|
||||
{
|
||||
replyTo.Tell(new IngestCachedTelemetryReply(accepted));
|
||||
return;
|
||||
}
|
||||
|
||||
foreach (var entry in cmd.Entries)
|
||||
{
|
||||
try
|
||||
@@ -330,6 +460,90 @@ public class AuditLogIngestActor : ReceiveActor
|
||||
replyTo.Tell(new IngestCachedTelemetryReply(accepted));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Attempts the whole cached-telemetry packet as ONE transaction: a single
|
||||
/// set-based audit insert followed by one monotonic <c>SiteCalls</c> upsert
|
||||
/// per entry. Returns <see langword="true"/> when it committed (and only then
|
||||
/// appends to <paramref name="accepted"/>), <see langword="false"/> when the
|
||||
/// caller should fall back to the per-entry transaction loop.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// The batch is all-or-nothing by construction — it is a single transaction —
|
||||
/// which is why a failure MUST fall back rather than report partial success:
|
||||
/// the per-entry loop is what turns one poison entry into one lost entry
|
||||
/// instead of a lost packet. Nothing is appended to
|
||||
/// <paramref name="accepted"/> until the commit returns, so a failed attempt
|
||||
/// leaves the caller's list untouched and the retry starts from a clean slate.
|
||||
/// </remarks>
|
||||
private async Task<bool> TryIngestCachedBatchAsync(
|
||||
Microsoft.EntityFrameworkCore.Storage.IExecutionStrategy strategy,
|
||||
ScadaBridgeDbContext dbContext,
|
||||
IAuditLogRepository auditRepo,
|
||||
ISiteCallAuditRepository siteCallRepo,
|
||||
IAuditRedactor? redactor,
|
||||
IngestCachedTelemetryCommand cmd,
|
||||
List<Guid> accepted,
|
||||
CancellationToken ct)
|
||||
{
|
||||
try
|
||||
{
|
||||
await strategy.ExecuteAsync(async () =>
|
||||
{
|
||||
await using var tx = await dbContext.Database
|
||||
.BeginTransactionAsync(ct)
|
||||
.ConfigureAwait(false);
|
||||
|
||||
// One central-side instant for the whole packet so a join across
|
||||
// the two tables sees matching timestamps (debugging convenience,
|
||||
// not a correctness invariant).
|
||||
var ingestedAt = DateTime.UtcNow;
|
||||
var safeRedactor = redactor ?? SafeDefaultAuditRedactor.Instance;
|
||||
|
||||
var auditRows = new List<AuditEvent>(cmd.Entries.Count);
|
||||
foreach (var entry in cmd.Entries)
|
||||
{
|
||||
// Only the AuditLog row's payload columns are redactable;
|
||||
// SiteCalls carries operational state only (status, retry
|
||||
// count) and is left untouched.
|
||||
var filteredAudit = safeRedactor.Apply(entry.Audit);
|
||||
auditRows.Add(AuditRowProjection.WithIngestedAtUtc(filteredAudit, ingestedAt));
|
||||
}
|
||||
|
||||
await auditRepo
|
||||
.InsertManyIfNotExistsAsync(auditRows, IngestSqlCommandTimeout, ct)
|
||||
.ConfigureAwait(false);
|
||||
|
||||
foreach (var entry in cmd.Entries)
|
||||
{
|
||||
await siteCallRepo
|
||||
.UpsertAsync(entry.SiteCall with { IngestedAtUtc = ingestedAt }, ct)
|
||||
.ConfigureAwait(false);
|
||||
}
|
||||
|
||||
await tx.CommitAsync(ct).ConfigureAwait(false);
|
||||
}).ConfigureAwait(false);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
// No health-counter bump here — the per-entry retry is the authority
|
||||
// on whether this packet genuinely failed, and double-counting a
|
||||
// batch that the fallback then writes successfully would make the
|
||||
// dashboard read as a sustained fault during normal contention.
|
||||
_logger.LogWarning(
|
||||
ex,
|
||||
"Batched cached-telemetry dual-write of {Count} entr(ies) failed; falling back to per-entry transactions.",
|
||||
cmd.Entries.Count);
|
||||
return false;
|
||||
}
|
||||
|
||||
foreach (var entry in cmd.Entries)
|
||||
{
|
||||
accepted.Add(entry.Audit.EventId);
|
||||
}
|
||||
|
||||
return true;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Fallback handler installed on the single-repository test ctor — that
|
||||
/// ctor has no DbContext and no <see cref="ISiteCallAuditRepository"/>, so
|
||||
|
||||
@@ -17,8 +17,9 @@ namespace ZB.MOM.WW.ScadaBridge.AuditLog.Central;
|
||||
/// for monthly boundaries whose latest <c>OccurredAtUtc</c> is older
|
||||
/// than <c>DateTime.UtcNow - RetentionDays</c>.</item>
|
||||
/// <item>For each eligible boundary, calls
|
||||
/// <see cref="IAuditLogRepository.SwitchOutPartitionAsync"/> which runs
|
||||
/// the drop-and-rebuild dance around <c>UX_AuditLog_EventId</c>.</item>
|
||||
/// <see cref="IAuditLogRepository.SwitchOutPartitionAsync"/>, a
|
||||
/// metadata-only staging-table switch (WP2.2 removed the index
|
||||
/// drop/rebuild that used to bracket it).</item>
|
||||
/// <item>Publishes <see cref="AuditLogPurgedEvent"/> on the actor-system
|
||||
/// EventStream so the central health collector + ops surfaces
|
||||
/// can subscribe without coupling to this actor.</item>
|
||||
@@ -26,11 +27,10 @@ namespace ZB.MOM.WW.ScadaBridge.AuditLog.Central;
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// <b>Daily cadence.</b> Partition switch is metadata-only but the
|
||||
/// drop-and-rebuild dance briefly removes <c>UX_AuditLog_EventId</c>; running
|
||||
/// more often than necessary trades unique-index rebuild outages for
|
||||
/// negligible freshness wins. The default 24-hour interval matches
|
||||
/// alog.md §10's retention policy.
|
||||
/// <b>Daily cadence.</b> The partition switch is metadata-only, but it still
|
||||
/// takes schema-modification locks on a table the ingest path is writing to;
|
||||
/// running more often than necessary trades contention for negligible freshness
|
||||
/// wins. The default 24-hour interval matches alog.md §10's retention policy.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Continue-on-error.</b> A single boundary that throws (transient SQL
|
||||
|
||||
@@ -10,12 +10,12 @@ namespace ZB.MOM.WW.ScadaBridge.AuditLog.Central;
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// The purge actor is a daily-cadence singleton, not a hot-loop, because
|
||||
/// partition-switch I/O is metadata-only but the drop-and-rebuild dance
|
||||
/// briefly removes the <c>UX_AuditLog_EventId</c> unique index — running
|
||||
/// more often than necessary trades index-rebuild outages for marginal
|
||||
/// freshness gains. Lower this only when an operator can prove they need
|
||||
/// sub-daily purge granularity.
|
||||
/// The purge actor is a daily-cadence singleton, not a hot-loop. The
|
||||
/// partition switch itself is metadata-only (since WP2.2's
|
||||
/// <c>AlignAuditLogEventIdUniqueness</c> it no longer drops and rebuilds a
|
||||
/// non-aligned unique index around it), but it still takes schema-modification
|
||||
/// locks and competes with ingest for the same table. Lower this only when an
|
||||
/// operator can prove they need sub-daily purge granularity.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <see cref="IntervalOverride"/> exists for tests to drop the cadence to
|
||||
@@ -58,15 +58,14 @@ public sealed class AuditLogPurgeOptions
|
||||
|
||||
/// <summary>
|
||||
/// Per-command timeout (in minutes) for the maintenance SQL the purge tick issues —
|
||||
/// both the partition switch-out drop-and-rebuild dance
|
||||
/// both the partition switch-out staging batch
|
||||
/// (<see cref="ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories.IAuditLogRepository.SwitchOutPartitionAsync"/>)
|
||||
/// and each per-channel <c>DELETE TOP</c> batch. Default 30 minutes.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// The ADO.NET default command timeout is ~30 seconds. On a large or contended partition the
|
||||
/// SWITCH dance (which briefly drops <c>UX_AuditLog_EventId</c>) can exceed that and abort
|
||||
/// mid-flight — leaving the live table without its idempotency-supporting unique index until a
|
||||
/// later tick's CATCH branch rebuilds it (arch-review 04, S2). A generous maintenance timeout
|
||||
/// SWITCH batch can exceed that and abort mid-flight, leaving an orphaned staging table for the
|
||||
/// next tick's CATCH branch to clean up (arch-review 04, S2). A generous maintenance timeout
|
||||
/// lets the metadata-only switch complete rather than self-locking. Resolved via
|
||||
/// <see cref="ResolvedMaintenanceCommandTimeout"/>, clamped to a 1-minute floor.
|
||||
/// </remarks>
|
||||
|
||||
@@ -73,6 +73,7 @@ public sealed class GrpcPullAuditEventsClient : IPullAuditEventsClient
|
||||
public async Task<PullAuditEventsResponse> PullAsync(
|
||||
string siteId,
|
||||
DateTime sinceUtc,
|
||||
string? afterId,
|
||||
int batchSize,
|
||||
CancellationToken ct)
|
||||
{
|
||||
@@ -93,6 +94,10 @@ public sealed class GrpcPullAuditEventsClient : IPullAuditEventsClient
|
||||
// EnsureUtc keeps Timestamp.FromDateTime happy (it requires UTC kind).
|
||||
SinceUtc = Timestamp.FromDateTime(EnsureUtc(sinceUtc)),
|
||||
BatchSize = batchSize,
|
||||
// Composite-keyset tiebreak (proto field 3), mirroring PullSiteCalls exactly.
|
||||
// proto3 has no nullable string — an unset/empty AfterId is the site's signal to
|
||||
// keep the legacy inclusive-timestamp contract (also what a first pull sends).
|
||||
AfterId = afterId ?? string.Empty,
|
||||
};
|
||||
|
||||
var (reply, transportFault) = await TryInvokeAsync(endpoint, request, siteId, ct)
|
||||
@@ -121,10 +126,13 @@ public sealed class GrpcPullAuditEventsClient : IPullAuditEventsClient
|
||||
|
||||
// Map proto DTOs to canonical AuditEvent records and order oldest-first
|
||||
// (the wire is already ordered by the site queue, but the
|
||||
// IPullAuditEventsClient contract is explicit, so sort defensively).
|
||||
// IPullAuditEventsClient contract is explicit, so sort defensively). The EventId
|
||||
// tiebreak matches the site's own composite ordering — ordinal over the "D" GUID
|
||||
// text, which is what SQLite's BINARY collation compares.
|
||||
var events = reply.Events
|
||||
.Select(AuditEventDtoMapper.FromDto)
|
||||
.OrderBy(e => e.OccurredAtUtc)
|
||||
.ThenBy(e => e.EventId.ToString(), StringComparer.Ordinal)
|
||||
.ToList();
|
||||
|
||||
return new PullAuditEventsResponse(events, reply.MoreAvailable);
|
||||
|
||||
@@ -39,12 +39,24 @@ public interface IPullAuditEventsClient
|
||||
/// </summary>
|
||||
/// <param name="siteId">The identifier of the site to pull audit events from.</param>
|
||||
/// <param name="sinceUtc">Only events with an <c>OccurredAtUtc</c> at or after this cursor time are returned.</param>
|
||||
/// <param name="afterId">
|
||||
/// The composite-keyset tiebreak cursor, mirroring
|
||||
/// <see cref="IPullSiteCallsClient.PullAsync"/>. When non-null it is the
|
||||
/// <c>EventId</c> ("D" GUID form) of the last row already consumed at
|
||||
/// <paramref name="sinceUtc"/>; the site returns only rows strictly greater than the
|
||||
/// composite <c>(OccurredAtUtc, EventId)</c> pair, so a burst sharing one exact instant
|
||||
/// drains via the id tiebreak instead of pinning the inclusive-timestamp cursor — and the
|
||||
/// site's <c>MarkReconciledUpToAsync</c> can retire the rows AT that instant, which a bare
|
||||
/// timestamp can never prove received. Null on the first pull (or against a legacy site)
|
||||
/// preserves the inclusive <c>>=</c> contract.
|
||||
/// </param>
|
||||
/// <param name="batchSize">Maximum number of events to return per call.</param>
|
||||
/// <param name="ct">Cancellation token.</param>
|
||||
/// <returns>A task that resolves to the next reconciliation batch with a <c>MoreAvailable</c> flag.</returns>
|
||||
Task<PullAuditEventsResponse> PullAsync(
|
||||
string siteId,
|
||||
DateTime sinceUtc,
|
||||
string? afterId,
|
||||
int batchSize,
|
||||
CancellationToken ct);
|
||||
}
|
||||
|
||||
@@ -76,14 +76,28 @@ public class SiteAuditReconciliationActor : ReceiveActor
|
||||
private readonly ILogger<SiteAuditReconciliationActor> _logger;
|
||||
|
||||
/// <summary>
|
||||
/// Per-site reconciliation watermark — the highest
|
||||
/// <see cref="AuditEvent.OccurredAtUtc"/> seen for that site on a previous
|
||||
/// tick. Asking for <c>OccurredAtUtc >= cursor</c> rather than >
|
||||
/// is the site contract (<see cref="ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services.ISiteAuditQueue.ReadPendingSinceAsync"/>);
|
||||
/// duplicate-with-same-timestamp rows are filtered out by the idempotent
|
||||
/// repository write.
|
||||
/// Per-site reconciliation watermark — the COMPOSITE
|
||||
/// <c>(OccurredAtUtc, EventId)</c> of the highest row seen for that site on a previous tick,
|
||||
/// mirroring <c>SiteCallAuditActor</c>'s <c>PullSiteCalls</c> cursor.
|
||||
/// </summary>
|
||||
private readonly Dictionary<string, DateTime> _cursors = new();
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// The site's
|
||||
/// <see cref="ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services.ISiteAuditQueue.ReadPendingSinceAsync"/>
|
||||
/// serves rows strictly after the pair when <c>AfterId</c> is set, and falls back to the
|
||||
/// legacy inclusive <c>OccurredAtUtc >= since</c> when it is null (the first pull).
|
||||
/// Sending the id half matters twice over: a burst sharing one exact instant drains via the
|
||||
/// tiebreak instead of pinning the timestamp forever, and
|
||||
/// <see cref="ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services.ISiteAuditQueue.MarkReconciledUpToAsync"/>
|
||||
/// can retire the rows AT the cursor instant — a bare timestamp can only prove receipt of
|
||||
/// rows strictly older than itself, so the boundary rows stayed servable forever.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Duplicates that a re-pull does produce are still filtered by the idempotent repository
|
||||
/// write, so the cursor is an optimization for exactness, never a correctness crutch.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
private readonly Dictionary<string, (DateTime Since, string? AfterId)> _cursors = new();
|
||||
|
||||
/// <summary>
|
||||
/// Per-site count of consecutive non-draining cycles. Resets to zero on the
|
||||
@@ -239,20 +253,33 @@ public class SiteAuditReconciliationActor : ReceiveActor
|
||||
/// <summary>
|
||||
/// Issues one <c>PullAuditEvents</c> RPC against the site, ingests the
|
||||
/// returned rows idempotently into the central repository, and advances
|
||||
/// the cursor based on the maximum <see cref="AuditEvent.OccurredAtUtc"/>
|
||||
/// observed. The brief's "saturate until backlog clears" intent is met by
|
||||
/// the natural cadence — each tick issues one pull, and a backed-up site
|
||||
/// the composite <c>(<see cref="AuditEvent.OccurredAtUtc"/>, <see cref="AuditEvent.EventId"/>)</c>
|
||||
/// cursor to the maximum row observed. The brief's "saturate until backlog clears" intent is
|
||||
/// met by the natural cadence — each tick issues one pull, and a backed-up site
|
||||
/// drains across consecutive ticks. The stalled signal (two non-draining
|
||||
/// ticks in a row) surfaces when that drain isn't keeping up.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Unlike <c>SiteCallAuditActor</c> this does NOT page within a tick; one pull per tick is
|
||||
/// deliberate (the stalled signal is how a lagging drain surfaces). The composite cursor is
|
||||
/// what makes that safe against a same-instant burst larger than one batch: the id tiebreak
|
||||
/// advances even when the timestamp cannot, so consecutive ticks make real progress instead
|
||||
/// of re-serving the same window forever.
|
||||
/// </remarks>
|
||||
private async Task PullSiteAsync(SiteEntry site, IAuditLogRepository repository, Akka.Event.EventStream eventStream)
|
||||
{
|
||||
var since = _cursors.TryGetValue(site.SiteId, out var c) ? c : DateTime.MinValue;
|
||||
var cursor = _cursors.TryGetValue(site.SiteId, out var c)
|
||||
? c
|
||||
: (Since: DateTime.MinValue, AfterId: (string?)null);
|
||||
var since = cursor.Since;
|
||||
var afterId = cursor.AfterId;
|
||||
|
||||
var response = await _client.PullAsync(
|
||||
site.SiteId, since, _options.BatchSize, CancellationToken.None)
|
||||
site.SiteId, since, afterId, _options.BatchSize, CancellationToken.None)
|
||||
.ConfigureAwait(false);
|
||||
|
||||
var maxOccurred = since;
|
||||
var maxAfterId = afterId;
|
||||
var hasUnresolvedFailure = false;
|
||||
var nowUtc = DateTime.UtcNow;
|
||||
foreach (var evt in response.Events)
|
||||
@@ -310,10 +337,18 @@ public class SiteAuditReconciliationActor : ReceiveActor
|
||||
}
|
||||
|
||||
// Canonical OccurredAtUtc is a DateTimeOffset; the cursor is a UTC DateTime.
|
||||
// Advance the COMPOSITE max, exactly as SiteCallAuditActor does: a greater
|
||||
// timestamp wins; on a tie the greater EventId (ordinal over the "D" GUID text,
|
||||
// matching the site's SQLite BINARY collation) wins. CompareOrdinal handles the
|
||||
// null seed — CompareOrdinal(x, null) > 0 for any non-null x.
|
||||
var occurredUtc = evt.OccurredAtUtc.UtcDateTime;
|
||||
if (advanceForThisRow && occurredUtc > maxOccurred)
|
||||
var rowId = evt.EventId.ToString();
|
||||
if (advanceForThisRow &&
|
||||
(occurredUtc > maxOccurred ||
|
||||
(occurredUtc == maxOccurred && string.CompareOrdinal(rowId, maxAfterId) > 0)))
|
||||
{
|
||||
maxOccurred = occurredUtc;
|
||||
maxAfterId = rowId;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -322,8 +357,11 @@ public class SiteAuditReconciliationActor : ReceiveActor
|
||||
// the whole batch next tick — successful rows are no-ops thanks to
|
||||
// InsertIfNotExistsAsync's idempotency, and the failing row gets
|
||||
// another attempt. Once it succeeds (or hits the permanent-abandon
|
||||
// threshold) the cursor unblocks naturally.
|
||||
_cursors[site.SiteId] = hasUnresolvedFailure ? since : maxOccurred;
|
||||
// threshold) the cursor unblocks naturally. Both halves move together: holding the
|
||||
// timestamp back while advancing the id would skip the very rows being retried.
|
||||
_cursors[site.SiteId] = hasUnresolvedFailure
|
||||
? (since, afterId)
|
||||
: (maxOccurred, maxAfterId);
|
||||
|
||||
var nonDraining = response.MoreAvailable && response.Events.Count > 0;
|
||||
UpdateStalledState(site.SiteId, draining: !nonDraining, eventStream);
|
||||
|
||||
@@ -45,12 +45,27 @@ public sealed class SiteAuditBacklogReporter : IHostedService, IDisposable
|
||||
/// </summary>
|
||||
internal static readonly TimeSpan DefaultRefreshInterval = TimeSpan.FromSeconds(30);
|
||||
|
||||
/// <summary>
|
||||
/// Age at which the oldest still-Pending row is called out in the log. A Pending row is
|
||||
/// one central has not acknowledged through EITHER path (telemetry push or reconciliation
|
||||
/// pull), and <c>PurgeExpiredAsync</c> deliberately never purges Pending — so the site
|
||||
/// store's floor depends on reconciliation actually running. Rows served by a pull but
|
||||
/// never covered by a later cursor sit in exactly this state, which is why the age is
|
||||
/// worth a signal rather than only a dashboard number. One day is comfortably longer than
|
||||
/// any normal drain/reconcile outage and well inside the ~7-day site retention window.
|
||||
/// </summary>
|
||||
internal static readonly TimeSpan StalePendingThreshold = TimeSpan.FromHours(24);
|
||||
|
||||
/// <summary>How often the stale-pending warning may repeat (it is a standing condition).</summary>
|
||||
internal static readonly TimeSpan StalePendingWarnInterval = TimeSpan.FromHours(1);
|
||||
|
||||
private readonly ISiteAuditQueue _queue;
|
||||
private readonly ISiteHealthCollector _collector;
|
||||
private readonly ILogger<SiteAuditBacklogReporter> _logger;
|
||||
private readonly TimeSpan _refreshInterval;
|
||||
private CancellationTokenSource? _cts;
|
||||
private Task? _loop;
|
||||
private DateTime _lastStalePendingWarnUtc = DateTime.MinValue;
|
||||
|
||||
/// <summary>Initializes a new instance of <see cref="SiteAuditBacklogReporter"/>.</summary>
|
||||
/// <param name="queue">The site audit queue used to probe the backlog count.</param>
|
||||
@@ -151,6 +166,7 @@ public sealed class SiteAuditBacklogReporter : IHostedService, IDisposable
|
||||
{
|
||||
var snapshot = await _queue.GetBacklogStatsAsync(ct).ConfigureAwait(false);
|
||||
_collector.UpdateSiteAuditBacklog(snapshot);
|
||||
WarnIfPendingIsStale(snapshot.OldestPendingUtc, snapshot.PendingCount);
|
||||
}
|
||||
catch (OperationCanceledException)
|
||||
{
|
||||
@@ -166,6 +182,34 @@ public sealed class SiteAuditBacklogReporter : IHostedService, IDisposable
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Logs a rate-limited warning when the oldest still-Pending audit row is older than
|
||||
/// <see cref="StalePendingThreshold"/>. Pending rows are exempt from the retention purge
|
||||
/// by design, so a standing Pending backlog is the one site-store condition that does not
|
||||
/// self-heal on age — it clears only when central acknowledges the rows (telemetry ack or
|
||||
/// a reconciliation cursor that covers them). Internal so tests can drive it directly.
|
||||
/// </summary>
|
||||
/// <param name="oldestPendingUtc">Oldest pending row's occurrence instant, or null when none.</param>
|
||||
/// <param name="pendingCount">Number of rows currently pending.</param>
|
||||
internal void WarnIfPendingIsStale(DateTime? oldestPendingUtc, int pendingCount)
|
||||
{
|
||||
if (oldestPendingUtc is null) return;
|
||||
|
||||
var age = DateTime.UtcNow - DateTime.SpecifyKind(oldestPendingUtc.Value, DateTimeKind.Utc);
|
||||
if (age < StalePendingThreshold) return;
|
||||
|
||||
var now = DateTime.UtcNow;
|
||||
if (now - _lastStalePendingWarnUtc < StalePendingWarnInterval) return;
|
||||
_lastStalePendingWarnUtc = now;
|
||||
|
||||
_logger.LogWarning(
|
||||
"Site audit backlog: oldest pending row is {AgeHours:F1}h old ({PendingCount} pending). " +
|
||||
"Pending rows are never purged on age, so this backlog only clears when central " +
|
||||
"acknowledges them — check the site→central audit telemetry drain and the central " +
|
||||
"reconciliation pull.",
|
||||
age.TotalHours, pendingCount);
|
||||
}
|
||||
|
||||
/// <summary>Signals the polling loop to stop and waits for it to complete.</summary>
|
||||
/// <param name="ct">Cancellation token (not used; the internal CTS governs shutdown).</param>
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
|
||||
@@ -174,6 +174,24 @@ public class SqliteAuditWriter : IAuditWriter, ISiteAuditQueue, IAsyncDisposable
|
||||
pragmaCmd.ExecuteNonQuery();
|
||||
}
|
||||
|
||||
// synchronous=NORMAL (arch-review remediation WP1.2): SQLite's FULL default
|
||||
// fsyncs on every transaction commit, which is the dominant cost of the
|
||||
// per-event write path this same change coalesces via FlushIntervalMs — FULL
|
||||
// would still fsync once per FLUSH even after that fix. NORMAL fsyncs at
|
||||
// fewer, WAL-checkpoint-aligned points instead of every commit; the window
|
||||
// this opens is a handful of the most recent commits lost on OS crash / power
|
||||
// loss (WAL mode itself still guarantees no *corruption*, only a possible
|
||||
// short rollback on unclean restart). That trade is acceptable here because
|
||||
// audit writes are explicitly best-effort by design (CLAUDE.md: "Audit-write
|
||||
// failure NEVER aborts the user-facing action — audit is best-effort, the
|
||||
// action's own success/failure path is authoritative") — this is the same
|
||||
// durability class as a dropped audit row, not a new risk class.
|
||||
using (var pragmaCmd = _connection.CreateCommand())
|
||||
{
|
||||
pragmaCmd.CommandText = "PRAGMA synchronous = NORMAL";
|
||||
pragmaCmd.ExecuteNonQuery();
|
||||
}
|
||||
|
||||
// Enable FK enforcement on the WRITE connection. PRAGMA foreign_keys is
|
||||
// a per-connection, per-session setting in SQLite — it is NOT persisted
|
||||
// in the database file, so every new connection that may INSERT into
|
||||
@@ -306,20 +324,67 @@ public class SqliteAuditWriter : IAuditWriter, ISiteAuditQueue, IAsyncDisposable
|
||||
{
|
||||
batch.Clear();
|
||||
batch.Add(first);
|
||||
var deadline = DateTime.UtcNow.AddMilliseconds(_options.FlushIntervalMs);
|
||||
|
||||
// Pull additional ready events up to BatchSize. TryRead is non-
|
||||
// blocking and lets us amortise the transaction overhead across a
|
||||
// burst of concurrent enqueues.
|
||||
while (batch.Count < _options.BatchSize &&
|
||||
_writeQueue.Reader.TryRead(out var next))
|
||||
// burst of concurrent enqueues. Once the immediately-available
|
||||
// burst is drained, keep the transaction open for up to
|
||||
// FlushIntervalMs waiting for MORE events to trickle in — this is
|
||||
// the soft-flush coalescing arch-review remediation WP1.2 adds:
|
||||
// without it, a low-rate trickle of writes (one script call every
|
||||
// few ms, well under BatchSize) fsyncs once PER EVENT, because the
|
||||
// writer loop reaches an empty channel and flushes immediately.
|
||||
// FlushIntervalMs bounds the worst-case added latency any single
|
||||
// event can see from this coalescing.
|
||||
while (batch.Count < _options.BatchSize)
|
||||
{
|
||||
batch.Add(next);
|
||||
if (_writeQueue.Reader.TryRead(out var next))
|
||||
{
|
||||
batch.Add(next);
|
||||
continue;
|
||||
}
|
||||
|
||||
var remaining = deadline - DateTime.UtcNow;
|
||||
if (remaining <= TimeSpan.Zero)
|
||||
{
|
||||
break;
|
||||
}
|
||||
|
||||
var waitToRead = _writeQueue.Reader.WaitToReadAsync().AsTask();
|
||||
var completed = await Task.WhenAny(
|
||||
waitToRead, Task.Delay(remaining)).ConfigureAwait(false);
|
||||
|
||||
if (completed != waitToRead)
|
||||
{
|
||||
// Deadline elapsed with nothing new — flush what we have.
|
||||
break;
|
||||
}
|
||||
|
||||
// WaitToReadAsync resolving false means the channel completed
|
||||
// (Dispose) with nothing left to read; resolving true means at
|
||||
// least one item is available, so the top of the loop's
|
||||
// TryRead will pick it up.
|
||||
if (!await waitToRead.ConfigureAwait(false))
|
||||
{
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
FlushBatch(batch);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Total number of <see cref="FlushBatch"/> transactions committed since
|
||||
/// construction. Test-only observability seam (arch-review remediation
|
||||
/// WP1.2) for asserting that FlushIntervalMs coalesces a trickle of
|
||||
/// near-simultaneous writes into a single transaction rather than one
|
||||
/// commit per event; production code never reads this.
|
||||
/// </summary>
|
||||
internal int FlushCountForTests => _flushCountForTests;
|
||||
private int _flushCountForTests;
|
||||
|
||||
private void FlushBatch(IReadOnlyList<PendingAuditEvent> batch)
|
||||
{
|
||||
lock (_writeLock)
|
||||
@@ -436,6 +501,7 @@ public class SqliteAuditWriter : IAuditWriter, ISiteAuditQueue, IAsyncDisposable
|
||||
}
|
||||
|
||||
transaction.Commit();
|
||||
_flushCountForTests++;
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
@@ -692,10 +758,17 @@ public class SqliteAuditWriter : IAuditWriter, ISiteAuditQueue, IAsyncDisposable
|
||||
/// </summary>
|
||||
/// <param name="sinceUtc">Lower bound timestamp (UTC) for event occurrence.</param>
|
||||
/// <param name="batchSize">Maximum number of rows to return.</param>
|
||||
/// <param name="afterId">
|
||||
/// Composite-keyset tiebreak: the EventId of the last row already consumed at
|
||||
/// <paramref name="sinceUtc"/>. Non-null switches the predicate from the inclusive
|
||||
/// <c>OccurredAtUtc >= $since</c> to the strict composite
|
||||
/// <c>(OccurredAtUtc, EventId) > ($since, $afterId)</c>, matching the query's own
|
||||
/// ORDER BY so a batch cannot stall on rows sharing one instant.
|
||||
/// </param>
|
||||
/// <param name="ct">Cancellation token.</param>
|
||||
/// <returns>A task that resolves to a read-only list of audit events since the given timestamp.</returns>
|
||||
public Task<IReadOnlyList<AuditEvent>> ReadPendingSinceAsync(
|
||||
DateTime sinceUtc, int batchSize, CancellationToken ct = default)
|
||||
DateTime sinceUtc, int batchSize, string? afterId = null, CancellationToken ct = default)
|
||||
{
|
||||
if (batchSize <= 0)
|
||||
{
|
||||
@@ -713,13 +786,24 @@ public class SqliteAuditWriter : IAuditWriter, ISiteAuditQueue, IAsyncDisposable
|
||||
ObjectDisposedException.ThrowIf(_disposed, this);
|
||||
|
||||
using var cmd = _readConnection.CreateCommand();
|
||||
cmd.CommandText = """
|
||||
var cursorPredicate = afterId is null
|
||||
? "fs.OccurredAtUtc >= $since"
|
||||
// Composite keyset, lexicographic on (OccurredAtUtc, EventId) — the same
|
||||
// ordering the query applies, so it is a strict "everything after this row".
|
||||
: "(fs.OccurredAtUtc > $since OR (fs.OccurredAtUtc = $since AND ae.EventId > $afterId))";
|
||||
|
||||
// fs.rowid rides along as an 11th column purely to maintain
|
||||
// _maxServedRowId — the insertion-order high-water mark that bounds
|
||||
// MarkReconciledUpToAsync (see that method). It is never mapped onto
|
||||
// the returned AuditEvent; rowid stays a storage-layer concept.
|
||||
cmd.CommandText = $"""
|
||||
SELECT ae.EventId, ae.OccurredAtUtc, ae.Actor, ae.Action, ae.Outcome,
|
||||
ae.Category, ae.Target, ae.SourceNode, ae.CorrelationId, ae.DetailsJson
|
||||
ae.Category, ae.Target, ae.SourceNode, ae.CorrelationId, ae.DetailsJson,
|
||||
fs.rowid
|
||||
FROM audit_event ae
|
||||
JOIN audit_forward_state fs ON fs.EventId = ae.EventId
|
||||
WHERE fs.ForwardState IN ($pending, $forwarded)
|
||||
AND fs.OccurredAtUtc >= $since
|
||||
AND {cursorPredicate}
|
||||
ORDER BY fs.OccurredAtUtc ASC, ae.EventId ASC
|
||||
LIMIT $limit;
|
||||
""";
|
||||
@@ -730,9 +814,121 @@ public class SqliteAuditWriter : IAuditWriter, ISiteAuditQueue, IAsyncDisposable
|
||||
// that encoding so we can index-scan against it.
|
||||
cmd.Parameters.AddWithValue("$since", EnsureUtc(sinceUtc).ToString(
|
||||
"o", System.Globalization.CultureInfo.InvariantCulture));
|
||||
if (afterId is not null)
|
||||
{
|
||||
// EventIds are stored as Guid.ToString() ("D"), so compare in that form.
|
||||
cmd.Parameters.AddWithValue("$afterId", NormalizeEventId(afterId));
|
||||
}
|
||||
cmd.Parameters.AddWithValue("$limit", batchSize);
|
||||
|
||||
return Task.FromResult(ReadRows(cmd, batchSize));
|
||||
return Task.FromResult(ReadServedRows(cmd, batchSize));
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Highest <c>audit_forward_state.rowid</c> this instance has ever SERVED from
|
||||
/// <see cref="ReadPendingSinceAsync"/> — i.e. the insertion-order high-water mark of
|
||||
/// rows central could possibly have received through the pull path. Bounds
|
||||
/// <see cref="MarkReconciledUpToAsync"/> so a row inserted after that point can never be
|
||||
/// retired by a cursor that happens to sit above its timestamp. Monotonic (a later, smaller
|
||||
/// batch never lowers it) except for the purge clamp. Written under <c>_readLock</c> (the
|
||||
/// read path) and read under <c>_writeLock</c> (the flip/purge paths) — two different
|
||||
/// locks, so the accesses go through <see cref="Volatile"/> for visibility. A stale read
|
||||
/// can only make the bound smaller, i.e. more conservative, never unsafe.
|
||||
/// </summary>
|
||||
private long _maxServedRowId;
|
||||
|
||||
/// <summary>
|
||||
/// Executes a reconciliation-pull read whose projection carries <c>fs.rowid</c> as an
|
||||
/// 11th column, materialising the canonical rows while advancing
|
||||
/// <see cref="_maxServedRowId"/>. Serving a row is what makes it eligible for
|
||||
/// cursor-proved retirement, so the two must happen in the same step.
|
||||
/// </summary>
|
||||
private IReadOnlyList<AuditEvent> ReadServedRows(SqliteCommand cmd, int capacityHint)
|
||||
{
|
||||
var rows = new List<AuditEvent>(Math.Min(capacityHint, 256));
|
||||
var maxRowId = Volatile.Read(ref _maxServedRowId);
|
||||
using (var reader = cmd.ExecuteReader())
|
||||
{
|
||||
while (reader.Read())
|
||||
{
|
||||
rows.Add(MapRow(reader));
|
||||
var rowId = reader.GetInt64(10);
|
||||
if (rowId > maxRowId) maxRowId = rowId;
|
||||
}
|
||||
}
|
||||
|
||||
Volatile.Write(ref _maxServedRowId, maxRowId);
|
||||
return rows;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Normalises a wire-supplied event id to the exact textual form stored in
|
||||
/// <c>audit_event.EventId</c> so the keyset comparison is apples-to-apples. An
|
||||
/// unparseable value is passed through verbatim rather than throwing — a malformed
|
||||
/// cursor must degrade to "serves a bit too much", never to a failed pull.
|
||||
/// </summary>
|
||||
private static string NormalizeEventId(string afterId) =>
|
||||
Guid.TryParse(afterId, out var g) ? g.ToString() : afterId;
|
||||
|
||||
/// <inheritdoc />
|
||||
public Task<int> MarkReconciledUpToAsync(
|
||||
DateTime sinceUtc, string? afterId, CancellationToken ct = default)
|
||||
{
|
||||
lock (_writeLock)
|
||||
{
|
||||
ObjectDisposedException.ThrowIf(_disposed, this);
|
||||
|
||||
// Everything at or before central's cursor is proven received. With no
|
||||
// afterId the read contract is inclusive (>= $since), so only rows STRICTLY
|
||||
// older than the cursor instant are proven — the boundary instant may be
|
||||
// half-consumed and must stay servable.
|
||||
var predicate = afterId is null
|
||||
? "fs.OccurredAtUtc < $since"
|
||||
: "(fs.OccurredAtUtc < $since OR (fs.OccurredAtUtc = $since AND fs.EventId <= $afterId))";
|
||||
|
||||
// INSERTION-ORDER BOUND — "only rows we actually served can retire".
|
||||
//
|
||||
// The timestamp cursor alone is not enough. OccurredAtUtc is stamped by the
|
||||
// caller, so a row can be INSERTED after a batch was served yet carry a
|
||||
// timestamp BELOW central's (by then advanced) cursor — a late-stamped insert.
|
||||
// The blanket cursor UPDATE retired exactly those rows: never served, never
|
||||
// servable again (ReadPendingSinceAsync's keyset has moved past them), and,
|
||||
// being Reconciled, purged on age — silent audit loss, a failure mode the old
|
||||
// explicit id-set flip could not produce.
|
||||
//
|
||||
// fs.rowid is insertion order, so bounding the flip at the highest rowid this
|
||||
// instance has ever served restores the invariant exactly: rows inserted after
|
||||
// that point are out of the flip's reach no matter where the cursor sits. A
|
||||
// FORWARDED row is exempt from the bound — central ACKED it through the
|
||||
// telemetry push path, which is receipt-proof independent of the pull.
|
||||
//
|
||||
// Bound state is per-process: after a restart it is 0 until this instance serves
|
||||
// a batch, so the first pull retires only Forwarded rows and the pull after it
|
||||
// resumes normal retirement. Conservative in the safe direction (delays
|
||||
// retirement, never loses a row); the liveness note in MarkReconciledUpToAsync's
|
||||
// interface doc + Component-AuditLog.md covers what a permanently halted
|
||||
// reconciliation means for such rows.
|
||||
using var cmd = _connection.CreateCommand();
|
||||
cmd.CommandText = $"""
|
||||
UPDATE audit_forward_state AS fs
|
||||
SET ForwardState = $reconciled
|
||||
WHERE fs.ForwardState IN ($pending, $forwarded)
|
||||
AND (fs.ForwardState = $forwarded OR fs.rowid <= $maxServedRowId)
|
||||
AND {predicate};
|
||||
""";
|
||||
cmd.Parameters.AddWithValue("$reconciled", AuditForwardState.Reconciled.ToString());
|
||||
cmd.Parameters.AddWithValue("$pending", AuditForwardState.Pending.ToString());
|
||||
cmd.Parameters.AddWithValue("$forwarded", AuditForwardState.Forwarded.ToString());
|
||||
cmd.Parameters.AddWithValue("$maxServedRowId", Volatile.Read(ref _maxServedRowId));
|
||||
cmd.Parameters.AddWithValue("$since", EnsureUtc(sinceUtc).ToString(
|
||||
"o", System.Globalization.CultureInfo.InvariantCulture));
|
||||
if (afterId is not null)
|
||||
{
|
||||
cmd.Parameters.AddWithValue("$afterId", NormalizeEventId(afterId));
|
||||
}
|
||||
|
||||
return Task.FromResult(cmd.ExecuteNonQuery());
|
||||
}
|
||||
}
|
||||
|
||||
@@ -933,6 +1129,22 @@ public class SqliteAuditWriter : IAuditWriter, ISiteAuditQueue, IAsyncDisposable
|
||||
dropCmd.ExecuteNonQuery();
|
||||
}
|
||||
|
||||
// Clamp the served-rows high-water mark to what is still in the table.
|
||||
// SQLite hands a fresh insert MAX(rowid)+1, so a purge that empties the
|
||||
// sidecar (or removes its top rows) makes rowids REUSABLE — a stale-high
|
||||
// _maxServedRowId would then vouch for rows nobody has served. Cheap:
|
||||
// MAX(rowid) is an index-less O(1) lookup.
|
||||
if (purged > 0)
|
||||
{
|
||||
using var maxCmd = _connection.CreateCommand();
|
||||
maxCmd.Transaction = transaction;
|
||||
maxCmd.CommandText = "SELECT IFNULL(MAX(rowid), 0) FROM audit_forward_state;";
|
||||
var liveMax = Convert.ToInt64(maxCmd.ExecuteScalar(),
|
||||
System.Globalization.CultureInfo.InvariantCulture);
|
||||
if (liveMax < Volatile.Read(ref _maxServedRowId))
|
||||
Volatile.Write(ref _maxServedRowId, liveMax);
|
||||
}
|
||||
|
||||
transaction.Commit();
|
||||
}
|
||||
catch
|
||||
|
||||
@@ -9,8 +9,23 @@ namespace ZB.MOM.WW.ScadaBridge.AuditLog.Site;
|
||||
/// </summary>
|
||||
public sealed class SqliteAuditWriterOptions
|
||||
{
|
||||
/// <summary>SQLite database path (or in-memory URI for tests).</summary>
|
||||
public string DatabasePath { get; set; } = "auditlog.db";
|
||||
/// <summary>
|
||||
/// SQLite database path (or in-memory URI for tests). No default —
|
||||
/// mirrors <c>LocalDb:Path</c> (<c>ZB.MOM.WW.LocalDb.LocalDbOptions.Path</c>): a
|
||||
/// bare filename here would resolve relative to the process CWD, which on the
|
||||
/// docker rig is the container's ephemeral overlayfs, not the mounted
|
||||
/// <c>/app/data</c> volume — the file (and its pending forward-state backlog)
|
||||
/// would be silently discarded on every container recreate. Site nodes MUST set
|
||||
/// this explicitly (e.g. <c>/app/data/auditlog.db</c>); <c>StartupValidator</c>
|
||||
/// in ZB.MOM.WW.ScadaBridge.Host enforces the requirement pre-host for Site nodes only,
|
||||
/// the same way it enforces <c>Communication:GrpcPsk</c>. Central composition roots also
|
||||
/// bind this options type (<c>AddAuditLog</c> is shared between roles) but never resolve
|
||||
/// <see cref="SqliteAuditWriter"/>, so the empty default is inert there and deliberately
|
||||
/// left unvalidated by <see cref="SqliteAuditWriterOptionsValidator"/> (which only checks
|
||||
/// the role-agnostic channel/batch/flush knobs) — enforcing it there too would fail
|
||||
/// central's boot via <c>ValidateOnStart</c>, since <c>AddAuditLog</c> runs on both roles.
|
||||
/// </summary>
|
||||
public string DatabasePath { get; set; } = "";
|
||||
|
||||
/// <summary>
|
||||
/// Capacity of the bounded write queue. Set high enough that ordinary
|
||||
|
||||
@@ -6,19 +6,25 @@ namespace ZB.MOM.WW.ScadaBridge.AuditLog.Site;
|
||||
/// Validates <see cref="SqliteAuditWriterOptions"/> at host startup
|
||||
/// (arch-review 08 round 2 NF4). The channel/batch/flush knobs feed the
|
||||
/// background writer task; a zero anywhere stalls it (nothing drains, nothing
|
||||
/// flushes), and an empty <c>DatabasePath</c> leaves the SQLite writer with no
|
||||
/// backing store. <see cref="SqliteAuditWriterOptions.BacklogPollIntervalSeconds"/>
|
||||
/// flushes). <see cref="SqliteAuditWriterOptions.BacklogPollIntervalSeconds"/>
|
||||
/// is intentionally NOT validated — a non-positive value has a documented
|
||||
/// fall-back-to-30s contract in <c>SiteAuditBacklogReporter</c>.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <see cref="SqliteAuditWriterOptions.DatabasePath"/> is deliberately NOT
|
||||
/// validated here (arch-review remediation WP1.2). <c>AddAuditLog</c> runs on
|
||||
/// BOTH Central and Site composition roots and binds this options type with
|
||||
/// <c>ValidateOnStart</c> either way, but only Site nodes ever resolve
|
||||
/// <see cref="SqliteAuditWriter"/> — requiring a non-empty path in this
|
||||
/// role-agnostic validator would fail Central's boot too. The requirement is
|
||||
/// instead enforced pre-host, Site-role-only, by <c>StartupValidator</c> in
|
||||
/// ZB.MOM.WW.ScadaBridge.Host (mirrors how it gates <c>Communication:GrpcPsk</c>).
|
||||
/// </remarks>
|
||||
public sealed class SqliteAuditWriterOptionsValidator : OptionsValidatorBase<SqliteAuditWriterOptions>
|
||||
{
|
||||
/// <inheritdoc />
|
||||
protected override void Validate(ValidationBuilder builder, SqliteAuditWriterOptions options)
|
||||
{
|
||||
builder.RequireThat(!string.IsNullOrWhiteSpace(options.DatabasePath),
|
||||
$"AuditLog:SiteWriter:{nameof(SqliteAuditWriterOptions.DatabasePath)} must be a non-empty path.");
|
||||
|
||||
builder.RequireThat(options.ChannelCapacity > 0,
|
||||
$"AuditLog:SiteWriter:{nameof(SqliteAuditWriterOptions.ChannelCapacity)} " +
|
||||
$"({options.ChannelCapacity}) must be > 0.");
|
||||
|
||||
@@ -19,6 +19,7 @@ public static class DeployCommands
|
||||
var command = new Command("deploy") { Description = "Deployment operations" };
|
||||
|
||||
command.Add(BuildInstance(urlOption, formatOption, usernameOption, passwordOption));
|
||||
command.Add(BuildSite(urlOption, formatOption, usernameOption, passwordOption));
|
||||
command.Add(BuildArtifacts(urlOption, formatOption, usernameOption, passwordOption));
|
||||
command.Add(BuildStatus(urlOption, formatOption, usernameOption, passwordOption));
|
||||
|
||||
@@ -39,6 +40,43 @@ public static class DeployCommands
|
||||
return cmd;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Builds <c>deploy site</c> — bulk-deploy every instance at one site.
|
||||
///
|
||||
/// <para>
|
||||
/// Sits under <c>deploy</c> alongside <c>deploy instance</c> and
|
||||
/// <c>deploy artifacts</c>, naming the SCOPE of the deploy as the verb, which
|
||||
/// is the convention the group already follows. Note it is deliberately NOT
|
||||
/// fleet-wide when <c>--site-id</c> is omitted (the pattern <c>deploy
|
||||
/// artifacts</c> uses): a bulk instance deploy is far more consequential than
|
||||
/// an artifact push, so the target site is required rather than defaulted.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
private static Command BuildSite(Option<string> urlOption, Option<string> formatOption, Option<string> usernameOption, Option<string> passwordOption)
|
||||
{
|
||||
var siteIdOption = new Option<int>("--site-id") { Description = "Target site ID", Required = true };
|
||||
var cmd = new Command("site") { Description = "Deploy every deployable instance at a site" };
|
||||
cmd.Add(siteIdOption);
|
||||
cmd.SetAction(async (ParseResult result) =>
|
||||
{
|
||||
var siteId = result.GetValue(siteIdOption);
|
||||
return await CommandHelpers.ExecuteCommandAsync(
|
||||
result, urlOption, formatOption, usernameOption, passwordOption,
|
||||
new MgmtDeploySiteCommand(siteId),
|
||||
// A bulk deploy is N instance round-trips; the default 30 s client
|
||||
// timeout would abandon the request while the server is still
|
||||
// applying. Matches the server-side long-running Ask window.
|
||||
timeout: BulkDeployTimeout);
|
||||
});
|
||||
return cmd;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Client-side timeout for the bulk site deploy, matching the management
|
||||
/// service's long-running command Ask window.
|
||||
/// </summary>
|
||||
internal static readonly TimeSpan BulkDeployTimeout = TimeSpan.FromMinutes(5);
|
||||
|
||||
private static Command BuildArtifacts(Option<string> urlOption, Option<string> formatOption, Option<string> usernameOption, Option<string> passwordOption)
|
||||
{
|
||||
var siteIdOption = new Option<int?>("--site-id") { Description = "Target site ID (all sites if omitted)" };
|
||||
|
||||
@@ -159,14 +159,20 @@ public static class TemplateCommands
|
||||
|
||||
private static Command BuildList(Option<string> urlOption, Option<string> formatOption, Option<string> usernameOption, Option<string> passwordOption)
|
||||
{
|
||||
// The server projects this listing in the database and returns
|
||||
// TemplateSummary rows (counts, no child collections), so --detail cannot
|
||||
// produce full definitions here — it only skips the CLI-side column
|
||||
// projection and renders the summary payload verbatim. Full definitions
|
||||
// come from `template get --id`.
|
||||
var detailOption = new Option<bool>("--detail")
|
||||
{
|
||||
Description = "Include full template definitions (all attributes/alarms/scripts) in table output. "
|
||||
+ "Without it, table output is a compact summary (counts only). JSON output is always full."
|
||||
Description = "Render the raw list payload in table output instead of the compact column projection. "
|
||||
+ "The server returns summary rows (member counts only) for a listing, so this does NOT include "
|
||||
+ "attribute/alarm/script definitions — use 'template get --id <int>' for those. No effect on JSON output."
|
||||
};
|
||||
var skipOption = new Option<int>("--skip") { Description = "Offset paging: number of items to skip (default 0)" };
|
||||
var takeOption = new Option<int?>("--take") { Description = "Offset paging: max items to return (1..1000; omit for unlimited)" };
|
||||
var cmd = new Command("list") { Description = "List all templates (compact table summary; use --detail for the full dump)" };
|
||||
var cmd = new Command("list") { Description = "List all templates (compact table summary with member counts; definitions via 'template get')" };
|
||||
cmd.Add(detailOption);
|
||||
cmd.Add(skipOption);
|
||||
cmd.Add(takeOption);
|
||||
|
||||
@@ -5,12 +5,30 @@ namespace ZB.MOM.WW.ScadaBridge.CLI.Commands;
|
||||
|
||||
/// <summary>
|
||||
/// Compact table projection for <c>template list</c> / <c>template get</c>.
|
||||
/// The management API returns full <c>Template</c> entities — every attribute, alarm,
|
||||
///
|
||||
/// <para>
|
||||
/// <c>template get</c> returns a full <c>Template</c> entity — every attribute, alarm,
|
||||
/// script, and composition inline — which the generic table renderer dumps as one giant
|
||||
/// cell per template (~171 KB for a real catalogue, unusable in a terminal). This
|
||||
/// projector reduces each template to id / name / description / parent / derived plus
|
||||
/// member <em>counts</em>, leaving JSON output untouched (callers pass this only on the
|
||||
/// table path) and the full dump available via the command's <c>--detail</c> flag.
|
||||
/// cell per template (~171 KB for a real catalogue, unusable in a terminal).
|
||||
/// <c>template list</c> no longer returns entities at all: the management
|
||||
/// <c>ListTemplates</c> handler projects in the DATABASE and returns
|
||||
/// <c>TemplateSummary</c> rows whose children are already reduced to
|
||||
/// <c>attributeCount</c>/<c>alarmCount</c>/… scalars.
|
||||
/// </para>
|
||||
///
|
||||
/// <para>
|
||||
/// This projector handles BOTH shapes: it prefers the pre-computed <c>*Count</c>
|
||||
/// scalar and falls back to the length of the matching child array, so the same
|
||||
/// column set renders for a summary row and for a full entity. Reading only the
|
||||
/// arrays (as it originally did) made <c>template list</c> print a wall of zeros
|
||||
/// once the server switched to summaries.
|
||||
/// </para>
|
||||
///
|
||||
/// <para>
|
||||
/// JSON output is left untouched (callers pass this only on the table path), and
|
||||
/// the command's <c>--detail</c> flag skips the projection to render the raw
|
||||
/// payload as-is.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
internal static class TemplateTableProjection
|
||||
{
|
||||
@@ -67,11 +85,11 @@ internal static class TemplateTableProjection
|
||||
["description"] = Str(element, "description"),
|
||||
["parentTemplateId"] = Int(element, "parentTemplateId"),
|
||||
["isDerived"] = Bool(element, "isDerived"),
|
||||
["#attrs"] = Count(element, "attributes"),
|
||||
["#alarms"] = Count(element, "alarms"),
|
||||
["#scripts"] = Count(element, "scripts"),
|
||||
["#comps"] = Count(element, "compositions"),
|
||||
["#nativeAlarms"] = Count(element, "nativeAlarmSources"),
|
||||
["#attrs"] = Count(element, "attributeCount", "attributes"),
|
||||
["#alarms"] = Count(element, "alarmCount", "alarms"),
|
||||
["#scripts"] = Count(element, "scriptCount", "scripts"),
|
||||
["#comps"] = Count(element, "compositionCount", "compositions"),
|
||||
["#nativeAlarms"] = Count(element, "nativeAlarmSourceCount", "nativeAlarmSources"),
|
||||
};
|
||||
}
|
||||
|
||||
@@ -104,9 +122,28 @@ internal static class TemplateTableProjection
|
||||
? JsonValue.Create(v.GetBoolean())
|
||||
: null;
|
||||
|
||||
private static JsonNode Count(JsonElement obj, string name)
|
||||
=> JsonValue.Create(
|
||||
TryGetPropertyCI(obj, name, out var v) && v.ValueKind == JsonValueKind.Array
|
||||
/// <summary>
|
||||
/// Member count for a column, preferring the summary payload's pre-computed
|
||||
/// scalar (<paramref name="countName"/>) and falling back to the length of the
|
||||
/// full entity's child array (<paramref name="arrayName"/>). Zero when neither
|
||||
/// is present.
|
||||
/// </summary>
|
||||
/// <param name="obj">The template object being projected.</param>
|
||||
/// <param name="countName">Scalar count property on a <c>TemplateSummary</c> row.</param>
|
||||
/// <param name="arrayName">Child-collection property on a full <c>Template</c> entity.</param>
|
||||
/// <returns>The member count as a JSON number.</returns>
|
||||
private static JsonNode Count(JsonElement obj, string countName, string arrayName)
|
||||
{
|
||||
if (TryGetPropertyCI(obj, countName, out var scalar)
|
||||
&& scalar.ValueKind == JsonValueKind.Number
|
||||
&& scalar.TryGetInt32(out var n))
|
||||
{
|
||||
return JsonValue.Create(n);
|
||||
}
|
||||
|
||||
return JsonValue.Create(
|
||||
TryGetPropertyCI(obj, arrayName, out var v) && v.ValueKind == JsonValueKind.Array
|
||||
? v.GetArrayLength()
|
||||
: 0);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -9,13 +9,79 @@ public class ManagementHttpClient : IDisposable
|
||||
private readonly HttpClient _httpClient;
|
||||
|
||||
/// <summary>
|
||||
/// Initializes a new instance of the <see cref="ManagementHttpClient"/> class.
|
||||
/// Default bound on the CONNECT phase only (30 s) — how long a TCP/TLS
|
||||
/// connection attempt to a black-holed management endpoint may hang before the
|
||||
/// call fails.
|
||||
///
|
||||
/// <para>
|
||||
/// It is deliberately NOT an overall request timeout.
|
||||
/// <see cref="HttpClient.Timeout"/> caps the whole request/response, so setting
|
||||
/// it to any fixed value silently truncates every caller whose own per-call
|
||||
/// <c>TimeSpan timeout</c> argument is longer: the effective deadline becomes
|
||||
/// <c>min(HttpClient.Timeout, caller timeout)</c>. That is exactly what a 30 s
|
||||
/// client timeout did to <c>deploy site</c>'s 5-minute bulk deploy and to the
|
||||
/// five-minute <c>bundle</c> export/preview/import calls — each printed a fake
|
||||
/// <c>504 Request timed out</c> at 30 s while the server carried on working.
|
||||
/// <see cref="HttpClient.Timeout"/> is therefore
|
||||
/// <see cref="Timeout.InfiniteTimeSpan"/> and the per-call
|
||||
/// <see cref="CancellationTokenSource"/> in
|
||||
/// <see cref="SendCommandAsync"/>/<see cref="SendGetAsync"/>/<see cref="SendPostAsync"/>
|
||||
/// is the SINGLE overall deadline — it bounds connect too, since the token is
|
||||
/// passed into the send itself.
|
||||
/// </para>
|
||||
///
|
||||
/// <para>
|
||||
/// The connect bound lives on <see cref="SocketsHttpHandler.ConnectTimeout"/>
|
||||
/// instead, which is connect-scoped and so cannot truncate a long-running
|
||||
/// request that has already reached the server. Overridable via the
|
||||
/// <c>SCADABRIDGE_HTTP_CONNECT_TIMEOUT_SECONDS</c> environment variable,
|
||||
/// consistent with how every other CLI setting is overridden (see
|
||||
/// <see cref="CliConfig"/>) — kept self-contained here (no
|
||||
/// <see cref="CliConfig"/>/command-file plumbing) since CLI commands are owned
|
||||
/// by a separate work package this phase.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public static readonly TimeSpan DefaultConnectTimeout = TimeSpan.FromSeconds(30);
|
||||
|
||||
/// <summary>Test seam — the effective <see cref="HttpClient.Timeout"/> this instance was constructed with.</summary>
|
||||
internal TimeSpan EffectiveTimeout { get; }
|
||||
|
||||
/// <summary>
|
||||
/// Resolves the effective connect timeout: the
|
||||
/// <c>SCADABRIDGE_HTTP_CONNECT_TIMEOUT_SECONDS</c> environment variable when set
|
||||
/// to a positive integer, otherwise <see cref="DefaultConnectTimeout"/>.
|
||||
/// </summary>
|
||||
/// <returns>The connect timeout to apply to the socket handler.</returns>
|
||||
internal static TimeSpan ResolveConnectTimeout()
|
||||
{
|
||||
var env = Environment.GetEnvironmentVariable("SCADABRIDGE_HTTP_CONNECT_TIMEOUT_SECONDS");
|
||||
if (!string.IsNullOrWhiteSpace(env)
|
||||
&& int.TryParse(env, out var seconds)
|
||||
&& seconds > 0)
|
||||
{
|
||||
return TimeSpan.FromSeconds(seconds);
|
||||
}
|
||||
|
||||
return DefaultConnectTimeout;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Initializes a new instance of the <see cref="ManagementHttpClient"/> class with
|
||||
/// an INFINITE <see cref="HttpClient.Timeout"/> (each call supplies its own
|
||||
/// deadline) over a <see cref="SocketsHttpHandler"/> whose
|
||||
/// <see cref="SocketsHttpHandler.ConnectTimeout"/> is
|
||||
/// <see cref="ResolveConnectTimeout"/>.
|
||||
/// </summary>
|
||||
/// <param name="baseUrl">The base URL for the management API.</param>
|
||||
/// <param name="username">The username for HTTP Basic authentication.</param>
|
||||
/// <param name="password">The password for HTTP Basic authentication.</param>
|
||||
public ManagementHttpClient(string baseUrl, string username, string password)
|
||||
: this(new HttpClient(), baseUrl, username, password)
|
||||
: this(
|
||||
new HttpClient(new SocketsHttpHandler { ConnectTimeout = ResolveConnectTimeout() })
|
||||
{
|
||||
Timeout = Timeout.InfiniteTimeSpan
|
||||
},
|
||||
baseUrl, username, password)
|
||||
{
|
||||
}
|
||||
|
||||
@@ -31,6 +97,9 @@ public class ManagementHttpClient : IDisposable
|
||||
internal ManagementHttpClient(HttpClient httpClient, string baseUrl, string username, string password)
|
||||
{
|
||||
_httpClient = httpClient;
|
||||
// Test seam (WP2.6e): exposes the constructed HttpClient's effective Timeout
|
||||
// without requiring reflection.
|
||||
EffectiveTimeout = httpClient.Timeout;
|
||||
_httpClient.BaseAddress = new Uri(baseUrl.TrimEnd('/') + "/");
|
||||
var credentials = Convert.ToBase64String(Encoding.UTF8.GetBytes($"{username}:{password}"));
|
||||
_httpClient.DefaultRequestHeaders.Authorization =
|
||||
|
||||
@@ -86,28 +86,33 @@ Exit codes:
|
||||
|
||||
#### `template list`
|
||||
|
||||
List all templates. **Table** output (`--format table`) shows a compact summary — id,
|
||||
name, description, parent, and member **counts** (`#attrs`, `#alarms`, `#scripts`,
|
||||
`#comps`, `#nativeAlarms`) — so it stays readable in a terminal. Add `--detail` to dump
|
||||
the full attribute/alarm/script/composition definitions in the table. **JSON** output
|
||||
(`--format json`) is always the full, unmodified payload regardless of `--detail`.
|
||||
List all templates. The server projects this listing in the database and returns
|
||||
**summary rows** — no child collections, member counts only. **Table** output
|
||||
(`--format table`) renders them as id, name, description, parent, and member **counts**
|
||||
(`#attrs`, `#alarms`, `#scripts`, `#comps`, `#nativeAlarms`), so it stays readable in a
|
||||
terminal. `--detail` skips that column projection and renders the summary payload
|
||||
verbatim; it does **not** add attribute/alarm/script definitions — a listing never
|
||||
carries them. Use `template get --id <int>` for a template's full definition. **JSON**
|
||||
output (`--format json`) is always the unmodified payload regardless of `--detail`.
|
||||
|
||||
```sh
|
||||
scadabridge --url <url> template list # compact table
|
||||
scadabridge --url <url> --format table template list --detail # full table dump
|
||||
scadabridge --url <url> --format json template list # full JSON (always)
|
||||
scadabridge --url <url> template list # compact table
|
||||
scadabridge --url <url> --format table template list --detail # raw summary rows
|
||||
scadabridge --url <url> --format json template list # unmodified JSON (always)
|
||||
```
|
||||
|
||||
| Option | Required | Description |
|
||||
|--------|----------|-------------|
|
||||
| `--detail` | no | Include full definitions in table output (no effect on JSON) |
|
||||
| `--detail` | no | Render the raw summary payload instead of the compact column projection (no effect on JSON; does not add definitions) |
|
||||
| `--skip` | no | Offset paging: number of items to skip (default 0) |
|
||||
| `--take` | no | Offset paging: max items to return (clamped 1..1000; omit for unlimited) |
|
||||
|
||||
#### `template get`
|
||||
|
||||
Get a single template by ID. Like `template list`, table output is a compact summary
|
||||
unless `--detail` is supplied; JSON output is always full.
|
||||
Get a single template by ID. Unlike `template list`, this returns the **full** template
|
||||
entity with every attribute, alarm, script and composition inline. Table output is a
|
||||
compact summary (the same counts as `template list`) unless `--detail` is supplied, which
|
||||
dumps the full definitions; JSON output is always full.
|
||||
|
||||
```sh
|
||||
scadabridge --url <url> template get --id <int> [--detail]
|
||||
@@ -952,6 +957,37 @@ scadabridge --url <url> deploy instance --id <int>
|
||||
|--------|----------|-------------|
|
||||
| `--id` | yes | Instance ID |
|
||||
|
||||
#### `deploy site`
|
||||
|
||||
Deploy **every deployable instance** at one site in a single batched operation.
|
||||
|
||||
Each instance keeps the full single-instance semantics — its own deployment ID and
|
||||
revision hash, its own per-instance operation lock, its own optimistically-concurrent
|
||||
status record, and the same query-before-redeploy idempotency check. The batch is
|
||||
**not** all-or-nothing: an instance that cannot be deployed (wrong state, failed
|
||||
validation, lock already held, site round-trip timed out) is reported as a failed row
|
||||
and the rest proceed. Each failed instance is individually retryable with
|
||||
`instance deploy --id`.
|
||||
|
||||
Site round-trips run with bounded concurrency
|
||||
(`ScadaBridge:DeploymentManager:SiteDeploymentMaxParallelism`, default 4), each under
|
||||
`SiteDeploymentTimeoutPerInstance` (default 120 s), so one wedged instance cannot stall
|
||||
the batch. The client waits up to 5 minutes.
|
||||
|
||||
`--site-id` is **required** — unlike `deploy artifacts`, omitting it does not mean
|
||||
fleet-wide.
|
||||
|
||||
```sh
|
||||
scadabridge --url <url> deploy site --site-id <int>
|
||||
```
|
||||
|
||||
| Option | Required | Description |
|
||||
|--------|----------|-------------|
|
||||
| `--site-id` | yes | Target site ID. A site-scoped (non-Administrator) Deployer must supply an in-scope site. |
|
||||
|
||||
Output is a per-instance result matrix (`instanceId`, `uniqueName`, `deploymentId`,
|
||||
`success`, `errorMessage`) plus `successCount` / `failureCount`.
|
||||
|
||||
#### `deploy artifacts`
|
||||
|
||||
Deploy compiled artifacts to one or all sites (same as `site deploy-artifacts`).
|
||||
|
||||
@@ -291,13 +291,56 @@
|
||||
.OrderBy(a => a.AttributeName)
|
||||
.ToList();
|
||||
|
||||
/// <summary>Attribute composition forest, rebuilt from the live latest-per-name dictionary.</summary>
|
||||
private IReadOnlyList<DebugTreeNode> AttributeForest =>
|
||||
DebugTreeBuilder.BuildAttributeTree(_attributeValues.Values, _attrFilter);
|
||||
// ── Tree memoization (arch-review WP2.4) ──────────────────────────────────
|
||||
// Both forests used to be rebuilt from scratch on EVERY render — and a render
|
||||
// fired per streamed event. A busy instance therefore rebuilt two whole
|
||||
// composition trees hundreds of times a second, on the circuit's single
|
||||
// dispatcher thread. The rebuild now sits behind a version stamp bumped
|
||||
// whenever the underlying dictionaries change (a coalesced stream flush,
|
||||
// Connect, Disconnect), so an unrelated re-render (tab switch, toast, status
|
||||
// strip tick) reuses the forest instead of rebuilding it. Reusing the same
|
||||
// instance also lets Blazor skip re-rendering the TreeView subtree entirely.
|
||||
//
|
||||
// The cached key includes the dictionary Count as well as the version: the
|
||||
// trees must never go stale if the dictionaries are populated by some path
|
||||
// that forgets to bump the stamp.
|
||||
private int _dataVersion;
|
||||
|
||||
/// <summary>Alarm composition forest, rebuilt from the live latest-per-name dictionary.</summary>
|
||||
private IReadOnlyList<DebugTreeNode> AlarmForest =>
|
||||
DebugTreeBuilder.BuildAlarmTree(_alarmStates.Values, _alarmFilter);
|
||||
private IReadOnlyList<DebugTreeNode>? _attrForestCache;
|
||||
private (int Version, int Count, string Filter) _attrForestKey = (-1, -1, string.Empty);
|
||||
|
||||
private IReadOnlyList<DebugTreeNode>? _alarmForestCache;
|
||||
private (int Version, int Count, string Filter) _alarmForestKey = (-1, -1, string.Empty);
|
||||
|
||||
/// <summary>Attribute composition forest, rebuilt only when the underlying data or filter changed.</summary>
|
||||
private IReadOnlyList<DebugTreeNode> AttributeForest
|
||||
{
|
||||
get
|
||||
{
|
||||
var key = (_dataVersion, _attributeValues.Count, _attrFilter);
|
||||
if (_attrForestCache is null || _attrForestKey != key)
|
||||
{
|
||||
_attrForestCache = DebugTreeBuilder.BuildAttributeTree(_attributeValues.Values, _attrFilter);
|
||||
_attrForestKey = key;
|
||||
}
|
||||
return _attrForestCache;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Alarm composition forest, rebuilt only when the underlying data or filter changed.</summary>
|
||||
private IReadOnlyList<DebugTreeNode> AlarmForest
|
||||
{
|
||||
get
|
||||
{
|
||||
var key = (_dataVersion, _alarmStates.Count, _alarmFilter);
|
||||
if (_alarmForestCache is null || _alarmForestKey != key)
|
||||
{
|
||||
_alarmForestCache = DebugTreeBuilder.BuildAlarmTree(_alarmStates.Values, _alarmFilter);
|
||||
_alarmForestKey = key;
|
||||
}
|
||||
return _alarmForestCache;
|
||||
}
|
||||
}
|
||||
|
||||
private DebugStreamSession? _session;
|
||||
private ToastNotification _toast = default!;
|
||||
@@ -458,6 +501,9 @@
|
||||
foreach (var al in session.InitialSnapshot.AlarmStates)
|
||||
_alarmStates[al.AlarmName] = al;
|
||||
|
||||
// Snapshot seeding is a data change like any other — bump the stamp so
|
||||
// the memoized forests rebuild from it.
|
||||
_dataVersion++;
|
||||
_snapshot = session.InitialSnapshot;
|
||||
_connected = true;
|
||||
|
||||
@@ -492,6 +538,11 @@
|
||||
_snapshot = null;
|
||||
_attributeValues.Clear();
|
||||
_alarmStates.Clear();
|
||||
// Drop anything the stream buffered for the session we just tore down, so a
|
||||
// late flush can't repopulate a disconnected view.
|
||||
_pendingAttributes.Clear();
|
||||
_pendingAlarms.Clear();
|
||||
_dataVersion++;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
@@ -508,37 +559,108 @@
|
||||
_toast.ShowInfo("Cleared previous session — select a site and instance to begin.", autoDismissMs: 5000);
|
||||
}
|
||||
|
||||
// ── Render coalescing (arch-review WP2.4) ─────────────────────────────────
|
||||
// Every streamed event used to marshal onto the dispatcher and call
|
||||
// StateHasChanged individually, so a chatty instance drove one full render
|
||||
// (and two full tree rebuilds) per value change. Events now land in a
|
||||
// thread-safe pending map keyed by name — repeated updates to the same tag
|
||||
// inside one window collapse to the latest — and exactly ONE dispatcher
|
||||
// marshal per window drains them, bumps the version stamp and renders once.
|
||||
private const int RenderCoalesceMs = 250;
|
||||
|
||||
private readonly System.Collections.Concurrent.ConcurrentDictionary<string, AttributeValueChanged> _pendingAttributes = new();
|
||||
private readonly System.Collections.Concurrent.ConcurrentDictionary<string, AlarmStateChanged> _pendingAlarms = new();
|
||||
|
||||
/// <summary>0/1 flag: whether a coalesce window is already armed. Interlocked-guarded.</summary>
|
||||
private int _flushArmed;
|
||||
|
||||
/// <summary>
|
||||
/// Handles one debug-stream event. The callback is invoked on an Akka/gRPC
|
||||
/// thread, but <see cref="_attributeValues"/>/<see cref="_alarmStates"/> are
|
||||
/// <see cref="Dictionary{TKey,TValue}"/> instances also enumerated by the
|
||||
/// render thread (the tree forests + <see cref="FilteredAttributeValues"/> are
|
||||
/// built from them). <c>Dictionary</c> is not thread-safe (CentralUI-021): a
|
||||
/// write racing an enumeration can throw or corrupt the buckets. The mutation
|
||||
/// is therefore marshalled onto the renderer's dispatcher via
|
||||
/// <see cref="SafeInvokeAsync"/> so every access to the dictionaries — read
|
||||
/// and write — happens on the render thread.
|
||||
/// write racing an enumeration can throw or corrupt the buckets. The event is
|
||||
/// therefore parked in a concurrent pending map here and applied on the
|
||||
/// renderer's dispatcher by <see cref="FlushPendingAsync"/>, so every access to
|
||||
/// the render dictionaries — read and write — still happens on one thread.
|
||||
/// </summary>
|
||||
private void HandleStreamEvent(object evt)
|
||||
{
|
||||
// CentralUI-009: the component may have been disposed while this event
|
||||
// was in flight on the Akka/gRPC thread.
|
||||
if (_disposed) return;
|
||||
_ = SafeInvokeAsync(() =>
|
||||
|
||||
switch (evt)
|
||||
{
|
||||
case AttributeValueChanged av:
|
||||
_pendingAttributes[av.AttributeName] = av;
|
||||
break;
|
||||
case AlarmStateChanged al:
|
||||
_pendingAlarms[al.AlarmName] = al;
|
||||
break;
|
||||
default:
|
||||
// Unknown event type — nothing to apply, no render needed.
|
||||
return;
|
||||
}
|
||||
|
||||
// Arm the window if it isn't already. Losers of the CAS have nothing to do:
|
||||
// their event is in the pending map and the armed flush will pick it up.
|
||||
if (Interlocked.CompareExchange(ref _flushArmed, 1, 0) == 0)
|
||||
{
|
||||
_ = FlushPendingAsync();
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Waits out the coalesce window, then applies everything buffered in one
|
||||
/// dispatcher pass and renders once.
|
||||
/// </summary>
|
||||
private async Task FlushPendingAsync()
|
||||
{
|
||||
try
|
||||
{
|
||||
await Task.Delay(RenderCoalesceMs);
|
||||
}
|
||||
catch
|
||||
{
|
||||
// Delay never faults in practice; if it ever did, fall through and drain.
|
||||
}
|
||||
|
||||
if (_disposed)
|
||||
{
|
||||
Volatile.Write(ref _flushArmed, 0);
|
||||
return;
|
||||
}
|
||||
|
||||
await SafeInvokeAsync(() =>
|
||||
{
|
||||
// Disarm BEFORE draining: an event arriving mid-drain then arms a fresh
|
||||
// window rather than being stranded until the next unrelated event.
|
||||
Volatile.Write(ref _flushArmed, 0);
|
||||
if (_disposed) return;
|
||||
switch (evt)
|
||||
|
||||
var applied = false;
|
||||
foreach (var key in _pendingAttributes.Keys)
|
||||
{
|
||||
case AttributeValueChanged av:
|
||||
if (_pendingAttributes.TryRemove(key, out var av))
|
||||
{
|
||||
_attributeValues[av.AttributeName] = av;
|
||||
break;
|
||||
case AlarmStateChanged al:
|
||||
_alarmStates[al.AlarmName] = al;
|
||||
break;
|
||||
default:
|
||||
// Unknown event type — no re-render needed.
|
||||
return;
|
||||
applied = true;
|
||||
}
|
||||
}
|
||||
foreach (var key in _pendingAlarms.Keys)
|
||||
{
|
||||
if (_pendingAlarms.TryRemove(key, out var al))
|
||||
{
|
||||
_alarmStates[al.AlarmName] = al;
|
||||
applied = true;
|
||||
}
|
||||
}
|
||||
|
||||
if (!applied) return;
|
||||
|
||||
_dataVersion++;
|
||||
StateHasChanged();
|
||||
});
|
||||
}
|
||||
|
||||
@@ -236,13 +236,66 @@
|
||||
DeploymentStatusNotifier.StatusChanged += OnDeploymentStatusChanged;
|
||||
}
|
||||
|
||||
// ── Push coalescing (arch-review WP2.4) ───────────────────────────────────
|
||||
// One reload = every deployment record + every instance, re-read and re-filtered.
|
||||
// The notifier fires per status WRITE, so a site deploy of N instances produced
|
||||
// 2N+ of those full reloads back to back, per circuit. The reload is now
|
||||
// leading-edge debounced: the first push after an idle gap still reloads
|
||||
// immediately (a single deployment stays as responsive as before), and every
|
||||
// push inside the window is absorbed into ONE trailing reload.
|
||||
private const int ReloadDebounceMs = 500;
|
||||
|
||||
private DateTimeOffset _lastReloadAt = DateTimeOffset.MinValue;
|
||||
|
||||
/// <summary>Coalescing timer for pushes arriving inside the debounce window. Disposed with the component.</summary>
|
||||
private Timer? _coalesceTimer;
|
||||
|
||||
private void OnDeploymentStatusChanged(ZB.MOM.WW.ScadaBridge.DeploymentManager.DeploymentStatusChange change)
|
||||
{
|
||||
// CentralUI-022: a callback racing disposal must not touch the component.
|
||||
if (_disposed || !_autoRefresh) return;
|
||||
|
||||
lock (_coalesceLock)
|
||||
{
|
||||
if (_disposed) return;
|
||||
|
||||
var sinceLast = DateTimeOffset.UtcNow - _lastReloadAt;
|
||||
if (sinceLast >= TimeSpan.FromMilliseconds(ReloadDebounceMs))
|
||||
{
|
||||
// Idle — reload straight away (leading edge).
|
||||
_lastReloadAt = DateTimeOffset.UtcNow;
|
||||
}
|
||||
else
|
||||
{
|
||||
// Inside the window: arm one trailing reload for the remainder, and
|
||||
// let any further push in this window ride it.
|
||||
if (_coalesceTimer is not null) return;
|
||||
|
||||
var delay = TimeSpan.FromMilliseconds(ReloadDebounceMs) - sinceLast;
|
||||
_coalesceTimer = new Timer(_ => OnCoalesceElapsed(), null, delay, Timeout.InfiniteTimeSpan);
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
_ = DispatchReloadAsync();
|
||||
}
|
||||
|
||||
/// <summary>Trailing edge of the debounce: disarm, stamp, and run the one coalesced reload.</summary>
|
||||
private void OnCoalesceElapsed()
|
||||
{
|
||||
lock (_coalesceLock)
|
||||
{
|
||||
_coalesceTimer?.Dispose();
|
||||
_coalesceTimer = null;
|
||||
if (_disposed) return;
|
||||
_lastReloadAt = DateTimeOffset.UtcNow;
|
||||
}
|
||||
|
||||
_ = DispatchReloadAsync();
|
||||
}
|
||||
|
||||
private readonly object _coalesceLock = new();
|
||||
|
||||
/// <summary>
|
||||
/// Reloads the deployment table on the renderer's dispatcher, guarded
|
||||
/// against the component being disposed mid-flight (CentralUI-022):
|
||||
@@ -361,5 +414,10 @@
|
||||
// status change reaches this disposed component.
|
||||
_disposed = true;
|
||||
DeploymentStatusNotifier.StatusChanged -= OnDeploymentStatusChanged;
|
||||
lock (_coalesceLock)
|
||||
{
|
||||
_coalesceTimer?.Dispose();
|
||||
_coalesceTimer = null;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
+39
-10
@@ -1,5 +1,6 @@
|
||||
@page "/monitoring/alarms"
|
||||
@attribute [Authorize(Policy = ZB.MOM.WW.ScadaBridge.Security.AuthorizationPolicies.RequireDeployment)]
|
||||
@using Microsoft.AspNetCore.Components.Web.Virtualization
|
||||
@using ZB.MOM.WW.ScadaBridge.CentralUI.Components.Shared
|
||||
@using ZB.MOM.WW.ScadaBridge.CentralUI.Services
|
||||
@using ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites
|
||||
@@ -196,15 +197,25 @@
|
||||
@onkeydown='e => OnHeaderKeyDown(e, "severity")' @onkeydown:preventDefault="_preventHeaderDefault">Severity @SortGlyph("severity")</th>
|
||||
</tr>
|
||||
</thead>
|
||||
@* Row markup is identical on both paths — the same <tr>, the same cells,
|
||||
the same data-test hook. Only WHO enumerates it differs: a plain
|
||||
foreach for the ordinary case, and Virtualize once the flat table
|
||||
gets long enough that rendering every row per delta dominates the
|
||||
circuit (arch-review WP2.4). Virtualize needs <tr> spacers or the
|
||||
browser hoists its <div>s out of the table and the layout collapses. *@
|
||||
<tbody>
|
||||
@foreach (var row in _visibleRows)
|
||||
@if (_visibleRows.Count <= VirtualizeThreshold)
|
||||
{
|
||||
<tr data-test="alarm-summary-row">
|
||||
<td class="font-monospace small">@row.InstanceUniqueName</td>
|
||||
<td>@row.Alarm.AlarmName</td>
|
||||
<td><AlarmStateBadges Alarm="row.Alarm" /></td>
|
||||
<td class="text-end font-monospace">@row.Alarm.Condition.Severity</td>
|
||||
</tr>
|
||||
@foreach (var row in _visibleRows)
|
||||
{
|
||||
@AlarmRow(row)
|
||||
}
|
||||
}
|
||||
else
|
||||
{
|
||||
<Virtualize Items="_visibleRows" Context="row" ItemSize="37" SpacerElement="tr">
|
||||
@AlarmRow(row)
|
||||
</Virtualize>
|
||||
}
|
||||
</tbody>
|
||||
</table>
|
||||
@@ -222,7 +233,23 @@
|
||||
// P4 (arch-review): the filtered+sorted view is memoized rather than recomputed
|
||||
// on every render. RecomputeVisibleRows() refreshes it whenever the inputs change
|
||||
// (a fresh snapshot in RefreshAsync, a filter change via @bind:after, or a sort).
|
||||
private IReadOnlyList<AlarmSummaryRow> _visibleRows = Array.Empty<AlarmSummaryRow>();
|
||||
// Concrete List because Virtualize binds ICollection<T>, not IReadOnlyList<T>.
|
||||
private List<AlarmSummaryRow> _visibleRows = new();
|
||||
|
||||
// Above this many visible rows the flat table switches to Virtualize. Below it the
|
||||
// plain foreach is cheaper than the virtualization machinery (and needs no JS), and
|
||||
// an operator filtered down to a handful of alarms should never pay for either.
|
||||
private const int VirtualizeThreshold = 150;
|
||||
|
||||
// One row template, shared by the plain and virtualized paths so the two can never
|
||||
// drift visually.
|
||||
private RenderFragment<AlarmSummaryRow> AlarmRow => row =>
|
||||
@<tr data-test="alarm-summary-row">
|
||||
<td class="font-monospace small">@row.InstanceUniqueName</td>
|
||||
<td>@row.Alarm.AlarmName</td>
|
||||
<td><AlarmStateBadges Alarm="row.Alarm" /></td>
|
||||
<td class="text-end font-monospace">@row.Alarm.Condition.Severity</td>
|
||||
</tr>;
|
||||
|
||||
private IReadOnlyList<string> _notReporting = Array.Empty<string>();
|
||||
private AlarmRollup _rollup = new(0, 0, 0, new Dictionary<AlarmKind, int>());
|
||||
@@ -304,8 +331,10 @@
|
||||
return;
|
||||
}
|
||||
|
||||
// _notReporting is the poll's unique authority — the alarm-only live cache
|
||||
// cannot compute it — so it is always refreshed.
|
||||
// _notReporting always comes from this call, live or not. While the cache is
|
||||
// serving the site the façade sources it from the aggregator's own fan-out (no
|
||||
// second fan-out is run); while it is cold the poll computes it. Either way the
|
||||
// shape and ordering are identical.
|
||||
_notReporting = result.NotReportingInstances;
|
||||
|
||||
// While the cache is live, the live deltas own the row set: a poll whose fan-out
|
||||
|
||||
@@ -15,8 +15,7 @@
|
||||
@implements IDisposable
|
||||
@inject ICentralHealthAggregator HealthAggregator
|
||||
@inject ISiteRepository SiteRepository
|
||||
@inject CommunicationService CommunicationService
|
||||
@inject IAuditLogQueryService AuditLogQueryService
|
||||
@inject IKpiSnapshotCache KpiSnapshots
|
||||
@inject IKpiHistoryQueryService KpiHistory
|
||||
@inject Microsoft.Extensions.Options.IOptions<HealthMonitoringOptions> HealthOptions
|
||||
|
||||
@@ -640,7 +639,9 @@
|
||||
// Non-fatal — fall back to showing siteId only
|
||||
}
|
||||
|
||||
await RefreshNow();
|
||||
// First load rides whatever the shared KPI cache already has fresh — a second
|
||||
// operator opening the dashboard inside the window costs no extra query.
|
||||
await RefreshAllAsync(forceRefresh: false);
|
||||
|
||||
// Site Health Trends (M6) load on their own path — never from the
|
||||
// timer tick below — so a trend-query fault can't disturb the live tile
|
||||
@@ -655,7 +656,7 @@
|
||||
{
|
||||
try
|
||||
{
|
||||
await RefreshNow();
|
||||
await RefreshAllAsync(forceRefresh: false);
|
||||
StateHasChanged();
|
||||
}
|
||||
finally
|
||||
@@ -769,20 +770,38 @@
|
||||
? "Central Cluster"
|
||||
: $"{GetSiteName(siteKey)} ({siteKey})";
|
||||
|
||||
private async Task RefreshNow()
|
||||
// Per-tile budget for one refresh tick. The underlying KPI Asks carry the
|
||||
// 30s CommunicationOptions.QueryTimeout, so without this a single hung
|
||||
// singleton stalled the WHOLE tick (and, with a 10s timer, stacked ticks)
|
||||
// rather than degrading its own tile. Cancelling here abandons only this
|
||||
// circuit's wait — the shared KPI flight continues and its result lands in
|
||||
// the process-level cache for the next tick.
|
||||
private static readonly TimeSpan TileTimeout = TimeSpan.FromSeconds(5);
|
||||
|
||||
// The operator's explicit "Refresh Now" bypasses the shared cache's freshness
|
||||
// window; the timer tick and the first load ride whatever is already fresh
|
||||
// (which is the whole point — N dashboards share one query per window).
|
||||
private Task RefreshNow() => RefreshAllAsync(forceRefresh: true);
|
||||
|
||||
// Every tile loads concurrently, each with its own short budget, so one slow
|
||||
// KPI degrades one tile instead of the tick. Every loader swallows its own
|
||||
// faults, so Task.WhenAll here can never observe an exception.
|
||||
private async Task RefreshAllAsync(bool forceRefresh)
|
||||
{
|
||||
_siteStates = HealthAggregator.GetAllSiteStates();
|
||||
await LoadOutboxKpis();
|
||||
await Task.WhenAll(LoadSiteCallKpis(), LoadSiteCallNodeKpis());
|
||||
await LoadAuditKpis();
|
||||
await Task.WhenAll(
|
||||
LoadOutboxKpis(forceRefresh),
|
||||
LoadSiteCallKpis(forceRefresh),
|
||||
LoadSiteCallNodeKpis(forceRefresh),
|
||||
LoadAuditKpis(forceRefresh));
|
||||
}
|
||||
|
||||
private async Task LoadOutboxKpis()
|
||||
private async Task LoadOutboxKpis(bool forceRefresh)
|
||||
{
|
||||
try
|
||||
{
|
||||
var response = await CommunicationService.GetNotificationKpisAsync(
|
||||
new NotificationKpiRequest(Guid.NewGuid().ToString("N")));
|
||||
using var timeout = new CancellationTokenSource(TileTimeout);
|
||||
var response = await KpiSnapshots.GetNotificationKpisAsync(forceRefresh, timeout.Token);
|
||||
if (response.Success)
|
||||
{
|
||||
_outboxKpi = response;
|
||||
@@ -795,6 +814,11 @@
|
||||
_outboxKpiError = response.ErrorMessage ?? "KPI query failed.";
|
||||
}
|
||||
}
|
||||
catch (OperationCanceledException)
|
||||
{
|
||||
_outboxKpiAvailable = false;
|
||||
_outboxKpiError = "KPI query timed out.";
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_outboxKpiAvailable = false;
|
||||
@@ -807,12 +831,12 @@
|
||||
// killing the dashboard. Mirrors LoadOutboxKpis's error handling shape — a
|
||||
// response with Success == false (repository fault) and an Ask that threw
|
||||
// (transport fault) both collapse to "unavailable".
|
||||
private async Task LoadSiteCallKpis()
|
||||
private async Task LoadSiteCallKpis(bool forceRefresh)
|
||||
{
|
||||
try
|
||||
{
|
||||
var response = await CommunicationService.GetSiteCallKpisAsync(
|
||||
new SiteCallKpiRequest(Guid.NewGuid().ToString("N")));
|
||||
using var timeout = new CancellationTokenSource(TileTimeout);
|
||||
var response = await KpiSnapshots.GetSiteCallKpisAsync(forceRefresh, timeout.Token);
|
||||
if (response.Success)
|
||||
{
|
||||
_siteCallKpi = response;
|
||||
@@ -825,6 +849,11 @@
|
||||
_siteCallKpiError = response.ErrorMessage ?? "KPI query failed.";
|
||||
}
|
||||
}
|
||||
catch (OperationCanceledException)
|
||||
{
|
||||
_siteCallKpiAvailable = false;
|
||||
_siteCallKpiError = "KPI query timed out.";
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_siteCallKpiAvailable = false;
|
||||
@@ -834,12 +863,12 @@
|
||||
|
||||
// Per-node site-call KPI loader (M5.2). Best-effort; a fault silently
|
||||
// suppresses the per-node sub-table rather than degrading the dashboard.
|
||||
private async Task LoadSiteCallNodeKpis()
|
||||
private async Task LoadSiteCallNodeKpis(bool forceRefresh)
|
||||
{
|
||||
try
|
||||
{
|
||||
var response = await CommunicationService.GetPerNodeSiteCallKpisAsync(
|
||||
new PerNodeSiteCallKpiRequest(Guid.NewGuid().ToString("N")));
|
||||
using var timeout = new CancellationTokenSource(TileTimeout);
|
||||
var response = await KpiSnapshots.GetPerNodeSiteCallKpisAsync(forceRefresh, timeout.Token);
|
||||
if (response.Success)
|
||||
{
|
||||
_siteCallNodeKpis = response.Nodes;
|
||||
@@ -864,14 +893,20 @@
|
||||
// Audit KPI loader: wraps the service call so a transient DB outage degrades
|
||||
// the three tiles to em dashes with an inline error rather than killing the
|
||||
// dashboard. Mirrors LoadOutboxKpis's error handling shape.
|
||||
private async Task LoadAuditKpis()
|
||||
private async Task LoadAuditKpis(bool forceRefresh)
|
||||
{
|
||||
try
|
||||
{
|
||||
_auditKpi = await AuditLogQueryService.GetKpiSnapshotAsync();
|
||||
using var timeout = new CancellationTokenSource(TileTimeout);
|
||||
_auditKpi = await KpiSnapshots.GetAuditKpiSnapshotAsync(forceRefresh, timeout.Token);
|
||||
_auditKpiAvailable = true;
|
||||
_auditKpiError = null;
|
||||
}
|
||||
catch (OperationCanceledException)
|
||||
{
|
||||
_auditKpiAvailable = false;
|
||||
_auditKpiError = "KPI query timed out.";
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_auditKpiAvailable = false;
|
||||
|
||||
+19
-13
@@ -7,7 +7,7 @@
|
||||
@using ZB.MOM.WW.ScadaBridge.Commons.Types.Kpi
|
||||
@using ZB.MOM.WW.ScadaBridge.Communication
|
||||
@using ZB.MOM.WW.ScadaBridge.CentralUI.Services
|
||||
@inject CommunicationService CommunicationService
|
||||
@inject IKpiSnapshotCache KpiSnapshots
|
||||
@inject ISiteRepository SiteRepository
|
||||
@inject IKpiHistoryQueryService KpiHistory
|
||||
@inject ILogger<NotificationKpis> Logger
|
||||
@@ -250,15 +250,24 @@
|
||||
Logger.LogWarning(ex, "Failed to load sites for the KPI per-site breakdown.");
|
||||
}
|
||||
|
||||
await RefreshAll();
|
||||
// First load rides whatever the shared KPI cache already holds fresh; the
|
||||
// Refresh button forces a new query (see RefreshAll).
|
||||
await LoadAllAsync(forceRefresh: false);
|
||||
}
|
||||
|
||||
private async Task RefreshAll()
|
||||
// The operator's explicit Refresh bypasses the shared cache's freshness window.
|
||||
private Task RefreshAll() => LoadAllAsync(forceRefresh: true);
|
||||
|
||||
private async Task LoadAllAsync(bool forceRefresh)
|
||||
{
|
||||
_loading = true;
|
||||
// Race-free despite all tasks mutating component fields: Blazor Server runs
|
||||
// every continuation on the circuit's single-threaded synchronization context.
|
||||
await Task.WhenAll(LoadGlobalKpis(), LoadPerSiteKpis(), LoadPerNodeKpis(), LoadTrends());
|
||||
await Task.WhenAll(
|
||||
LoadGlobalKpis(forceRefresh),
|
||||
LoadPerSiteKpis(forceRefresh),
|
||||
LoadPerNodeKpis(forceRefresh),
|
||||
LoadTrends());
|
||||
_loading = false;
|
||||
}
|
||||
|
||||
@@ -320,12 +329,11 @@
|
||||
}
|
||||
}
|
||||
|
||||
private async Task LoadGlobalKpis()
|
||||
private async Task LoadGlobalKpis(bool forceRefresh)
|
||||
{
|
||||
try
|
||||
{
|
||||
var response = await CommunicationService.GetNotificationKpisAsync(
|
||||
new NotificationKpiRequest(Guid.NewGuid().ToString("N")));
|
||||
var response = await KpiSnapshots.GetNotificationKpisAsync(forceRefresh);
|
||||
if (response.Success)
|
||||
{
|
||||
_kpi = response;
|
||||
@@ -342,12 +350,11 @@
|
||||
}
|
||||
}
|
||||
|
||||
private async Task LoadPerSiteKpis()
|
||||
private async Task LoadPerSiteKpis(bool forceRefresh)
|
||||
{
|
||||
try
|
||||
{
|
||||
var response = await CommunicationService.GetPerSiteNotificationKpisAsync(
|
||||
new PerSiteNotificationKpiRequest(Guid.NewGuid().ToString("N")));
|
||||
var response = await KpiSnapshots.GetPerSiteNotificationKpisAsync(forceRefresh);
|
||||
if (response.Success)
|
||||
{
|
||||
_perSite = response.Sites;
|
||||
@@ -364,12 +371,11 @@
|
||||
}
|
||||
}
|
||||
|
||||
private async Task LoadPerNodeKpis()
|
||||
private async Task LoadPerNodeKpis(bool forceRefresh)
|
||||
{
|
||||
try
|
||||
{
|
||||
var response = await CommunicationService.GetPerNodeNotificationKpisAsync(
|
||||
new PerNodeNotificationKpiRequest(Guid.NewGuid().ToString("N")));
|
||||
var response = await KpiSnapshots.GetPerNodeNotificationKpisAsync(forceRefresh);
|
||||
if (response.Success)
|
||||
{
|
||||
_perNode = response.Nodes;
|
||||
|
||||
@@ -177,8 +177,42 @@ public class ScriptAnalysisService
|
||||
private const int SandboxMaxConsoleChars = 32_000;
|
||||
private const int SandboxMaxReturnJsonChars = 32_000;
|
||||
|
||||
/// <summary>
|
||||
/// Shared options for the per-Test-Run return-value JSON serialize in
|
||||
/// <see cref="SerializeReturn"/> (perf remediation, arch-review WP1.5) — avoids
|
||||
/// allocating a new <see cref="JsonSerializerOptions"/> per run. Settings preserved
|
||||
/// exactly.
|
||||
/// </summary>
|
||||
private static readonly JsonSerializerOptions SandboxReturnJsonOptions =
|
||||
new() { WriteIndented = true };
|
||||
|
||||
private const int SandboxMaxCallSharedDepth = 16;
|
||||
|
||||
/// <summary>
|
||||
/// Process-wide cap on concurrently executing Test Runs (arch-review WP2.4).
|
||||
/// <para>
|
||||
/// A sandbox run occupies a thread-pool thread for its whole duration — the
|
||||
/// <c>SandboxScriptHost</c> attribute accessors are synchronous by contract and block on
|
||||
/// cross-site I/O (<c>GetAwaiter().GetResult()</c>), so the thread is <em>parked</em>, not
|
||||
/// merely busy, for up to the 10s sandbox timeout. Unbounded, a handful of Designers
|
||||
/// each hitting Run on a slow-bound script could park enough pool threads to starve the
|
||||
/// Blazor circuits sharing this process. The gate bounds that: excess runs wait for a
|
||||
/// slot instead of taking a thread, and the wait is charged to the caller's own timeout.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Static because <c>ScriptAnalysisService</c> is registered scoped — a per-instance gate
|
||||
/// would be per-request and bound nothing.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
private static readonly SemaphoreSlim SandboxRunGate =
|
||||
new(SandboxMaxConcurrentRuns, SandboxMaxConcurrentRuns);
|
||||
|
||||
/// <summary>
|
||||
/// Slot count for <see cref="SandboxRunGate"/>. Deliberately small: Test Run is an
|
||||
/// authoring convenience on a node whose real job is serving circuits.
|
||||
/// </summary>
|
||||
private const int SandboxMaxConcurrentRuns = 4;
|
||||
|
||||
/// <summary>
|
||||
/// Compiles and runs a script in the central process. The globals surface
|
||||
/// depends on <see cref="SandboxRunRequest.Kind"/>: template and shared
|
||||
@@ -428,8 +462,15 @@ public class ScriptAnalysisService
|
||||
using var errorScope = captureError.BeginCapture(captured);
|
||||
|
||||
var stopwatch = Stopwatch.StartNew();
|
||||
var gateAcquired = false;
|
||||
try
|
||||
{
|
||||
// Bound concurrent Test Runs (see SandboxRunGate). The wait is charged to the
|
||||
// caller's own linked token, so queue time counts against the sandbox timeout
|
||||
// and a queued run reports Timeout rather than hanging.
|
||||
await SandboxRunGate.WaitAsync(linkedCts.Token).ConfigureAwait(false);
|
||||
gateAcquired = true;
|
||||
|
||||
// Run on a thread-pool thread with no SynchronizationContext: a
|
||||
// bound script's Instance.SetAttribute / Attributes[...] block
|
||||
// synchronously on cross-site I/O (the API surface is sync by
|
||||
@@ -480,6 +521,10 @@ public class ScriptAnalysisService
|
||||
$"{inner.GetType().Name}: {inner.Message}",
|
||||
SandboxErrorKind.RuntimeError, stopwatch.ElapsedMilliseconds, null);
|
||||
}
|
||||
finally
|
||||
{
|
||||
if (gateAcquired) SandboxRunGate.Release();
|
||||
}
|
||||
// outScope / errorScope are disposed by their `using` declarations when the
|
||||
// method returns, restoring the previous capture scope on this call-tree
|
||||
// without touching process-global Console state.
|
||||
@@ -520,7 +565,7 @@ public class ScriptAnalysisService
|
||||
var typeName = value.GetType().Name;
|
||||
try
|
||||
{
|
||||
var json = JsonSerializer.Serialize(value, new JsonSerializerOptions { WriteIndented = true });
|
||||
var json = JsonSerializer.Serialize(value, SandboxReturnJsonOptions);
|
||||
if (json.Length > SandboxMaxReturnJsonChars)
|
||||
json = json[..SandboxMaxReturnJsonChars] + "\n… (truncated)";
|
||||
return (json, typeName);
|
||||
|
||||
@@ -99,7 +99,17 @@ public static class ServiceCollectionExtensions
|
||||
// facade over CommunicationService.RequestDebugSnapshotAsync (the same
|
||||
// single-shot Ask the Debug View uses) — and flattens the alarm states.
|
||||
services.AddScoped<IInstanceSnapshotClient, CommunicationInstanceSnapshotClient>();
|
||||
services.AddScoped<IAlarmSummaryService, AlarmSummaryService>();
|
||||
// The fan-out itself stays scoped (it holds scoped repositories + the snapshot
|
||||
// client), but the page resolves it through SharedAlarmSummaryService — a process
|
||||
// singleton that memoizes the fan-out per site with single-flight, so N circuits
|
||||
// watching one site cost ONE fan-out per window instead of N (arch-review WP2.4).
|
||||
services.AddScoped<AlarmSummaryService>();
|
||||
services.AddSingleton<IAlarmSummaryService, SharedAlarmSummaryService>();
|
||||
|
||||
// Process-level memoized point-in-time KPI snapshots (arch-review WP2.4, finding
|
||||
// #8's per-circuit half). Singleton on purpose: the memo has to outlive any one
|
||||
// circuit for N dashboards to share a single SQL round per refresh window.
|
||||
services.AddSingleton<IKpiSnapshotCache, KpiSnapshotCache>();
|
||||
|
||||
// Secured Writes: dispatches the two-person secured-write commands
|
||||
// (submit / approve / reject / list) to the central ManagementActor through the
|
||||
|
||||
@@ -128,7 +128,20 @@ public sealed class AlarmSummaryService : IAlarmSummaryService
|
||||
}
|
||||
|
||||
/// <inheritdoc/>
|
||||
public AlarmSummaryResult BuildFromLiveAlarms(IReadOnlyList<AlarmStateChanged> alarms)
|
||||
public AlarmSummaryResult BuildFromLiveAlarms(IReadOnlyList<AlarmStateChanged> alarms) =>
|
||||
BuildFromLiveAlarmsCore(alarms);
|
||||
|
||||
/// <inheritdoc/>
|
||||
public AlarmRollup ComputeRollup(IReadOnlyList<AlarmSummaryRow> rows) => ComputeRollupCore(rows);
|
||||
|
||||
/// <summary>
|
||||
/// Pure flattening of a live-cache snapshot into summary rows. Shared with
|
||||
/// <see cref="SharedAlarmSummaryService"/>, which memoizes the fan-out but must produce
|
||||
/// byte-identical rows on the live path — one implementation, no drift.
|
||||
/// </summary>
|
||||
/// <param name="alarms">The live-cache alarm snapshot.</param>
|
||||
/// <returns>The flattened, deterministically ordered rows (never any not-reporting names).</returns>
|
||||
internal static AlarmSummaryResult BuildFromLiveAlarmsCore(IReadOnlyList<AlarmStateChanged> alarms)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(alarms);
|
||||
|
||||
@@ -141,14 +154,22 @@ public sealed class AlarmSummaryService : IAlarmSummaryService
|
||||
.ThenBy(r => r.Alarm.AlarmName, StringComparer.OrdinalIgnoreCase)
|
||||
.ToList();
|
||||
|
||||
// The live cache is alarm-only, so it cannot enumerate "not reporting"
|
||||
// instances — left empty here; the periodic poll (GetSiteAlarmsAsync)
|
||||
// remains the authority for that list.
|
||||
// Alarm rows alone cannot name the instances that failed to answer, so this pure
|
||||
// flattening leaves the list empty. The names come from the fan-out that produced the
|
||||
// snapshot: while the live cache is serving a site, SharedAlarmSummaryService pairs
|
||||
// these rows with ISiteAlarmLiveCache.GetNotReportingInstances (the aggregator's own
|
||||
// seed/reconcile result); while it is cold, GetSiteAlarmsAsync's poll supplies both.
|
||||
return new AlarmSummaryResult(orderedRows, Array.Empty<string>());
|
||||
}
|
||||
|
||||
/// <inheritdoc/>
|
||||
public AlarmRollup ComputeRollup(IReadOnlyList<AlarmSummaryRow> rows)
|
||||
/// <summary>
|
||||
/// Pure roll-up over already-flattened rows. Shared with
|
||||
/// <see cref="SharedAlarmSummaryService"/> for the same one-implementation reason as
|
||||
/// <see cref="BuildFromLiveAlarmsCore"/>.
|
||||
/// </summary>
|
||||
/// <param name="rows">The rows to roll up.</param>
|
||||
/// <returns>The active/worst-severity/unacked/by-kind roll-up.</returns>
|
||||
internal static AlarmRollup ComputeRollupCore(IReadOnlyList<AlarmSummaryRow> rows)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(rows);
|
||||
|
||||
|
||||
@@ -0,0 +1,70 @@
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Audit;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Notification;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Types;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.CentralUI.Services;
|
||||
|
||||
/// <summary>
|
||||
/// Process-level memoized point-in-time KPI snapshots shared by every Blazor circuit on
|
||||
/// this node (arch-review WP2.4, finding #8 — the per-circuit polling half).
|
||||
/// <para>
|
||||
/// Every KPI here is a <em>global, point-in-time aggregate</em>: it does not vary by user,
|
||||
/// site scope, or role, so N circuits asking for it inside one refresh window are asking
|
||||
/// the identical question. Without memoization, ten Health dashboards meant ten Asks →
|
||||
/// ten aggregate SQL round trips every ten seconds. Each accessor is independently
|
||||
/// memoized with a short TTL and single-flight production, so the fleet costs ONE round
|
||||
/// trip per KPI per window regardless of viewer count.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Faults are never memoized: a failed query is re-attempted by the next caller, and the
|
||||
/// caller still sees the exception exactly as it would from a direct query — the pages'
|
||||
/// existing per-tile "unavailable" degradation is unchanged.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public interface IKpiSnapshotCache
|
||||
{
|
||||
/// <summary>Global Notification Outbox KPIs (queue depth, stuck, parked, delivered-last-interval).</summary>
|
||||
/// <param name="forceRefresh">Bypass a still-fresh value (operator "Refresh"); an in-flight round is still joined.</param>
|
||||
/// <param name="cancellationToken">Cancels this caller's wait, never the shared flight.</param>
|
||||
/// <returns>The memoized notification KPI response.</returns>
|
||||
Task<NotificationKpiResponse> GetNotificationKpisAsync(
|
||||
bool forceRefresh = false, CancellationToken cancellationToken = default);
|
||||
|
||||
/// <summary>Per-source-site Notification Outbox KPI breakdown.</summary>
|
||||
/// <param name="forceRefresh">Bypass a still-fresh value; an in-flight round is still joined.</param>
|
||||
/// <param name="cancellationToken">Cancels this caller's wait, never the shared flight.</param>
|
||||
/// <returns>The memoized per-site notification KPI response.</returns>
|
||||
Task<PerSiteNotificationKpiResponse> GetPerSiteNotificationKpisAsync(
|
||||
bool forceRefresh = false, CancellationToken cancellationToken = default);
|
||||
|
||||
/// <summary>Per-node Notification Outbox KPI breakdown (grouped by <c>SourceNode</c>).</summary>
|
||||
/// <param name="forceRefresh">Bypass a still-fresh value; an in-flight round is still joined.</param>
|
||||
/// <param name="cancellationToken">Cancels this caller's wait, never the shared flight.</param>
|
||||
/// <returns>The memoized per-node notification KPI response.</returns>
|
||||
Task<PerNodeNotificationKpiResponse> GetPerNodeNotificationKpisAsync(
|
||||
bool forceRefresh = false, CancellationToken cancellationToken = default);
|
||||
|
||||
/// <summary>Global Site Call Audit KPIs (buffered, stuck, parked).</summary>
|
||||
/// <param name="forceRefresh">Bypass a still-fresh value; an in-flight round is still joined.</param>
|
||||
/// <param name="cancellationToken">Cancels this caller's wait, never the shared flight.</param>
|
||||
/// <returns>The memoized site call KPI response.</returns>
|
||||
Task<SiteCallKpiResponse> GetSiteCallKpisAsync(
|
||||
bool forceRefresh = false, CancellationToken cancellationToken = default);
|
||||
|
||||
/// <summary>Per-node Site Call Audit KPI breakdown (grouped by <c>SourceNode</c>).</summary>
|
||||
/// <param name="forceRefresh">Bypass a still-fresh value; an in-flight round is still joined.</param>
|
||||
/// <param name="cancellationToken">Cancels this caller's wait, never the shared flight.</param>
|
||||
/// <returns>The memoized per-node site call KPI response.</returns>
|
||||
Task<PerNodeSiteCallKpiResponse> GetPerNodeSiteCallKpisAsync(
|
||||
bool forceRefresh = false, CancellationToken cancellationToken = default);
|
||||
|
||||
/// <summary>
|
||||
/// Audit Log KPI snapshot (1h volume + error rate over the central <c>AuditLog</c> table,
|
||||
/// plus the summed per-site backlog).
|
||||
/// </summary>
|
||||
/// <param name="forceRefresh">Bypass a still-fresh value; an in-flight round is still joined.</param>
|
||||
/// <param name="cancellationToken">Cancels this caller's wait, never the shared flight.</param>
|
||||
/// <returns>The memoized audit KPI snapshot.</returns>
|
||||
Task<AuditLogKpiSnapshot> GetAuditKpiSnapshotAsync(
|
||||
bool forceRefresh = false, CancellationToken cancellationToken = default);
|
||||
}
|
||||
@@ -0,0 +1,122 @@
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Audit;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Notification;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Types;
|
||||
using ZB.MOM.WW.ScadaBridge.Communication;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.CentralUI.Services;
|
||||
|
||||
/// <summary>
|
||||
/// Default <see cref="IKpiSnapshotCache"/> — a DI <b>singleton</b> holding one
|
||||
/// <see cref="SingleFlightMemo{T}"/> per KPI query.
|
||||
/// <para>
|
||||
/// Registered as a singleton on purpose: the whole point is that the memo outlives any one
|
||||
/// circuit. It depends only on <see cref="CommunicationService"/> (itself a singleton) and
|
||||
/// <see cref="IServiceScopeFactory"/> — the audit KPI resolves its scoped query service from
|
||||
/// a fresh scope per flight, the same shape <c>AuditLogQueryService</c> uses internally so a
|
||||
/// KPI round never shares a circuit-scoped <c>DbContext</c>.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public sealed class KpiSnapshotCache : IKpiSnapshotCache
|
||||
{
|
||||
/// <summary>
|
||||
/// Freshness window. Sized just under the Health dashboard's 10s tick so each tick still
|
||||
/// yields ONE fresh query for the whole node (rather than one per circuit), while a
|
||||
/// circuit whose tick lands mid-window is served the memoized value instead of duplicating it.
|
||||
/// </summary>
|
||||
internal static readonly TimeSpan Ttl = TimeSpan.FromSeconds(8);
|
||||
|
||||
private readonly CommunicationService _communication;
|
||||
private readonly IServiceScopeFactory _scopeFactory;
|
||||
|
||||
private readonly SingleFlightMemo<NotificationKpiResponse> _notificationKpis;
|
||||
private readonly SingleFlightMemo<PerSiteNotificationKpiResponse> _perSiteNotificationKpis;
|
||||
private readonly SingleFlightMemo<PerNodeNotificationKpiResponse> _perNodeNotificationKpis;
|
||||
private readonly SingleFlightMemo<SiteCallKpiResponse> _siteCallKpis;
|
||||
private readonly SingleFlightMemo<PerNodeSiteCallKpiResponse> _perNodeSiteCallKpis;
|
||||
private readonly SingleFlightMemo<AuditLogKpiSnapshot> _auditKpis;
|
||||
|
||||
/// <summary>
|
||||
/// Initializes the shared KPI snapshot cache.
|
||||
/// </summary>
|
||||
/// <param name="communication">Central-side comms used to Ask the outbox / site-call singletons.</param>
|
||||
/// <param name="scopeFactory">Opens a fresh DI scope per audit-KPI flight.</param>
|
||||
public KpiSnapshotCache(CommunicationService communication, IServiceScopeFactory scopeFactory)
|
||||
: this(communication, scopeFactory, Ttl, clock: null)
|
||||
{
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Test seam: same cache with an explicit TTL and clock so freshness can be asserted
|
||||
/// without real delays.
|
||||
/// </summary>
|
||||
/// <param name="communication">Central-side comms used to Ask the outbox / site-call singletons.</param>
|
||||
/// <param name="scopeFactory">Opens a fresh DI scope per audit-KPI flight.</param>
|
||||
/// <param name="ttl">Freshness window.</param>
|
||||
/// <param name="clock">Clock used for freshness.</param>
|
||||
internal KpiSnapshotCache(
|
||||
CommunicationService communication,
|
||||
IServiceScopeFactory scopeFactory,
|
||||
TimeSpan ttl,
|
||||
Func<DateTimeOffset>? clock)
|
||||
{
|
||||
_communication = communication ?? throw new ArgumentNullException(nameof(communication));
|
||||
_scopeFactory = scopeFactory ?? throw new ArgumentNullException(nameof(scopeFactory));
|
||||
|
||||
_notificationKpis = new SingleFlightMemo<NotificationKpiResponse>(ttl, clock);
|
||||
_perSiteNotificationKpis = new SingleFlightMemo<PerSiteNotificationKpiResponse>(ttl, clock);
|
||||
_perNodeNotificationKpis = new SingleFlightMemo<PerNodeNotificationKpiResponse>(ttl, clock);
|
||||
_siteCallKpis = new SingleFlightMemo<SiteCallKpiResponse>(ttl, clock);
|
||||
_perNodeSiteCallKpis = new SingleFlightMemo<PerNodeSiteCallKpiResponse>(ttl, clock);
|
||||
_auditKpis = new SingleFlightMemo<AuditLogKpiSnapshot>(ttl, clock);
|
||||
}
|
||||
|
||||
/// <inheritdoc/>
|
||||
public Task<NotificationKpiResponse> GetNotificationKpisAsync(
|
||||
bool forceRefresh = false, CancellationToken cancellationToken = default) =>
|
||||
_notificationKpis.GetAsync(
|
||||
() => _communication.GetNotificationKpisAsync(new NotificationKpiRequest(NewCorrelationId())),
|
||||
forceRefresh, cancellationToken);
|
||||
|
||||
/// <inheritdoc/>
|
||||
public Task<PerSiteNotificationKpiResponse> GetPerSiteNotificationKpisAsync(
|
||||
bool forceRefresh = false, CancellationToken cancellationToken = default) =>
|
||||
_perSiteNotificationKpis.GetAsync(
|
||||
() => _communication.GetPerSiteNotificationKpisAsync(new PerSiteNotificationKpiRequest(NewCorrelationId())),
|
||||
forceRefresh, cancellationToken);
|
||||
|
||||
/// <inheritdoc/>
|
||||
public Task<PerNodeNotificationKpiResponse> GetPerNodeNotificationKpisAsync(
|
||||
bool forceRefresh = false, CancellationToken cancellationToken = default) =>
|
||||
_perNodeNotificationKpis.GetAsync(
|
||||
() => _communication.GetPerNodeNotificationKpisAsync(new PerNodeNotificationKpiRequest(NewCorrelationId())),
|
||||
forceRefresh, cancellationToken);
|
||||
|
||||
/// <inheritdoc/>
|
||||
public Task<SiteCallKpiResponse> GetSiteCallKpisAsync(
|
||||
bool forceRefresh = false, CancellationToken cancellationToken = default) =>
|
||||
_siteCallKpis.GetAsync(
|
||||
() => _communication.GetSiteCallKpisAsync(new SiteCallKpiRequest(NewCorrelationId())),
|
||||
forceRefresh, cancellationToken);
|
||||
|
||||
/// <inheritdoc/>
|
||||
public Task<PerNodeSiteCallKpiResponse> GetPerNodeSiteCallKpisAsync(
|
||||
bool forceRefresh = false, CancellationToken cancellationToken = default) =>
|
||||
_perNodeSiteCallKpis.GetAsync(
|
||||
() => _communication.GetPerNodeSiteCallKpisAsync(new PerNodeSiteCallKpiRequest(NewCorrelationId())),
|
||||
forceRefresh, cancellationToken);
|
||||
|
||||
/// <inheritdoc/>
|
||||
public Task<AuditLogKpiSnapshot> GetAuditKpiSnapshotAsync(
|
||||
bool forceRefresh = false, CancellationToken cancellationToken = default) =>
|
||||
_auditKpis.GetAsync(LoadAuditKpisAsync, forceRefresh, cancellationToken);
|
||||
|
||||
private async Task<AuditLogKpiSnapshot> LoadAuditKpisAsync()
|
||||
{
|
||||
await using var scope = _scopeFactory.CreateAsyncScope();
|
||||
var queryService = scope.ServiceProvider.GetRequiredService<IAuditLogQueryService>();
|
||||
return await queryService.GetKpiSnapshotAsync();
|
||||
}
|
||||
|
||||
private static string NewCorrelationId() => Guid.NewGuid().ToString("N");
|
||||
}
|
||||
@@ -0,0 +1,132 @@
|
||||
using System.Collections.Concurrent;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Streaming;
|
||||
using ZB.MOM.WW.ScadaBridge.Communication;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.CentralUI.Services;
|
||||
|
||||
/// <summary>
|
||||
/// Process-level memoizing façade over <see cref="AlarmSummaryService"/> (arch-review WP2.4).
|
||||
/// <para>
|
||||
/// The Alarm Summary page polls every 15s <em>per circuit</em>, and each poll fans one debug
|
||||
/// snapshot Ask out to every Enabled instance on the site. Ten operators watching one site
|
||||
/// meant ten independent fan-outs per 15s across the same instances — on top of the shared
|
||||
/// live-alarm aggregator already seeding and reconciling from the identical fan-out. The
|
||||
/// question is not user-specific, so this service collapses it: one memo slot per site,
|
||||
/// single-flight, so the node costs ONE fan-out per site per window no matter how many
|
||||
/// viewers are watching.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>While the live cache is serving the site there is NO fan-out at all.</b> The page already
|
||||
/// ignores the poll's alarm rows in that state (the live deltas own them — arch-review R2 N5),
|
||||
/// and the aggregator's own seed/reconcile fan-out now publishes the not-reporting set through
|
||||
/// <see cref="ISiteAlarmLiveCache.GetNotReportingInstances"/> — the one thing the poll used to
|
||||
/// still supply. So the live path is answered entirely from the cache: rows flattened from
|
||||
/// <see cref="ISiteAlarmLiveCache.GetCurrentAlarms"/> through the same
|
||||
/// <c>AlarmSummaryService.BuildFromLiveAlarmsCore</c> the page's live subscription uses, plus the
|
||||
/// cached not-reporting names. One fan-out per site per reconcile window, run by the aggregator,
|
||||
/// instead of two.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// The rows are returned rather than left empty on purpose: liveness can flip to false in the
|
||||
/// window between this call and the page's own <c>IsLive</c> re-check, and the page would then
|
||||
/// adopt whatever came back. Handing it the live snapshot makes that race land on last-known
|
||||
/// state instead of a blank grid.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// While the cache is cold the poll is the page's full-rebuild safety net, so it still fans out,
|
||||
/// memoized with a window just under the page's 15s tick so every tick gets fresh data.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public sealed class SharedAlarmSummaryService : IAlarmSummaryService
|
||||
{
|
||||
/// <summary>
|
||||
/// Freshness window while the live alarm cache is NOT serving this site: the poll is the
|
||||
/// page's authoritative rebuild path, so it must stay under the 15s page tick.
|
||||
/// </summary>
|
||||
internal static readonly TimeSpan ColdCacheTtl = TimeSpan.FromSeconds(12);
|
||||
|
||||
private readonly IServiceScopeFactory _scopeFactory;
|
||||
private readonly ISiteAlarmLiveCache _liveCache;
|
||||
private readonly Func<DateTimeOffset>? _clock;
|
||||
|
||||
private readonly ConcurrentDictionary<int, SingleFlightMemo<AlarmSummaryResult>> _bySite = new();
|
||||
|
||||
/// <summary>
|
||||
/// Initializes the shared alarm summary façade.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// There is no live-cache freshness window any more: the live path reads the aggregator's
|
||||
/// cache directly and never fans out, so the only window left is <see cref="ColdCacheTtl"/>.
|
||||
/// The reconcile interval this ctor used to read from <c>CommunicationOptions</c> is now
|
||||
/// implicit — it IS the cadence at which the aggregator refreshes what the live path returns.
|
||||
/// </remarks>
|
||||
/// <param name="scopeFactory">Opens a fresh DI scope per fan-out (fresh repositories, off any circuit scope).</param>
|
||||
/// <param name="liveCache">The shared live alarm cache: liveness, alarm rows, and not-reporting names.</param>
|
||||
public SharedAlarmSummaryService(
|
||||
IServiceScopeFactory scopeFactory,
|
||||
ISiteAlarmLiveCache liveCache)
|
||||
: this(scopeFactory, liveCache, clock: null)
|
||||
{
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Test seam: same façade with an explicit clock.
|
||||
/// </summary>
|
||||
/// <param name="scopeFactory">Opens a fresh DI scope per fan-out.</param>
|
||||
/// <param name="liveCache">The shared live alarm cache.</param>
|
||||
/// <param name="clock">Clock used for freshness.</param>
|
||||
internal SharedAlarmSummaryService(
|
||||
IServiceScopeFactory scopeFactory,
|
||||
ISiteAlarmLiveCache liveCache,
|
||||
Func<DateTimeOffset>? clock)
|
||||
{
|
||||
_scopeFactory = scopeFactory ?? throw new ArgumentNullException(nameof(scopeFactory));
|
||||
_liveCache = liveCache ?? throw new ArgumentNullException(nameof(liveCache));
|
||||
_clock = clock;
|
||||
}
|
||||
|
||||
/// <inheritdoc/>
|
||||
public Task<AlarmSummaryResult> GetSiteAlarmsAsync(
|
||||
int siteId, CancellationToken cancellationToken = default)
|
||||
{
|
||||
// Live: answer from the shared aggregator's cache and skip the fan-out entirely. Not
|
||||
// memoized — there is nothing to amortize, and reading the cache directly keeps the
|
||||
// answer as fresh as the aggregator itself.
|
||||
if (_liveCache.IsLive(siteId))
|
||||
{
|
||||
var live = AlarmSummaryService.BuildFromLiveAlarmsCore(_liveCache.GetCurrentAlarms(siteId));
|
||||
return Task.FromResult(new AlarmSummaryResult(
|
||||
live.Alarms, _liveCache.GetNotReportingInstances(siteId)));
|
||||
}
|
||||
|
||||
var memo = _bySite.GetOrAdd(
|
||||
siteId,
|
||||
_ => new SingleFlightMemo<AlarmSummaryResult>(ColdCacheTtl, _clock));
|
||||
|
||||
return memo.GetAsync(
|
||||
() => FanOutAsync(siteId),
|
||||
forceRefresh: false,
|
||||
cancellationToken,
|
||||
ttlOverride: ColdCacheTtl);
|
||||
}
|
||||
|
||||
/// <inheritdoc/>
|
||||
public AlarmSummaryResult BuildFromLiveAlarms(IReadOnlyList<AlarmStateChanged> alarms) =>
|
||||
AlarmSummaryService.BuildFromLiveAlarmsCore(alarms);
|
||||
|
||||
/// <inheritdoc/>
|
||||
public AlarmRollup ComputeRollup(IReadOnlyList<AlarmSummaryRow> rows) =>
|
||||
AlarmSummaryService.ComputeRollupCore(rows);
|
||||
|
||||
/// <summary>
|
||||
/// One shared fan-out. Deliberately runs with no caller cancellation token: the flight is
|
||||
/// shared, so one circuit navigating away must not cancel the round the others are awaiting.
|
||||
/// </summary>
|
||||
private async Task<AlarmSummaryResult> FanOutAsync(int siteId)
|
||||
{
|
||||
await using var scope = _scopeFactory.CreateAsyncScope();
|
||||
var inner = scope.ServiceProvider.GetRequiredService<AlarmSummaryService>();
|
||||
return await inner.GetSiteAlarmsAsync(siteId, CancellationToken.None);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,136 @@
|
||||
namespace ZB.MOM.WW.ScadaBridge.CentralUI.Services;
|
||||
|
||||
/// <summary>
|
||||
/// A process-level memo slot: one value, a time-to-live, and single-flight production.
|
||||
/// <para>
|
||||
/// Blazor Server runs one circuit per connected browser, so a page that polls on a timer
|
||||
/// runs its query <em>per circuit</em>. Ten operators on the Health dashboard meant ten
|
||||
/// identical KPI round trips every ten seconds. This slot collapses them: the first caller
|
||||
/// in a TTL window starts the flight, every concurrent caller awaits the <em>same</em>
|
||||
/// <see cref="Task{TResult}"/>, and callers arriving after it completes are served the
|
||||
/// memoized value until the TTL expires.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Failures are never cached — a faulted flight is discarded so the next caller retries.
|
||||
/// The caller's own exception handling is therefore unchanged: it still sees the fault
|
||||
/// (all callers of that one flight see it), and the next window starts clean.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
/// <typeparam name="T">The memoized value type.</typeparam>
|
||||
internal sealed class SingleFlightMemo<T>
|
||||
{
|
||||
private readonly object _lock = new();
|
||||
private readonly TimeSpan _ttl;
|
||||
private readonly Func<DateTimeOffset> _clock;
|
||||
|
||||
/// <summary>The current flight, completed or not. Null until the first call.</summary>
|
||||
private Task<T>? _current;
|
||||
|
||||
/// <summary>Wall-clock instant after which <see cref="_current"/> is stale. Only meaningful when it completed successfully.</summary>
|
||||
private DateTimeOffset _freshUntil;
|
||||
|
||||
/// <summary>
|
||||
/// Initializes a memo slot.
|
||||
/// </summary>
|
||||
/// <param name="ttl">How long a successfully produced value stays fresh.</param>
|
||||
/// <param name="clock">Clock used for freshness, injectable so tests need no real delay.</param>
|
||||
public SingleFlightMemo(TimeSpan ttl, Func<DateTimeOffset>? clock = null)
|
||||
{
|
||||
_ttl = ttl;
|
||||
_clock = clock ?? (() => DateTimeOffset.UtcNow);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Number of times the factory has actually been invoked. Diagnostics and tests only.
|
||||
/// </summary>
|
||||
public int FlightCount { get; private set; }
|
||||
|
||||
/// <summary>
|
||||
/// Returns the memoized value, starting a flight if none is fresh.
|
||||
/// </summary>
|
||||
/// <param name="factory">Produces a fresh value. Invoked outside the slot's lock.</param>
|
||||
/// <param name="forceRefresh">
|
||||
/// When <see langword="true"/>, a completed-but-still-fresh value is discarded and a new
|
||||
/// flight starts. An <em>in-flight</em> round is still joined rather than duplicated, so a
|
||||
/// burst of operator "Refresh" clicks remains one round trip.
|
||||
/// </param>
|
||||
/// <param name="cancellationToken">
|
||||
/// Cancels this caller's wait only — never the shared flight, which other callers may still
|
||||
/// be awaiting.
|
||||
/// </param>
|
||||
/// <param name="ttlOverride">
|
||||
/// Freshness window for <em>this</em> flight, overriding the slot's default. Used where the
|
||||
/// acceptable staleness depends on state the caller knows (the Alarm Summary poll needs a
|
||||
/// far shorter window while the live alarm cache is cold than while it is serving deltas).
|
||||
/// </param>
|
||||
/// <returns>The memoized (or freshly produced) value.</returns>
|
||||
public Task<T> GetAsync(
|
||||
Func<Task<T>> factory,
|
||||
bool forceRefresh = false,
|
||||
CancellationToken cancellationToken = default,
|
||||
TimeSpan? ttlOverride = null)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(factory);
|
||||
|
||||
TaskCompletionSource<T>? started = null;
|
||||
Task<T> result;
|
||||
|
||||
lock (_lock)
|
||||
{
|
||||
var current = _current;
|
||||
if (current is { IsCompleted: false })
|
||||
{
|
||||
// A round is already in progress — join it, force or not.
|
||||
result = current;
|
||||
}
|
||||
else if (!forceRefresh
|
||||
&& current is { IsCompletedSuccessfully: true }
|
||||
&& _clock() < _freshUntil)
|
||||
{
|
||||
result = current;
|
||||
}
|
||||
else
|
||||
{
|
||||
// RunContinuationsAsynchronously: a caller's continuation must never run
|
||||
// inline on the thread that completes the flight (that thread is another
|
||||
// circuit's dispatcher).
|
||||
started = new TaskCompletionSource<T>(TaskCreationOptions.RunContinuationsAsynchronously);
|
||||
_current = started.Task;
|
||||
FlightCount++;
|
||||
result = started.Task;
|
||||
}
|
||||
}
|
||||
|
||||
// The factory runs OUTSIDE the lock: it does I/O (an Ask, a SQL round trip) and must
|
||||
// never be able to re-enter this slot on the calling thread while the lock is held.
|
||||
if (started is not null)
|
||||
{
|
||||
_ = RunAsync(factory, started, ttlOverride ?? _ttl);
|
||||
}
|
||||
|
||||
return cancellationToken.CanBeCanceled ? result.WaitAsync(cancellationToken) : result;
|
||||
}
|
||||
|
||||
private async Task RunAsync(Func<Task<T>> factory, TaskCompletionSource<T> completion, TimeSpan ttl)
|
||||
{
|
||||
try
|
||||
{
|
||||
var value = await factory().ConfigureAwait(false);
|
||||
lock (_lock)
|
||||
{
|
||||
_freshUntil = _clock() + ttl;
|
||||
}
|
||||
completion.SetResult(value);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
// A failure is not memoized: leave _freshUntil in the past so the next caller
|
||||
// starts a new flight instead of re-serving the fault for the whole TTL.
|
||||
lock (_lock)
|
||||
{
|
||||
_freshUntil = default;
|
||||
}
|
||||
completion.SetException(ex);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,60 @@
|
||||
namespace ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Protocol;
|
||||
|
||||
/// <summary>
|
||||
/// Per-tag outcome of a batch subscribe. A failed row is a genuine per-tag fault
|
||||
/// (bad node id, unresolvable path) and never aborts the batch — mirroring the
|
||||
/// MxAccess Gateway's <c>SubscribeResult</c> and OPC UA's per-monitored-item create
|
||||
/// status. A batch that fails at connection level throws instead, so the
|
||||
/// DataConnectionActor can drive the reconnect state machine.
|
||||
/// </summary>
|
||||
/// <param name="TagPath">The requested tag path.</param>
|
||||
/// <param name="Success">Whether the tag was subscribed.</param>
|
||||
/// <param name="SubscriptionId">Adapter subscription handle when <paramref name="Success"/>; otherwise <c>null</c>.</param>
|
||||
/// <param name="ErrorMessage">Per-tag failure reason when not successful.</param>
|
||||
public record TagSubscribeResult(string TagPath, bool Success, string? SubscriptionId, string? ErrorMessage);
|
||||
|
||||
/// <summary>
|
||||
/// Optional capability for an <see cref="IDataConnection"/> implementation whose
|
||||
/// protocol can subscribe/unsubscribe MANY tags in one round trip, and whose
|
||||
/// <see cref="IDataConnection.ReadBatchAsync"/> / <see cref="IDataConnection.WriteBatchAsync"/>
|
||||
/// are TRUE bulk service calls rather than a loop over the single-tag methods.
|
||||
/// Mirrors the <see cref="IBrowsableDataConnection"/> / <see cref="IAlarmSubscribableConnection"/>
|
||||
/// capability-interface pattern; consumed by the DataConnectionActor only.
|
||||
///
|
||||
/// <para>
|
||||
/// Implementing this interface is the adapter's assertion that batching is genuinely
|
||||
/// cheaper than N single calls: the actor then subscribes, unsubscribes and seed-reads
|
||||
/// in bounded chunks instead of per tag. An adapter that does NOT implement it keeps
|
||||
/// the historical per-tag behaviour, so the capability is purely additive.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public interface IBatchSubscribableConnection
|
||||
{
|
||||
/// <summary>
|
||||
/// Subscribes every tag in <paramref name="tagPaths"/> in as few protocol round
|
||||
/// trips as the adapter allows, returning one <see cref="TagSubscribeResult"/> per
|
||||
/// requested tag (in any order — callers key by <see cref="TagSubscribeResult.TagPath"/>).
|
||||
/// All tags share ONE <paramref name="callback"/>; the callback already carries the
|
||||
/// tag path, so per-tag delegates would carry nothing extra.
|
||||
/// </summary>
|
||||
/// <param name="tagPaths">The tag paths to subscribe.</param>
|
||||
/// <param name="callback">Callback invoked for every value change on any of the tags.</param>
|
||||
/// <param name="cancellationToken">Cancellation token.</param>
|
||||
/// <returns>One result row per requested tag path.</returns>
|
||||
/// <exception cref="Exception">
|
||||
/// A connection-level fault (adapter not connected / transport down) is thrown so the
|
||||
/// caller classifies it as a connection failure; per-tag faults are result rows.
|
||||
/// </exception>
|
||||
Task<IReadOnlyList<TagSubscribeResult>> SubscribeBatchAsync(
|
||||
IReadOnlyList<string> tagPaths, SubscriptionCallback callback, CancellationToken cancellationToken = default);
|
||||
|
||||
/// <summary>
|
||||
/// Releases every supplied subscription id in as few protocol round trips as the
|
||||
/// adapter allows. Unknown/stale ids are ignored, mirroring
|
||||
/// <see cref="IDataConnection.UnsubscribeAsync"/>.
|
||||
/// </summary>
|
||||
/// <param name="subscriptionIds">Subscription ids previously returned by a subscribe call.</param>
|
||||
/// <param name="cancellationToken">Cancellation token.</param>
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
Task UnsubscribeBatchAsync(IReadOnlyList<string> subscriptionIds, CancellationToken cancellationToken = default);
|
||||
}
|
||||
@@ -34,6 +34,62 @@ public interface IAuditLogRepository
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
Task InsertIfNotExistsAsync(AuditEvent evt, CancellationToken ct = default);
|
||||
|
||||
/// <summary>
|
||||
/// Set-based form of <see cref="InsertIfNotExistsAsync"/>: inserts every
|
||||
/// event in <paramref name="events"/> that does not already exist, in as few
|
||||
/// round trips as the provider allows, and returns the number of rows
|
||||
/// actually written. First-write-wins idempotency is unchanged — an EventId
|
||||
/// already present is silently skipped.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// <b>Duplicate EventIds are tolerated both within and across packets.</b>
|
||||
/// Duplicates inside one call collapse to the first occurrence before the
|
||||
/// statement is built (an append-only row is immutable, so later copies of an
|
||||
/// EventId carry the same content); duplicates against already-committed rows
|
||||
/// are eliminated by the anti-semi-join. A concurrent writer that commits a
|
||||
/// row mid-statement is handled by falling back to the per-row path, so the
|
||||
/// batch is a throughput optimisation and never a correctness dependency.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Default implementation.</b> The interface supplies a per-row loop so a
|
||||
/// test double or an alternative store keeps working unchanged; the EF Core
|
||||
/// implementation overrides it with a genuine set-based statement. Callers on
|
||||
/// the ingest hot path should always prefer this method — the per-row form
|
||||
/// cost one <c>IF NOT EXISTS … INSERT</c> round trip per audit event.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
/// <param name="events">Audit events to insert; may contain duplicates.</param>
|
||||
/// <param name="commandTimeout">
|
||||
/// Optional per-statement timeout. The ingest actor passes a budget strictly
|
||||
/// SHORTER than the gRPC Ask that wraps it, so the reply is produced before
|
||||
/// the caller gives up rather than at the same instant.
|
||||
/// </param>
|
||||
/// <param name="ct">Cancellation token.</param>
|
||||
/// <returns>A task that resolves to the number of rows inserted (duplicates excluded).</returns>
|
||||
async Task<int> InsertManyIfNotExistsAsync(
|
||||
IReadOnlyList<AuditEvent> events,
|
||||
TimeSpan? commandTimeout = null,
|
||||
CancellationToken ct = default)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(events);
|
||||
|
||||
var inserted = 0;
|
||||
var seen = new HashSet<Guid>(events.Count);
|
||||
foreach (var evt in events)
|
||||
{
|
||||
if (evt is null || !seen.Add(evt.EventId))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
await InsertIfNotExistsAsync(evt, ct).ConfigureAwait(false);
|
||||
inserted++;
|
||||
}
|
||||
|
||||
return inserted;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Returns up to <see cref="AuditLogPaging.PageSize"/> rows matching
|
||||
/// <paramref name="filter"/>, ordered by <c>(OccurredAtUtc DESC, EventId DESC)</c>.
|
||||
@@ -60,36 +116,32 @@ public interface IAuditLogRepository
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// <b>Drop-and-rebuild dance.</b> <c>UX_AuditLog_EventId</c> is intentionally
|
||||
/// non-partition-aligned (it lives on <c>[PRIMARY]</c> so single-column
|
||||
/// EventId uniqueness — required by <see cref="InsertIfNotExistsAsync"/> —
|
||||
/// can be enforced cheaply). SQL Server rejects
|
||||
/// <c>ALTER TABLE … SWITCH PARTITION</c> while a non-aligned unique index
|
||||
/// is present, so the implementation drops the index, creates a staging
|
||||
/// table with byte-identical schema, switches the partition's data into
|
||||
/// staging, drops staging (discarding the rows), and rebuilds the unique
|
||||
/// index. The CATCH branch guarantees the index is rebuilt even on partial
|
||||
/// failure so the table never returns to live traffic without its
|
||||
/// idempotency-supporting index.
|
||||
/// <b>Partition-aligned uniqueness — no index drop.</b> EventId uniqueness is
|
||||
/// enforced by the clustered <c>PK_AuditLog (EventId, OccurredAtUtc)</c>,
|
||||
/// which is aligned on <c>ps_AuditLog_Month(OccurredAtUtc)</c>, so
|
||||
/// <c>ALTER TABLE … SWITCH PARTITION</c> has no non-aligned unique index to
|
||||
/// object to. The implementation creates a staging table with byte-identical
|
||||
/// schema, switches the partition's data into staging, and drops staging
|
||||
/// (discarding the rows). Nothing is dropped from the live table, so there is
|
||||
/// no window in which the idempotency enforcement is absent and no offline
|
||||
/// index rebuild inside the switch transaction.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Outage window.</b> The dance briefly removes the unique index, so
|
||||
/// concurrent <see cref="InsertIfNotExistsAsync"/> calls during the switch
|
||||
/// could in principle race past the IF NOT EXISTS check without the index
|
||||
/// catching the duplicate. This is acceptable for the daily purge cadence
|
||||
/// — the inserts that the IF NOT EXISTS check guards are themselves rare
|
||||
/// enough that a sub-second collision window is operationally negligible,
|
||||
/// and the composite PK still rejects same-(EventId, OccurredAtUtc) rows.
|
||||
/// The predecessor design carried a non-aligned
|
||||
/// <c>UX_AuditLog_EventId</c> on <c>[PRIMARY]</c> that had to be dropped and
|
||||
/// rebuilt around every switch. A defensive guarded <c>DROP INDEX</c> remains
|
||||
/// in the batch so a database restored from a pre-alignment backup still
|
||||
/// purges; it is a one-way cleanup, never rebuilt.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
/// <param name="monthBoundary">Lower-bound datetime of the monthly partition to switch out.</param>
|
||||
/// <param name="commandTimeout">
|
||||
/// Optional per-command timeout for the maintenance SQL (the row-count sample plus the
|
||||
/// drop-and-rebuild dance). When null the provider default applies. The purge actor passes
|
||||
/// staging/switch batch). When null the provider default applies. The purge actor passes
|
||||
/// <see cref="ZB.MOM.WW.ScadaBridge.AuditLog.Central.AuditLogPurgeOptions.ResolvedMaintenanceCommandTimeout"/>
|
||||
/// (default 30 min) because the ~30s ADO.NET default can abort the switch mid-dance on a large
|
||||
/// or contended partition, leaving the table without <c>UX_AuditLog_EventId</c> until the next
|
||||
/// tick recovers (arch-review 04, S2).
|
||||
/// (default 30 min) because the ~30s ADO.NET default can abort the switch mid-batch on a large
|
||||
/// or contended partition, leaving an orphaned staging table for the next tick's CATCH to clean
|
||||
/// up (arch-review 04, S2).
|
||||
/// </param>
|
||||
/// <param name="ct">Cancellation token.</param>
|
||||
/// <returns>A task that resolves to the approximate number of rows discarded by the partition switch.</returns>
|
||||
|
||||
+42
@@ -22,6 +22,48 @@ public interface IDeploymentManagerRepository
|
||||
/// <returns>A read-only list of all deployment records.</returns>
|
||||
Task<IReadOnlyList<DeploymentRecord>> GetAllDeploymentRecordsAsync(CancellationToken cancellationToken = default);
|
||||
/// <summary>
|
||||
/// Database-side paged and filtered deployment query returning row summaries.
|
||||
/// Backs the <c>QueryDeployments</c> management command, which previously
|
||||
/// loaded the ENTIRE deployment-record table (insert-only, one row per deploy
|
||||
/// attempt for the life of the system) and ignored the page arguments the CLI
|
||||
/// was already sending.
|
||||
/// </summary>
|
||||
/// <param name="instanceId">Restrict to one instance's history, or <see langword="null"/> for all.</param>
|
||||
/// <param name="status">Restrict to one deployment status, or <see langword="null"/> for all.</param>
|
||||
/// <param name="instanceIdScope">
|
||||
/// When non-null, restrict to these instance ids. Used to apply a site-scoped
|
||||
/// user's permitted-site filter IN THE DATABASE rather than by loading every
|
||||
/// record and every instance and filtering in memory.
|
||||
/// </param>
|
||||
/// <param name="skip">Rows to skip; negative values are floored at 0.</param>
|
||||
/// <param name="take">Page size; non-positive or null returns every remaining row.</param>
|
||||
/// <param name="cancellationToken">A cancellation token that can be used to cancel the operation.</param>
|
||||
/// <returns>One page of deployment summaries, newest first.</returns>
|
||||
Task<IReadOnlyList<Types.Deployment.DeploymentRecordSummary>> QueryDeploymentSummariesAsync(
|
||||
int? instanceId,
|
||||
DeploymentStatus? status,
|
||||
IReadOnlyCollection<int>? instanceIdScope,
|
||||
int skip,
|
||||
int? take,
|
||||
CancellationToken cancellationToken = default);
|
||||
/// <summary>
|
||||
/// Deletes TERMINAL deployment records (<see cref="DeploymentStatus.Success"/> /
|
||||
/// <see cref="DeploymentStatus.Failed"/>) that completed before
|
||||
/// <paramref name="cutoffUtc"/>, in one bounded batch.
|
||||
///
|
||||
/// <para>
|
||||
/// Non-terminal rows are never touched: an <c>InProgress</c> record is what the
|
||||
/// query-before-redeploy reconciliation path reads to decide whether a prior
|
||||
/// deploy actually landed at the site, so purging one by age would silently
|
||||
/// disable that idempotency guard.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
/// <param name="cutoffUtc">Records completing before this instant are eligible.</param>
|
||||
/// <param name="batchSize">Maximum rows deleted in this call.</param>
|
||||
/// <param name="cancellationToken">A cancellation token that can be used to cancel the operation.</param>
|
||||
/// <returns>The number of rows deleted; fewer than <paramref name="batchSize"/> means the sweep is complete.</returns>
|
||||
Task<int> PurgeTerminalDeploymentRecordsAsync(DateTimeOffset cutoffUtc, int batchSize, CancellationToken cancellationToken = default);
|
||||
/// <summary>
|
||||
/// Gets all deployment records for a specific instance.
|
||||
/// </summary>
|
||||
/// <param name="instanceId">The instance ID.</param>
|
||||
|
||||
+34
-5
@@ -46,13 +46,42 @@ public interface INotificationOutboxRepository
|
||||
Task<Notification?> GetByIdAsync(string notificationId, CancellationToken cancellationToken = default);
|
||||
|
||||
/// <summary>
|
||||
/// Marks <paramref name="n"/> modified and persists it (status transitions).
|
||||
/// Commits internally — this call is its own transaction.
|
||||
/// Persists <paramref name="n"/>'s <b>delivery-state</b> columns —
|
||||
/// <c>Status</c>, <c>RetryCount</c>, <c>LastError</c>, <c>ResolvedTargets</c>,
|
||||
/// <c>LastAttemptAt</c>, <c>NextAttemptAt</c>, <c>DeliveredAt</c>. Commits
|
||||
/// internally — this call is its own transaction.
|
||||
/// </summary>
|
||||
/// <param name="n">The notification to update.</param>
|
||||
/// <remarks>
|
||||
/// <b>Scope is deliberately narrow.</b> Those seven columns are the ONLY
|
||||
/// mutable state a notification has: everything else (identity, type, list,
|
||||
/// subject, body, type data, source/origin attribution, enqueue and creation
|
||||
/// timestamps) is written once at ingest and is immutable by contract. Every
|
||||
/// caller — the dispatcher's per-attempt write and the operator retry/discard
|
||||
/// one-shots — touches only this set. Implementations are therefore free to
|
||||
/// issue a targeted UPDATE of these columns rather than rewriting the whole
|
||||
/// row, which matters because the row carries <c>nvarchar(max)</c> body and
|
||||
/// payload columns and the dispatcher writes on EVERY attempt. A future
|
||||
/// caller that needs to change an immutable column must add its own method
|
||||
/// rather than widening this one.
|
||||
/// <para>
|
||||
/// <b>The return value is the not-found signal.</b> A targeted UPDATE against
|
||||
/// a row that no longer exists (the retention purge removed it between the
|
||||
/// read and the write) affects zero rows and is otherwise indistinguishable
|
||||
/// from success — which is how an operator Retry/Discard came to report
|
||||
/// success against a vanished notification. Implementations MUST return
|
||||
/// <see langword="false"/> when no row matched. Callers that speak to a human
|
||||
/// (the operator one-shots) must surface that as "not found"; the dispatcher
|
||||
/// treats it as a lost race and logs.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
/// <param name="n">The notification whose delivery state should be persisted.</param>
|
||||
/// <param name="cancellationToken">Cancellation token.</param>
|
||||
/// <returns>A task that completes when the notification has been persisted.</returns>
|
||||
Task UpdateAsync(Notification n, CancellationToken cancellationToken = default);
|
||||
/// <returns>
|
||||
/// <see langword="true"/> when the row was found and its delivery state
|
||||
/// persisted; <see langword="false"/> when no row with that
|
||||
/// <c>NotificationId</c> exists any more.
|
||||
/// </returns>
|
||||
Task<bool> UpdateAsync(Notification n, CancellationToken cancellationToken = default);
|
||||
|
||||
/// <summary>
|
||||
/// Returns a page of notifications matching <paramref name="filter"/>, ordered by
|
||||
|
||||
+35
@@ -35,6 +35,41 @@ public interface ITemplateEngineRepository
|
||||
/// <returns>A task that resolves to a read-only list of all templates.</returns>
|
||||
Task<IReadOnlyList<Template>> GetAllTemplatesAsync(CancellationToken cancellationToken = default);
|
||||
/// <summary>
|
||||
/// Read-only, no-tracking variant of <see cref="GetAllTemplatesAsync"/> for the
|
||||
/// design-time ANALYSIS walks — acyclicity (<c>CycleDetector</c>), naming
|
||||
/// collisions (<c>CollisionDetector</c>) and canonical-name resolution
|
||||
/// (<c>TemplateResolver</c>). Script bodies are NOT loaded
|
||||
/// (<see cref="TemplateScript.Code"/> comes back empty) because no analysis
|
||||
/// consumer reads them; everything those walks do read — ids, parent and
|
||||
/// composition edges, member names and lock flags — is present.
|
||||
///
|
||||
/// <para>
|
||||
/// Do NOT use this for the inheritance reconciler or the flattener: the
|
||||
/// reconciler compares and copies script bodies and writes through tracked
|
||||
/// entities, and the flattener must observe rows an in-flight bundle import
|
||||
/// has staged on the shared change tracker. Both keep using
|
||||
/// <see cref="GetAllTemplatesAsync"/> / <see cref="GetTemplateWithChildrenAsync"/>.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
/// <param name="cancellationToken">Cancellation token.</param>
|
||||
/// <returns>A task that resolves to a read-only list of all templates without script bodies.</returns>
|
||||
Task<IReadOnlyList<Template>> GetAllTemplatesForAnalysisAsync(CancellationToken cancellationToken = default);
|
||||
/// <summary>
|
||||
/// Database-side paged listing of templates as row summaries (child
|
||||
/// collections reduced to counts, no script bodies). Backs the
|
||||
/// <c>ListTemplates</c> management command, which previously materialised
|
||||
/// every template's full child graph and then paged the list in memory.
|
||||
/// </summary>
|
||||
/// <param name="skip">Number of rows to skip; negative values are floored at 0.</param>
|
||||
/// <param name="take">
|
||||
/// Page size. <see langword="null"/> or non-positive returns every remaining
|
||||
/// row (the historical unpaged behaviour); otherwise clamped to
|
||||
/// <see cref="Types.Templates.TemplateSummary.MaxPageSize"/>.
|
||||
/// </param>
|
||||
/// <param name="cancellationToken">Cancellation token.</param>
|
||||
/// <returns>A task that resolves to one page of template summaries, ordered by id.</returns>
|
||||
Task<IReadOnlyList<Types.Templates.TemplateSummary>> GetTemplateSummariesAsync(int skip, int? take, CancellationToken cancellationToken = default);
|
||||
/// <summary>
|
||||
/// Returns every template that contains a composition referencing
|
||||
/// <paramref name="composedTemplateId"/>. Each result is eager-loaded with
|
||||
/// its Attributes / Scripts / Compositions so the caller can build a
|
||||
|
||||
@@ -109,10 +109,70 @@ public interface ISiteAuditQueue
|
||||
/// </remarks>
|
||||
/// <param name="sinceUtc">Lower bound timestamp (UTC).</param>
|
||||
/// <param name="batchSize">Maximum number of rows to return.</param>
|
||||
/// <param name="afterId">
|
||||
/// Composite-keyset tiebreak cursor (WP2.3), mirroring
|
||||
/// <see cref="IOperationTrackingStore.ReadChangedSinceAsync"/>: when non-null it is the
|
||||
/// <see cref="AuditEvent.EventId"/> ("D" GUID form) of the last row already consumed at
|
||||
/// <paramref name="sinceUtc"/>, and only rows strictly after the composite
|
||||
/// <c>(OccurredAtUtc, EventId)</c> pair are returned — so a burst sharing one exact
|
||||
/// instant drains via the id tiebreak instead of pinning the inclusive-timestamp cursor.
|
||||
/// Null (the first pull, or a central that never sets it) keeps the inclusive
|
||||
/// <c>>=</c> contract.
|
||||
/// </param>
|
||||
/// <param name="ct">Cancellation token.</param>
|
||||
/// <returns>A task that resolves to audit events at or after <paramref name="sinceUtc"/> in pending or forwarded state, up to <paramref name="batchSize"/>.</returns>
|
||||
Task<IReadOnlyList<AuditEvent>> ReadPendingSinceAsync(
|
||||
DateTime sinceUtc, int batchSize, CancellationToken ct = default);
|
||||
DateTime sinceUtc, int batchSize, string? afterId = null, CancellationToken ct = default);
|
||||
|
||||
/// <summary>
|
||||
/// Reconciliation-pull commit surface, cursor form (WP2.3): flips every row at or before
|
||||
/// the composite cursor <c>(<paramref name="sinceUtc"/>, <paramref name="afterId"/>)</c>
|
||||
/// from <see cref="ZB.MOM.WW.ScadaBridge.Commons.Types.Enums.AuditForwardState.Pending"/>/
|
||||
/// <see cref="ZB.MOM.WW.ScadaBridge.Commons.Types.Enums.AuditForwardState.Forwarded"/> to
|
||||
/// <see cref="ZB.MOM.WW.ScadaBridge.Commons.Types.Enums.AuditForwardState.Reconciled"/>.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// The cursor is central's proof of receipt: it only advances past rows central has
|
||||
/// actually ingested. Flipping on THAT — rather than on having merely served the rows —
|
||||
/// is what makes the pull at-least-once: a fault between the response leaving the site
|
||||
/// and central committing it leaves the rows unflipped, so the next pull re-serves them
|
||||
/// (central dedups on <see cref="AuditEvent.EventId"/>).
|
||||
/// <para>
|
||||
/// With <paramref name="afterId"/> null the cursor is a bare timestamp under the legacy
|
||||
/// inclusive <c>>=</c> read contract, so only rows STRICTLY older than
|
||||
/// <paramref name="sinceUtc"/> are proven received; rows at the boundary instant are left
|
||||
/// alone. Idempotent; already-Reconciled rows are untouched.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Only-served-rows-retire.</b> The cursor is a timestamp, and
|
||||
/// <see cref="AuditEvent.OccurredAtUtc"/> is stamped by the caller, so a row can be
|
||||
/// INSERTED after a batch was served yet carry a timestamp below the (by then advanced)
|
||||
/// cursor. Implementations MUST NOT retire such a row: it was never served, the keyset
|
||||
/// read has moved past it, and retiring it would make it purgeable — silent audit loss.
|
||||
/// The SQLite implementation bounds the flip by insertion order (<c>rowid</c> high-water
|
||||
/// mark of rows it has served), exempting rows already
|
||||
/// <see cref="ZB.MOM.WW.ScadaBridge.Commons.Types.Enums.AuditForwardState.Forwarded"/>
|
||||
/// because central ACKED those through the telemetry push path.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Liveness dependency (documented, accepted).</b> A row that WAS served but is never
|
||||
/// covered by a later cursor — central reconciliation stopped for good, or the site node
|
||||
/// restarted and its bound reset — stays
|
||||
/// <see cref="ZB.MOM.WW.ScadaBridge.Commons.Types.Enums.AuditForwardState.Pending"/>
|
||||
/// indefinitely, and <see cref="PurgeExpiredAsync"/> never purges Pending. That is the
|
||||
/// retention invariant working as designed (an unacknowledged row is not droppable), but
|
||||
/// it means the site store's floor is bounded by reconciliation actually running. The
|
||||
/// backlog is observable: <see cref="GetBacklogStatsAsync"/> reports the pending count and
|
||||
/// oldest-pending instant on every site health report, and the site reporter logs a
|
||||
/// warning once the oldest pending row exceeds its stale threshold.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
/// <param name="sinceUtc">The cursor timestamp central has consumed up to (UTC).</param>
|
||||
/// <param name="afterId">The last consumed <see cref="AuditEvent.EventId"/> at that instant, or null.</param>
|
||||
/// <param name="ct">Cancellation token.</param>
|
||||
/// <returns>A task that resolves to the number of rows flipped.</returns>
|
||||
Task<int> MarkReconciledUpToAsync(
|
||||
DateTime sinceUtc, string? afterId, CancellationToken ct = default);
|
||||
|
||||
/// <summary>
|
||||
/// Reconciliation-pull commit surface: flips the supplied EventIds to
|
||||
|
||||
@@ -0,0 +1,82 @@
|
||||
namespace ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services;
|
||||
|
||||
/// <summary>
|
||||
/// Process-wide monotonic version watermark over the template/instance
|
||||
/// configuration graph.
|
||||
///
|
||||
/// <para>
|
||||
/// Every mutation that lands through the configuration database bumps the
|
||||
/// version of the affected template (or instance), plus a global counter and —
|
||||
/// for edge-shaped changes (template added/removed, parent or composition edge
|
||||
/// changed) — a separate <see cref="StructureVersion"/>. Consumers use the
|
||||
/// watermark two ways:
|
||||
/// </para>
|
||||
/// <list type="bullet">
|
||||
/// <item>
|
||||
/// as the discriminator in a flatten-session cache key
|
||||
/// (<c>templateId + version</c>), so a memoised template chain can never be
|
||||
/// served after that template changed; and
|
||||
/// </item>
|
||||
/// <item>
|
||||
/// as a staleness fast path — when neither the instance nor any template in
|
||||
/// its chain has moved since a previously computed revision hash, the hash
|
||||
/// is still current and a full re-flatten can be skipped.
|
||||
/// </item>
|
||||
/// </list>
|
||||
///
|
||||
/// <para>
|
||||
/// Versions are in-memory and process-local. They are deliberately NOT
|
||||
/// persisted: a restart (or the standby central node) simply starts from zero,
|
||||
/// which invalidates every cached entry and falls back to the authoritative
|
||||
/// full flatten. The watermark is therefore only ever able to cause EXTRA work,
|
||||
/// never stale work — a correctness property that matters because the value is
|
||||
/// not replicated between the two central nodes.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public interface ITemplateGraphWatermark
|
||||
{
|
||||
/// <summary>
|
||||
/// Monotonic counter bumped on every template- or instance-graph mutation.
|
||||
/// Useful as a coarse "did anything change at all" gate.
|
||||
/// </summary>
|
||||
long Global { get; }
|
||||
|
||||
/// <summary>
|
||||
/// Monotonic counter bumped only when the SHAPE of the graph changes — a
|
||||
/// template row added or removed, a parent-template edge changed, or a
|
||||
/// composition row added/removed/repointed. A cached template-chain
|
||||
/// MEMBERSHIP is valid only while this value is unchanged; the per-template
|
||||
/// versions then cover the contents of each member.
|
||||
/// </summary>
|
||||
long StructureVersion { get; }
|
||||
|
||||
/// <summary>Current version of a single template. Zero when never mutated in this process.</summary>
|
||||
/// <param name="templateId">Template id to read.</param>
|
||||
/// <returns>The template's monotonic version.</returns>
|
||||
long GetTemplateVersion(int templateId);
|
||||
|
||||
/// <summary>Current version of a single instance (its override/binding rows). Zero when never mutated in this process.</summary>
|
||||
/// <param name="instanceId">Instance id to read.</param>
|
||||
/// <returns>The instance's monotonic version.</returns>
|
||||
long GetInstanceVersion(int instanceId);
|
||||
|
||||
/// <summary>Records a mutation of the given template.</summary>
|
||||
/// <param name="templateId">Template whose version is bumped.</param>
|
||||
/// <param name="structural">
|
||||
/// When <c>true</c> the change altered the graph shape (add/remove/parent or
|
||||
/// composition edge) and <see cref="StructureVersion"/> is bumped too.
|
||||
/// </param>
|
||||
void BumpTemplate(int templateId, bool structural = false);
|
||||
|
||||
/// <summary>Records a mutation of the given instance.</summary>
|
||||
/// <param name="instanceId">Instance whose version is bumped.</param>
|
||||
void BumpInstance(int instanceId);
|
||||
|
||||
/// <summary>
|
||||
/// Records a mutation whose affected template/instance could not be
|
||||
/// attributed to a specific id. Bumps <see cref="Global"/> and
|
||||
/// <see cref="StructureVersion"/>, invalidating every chain-membership and
|
||||
/// staleness fast path — the safe, conservative fallback.
|
||||
/// </summary>
|
||||
void BumpAll();
|
||||
}
|
||||
@@ -1,5 +1,19 @@
|
||||
namespace ZB.MOM.WW.ScadaBridge.Commons.Messages.DebugView;
|
||||
|
||||
/// <summary>
|
||||
/// Asks one instance for a point-in-time <see cref="DebugViewSnapshot"/>.
|
||||
/// </summary>
|
||||
/// <param name="InstanceUniqueName">Unique name of the instance to snapshot.</param>
|
||||
/// <param name="CorrelationId">Correlation id echoed on the reply.</param>
|
||||
/// <param name="AlarmsOnly">
|
||||
/// When <c>true</c> the site builds ONLY the alarm half of the snapshot and returns an
|
||||
/// empty <see cref="DebugViewSnapshot.AttributeValues"/> list (wire efficiency, WP2.3).
|
||||
/// Set by the central per-site live alarm cache, whose seed/reconcile fan-out discards
|
||||
/// every attribute row anyway. Defaults to <c>false</c> so the Debug View and every other
|
||||
/// caller keep the full snapshot; additive on the wire
|
||||
/// (<c>DebugSnapshotRequestDto.alarms_only</c>, field 3).
|
||||
/// </param>
|
||||
public record DebugSnapshotRequest(
|
||||
string InstanceUniqueName,
|
||||
string CorrelationId);
|
||||
string CorrelationId,
|
||||
bool AlarmsOnly = false);
|
||||
|
||||
@@ -1,7 +1,27 @@
|
||||
namespace ZB.MOM.WW.ScadaBridge.Commons.Messages.Health;
|
||||
|
||||
/// <summary>
|
||||
/// Application-level heartbeat a site node sends to central on its heartbeat interval.
|
||||
/// Central's health aggregator stamps the site's last-seen time from it, and the offline
|
||||
/// timeout fires when they stop landing.
|
||||
/// </summary>
|
||||
/// <param name="SiteId">The reporting site's identifier.</param>
|
||||
/// <param name="NodeHostname">The reporting node's hostname.</param>
|
||||
/// <param name="IsActive">Whether the reporting node is the site's active (oldest-Up) node.</param>
|
||||
/// <param name="Timestamp">When the heartbeat was produced (UTC).</param>
|
||||
/// <param name="Synthetic">
|
||||
/// <c>true</c> marks a heartbeat that carries NO liveness meaning and that central must skip for
|
||||
/// all health bookkeeping — it neither stamps a last-seen time nor replicates to the peer node.
|
||||
/// The only producer today is <c>CentralChannelProvider</c>'s failback probe, which reuses the
|
||||
/// heartbeat RPC purely to ask "does the preferred central endpoint answer again?"; it is emitted
|
||||
/// by a SITE's transport layer, not by any node's heartbeat timer, and counting it as liveness
|
||||
/// would keep a site looking alive on the health dashboard from the probe alone. Additive with a
|
||||
/// <c>false</c> default (and proto field 5 on <c>HeartbeatDto</c>), so every real heartbeat —
|
||||
/// including one from a node that predates the flag — is unaffected.
|
||||
/// </param>
|
||||
public record HeartbeatMessage(
|
||||
string SiteId,
|
||||
string NodeHostname,
|
||||
bool IsActive,
|
||||
DateTimeOffset Timestamp);
|
||||
DateTimeOffset Timestamp,
|
||||
bool Synthetic = false);
|
||||
|
||||
@@ -74,6 +74,25 @@ public record SiteHealthReport(
|
||||
/// </summary>
|
||||
public double? ScriptOldestBusyAgeSeconds { get; init; }
|
||||
|
||||
/// <summary>
|
||||
/// WP3.1: script-execution worker threads the stuck-script watchdog has DETACHED and
|
||||
/// replaced — a script wedged past its deadline plus grace in uninterruptible blocking
|
||||
/// I/O, so the pool started a fresh thread and the wedged one will exit only when its
|
||||
/// body finally returns. Point-in-time, refreshed by <c>ScriptSchedulerStatsReporter</c>.
|
||||
/// Zero is the healthy state; a value that climbs and never drains means script bodies
|
||||
/// are permanently consuming threads and the pool is being repeatedly rebuilt around them.
|
||||
/// </summary>
|
||||
public int DetachedScriptThreads { get; init; }
|
||||
|
||||
/// <summary>
|
||||
/// WP3.1: per-interval count of script and alarm on-trigger runs SHED because
|
||||
/// <c>MaxConcurrentRunsPerScript</c> runs were already in flight for that script. Raw
|
||||
/// per-interval count (drained on collect) like <see cref="ScriptErrorCount"/>. A
|
||||
/// sustained non-zero value means a trigger is firing faster than its script completes;
|
||||
/// the shed itself is the designed back-pressure, not an error.
|
||||
/// </summary>
|
||||
public int ScriptRunShedCount { get; init; }
|
||||
|
||||
// LocalDb 2-node replication of the consolidated site database (Phase 1).
|
||||
// Additive init properties for the same reason as the scheduler gauges above:
|
||||
// the positional constructor stays untouched. Refreshed on the site by
|
||||
@@ -104,6 +123,17 @@ public record SiteHealthReport(
|
||||
/// pair as perfectly healthy. Collapsing null to 0 anywhere on this path is a bug.
|
||||
/// </remarks>
|
||||
public long? LocalDbOplogBacklog { get; init; }
|
||||
|
||||
/// <summary>
|
||||
/// WP2.6d (arch-review misc — site stream alarm-vs-attribute shared buffer):
|
||||
/// cumulative count of alarm state changes dropped from
|
||||
/// <c>SiteStreamManager</c>'s dedicated alarm publish queue because it was at
|
||||
/// capacity — i.e. the site-wide alarm hand-off itself fell behind, not a
|
||||
/// per-subscriber buffer. Point-in-time (not reset on collect), refreshed by
|
||||
/// the site-side <c>SiteStreamAlarmDropReporter</c> hosted service. Defaults to
|
||||
/// 0 so existing producers/tests that don't populate it stay valid.
|
||||
/// </summary>
|
||||
public long SiteStreamAlarmDropCount { get; init; }
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
|
||||
@@ -1,5 +1,26 @@
|
||||
namespace ZB.MOM.WW.ScadaBridge.Commons.Messages.Management;
|
||||
|
||||
public record MgmtDeployArtifactsCommand(int? SiteId = null);
|
||||
|
||||
/// <summary>
|
||||
/// Deploys every deployable instance at one site in a single operation.
|
||||
///
|
||||
/// <para>
|
||||
/// Distinct from <see cref="MgmtDeployArtifactsCommand"/>, which ships the
|
||||
/// SYSTEM-WIDE artifact sets (shared scripts, external systems, DB/data
|
||||
/// connections) to a site. This command deploys the site's INSTANCE
|
||||
/// configurations — the same per-instance pipeline
|
||||
/// <c>MgmtDeployInstanceCommand</c> runs, batched.
|
||||
/// </para>
|
||||
///
|
||||
/// <para>
|
||||
/// Not all-or-nothing: each instance keeps its own deployment id, revision hash,
|
||||
/// operation lock and status record, and a failure on one is reported as a failed
|
||||
/// row while the rest proceed. Individually retryable via the single-instance
|
||||
/// deploy. Additive contract; new fields go on the end with a default.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
/// <param name="SiteId">Database id of the target site. Required — this command is never fleet-wide.</param>
|
||||
public record MgmtDeploySiteCommand(int SiteId);
|
||||
public record QueryDeploymentsCommand(int? InstanceId = null, string? Status = null, int Page = 1, int PageSize = 50);
|
||||
public record GetDeploymentDiffCommand(int InstanceId);
|
||||
|
||||
@@ -64,6 +64,35 @@ public static class ScadaBridgeTelemetry
|
||||
Meter.CreateCounter<long>("scadabridge.site.alarm_cache.reconnects", unit: "1",
|
||||
description: "Live-alarm aggregator site-wide gRPC stream reconnects (NodeA↔NodeB flip or reconcile-driven reopen).");
|
||||
|
||||
/// <summary>
|
||||
/// Incremented for each live delta evicted from a per-site live-alarm aggregator's
|
||||
/// bounded pre-seed buffer (drop-oldest, WP2.3). Non-zero means a seed/reconcile
|
||||
/// fan-out ran long enough for the delta storm behind it to exceed the cap — the
|
||||
/// dropped transitions are recovered by the fan-out's authoritative snapshot, but a
|
||||
/// sustained climb points at a slow site.
|
||||
/// </summary>
|
||||
private static readonly Counter<long> _liveAlarmBufferDrops =
|
||||
Meter.CreateCounter<long>("scadabridge.site.alarm_cache.buffer_dropped", unit: "1",
|
||||
description: "Live deltas evicted from a live-alarm aggregator's bounded pre-seed buffer (drop-oldest).");
|
||||
|
||||
/// <summary>
|
||||
/// Incremented for each debug event evicted from a central debug session's bounded
|
||||
/// pre-snapshot buffer (drop-oldest, WP2.3). The Debug View is lossy-under-backpressure
|
||||
/// by design; this makes the loss measurable instead of unbounded memory growth.
|
||||
/// </summary>
|
||||
private static readonly Counter<long> _debugPreSnapshotDrops =
|
||||
Meter.CreateCounter<long>("scadabridge.central.debug_view.presnapshot_dropped", unit: "1",
|
||||
description: "Debug events evicted from a central debug session's bounded pre-snapshot buffer (drop-oldest).");
|
||||
|
||||
/// <summary>
|
||||
/// Incremented for each event evicted from a site-hosted gRPC stream's bounded send
|
||||
/// channel, tagged by stream kind (<c>instance</c> = Debug View, <c>site-alarms</c> =
|
||||
/// the site-wide alarm feed behind the operator Alarm Summary).
|
||||
/// </summary>
|
||||
private static readonly Counter<long> _siteStreamEventDrops =
|
||||
Meter.CreateCounter<long>("scadabridge.site.stream.events_dropped", unit: "1",
|
||||
description: "Events evicted from a site gRPC stream's bounded send channel, tagged by stream kind.");
|
||||
|
||||
// ---------------- Observable gauges ----------------
|
||||
|
||||
/// <summary>Current count of open site connections, mutated via <see cref="Interlocked"/>.</summary>
|
||||
@@ -136,6 +165,19 @@ public static class ScadaBridgeTelemetry
|
||||
/// <summary>Records that a per-site live-alarm aggregator re-established its site-wide gRPC stream.</summary>
|
||||
public static void RecordLiveAlarmStreamReconnect() => _liveAlarmStreamReconnects.Add(1);
|
||||
|
||||
/// <summary>Records live deltas evicted from a live-alarm aggregator's bounded pre-seed buffer.</summary>
|
||||
/// <param name="count">Number of deltas evicted.</param>
|
||||
public static void RecordLiveAlarmBufferDrop(long count = 1) => _liveAlarmBufferDrops.Add(count);
|
||||
|
||||
/// <summary>Records debug events evicted from a debug session's bounded pre-snapshot buffer.</summary>
|
||||
/// <param name="count">Number of events evicted.</param>
|
||||
public static void RecordDebugPreSnapshotDrop(long count = 1) => _debugPreSnapshotDrops.Add(count);
|
||||
|
||||
/// <summary>Records an event evicted from a site gRPC stream's bounded send channel.</summary>
|
||||
/// <param name="streamKind">Stream kind tag (<c>instance</c> or <c>site-alarms</c>).</param>
|
||||
public static void RecordSiteStreamEventDropped(string streamKind) =>
|
||||
_siteStreamEventDrops.Add(1, new KeyValuePair<string, object?>("stream", streamKind));
|
||||
|
||||
/// <summary>
|
||||
/// Registers the provider the StoreAndForward queue-depth gauge reads on each observation.
|
||||
/// A later task supplies a provider that reads the real StoreAndForward depth. A null
|
||||
|
||||
@@ -195,6 +195,7 @@ public static class OpcUaEndpointConfigSerializer
|
||||
["DiscardOldest"] = config.DiscardOldest.ToString(),
|
||||
["SubscriptionPriority"] = config.SubscriptionPriority.ToString(),
|
||||
["SubscriptionDisplayName"] = config.SubscriptionDisplayName,
|
||||
["MaxMonitoredItemsPerSubscription"] = config.MaxMonitoredItemsPerSubscription.ToString(),
|
||||
["TimestampsToReturn"] = config.TimestampsToReturn.ToString(),
|
||||
};
|
||||
if (config.Heartbeat is { } hb)
|
||||
@@ -248,6 +249,7 @@ public static class OpcUaEndpointConfigSerializer
|
||||
TryAssignInt(dict, "KeepAliveCount", v => c.KeepAliveCount = v);
|
||||
TryAssignInt(dict, "LifetimeCount", v => c.LifetimeCount = v);
|
||||
TryAssignInt(dict, "MaxNotificationsPerPublish", v => c.MaxNotificationsPerPublish = v);
|
||||
TryAssignInt(dict, "MaxMonitoredItemsPerSubscription", v => c.MaxMonitoredItemsPerSubscription = v);
|
||||
|
||||
if (dict.TryGetValue("DiscardOldest", out var doStr) && bool.TryParse(doStr, out var doVal))
|
||||
c.DiscardOldest = doVal;
|
||||
|
||||
@@ -66,6 +66,14 @@ public sealed class OpcUaEndpointConfig
|
||||
/// Display name for the subscription.
|
||||
/// </summary>
|
||||
public string SubscriptionDisplayName { get; set; } = "ScadaBridge";
|
||||
/// <summary>
|
||||
/// Maximum monitored items placed on a single OPC UA subscription before the
|
||||
/// adapter shards onto an additional subscription. Item-count ceilings are a
|
||||
/// property of the target server, so this is per endpoint rather than global.
|
||||
/// A site sized at 37,500 tags shards into 8 subscriptions at the default.
|
||||
/// Values <= 0 are treated as the default.
|
||||
/// </summary>
|
||||
public int MaxMonitoredItemsPerSubscription { get; set; } = 5000;
|
||||
|
||||
// Read / filter
|
||||
/// <summary>
|
||||
|
||||
@@ -0,0 +1,54 @@
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.Commons.Types.Deployment;
|
||||
|
||||
/// <summary>
|
||||
/// Row-shaped projection of a <c>DeploymentRecord</c> for list surfaces (CLI
|
||||
/// <c>deploy list</c> / <c>instance</c> history, the Central UI deployment-status
|
||||
/// page).
|
||||
///
|
||||
/// <para>
|
||||
/// Exists so a deployment listing can be paged and filtered in the DATABASE.
|
||||
/// Deployment records are insert-only — one row per deploy attempt, retained for
|
||||
/// the configured retention window — so the previous "load the whole table, then
|
||||
/// filter in memory" read path degraded with the age of the system rather than
|
||||
/// with the size of the requested page.
|
||||
/// </para>
|
||||
///
|
||||
/// <para>
|
||||
/// The <c>RowVersion</c> optimistic-concurrency token is deliberately absent: this
|
||||
/// is a READ projection, and a summary must never be mistaken for something that
|
||||
/// can be written back. Mutating paths keep loading the tracked entity.
|
||||
/// </para>
|
||||
///
|
||||
/// <para>
|
||||
/// Message-contract evolution rule: additive-only. New fields go on the end with a
|
||||
/// default.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
/// <param name="Id">Deployment record row id.</param>
|
||||
/// <param name="DeploymentId">The logical deployment id (GUID, "N" format).</param>
|
||||
/// <param name="InstanceId">Instance the deployment targeted.</param>
|
||||
/// <param name="Status">Terminal or in-flight deployment status.</param>
|
||||
/// <param name="RevisionHash">Revision hash of the deployed configuration.</param>
|
||||
/// <param name="DeployedBy">User who initiated the deployment.</param>
|
||||
/// <param name="DeployedAt">When the deployment was initiated.</param>
|
||||
/// <param name="CompletedAt">When the deployment reached a terminal status, if it has.</param>
|
||||
/// <param name="ErrorMessage">Failure detail when the deployment did not succeed.</param>
|
||||
public record DeploymentRecordSummary(
|
||||
int Id,
|
||||
string DeploymentId,
|
||||
int InstanceId,
|
||||
DeploymentStatus Status,
|
||||
string? RevisionHash,
|
||||
string DeployedBy,
|
||||
DateTimeOffset DeployedAt,
|
||||
DateTimeOffset? CompletedAt,
|
||||
string? ErrorMessage)
|
||||
{
|
||||
/// <summary>
|
||||
/// Hard ceiling applied to a requested page size, mirroring the in-memory
|
||||
/// <c>Page</c> clamp the management actor already applies elsewhere.
|
||||
/// </summary>
|
||||
public const int MaxPageSize = 1000;
|
||||
}
|
||||
@@ -0,0 +1,72 @@
|
||||
using System.Collections.Concurrent;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.Commons.Types.Templates;
|
||||
|
||||
/// <summary>
|
||||
/// In-memory <see cref="ITemplateGraphWatermark"/>. Registered as a singleton so
|
||||
/// the writer (the configuration-database repository's unit-of-work commit) and
|
||||
/// the readers (the flatten-session cache, the staleness fast path) observe the
|
||||
/// same counters within a process.
|
||||
///
|
||||
/// <para>
|
||||
/// Lives in Commons rather than a component project because both the
|
||||
/// ConfigurationDatabase writer and the DeploymentManager readers need it and
|
||||
/// neither references the other. It is a pure in-memory primitive with no
|
||||
/// infrastructure dependency, so it does not violate the "implementations live
|
||||
/// with their component" convention the way a repository would.
|
||||
/// </para>
|
||||
///
|
||||
/// <para>
|
||||
/// All counters are <see cref="Interlocked"/>-updated and the per-id maps are
|
||||
/// <see cref="ConcurrentDictionary{TKey,TValue}"/>, so concurrent management
|
||||
/// commands (which each run on their own DI scope) can bump safely. Reads are
|
||||
/// deliberately unsynchronised relative to each other: a reader that observes a
|
||||
/// half-applied batch of bumps sees a CHANGED value for at least one member,
|
||||
/// which invalidates rather than falsely validates a cache entry.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public sealed class TemplateGraphWatermark : ITemplateGraphWatermark
|
||||
{
|
||||
private readonly ConcurrentDictionary<int, long> _templateVersions = new();
|
||||
private readonly ConcurrentDictionary<int, long> _instanceVersions = new();
|
||||
private long _global;
|
||||
private long _structure;
|
||||
|
||||
/// <inheritdoc />
|
||||
public long Global => Interlocked.Read(ref _global);
|
||||
|
||||
/// <inheritdoc />
|
||||
public long StructureVersion => Interlocked.Read(ref _structure);
|
||||
|
||||
/// <inheritdoc />
|
||||
public long GetTemplateVersion(int templateId) =>
|
||||
_templateVersions.TryGetValue(templateId, out var v) ? v : 0L;
|
||||
|
||||
/// <inheritdoc />
|
||||
public long GetInstanceVersion(int instanceId) =>
|
||||
_instanceVersions.TryGetValue(instanceId, out var v) ? v : 0L;
|
||||
|
||||
/// <inheritdoc />
|
||||
public void BumpTemplate(int templateId, bool structural = false)
|
||||
{
|
||||
_templateVersions.AddOrUpdate(templateId, 1L, static (_, current) => current + 1);
|
||||
Interlocked.Increment(ref _global);
|
||||
if (structural)
|
||||
Interlocked.Increment(ref _structure);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public void BumpInstance(int instanceId)
|
||||
{
|
||||
_instanceVersions.AddOrUpdate(instanceId, 1L, static (_, current) => current + 1);
|
||||
Interlocked.Increment(ref _global);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public void BumpAll()
|
||||
{
|
||||
Interlocked.Increment(ref _global);
|
||||
Interlocked.Increment(ref _structure);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,53 @@
|
||||
namespace ZB.MOM.WW.ScadaBridge.Commons.Types.Templates;
|
||||
|
||||
/// <summary>
|
||||
/// Row-shaped projection of a template for list surfaces (CLI <c>template list</c>,
|
||||
/// the Central UI template tree, the <c>ListTemplates</c> management command).
|
||||
///
|
||||
/// <para>
|
||||
/// Exists so a template listing can be paged in the DATABASE rather than by
|
||||
/// materialising every template's full child graph — attributes, alarms, script
|
||||
/// bodies, compositions and native alarm sources — and then discarding all but
|
||||
/// one page of it in memory. Child collections are reduced to counts, which is
|
||||
/// what every list surface actually renders.
|
||||
/// </para>
|
||||
///
|
||||
/// <para>
|
||||
/// Message-contract evolution rule: this record is additive-only. New fields go
|
||||
/// on the end with a default so an older peer deserialising a newer payload is
|
||||
/// unaffected.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
/// <param name="Id">Template id.</param>
|
||||
/// <param name="Name">Template name.</param>
|
||||
/// <param name="Description">Optional description.</param>
|
||||
/// <param name="ParentTemplateId">Parent template id when this template inherits.</param>
|
||||
/// <param name="FolderId">Containing folder id.</param>
|
||||
/// <param name="IsDerived">True when the template was auto-derived to back a composition slot.</param>
|
||||
/// <param name="OwnerCompositionId">Owning composition row for a derived template.</param>
|
||||
/// <param name="AttributeCount">Number of directly declared attributes.</param>
|
||||
/// <param name="AlarmCount">Number of directly declared alarms.</param>
|
||||
/// <param name="ScriptCount">Number of directly declared scripts.</param>
|
||||
/// <param name="CompositionCount">Number of composition slots.</param>
|
||||
/// <param name="NativeAlarmSourceCount">Number of native alarm source bindings.</param>
|
||||
public record TemplateSummary(
|
||||
int Id,
|
||||
string Name,
|
||||
string? Description,
|
||||
int? ParentTemplateId,
|
||||
int? FolderId,
|
||||
bool IsDerived,
|
||||
int? OwnerCompositionId,
|
||||
int AttributeCount,
|
||||
int AlarmCount,
|
||||
int ScriptCount,
|
||||
int CompositionCount,
|
||||
int NativeAlarmSourceCount)
|
||||
{
|
||||
/// <summary>
|
||||
/// Hard ceiling applied to a requested page size, mirroring the existing
|
||||
/// in-memory <c>Page</c> clamp in the management actor so DB-side paging
|
||||
/// cannot be used to pull an unbounded result set.
|
||||
/// </summary>
|
||||
public const int MaxPageSize = 1000;
|
||||
}
|
||||
@@ -67,36 +67,6 @@ public class CentralCommunicationActor : ReceiveActor
|
||||
/// </summary>
|
||||
private IActorRef? _notificationOutboxProxy;
|
||||
|
||||
/// <summary>
|
||||
/// Proxy <see cref="IActorRef"/> for the central AuditLogIngestActor cluster
|
||||
/// singleton. Set via <see cref="RegisterAuditIngest"/> — the Host creates the
|
||||
/// singleton proxy after this actor and registers it (mirrors
|
||||
/// <see cref="_notificationOutboxProxy"/>). Null until registration completes;
|
||||
/// an audit ingest command arriving before then is answered with an empty
|
||||
/// reply so the site keeps its rows Pending and retries.
|
||||
///
|
||||
/// Once registered, the handler Asks this proxy and pipes the reply straight
|
||||
/// back to the caller. On an Ask timeout or a faulted reply, PipeTo forwards a
|
||||
/// <see cref="Status.Failure"/> to the caller — the fault propagates rather
|
||||
/// than being swallowed. This differs from the gRPC handler
|
||||
/// (<c>SiteStreamGrpcServer</c>), which catches the exception and returns an
|
||||
/// empty ack; here the faulted Ask is the transient signal the site relies on
|
||||
/// (see <see cref="HandleIngestAuditEvents"/>).
|
||||
/// </summary>
|
||||
private IActorRef? _auditIngestProxy;
|
||||
|
||||
/// <summary>
|
||||
/// Default Ask timeout for routing audit ingest commands to the
|
||||
/// Effective Ask timeout for audit ingest routing. Defaults to
|
||||
/// <see cref="Grpc.SiteStreamGrpcServer.AuditIngestAskTimeout"/> (30 s) — the two
|
||||
/// audit-ingest entry points (the site stream server and the control plane) share one source of truth
|
||||
/// for the timeout. Overridable via the constructor so tests can exercise the
|
||||
/// timeout/fault path without waiting 30 s. When the window is exceeded the Ask
|
||||
/// faults and that fault is piped back to the caller as a
|
||||
/// <see cref="Status.Failure"/> (see <see cref="HandleIngestAuditEvents"/>).
|
||||
/// </summary>
|
||||
private readonly TimeSpan _auditIngestAskTimeout;
|
||||
|
||||
/// <summary>
|
||||
/// DistributedPubSub topic used to fan health reports out to the peer
|
||||
/// central node so both per-node aggregators stay in sync. See
|
||||
@@ -111,12 +81,10 @@ public class CentralCommunicationActor : ReceiveActor
|
||||
/// </summary>
|
||||
/// <param name="serviceProvider">DI service provider for scoped repository and aggregator access.</param>
|
||||
/// <param name="transport">The central→site command transport to route every <see cref="SiteEnvelope"/> through.</param>
|
||||
/// <param name="auditIngestAskTimeout">Optional override for the audit-ingest Ask timeout (test hook).</param>
|
||||
public CentralCommunicationActor(
|
||||
IServiceProvider serviceProvider,
|
||||
ISiteCommandTransport transport,
|
||||
TimeSpan? auditIngestAskTimeout = null)
|
||||
: this(serviceProvider, auditIngestAskTimeout)
|
||||
ISiteCommandTransport transport)
|
||||
: this(serviceProvider)
|
||||
{
|
||||
_transport = transport ?? throw new ArgumentNullException(nameof(transport));
|
||||
}
|
||||
@@ -125,13 +93,9 @@ public class CentralCommunicationActor : ReceiveActor
|
||||
/// <see cref="_transport"/> is assigned by the delegating public constructor before any message
|
||||
/// is dispatched.</summary>
|
||||
/// <param name="serviceProvider">DI service provider.</param>
|
||||
/// <param name="auditIngestAskTimeout">Optional audit-ingest Ask timeout override.</param>
|
||||
private CentralCommunicationActor(
|
||||
IServiceProvider serviceProvider,
|
||||
TimeSpan? auditIngestAskTimeout)
|
||||
private CentralCommunicationActor(IServiceProvider serviceProvider)
|
||||
{
|
||||
_serviceProvider = serviceProvider;
|
||||
_auditIngestAskTimeout = auditIngestAskTimeout ?? Grpc.SiteStreamGrpcServer.AuditIngestAskTimeout;
|
||||
|
||||
// Site address cache loaded from database
|
||||
Receive<SiteAddressCacheLoaded>(HandleSiteAddressCacheLoaded);
|
||||
@@ -176,24 +140,11 @@ public class CentralCommunicationActor : ReceiveActor
|
||||
// so the NotificationStatusResponse routes back to the querying site.
|
||||
Receive<NotificationStatusQuery>(HandleNotificationStatusQuery);
|
||||
|
||||
// Audit Log: the Host registers the AuditLogIngestActor singleton
|
||||
// proxy after this actor is created (the proxy cannot exist before this
|
||||
// actor's construction).
|
||||
Receive<RegisterAuditIngest>(msg =>
|
||||
{
|
||||
_auditIngestProxy = msg.AuditIngestActor;
|
||||
_log.Info("Registered audit ingest proxy");
|
||||
});
|
||||
|
||||
// Audit Log site→central ingest: a site forwards a batch of audit
|
||||
// events to the central cluster. Ask the ingest proxy
|
||||
// and pipe the IngestAuditEventsReply back to the original Sender (the
|
||||
// site's transport path) so the site can flip its rows to Forwarded.
|
||||
Receive<IngestAuditEventsCommand>(HandleIngestAuditEvents);
|
||||
|
||||
// Audit Log combined-telemetry ingest: routes to the same proxy
|
||||
// the same way; the proxy replies with an IngestCachedTelemetryReply.
|
||||
Receive<IngestCachedTelemetryCommand>(HandleIngestCachedTelemetry);
|
||||
// Audit Log site→central ingest is NOT relayed here. The central-hosted
|
||||
// CentralControlGrpcService Asks the audit-log-ingest singleton proxy directly
|
||||
// (as SiteStreamGrpcServer always has), so this actor no longer sits in that
|
||||
// path — the relay was a second hop with an identical 30 s Ask timeout that
|
||||
// could only add latency. Do not reintroduce it.
|
||||
|
||||
// Startup reconciliation: a site node forwards its local deployed inventory on
|
||||
// startup. Resolve the scoped ReconcileService, diff the
|
||||
@@ -239,51 +190,6 @@ public class CentralCommunicationActor : ReceiveActor
|
||||
_notificationOutboxProxy.Forward(msg);
|
||||
}
|
||||
|
||||
private void HandleIngestAuditEvents(IngestAuditEventsCommand msg)
|
||||
{
|
||||
if (_auditIngestProxy == null)
|
||||
{
|
||||
// No ingest proxy registered yet (host startup race). Reply with an
|
||||
// empty IngestAuditEventsReply so the site keeps its rows Pending and
|
||||
// retries — the same behaviour as the gRPC handler's wiring-race path.
|
||||
_log.Warning(
|
||||
"Cannot route IngestAuditEventsCommand ({0} events) — audit ingest not available",
|
||||
msg.Events.Count);
|
||||
Sender.Tell(new IngestAuditEventsReply(Array.Empty<Guid>()));
|
||||
return;
|
||||
}
|
||||
|
||||
// Capture Sender before the async/PipeTo — Akka resets Sender between
|
||||
// dispatches. The reply is piped straight back to the calling site node.
|
||||
// On an Ask timeout or a faulted reply, PipeTo delivers a Status.Failure to
|
||||
// replyTo: the fault propagates to the caller rather than being swallowed.
|
||||
// The site's own Ask through this path then faults, and the site drain loop
|
||||
// treats that as a transient failure — rows stay Pending and are retried on
|
||||
// the next tick. (The gRPC handler instead returns an empty ack on fault;
|
||||
// propagating the fault here is the cleaner transient signal.)
|
||||
var replyTo = Sender;
|
||||
_log.Debug("Routing IngestAuditEventsCommand ({0} events) to the audit ingest actor", msg.Events.Count);
|
||||
_auditIngestProxy.Ask<IngestAuditEventsReply>(msg, _auditIngestAskTimeout)
|
||||
.PipeTo(replyTo);
|
||||
}
|
||||
|
||||
private void HandleIngestCachedTelemetry(IngestCachedTelemetryCommand msg)
|
||||
{
|
||||
if (_auditIngestProxy == null)
|
||||
{
|
||||
_log.Warning(
|
||||
"Cannot route IngestCachedTelemetryCommand ({0} entries) — audit ingest not available",
|
||||
msg.Entries.Count);
|
||||
Sender.Tell(new IngestCachedTelemetryReply(Array.Empty<Guid>()));
|
||||
return;
|
||||
}
|
||||
|
||||
var replyTo = Sender;
|
||||
_log.Debug("Routing IngestCachedTelemetryCommand ({0} entries) to the audit ingest actor", msg.Entries.Count);
|
||||
_auditIngestProxy.Ask<IngestCachedTelemetryReply>(msg, _auditIngestAskTimeout)
|
||||
.PipeTo(replyTo);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Startup reconciliation (site→central): resolve the scoped
|
||||
/// <see cref="ReconcileService"/> in a DI scope, diff the node's reported inventory
|
||||
@@ -291,7 +197,7 @@ public class CentralCommunicationActor : ReceiveActor
|
||||
/// back to the site node's transport path. The actor stays thin — all the diff
|
||||
/// and staging logic lives in the service. Mirrors the DB-access pattern used by
|
||||
/// <see cref="LoadSiteAddressesFromDb"/> (Task.Run + CreateScope + PipeTo) and the
|
||||
/// Sender-preservation pattern of <see cref="HandleIngestAuditEvents"/>.
|
||||
/// Sender-preservation pattern of <see cref="HandleNotificationSubmit"/>.
|
||||
///
|
||||
/// On a faulted task PipeTo delivers a <see cref="Status.Failure"/> to the node; its
|
||||
/// Ask faults and it simply retries reconcile on the next startup — reconcile is
|
||||
@@ -328,6 +234,21 @@ public class CentralCommunicationActor : ReceiveActor
|
||||
|
||||
private void HandleHeartbeat(HeartbeatMessage heartbeat)
|
||||
{
|
||||
// Synthetic heartbeats carry no liveness meaning — today only
|
||||
// CentralChannelProvider's failback probe, which reuses this RPC to ask whether the
|
||||
// preferred central endpoint answers again. It is emitted by a site's TRANSPORT layer,
|
||||
// not by any node's heartbeat timer, so stamping it would let a site whose real
|
||||
// heartbeats had stopped keep looking alive on the health dashboard for as long as its
|
||||
// transport kept probing. Dropped before the local mark AND before the peer fan-out,
|
||||
// so neither central node's aggregator ever sees it.
|
||||
if (heartbeat.Synthetic)
|
||||
{
|
||||
_log.Debug(
|
||||
"Ignoring synthetic heartbeat from site {0} (node '{1}') — probe traffic, not liveness",
|
||||
heartbeat.SiteId, heartbeat.NodeHostname);
|
||||
return;
|
||||
}
|
||||
|
||||
MarkHeartbeatLocally(heartbeat);
|
||||
|
||||
// Fan the heartbeat out to the peer central node so BOTH aggregators mark
|
||||
@@ -350,9 +271,17 @@ public class CentralCommunicationActor : ReceiveActor
|
||||
/// <summary>
|
||||
/// Marks a site heartbeat on the local aggregator without re-broadcasting.
|
||||
/// Used for both site-originated heartbeats and peer-replicated ones.
|
||||
/// A synthetic heartbeat is never marked — <see cref="HandleHeartbeat"/> already drops it
|
||||
/// before the fan-out, so a replica can only carry one if a peer node predates the flag;
|
||||
/// the guard here is belt-and-braces on the last hop before the aggregator.
|
||||
/// </summary>
|
||||
private void MarkHeartbeatLocally(HeartbeatMessage heartbeat)
|
||||
{
|
||||
if (heartbeat.Synthetic)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
var aggregator = _serviceProvider.GetService<ICentralHealthAggregator>();
|
||||
aggregator?.MarkHeartbeat(heartbeat.SiteId, heartbeat.Timestamp);
|
||||
}
|
||||
@@ -623,13 +552,7 @@ public record DebugStreamTerminated(string SiteId, string CorrelationId);
|
||||
/// </summary>
|
||||
public record RegisterNotificationOutbox(IActorRef OutboxProxy);
|
||||
|
||||
/// <summary>
|
||||
/// Registers the central AuditLogIngestActor singleton proxy with the
|
||||
/// <see cref="CentralCommunicationActor"/> so site-forwarded
|
||||
/// <see cref="IngestAuditEventsCommand"/> and <see cref="IngestCachedTelemetryCommand"/>
|
||||
/// messages can be routed to it. Sent by the Host after the audit-ingest
|
||||
/// singleton proxy is created. Lives here (not in Commons) because
|
||||
/// <c>ZB.MOM.WW.ScadaBridge.Commons</c> has no Akka package reference and cannot hold an
|
||||
/// <see cref="IActorRef"/> field.
|
||||
/// </summary>
|
||||
public sealed record RegisterAuditIngest(IActorRef AuditIngestActor);
|
||||
// NOTE: there is deliberately no RegisterAuditIngest counterpart. The audit-log-ingest
|
||||
// singleton proxy goes STRAIGHT to the two gRPC servers that need it
|
||||
// (CentralControlGrpcService.SetAuditIngestActor / SiteStreamGrpcServer.SetAuditIngestActor);
|
||||
// routing audit batches through this actor was a redundant hop and was removed.
|
||||
|
||||
@@ -2,6 +2,7 @@ using Akka.Actor;
|
||||
using Akka.Event;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Messages.DebugView;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Streaming;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Observability;
|
||||
using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.Communication.Actors;
|
||||
@@ -45,9 +46,20 @@ public class DebugStreamBridgeActor : ReceiveActor, IWithTimers
|
||||
private const int MaxRetries = 3;
|
||||
private const string ReconnectTimerKey = "grpc-reconnect";
|
||||
private const string StabilityTimerKey = "grpc-stability";
|
||||
private const string SnapshotTimerKey = "debug-snapshot-deadline";
|
||||
private const string ConsumerLivenessTimerKey = "debug-consumer-liveness";
|
||||
/// <summary>Delay between gRPC reconnection attempts.</summary>
|
||||
internal static TimeSpan ReconnectDelay { get; set; } = TimeSpan.FromSeconds(5);
|
||||
|
||||
/// <summary>
|
||||
/// Hard deadline on the initial <see cref="DebugViewSnapshot"/> (WP2.3). The site builds
|
||||
/// it in milliseconds; if none arrives inside this window the site never answered (the
|
||||
/// Ask was lost, the singleton moved mid-request, the instance actor is wedged) and the
|
||||
/// session must FAIL rather than sit in the buffering phase accumulating live events
|
||||
/// behind a snapshot that is never coming. Settable for tests.
|
||||
/// </summary>
|
||||
internal static TimeSpan SnapshotTimeout { get; set; } = TimeSpan.FromSeconds(60);
|
||||
|
||||
/// <summary>
|
||||
/// How long a freshly-opened gRPC stream must stay up before its retry budget
|
||||
/// is considered "recovered" and <see cref="_retryCount"/> is reset to 0.
|
||||
@@ -58,11 +70,46 @@ public class DebugStreamBridgeActor : ReceiveActor, IWithTimers
|
||||
/// </summary>
|
||||
internal static TimeSpan StabilityWindow { get; set; } = TimeSpan.FromSeconds(60);
|
||||
|
||||
/// <summary>
|
||||
/// Orphan window: how long the session may go without ANY sign of life from its CONSUMER
|
||||
/// before it self-terminates. Renewed by <see cref="DebugStreamConsumerAlive"/>, which
|
||||
/// <c>DebugStreamService</c> Tells on a timer for every session still attached to a
|
||||
/// consumer (Blazor debug view or the SignalR hub) — so it measures the consumer, never
|
||||
/// the stream. Settable for tests.
|
||||
/// </summary>
|
||||
internal static TimeSpan ConsumerIdleTimeout { get; set; } = TimeSpan.FromMinutes(5);
|
||||
|
||||
/// <summary>
|
||||
/// How often the actor checks the consumer-last-seen stamp against
|
||||
/// <see cref="ConsumerIdleTimeout"/>. A self-tick rather than <c>SetReceiveTimeout</c>:
|
||||
/// the receive timeout measures the MAILBOX, which conflates site chatter with consumer
|
||||
/// liveness — and once stream events were correctly excluded from it (via
|
||||
/// <see cref="LiveDebugStreamEvent"/>) nothing recurring reset it at all, so every healthy
|
||||
/// session self-terminated one window after its snapshot with a false "Site disconnected".
|
||||
/// Settable for tests.
|
||||
/// </summary>
|
||||
internal static TimeSpan ConsumerLivenessCheckInterval { get; set; } = TimeSpan.FromSeconds(30);
|
||||
|
||||
/// <summary>
|
||||
/// When the consumer was last known to be attached (UTC). Seeded in <see cref="PreStart"/>
|
||||
/// so a session gets a full window to receive its first keepalive, then refreshed by every
|
||||
/// <see cref="DebugStreamConsumerAlive"/>. Actor-thread only.
|
||||
/// </summary>
|
||||
private DateTime _consumerLastSeenUtc = DateTime.UtcNow;
|
||||
|
||||
private int _retryCount;
|
||||
private bool _useNodeA = true;
|
||||
private bool _stopped;
|
||||
private CancellationTokenSource? _grpcCts;
|
||||
|
||||
/// <summary>
|
||||
/// Monotonic stream generation stamped on each opened gRPC stream and echoed back on its
|
||||
/// error/completion callbacks: a late callback raced out of a previous (cancelled) stream
|
||||
/// carries a stale generation and is ignored, so it can neither burn retry budget nor
|
||||
/// open a duplicate stream. Mirrors <c>SiteAlarmAggregatorActor</c>. Actor-thread only.
|
||||
/// </summary>
|
||||
private int _streamGeneration;
|
||||
|
||||
/// <summary>
|
||||
/// Phase flag. <see langword="false"/> until the initial
|
||||
/// <see cref="DebugViewSnapshot"/> has been delivered and the pre-snapshot buffer
|
||||
@@ -77,17 +124,35 @@ public class DebugStreamBridgeActor : ReceiveActor, IWithTimers
|
||||
/// Ordered buffer of live gRPC events (<see cref="AttributeValueChanged"/>/
|
||||
/// <see cref="AlarmStateChanged"/>) that arrived before the snapshot was delivered.
|
||||
/// Flushed (with per-entity dedup against the snapshot) when the snapshot arrives,
|
||||
/// then never used again. Mutated only on the actor thread.
|
||||
/// then never used again. Bounded by <see cref="MaxPreSnapshotBuffer"/> with drop-oldest
|
||||
/// eviction (WP2.3): a snapshot that never arrives used to buffer without limit on the
|
||||
/// central node. Mutated only on the actor thread.
|
||||
/// </summary>
|
||||
private readonly List<object> _preSnapshotBuffer = new();
|
||||
private readonly Queue<object> _preSnapshotBuffer = new();
|
||||
|
||||
/// <summary>
|
||||
/// Defensive log threshold: if the pre-snapshot buffer grows past this many events
|
||||
/// during a slow snapshot we log once (events are NOT dropped — the window is short).
|
||||
/// Defensive log threshold: the first warning fires when the pre-snapshot buffer grows
|
||||
/// past this many events during a slow snapshot, before the hard cap starts evicting.
|
||||
/// </summary>
|
||||
private const int BufferWarnThreshold = 10_000;
|
||||
private bool _bufferWarned;
|
||||
|
||||
/// <summary>
|
||||
/// Hard cap on the pre-snapshot buffer. Beyond it the OLDEST event is evicted — the
|
||||
/// snapshot that ends the buffering phase is authoritative for anything that old, so
|
||||
/// keeping the newest events is what preserves the post-snapshot delta chain.
|
||||
/// </summary>
|
||||
private const int MaxPreSnapshotBuffer = 20_000;
|
||||
|
||||
/// <summary>Events evicted from the pre-snapshot buffer in this session. Actor-thread only.</summary>
|
||||
private long _preSnapshotDropped;
|
||||
|
||||
/// <summary>
|
||||
/// Total pre-snapshot events dropped across all debug sessions on this node — the raw
|
||||
/// counter behind <c>scadabridge.central.debug_view.presnapshot_dropped</c>.
|
||||
/// </summary>
|
||||
internal static long TotalPreSnapshotDropped;
|
||||
|
||||
/// <summary>Timer scheduler for reconnect and stability window timers.</summary>
|
||||
public ITimerScheduler Timers { get; set; } = null!;
|
||||
|
||||
@@ -151,7 +216,7 @@ public class DebugStreamBridgeActor : ReceiveActor, IWithTimers
|
||||
// non-deployed instance — cancel it (and any buffered gap events are
|
||||
// discarded with the actor). No pass-through.
|
||||
// _stopped is set AFTER CleanupGrpc() to match the ordering in the
|
||||
// DebugStreamTerminated and ReceiveTimeout handlers (cosmetic consistency).
|
||||
// DebugStreamTerminated and consumer-liveness handlers (cosmetic consistency).
|
||||
CleanupGrpc();
|
||||
_stopped = true;
|
||||
_preSnapshotBuffer.Clear();
|
||||
@@ -168,6 +233,9 @@ public class DebugStreamBridgeActor : ReceiveActor, IWithTimers
|
||||
_instanceUniqueName, snapshot.AttributeValues.Count, snapshot.AlarmStates.Count,
|
||||
_preSnapshotBuffer.Count);
|
||||
|
||||
// The snapshot arrived — stand the hard deadline down.
|
||||
Timers.Cancel(SnapshotTimerKey);
|
||||
|
||||
// Deliver the snapshot, then flush the gap-window buffer (deduped), then
|
||||
// switch to pass-through. Order matters: snapshot first, buffered events next.
|
||||
_onEvent(snapshot);
|
||||
@@ -175,14 +243,38 @@ public class DebugStreamBridgeActor : ReceiveActor, IWithTimers
|
||||
_snapshotDelivered = true;
|
||||
});
|
||||
|
||||
// Domain events arriving via Self.Tell from gRPC callback.
|
||||
// Receiving an event must NOT reset _retryCount — a
|
||||
// flapping stream that delivers a single event between failures would
|
||||
// otherwise never trip MaxRetries. The retry budget is recovered only by
|
||||
// GrpcStreamStable (a stream that has stayed up for StabilityWindow).
|
||||
// Before the snapshot has been delivered, BUFFER (in arrival order)
|
||||
// rather than deliver — these may be gap-window events. After the snapshot has
|
||||
// been flushed, pass through directly (same handler, phase-dependent behavior).
|
||||
// Hard snapshot deadline (WP2.3). Nothing else ends a session stuck in the
|
||||
// buffering phase: the site's reply was lost, so no gRPC error fires, the stream
|
||||
// keeps delivering events, and stream traffic does not renew the orphan net (which
|
||||
// measures the consumer). Fail the session so the consumer is told and can reopen.
|
||||
Receive<DebugSnapshotDeadline>(_ =>
|
||||
{
|
||||
if (_stopped || _snapshotDelivered) return;
|
||||
_log.Error(
|
||||
"No debug snapshot for {0} within {1}s ({2} event(s) buffered, {3} dropped); failing the session",
|
||||
_instanceUniqueName, SnapshotTimeout.TotalSeconds,
|
||||
_preSnapshotBuffer.Count, _preSnapshotDropped);
|
||||
CleanupGrpc();
|
||||
SendUnsubscribe();
|
||||
_stopped = true;
|
||||
_preSnapshotBuffer.Clear();
|
||||
_onTerminated();
|
||||
Context.Stop(Self);
|
||||
});
|
||||
|
||||
// Domain events arriving via Self.Tell from the gRPC callback. Stream traffic never
|
||||
// proves the CONSUMER is still there, so it deliberately does not touch the orphan
|
||||
// net (which now measures the consumer keepalive, not the mailbox). Receiving an
|
||||
// event must not reset _retryCount either: a flapping stream that delivers a single
|
||||
// event between failures would otherwise never trip MaxRetries. The retry budget is
|
||||
// recovered only by GrpcStreamStable (a stream that has stayed up for
|
||||
// StabilityWindow). Before the snapshot has been delivered, BUFFER (in arrival order)
|
||||
// rather than deliver — these may be gap-window events; after the snapshot has been
|
||||
// flushed, pass through directly (phase-dependent behavior).
|
||||
Receive<LiveDebugStreamEvent>(wrapped => HandleStreamEvent(wrapped.Event));
|
||||
|
||||
// Unwrapped forms are still accepted (a direct Tell from a test or a future
|
||||
// in-process producer) and take the identical path.
|
||||
Receive<AttributeValueChanged>(changed => HandleStreamEvent(changed));
|
||||
Receive<AlarmStateChanged>(changed => HandleStreamEvent(changed));
|
||||
|
||||
@@ -198,10 +290,33 @@ public class DebugStreamBridgeActor : ReceiveActor, IWithTimers
|
||||
// gRPC stream error — attempt reconnection
|
||||
Receive<GrpcStreamError>(msg =>
|
||||
{
|
||||
// Ignore a late error raced out of a previous (cancelled) stream: it must not
|
||||
// burn retry budget or flip the node a second time.
|
||||
if (msg.Generation != _streamGeneration)
|
||||
{
|
||||
_log.Debug("Ignoring stale gRPC error from stream generation {0} (current {1})",
|
||||
msg.Generation, _streamGeneration);
|
||||
return;
|
||||
}
|
||||
_log.Warning("gRPC stream error for {0}: {1}", _instanceUniqueName, msg.Exception.Message);
|
||||
HandleGrpcError();
|
||||
});
|
||||
|
||||
// gRPC stream ended GRACEFULLY (server status OK) — the site's 4h max stream
|
||||
// lifetime elapsing or a graceful site shutdown. Not a fault: reopen on the SAME
|
||||
// node without spending retry budget. Without this the session went silently deaf.
|
||||
Receive<GrpcStreamCompleted>(msg =>
|
||||
{
|
||||
if (_stopped) return;
|
||||
if (msg.Generation != _streamGeneration)
|
||||
{
|
||||
_log.Debug("Ignoring stale gRPC completion from stream generation {0} (current {1})",
|
||||
msg.Generation, _streamGeneration);
|
||||
return;
|
||||
}
|
||||
HandleGrpcCompleted();
|
||||
});
|
||||
|
||||
// Scheduled reconnection
|
||||
Receive<ReconnectGrpcStream>(_ => OpenGrpcStream());
|
||||
|
||||
@@ -226,11 +341,32 @@ public class DebugStreamBridgeActor : ReceiveActor, IWithTimers
|
||||
Context.Stop(Self);
|
||||
});
|
||||
|
||||
// Orphan safety net — if nobody stops us within 5 minutes, self-terminate
|
||||
Context.SetReceiveTimeout(TimeSpan.FromMinutes(5));
|
||||
Receive<ReceiveTimeout>(_ =>
|
||||
// Consumer keepalive: DebugStreamService Tells this on a timer for every session it
|
||||
// still holds (i.e. still attached to a Blazor debug view / SignalR connection). It
|
||||
// is the ONLY thing that renews the orphan window — deliberately, so neither a chatty
|
||||
// site nor a silent one can influence it.
|
||||
Receive<DebugStreamConsumerAlive>(_ =>
|
||||
{
|
||||
_log.Warning("Debug stream for {0} timed out (orphaned session), stopping", _instanceUniqueName);
|
||||
if (_stopped) return;
|
||||
_consumerLastSeenUtc = DateTime.UtcNow;
|
||||
});
|
||||
|
||||
// Orphan safety net, CONSUMER-measured (WP2.3 follow-up). A periodic self-tick
|
||||
// compares the consumer-last-seen stamp against ConsumerIdleTimeout; the previous
|
||||
// SetReceiveTimeout(5 min) measured the mailbox instead, and once stream events were
|
||||
// (correctly) marked INotInfluenceReceiveTimeout nothing recurring reset it — the
|
||||
// snapshot arrives once and GrpcStreamStable once, so EVERY healthy session died at
|
||||
// ~6 minutes and the consumer was told "Site disconnected".
|
||||
Receive<ConsumerLivenessTick>(_ =>
|
||||
{
|
||||
if (_stopped) return;
|
||||
var idle = DateTime.UtcNow - _consumerLastSeenUtc;
|
||||
if (idle < ConsumerIdleTimeout) return;
|
||||
|
||||
_log.Warning(
|
||||
"Debug stream for {0} has had no consumer activity for {1:F0}s (orphaned session), stopping",
|
||||
_instanceUniqueName, idle.TotalSeconds);
|
||||
Timers.Cancel(ConsumerLivenessTimerKey);
|
||||
CleanupGrpc();
|
||||
SendUnsubscribe();
|
||||
_stopped = true;
|
||||
@@ -255,15 +391,30 @@ public class DebugStreamBridgeActor : ReceiveActor, IWithTimers
|
||||
return;
|
||||
}
|
||||
|
||||
_preSnapshotBuffer.Add(evt);
|
||||
if (!_bufferWarned && _preSnapshotBuffer.Count > BufferWarnThreshold)
|
||||
if (!_bufferWarned && _preSnapshotBuffer.Count + 1 > BufferWarnThreshold)
|
||||
{
|
||||
_bufferWarned = true;
|
||||
_log.Warning(
|
||||
"Pre-snapshot debug-event buffer for {0} exceeded {1} events while awaiting the snapshot; " +
|
||||
"events are still retained (not dropped).",
|
||||
_instanceUniqueName, BufferWarnThreshold);
|
||||
"Pre-snapshot debug-event buffer for {0} exceeded {1} events while awaiting the snapshot " +
|
||||
"(hard cap {2}, drop-oldest beyond it).",
|
||||
_instanceUniqueName, BufferWarnThreshold, MaxPreSnapshotBuffer);
|
||||
}
|
||||
|
||||
while (_preSnapshotBuffer.Count >= MaxPreSnapshotBuffer)
|
||||
{
|
||||
_preSnapshotBuffer.Dequeue();
|
||||
_preSnapshotDropped++;
|
||||
Interlocked.Increment(ref TotalPreSnapshotDropped);
|
||||
ScadaBridgeTelemetry.RecordDebugPreSnapshotDrop();
|
||||
if (_preSnapshotDropped == 1 || _preSnapshotDropped % 500 == 0)
|
||||
{
|
||||
_log.Warning(
|
||||
"Pre-snapshot debug-event buffer for {0} is at its {1}-event cap; {2} event(s) evicted so far",
|
||||
_instanceUniqueName, MaxPreSnapshotBuffer, _preSnapshotDropped);
|
||||
}
|
||||
}
|
||||
|
||||
_preSnapshotBuffer.Enqueue(evt);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
@@ -321,8 +472,9 @@ public class DebugStreamBridgeActor : ReceiveActor, IWithTimers
|
||||
|
||||
if (dropped > 0 || flushed > 0)
|
||||
{
|
||||
_log.Debug("Flushed {0} buffered debug event(s) for {1}, dropped {2} as already-in-snapshot",
|
||||
flushed, _instanceUniqueName, dropped);
|
||||
_log.Debug("Flushed {0} buffered debug event(s) for {1}, dropped {2} as already-in-snapshot" +
|
||||
" ({3} previously evicted at the buffer cap)",
|
||||
flushed, _instanceUniqueName, dropped, _preSnapshotDropped);
|
||||
}
|
||||
|
||||
_preSnapshotBuffer.Clear();
|
||||
@@ -398,6 +550,19 @@ public class DebugStreamBridgeActor : ReceiveActor, IWithTimers
|
||||
var request = new SubscribeDebugViewRequest(_instanceUniqueName, _correlationId);
|
||||
var envelope = new SiteEnvelope(_siteIdentifier, request);
|
||||
_centralCommunicationActor.Tell(envelope, Self);
|
||||
|
||||
// Arm the hard snapshot deadline alongside the request.
|
||||
if (SnapshotTimeout > TimeSpan.Zero)
|
||||
Timers.StartSingleTimer(SnapshotTimerKey, new DebugSnapshotDeadline(), SnapshotTimeout);
|
||||
|
||||
// Arm the consumer-liveness net. The stamp starts now, so the session always gets a
|
||||
// full ConsumerIdleTimeout to see its first keepalive from DebugStreamService.
|
||||
_consumerLastSeenUtc = DateTime.UtcNow;
|
||||
if (ConsumerIdleTimeout > TimeSpan.Zero && ConsumerLivenessCheckInterval > TimeSpan.Zero)
|
||||
{
|
||||
Timers.StartPeriodicTimer(
|
||||
ConsumerLivenessTimerKey, new ConsumerLivenessTick(), ConsumerLivenessCheckInterval);
|
||||
}
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
@@ -424,20 +589,58 @@ public class DebugStreamBridgeActor : ReceiveActor, IWithTimers
|
||||
// retry budget is recovered. Cancelled by HandleGrpcError.
|
||||
Timers.StartSingleTimer(StabilityTimerKey, new GrpcStreamStable(), StabilityWindow);
|
||||
|
||||
var generation = ++_streamGeneration;
|
||||
var client = _grpcFactory.GetOrCreate(_siteIdentifier, endpoint);
|
||||
var self = Self;
|
||||
var ct = _grpcCts.Token;
|
||||
|
||||
// Launch as background task — onEvent and onError marshal back to actor via Tell
|
||||
// Launch as background task — the callbacks marshal back to the actor via Tell.
|
||||
// The task itself is observed below: a fault escaping SubscribeAsync would otherwise
|
||||
// leave the session waiting on a stream that does not exist, exception unobserved.
|
||||
Task.Run(async () =>
|
||||
{
|
||||
await client.SubscribeAsync(
|
||||
_correlationId,
|
||||
_instanceUniqueName,
|
||||
evt => self.Tell(evt),
|
||||
ex => self.Tell(new GrpcStreamError(ex)),
|
||||
// Wrapped so the stream path is explicit (it never renews the orphan net).
|
||||
evt => self.Tell(new LiveDebugStreamEvent(evt)),
|
||||
ex => self.Tell(new GrpcStreamError(ex, generation)),
|
||||
() => self.Tell(new GrpcStreamCompleted(generation)),
|
||||
ct);
|
||||
}, ct);
|
||||
}, ct).ContinueWith(t =>
|
||||
{
|
||||
if (t.IsFaulted)
|
||||
self.Tell(new GrpcStreamError(t.Exception!.GetBaseException(), generation));
|
||||
else if (t.IsCanceled && !ct.IsCancellationRequested)
|
||||
self.Tell(new GrpcStreamCompleted(generation));
|
||||
// RanToCompletion: SubscribeAsync already reported its own outcome.
|
||||
}, TaskContinuationOptions.ExecuteSynchronously);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Handles a graceful end of stream (server status OK). The stream simply expired or the
|
||||
/// site shut down cleanly, so the retry budget is left untouched and the endpoint is not
|
||||
/// flipped; the reopen is scheduled through the existing reconnect timer, which also
|
||||
/// rate-limits a pathological site that keeps closing streams immediately.
|
||||
/// </summary>
|
||||
private void HandleGrpcCompleted()
|
||||
{
|
||||
// The stream is gone, so its armed stability timer must not later "recover" a
|
||||
// budget that its successor has since spent.
|
||||
Timers.Cancel(StabilityTimerKey);
|
||||
|
||||
_log.Info("gRPC stream for {0} completed gracefully (server end of stream); reopening",
|
||||
_instanceUniqueName);
|
||||
|
||||
// Release the site-side relay for the finished stream before reopening, so the site
|
||||
// is not left with a zombie relay actor for this correlation id.
|
||||
_grpcCts?.Cancel();
|
||||
_grpcCts?.Dispose();
|
||||
_grpcCts = null;
|
||||
_grpcFactory.TryGet(_siteIdentifier, _useNodeA ? _grpcNodeAAddress : _grpcNodeBAddress)
|
||||
?.Unsubscribe(_correlationId);
|
||||
|
||||
Timers.StartSingleTimer(ReconnectTimerKey, new ReconnectGrpcStream(), ReconnectDelay);
|
||||
}
|
||||
|
||||
private void HandleGrpcError()
|
||||
@@ -512,9 +715,45 @@ public class DebugStreamBridgeActor : ReceiveActor, IWithTimers
|
||||
public record StopDebugStream;
|
||||
|
||||
/// <summary>
|
||||
/// Internal message indicating a gRPC stream error occurred.
|
||||
/// Envelope for a live gRPC stream event (<c>AttributeValueChanged</c>/
|
||||
/// <c>AlarmStateChanged</c>). Kept as a distinct envelope so the high-volume stream path is
|
||||
/// explicit at the call site; the orphan net no longer keys off the mailbox at all (it
|
||||
/// measures the consumer keepalive), so a busy site's event flood can neither hold a dead
|
||||
/// session open nor — as briefly happened — be the only thing keeping a healthy one alive.
|
||||
/// </summary>
|
||||
internal record GrpcStreamError(Exception Exception);
|
||||
internal record LiveDebugStreamEvent(object Event);
|
||||
|
||||
/// <summary>
|
||||
/// Consumer keepalive: <c>DebugStreamService</c> Tells one of these to every bridge actor
|
||||
/// whose session is still attached to a consumer, on
|
||||
/// <see cref="DebugStreamBridgeActor.ConsumerLivenessCheckInterval"/>-scale cadence. Renewing
|
||||
/// the consumer-last-seen stamp is its ONLY effect.
|
||||
/// </summary>
|
||||
public record DebugStreamConsumerAlive;
|
||||
|
||||
/// <summary>
|
||||
/// Internal self-tick that checks the consumer-last-seen stamp against
|
||||
/// <see cref="DebugStreamBridgeActor.ConsumerIdleTimeout"/>.
|
||||
/// </summary>
|
||||
internal record ConsumerLivenessTick;
|
||||
|
||||
/// <summary>
|
||||
/// Internal message: the hard deadline for the initial <c>DebugViewSnapshot</c> expired.
|
||||
/// </summary>
|
||||
internal record DebugSnapshotDeadline;
|
||||
|
||||
/// <summary>
|
||||
/// Internal message indicating a gRPC stream error occurred, stamped with the stream
|
||||
/// generation it came from so a late error out of a cancelled stream can be ignored.
|
||||
/// </summary>
|
||||
internal record GrpcStreamError(Exception Exception, int Generation);
|
||||
|
||||
/// <summary>
|
||||
/// Internal message indicating the gRPC stream ended gracefully (server status OK — the
|
||||
/// site's max stream lifetime elapsed, or the site shut down cleanly), stamped with its
|
||||
/// stream generation.
|
||||
/// </summary>
|
||||
internal record GrpcStreamCompleted(int Generation);
|
||||
|
||||
/// <summary>
|
||||
/// Internal message to trigger gRPC stream reconnection.
|
||||
|
||||
@@ -24,8 +24,10 @@ public interface ISiteCommandTransport
|
||||
/// Routes <paramref name="envelope"/>'s message to its site. Any reply the site produces is
|
||||
/// delivered to <paramref name="replyTo"/> — for an <c>Ask</c> that is the temporary ask actor
|
||||
/// (completing the caller's task); for a <c>Tell</c>-with-sender (the debug bridge) that is the
|
||||
/// originating actor. A message with no route (an unknown site) is warned and dropped so the
|
||||
/// caller's <c>Ask</c> times out — central never buffers for an unreachable site.
|
||||
/// originating actor. A message with no route (an unknown/unconfigured site) fails fast: a
|
||||
/// <see cref="Status.Failure"/> is delivered to <paramref name="replyTo"/> immediately instead of
|
||||
/// letting the caller's <c>Ask</c> burn its full timeout — central still never buffers for an
|
||||
/// unreachable site.
|
||||
/// </summary>
|
||||
/// <param name="envelope">The site-addressed command envelope.</param>
|
||||
/// <param name="replyTo">Where a reply (or a <see cref="Status.Failure"/>) is delivered.</param>
|
||||
|
||||
@@ -31,10 +31,14 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Actors;
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Failover + drift:</b> a gRPC error flips NodeA↔NodeB with the same retry budget +
|
||||
/// stability window as <see cref="DebugStreamBridgeActor"/>, and each reconnect triggers
|
||||
/// a RE-SEED (never silently serve stale). A periodic reconcile snapshot
|
||||
/// (<see cref="_reconcileInterval"/>, default 60s) corrects instance-set drift and any
|
||||
/// missed delta.
|
||||
/// stability window as <see cref="DebugStreamBridgeActor"/>. A re-seed runs <b>once per
|
||||
/// successful (re)connect</b> — driven by the stream's connected callback, not by each
|
||||
/// reconnect ATTEMPT (WP2.3: a site outage used to fan a full snapshot out per retry, all
|
||||
/// of them against the very site that is unreachable). A periodic reconcile snapshot
|
||||
/// (<see cref="_reconcileInterval"/>, default 60s, jittered per site so N aggregators do
|
||||
/// not fan out in lockstep) remains the drift/backstop: it corrects instance-set drift and
|
||||
/// any missed delta, but it is <em>skipped</em> when a fan-out already completed inside the
|
||||
/// window, and it publishes to viewers only when the snapshot actually changed the cache.
|
||||
/// </para>
|
||||
/// All state is mutated only on the actor thread: gRPC callbacks and fan-out results are
|
||||
/// marshalled back via <c>Self.Tell</c>, so the cache needs no internal lock. The
|
||||
@@ -47,18 +51,20 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
|
||||
private readonly string _siteIdentifier;
|
||||
private readonly string _correlationId;
|
||||
private readonly Func<CancellationToken, Task<IReadOnlyList<AlarmStateChanged>>> _seedFn;
|
||||
private readonly Action<IReadOnlyList<AlarmStateChanged>> _publish;
|
||||
private readonly Action<IReadOnlyList<AlarmStateChanged>, bool> _publish;
|
||||
private readonly SiteStreamGrpcClientFactory _grpcFactory;
|
||||
private readonly string _grpcNodeAAddress;
|
||||
private readonly string _grpcNodeBAddress;
|
||||
private readonly TimeSpan _reconcileInterval;
|
||||
private readonly TimeSpan _publishCoalesce;
|
||||
private readonly double _reconcileJitterFraction;
|
||||
|
||||
private const int MaxRetries = 3;
|
||||
private const string ReconnectTimerKey = "alarm-grpc-reconnect";
|
||||
private const string StabilityTimerKey = "alarm-grpc-stability";
|
||||
private const string ReconcileTimerKey = "alarm-reconcile";
|
||||
private const string PublishTimerKey = "alarm-publish-coalesce";
|
||||
private const string SeedRetryTimerKey = "alarm-seed-retry";
|
||||
|
||||
/// <summary>True while a coalesced publish is armed (dirty deltas awaiting one tick). Actor-thread only.</summary>
|
||||
private bool _publishPending;
|
||||
@@ -78,12 +84,24 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
|
||||
private bool _stopped;
|
||||
|
||||
/// <summary>
|
||||
/// True once the live gRPC stream has been given up (retry budget exhausted). Reconcile
|
||||
/// snapshots keep serving in the meantime; the next reconcile tick self-heals the stream
|
||||
/// by resetting the retry budget and reopening it, so a sustained site outage does not
|
||||
/// permanently drop the live feed. Actor-thread only.
|
||||
/// True while there is no live gRPC stream — either it was given up (retry budget
|
||||
/// exhausted) or it ended gracefully (server status OK at the site's max stream
|
||||
/// lifetime). Reconcile snapshots keep serving in the meantime; the next reconcile tick
|
||||
/// self-heals the stream by resetting the retry budget and reopening it, so neither a
|
||||
/// sustained site outage nor a routine 4h stream expiry permanently drops the live feed.
|
||||
/// Starts <c>true</c>: until the site accepts the first subscription there is no live
|
||||
/// feed, and the owning cache must not advertise one. Actor-thread only.
|
||||
/// </summary>
|
||||
private bool _streamDown;
|
||||
private bool _streamDown = true;
|
||||
|
||||
/// <summary>
|
||||
/// Why the stream is down: <c>true</c> = the retry budget was exhausted, so the
|
||||
/// self-healing reopen must also reset it; <c>false</c> = it ended gracefully, and the
|
||||
/// budget — which a completion neither spends nor refunds — is carried across the reopen
|
||||
/// untouched (a stream flapping between faults and clean closes must still trip
|
||||
/// MaxRetries). Actor-thread only.
|
||||
/// </summary>
|
||||
private bool _retryBudgetExhausted;
|
||||
private CancellationTokenSource? _grpcCts;
|
||||
private CancellationTokenSource? _lifetimeCts;
|
||||
|
||||
@@ -96,8 +114,80 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
|
||||
/// <summary>True while a seed/reconcile snapshot fan-out is in flight (deltas buffer). Actor-thread only.</summary>
|
||||
private bool _fanoutInFlight;
|
||||
|
||||
/// <summary>Ordered buffer of live deltas that arrived while a fan-out was in flight. Actor-thread only.</summary>
|
||||
private readonly List<AlarmStateChanged> _buffer = new();
|
||||
/// <summary>
|
||||
/// Ordered buffer of live deltas that arrived while a fan-out was in flight, capped at
|
||||
/// <see cref="MaxBufferedDeltas"/> with drop-oldest eviction (WP2.3 — a stuck fan-out
|
||||
/// used to buffer without bound). Dropping the OLDEST is the right eviction here: the
|
||||
/// fan-out that follows rebuilds the cache authoritatively, so an evicted delta is
|
||||
/// superseded rather than lost, while the newest transitions — the ones the snapshot may
|
||||
/// predate — are the ones kept. Actor-thread only.
|
||||
/// </summary>
|
||||
private readonly Queue<AlarmStateChanged> _buffer = new();
|
||||
|
||||
/// <summary>Total deltas evicted from <see cref="_buffer"/> over this actor's life. Actor-thread only.</summary>
|
||||
private long _bufferDropped;
|
||||
|
||||
/// <summary>
|
||||
/// Hard cap on the pre-fan-out delta buffer. Beyond this the oldest entry is evicted and
|
||||
/// counted; the counter is logged (first drop + every 500th) and exported as
|
||||
/// <c>scadabridge.site.alarm_cache.buffer_dropped</c>.
|
||||
/// </summary>
|
||||
private const int MaxBufferedDeltas = 20_000;
|
||||
|
||||
/// <summary>
|
||||
/// True when the next successful (re)connect must run a seed fan-out: set whenever the
|
||||
/// stream is lost (fault or graceful completion) and cleared by the connect that consumes
|
||||
/// it. This is what makes the re-seed happen once per successful reconnect rather than
|
||||
/// once per reconnect ATTEMPT — the old code fanned a full snapshot out on every retry,
|
||||
/// against a site that was by definition unreachable. Actor-thread only.
|
||||
/// </summary>
|
||||
private bool _seedOnConnect;
|
||||
|
||||
/// <summary>
|
||||
/// Set when a CONNECT- OR FAILOVER-DRIVEN fan-out finishes (the initial seed, a re-seed
|
||||
/// consumed by a successful (re)connect, or a re-seed queued behind one); consumed by the
|
||||
/// next reconcile tick, which skips its own fan-out when it finds the flag set. That makes
|
||||
/// the connect-driven seed and the periodic backstop mutually exclusive — the pair used to
|
||||
/// run BOTH, which is the "full unconditional snapshot every 60s" waste — while bounding
|
||||
/// staleness at two intervals (a tick can be skipped at most once in a row).
|
||||
/// <para>
|
||||
/// A TICK-driven fan-out deliberately does NOT set it. It used to: the tick's own fan-out
|
||||
/// completion armed the skip, so steady state ran fan-out → skip → fan-out → skip, i.e.
|
||||
/// one reconcile per TWO intervals — halving the not-reporting refresh rate and the alarm
|
||||
/// reconcile backstop for no reason (nothing was duplicated to suppress).
|
||||
/// </para>
|
||||
/// Actor-thread only.
|
||||
/// </summary>
|
||||
private bool _fanoutSinceLastTick;
|
||||
|
||||
/// <summary>
|
||||
/// Whether the in-flight fan-out is connect/failover-driven and therefore makes the next
|
||||
/// reconcile tick redundant (see <see cref="_fanoutSinceLastTick"/>). Stable for the
|
||||
/// fan-out's lifetime: <see cref="StartFanout"/> never overwrites it while one is in
|
||||
/// flight (a colliding request queues instead). Actor-thread only.
|
||||
/// </summary>
|
||||
private bool _fanoutSuppressesNextTick;
|
||||
|
||||
/// <summary>
|
||||
/// Same flag for a fan-out that was queued behind an in-flight one
|
||||
/// (<see cref="_reseedQueued"/>) — OR-ed across colliding requests so a queued
|
||||
/// connect/failover re-seed keeps its suppression even if a tick collided too.
|
||||
/// Actor-thread only.
|
||||
/// </summary>
|
||||
private bool _queuedFanoutSuppressesNextTick;
|
||||
|
||||
/// <summary>
|
||||
/// True between issuing a stream open and its first outcome (connected / error /
|
||||
/// completed), so a reconcile tick never stacks a second open on an in-flight one.
|
||||
/// Actor-thread only.
|
||||
/// </summary>
|
||||
private bool _openInFlight;
|
||||
|
||||
/// <summary>Consecutive whole-fan-out failures, driving the seed-retry backoff. Actor-thread only.</summary>
|
||||
private int _consecutiveSeedFailures;
|
||||
|
||||
/// <summary>Upper bound on the seed-retry backoff, as a multiple of the reconcile interval.</summary>
|
||||
private const int MaxSeedRetryBackoffMultiplier = 8;
|
||||
|
||||
/// <summary>
|
||||
/// A failover re-seed was requested while a fan-out was already in flight; it must run
|
||||
@@ -133,6 +223,10 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
|
||||
/// <param name="publish">
|
||||
/// Publishes a fresh immutable snapshot of the cache to the owning service, which
|
||||
/// stores it and raises viewer <c>onChanged</c> callbacks. Invoked on the actor thread.
|
||||
/// The second argument is the live-stream liveness flag: <c>false</c> means the site-wide
|
||||
/// gRPC stream is currently down (fault-exhausted or gracefully completed), so the cache
|
||||
/// is only as fresh as the last reconcile and the page must fall back to polling
|
||||
/// (WP2.3 carried residual — <c>IsLive</c> used to stay true across a dead stream).
|
||||
/// </param>
|
||||
/// <param name="grpcFactory">Factory caching one gRPC client per (site, endpoint).</param>
|
||||
/// <param name="grpcNodeAAddress">gRPC address of the site's node A.</param>
|
||||
@@ -147,18 +241,24 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
|
||||
/// <param name="stabilityWindow">
|
||||
/// How long a fresh gRPC stream must stay up before its retry budget recovers (production 60s).
|
||||
/// </param>
|
||||
/// <param name="reconcileJitterFraction">
|
||||
/// Fraction of <paramref name="reconcileInterval"/> added as a per-tick random offset so
|
||||
/// N per-site aggregators on one central node do not fan out in lockstep (production 0.2 =
|
||||
/// up to +20%). Zero disables jitter, which is what tests want for determinism.
|
||||
/// </param>
|
||||
public SiteAlarmAggregatorActor(
|
||||
string siteIdentifier,
|
||||
string correlationId,
|
||||
Func<CancellationToken, Task<IReadOnlyList<AlarmStateChanged>>> seedFn,
|
||||
Action<IReadOnlyList<AlarmStateChanged>> publish,
|
||||
Action<IReadOnlyList<AlarmStateChanged>, bool> publish,
|
||||
SiteStreamGrpcClientFactory grpcFactory,
|
||||
string grpcNodeAAddress,
|
||||
string grpcNodeBAddress,
|
||||
TimeSpan reconcileInterval,
|
||||
TimeSpan publishCoalesce,
|
||||
TimeSpan reconnectDelay,
|
||||
TimeSpan stabilityWindow)
|
||||
TimeSpan stabilityWindow,
|
||||
double reconcileJitterFraction = 0.0)
|
||||
{
|
||||
_siteIdentifier = siteIdentifier;
|
||||
_correlationId = correlationId;
|
||||
@@ -171,6 +271,7 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
|
||||
_publishCoalesce = publishCoalesce;
|
||||
_reconnectDelay = reconnectDelay;
|
||||
_stabilityWindow = stabilityWindow;
|
||||
_reconcileJitterFraction = Math.Clamp(reconcileJitterFraction, 0.0, 1.0);
|
||||
|
||||
// Live delta from the site-wide alarm stream (marshalled in via Self.Tell).
|
||||
// A received delta must NOT reset the retry budget (a flapping stream that
|
||||
@@ -187,6 +288,24 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
|
||||
// Periodic reconcile tick (and the re-seed kicked after a reconnect).
|
||||
Receive<RunReconcile>(_ => OnReconcileTick());
|
||||
|
||||
// Backoff-scheduled retry of a fan-out that failed as a whole.
|
||||
Receive<RetrySeed>(_ =>
|
||||
{
|
||||
if (_stopped) return;
|
||||
// A backoff retry of a failed fan-out is not a connect/failover seed: it must
|
||||
// not stand the next reconcile tick down.
|
||||
StartFanout(isInitial: false, suppressesNextTick: false);
|
||||
});
|
||||
|
||||
// The site-wide stream is confirmed established (response headers received). This —
|
||||
// not the reconnect attempt — is the "successful (re)connect" that earns one re-seed.
|
||||
Receive<GrpcAlarmStreamConnected>(msg =>
|
||||
{
|
||||
if (_stopped) return;
|
||||
if (msg.Generation != _streamGeneration) return;
|
||||
OnStreamConnected();
|
||||
});
|
||||
|
||||
// Coalesced-publish tick: one publish for a batch of dirtying deltas (N6).
|
||||
Receive<PublishCoalesced>(_ =>
|
||||
{
|
||||
@@ -218,6 +337,22 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
|
||||
HandleGrpcError();
|
||||
});
|
||||
|
||||
// gRPC stream ended GRACEFULLY (server status OK) — the site's 4h max stream
|
||||
// lifetime elapsing or a graceful site shutdown. Not a fault: the stream is marked
|
||||
// down so the reconcile tick reopens it (which also re-seeds), but the retry budget
|
||||
// is untouched and the node is not flipped. Same generation fence as the error path.
|
||||
Receive<GrpcAlarmStreamCompleted>(msg =>
|
||||
{
|
||||
if (_stopped) return;
|
||||
if (msg.Generation != _streamGeneration)
|
||||
{
|
||||
_log.Debug("Ignoring stale gRPC completion from stream generation {0} (current {1})",
|
||||
msg.Generation, _streamGeneration);
|
||||
return;
|
||||
}
|
||||
HandleGrpcCompleted();
|
||||
});
|
||||
|
||||
Receive<ReconnectAlarmStream>(_ => OpenGrpcStream());
|
||||
|
||||
// Owning service asks us to stop (last viewer left + linger elapsed).
|
||||
@@ -243,11 +378,31 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
|
||||
// in the seed window are captured (buffered) rather than lost.
|
||||
OpenGrpcStream();
|
||||
|
||||
// Kick the initial seed fan-out.
|
||||
StartFanout(isInitial: true);
|
||||
// Kick the initial seed fan-out. The connect callback for this first stream must
|
||||
// NOT seed again on top of it, so the flag starts cleared.
|
||||
_seedOnConnect = false;
|
||||
StartFanout(isInitial: true, suppressesNextTick: true);
|
||||
|
||||
// Periodic reconcile backstop.
|
||||
Timers.StartPeriodicTimer(ReconcileTimerKey, new RunReconcile(), _reconcileInterval, _reconcileInterval);
|
||||
// Periodic reconcile backstop. Single-shot and re-armed with fresh jitter each
|
||||
// tick (a periodic timer would lock every site to the same phase forever).
|
||||
ArmReconcileTimer();
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Arms the next reconcile tick at <see cref="_reconcileInterval"/> plus a random
|
||||
/// offset of up to <see cref="_reconcileJitterFraction"/> of it, so N per-site
|
||||
/// aggregators started together (a central failover restarts them all at once) spread
|
||||
/// their fan-outs instead of stampeding the same 60s boundary.
|
||||
/// </summary>
|
||||
private void ArmReconcileTimer()
|
||||
{
|
||||
var delay = _reconcileInterval;
|
||||
if (_reconcileJitterFraction > 0 && _reconcileInterval > TimeSpan.Zero)
|
||||
{
|
||||
var jitterTicks = (long)(_reconcileInterval.Ticks * _reconcileJitterFraction * Random.Shared.NextDouble());
|
||||
delay += TimeSpan.FromTicks(jitterTicks);
|
||||
}
|
||||
Timers.StartSingleTimer(ReconcileTimerKey, new RunReconcile(), delay);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
@@ -267,22 +422,56 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
|
||||
// ── Reconcile tick ──────────────────────────────────────────────────────────
|
||||
|
||||
/// <summary>
|
||||
/// Periodic reconcile: always re-run the snapshot fan-out (corrects drift + missed
|
||||
/// deltas), and if the live stream was previously given up, self-heal it by resetting
|
||||
/// the retry budget and reopening — so a sustained outage never permanently kills the
|
||||
/// live feed.
|
||||
/// Periodic reconcile backstop: re-run the snapshot fan-out (corrects instance-set drift
|
||||
/// + any missed delta) UNLESS a CONNECT- OR FAILOVER-DRIVEN seed already completed inside
|
||||
/// this window — that seed makes the tick redundant, and running both was the "full
|
||||
/// unconditional snapshot every 60s" waste (WP2.3). A tick's OWN fan-out never arms that
|
||||
/// skip: it used to, which made steady state alternate fan-out/skip and halved the
|
||||
/// effective reconcile rate. If the live stream was previously given up, self-heal it by
|
||||
/// resetting the retry budget and reopening, so a sustained outage never permanently kills
|
||||
/// the live feed.
|
||||
/// </summary>
|
||||
private void OnReconcileTick()
|
||||
{
|
||||
if (_stopped) return;
|
||||
StartFanout(isInitial: false);
|
||||
if (_streamDown)
|
||||
|
||||
// Re-arm first: every exit path below must leave the backstop running.
|
||||
ArmReconcileTimer();
|
||||
|
||||
// A down stream is reopened in preference to fanning out: the reopen's connect
|
||||
// callback seeds by itself, so doing both would double the work.
|
||||
var reopening = false;
|
||||
if (_streamDown && !_openInFlight)
|
||||
{
|
||||
_log.Info("Site-alarm gRPC stream for {0} was down; reopening on reconcile tick", _siteIdentifier);
|
||||
_retryCount = 0;
|
||||
if (_retryBudgetExhausted)
|
||||
{
|
||||
_retryBudgetExhausted = false;
|
||||
_retryCount = 0;
|
||||
}
|
||||
// Telemetry: a reconcile-driven reopen after the stream was given up is a reconnect.
|
||||
ScadaBridgeTelemetry.RecordLiveAlarmStreamReconnect();
|
||||
_seedOnConnect = true;
|
||||
OpenGrpcStream();
|
||||
reopening = true;
|
||||
}
|
||||
|
||||
if (_fanoutSinceLastTick)
|
||||
{
|
||||
// A connect-driven seed (or the initial seed) already refreshed the cache inside
|
||||
// this window. Clear the flag so the NEXT tick fans out regardless — staleness
|
||||
// stays bounded at two intervals even if reconnects keep arriving.
|
||||
_fanoutSinceLastTick = false;
|
||||
_log.Debug("Site-alarm reconcile for {0} skipped; a fan-out already ran this window",
|
||||
_siteIdentifier);
|
||||
return;
|
||||
}
|
||||
|
||||
if (!reopening)
|
||||
{
|
||||
// Tick-driven: deliberately does NOT set the skip flag, so the NEXT tick fans
|
||||
// out too. Setting it here was the "one reconcile per two intervals" bug.
|
||||
StartFanout(isInitial: false, suppressesNextTick: false);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -293,7 +482,7 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
|
||||
/// <c>Self.Tell</c>. While in flight, live deltas buffer. A reconcile that arrives
|
||||
/// while a fan-out is already running is skipped (no stacking).
|
||||
/// </summary>
|
||||
private void StartFanout(bool isInitial)
|
||||
private void StartFanout(bool isInitial, bool suppressesNextTick)
|
||||
{
|
||||
if (_stopped) return;
|
||||
if (_fanoutInFlight)
|
||||
@@ -302,11 +491,16 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
|
||||
// one — its snapshot read-time predates the stream death, so skipping it would
|
||||
// serve stale up to the next 60s reconcile (N7.1). An initial-seed collision
|
||||
// never queues (there is only ever one).
|
||||
if (!isInitial) _reseedQueued = true;
|
||||
if (!isInitial)
|
||||
{
|
||||
_reseedQueued = true;
|
||||
_queuedFanoutSuppressesNextTick |= suppressesNextTick;
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
_fanoutInFlight = true;
|
||||
_fanoutSuppressesNextTick = suppressesNextTick;
|
||||
var self = Self;
|
||||
var ct = _lifetimeCts?.Token ?? CancellationToken.None;
|
||||
|
||||
@@ -335,6 +529,12 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
|
||||
// Rebuild the cache authoritatively from the fresh snapshot (this is what makes
|
||||
// reconcile able to DROP rows for instances/alarms that disappeared — a merge
|
||||
// could never remove a stale row since the live stream sends no "removed" event).
|
||||
// The previous cache is kept alongside so the reconcile can publish as a DIFF:
|
||||
// an unchanged snapshot must not wake every viewer's render path once a minute.
|
||||
var previous = _cache.Count == 0
|
||||
? null
|
||||
: new Dictionary<string, AlarmStateChanged>(_cache);
|
||||
|
||||
_cache.Clear();
|
||||
foreach (var alarm in msg.Alarms)
|
||||
{
|
||||
@@ -347,44 +547,85 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
|
||||
// buffered delta whose key is in the fresh snapshot with an equal-or-newer
|
||||
// timestamp is already reflected → drop; a strictly-newer (or new-key) delta is
|
||||
// applied. Inclusive-on-snapshot boundary matches DebugStreamBridgeActor.
|
||||
FlushBuffer();
|
||||
var flushChanged = FlushBuffer();
|
||||
|
||||
_fanoutInFlight = false;
|
||||
// Only a connect/failover-driven seed makes the next tick redundant — a tick's own
|
||||
// fan-out must never suppress the tick after it (that halved the reconcile rate).
|
||||
if (_fanoutSuppressesNextTick) _fanoutSinceLastTick = true;
|
||||
_consecutiveSeedFailures = 0;
|
||||
Timers.Cancel(SeedRetryTimerKey);
|
||||
var firstSeed = !_seeded;
|
||||
_seeded = true;
|
||||
|
||||
_log.Debug("Site-alarm {0} {1} complete: {2} alarm row(s)",
|
||||
_siteIdentifier, msg.IsInitial ? "seed" : "reconcile", _cache.Count);
|
||||
|
||||
// The fresh snapshot already carries the buffered deltas; drop any armed coalesce
|
||||
// tick so we publish once, immediately.
|
||||
// tick so we publish once, immediately — but only when something actually moved.
|
||||
Timers.Cancel(PublishTimerKey);
|
||||
_publishPending = false;
|
||||
Publish();
|
||||
if (firstSeed || flushChanged || DiffersFrom(previous))
|
||||
Publish();
|
||||
|
||||
// A failover re-seed requested while this fan-out was in flight runs now (N7.1).
|
||||
if (_reseedQueued)
|
||||
RunQueuedFanoutIfAny();
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Runs a fan-out that collided with an in-flight one, carrying the suppression flag it
|
||||
/// was queued with so a queued connect/failover re-seed still stands the next tick down
|
||||
/// (and a queued TICK fan-out still does not).
|
||||
/// </summary>
|
||||
private void RunQueuedFanoutIfAny()
|
||||
{
|
||||
if (!_reseedQueued) return;
|
||||
_reseedQueued = false;
|
||||
var suppresses = _queuedFanoutSuppressesNextTick;
|
||||
_queuedFanoutSuppressesNextTick = false;
|
||||
StartFanout(isInitial: false, suppressesNextTick: suppresses);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// True when the just-installed snapshot differs from <paramref name="previous"/> in any
|
||||
/// key or any row value — the reconcile's "is this publish worth a viewer fan-out?" test.
|
||||
/// A null previous cache (nothing published yet) always counts as different.
|
||||
/// </summary>
|
||||
private bool DiffersFrom(Dictionary<string, AlarmStateChanged>? previous)
|
||||
{
|
||||
if (previous is null || previous.Count != _cache.Count) return true;
|
||||
|
||||
foreach (var (key, current) in _cache)
|
||||
{
|
||||
_reseedQueued = false;
|
||||
StartFanout(isInitial: false);
|
||||
if (!previous.TryGetValue(key, out var old)) return true;
|
||||
// AlarmStateChanged is a record: structural equality covers state, level,
|
||||
// timestamp and every native-alarm enrichment field.
|
||||
if (!Equals(old, current)) return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
private void OnSeedFailed(SeedFailed msg)
|
||||
{
|
||||
if (_stopped) return;
|
||||
|
||||
_consecutiveSeedFailures++;
|
||||
|
||||
_log.Warning(msg.Exception,
|
||||
"Site-alarm {0} {1} fan-out failed; keeping current cache and relying on the next reconcile",
|
||||
_siteIdentifier, msg.IsInitial ? "seed" : "reconcile");
|
||||
"Site-alarm {0} {1} fan-out failed ({2} consecutive); keeping current cache and retrying with backoff",
|
||||
_siteIdentifier, msg.IsInitial ? "seed" : "reconcile", _consecutiveSeedFailures);
|
||||
|
||||
// Don't lose deltas captured during the failed window — apply them pass-through
|
||||
// into the (possibly stale/empty) cache. The next reconcile re-seeds authoritatively.
|
||||
_fanoutInFlight = false;
|
||||
FlushBuffer(dedupAgainstSeed: false);
|
||||
// Same rule as the success path: only a connect/failover-driven fan-out stands the
|
||||
// next reconcile tick down.
|
||||
if (_fanoutSuppressesNextTick) _fanoutSinceLastTick = true;
|
||||
var flushChanged = FlushBuffer(dedupAgainstSeed: false);
|
||||
|
||||
// Only publish if we already had a seed (so IsLive doesn't flip true on a
|
||||
// failed initial seed — the page keeps its poll fallback until we truly seed).
|
||||
if (_seeded)
|
||||
if (_seeded && flushChanged)
|
||||
{
|
||||
Timers.Cancel(PublishTimerKey);
|
||||
_publishPending = false;
|
||||
@@ -394,29 +635,55 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
|
||||
// A failover re-seed requested while this fan-out was in flight runs now (N7.1).
|
||||
if (_reseedQueued)
|
||||
{
|
||||
_reseedQueued = false;
|
||||
StartFanout(isInitial: false);
|
||||
RunQueuedFanoutIfAny();
|
||||
return;
|
||||
}
|
||||
|
||||
// Backoff retry (WP2.3): the seed leg used to have none — a site returning errors
|
||||
// was re-fanned at full reconcile cadence forever. Retry at reconnectDelay doubling
|
||||
// up to MaxSeedRetryBackoffMultiplier × the reconcile interval; the periodic
|
||||
// reconcile remains the floor once the backoff saturates.
|
||||
ScheduleSeedRetry();
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Arms the backoff retry for a failed fan-out: <c>reconnectDelay × 2^(failures-1)</c>,
|
||||
/// capped at <see cref="MaxSeedRetryBackoffMultiplier"/> × the reconcile interval. Once
|
||||
/// the cap is reached the periodic reconcile tick is doing the same work anyway, so no
|
||||
/// extra timer is armed.
|
||||
/// </summary>
|
||||
private void ScheduleSeedRetry()
|
||||
{
|
||||
var cap = _reconcileInterval > TimeSpan.Zero
|
||||
? TimeSpan.FromTicks(_reconcileInterval.Ticks * MaxSeedRetryBackoffMultiplier)
|
||||
: TimeSpan.FromMinutes(8);
|
||||
|
||||
var shift = Math.Min(_consecutiveSeedFailures - 1, 16);
|
||||
var delayTicks = _reconnectDelay.Ticks * (1L << shift);
|
||||
var delay = TimeSpan.FromTicks(Math.Min(delayTicks, cap.Ticks));
|
||||
if (delay <= TimeSpan.Zero) return;
|
||||
|
||||
Timers.StartSingleTimer(SeedRetryTimerKey, new RetrySeed(), delay);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Flushes the pre-fan-out buffer in arrival order. When <paramref name="dedupAgainstSeed"/>
|
||||
/// is true (normal completion) a buffered delta already reflected in the just-installed
|
||||
/// snapshot (same key, timestamp <= cache entry) is dropped; otherwise every buffered
|
||||
/// delta is applied pass-through.
|
||||
/// delta is applied pass-through. Returns <c>true</c> when at least one delta changed the
|
||||
/// cache (the reconcile's publish-only-on-change test must count these too).
|
||||
/// </summary>
|
||||
private void FlushBuffer(bool dedupAgainstSeed = true)
|
||||
private bool FlushBuffer(bool dedupAgainstSeed = true)
|
||||
{
|
||||
if (_buffer.Count == 0) return;
|
||||
if (_buffer.Count == 0) return false;
|
||||
|
||||
var changed = false;
|
||||
foreach (var delta in _buffer)
|
||||
{
|
||||
if (dedupAgainstSeed)
|
||||
ApplyDelta(delta, requireStrictlyNewer: true);
|
||||
else
|
||||
ApplyDelta(delta, requireStrictlyNewer: false);
|
||||
changed |= ApplyDelta(delta, requireStrictlyNewer: dedupAgainstSeed);
|
||||
}
|
||||
_buffer.Clear();
|
||||
return changed;
|
||||
}
|
||||
|
||||
// ── Live delta handling ─────────────────────────────────────────────────────
|
||||
@@ -427,15 +694,32 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
|
||||
|
||||
if (_fanoutInFlight)
|
||||
{
|
||||
_buffer.Add(delta);
|
||||
if (!_bufferWarned && _buffer.Count > BufferWarnThreshold)
|
||||
if (!_bufferWarned && _buffer.Count + 1 > BufferWarnThreshold)
|
||||
{
|
||||
_bufferWarned = true;
|
||||
_log.Warning(
|
||||
"Site-alarm pre-seed buffer for {0} exceeded {1} deltas while a fan-out was in flight " +
|
||||
"(deltas retained, not dropped).",
|
||||
_siteIdentifier, BufferWarnThreshold);
|
||||
"(hard cap {2}, drop-oldest beyond it).",
|
||||
_siteIdentifier, BufferWarnThreshold, MaxBufferedDeltas);
|
||||
}
|
||||
|
||||
// Hard cap with drop-oldest: an unbounded buffer behind a stuck fan-out was a
|
||||
// straight memory leak. The evicted rows are superseded by the snapshot that
|
||||
// ends the fan-out, so the drop costs freshness, never correctness.
|
||||
while (_buffer.Count >= MaxBufferedDeltas)
|
||||
{
|
||||
_buffer.Dequeue();
|
||||
_bufferDropped++;
|
||||
ScadaBridgeTelemetry.RecordLiveAlarmBufferDrop();
|
||||
if (_bufferDropped == 1 || _bufferDropped % 500 == 0)
|
||||
{
|
||||
_log.Warning(
|
||||
"Site-alarm pre-seed buffer for {0} is at its {1}-delta cap; {2} delta(s) evicted so far",
|
||||
_siteIdentifier, MaxBufferedDeltas, _bufferDropped);
|
||||
}
|
||||
}
|
||||
|
||||
_buffer.Enqueue(delta);
|
||||
return;
|
||||
}
|
||||
|
||||
@@ -485,7 +769,10 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
|
||||
var snapshot = _cache.Values.ToList();
|
||||
try
|
||||
{
|
||||
_publish(snapshot);
|
||||
// Liveness rides every publish so the owning service's IsLive tracks the STREAM,
|
||||
// not merely "a snapshot was published once". A down stream means the cache is
|
||||
// only as fresh as the last reconcile and the page must keep polling.
|
||||
_publish(snapshot, !_streamDown);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
@@ -499,7 +786,8 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
|
||||
{
|
||||
if (_stopped) return;
|
||||
|
||||
_streamDown = false;
|
||||
// The stream is not live again until the server actually accepts it — liveness flips
|
||||
// on GrpcAlarmStreamConnected, not on the attempt (WP2.3 carried residual).
|
||||
var endpoint = _useNodeA ? _grpcNodeAAddress : _grpcNodeBAddress;
|
||||
_log.Info("Opening site-alarm gRPC stream for {0} to {1}", _siteIdentifier, endpoint);
|
||||
|
||||
@@ -509,19 +797,59 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
|
||||
|
||||
Timers.StartSingleTimer(StabilityTimerKey, new GrpcAlarmStreamStable(), _stabilityWindow);
|
||||
|
||||
_openInFlight = true;
|
||||
var generation = ++_streamGeneration;
|
||||
var client = _grpcFactory.GetOrCreate(_siteIdentifier, endpoint);
|
||||
var self = Self;
|
||||
var ct = _grpcCts.Token;
|
||||
|
||||
// The subscription task itself is observed (below): a fault escaping
|
||||
// SubscribeSiteAsync — or a Task.Run that never started because ct was already
|
||||
// cancelled — would otherwise leave the actor waiting on a stream that does not
|
||||
// exist, with the exception silently unobserved.
|
||||
Task.Run(async () =>
|
||||
{
|
||||
await client.SubscribeSiteAsync(
|
||||
_correlationId,
|
||||
alarm => self.Tell(alarm),
|
||||
ex => self.Tell(new GrpcAlarmStreamError(ex, generation)),
|
||||
ct);
|
||||
}, ct);
|
||||
() => self.Tell(new GrpcAlarmStreamCompleted(generation)),
|
||||
ct,
|
||||
() => self.Tell(new GrpcAlarmStreamConnected(generation)));
|
||||
}, ct).ContinueWith(t =>
|
||||
{
|
||||
if (t.IsFaulted)
|
||||
self.Tell(new GrpcAlarmStreamError(t.Exception!.GetBaseException(), generation));
|
||||
else if (t.IsCanceled && !ct.IsCancellationRequested)
|
||||
self.Tell(new GrpcAlarmStreamCompleted(generation));
|
||||
// RanToCompletion: SubscribeSiteAsync already reported its own outcome.
|
||||
}, TaskContinuationOptions.ExecuteSynchronously);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The site accepted the subscription — the only point at which the stream is genuinely
|
||||
/// live. Consumes the pending re-seed (so exactly ONE fan-out runs per successful
|
||||
/// (re)connect, instead of one per reconnect attempt against a site that is by definition
|
||||
/// unreachable) and republishes so <c>IsLive</c> recovers.
|
||||
/// </summary>
|
||||
private void OnStreamConnected()
|
||||
{
|
||||
_openInFlight = false;
|
||||
|
||||
var wasDown = _streamDown;
|
||||
_streamDown = false;
|
||||
|
||||
if (_seedOnConnect)
|
||||
{
|
||||
_seedOnConnect = false;
|
||||
_log.Info("Site-alarm gRPC stream for {0} (re)connected; running one re-seed", _siteIdentifier);
|
||||
StartFanout(isInitial: false, suppressesNextTick: true);
|
||||
}
|
||||
|
||||
// Liveness recovered — republish so viewers stop falling back to polling even if the
|
||||
// seed that follows finds nothing changed.
|
||||
if (wasDown && _seeded)
|
||||
Publish();
|
||||
}
|
||||
|
||||
private void HandleGrpcError()
|
||||
@@ -531,6 +859,19 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
|
||||
// Stream failed before the stability window — retry budget NOT recovered.
|
||||
Timers.Cancel(StabilityTimerKey);
|
||||
|
||||
_openInFlight = false;
|
||||
var wasLive = !_streamDown;
|
||||
_streamDown = true;
|
||||
|
||||
// The next successful connect owes exactly one re-seed. Setting it here (rather than
|
||||
// fanning out now) is the fix for the per-attempt re-fan-out: a site outage used to
|
||||
// cost one whole-site snapshot per retry.
|
||||
_seedOnConnect = true;
|
||||
|
||||
// Liveness lost — tell viewers immediately so a dead stream stops reading as live.
|
||||
if (wasLive && _seeded)
|
||||
Publish();
|
||||
|
||||
_retryCount++;
|
||||
|
||||
if (_retryCount > MaxRetries)
|
||||
@@ -542,7 +883,7 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
|
||||
_log.Error("Site-alarm gRPC stream for {0} exceeded max retries ({1}); leaving stream down, " +
|
||||
"reconcile snapshots continue and the next reconcile tick will retry the stream",
|
||||
_siteIdentifier, MaxRetries);
|
||||
_streamDown = true;
|
||||
_retryBudgetExhausted = true;
|
||||
CleanupGrpc();
|
||||
return;
|
||||
}
|
||||
@@ -558,17 +899,44 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
|
||||
// Telemetry: a NodeA↔NodeB failover flip is a reconnect + re-seed.
|
||||
ScadaBridgeTelemetry.RecordLiveAlarmStreamReconnect();
|
||||
|
||||
// A failover flip must RE-SEED (never silently serve stale) — kick a reconcile
|
||||
// fan-out alongside the reconnect. Buffering during the fan-out keeps the new
|
||||
// stream's deltas coherent with the fresh snapshot.
|
||||
StartFanout(isInitial: false);
|
||||
|
||||
// The re-seed is owed to the CONNECT (see _seedOnConnect above), not to this
|
||||
// attempt: a snapshot fan-out issued while the site is unreachable degrades to
|
||||
// empty rows and would have to be redone on reconnect anyway.
|
||||
if (_retryCount == 1)
|
||||
Self.Tell(new ReconnectAlarmStream());
|
||||
else
|
||||
Timers.StartSingleTimer(ReconnectTimerKey, new ReconnectAlarmStream(), _reconnectDelay);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Handles a graceful end of stream. The stream is torn down and left down for the
|
||||
/// reconcile tick to reopen — deliberately NOT reopened inline, so a site that keeps
|
||||
/// completing streams immediately can never spin this actor into a hot reconnect loop.
|
||||
/// The retry budget is neither spent nor reset here (completion is not a fault), and the
|
||||
/// endpoint is not flipped: the node that just closed a stream cleanly is healthy.
|
||||
/// </summary>
|
||||
private void HandleGrpcCompleted()
|
||||
{
|
||||
// The stream is gone, so its armed stability timer must not later "recover" a
|
||||
// budget that its successor has since spent.
|
||||
Timers.Cancel(StabilityTimerKey);
|
||||
|
||||
_log.Info("Site-alarm gRPC stream for {0} completed gracefully (server end of stream); " +
|
||||
"reopening on the next reconcile tick", _siteIdentifier);
|
||||
|
||||
_openInFlight = false;
|
||||
var wasLive = !_streamDown;
|
||||
_streamDown = true;
|
||||
// The reopen owes one re-seed, run when it actually connects.
|
||||
_seedOnConnect = true;
|
||||
CleanupGrpc();
|
||||
|
||||
// A completed stream is NOT live: without this the cache kept reporting live for up
|
||||
// to a full reconcile interval after the site closed the feed (WP2.3 residual).
|
||||
if (wasLive && _seeded)
|
||||
Publish();
|
||||
}
|
||||
|
||||
private void CleanupGrpc()
|
||||
{
|
||||
_grpcCts?.Cancel();
|
||||
@@ -612,6 +980,14 @@ internal sealed record SeedFailed(Exception Exception, bool IsInitial);
|
||||
/// <summary>Internal: periodic reconcile tick (and the re-seed kicked after a reconnect).</summary>
|
||||
internal sealed record RunReconcile;
|
||||
|
||||
/// <summary>Internal: backoff-scheduled retry of a fan-out that failed as a whole.</summary>
|
||||
internal sealed record RetrySeed;
|
||||
|
||||
/// <summary>Internal: the site accepted the site-wide alarm subscription (response headers
|
||||
/// received), stamped with its stream generation so a late connect from a cancelled stream
|
||||
/// is ignored. This — not the reconnect attempt — is what earns a re-seed.</summary>
|
||||
internal sealed record GrpcAlarmStreamConnected(int Generation);
|
||||
|
||||
/// <summary>Internal: coalesced-publish tick — flush the dirty cache to viewers once (N6).</summary>
|
||||
internal sealed record PublishCoalesced;
|
||||
|
||||
@@ -619,6 +995,11 @@ internal sealed record PublishCoalesced;
|
||||
/// generation it came from so a late error from a cancelled stream can be ignored (N7.2).</summary>
|
||||
internal sealed record GrpcAlarmStreamError(Exception Exception, int Generation);
|
||||
|
||||
/// <summary>Internal: the site-alarm gRPC stream ended gracefully (server status OK — max
|
||||
/// stream lifetime or site shutdown), stamped with its stream generation so a late completion
|
||||
/// from a cancelled stream can be ignored.</summary>
|
||||
internal sealed record GrpcAlarmStreamCompleted(int Generation);
|
||||
|
||||
/// <summary>Internal: reconnect the site-alarm gRPC stream (flip node).</summary>
|
||||
internal sealed record ReconnectAlarmStream;
|
||||
|
||||
|
||||
@@ -114,33 +114,34 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
|
||||
"c0R0bzoCOAEaQgogU3RvcmVBbmRGb3J3YXJkQnVmZmVyRGVwdGhzRW50cnkS",
|
||||
"CwoDa2V5GAEgASgJEg0KBXZhbHVlGAIgASgFOgI4ASJjChZTaXRlSGVhbHRo",
|
||||
"UmVwb3J0QWNrRHRvEg8KB3NpdGVfaWQYASABKAkSFwoPc2VxdWVuY2VfbnVt",
|
||||
"YmVyGAIgASgDEhAKCGFjY2VwdGVkGAMgASgIEg0KBWVycm9yGAQgASgJIngK",
|
||||
"DEhlYXJ0YmVhdER0bxIPCgdzaXRlX2lkGAEgASgJEhUKDW5vZGVfaG9zdG5h",
|
||||
"bWUYAiABKAkSEQoJaXNfYWN0aXZlGAMgASgIEi0KCXRpbWVzdGFtcBgEIAEo",
|
||||
"CzIaLmdvb2dsZS5wcm90b2J1Zi5UaW1lc3RhbXAqvQEKFENvbm5lY3Rpb25I",
|
||||
"ZWFsdGhFbnVtEiEKHUNPTk5FQ1RJT05fSEVBTFRIX1VOU1BFQ0lGSUVEEAAS",
|
||||
"HwobQ09OTkVDVElPTl9IRUFMVEhfQ09OTkVDVEVEEAESIgoeQ09OTkVDVElP",
|
||||
"Tl9IRUFMVEhfRElTQ09OTkVDVEVEEAISIAocQ09OTkVDVElPTl9IRUFMVEhf",
|
||||
"Q09OTkVDVElORxADEhsKF0NPTk5FQ1RJT05fSEVBTFRIX0VSUk9SEAQyoQYK",
|
||||
"FUNlbnRyYWxDb250cm9sU2VydmljZRKDAQoSU3VibWl0Tm90aWZpY2F0aW9u",
|
||||
"EjQuc2NhZGFicmlkZ2UuY2VudHJhbGNvbnRyb2wudjEuTm90aWZpY2F0aW9u",
|
||||
"U3VibWl0RHRvGjcuc2NhZGFicmlkZ2UuY2VudHJhbGNvbnRyb2wudjEuTm90",
|
||||
"aWZpY2F0aW9uU3VibWl0QWNrRHRvEpIBChdRdWVyeU5vdGlmaWNhdGlvblN0",
|
||||
"YXR1cxI5LnNjYWRhYnJpZGdlLmNlbnRyYWxjb250cm9sLnYxLk5vdGlmaWNh",
|
||||
"dGlvblN0YXR1c1F1ZXJ5RHRvGjwuc2NhZGFicmlkZ2UuY2VudHJhbGNvbnRy",
|
||||
"b2wudjEuTm90aWZpY2F0aW9uU3RhdHVzUmVzcG9uc2VEdG8SRwoRSW5nZXN0",
|
||||
"QXVkaXRFdmVudHMSGy5zaXRlc3RyZWFtLkF1ZGl0RXZlbnRCYXRjaBoVLnNp",
|
||||
"dGVzdHJlYW0uSW5nZXN0QWNrElAKFUluZ2VzdENhY2hlZFRlbGVtZXRyeRIg",
|
||||
"LnNpdGVzdHJlYW0uQ2FjaGVkVGVsZW1ldHJ5QmF0Y2gaFS5zaXRlc3RyZWFt",
|
||||
"LkluZ2VzdEFjaxKAAQoNUmVjb25jaWxlU2l0ZRI2LnNjYWRhYnJpZGdlLmNl",
|
||||
"bnRyYWxjb250cm9sLnYxLlJlY29uY2lsZVNpdGVSZXF1ZXN0RHRvGjcuc2Nh",
|
||||
"ZGFicmlkZ2UuY2VudHJhbGNvbnRyb2wudjEuUmVjb25jaWxlU2l0ZVJlc3Bv",
|
||||
"bnNlRHRvEn0KEFJlcG9ydFNpdGVIZWFsdGgSMi5zY2FkYWJyaWRnZS5jZW50",
|
||||
"cmFsY29udHJvbC52MS5TaXRlSGVhbHRoUmVwb3J0RHRvGjUuc2NhZGFicmlk",
|
||||
"Z2UuY2VudHJhbGNvbnRyb2wudjEuU2l0ZUhlYWx0aFJlcG9ydEFja0R0bxJQ",
|
||||
"CglIZWFydGJlYXQSKy5zY2FkYWJyaWRnZS5jZW50cmFsY29udHJvbC52MS5I",
|
||||
"ZWFydGJlYXREdG8aFi5nb29nbGUucHJvdG9idWYuRW1wdHlCK6oCKFpCLk1P",
|
||||
"TS5XVy5TY2FkYUJyaWRnZS5Db21tdW5pY2F0aW9uLkdycGNiBnByb3RvMw=="));
|
||||
"YmVyGAIgASgDEhAKCGFjY2VwdGVkGAMgASgIEg0KBWVycm9yGAQgASgJIosB",
|
||||
"CgxIZWFydGJlYXREdG8SDwoHc2l0ZV9pZBgBIAEoCRIVCg1ub2RlX2hvc3Ru",
|
||||
"YW1lGAIgASgJEhEKCWlzX2FjdGl2ZRgDIAEoCBItCgl0aW1lc3RhbXAYBCAB",
|
||||
"KAsyGi5nb29nbGUucHJvdG9idWYuVGltZXN0YW1wEhEKCXN5bnRoZXRpYxgF",
|
||||
"IAEoCCq9AQoUQ29ubmVjdGlvbkhlYWx0aEVudW0SIQodQ09OTkVDVElPTl9I",
|
||||
"RUFMVEhfVU5TUEVDSUZJRUQQABIfChtDT05ORUNUSU9OX0hFQUxUSF9DT05O",
|
||||
"RUNURUQQARIiCh5DT05ORUNUSU9OX0hFQUxUSF9ESVNDT05ORUNURUQQAhIg",
|
||||
"ChxDT05ORUNUSU9OX0hFQUxUSF9DT05ORUNUSU5HEAMSGwoXQ09OTkVDVElP",
|
||||
"Tl9IRUFMVEhfRVJST1IQBDKhBgoVQ2VudHJhbENvbnRyb2xTZXJ2aWNlEoMB",
|
||||
"ChJTdWJtaXROb3RpZmljYXRpb24SNC5zY2FkYWJyaWRnZS5jZW50cmFsY29u",
|
||||
"dHJvbC52MS5Ob3RpZmljYXRpb25TdWJtaXREdG8aNy5zY2FkYWJyaWRnZS5j",
|
||||
"ZW50cmFsY29udHJvbC52MS5Ob3RpZmljYXRpb25TdWJtaXRBY2tEdG8SkgEK",
|
||||
"F1F1ZXJ5Tm90aWZpY2F0aW9uU3RhdHVzEjkuc2NhZGFicmlkZ2UuY2VudHJh",
|
||||
"bGNvbnRyb2wudjEuTm90aWZpY2F0aW9uU3RhdHVzUXVlcnlEdG8aPC5zY2Fk",
|
||||
"YWJyaWRnZS5jZW50cmFsY29udHJvbC52MS5Ob3RpZmljYXRpb25TdGF0dXNS",
|
||||
"ZXNwb25zZUR0bxJHChFJbmdlc3RBdWRpdEV2ZW50cxIbLnNpdGVzdHJlYW0u",
|
||||
"QXVkaXRFdmVudEJhdGNoGhUuc2l0ZXN0cmVhbS5Jbmdlc3RBY2sSUAoVSW5n",
|
||||
"ZXN0Q2FjaGVkVGVsZW1ldHJ5EiAuc2l0ZXN0cmVhbS5DYWNoZWRUZWxlbWV0",
|
||||
"cnlCYXRjaBoVLnNpdGVzdHJlYW0uSW5nZXN0QWNrEoABCg1SZWNvbmNpbGVT",
|
||||
"aXRlEjYuc2NhZGFicmlkZ2UuY2VudHJhbGNvbnRyb2wudjEuUmVjb25jaWxl",
|
||||
"U2l0ZVJlcXVlc3REdG8aNy5zY2FkYWJyaWRnZS5jZW50cmFsY29udHJvbC52",
|
||||
"MS5SZWNvbmNpbGVTaXRlUmVzcG9uc2VEdG8SfQoQUmVwb3J0U2l0ZUhlYWx0",
|
||||
"aBIyLnNjYWRhYnJpZGdlLmNlbnRyYWxjb250cm9sLnYxLlNpdGVIZWFsdGhS",
|
||||
"ZXBvcnREdG8aNS5zY2FkYWJyaWRnZS5jZW50cmFsY29udHJvbC52MS5TaXRl",
|
||||
"SGVhbHRoUmVwb3J0QWNrRHRvElAKCUhlYXJ0YmVhdBIrLnNjYWRhYnJpZGdl",
|
||||
"LmNlbnRyYWxjb250cm9sLnYxLkhlYXJ0YmVhdER0bxoWLmdvb2dsZS5wcm90",
|
||||
"b2J1Zi5FbXB0eUIrqgIoWkIuTU9NLldXLlNjYWRhQnJpZGdlLkNvbW11bmlj",
|
||||
"YXRpb24uR3JwY2IGcHJvdG8z"));
|
||||
descriptor = pbr::FileDescriptor.FromGeneratedCode(descriptorData,
|
||||
new pbr::FileDescriptor[] { global::Google.Protobuf.WellKnownTypes.EmptyReflection.Descriptor, global::Google.Protobuf.WellKnownTypes.TimestampReflection.Descriptor, global::Google.Protobuf.WellKnownTypes.WrappersReflection.Descriptor, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor, },
|
||||
new pbr::GeneratedClrTypeInfo(new[] {typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ConnectionHealthEnum), }, null, new pbr::GeneratedClrTypeInfo[] {
|
||||
@@ -160,7 +161,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
|
||||
new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NodeStatusListDto), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NodeStatusListDto.Parser, new[]{ "Nodes" }, null, null, null, null),
|
||||
new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportDto), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportDto.Parser, new[]{ "SiteId", "SequenceNumber", "ReportTimestamp", "DataConnectionStatuses", "TagResolutionCounts", "ScriptErrorCount", "AlarmEvaluationErrorCount", "StoreAndForwardBufferDepths", "DeadLetterCount", "DeployedInstanceCount", "EnabledInstanceCount", "DisabledInstanceCount", "NodeRole", "NodeHostname", "DataConnectionEndpoints", "DataConnectionTagQuality", "ParkedMessageCount", "ClusterNodes", "SiteAuditWriteFailures", "AuditRedactionFailure", "SiteAuditBacklog", "SiteEventLogWriteFailures", "OldestParkedMessageAgeSeconds", "ScriptQueueDepth", "ScriptBusyThreads", "ScriptOldestBusyAgeSeconds", "LocalDbReplicationConnected", "LocalDbOplogBacklog" }, null, null, null, new pbr::GeneratedClrTypeInfo[] { null, null, null, }),
|
||||
new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportAckDto), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportAckDto.Parser, new[]{ "SiteId", "SequenceNumber", "Accepted", "Error" }, null, null, null, null),
|
||||
new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.HeartbeatDto), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.HeartbeatDto.Parser, new[]{ "SiteId", "NodeHostname", "IsActive", "Timestamp" }, null, null, null, null)
|
||||
new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.HeartbeatDto), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.HeartbeatDto.Parser, new[]{ "SiteId", "NodeHostname", "IsActive", "Timestamp", "Synthetic" }, null, null, null, null)
|
||||
}));
|
||||
}
|
||||
#endregion
|
||||
@@ -5826,6 +5827,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
|
||||
nodeHostname_ = other.nodeHostname_;
|
||||
isActive_ = other.isActive_;
|
||||
timestamp_ = other.timestamp_ != null ? other.timestamp_.Clone() : null;
|
||||
synthetic_ = other.synthetic_;
|
||||
_unknownFields = pb::UnknownFieldSet.Clone(other._unknownFields);
|
||||
}
|
||||
|
||||
@@ -5883,6 +5885,27 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Field number for the "synthetic" field.</summary>
|
||||
public const int SyntheticFieldNumber = 5;
|
||||
private bool synthetic_;
|
||||
/// <summary>
|
||||
/// Additive (field 5): true marks a SYNTHETIC heartbeat that carries no liveness
|
||||
/// meaning — today only CentralChannelProvider's failback probe, which reuses this
|
||||
/// RPC purely to test whether the preferred central endpoint answers again. Central
|
||||
/// must skip ALL liveness/health bookkeeping for such a heartbeat: it does not come
|
||||
/// from a site node's heartbeat timer, and treating it as one would keep a site
|
||||
/// looking alive on the health dashboard from the probe alone. proto3 defaults it to
|
||||
/// false, so a pre-existing site's heartbeat is a real one, as before.
|
||||
/// </summary>
|
||||
[global::System.Diagnostics.DebuggerNonUserCodeAttribute]
|
||||
[global::System.CodeDom.Compiler.GeneratedCode("protoc", null)]
|
||||
public bool Synthetic {
|
||||
get { return synthetic_; }
|
||||
set {
|
||||
synthetic_ = value;
|
||||
}
|
||||
}
|
||||
|
||||
[global::System.Diagnostics.DebuggerNonUserCodeAttribute]
|
||||
[global::System.CodeDom.Compiler.GeneratedCode("protoc", null)]
|
||||
public override bool Equals(object other) {
|
||||
@@ -5902,6 +5925,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
|
||||
if (NodeHostname != other.NodeHostname) return false;
|
||||
if (IsActive != other.IsActive) return false;
|
||||
if (!object.Equals(Timestamp, other.Timestamp)) return false;
|
||||
if (Synthetic != other.Synthetic) return false;
|
||||
return Equals(_unknownFields, other._unknownFields);
|
||||
}
|
||||
|
||||
@@ -5913,6 +5937,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
|
||||
if (NodeHostname.Length != 0) hash ^= NodeHostname.GetHashCode();
|
||||
if (IsActive != false) hash ^= IsActive.GetHashCode();
|
||||
if (timestamp_ != null) hash ^= Timestamp.GetHashCode();
|
||||
if (Synthetic != false) hash ^= Synthetic.GetHashCode();
|
||||
if (_unknownFields != null) {
|
||||
hash ^= _unknownFields.GetHashCode();
|
||||
}
|
||||
@@ -5947,6 +5972,10 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
|
||||
output.WriteRawTag(34);
|
||||
output.WriteMessage(Timestamp);
|
||||
}
|
||||
if (Synthetic != false) {
|
||||
output.WriteRawTag(40);
|
||||
output.WriteBool(Synthetic);
|
||||
}
|
||||
if (_unknownFields != null) {
|
||||
_unknownFields.WriteTo(output);
|
||||
}
|
||||
@@ -5973,6 +6002,10 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
|
||||
output.WriteRawTag(34);
|
||||
output.WriteMessage(Timestamp);
|
||||
}
|
||||
if (Synthetic != false) {
|
||||
output.WriteRawTag(40);
|
||||
output.WriteBool(Synthetic);
|
||||
}
|
||||
if (_unknownFields != null) {
|
||||
_unknownFields.WriteTo(ref output);
|
||||
}
|
||||
@@ -5995,6 +6028,9 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
|
||||
if (timestamp_ != null) {
|
||||
size += 1 + pb::CodedOutputStream.ComputeMessageSize(Timestamp);
|
||||
}
|
||||
if (Synthetic != false) {
|
||||
size += 1 + 1;
|
||||
}
|
||||
if (_unknownFields != null) {
|
||||
size += _unknownFields.CalculateSize();
|
||||
}
|
||||
@@ -6022,6 +6058,9 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
|
||||
}
|
||||
Timestamp.MergeFrom(other.Timestamp);
|
||||
}
|
||||
if (other.Synthetic != false) {
|
||||
Synthetic = other.Synthetic;
|
||||
}
|
||||
_unknownFields = pb::UnknownFieldSet.MergeFrom(_unknownFields, other._unknownFields);
|
||||
}
|
||||
|
||||
@@ -6060,6 +6099,10 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
|
||||
input.ReadMessage(Timestamp);
|
||||
break;
|
||||
}
|
||||
case 40: {
|
||||
Synthetic = input.ReadBool();
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
#endif
|
||||
@@ -6098,6 +6141,10 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
|
||||
input.ReadMessage(Timestamp);
|
||||
break;
|
||||
}
|
||||
case 40: {
|
||||
Synthetic = input.ReadBool();
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -47,6 +47,36 @@ public class CommunicationOptions
|
||||
/// </summary>
|
||||
public TimeSpan NotificationForwardTimeout { get; set; } = TimeSpan.FromSeconds(30);
|
||||
|
||||
/// <summary>
|
||||
/// Audit Log: timeout for the SITE-side Ask that forwards one audit-telemetry batch
|
||||
/// (<c>IngestAuditEventsCommand</c> / <c>IngestCachedTelemetryCommand</c>) through the site's
|
||||
/// <c>SiteCommunicationActor</c> and awaits central's ack. Deliberately the LONGEST rung of the
|
||||
/// ingest timeout ladder.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// The ladder is strictly monotonic, outermost first:
|
||||
/// <c>AuditForwardTimeout</c> (35 s, this option) >
|
||||
/// <c>SiteStreamGrpcServer.AuditIngestAskTimeout</c> (30 s — both the gRPC call deadline and
|
||||
/// central's own Ask of the ingest singleton) >
|
||||
/// <c>AuditLogIngestActor.IngestBudget</c> (20 s) >
|
||||
/// <c>AuditLogIngestActor.IngestSqlCommandTimeout</c> (15 s).
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Strictness matters: it used to reuse <see cref="NotificationForwardTimeout"/> (30 s), the
|
||||
/// same value as the rung below it, so a slow-but-succeeding central write could be acked to a
|
||||
/// caller whose Ask had already expired — the drain loop would treat the batch as unsent and
|
||||
/// re-ship it. Central dedups on <c>EventId</c>, so the duplicate was harmless but the retry
|
||||
/// traffic and the "stalled" telemetry signal were not. A timeout here is still transient: the
|
||||
/// rows stay <c>Pending</c> and drain on the next tick.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <see cref="CommunicationOptionsValidator"/> enforces the outermost inequality — this value
|
||||
/// must be strictly greater than the 30 s gRPC/central Ask rung.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public TimeSpan AuditForwardTimeout { get; set; } = TimeSpan.FromSeconds(35);
|
||||
|
||||
/// <summary>
|
||||
/// Preshared key authenticating this node's gRPC control plane — the site↔central
|
||||
/// boundary. On a site node this is the key its inbound gate
|
||||
@@ -107,6 +137,21 @@ public class CommunicationOptions
|
||||
/// <summary>Maximum number of concurrent gRPC streaming subscriptions per site node.</summary>
|
||||
public int GrpcMaxConcurrentStreams { get; set; } = 100;
|
||||
|
||||
/// <summary>
|
||||
/// Send-channel capacity for a per-instance Debug View stream (<c>SubscribeInstance</c>).
|
||||
/// Lossy by design: the Debug View is a diagnostic surface and drops its oldest events
|
||||
/// under backpressure rather than stalling the site's event hub.
|
||||
/// </summary>
|
||||
public int GrpcInstanceStreamChannelCapacity { get; set; } = 1000;
|
||||
|
||||
/// <summary>
|
||||
/// Send-channel capacity for the site-wide alarm stream (<c>SubscribeSite</c>). Larger
|
||||
/// than the Debug View's (WP2.3): that feed backs the operator Alarm Summary, where a
|
||||
/// silently dropped transition is a missed alarm rather than a missed diagnostic frame,
|
||||
/// and an alarm burst arriving during a WAN stall must survive the stall.
|
||||
/// </summary>
|
||||
public int GrpcSiteAlarmStreamChannelCapacity { get; set; } = 20_000;
|
||||
|
||||
/// <summary>Akka.Remote transport heartbeat interval.</summary>
|
||||
public TimeSpan TransportHeartbeatInterval { get; set; } = TimeSpan.FromSeconds(5);
|
||||
|
||||
@@ -185,4 +230,12 @@ public class CommunicationOptions
|
||||
/// reconcile publishes are always immediate. Default 250 ms.
|
||||
/// </summary>
|
||||
public TimeSpan LiveAlarmCachePublishCoalesce { get; set; } = TimeSpan.FromMilliseconds(250);
|
||||
|
||||
/// <summary>
|
||||
/// Random jitter added to each per-site reconcile tick, as a fraction of
|
||||
/// <see cref="LiveAlarmCacheReconcileInterval"/> (WP2.3). Without it every aggregator
|
||||
/// started by one central failover fans its whole-site snapshot out on the same 60s
|
||||
/// boundary forever. Zero disables jitter.
|
||||
/// </summary>
|
||||
public double LiveAlarmCacheReconcileJitterFraction { get; set; } = 0.2;
|
||||
}
|
||||
|
||||
@@ -39,6 +39,15 @@ public sealed class CommunicationOptionsValidator : OptionsValidatorBase<Communi
|
||||
builder.RequireThat(options.NotificationForwardTimeout > TimeSpan.Zero,
|
||||
$"ScadaBridge:Communication:NotificationForwardTimeout must be a positive duration (was {options.NotificationForwardTimeout}).");
|
||||
|
||||
// Outermost rung of the audit-ingest timeout ladder. It must be strictly greater than the
|
||||
// gRPC deadline / central Ask beneath it, or a slow-but-succeeding central write gets acked
|
||||
// to a caller that already gave up and re-sent the batch. See CommunicationOptions.
|
||||
builder.RequireThat(
|
||||
options.AuditForwardTimeout > Grpc.SiteStreamGrpcServer.AuditIngestAskTimeout,
|
||||
$"ScadaBridge:Communication:AuditForwardTimeout must be strictly greater than the "
|
||||
+ $"central audit-ingest Ask timeout ({Grpc.SiteStreamGrpcServer.AuditIngestAskTimeout}) "
|
||||
+ $"so the ingest timeout ladder stays monotonic (was {options.AuditForwardTimeout}).");
|
||||
|
||||
builder.RequireThat(options.GrpcKeepAlivePingDelay > TimeSpan.Zero,
|
||||
$"ScadaBridge:Communication:GrpcKeepAlivePingDelay must be a positive duration (was {options.GrpcKeepAlivePingDelay}).");
|
||||
|
||||
|
||||
@@ -13,14 +13,34 @@ namespace ZB.MOM.WW.ScadaBridge.Communication;
|
||||
/// Manages debug stream sessions by creating DebugStreamBridgeActors that persist
|
||||
/// as subscribers on the site side. Both the Blazor debug view and the SignalR hub
|
||||
/// use this service to start/stop streams.
|
||||
/// <para>
|
||||
/// <b>Consumer keepalive.</b> This service is the session registry, and holding a session
|
||||
/// here IS what "a consumer is attached" means: the Blazor debug view and the SignalR hub
|
||||
/// both release their session (<see cref="StopStream"/>) on dispose/disconnect. A single
|
||||
/// shared timer therefore Tells <see cref="DebugStreamConsumerAlive"/> to every registered
|
||||
/// bridge actor, which is the only thing that renews each actor's orphan window. A bridge
|
||||
/// actor that outlives its registration — the leak the orphan net exists for — stops
|
||||
/// receiving keepalives and self-terminates. The keepalive is independent of stream traffic
|
||||
/// by design: neither a chatty nor a silent site can influence the orphan decision.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public class DebugStreamService
|
||||
public class DebugStreamService : IDisposable
|
||||
{
|
||||
/// <summary>
|
||||
/// Cadence of the shared consumer keepalive. Comfortably shorter than
|
||||
/// <see cref="DebugStreamBridgeActor.ConsumerIdleTimeout"/> so a few missed ticks (GC
|
||||
/// pause, thread-pool starvation) can never orphan a live session. Settable for tests.
|
||||
/// </summary>
|
||||
internal static TimeSpan KeepaliveInterval { get; set; } = TimeSpan.FromSeconds(30);
|
||||
|
||||
private readonly CommunicationService _communicationService;
|
||||
private readonly IServiceProvider _serviceProvider;
|
||||
private readonly SiteStreamGrpcClientFactory _grpcClientFactory;
|
||||
private readonly ILogger<DebugStreamService> _logger;
|
||||
private readonly ConcurrentDictionary<string, IActorRef> _sessions = new();
|
||||
private readonly object _keepaliveLock = new();
|
||||
private Timer? _keepaliveTimer;
|
||||
private bool _disposed;
|
||||
private ActorSystem? _actorSystem;
|
||||
|
||||
/// <summary>
|
||||
@@ -135,6 +155,7 @@ public class DebugStreamService
|
||||
var bridgeActor = system.ActorOf(props, $"debug-stream-{sessionId}");
|
||||
|
||||
_sessions[sessionId] = bridgeActor;
|
||||
EnsureKeepaliveTimer();
|
||||
|
||||
// Wait for the initial snapshot (with timeout)
|
||||
using var timeoutCts = CancellationTokenSource.CreateLinkedTokenSource(ct);
|
||||
@@ -184,6 +205,73 @@ public class DebugStreamService
|
||||
_logger.LogInformation("Debug stream {SessionId} stopped", sessionId);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Sends one keepalive round to every attached session. Exposed (internal) so tests can
|
||||
/// drive the keepalive deterministically instead of waiting on the timer.
|
||||
/// </summary>
|
||||
internal void SendConsumerKeepalives()
|
||||
{
|
||||
foreach (var session in _sessions)
|
||||
{
|
||||
session.Value.Tell(new DebugStreamConsumerAlive());
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Starts the shared keepalive timer on first use. One timer for the whole service (not
|
||||
/// one per session): the payload is a Tell per attached session, so a single periodic
|
||||
/// callback is the cheapest shape that covers every consumer surface.
|
||||
/// </summary>
|
||||
private void EnsureKeepaliveTimer()
|
||||
{
|
||||
if (_keepaliveTimer is not null) return;
|
||||
lock (_keepaliveLock)
|
||||
{
|
||||
if (_keepaliveTimer is not null || _disposed) return;
|
||||
_keepaliveTimer = new Timer(
|
||||
_ =>
|
||||
{
|
||||
try
|
||||
{
|
||||
SendConsumerKeepalives();
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
// A keepalive round must never take the timer down — a thrown Tell
|
||||
// would silently stop renewing EVERY session's orphan window.
|
||||
_logger.LogWarning(ex, "Debug stream consumer keepalive round failed");
|
||||
}
|
||||
},
|
||||
null,
|
||||
KeepaliveInterval,
|
||||
KeepaliveInterval);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Stops the keepalive timer.</summary>
|
||||
public void Dispose()
|
||||
{
|
||||
Dispose(true);
|
||||
GC.SuppressFinalize(this);
|
||||
}
|
||||
|
||||
/// <summary>Stops the keepalive timer.</summary>
|
||||
/// <param name="disposing">True when called from <see cref="Dispose()"/>.</param>
|
||||
protected virtual void Dispose(bool disposing)
|
||||
{
|
||||
if (_disposed) return;
|
||||
lock (_keepaliveLock)
|
||||
{
|
||||
if (_disposed) return;
|
||||
_disposed = true;
|
||||
if (disposing)
|
||||
{
|
||||
_keepaliveTimer?.Dispose();
|
||||
_keepaliveTimer = null;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
public record DebugStreamSession(string SessionId, DebugViewSnapshot InitialSnapshot);
|
||||
|
||||
@@ -38,6 +38,15 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc;
|
||||
/// </remarks>
|
||||
public sealed class CentralChannelProvider : IDisposable
|
||||
{
|
||||
/// <summary>
|
||||
/// Hostname stamped on the failback probe's heartbeat, so a central-side log line names it
|
||||
/// for what it is. It is a LABEL only — the machine-readable contract is
|
||||
/// <see cref="ZB.MOM.WW.ScadaBridge.Commons.Messages.Health.HeartbeatMessage.Synthetic"/>
|
||||
/// (<c>HeartbeatDto.synthetic</c>, proto field 5), which central keys its skip on. Never
|
||||
/// filter on this string.
|
||||
/// </summary>
|
||||
public const string SyntheticProbeHostname = "failback-probe";
|
||||
|
||||
private static readonly TimeSpan DefaultBackoffBase = TimeSpan.FromSeconds(1);
|
||||
private static readonly TimeSpan DefaultBackoffCap = TimeSpan.FromSeconds(60);
|
||||
private static readonly TimeSpan DefaultProbeDeadline = TimeSpan.FromSeconds(5);
|
||||
@@ -202,9 +211,16 @@ public sealed class CentralChannelProvider : IDisposable
|
||||
new HeartbeatDto
|
||||
{
|
||||
SiteId = _siteId,
|
||||
NodeHostname = "failback-probe",
|
||||
NodeHostname = SyntheticProbeHostname,
|
||||
IsActive = false,
|
||||
Timestamp = Timestamp.FromDateTimeOffset(DateTimeOffset.UtcNow),
|
||||
// Explicit contract, not a hostname convention: central skips ALL
|
||||
// liveness/health bookkeeping for a synthetic heartbeat. Without it the
|
||||
// probe would stamp the site's last-seen time on the central health
|
||||
// aggregator every backoff tick, so a site whose real heartbeats had
|
||||
// stopped could still look alive purely because its transport layer was
|
||||
// probing for failback.
|
||||
Synthetic = true,
|
||||
},
|
||||
deadline: DateTime.UtcNow.Add(_probeDeadline)).ConfigureAwait(false);
|
||||
|
||||
|
||||
@@ -695,6 +695,7 @@ public static class CentralControlDtoMapper
|
||||
NodeHostname = msg.NodeHostname,
|
||||
IsActive = msg.IsActive,
|
||||
Timestamp = Timestamp.FromDateTimeOffset(msg.Timestamp),
|
||||
Synthetic = msg.Synthetic,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -709,7 +710,8 @@ public static class CentralControlDtoMapper
|
||||
SiteId: dto.SiteId,
|
||||
NodeHostname: dto.NodeHostname,
|
||||
IsActive: dto.IsActive,
|
||||
Timestamp: dto.Timestamp.ToDateTimeOffset());
|
||||
Timestamp: dto.Timestamp.ToDateTimeOffset(),
|
||||
Synthetic: dto.Synthetic);
|
||||
}
|
||||
|
||||
/// <summary>Projects a <see cref="ConnectionHealth"/> onto its wire enum.</summary>
|
||||
|
||||
@@ -33,13 +33,23 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc;
|
||||
/// host's Akka bootstrap, not by the container.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>The two audit-ingest RPCs are the exception: they Ask the ingest singleton DIRECTLY.</b>
|
||||
/// <c>CentralCommunicationActor</c> used to relay them — an extra hop that re-<c>Ask</c>ed the
|
||||
/// same <c>audit-log-ingest</c> proxy with the same 30 s
|
||||
/// <see cref="SiteStreamGrpcServer.AuditIngestAskTimeout"/>, so the inner Ask expired at the same
|
||||
/// instant as the outer one and could only add latency plus a
|
||||
/// <see cref="Akka.Actor.Status.Failure"/> repackaging. The proxy now arrives through
|
||||
/// <see cref="SetAuditIngestActor"/>, exactly as <see cref="SiteStreamGrpcServer"/> has always
|
||||
/// taken it, and the relay handlers were deleted.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Fault semantics deliberately differ from <see cref="SiteStreamGrpcServer"/>'s ingest
|
||||
/// RPCs.</b> That server answers a failed audit ingest with an EMPTY <c>IngestAck</c>; this one
|
||||
/// fails the call with a non-OK status. Both leave the site's rows <c>Pending</c> for the next
|
||||
/// drain, so the outcome is the same — but this service replaced the ClusterClient path, whose
|
||||
/// documented behaviour was to propagate the fault (<c>CentralCommunicationActor</c>'s
|
||||
/// <c>HandleIngestAuditEvents</c> pipes a <c>Status.Failure</c> back), and preserving that keeps
|
||||
/// a lost batch visible as a failure rather than as a successful call that acked nothing.
|
||||
/// documented behaviour was to propagate the fault, and preserving that keeps a lost batch
|
||||
/// visible as a failure rather than as a successful call that acked nothing. A proxy that has
|
||||
/// not been handed over yet is NOT a fault and still answers with an empty ack.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Status mapping, and why it is not uniform.</b> A site transport may safely re-send a call
|
||||
@@ -66,6 +76,14 @@ public sealed class CentralControlGrpcService : CentralControlService.CentralCon
|
||||
// Kestrel's thread pool.
|
||||
private volatile IActorRef? _central;
|
||||
|
||||
// Central-side audit-ingest singleton proxy (audit-log-ingest). Handed over by the host
|
||||
// AFTER the singleton starts, so it arrives strictly later than _central. The two ingest
|
||||
// RPCs Ask THIS proxy directly rather than relaying through CentralCommunicationActor —
|
||||
// see the class remarks. Null until registration completes; an ingest arriving before then
|
||||
// gets an empty ack (leave the site's rows Pending, retry next drain), which is exactly
|
||||
// what the removed relay replied in the same window.
|
||||
private volatile IActorRef? _auditIngest;
|
||||
|
||||
/// <summary>
|
||||
/// Creates the service. <b>This must remain the only public constructor</b> — see
|
||||
/// <see cref="SetReady"/> for how the actor arrives, and the Host's
|
||||
@@ -105,9 +123,32 @@ public sealed class CentralControlGrpcService : CentralControlService.CentralCon
|
||||
_central = centralCommunicationActor;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Hands the central <c>AuditLogIngestActor</c> singleton proxy to the service so the two
|
||||
/// ingest RPCs can Ask it DIRECTLY. Mirrors
|
||||
/// <see cref="SiteStreamGrpcServer.SetAuditIngestActor"/> — the site-stream server has always
|
||||
/// dispatched straight to the singleton, and this service now matches it.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Removing the <c>CentralCommunicationActor</c> relay removes an actor hop, a second Ask,
|
||||
/// and a second timeout from every audit batch: the relay's own Ask was armed with the same
|
||||
/// 30 s <see cref="SiteStreamGrpcServer.AuditIngestAskTimeout"/> as this service's, so the two
|
||||
/// expired together and the inner one could only ever add latency and a
|
||||
/// <see cref="Akka.Actor.Status.Failure"/> repackaging on the way out.
|
||||
/// </remarks>
|
||||
/// <param name="auditIngestProxy">The audit-log-ingest singleton proxy.</param>
|
||||
public void SetAuditIngestActor(IActorRef auditIngestProxy)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(auditIngestProxy);
|
||||
_auditIngest = auditIngestProxy;
|
||||
}
|
||||
|
||||
/// <summary>Exposed for wiring assertions in tests.</summary>
|
||||
internal bool IsReady => _central is not null;
|
||||
|
||||
/// <summary>Exposed for wiring assertions in tests.</summary>
|
||||
internal bool IsAuditIngestBound => _auditIngest is not null;
|
||||
|
||||
/// <inheritdoc />
|
||||
public override async Task<NotificationSubmitAckDto> SubmitNotification(
|
||||
NotificationSubmitDto request, ServerCallContext context)
|
||||
@@ -147,9 +188,16 @@ public sealed class CentralControlGrpcService : CentralControlService.CentralCon
|
||||
return new IngestAck();
|
||||
}
|
||||
|
||||
var central = RequireReady(context);
|
||||
// Node-readiness gate first (the actor system must be up), then the ingest proxy.
|
||||
RequireReady(context);
|
||||
var ingest = RequireAuditIngest(request.Events.Count, nameof(IngestAuditEvents));
|
||||
if (ingest is null)
|
||||
{
|
||||
return new IngestAck();
|
||||
}
|
||||
|
||||
var reply = await AskAsync<IngestAuditEventsReply>(
|
||||
central,
|
||||
ingest,
|
||||
CentralControlDtoMapper.FromDto(request),
|
||||
SiteStreamGrpcServer.AuditIngestAskTimeout,
|
||||
context).ConfigureAwait(false);
|
||||
@@ -166,9 +214,15 @@ public sealed class CentralControlGrpcService : CentralControlService.CentralCon
|
||||
return new IngestAck();
|
||||
}
|
||||
|
||||
var central = RequireReady(context);
|
||||
RequireReady(context);
|
||||
var ingest = RequireAuditIngest(request.Packets.Count, nameof(IngestCachedTelemetry));
|
||||
if (ingest is null)
|
||||
{
|
||||
return new IngestAck();
|
||||
}
|
||||
|
||||
var reply = await AskAsync<IngestCachedTelemetryReply>(
|
||||
central,
|
||||
ingest,
|
||||
CentralControlDtoMapper.FromDto(request),
|
||||
SiteStreamGrpcServer.AuditIngestAskTimeout,
|
||||
context).ConfigureAwait(false);
|
||||
@@ -176,6 +230,30 @@ public sealed class CentralControlGrpcService : CentralControlService.CentralCon
|
||||
return CentralControlDtoMapper.ToIngestAck(reply.AcceptedEventIds);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Returns the audit-ingest singleton proxy, or <c>null</c> when the host has not handed it
|
||||
/// over yet (it starts moments after <see cref="SetReady"/>). A null answer means the caller
|
||||
/// must reply with an EMPTY <see cref="IngestAck"/> — nothing accepted, rows stay
|
||||
/// <c>Pending</c> on the site and drain on the next tick. That is deliberately NOT
|
||||
/// <see cref="StatusCode.Unavailable"/>: readiness here is intentionally narrow (see
|
||||
/// <see cref="SetReady"/>), and this is the exact reply the removed
|
||||
/// <c>CentralCommunicationActor</c> relay produced in the same window.
|
||||
/// </summary>
|
||||
private IActorRef? RequireAuditIngest(int itemCount, string rpc)
|
||||
{
|
||||
var ingest = _auditIngest;
|
||||
if (ingest is not null)
|
||||
{
|
||||
return ingest;
|
||||
}
|
||||
|
||||
_logger.LogWarning(
|
||||
"{Rpc} received {Count} item(s) before SetAuditIngestActor was called; returning an "
|
||||
+ "empty ack so the site keeps its rows Pending and retries.",
|
||||
rpc, itemCount);
|
||||
return null;
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public override async Task<ReconcileSiteResponseDto> ReconcileSite(
|
||||
ReconcileSiteRequestDto request, ServerCallContext context)
|
||||
|
||||
@@ -22,9 +22,10 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc;
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// <b>Per-call deadlines match today's Ask timeouts exactly</b> — see <see cref="ResolveDeadline"/>.
|
||||
/// Behaviour is otherwise unchanged from the Akka path: an unknown/unconfigured site is warned and
|
||||
/// dropped (the caller's Ask times out), and a transport fault surfaces to the caller as a
|
||||
/// <see cref="Status.Failure"/>, which the S&F/audit layers already treat as transient.
|
||||
/// An unknown/unconfigured site now fails fast: it surfaces to the caller as a
|
||||
/// <see cref="Status.Failure"/> immediately (WP1.6) instead of being warned-and-dropped to let the
|
||||
/// caller's Ask burn its full timeout. A transport fault takes the same <see cref="Status.Failure"/>
|
||||
/// path, which the S&F/audit layers already treat as transient.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Cross-node retry is the channel provider's job</b> and happens only on
|
||||
@@ -79,13 +80,21 @@ public sealed class GrpcSiteTransport : ISiteCommandTransport
|
||||
replyTo.Tell(reply, ActorRefs.NoSender);
|
||||
}
|
||||
}
|
||||
catch (SiteChannelUnavailableException)
|
||||
catch (SiteChannelUnavailableException ex)
|
||||
{
|
||||
// Parity with the removed Akka "no ClusterClient for site" path: warn and drop, so the caller's
|
||||
// Ask times out. Central never buffers.
|
||||
// WP1.6: a known-dead send (no configured channel for this site) used to warn-and-drop,
|
||||
// leaving the caller's Ask to burn its full timeout for a failure we already know about.
|
||||
// Fail fast instead — same Status.Failure completion path a transport fault takes below —
|
||||
// so S&F/audit treat it as transient immediately rather than tens of seconds later.
|
||||
// Central still never buffers for an unreachable site; only the completion timing changed.
|
||||
_logger.LogWarning(
|
||||
"No gRPC channel for site {SiteId}; dropping {Message} (caller's Ask will time out)",
|
||||
"No gRPC channel for site {SiteId}; failing {Message} immediately instead of dropping it",
|
||||
envelope.SiteId, envelope.Message.GetType().Name);
|
||||
|
||||
if (!fireAndForget && !replyTo.IsNobody())
|
||||
{
|
||||
replyTo.Tell(new Status.Failure(ex), ActorRefs.NoSender);
|
||||
}
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
|
||||
@@ -1620,7 +1620,10 @@ public static class SiteCommandDtoMapper
|
||||
return new DebugSnapshotRequestDto
|
||||
{
|
||||
InstanceUniqueName = request.InstanceUniqueName,
|
||||
CorrelationId = request.CorrelationId
|
||||
CorrelationId = request.CorrelationId,
|
||||
// Additive field 3 — proto3 false default means an older site that does
|
||||
// not know the field simply returns the full snapshot (correct, just fatter).
|
||||
AlarmsOnly = request.AlarmsOnly
|
||||
};
|
||||
}
|
||||
|
||||
@@ -1631,7 +1634,7 @@ public static class SiteCommandDtoMapper
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(dto);
|
||||
|
||||
return new DebugSnapshotRequest(dto.InstanceUniqueName, dto.CorrelationId);
|
||||
return new DebugSnapshotRequest(dto.InstanceUniqueName, dto.CorrelationId, dto.AlarmsOnly);
|
||||
}
|
||||
|
||||
/// <summary>Projects a <see cref="SubscribeDebugViewRequest"/> onto the wire.</summary>
|
||||
|
||||
@@ -8,9 +8,11 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc;
|
||||
|
||||
/// <summary>
|
||||
/// Raised when a site has no usable gRPC channel — an unknown site, or a site with neither
|
||||
/// <c>GrpcNodeAAddress</c> nor <c>GrpcNodeBAddress</c> configured. The gRPC transport treats this
|
||||
/// the way the removed Akka path treated "no ClusterClient for site": warn and drop, so the caller's Ask
|
||||
/// times out (central never buffers).
|
||||
/// <c>GrpcNodeAAddress</c> nor <c>GrpcNodeBAddress</c> configured. <see cref="GrpcSiteTransport"/>
|
||||
/// treats this as a known-dead send: it fails the caller's <c>Ask</c> fast with an
|
||||
/// <c>Akka.Actor.Status.Failure</c> wrapping this exception (WP1.6) instead of warning and dropping
|
||||
/// the message, so the caller does not burn its full Ask timeout on a failure already known at send
|
||||
/// time (central still never buffers).
|
||||
/// </summary>
|
||||
public sealed class SiteChannelUnavailableException(string siteId)
|
||||
: Exception($"No gRPC channel is configured for site '{siteId}'.")
|
||||
|
||||
@@ -167,13 +167,19 @@ public class SiteStreamGrpcClient : IAsyncDisposable, IDisposable
|
||||
/// <summary>
|
||||
/// Opens a server-streaming subscription for a specific instance.
|
||||
/// This is a long-running async method; the caller launches it as a background task.
|
||||
/// The <paramref name="onEvent"/> callback delivers domain events, and
|
||||
/// <paramref name="onError"/> lets the caller handle reconnection.
|
||||
/// The <paramref name="onEvent"/> callback delivers domain events,
|
||||
/// <paramref name="onError"/> lets the caller handle reconnection after a fault, and
|
||||
/// <paramref name="onCompleted"/> reports a graceful end of stream.
|
||||
/// </summary>
|
||||
/// <param name="correlationId">Unique identifier for this subscription.</param>
|
||||
/// <param name="instanceUniqueName">Unique name of the instance to subscribe to.</param>
|
||||
/// <param name="onEvent">Callback invoked for each domain event received from the stream.</param>
|
||||
/// <param name="onError">Callback invoked when the subscription encounters an error.</param>
|
||||
/// <param name="onCompleted">
|
||||
/// Callback invoked when the server ended the RPC with OK — the site's max stream
|
||||
/// lifetime elapsing or a graceful site shutdown. Mutually exclusive with
|
||||
/// <paramref name="onError"/>; see <see cref="ConsumeStreamAsync"/>.
|
||||
/// </param>
|
||||
/// <param name="ct">Cancellation token to stop the subscription.</param>
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
public virtual async Task SubscribeAsync(
|
||||
@@ -181,6 +187,7 @@ public class SiteStreamGrpcClient : IAsyncDisposable, IDisposable
|
||||
string instanceUniqueName,
|
||||
Action<object> onEvent,
|
||||
Action<Exception> onError,
|
||||
Action onCompleted,
|
||||
CancellationToken ct)
|
||||
{
|
||||
if (_client is null)
|
||||
@@ -195,30 +202,18 @@ public class SiteStreamGrpcClient : IAsyncDisposable, IDisposable
|
||||
InstanceUniqueName = instanceUniqueName
|
||||
};
|
||||
|
||||
try
|
||||
{
|
||||
using var call = _client.SubscribeInstance(request, cancellationToken: cts.Token);
|
||||
|
||||
await foreach (var evt in call.ResponseStream.ReadAllAsync(cts.Token))
|
||||
await ConsumeStreamAsync(
|
||||
correlationId,
|
||||
cts,
|
||||
() => _client.SubscribeInstance(request, cancellationToken: cts.Token),
|
||||
evt =>
|
||||
{
|
||||
var domainEvent = ConvertToDomainEvent(evt);
|
||||
if (domainEvent != null)
|
||||
onEvent(domainEvent);
|
||||
}
|
||||
}
|
||||
catch (RpcException ex) when (ex.StatusCode == StatusCode.Cancelled)
|
||||
{
|
||||
// Normal cancellation — not an error
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
onError(ex);
|
||||
}
|
||||
finally
|
||||
{
|
||||
// Remove only our own entry -- a racing reconnect may already own the slot.
|
||||
RemoveSubscription(correlationId, cts);
|
||||
}
|
||||
},
|
||||
onError,
|
||||
onCompleted);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
@@ -244,13 +239,28 @@ public class SiteStreamGrpcClient : IAsyncDisposable, IDisposable
|
||||
/// <param name="correlationId">Unique identifier for this subscription.</param>
|
||||
/// <param name="onAlarmEvent">Callback invoked for each alarm delta received from the site-wide stream.</param>
|
||||
/// <param name="onError">Callback invoked when the subscription encounters an error.</param>
|
||||
/// <param name="onCompleted">
|
||||
/// Callback invoked when the server ended the RPC with OK — the site's max stream
|
||||
/// lifetime elapsing or a graceful site shutdown. Mutually exclusive with
|
||||
/// <paramref name="onError"/>; see <see cref="ConsumeStreamAsync"/>.
|
||||
/// </param>
|
||||
/// <param name="ct">Cancellation token to stop the subscription.</param>
|
||||
/// <param name="onConnected">
|
||||
/// Optional callback invoked once when the site has ACCEPTED the subscription — response
|
||||
/// headers received (the site writes them as soon as its relay actor is subscribed, so no
|
||||
/// event can be missed after this point), or the first event received if the peer defers
|
||||
/// its headers. A header timeout is NOT treated as connected. The per-site aggregator uses
|
||||
/// this to run exactly one re-seed per successful (re)connect instead of one per reconnect
|
||||
/// attempt. Never invoked more than once per call, and never after <paramref name="onError"/>.
|
||||
/// </param>
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
public virtual async Task SubscribeSiteAsync(
|
||||
string correlationId,
|
||||
Action<AlarmStateChanged> onAlarmEvent,
|
||||
Action<Exception> onError,
|
||||
CancellationToken ct)
|
||||
Action onCompleted,
|
||||
CancellationToken ct,
|
||||
Action? onConnected = null)
|
||||
{
|
||||
if (_client is null)
|
||||
throw new InvalidOperationException("Cannot subscribe on a test-only client.");
|
||||
@@ -263,20 +273,109 @@ public class SiteStreamGrpcClient : IAsyncDisposable, IDisposable
|
||||
CorrelationId = correlationId
|
||||
};
|
||||
|
||||
try
|
||||
{
|
||||
using var call = _client.SubscribeSite(request, cancellationToken: cts.Token);
|
||||
|
||||
await foreach (var evt in call.ResponseStream.ReadAllAsync(cts.Token))
|
||||
await ConsumeStreamAsync(
|
||||
correlationId,
|
||||
cts,
|
||||
() => _client.SubscribeSite(request, cancellationToken: cts.Token),
|
||||
evt =>
|
||||
{
|
||||
// Site-wide stream is alarm-only by contract; defensively ignore anything else.
|
||||
if (ConvertToAlarmEvent(evt) is { } alarm)
|
||||
onAlarmEvent(alarm);
|
||||
}
|
||||
}
|
||||
catch (RpcException ex) when (ex.StatusCode == StatusCode.Cancelled)
|
||||
},
|
||||
onError,
|
||||
onCompleted,
|
||||
onConnected);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Drains a server-streaming call to its end, classifying the outcome into exactly one
|
||||
/// of three terminations: a fault (<paramref name="onError"/>), a graceful server-side
|
||||
/// end of stream (<paramref name="onCompleted"/>), or our own cancellation (neither).
|
||||
/// <para>
|
||||
/// The graceful case is the one that matters operationally: the site server caps every
|
||||
/// stream at <c>CommunicationOptions.GrpcMaxStreamLifetime</c> (4 h) and, when that
|
||||
/// elapses, ends the RPC with status OK. The client-side <c>await foreach</c> then simply
|
||||
/// finishes, so before <c>onCompleted</c> existed the consuming actor observed nothing at
|
||||
/// all and the stream stayed silently dead until the process restarted.
|
||||
/// </para>
|
||||
/// Internal so the completion/fault classification can be unit-tested with a fake
|
||||
/// <see cref="IAsyncStreamReader{T}"/> and no live channel.
|
||||
/// </summary>
|
||||
/// <param name="correlationId">Unique identifier for this subscription.</param>
|
||||
/// <param name="cts">The subscription's linked token source (owned by the caller).</param>
|
||||
/// <param name="openCall">
|
||||
/// Opens the server-streaming call. Invoked inside the guarded region so a throw while
|
||||
/// opening is reported through <paramref name="onError"/> like any other stream fault.
|
||||
/// </param>
|
||||
/// <param name="onEvent">Invoked per wire event.</param>
|
||||
/// <param name="onError">Invoked once if the stream faulted.</param>
|
||||
/// <param name="onCompleted">Invoked once if the server ended the stream with OK.</param>
|
||||
/// <param name="onConnected">
|
||||
/// Optional; invoked AT MOST ONCE when the site has demonstrably accepted the
|
||||
/// subscription — either the server's response headers arrived (bounded by
|
||||
/// <see cref="ConnectedHeaderTimeout"/>) or, for a peer that defers headers until its
|
||||
/// first message, the first event was received. A header timeout alone is never
|
||||
/// reported as connected: that is also exactly what an unreachable site looks like.
|
||||
/// </param>
|
||||
/// <returns>A task that completes when the stream has ended and its outcome been reported.</returns>
|
||||
internal async Task ConsumeStreamAsync(
|
||||
string correlationId,
|
||||
CancellationTokenSource cts,
|
||||
Func<AsyncServerStreamingCall<SiteStreamEvent>> openCall,
|
||||
Action<SiteStreamEvent> onEvent,
|
||||
Action<Exception> onError,
|
||||
Action onCompleted,
|
||||
Action? onConnected = null)
|
||||
{
|
||||
var completedGracefully = false;
|
||||
var connectedReported = false;
|
||||
|
||||
// Fires onConnected AT MOST ONCE, from whichever proof of a live peer arrives
|
||||
// first: the response headers, or (for a peer that defers headers until its first
|
||||
// message) the first event. A header TIMEOUT is deliberately NOT such a proof —
|
||||
// see AwaitHeadersAsync.
|
||||
void ReportConnected()
|
||||
{
|
||||
// Normal cancellation — not an error
|
||||
if (connectedReported || onConnected is null) return;
|
||||
connectedReported = true;
|
||||
onConnected();
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
using (var call = openCall())
|
||||
{
|
||||
if (onConnected is not null && await AwaitHeadersAsync(call, cts.Token).ConfigureAwait(false))
|
||||
{
|
||||
ReportConnected();
|
||||
}
|
||||
|
||||
await foreach (var evt in call.ResponseStream.ReadAllAsync(cts.Token))
|
||||
{
|
||||
// Fallback connected signal: an event can only come from a peer that
|
||||
// accepted the subscription, so it proves what the headers would have.
|
||||
// Raised BEFORE the event is delivered so the consumer sees
|
||||
// connected-then-event ordering.
|
||||
ReportConnected();
|
||||
onEvent(evt);
|
||||
}
|
||||
}
|
||||
|
||||
completedGracefully = true;
|
||||
}
|
||||
catch (RpcException ex) when (ex.StatusCode == StatusCode.Cancelled && cts.IsCancellationRequested)
|
||||
{
|
||||
// OUR OWN cancellation (Unsubscribe / reconnect / channel teardown we asked
|
||||
// for) — not an error. The IsCancellationRequested guard matters: a Cancelled
|
||||
// status can also originate at the PEER or from a channel disposed underneath
|
||||
// us, and swallowing THAT fired none of onError/onCompleted/onConnected, so
|
||||
// the consuming actor kept a dead stream marked live forever. Foreign
|
||||
// Cancelled now falls through to the onError path below.
|
||||
}
|
||||
catch (OperationCanceledException) when (cts.IsCancellationRequested)
|
||||
{
|
||||
// Our own Unsubscribe/reconnect cancelled the read — not an error either.
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
@@ -287,6 +386,56 @@ public class SiteStreamGrpcClient : IAsyncDisposable, IDisposable
|
||||
// Remove only our own entry -- a racing reconnect may already own the slot.
|
||||
RemoveSubscription(correlationId, cts);
|
||||
}
|
||||
|
||||
// Raised outside the try so a throwing callback is not misreported as a stream
|
||||
// fault. Suppressed when our own token was cancelled as the stream ended: that is
|
||||
// a teardown, and the caller has already moved on to a newer stream.
|
||||
if (completedGracefully && !cts.IsCancellationRequested)
|
||||
onCompleted();
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// How long to wait for response headers before giving up on them as the connected
|
||||
/// signal and falling back to the first received event. A peer that only flushes headers
|
||||
/// with its first message would otherwise hold the connected signal — and with it the
|
||||
/// aggregator's re-seed — for as long as the site happens to be quiet.
|
||||
/// </summary>
|
||||
internal static TimeSpan ConnectedHeaderTimeout { get; set; } = TimeSpan.FromSeconds(10);
|
||||
|
||||
/// <summary>
|
||||
/// Awaits the call's response headers, bounded by <see cref="ConnectedHeaderTimeout"/>.
|
||||
/// Returns <see langword="true"/> when the headers arrived (the site accepted the
|
||||
/// subscription) and <see langword="false"/> on timeout. A fault propagates (the caller
|
||||
/// reports it through <c>onError</c> like any other stream fault); on timeout the
|
||||
/// abandoned headers task is observed so a later fault on it can never surface as an
|
||||
/// unobserved task exception.
|
||||
/// <para>
|
||||
/// A timeout must NOT be reported as connected: an unreachable/wedged site produces
|
||||
/// exactly that shape, and calling <c>onConnected</c> for it made the aggregator clear
|
||||
/// <c>_streamDown</c>, consume its pending re-seed and fan a full snapshot out at a site
|
||||
/// that never answered. The caller instead treats the FIRST RECEIVED EVENT as the
|
||||
/// fallback connected signal — real proof of a live peer, and the quiet-site case the
|
||||
/// timeout was added for is covered by the reconcile backstop.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
private static async Task<bool> AwaitHeadersAsync(
|
||||
AsyncServerStreamingCall<SiteStreamEvent> call, CancellationToken ct)
|
||||
{
|
||||
var headers = call.ResponseHeadersAsync;
|
||||
try
|
||||
{
|
||||
await headers.WaitAsync(ConnectedHeaderTimeout, ct).ConfigureAwait(false);
|
||||
return true;
|
||||
}
|
||||
catch (TimeoutException)
|
||||
{
|
||||
_ = headers.ContinueWith(
|
||||
t => _ = t.Exception,
|
||||
CancellationToken.None,
|
||||
TaskContinuationOptions.OnlyOnFaulted | TaskContinuationOptions.ExecuteSynchronously,
|
||||
TaskScheduler.Default);
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
|
||||
@@ -27,6 +27,8 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase
|
||||
private readonly ConcurrentDictionary<string, StreamEntry> _activeStreams = new();
|
||||
private readonly int _maxConcurrentStreams;
|
||||
private readonly TimeSpan _maxStreamLifetime;
|
||||
private readonly int _instanceChannelCapacity;
|
||||
private readonly int _siteAlarmChannelCapacity;
|
||||
private volatile bool _ready;
|
||||
// Flipped by CancelAllStreams() when the host enters
|
||||
// CoordinatedShutdown so SubscribeInstance refuses new streams with
|
||||
@@ -72,10 +74,17 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase
|
||||
ISiteStreamSubscriber streamSubscriber,
|
||||
ILogger<SiteStreamGrpcServer> logger,
|
||||
int maxConcurrentStreams = 100)
|
||||
: this(streamSubscriber, logger, maxConcurrentStreams, TimeSpan.FromHours(4))
|
||||
: this(streamSubscriber, logger, maxConcurrentStreams, TimeSpan.FromHours(4),
|
||||
DefaultInstanceChannelCapacity, DefaultSiteAlarmChannelCapacity)
|
||||
{
|
||||
}
|
||||
|
||||
/// <summary>Fallback Debug View send-channel capacity when no options are bound.</summary>
|
||||
internal const int DefaultInstanceChannelCapacity = 1000;
|
||||
|
||||
/// <summary>Fallback site-wide alarm send-channel capacity when no options are bound.</summary>
|
||||
internal const int DefaultSiteAlarmChannelCapacity = 20_000;
|
||||
|
||||
/// <summary>
|
||||
/// DI constructor — binds <see cref="CommunicationOptions.GrpcMaxConcurrentStreams"/>
|
||||
/// and <see cref="CommunicationOptions.GrpcMaxStreamLifetime"/> so the documented
|
||||
@@ -91,7 +100,9 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase
|
||||
IOptions<CommunicationOptions> options)
|
||||
: this(streamSubscriber, logger,
|
||||
options.Value.GrpcMaxConcurrentStreams,
|
||||
options.Value.GrpcMaxStreamLifetime)
|
||||
options.Value.GrpcMaxStreamLifetime,
|
||||
options.Value.GrpcInstanceStreamChannelCapacity,
|
||||
options.Value.GrpcSiteAlarmStreamChannelCapacity)
|
||||
{
|
||||
}
|
||||
|
||||
@@ -99,12 +110,16 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase
|
||||
ISiteStreamSubscriber streamSubscriber,
|
||||
ILogger<SiteStreamGrpcServer> logger,
|
||||
int maxConcurrentStreams,
|
||||
TimeSpan maxStreamLifetime)
|
||||
TimeSpan maxStreamLifetime,
|
||||
int instanceChannelCapacity,
|
||||
int siteAlarmChannelCapacity)
|
||||
{
|
||||
_streamSubscriber = streamSubscriber;
|
||||
_logger = logger;
|
||||
_maxConcurrentStreams = maxConcurrentStreams;
|
||||
_maxStreamLifetime = maxStreamLifetime;
|
||||
_instanceChannelCapacity = Math.Max(1, instanceChannelCapacity);
|
||||
_siteAlarmChannelCapacity = Math.Max(1, siteAlarmChannelCapacity);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
@@ -209,6 +224,21 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase
|
||||
/// <summary>Effective per-stream session lifetime. Exposed for tests.</summary>
|
||||
internal TimeSpan MaxStreamLifetime => _maxStreamLifetime;
|
||||
|
||||
/// <summary>Effective Debug View send-channel capacity. Exposed for tests.</summary>
|
||||
internal int InstanceChannelCapacity => _instanceChannelCapacity;
|
||||
|
||||
/// <summary>Effective site-wide alarm send-channel capacity. Exposed for tests.</summary>
|
||||
internal int SiteAlarmChannelCapacity => _siteAlarmChannelCapacity;
|
||||
|
||||
/// <summary>
|
||||
/// Total events evicted from stream send channels on this node since start (both stream
|
||||
/// kinds). Exported as <c>scadabridge.site.stream.events_dropped</c>; exposed here so a
|
||||
/// test — and an operator via diagnostics — can read the raw count.
|
||||
/// </summary>
|
||||
public long DroppedStreamEventCount => Interlocked.Read(ref _droppedStreamEvents);
|
||||
|
||||
private long _droppedStreamEvents;
|
||||
|
||||
/// <inheritdoc />
|
||||
public override Task SubscribeInstance(
|
||||
InstanceStreamRequest request,
|
||||
@@ -219,7 +249,9 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase
|
||||
responseStream,
|
||||
context,
|
||||
relay => _streamSubscriber.Subscribe(request.InstanceUniqueName, relay),
|
||||
request.InstanceUniqueName);
|
||||
request.InstanceUniqueName,
|
||||
_instanceChannelCapacity,
|
||||
streamKind: "instance");
|
||||
|
||||
/// <inheritdoc />
|
||||
public override Task SubscribeSite(
|
||||
@@ -234,7 +266,12 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase
|
||||
// already drops IsConfiguredPlaceholder rows and maps only the
|
||||
// enriched AlarmStateUpdate, so it is reused unchanged.
|
||||
_streamSubscriber.SubscribeSiteAlarms,
|
||||
"site-wide alarms");
|
||||
"site-wide alarms",
|
||||
// Its OWN, much larger channel (WP2.3): sharing the Debug View's 1000-slot
|
||||
// DropOldest meant an alarm burst during a WAN stall silently evicted operator-
|
||||
// visible transitions to make room for diagnostics traffic.
|
||||
_siteAlarmChannelCapacity,
|
||||
streamKind: "site-alarms");
|
||||
|
||||
/// <summary>
|
||||
/// Shared streaming pipeline behind <see cref="SubscribeInstance"/> and
|
||||
@@ -250,12 +287,16 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase
|
||||
/// <param name="context">The server call context (carries the client cancellation token).</param>
|
||||
/// <param name="subscribe">Subscribes the relay actor to the hub, returning a subscription id.</param>
|
||||
/// <param name="description">Human-readable subscription description for logging.</param>
|
||||
/// <param name="channelCapacity">Send-channel capacity for this stream kind.</param>
|
||||
/// <param name="streamKind">Telemetry tag for this stream kind (<c>instance</c>/<c>site-alarms</c>).</param>
|
||||
private async Task RunSubscriptionStreamAsync(
|
||||
string correlationId,
|
||||
IServerStreamWriter<SiteStreamEvent> responseStream,
|
||||
ServerCallContext context,
|
||||
Func<IActorRef, string> subscribe,
|
||||
string description)
|
||||
string description,
|
||||
int channelCapacity,
|
||||
string streamKind)
|
||||
{
|
||||
if (!_ready)
|
||||
throw new RpcException(new GrpcStatus(StatusCode.Unavailable, "Server not ready"));
|
||||
@@ -302,16 +343,20 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase
|
||||
|
||||
long dropped = 0;
|
||||
var channel = Channel.CreateBounded<SiteStreamEvent>(
|
||||
new BoundedChannelOptions(1000) { FullMode = BoundedChannelFullMode.DropOldest },
|
||||
new BoundedChannelOptions(channelCapacity) { FullMode = BoundedChannelFullMode.DropOldest },
|
||||
_ =>
|
||||
{
|
||||
// Lossy-under-backpressure is the debug-view spec; make the real
|
||||
// loss visible: first eviction + every 500th thereafter.
|
||||
// loss visible: first eviction + every 500th thereafter, plus a
|
||||
// per-kind counter on the node (WP2.3).
|
||||
var n = Interlocked.Increment(ref dropped);
|
||||
Interlocked.Increment(ref _droppedStreamEvents);
|
||||
ScadaBridgeTelemetry.RecordSiteStreamEventDropped(streamKind);
|
||||
if (n == 1 || n % 500 == 0)
|
||||
_logger.LogWarning(
|
||||
"Debug stream {CorrelationId} backpressure: {Dropped} oldest event(s) evicted so far",
|
||||
correlationId, n);
|
||||
"Stream {CorrelationId} ({StreamKind}) backpressure: {Dropped} oldest event(s) evicted so far " +
|
||||
"of a {Capacity}-slot channel",
|
||||
correlationId, streamKind, n, channelCapacity);
|
||||
});
|
||||
|
||||
var actorSeq = Interlocked.Increment(ref _actorCounter);
|
||||
@@ -348,6 +393,25 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase
|
||||
"Stream {CorrelationId} started for {Description} (subscription {SubscriptionId})",
|
||||
correlationId, description, subscriptionId);
|
||||
|
||||
// Flush response headers NOW, while the relay is attached and before the first event
|
||||
// (WP2.3). This is the client's "the site accepted my subscription" signal — the
|
||||
// per-site aggregator hangs its once-per-successful-reconnect re-seed off it. Without
|
||||
// an explicit flush, ASP.NET Core defers headers to the first written message, so a
|
||||
// quiet site would never report connected. Best-effort: a client that vanished between
|
||||
// Subscribe and here fails the write, and the read loop below handles the teardown.
|
||||
try
|
||||
{
|
||||
await context.WriteResponseHeadersAsync(Metadata.Empty);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
// Deliberately catch-all: an unsent header must never leak the relay actor and
|
||||
// the _activeStreams entry, which the finally below owns. Any real transport
|
||||
// failure resurfaces on the read/write loop immediately after.
|
||||
_logger.LogDebug(ex,
|
||||
"Could not flush response headers for stream {CorrelationId}; continuing.", correlationId);
|
||||
}
|
||||
|
||||
// Telemetry follow-on: the connection is now fully established (Subscribe
|
||||
// succeeded, so no leak via the catch above). Count it up here and balance
|
||||
// it in the finally below so the scadabridge.site.connection.up gauge is
|
||||
@@ -519,17 +583,46 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase
|
||||
? DateTime.SpecifyKind(request.SinceUtc.ToDateTime(), DateTimeKind.Utc)
|
||||
: DateTime.MinValue;
|
||||
|
||||
// Composite-keyset cursor (WP2.3), mirroring PullSiteCalls: proto3
|
||||
// defaults an unset string to "", which the queue treats as "no cursor"
|
||||
// (legacy inclusive >= behaviour), so an older central is unaffected.
|
||||
var afterId = string.IsNullOrEmpty(request.AfterId) ? null : request.AfterId;
|
||||
|
||||
// AT-LEAST-ONCE (WP2.3). The incoming cursor is central's receipt for
|
||||
// everything at or before it — that is the ONLY proof the site accepts.
|
||||
// Rows are flipped to Reconciled here, at the START of the NEXT pull,
|
||||
// instead of right after the previous response was projected: a fault
|
||||
// between the response leaving the site and central committing it used
|
||||
// to lose the rows outright, because they were already Reconciled and
|
||||
// ReadPendingSinceAsync would never serve them again. The flip runs
|
||||
// before the read so the rows it retires do not consume this batch's
|
||||
// budget. Best-effort — a failure here only costs a re-ship, which
|
||||
// central dedups on EventId.
|
||||
if (since > DateTime.MinValue)
|
||||
{
|
||||
try
|
||||
{
|
||||
await queue.MarkReconciledUpToAsync(since, afterId, context.CancellationToken);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogWarning(ex,
|
||||
"MarkReconciledUpToAsync failed for cursor since={Since} afterId={AfterId}; rows stay pending for the next pull.",
|
||||
since, afterId);
|
||||
}
|
||||
}
|
||||
|
||||
IReadOnlyList<AuditEvent> events;
|
||||
try
|
||||
{
|
||||
events = await queue.ReadPendingSinceAsync(
|
||||
since, request.BatchSize, context.CancellationToken);
|
||||
since, request.BatchSize, afterId, context.CancellationToken);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogError(ex,
|
||||
"ReadPendingSinceAsync failed for since={Since} batch={Batch}; returning empty response.",
|
||||
since, request.BatchSize);
|
||||
"ReadPendingSinceAsync failed for since={Since} batch={Batch} afterId={AfterId}; returning empty response.",
|
||||
since, request.BatchSize, afterId);
|
||||
return new PullAuditEventsResponse();
|
||||
}
|
||||
|
||||
@@ -537,7 +630,8 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase
|
||||
{
|
||||
// batch_size saturated → tell central to issue a follow-up pull
|
||||
// with an advanced cursor. The site doesn't compute the cursor —
|
||||
// central walks it forward from the last returned OccurredAtUtc.
|
||||
// central walks it forward from the last returned OccurredAtUtc
|
||||
// (plus, once it sets after_id, that row's EventId).
|
||||
MoreAvailable = events.Count >= request.BatchSize,
|
||||
};
|
||||
foreach (var evt in events)
|
||||
@@ -545,31 +639,6 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase
|
||||
response.Events.Add(AuditEventDtoMapper.ToDto(evt));
|
||||
}
|
||||
|
||||
// Flip to Reconciled AFTER projecting the response so a fault below the
|
||||
// try/catch (mid-response, mid-flip) leaves the rows in Pending/Forwarded
|
||||
// and central pulls them again next cycle. The flip itself is
|
||||
// best-effort — its failure is a warning, not a fault, because central
|
||||
// will dedup on EventId on the next pull.
|
||||
var ids = new List<Guid>(events.Count);
|
||||
foreach (var evt in events)
|
||||
{
|
||||
ids.Add(evt.EventId);
|
||||
}
|
||||
|
||||
if (ids.Count > 0)
|
||||
{
|
||||
try
|
||||
{
|
||||
await queue.MarkReconciledAsync(ids, context.CancellationToken);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogWarning(ex,
|
||||
"MarkReconciledAsync failed after PullAuditEvents response of {Count} rows; rows stay Pending for retry.",
|
||||
ids.Count);
|
||||
}
|
||||
}
|
||||
|
||||
return response;
|
||||
}
|
||||
|
||||
|
||||
@@ -62,4 +62,29 @@ public interface ISiteAlarmLiveCache
|
||||
/// <param name="siteId">The numeric site id.</param>
|
||||
/// <returns><c>true</c> while a <b>living</b> aggregator has seeded and published; reverts to false if the aggregator terminates (R2 N6) — a frozen snapshot is never reported as live.</returns>
|
||||
bool IsLive(int siteId);
|
||||
|
||||
/// <summary>
|
||||
/// The Enabled instances on the site that failed to answer the aggregator's most recent
|
||||
/// snapshot fan-out (seed or reconcile) — i.e. that reported <c>InstanceNotFound</c>, timed
|
||||
/// out, or faulted. Deterministically ordered by instance name, and empty when the site has
|
||||
/// no aggregator or has not yet completed a fan-out.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// Published alongside the alarm snapshot on every fan-out, because the aggregator already
|
||||
/// knows it: the seed/reconcile is the SAME per-instance debug-snapshot fan-out the Alarm
|
||||
/// Summary poll used to run a second time purely to compute this list. Exposing it here lets
|
||||
/// the UI-side poll be skipped entirely while <see cref="IsLive"/> is true, which is the
|
||||
/// whole point — one fan-out per site per reconcile window, not two.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// It is refreshed at the aggregator's fan-out cadence (seed, then each reconcile), NOT on
|
||||
/// every stream delta: an instance going silent is precisely what the stream cannot tell you.
|
||||
/// Callers must therefore treat it as reconcile-fresh, not delta-fresh — which matches the
|
||||
/// poll's own 15 s freshness, so nothing on screen changes.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
/// <param name="siteId">The numeric site id.</param>
|
||||
/// <returns>The not-reporting instance unique names (possibly empty), never <c>null</c>.</returns>
|
||||
IReadOnlyList<string> GetNotReportingInstances(int siteId);
|
||||
}
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user