docs: arch-review remediation — component docs sweep, execution log, residuals register
Final consistency sweep per plan §6: verified component docs against shipped WP1-WP3 + adversarial-review-fix state, corrected drift found in SiteRuntime (recursion-exempt run cap, stale ScriptExecutionActor/AlarmExecutionActor references), TemplateEngine (BundleImporter watermark path), DeploymentManager (phase-2 PendingDeployment staging), CentralUI (shared KPI cache, dedup'd alarm poll, render coalescing), StoreAndForward (rate-limited drop logging), and ConfigurationDatabase (documented DbContext-pooling non-adoption). Updated the docs/components/ developer-reference set (SiteRuntime, SiteEventLogging, InboundAPI) to drop the deleted per-run actor classes. Amended one known-issue for the superseding MaxBatchSize:64 read-page pin. Added CLAUDE.md bullets for stream graceful-completion reconnect, the required site audit DB path, honest CLI HTTP timeouts, bulk DeploySiteAsync, and LocalDb 0.2.1. New execution log records the phase→commit map, gate results, adversarial-review tally, the three test-flake root causes, and the nine-item residuals register.
This commit is contained in:
@@ -90,14 +90,14 @@ spec for each is `docs/requirements/Component-<Name>.md`, and `README.md` carrie
|
||||
- **`notification_lists` and `smtp_configurations` are created but deliberately NOT registered.** They are permanently empty on a site (no writer since 2026-07-10, the migrator skips them, the active-node purge keeps them empty), and registering them would open a standing replication channel whose only historical payload was plaintext SMTP passwords. Pinned by a security-named test, and verified live: those two tables have **no CDC triggers** on either rig node.
|
||||
- **Operational constraints (read before upgrading a site pair):** stop and start both nodes TOGETHER — rolling one at a time is no longer supported, since the legacy `SfBufferSnapshot` compatibility handler went with the replicator. And a node offline longer than `LocalDb:Replication:TombstoneRetention` (default 7 days) can resurrect deleted rows on rejoin. See `docs/deployment/topology-guide.md`.
|
||||
- **Batching is by BYTE BUDGET as of LocalDb 0.2.0** — `LocalDb:Replication:MaxBatchBytes` (default **2 MB**, sized under the 4 MB gRPC cap) bounds a delta/snapshot message by summed serialized size via a per-message split in `SyncSession.PumpLoopAsync`, with `MaxBatchSize` demoted to a secondary row cap; a single row over budget is sent alone rather than stalling the stream. **The rig's old `MaxBatchSize = 16` pin (a hand-computed byte-budget proxy: ~70 KB worst-case `config_json` x the 500 default is ~35 MB) is retired, but `MaxBatchSize` is NOT fully redundant with `MaxBatchBytes`** — it also bounds the separate DB READ page in `OplogStore.ReadBatchAboveAsync`/`SnapshotStreamer`, which materializes the whole page into memory *before* the byte-budget split runs, so an unset (500-default) `MaxBatchSize` still lets a reconnect drain transiently allocate ~35 MB per read even though every wire message stays under `MaxBatchBytes` (arch-review adversarial finding F2). Both site-a nodes on `docker/` therefore pin an explicit `"MaxBatchSize": 64` to bound that transient allocation, while `MaxBatchBytes` stays unset (its 2 MB default) to bound the wire message; site-b/site-c stay unreplicated so the key doesn't apply there.
|
||||
- **CDC registration is conditional, and both directions self-heal at boot** (arch-review WP1.3 + WP3.3, `Host/SiteLocalDbSetup.cs`). Capture triggers are installed only when this node has replication configured — `PeerAddress` **or** `ApiKey`, an OR because only the dialling half sets `PeerAddress` while the passive half carries the key alone. An unreplicated node calls **`DeregisterReplicated`** on all ten tables at boot, dropping triggers an earlier build left behind and pruning their oplog/row-version rows (idempotent; logs once at Information when something was actually cleaned). A replicating node registers with **`baselineExistingRows: true`**, which seeds `__localdb_row_version` for pre-existing rows at the LWW floor (HLC `0`, this node's id) and flags a snapshot resync — so turning replication ON for a site that has been running without it now converges on the rows already in the file instead of only on writes made after the restart. Deregistration must be symmetric (the handshake compares registered-table digests fail-closed), so replication is a both-nodes-together change in either direction. LocalDb 0.2.0 also makes backlog depth O(1) and drops the unused `__localdb_oplog_hlc` index; the on-disk bookkeeping schema goes to v2, upgraded in place on open, with no wire change (0.1.x peers still sync).
|
||||
- **CDC registration is conditional, and both directions self-heal at boot** (arch-review WP1.3 + WP3.3, `Host/SiteLocalDbSetup.cs`). Capture triggers are installed only when this node has replication configured — `PeerAddress` **or** `ApiKey`, an OR because only the dialling half sets `PeerAddress` while the passive half carries the key alone. An unreplicated node calls **`DeregisterReplicated`** on all ten tables at boot, dropping triggers an earlier build left behind and pruning their oplog/row-version rows (idempotent; logs once at Information when something was actually cleaned). A replicating node registers with **`baselineExistingRows: true`**, which seeds `__localdb_row_version` for pre-existing rows at the LWW floor (HLC `0`, this node's id) and flags a snapshot resync — so turning replication ON for a site that has been running without it now converges on the rows already in the file instead of only on writes made after the restart. Deregistration must be symmetric (the handshake compares registered-table digests fail-closed), so replication is a both-nodes-together change in either direction. LocalDb 0.2.0 also makes backlog depth O(1) and drops the unused `__localdb_oplog_hlc` index; the on-disk bookkeeping schema goes to v2, upgraded in place on open, with no wire change (0.1.x peers still sync). **LocalDb 0.2.1** (two adversarial-review findings against 0.2.0, wire-compatible, no schema change) fixes: (1) `DeregisterReplicated` was deleting the HLC clock's only durable crash anchor along with the table rows — an ungraceful exit (crash, SIGKILL, power loss) on a fully-deregistered file left nothing to recover from, so the reopened clock could restart from wall-clock time and re-issue stamps below HLCs peers already hold (silently discarded by LWW); the current clock is now flushed inside the deregistration transaction, before the prune. (2) The sync inbox was an **unbounded** channel, so inbound memory was a function of what the peer sends — a mutual full snapshot (both sides of a pair baseline unconditionally when replication is enabled late) buffered whole, in memory, on both nodes at once; it is now **bounded at 64 messages** with wait-mode backpressure (unrelated to, and not to be confused with, the rig's separate `MaxBatchSize: 64` DB-read-page pin above), which required moving the snapshot to run alongside the receive loop (not inside the pre-loop handshake) so the inbox is drained for the whole session and can't deadlock itself.
|
||||
- All timestamps are UTC throughout the system.
|
||||
- Inter-cluster communication uses **three** transports, not two — **all cross-cluster command/control and data now rides gRPC** after the ClusterClient→gRPC migration's Phase 4 (`docs/plans/2026-07-22-clusterclient-to-grpc-plan.md`) deleted Akka `ClusterClient`/`ClusterClientReceptionist`: (1) **gRPC command/control** — site→central over the central-hosted `CentralControlService` (`GrpcCentralTransport`, sticky central-a→central-b channel pair; deployments/notifications/health/heartbeat/audit-ingest/reconcile), and central→site over the site-hosted `SiteCommandService` (`GrpcSiteTransport`, per-site NodeA→NodeB channel pair; the 28 lifecycle/OPC-UA/query/parked/route/failover commands); (2) **gRPC** server-streaming for real-time data (attribute values, alarm states, `SiteStreamService`); and (3) **plain token-gated HTTP** for the deployment config itself — notify-and-fetch, the site pulls the config from `DeploymentConfigEndpoints` (`ManagementService/DeploymentConfigEndpoints.cs`) with an `X-Deployment-Token` header, `AllowAnonymous` with the per-deployment token as the entire security boundary. The gRPC boundary is per-site PSK-authenticated (`ControlPlaneAuthInterceptor`, unchanged). There is **no receptionist registration** — discovery is by dialling configured endpoints; central builds one `SitePairChannelProvider` per site (addresses from `Site.GrpcNodeAAddress`/`GrpcNodeBAddress`, refreshed from the DB every 60s and on admin changes), sites dial `ScadaBridge:Communication:CentralGrpcEndpoints` (both central nodes, h2c on `CentralGrpcPort` 8083, **NOT** via Traefik). **Discovery is asymmetric by design:** central discovers site gRPC addresses from the *database* (refreshable at runtime), sites discover central from *appsettings* (`CentralGrpcEndpoints`, static — restart required; `StartupValidator` requires a Site node to list at least one). `Akka.Cluster.Tools` stays for ClusterSingleton; only the ClusterClient part is gone. **Central never buffers for an unreachable site** — the send fails with the caller's Ask/deadline timing out; a `ConnectionStateChanged` mechanism built for this was deleted as dead code.
|
||||
- **All clusters share ONE ActorSystem name**, `"scadabridge"` — hardcoded in `AkkaHostedService` at the `ActorSystem.Create` call. Central and each site are separate clusters *only* by seed-node partitioning. The constraint originated with ClusterClient (Akka.Remote address matching meant it could not reach a differently-named system); whether it is still load-bearing after the gRPC migration has **not** been re-verified, so treat the name as fixed until someone checks.
|
||||
- **`ActiveNodeEvaluator.SelfIsOldestUp` is THE single definition of "active node"** (`Communication/ClusterState/ActiveNodeEvaluator.cs`) — the **oldest Up member** in a role scope, and explicitly **never `cluster.State.Leader`**: leadership (lowest address) is an Akka-internal concept that diverges from singleton placement permanently once the original first node restarts and rejoins, and both sides claim it during a partition. The equivalence *oldest-Up == where `ClusterSingletonManager` places singletons* **is** the design. `ClusterActivityEvaluator.SelfIsOldest`, the S&F delivery gate, `/health/active` and the heartbeat `IsActive` stamp all delegate here.
|
||||
- Site nodes carry **two Akka roles**: the base `Site` plus a site-specific `site-{SiteId}` (`AkkaHostedService.BuildRoles`). Singletons scope to the **site-specific** role.
|
||||
- **The gRPC boundary is authenticated (PSK) as of 2026-07-22; Akka remoting still is not, and nothing is encrypted.** Akka remoting sets no `enable-ssl`, no secure cookie, no `trusted-selection-paths` — so intra-cluster Akka remoting remains open to anyone who can reach the remoting port, and that boundary still assumes a trusted network. The gRPC listener stays **h2c**, but `SiteStreamService` is no longer open: `ControlPlaneAuthInterceptor` (`Host/ControlPlaneAuthInterceptor.cs`) gates `/sitestream.SiteStreamService/` — including the `PullAuditEvents`/`PullSiteCalls` RPCs that return audit rows — against a **per-site preshared key**, fail-closed, constant-time compared, alongside the separate `LocalDbSyncAuthInterceptor` on `/localdb_sync.v1.LocalDbSync/` with its own separate key. **Site side:** `ScadaBridge:Communication:GrpcPsk`, in production `${secret:SB-GRPC-PSK-<siteId>}`, and **`StartupValidator` refuses to boot a site node without it** (an unset key would leave the node healthy-looking but serving nothing). **Central side:** `SitePskProvider` resolves `SB-GRPC-PSK-{siteId}` from the secrets store at channel-build time (sites are added at runtime, so no boot-time expansion is possible), with `ScadaBridge:Communication:SitePsks:{siteId}` as an override for hosts running without a master key — the docker rig uses the latter. One key per site, never fleet-wide. A bearer token over h2c is readable and replayable on-path; TLS is the follow-on hardening and needs no change to this design. Introduced by Phase 0 of the ClusterClient→gRPC migration (`docs/plans/2026-07-22-clusterclient-to-grpc-plan.md`).
|
||||
- gRPC streaming channel — **note the direction is inverted from the data flow**: data moves site→central, but each **site node hosts the gRPC server** (`SiteStreamGrpcServer`, Kestrel h2c, port 8083, mapped **only in the Site branch** of `Program.cs`) and **central is the client**, dialling in. Central creates per-site `SiteStreamGrpcClient` via `SiteStreamGrpcClientFactory`, keyed **`(siteId, endpoint)`** — the key was widened from site-only to fix an arch-review High where one session's NodeA→NodeB flip disposed a channel another session was still using. Proto evolution is **additive only** and field numbers are never reused (`AlarmStateUpdate` grew 7→23 fields for the native-alarm mirror). Generated C# is **vendored** under `Communication/SiteStreamGrpc/` with the `<Protobuf>` include commented out — regeneration is a manual toggle-build-copy-untoggle.
|
||||
- gRPC streaming channel — **note the direction is inverted from the data flow**: data moves site→central, but each **site node hosts the gRPC server** (`SiteStreamGrpcServer`, Kestrel h2c, port 8083, mapped **only in the Site branch** of `Program.cs`) and **central is the client**, dialling in. Central creates per-site `SiteStreamGrpcClient` via `SiteStreamGrpcClientFactory`, keyed **`(siteId, endpoint)`** — the key was widened from site-only to fix an arch-review High where one session's NodeA→NodeB flip disposed a channel another session was still using. Proto evolution is **additive only** and field numbers are never reused (`AlarmStateUpdate` grew 7→23 fields for the native-alarm mirror). Generated C# is **vendored** under `Communication/SiteStreamGrpc/` with the `<Protobuf>` include commented out — regeneration is a manual toggle-build-copy-untoggle. **The client reconnects on graceful (OK-status) stream completion, not just on fault (arch-review remediation, commit `34a3f4bb`)** — Kestrel's `MaxStreamLifetime`/`Grpc.AspNetCore.Server` max-connection-age periodically ends a healthy stream with a normal completion, which the client used to treat as terminal (no reconnect attempt), silently killing a site's live feed until the next process restart (observed up to ~4h on the rig); live-probed at a 2-minute forced lifetime, reconnect lands within one reconcile tick and `IsLive` reflects the gap in between.
|
||||
- Native alarms are a **read-only** mirror of OPC UA Alarms & Conditions and MxAccess Gateway alarms — **no ack-back, no central tables**; state lives in the site's `native_alarm_state`, survives failover, and is cleared on redeploy/undeploy (mirrors static overrides). Central's per-site live alarm cache (`ISiteAlarmLiveCache`) is **transient in-memory only** — there is deliberately no persisted central alarm store, so the 15s poll remains the NotReporting authority behind the live stream. See `Component-DataConnectionLayer.md` / `Component-CentralUI.md` for the model and the authoring surface.
|
||||
- **`AckTime` mirror enrichment + the `Alarms` script accessor (MES alarm-status API Phase 1, 2026-08-01).** `AlarmStateChanged` carries an additive `AckTime` (`DateTimeOffset?`), mirrored on the vendored `AlarmStateUpdate` proto as **field 24** and persisted inside `native_alarm_state`'s `metadata_json` — deliberately NOT a new column, because that table is `RegisterReplicated` and LocalDb builds its CDC triggers from the column list at registration time. Set only while a condition is active AND acknowledged (so it is null while unacked and cleared on re-raise); the DCL stamps the source's own ack instant for OPC UA (new SelectClause **index 18** = `AckedState/TransitionTime`) and its observation time of the ack transition for MxGateway, which supplies none. Site `Call` scripts read alarms via the new **`Alarms.CurrentAsync()`** accessor (`ScriptRuntimeContext` + `ScriptGlobals`, local Ask on `GetAlarmSnapshotRequest`, returns `Commons.Types.Scripts.ScriptAlarm`), mirrored on `ScriptCompileSurface` AND the Central UI `SandboxScriptHost` editor surface. The trust model needed no change — it is a deny-list over API roots, not an allow-list of context members. Plan: `docs/plans/2026-06-30-mes-alarm-status-api.md` (Phases 2–4 are deployed config, not repo).
|
||||
- OPC UA cert trust is **site-local and not persisted centrally** (follow-up): the verify-endpoint probe captures an untrusted server cert but **NEVER trusts it**, and DeploymentManager broadcasts `TrustServerCertCommand`/`RemoveServerCertCommand` to **BOTH** site nodes — `CertStoreActor` runs on every site node, not as a singleton, so PKI stores stay consistent across failover.
|
||||
@@ -122,6 +122,7 @@ spec for each is `docs/requirements/Component-<Name>.md`, and `README.md` carrie
|
||||
- Flattened configs include a revision hash for staleness detection.
|
||||
- Deployment identity: unique deployment ID + revision hash for idempotency.
|
||||
- Per-instance operation lock covers all mutating commands (deploy, disable, enable, delete).
|
||||
- **Bulk site deployment (`DeploymentService.DeploySiteAsync`, arch-review WP2.5)** redeploys every out-of-date instance at a site through one flatten session (a template chain shared by N instances is walked once) in three phases — serial Prepare, bounded-parallel Send (`SiteDeploymentMaxParallelism`, default 4), serial Finalize — still **not all-or-nothing**: any one instance's failure doesn't block the rest. The `PendingDeployment` staging row (5-minute TTL) is written per-instance immediately before that instance's round-trip in the Send phase, not upfront in Prepare, so a large batch's tail instances don't have their TTL clock already half-spent by the time they're reached.
|
||||
- Site-side apply is all-or-nothing per instance.
|
||||
- System-wide artifact version skew across sites is supported.
|
||||
- Last-write-wins for concurrent template editing (no optimistic concurrency on templates).
|
||||
@@ -153,7 +154,7 @@ spec for each is `docs/requirements/Component-<Name>.md`, and `README.md` carrie
|
||||
- One row per lifecycle event; cached calls produce 4+ rows per operation (`Submitted`, `Forwarded`, `Attempted`, `Delivered`/`Parked`/`Discarded`).
|
||||
- `ExecutionId` (`uniqueidentifier NULL`) is the universal per-run correlation value — every audit row emitted by one script execution / inbound request shares it; `CorrelationId` remains the per-operation lifecycle id (NULL for sync one-shots).
|
||||
- `ParentExecutionId` (`uniqueidentifier NULL`) is the cross-execution spawn pointer — every row of a spawned run carries the spawner's `ExecutionId`; bridges inbound API → routed-site-script, alarm-triggered on-trigger scripts, and nested `CallScript`/`CallShared` invocations; `IX_AuditLog_ParentExecution` backs the filter + the recursive execution-tree walk. **Tag-cascade (alarm leg) is populated, not just plumbed** — M5.4 T4 threaded the `parentExecutionId` parameter but every `AlarmActor.SpawnAlarmExecution` call site passed null, so alarm runs were silently always roots; the id now rides site-locally as `SetStaticAttributeCommand.SourceExecutionId` → `AttributeValueChanged.SourceExecutionId` → `SpawnAlarmExecution` (all additive/nullable, no wire/proto/schema change; `Expression` triggers capture the writer *with* the evaluated snapshot since the eval completes off-dispatcher). Sources are `ScriptRuntimeContext.SetAttribute` (the run's own `ExecutionId`) and `Route.To(...).SetAttributes(...)` (the inbound request's). **Still roots by design, not omission:** alarms fired by DCL data (external values have no spawning execution — including the device echo of a script write to a *data-sourced* attribute, so only **static** writes cascade), and `ScriptActor` value-change/conditional/expression/timer trigger runs (a timer tick has no spawner; a `WhileTrue`/interval run has no single identifiable write).
|
||||
- Site SQLite hot-path first, then gRPC telemetry to central; ingest is idempotent on `EventId`; periodic reconciliation pull as fallback when telemetry is lost.
|
||||
- Site SQLite hot-path first, then gRPC telemetry to central; ingest is idempotent on `EventId`; periodic reconciliation pull as fallback when telemetry is lost. **The site audit DB is a separate, unreplicated SQLite file — NOT one of LocalDb's ten consolidated tables** — at the **required** `AuditLog:SiteWriter:DatabasePath` (arch-review WP1.2, finding #2 High), `/app/data/auditlog.db` on the rig; `StartupValidator` refuses to boot a Site node without it. `DatabasePath` used to default to CWD-relative `auditlog.db`, which on the docker rig landed on the container's ephemeral overlay, not the mounted volume — the same data-loss pattern the pre-LocalDb legacy S&F/tracking databases hit (see the LocalDb bullet above), independently rediscovered here because this file sits outside LocalDb's management. The writer loop now also honors `FlushIntervalMs` (was validated but never read — one fsync'ing commit per event even at trickle rate) and sets `PRAGMA synchronous = NORMAL` (audit is best-effort by design, so the narrower power-loss window is an acceptable trade for far fewer fsyncs; WAL still guarantees no corruption).
|
||||
- Cached operations: site emits a single additively-extended `CachedCallTelemetry` packet carrying both audit events and operational state; central writes `AuditLog` + `SiteCalls` in one transaction.
|
||||
- Payload cap 8 KB by default / 64 KB on error rows; auth headers redacted by default; SQL parameter values captured by default; per-target redaction opt-in. Inbound API: full verbatim capture up to `InboundMaxBytes` (default 1 MiB); request headers stored in `Extra.requestHeaders` (post-redaction); per-method `SkipBodyCapture` flag suppresses bodies while still recording headers + metadata; `AuditInboundCeilingHits` counter surfaced on health snapshot. (M5.3 T7)
|
||||
- Audit-write failure NEVER aborts the user-facing action — audit is best-effort, the action's own success/failure path is authoritative.
|
||||
@@ -214,6 +215,7 @@ spec for each is `docs/requirements/Component-<Name>.md`, and `README.md` carrie
|
||||
## Tool Usage
|
||||
|
||||
- When a task requires setting up or controlling system state (sites, templates, instances, data connections, deployments, security, etc.) and the Central UI is not needed, prefer the ScadaBridge CLI over manual DB edits or UI navigation. See [`src/ZB.MOM.WW.ScadaBridge.CLI/README.md`](src/ZB.MOM.WW.ScadaBridge.CLI/README.md) for the full command reference.
|
||||
- **CLI HTTP timeouts are honest (arch-review remediation, finding #1 High).** `ManagementHttpClient` used to cap every call at `min(30s, caller timeout)`, silently truncating `deploy site`'s 5-minute `BulkDeployTimeout` and the 5-minute bundle export/preview/import calls — the CLI printed a fake `504 Request timed out` while the server kept working to completion. `HttpClient.Timeout` is now `Timeout.InfiniteTimeSpan`; the per-call `CancellationTokenSource` is the single overall deadline (connect phase included), with connect bounded separately via `SocketsHttpHandler.ConnectTimeout` (env override renamed `SCADABRIDGE_HTTP_CONNECT_TIMEOUT_SECONDS` to match its narrower new meaning).
|
||||
|
||||
### CLI Quick Reference (Docker / OrbStack)
|
||||
|
||||
|
||||
@@ -241,7 +241,7 @@ The inbound body-capture cap for audit is configured separately under `AuditLog:
|
||||
|
||||
- [Commons (#16)](./Commons.md) — owns `ApiMethod`, `ParameterDefinition`, `ScriptParameters`, `ScriptParameterException`, the `RouteToCall*` / `RouteToGetAttributes*` / `RouteToSetAttributes*` message records, `IInboundApiRepository`, and `IInstanceLocator`. Also owns `ICentralAuditWriter` (via `ZB.MOM.WW.Audit`), `AuditChannel`, `AuditKind`, `AuditStatus`, and `ScadaBridgeAuditEventFactory`.
|
||||
- [Configuration Database (#17)](./ConfigurationDatabase.md) — provides the `IInboundApiRepository` implementation (`GetMethodByNameAsync`, `GetAllApiMethodsAsync`, CRUD). Method definitions persist in the central MS SQL configuration database.
|
||||
- [Central–Site Communication (#5)](./Communication.md) — `CommunicationServiceInstanceRouter` delegates every `Route.To()` operation to `CommunicationService`. The routed call travels from the central `CentralCommunicationActor` to the target site via `ClusterClient`, reaches the target `InstanceActor`, and a `ScriptExecutionActor` executes the named script. The return value flows back synchronously.
|
||||
- [Central–Site Communication (#5)](./Communication.md) — `CommunicationServiceInstanceRouter` delegates every `Route.To()` operation to `CommunicationService`. The routed call travels from the central `CentralCommunicationActor` to the target site over gRPC (`CentralControlService`/`SiteCommandService` — `ClusterClient` was removed in the ClusterClient→gRPC migration), reaches the target `InstanceActor`, and the named script runs via `ScriptRunLauncher` (no per-run child actor since WP3.1). The return value flows back synchronously.
|
||||
- [Audit Log (#23)](./AuditLog.md) — `AuditWriteMiddleware` resolves `ICentralAuditWriter` to emit the `ApiInbound` row via the central direct-write path. The inbound request is the parent execution for any site script it spawns: the middleware's `ExecutionId` becomes `RouteToCallRequest.ParentExecutionId` on every routed `Call`. Cross-link: `AuditWriteMiddleware.InboundExecutionIdItemKey` / `AuditWriteMiddleware.AuditActorItemKey` are the `HttpContext.Items` keys that tie the endpoint handler and middleware together.
|
||||
- [Security (#10)](./Security.md) — API key verification (`IApiKeyVerifier`, `AddZbApiKeyAuth`) is registered by the Host. The inbound API uses a dedicated key scheme independent of LDAP/AD session auth.
|
||||
- [Cluster Infrastructure (#13)](./ClusterInfrastructure.md) — `IActiveNodeGate` (interface in this project; implementation in the Host) gates the endpoint to the active central node. A standby returns `503` without running any script logic.
|
||||
|
||||
@@ -157,7 +157,7 @@ public class EventLogHandlerActor : ReceiveActor
|
||||
Callers resolve `ISiteEventLogger` from DI. Because the write is non-blocking and best-effort, site actors discard the returned `Task` with `_ =` rather than awaiting it on the hot path:
|
||||
|
||||
```csharp
|
||||
// ScriptExecutionActor — reporting a script failure
|
||||
// ScriptRunLauncher (via ScriptActor) — reporting a script failure
|
||||
_ = siteEventLogger?.LogEventAsync(
|
||||
"script", "Error", instanceName, $"ScriptActor:{scriptName}", errorMsg, ex.ToString());
|
||||
|
||||
@@ -196,7 +196,7 @@ The docker cluster appsettings (`ScadaBridge:SiteEventLog`) sets `RetentionDays:
|
||||
|
||||
- [Commons (#16)](./Commons.md) — defines the `EventLogQueryRequest` / `EventLogQueryResponse` / `EventLogEntry` message contracts in `ZB.MOM.WW.ScadaBridge.Commons.Messages.RemoteQuery`, shared across the site query path and the central dispatch path (`QueryEventLogsCommand`).
|
||||
- [Central–Site Communication (#5)](./Communication.md) — the `SiteCommunicationActor` dispatches inbound `EventLogQueryRequest` messages to `EventLogHandlerActor` and carries the `EventLogQueryResponse` back to central. The query timeout is 30 s.
|
||||
- [Site Runtime (#3)](./SiteRuntime.md) — `ScriptActor` and `ScriptExecutionActor` log `script`-type events: trigger expression failures, script execution errors, and timeouts. `ISiteEventLogger` is resolved from DI inside execution actors.
|
||||
- [Site Runtime (#3)](./SiteRuntime.md) — `ScriptActor` and `AlarmActor` log `script`-type events (via `ScriptRunLauncher`, no per-run child actor since WP3.1): trigger expression failures, script execution errors, and timeouts. `ISiteEventLogger` is resolved from DI for the run.
|
||||
- [Data Connection Layer (#4)](./DataConnectionLayer.md) — `DataConnectionActor` logs `connection`-type events: connection loss, reconnection, and endpoint failover. `DataConnectionManagerActor` may also log connection-category events.
|
||||
- [Store-and-Forward Engine (#6)](./StoreAndForward.md) — logs `store_and_forward`-type events on the site→central notification forward path (forward failures, long-buffered notifications). Routine enqueue and forward-success events are not logged; central's `Notifications` table is the authoritative record.
|
||||
- [Host (#15)](./Host.md) — `SiteServiceRegistration` calls `AddSiteEventLogging` and binds `SiteEventLogOptions`. `AkkaHostedService` wires `EventLogHandlerActor` as a cluster singleton scoped to `"site-{SiteId}"`. The `SiteEventLogActiveNodeCheck` delegate is an optional seam defined in `SiteEventLogging` for the Host to register when it wants to gate the purge to the active node only; the Host does not currently register it, so the purge defaults to always-active and runs on every node.
|
||||
|
||||
@@ -4,12 +4,12 @@ The Site Runtime component runs the site-side actor hierarchy that executes depl
|
||||
|
||||
## Overview
|
||||
|
||||
Site Runtime (#3) operates exclusively on site clusters. Its entry point is the `DeploymentManagerActor` cluster singleton, which re-creates the full actor hierarchy on every site startup or failover. Each deployed enabled instance gets an `InstanceActor` child; each `InstanceActor` spawns `ScriptActor` and `AlarmActor` coordinator children, plus a `NativeAlarmActor` peer for every configured native alarm source. Script invocations spawn short-lived `ScriptExecutionActor` children; alarm on-trigger invocations spawn short-lived `AlarmExecutionActor` children.
|
||||
Site Runtime (#3) operates exclusively on site clusters. Its entry point is the `DeploymentManagerActor` cluster singleton, which re-creates the full actor hierarchy on every site startup or failover. Each deployed enabled instance gets an `InstanceActor` child; each `InstanceActor` spawns `ScriptActor` and `AlarmActor` coordinator children, plus a `NativeAlarmActor` peer for every configured native alarm source. **As of WP3.1 (arch-review remediation, 2026-08-15) a script or alarm on-trigger invocation is no longer a per-run child actor** — `ScriptExecutionActor`/`AlarmExecutionActor` were deleted (they were already inert shells: no `Receive` handler, no `PostStop`, never a message target). `ScriptActor`/`AlarmActor` instead launch the run body directly onto the shared script-execution pool via `ScriptRunLauncher`, keyed by instance+script so `MaxConcurrentRunsPerScript` and the deadline-at-enqueue timeout still apply per script.
|
||||
|
||||
The component code lives in `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/`:
|
||||
|
||||
- `Actors/` — `DeploymentManagerActor`, `InstanceActor`, `ScriptActor`, `ScriptExecutionActor`, `AlarmActor`, `AlarmExecutionActor`, `NativeAlarmActor`.
|
||||
- `Scripts/` — `ScriptCompilationService`, `ScriptExecutionScheduler`, `SharedScriptLibrary`, `ScriptRuntimeContext`, `ScopeAccessors`, `TriggerExpressionGlobals`.
|
||||
- `Actors/` — `DeploymentManagerActor`, `InstanceActor`, `ScriptActor`, `AlarmActor`, `NativeAlarmActor`. (`ScriptExecutionActor`/`AlarmExecutionActor` were removed by WP3.1 — see Overview.)
|
||||
- `Scripts/` — `ScriptCompilationService`, `ScriptExecutionScheduler`, `ScriptRunLauncher`, `TriggerEvalGate`, `ScriptRunSummaryRecorder`, `SharedScriptLibrary`, `ScriptRuntimeContext`, `ScopeAccessors`, `TriggerExpressionGlobals`.
|
||||
- `Streaming/` — `SiteStreamManager` (the site-wide Akka broadcast stream).
|
||||
- `Persistence/` — `SiteStorageService` (raw SQLite via `Microsoft.Data.Sqlite`), `SiteStorageInitializer`.
|
||||
- `Repositories/` — `SiteExternalSystemRepository`. (The `SiteNotificationRepository` variant was removed 2026-07-10, arch-review 08 §1.3/#23, because notification config is central-only and never lives on a site.)
|
||||
@@ -36,14 +36,12 @@ The singleton reads all deployed configurations from SQLite in `PreStart`, compi
|
||||
| `ScriptActor` | long-lived coordinator | `OneForOneStrategy` | Stop execution child, keep self |
|
||||
| `AlarmActor` | long-lived coordinator | `OneForOneStrategy` | Stop execution child, keep self |
|
||||
| `NativeAlarmActor` | long-lived coordinator | — | Supervised by Instance Actor (Resume) |
|
||||
| `ScriptExecutionActor` | short-lived per invocation | — | Stops itself; parent logs failure |
|
||||
| `AlarmExecutionActor` | short-lived per invocation | — | Stops itself; parent logs failure |
|
||||
|
||||
Coordinator actors resume on exception because their in-memory state (trigger timers, last execution time, alarm level) must survive child crashes. Short-lived execution actors stop themselves on completion or exception — the coordinator remains available for the next trigger.
|
||||
Coordinator actors resume on exception because their in-memory state (trigger timers, last execution time, alarm level) must survive child crashes. **Since WP3.1 there is no per-run child actor** — `ScriptActor`/`AlarmActor` launch the run body directly via `ScriptRunLauncher` onto the shared pool and observe its completion `Task`; a run failure is caught and reported by the launcher/coordinator, not by an actor `PostStop`.
|
||||
|
||||
### Dedicated script-execution dispatcher
|
||||
|
||||
Script and alarm on-trigger bodies run on the `ScriptExecutionScheduler` (`SiteRuntime-009`): a custom `TaskScheduler` backed by a bounded set of dedicated threads (default 8, `ScriptExecutionThreadCount`). The script body is submitted to this scheduler via `Task.Factory.StartNew(..., scheduler)` inside `ScriptExecutionActor` and `AlarmExecutionActor`. Scripts that block on I/O (database connections, synchronous external system calls) block only the scheduler's threads, leaving the shared .NET thread pool and all Akka dispatchers unaffected.
|
||||
Script and alarm on-trigger bodies run on the `ScriptExecutionScheduler` (`SiteRuntime-009`): a custom `TaskScheduler` backed by a bounded, **instance-scaled, grow-only** set of dedicated threads (`clamp(max(ScriptExecutionThreadCount, ceil(enabledInstances/8)), 1, ScriptExecutionMaxThreadCount)`; floor default 8, ceiling default 32 — WP3.1). The script body is submitted to this scheduler via `Task.Factory.StartNew(..., scheduler)` from `ScriptRunLauncher`, called directly by `ScriptActor`/`AlarmActor` — no per-run actor is spawned. Scripts that block on I/O (database connections, synchronous external system calls) block only the scheduler's threads, leaving the shared .NET thread pool and all Akka dispatchers unaffected. **Expression-trigger evaluation runs on a separate path** — the shared .NET thread pool behind `TriggerEvalGate`, not this scheduler (WP3.1, arch-review finding #4) — so scripts blocked in synchronous I/O can no longer stall every Expression trigger on the node.
|
||||
|
||||
### Tell vs. Ask
|
||||
|
||||
@@ -52,7 +50,7 @@ Script and alarm on-trigger bodies run on the `ScriptExecutionScheduler` (`SiteR
|
||||
|
||||
### Attribute serialization through the Instance Actor
|
||||
|
||||
All in-memory state mutations (attribute values, qualities, alarm states) run inside `InstanceActor`'s mailbox. Multiple `ScriptExecutionActor` instances may run concurrently but all `SetAttribute` calls serialize through the `InstanceActor` mailbox, preventing race conditions. Concurrent script executions may interleave external side effects (HTTP calls, database writes, notifications); those are independent and intentionally not serialized.
|
||||
All in-memory state mutations (attribute values, qualities, alarm states) run inside `InstanceActor`'s mailbox. Multiple script/alarm runs may execute concurrently on the shared pool but all `SetAttribute` calls serialize through the `InstanceActor` mailbox, preventing race conditions. Concurrent script executions may interleave external side effects (HTTP calls, database writes, notifications); those are independent and intentionally not serialized.
|
||||
|
||||
## Architecture
|
||||
|
||||
@@ -61,16 +59,15 @@ All in-memory state mutations (attribute values, qualities, alarm states) run in
|
||||
```text
|
||||
DeploymentManagerActor (Akka.NET cluster singleton)
|
||||
└── InstanceActor "MachineA-001"
|
||||
├── ScriptActor "MonitorSpeed" (coordinator)
|
||||
│ └── ScriptExecutionActor (short-lived, per invocation)
|
||||
├── ScriptActor "MonitorSpeed" (coordinator — launches runs via ScriptRunLauncher, no per-run child)
|
||||
├── ScriptActor "CalculateOEE" (coordinator)
|
||||
│ └── ScriptExecutionActor (short-lived)
|
||||
├── AlarmActor "OverTemp" (coordinator, computed)
|
||||
│ └── AlarmExecutionActor (short-lived, on-trigger)
|
||||
├── AlarmActor "OverTemp" (coordinator, computed — on-trigger runs also via ScriptRunLauncher)
|
||||
├── AlarmActor "LowPressure" (coordinator, computed)
|
||||
└── NativeAlarmActor "OpcUaServer1" (read-only mirror, peer to AlarmActor)
|
||||
```
|
||||
|
||||
Prior to WP3.1 (2026-08-15) each invocation spawned a short-lived `ScriptExecutionActor`/`AlarmExecutionActor` child; those actor classes were deleted (see Overview) and every run now executes on the shared `ScriptExecutionScheduler` pool directly, tracked by `ScriptRunLauncher` rather than by an actor cell.
|
||||
|
||||
`NativeAlarmActor` is a sibling of `AlarmActor` — a peer under the same `InstanceActor` parent. It is not a child of `AlarmActor` and has no relationship to the script engine.
|
||||
|
||||
### Deployment flow
|
||||
@@ -234,17 +231,17 @@ Central sends commands to the site `DeploymentManagerActor` singleton over the C
|
||||
|
||||
### Script API surface
|
||||
|
||||
Scripts run inside `ScriptExecutionActor` with a `ScriptGlobals` object as the Roslyn host object. The `Instance` global is a `ScriptRuntimeContext`. Convenience top-level aliases (`ExternalSystem`, `Database`, `Notify`, `Scripts`, `Attributes`, `Children`, `Parent`) delegate to context methods. Key calls:
|
||||
Scripts run on the shared script-execution pool (via `ScriptRunLauncher`, launched directly by `ScriptActor` — no per-run child actor since WP3.1) with a `ScriptGlobals` object as the Roslyn host object. The `Instance` global is a `ScriptRuntimeContext`. Convenience top-level aliases (`ExternalSystem`, `Database`, `Notify`, `Scripts`, `Attributes`, `Children`, `Parent`) delegate to context methods. Key calls:
|
||||
|
||||
- `Instance.GetAttribute("name")` / `Instance.SetAttribute("name", value)` — Ask to `InstanceActor` for write, in-process for read.
|
||||
- `Instance.CallScript("scriptName", params)` — Ask from `ScriptExecutionActor` to sibling `ScriptActor`, which spawns a new `ScriptExecutionActor`.
|
||||
- `Instance.CallScript("scriptName", params)` — Ask from the running script (via `ScriptRuntimeContext.CallScript`) to sibling `ScriptActor`, which launches a new run via `ScriptRunLauncher`. Nested self-recursion is exempt from `MaxConcurrentRunsPerScript` (bounded instead by `MaxScriptCallDepth`), since the calling run is already holding one of the cap's slots while it awaits the callee.
|
||||
- `Scripts.CallShared("name", params)` — `SharedScriptLibrary.ExecuteAsync`, inline on the current scheduler thread.
|
||||
- `ExternalSystem.Call(...)` — synchronous HTTP call through `IExternalSystemClient`.
|
||||
- `ExternalSystem.CachedCall(...)` / `Database.CachedWrite(...)` — store-and-forwarded; returns a `TrackedOperationId`.
|
||||
- `Tracking.Status(id)` — reads the site-local `OperationTrackingStore` synchronously.
|
||||
- `Notify.To("list").Send(...)` — enqueues a notification in the Store-and-Forward Engine for delivery to central.
|
||||
|
||||
Alarm on-trigger scripts run in `AlarmExecutionActor` with a **restricted** context: they receive an `Alarm` global (`AlarmContext` carrying `Name`, `Level`, `Priority`, `Message`) and have access to the instance/shared-script surface (`Instance.*`, `Scripts.CallShared`, `Instance.CallScript`), but **not** the external-system, database, notification, or audit integration APIs. `AlarmExecutionActor` builds its `ScriptRuntimeContext` without a `serviceProvider`, so `ExternalSystem`, `Database`, `Notify`, and audit writes are unavailable to alarm on-trigger scripts — those APIs are only resolved inside `ScriptExecutionActor` (instance scripts).
|
||||
Alarm on-trigger scripts run via the same `ScriptRunLauncher` path (launched by `AlarmActor`) with a **restricted** context: they receive an `Alarm` global (`AlarmContext` carrying `Name`, `Level`, `Priority`, `Message`) and have access to the instance/shared-script surface (`Instance.*`, `Scripts.CallShared`, `Instance.CallScript`), but **not** the external-system, database, notification, or audit integration APIs. The alarm run's `ScriptRuntimeContext` is built without a `serviceProvider`, so `ExternalSystem`, `Database`, `Notify`, and audit writes are unavailable to alarm on-trigger scripts — those APIs are only resolved for instance-script runs.
|
||||
|
||||
### Debug view
|
||||
|
||||
@@ -261,7 +258,9 @@ All options live in the `ScadaBridge:SiteRuntime` section, bound to `SiteRuntime
|
||||
| `MaxScriptCallDepth` | `10` | Maximum `Instance.CallScript` / `Scripts.CallShared` recursion depth |
|
||||
| `ScriptExecutionTimeoutSeconds` | `30` | Per-script body execution timeout; exceeding it cancels and logs an error |
|
||||
| `StreamBufferSize` | `1000` | Per-subscriber drop-oldest buffer size for the Akka broadcast stream |
|
||||
| `ScriptExecutionThreadCount` | `8` | Dedicated threads in the `ScriptExecutionScheduler` (covers both scripts and alarm on-trigger bodies) |
|
||||
| `ScriptExecutionThreadCount` | `8` | Floor thread count in the `ScriptExecutionScheduler` (covers both scripts and alarm on-trigger bodies) — the pool grows above this with `enabledInstances`, see Dedicated script-execution dispatcher |
|
||||
| `ScriptExecutionMaxThreadCount` | `32` | Ceiling on the instance-scaled pool growth (WP3.1) |
|
||||
| `MaxConcurrentRunsPerScript` | `4` | Runs in flight (queued or executing) per script/alarm on-trigger script before the newest is shed (WP3.1) |
|
||||
| `MirroredAlarmCapPerSource` | `1000` | Maximum mirrored conditions per `NativeAlarmActor` source binding before oldest is dropped and logged |
|
||||
| `NativeAlarmRetryIntervalMs` | `5000` | Milliseconds before retrying a failed native alarm subscription |
|
||||
|
||||
@@ -271,11 +270,11 @@ The SQLite connection string is passed directly to `AddSiteRuntime(connectionStr
|
||||
|
||||
- [Data Connection Layer (#4)](./DataConnectionLayer.md) — supplies `TagValueUpdate` and `ConnectionQualityChanged` messages to `InstanceActor`; receives `SubscribeTagsRequest` and `WriteTagRequest`. Also supplies `NativeAlarmTransitionUpdate` and `NativeAlarmSourceUnavailable` to `NativeAlarmActor` via `SubscribeAlarmsRequest` (connections implementing `IAlarmSubscribableConnection`).
|
||||
- [Central–Site Communication (#5)](./Communication.md) — routes `DeployInstanceCommand`, `DisableInstanceCommand`, `EnableInstanceCommand`, `DeleteInstanceCommand`, `DeployArtifactsCommand`, debug view requests, and Inbound API `RouteToCallRequest` / `RouteToGetAttributesRequest` / `RouteToSetAttributesRequest` to the singleton; receives `DeploymentStatusResponse` and `ArtifactDeploymentResponse` back. The `SiteStreamManager` implements `ISiteStreamSubscriber` so the Communication Layer's `SiteStreamGrpcServer` can subscribe `StreamRelayActor` instances to the broadcast hub.
|
||||
- [Store-and-Forward Engine (#6)](./StoreAndForward.md) — `ScriptRuntimeContext` passes `StoreAndForwardService` (resolved from DI inside `ScriptExecutionActor`) for `ExternalSystem.CachedCall`, `Database.CachedWrite`, and `Notify.To().Send()`. Owns the site-local operation tracking table that `Tracking.Status(id)` reads.
|
||||
- [Store-and-Forward Engine (#6)](./StoreAndForward.md) — `ScriptRuntimeContext` passes `StoreAndForwardService` (resolved from DI for the run, via `ScriptRunLauncher`) for `ExternalSystem.CachedCall`, `Database.CachedWrite`, and `Notify.To().Send()`. Owns the site-local operation tracking table that `Tracking.Status(id)` reads.
|
||||
- [External System Gateway (#7)](./ExternalSystemGateway.md) — `IExternalSystemClient` called by `ScriptRuntimeContext.ExternalSystemHelper` for synchronous and cached external system calls.
|
||||
- [Site Event Logging (#12)](./SiteEventLogging.md) — `ISiteEventLogger` (resolved from DI inside execution actors) receives script error, alarm error, and script execution events.
|
||||
- [Site Event Logging (#12)](./SiteEventLogging.md) — `ISiteEventLogger` (resolved from DI for the run) receives script error, alarm error, and script execution events.
|
||||
- [Health Monitoring (#11)](./HealthMonitoring.md) — `ISiteHealthCollector` (injected into `DeploymentManagerActor`, `InstanceActor`, `ScriptActor`, `AlarmActor`) tracks instance counts (`SetInstanceCounts`), script errors (`IncrementScriptError`), and alarm errors (`IncrementAlarmError`); sets `SetActiveNode` in `DeploymentManagerActor.PreStart`/`PostStop` so the health report reflects which node holds the singleton.
|
||||
- [Audit Log (#23)](./AuditLog.md) — `IAuditWriter` (resolved from DI inside `ScriptExecutionActor`) receives one row per script-trust-boundary call; audit writes are best-effort and never abort the calling script.
|
||||
- [Audit Log (#23)](./AuditLog.md) — `IAuditWriter` (resolved from DI for the run) receives one row per script-trust-boundary call; audit writes are best-effort and never abort the calling script.
|
||||
- [Commons (#16)](./Commons.md) — owns all message contracts (`DeployInstanceCommand`, `AttributeValueChanged`, `AlarmStateChanged`, `ScriptCallRequest`, `NativeAlarmTransitionUpdate`, etc.), the `FlattenedConfiguration` / `ResolvedScript` / `ResolvedAlarm` / `ResolvedNativeAlarmSource` types, and the `AlarmKind` / `AlarmState` / `AlarmLevel` / `AlarmConditionState` / `AlarmTransitionKind` enums.
|
||||
- Local SQLite — `SiteStorageService` owns the site database. Peer SQLite stores (Store-and-Forward buffer, AuditLog, operation tracking, site event log) are owned by their respective components but share the same SQLite file path convention.
|
||||
- Design spec: [Component-SiteRuntime.md](../requirements/Component-SiteRuntime.md).
|
||||
|
||||
@@ -30,9 +30,20 @@ case). See the Phase 2 plan (D6) and `docs/plans/2026-07-19-localdb-phase2-live-
|
||||
**Closed by LocalDb 0.2.0** (arch-review WP3.3): batching is now bounded by
|
||||
`LocalDb:Replication:MaxBatchBytes` — summed serialized bytes, default **2 MB**, sized under the
|
||||
4 MB limit — with the row count demoted to a secondary cap, and a single row above the budget sent
|
||||
alone rather than stalling the stream. The rig's `MaxBatchSize = 16` pin is retired and both keys
|
||||
are left at their defaults; a deployment replicating wide rows no longer has to size a row count
|
||||
against its widest column by hand.
|
||||
alone rather than stalling the stream. The rig's old `MaxBatchSize = 16` pin (sized against the
|
||||
4 MB *wire* limit) is retired.
|
||||
|
||||
**Amended (2026-08-14) — `MaxBatchSize` is back, pinned to a different value for a different
|
||||
reason.** `MaxBatchBytes` bounds the wire message, but it doesn't bound the DB **read page**:
|
||||
`OplogStore.ReadBatchAboveAsync`/`SnapshotStreamer` materializes a whole `MaxBatchSize`-row page
|
||||
into memory *before* the byte-budget split runs, so an unset (500-default) `MaxBatchSize` still
|
||||
lets a reconnect drain transiently allocate ~35 MB per read even though every wire message stays
|
||||
under `MaxBatchBytes` (arch-review adversarial finding F2). Both site-a nodes on `docker/` now pin
|
||||
an explicit `"MaxBatchSize": 64` to bound that transient read-side allocation, while
|
||||
`MaxBatchBytes` is left at its 2 MB default to bound the wire message; site-b/site-c stay
|
||||
unreplicated so the key doesn't apply there. Not to be confused with LocalDb 0.2.1's unrelated
|
||||
sync-inbox bound (also 64, but message count on the *receive* side, hardcoded in the library, not
|
||||
a config key) — see CLAUDE.md's LocalDb bullet.
|
||||
|
||||
Note the failure mode differs from the one documented below: an oversized gRPC message is
|
||||
**rejected**, not silently dropped.
|
||||
|
||||
@@ -0,0 +1,135 @@
|
||||
# Arch-review remediation — execution log
|
||||
|
||||
**Ran:** 2026-08-14/15 · **Plan:** `docs/plans/2026-08-14-arch-review-remediation-plan.md` ·
|
||||
**Branch:** `arch-review-remediation` (off `main`) · **Status:** code-complete, docs-propagated,
|
||||
not yet pushed/merged.
|
||||
|
||||
Source: the 2026-08-14 eight-pass performance/architecture review (all 27 components +
|
||||
`ZB.MOM.WW.LocalDb`). This log is the compact record of what shipped, how it was gated, and
|
||||
what's left. Full package specs live in the plan; this is the after-action summary.
|
||||
|
||||
## Phases and package → commit map
|
||||
|
||||
Every package ran as an isolated-worktree subagent, merged `--no-ff` onto
|
||||
`arch-review-remediation`; commit hashes below are the substantive commit on the branch (merge
|
||||
commits omitted).
|
||||
|
||||
**Phase 0 — Preflight.** `0b201e41` docs(plans): the remediation plan itself.
|
||||
|
||||
**Phase 1 — Quick wins (7 packages, parallel).**
|
||||
| WP | Commit | What |
|
||||
|---|---|---|
|
||||
| 1.1 | `34a3f4bb` | Reconnect on graceful (OK-status) stream completion — closes the 4h silent stream death |
|
||||
| 1.2 | `2e4e41a8` | Site audit DB onto the mounted data volume; required path + soft flush |
|
||||
| 1.3 | `7ebdcd37` | CDC capture installed only when replication is configured |
|
||||
| 1.4 | `600659d5` | Sweep/KPI covering indexes + sliced notification terminal purge |
|
||||
| 1.5 | `125055d9` | O(1) attribute resolution, precomputed types, coalesced static writes, shared JSON options |
|
||||
| 1.6 | `c5e66ed4` | Fail known-dead sends immediately instead of burning Ask timeouts |
|
||||
| 1.7 | `2cfcd890` | Batched event-log commits + sliced retention purge |
|
||||
|
||||
**Phase 2 — Seam rework (6 packages, parallel after the WP2.1a design doc `1040dc0f`).**
|
||||
| WP | Commit | What |
|
||||
|---|---|---|
|
||||
| 2.1 | `d15c5f02` | DCL batch subscribe/read/write seam, bounded reconnect, sharded subscriptions |
|
||||
| 2.2 | `5db2a810` | Central set-based ingest, aligned partition purge, KPI query shapes, EF hygiene |
|
||||
| 2.3 | `2ce0ad7e` | Alarms-only stream seed, capped buffers, at-least-once audit pull |
|
||||
| 2.4 | `8c0b36b2` | Shared KPI cache, live-cache-backed alarm summary, coalesced Debug View renders |
|
||||
| 2.5 | `48b3c40a` | Flatten-session caching, bulk `DeploySiteAsync`, paged management queries |
|
||||
| 2.6 | `a2122831` | Cached hot-path lookups, bounded observer queue, alarm-priority stream path |
|
||||
|
||||
Plus `a5882753` closing Phase-2-gate residuals (direct ingest path, monotonic timeouts, synthetic
|
||||
probe, not-reporting set, cursor-exact audit pull) found while gating.
|
||||
|
||||
**Phase 3 — Structural (design-first).**
|
||||
| WP | Design | Commit | What |
|
||||
|---|---|---|---|
|
||||
| 3.1 | `312216ff` | `c4fc1f8e` | Script execution pool split — trigger evals off the blocking pool, bounded/deadline-aware execution |
|
||||
| 3.2 | `6cfb2dd8` | `c254d074` | `site_events` volume policy — sampled per-run events, interval summaries, replication pinned |
|
||||
| 3.3 | (scadaproj) | `cca7f178` + scadaproj `9377fa1` | LocalDb 0.2.0 — dereg cleanup, late-opt-in baselining, byte-budget replication |
|
||||
|
||||
**Phase 4 — Verification, adversarial review, docs.** Six parallel `code-reviewer` passes over
|
||||
the full diff, one per area (site runtime, DCL, comms, site persistence, central SQL, UI/deploy),
|
||||
each instructed to try to refute the fixes. Confirmed findings landed as targeted follow-ups:
|
||||
|
||||
| Area | Commit | What |
|
||||
|---|---|---|
|
||||
| Site runtime | `950c54c5` | Recursion-exempt run cap (nested `CallScript` no longer double-gated), atomic detach counter, summary edge cases, per-row event-log fallback |
|
||||
| DCL | `37f13e2e` | Discard in-flight subscribe results for tags unsubscribed mid-flight; release the orphaned handle |
|
||||
| Central SQL | `5d075f13` | No client-side audit truncation, insert-first upsert (Site Call Audit), QI-safe filtered-index scripts, honest operator-not-found replies |
|
||||
| Comms | `fd5e023d` | Consumer-based debug-stream orphan net, foreign-cancel triad, honest `onConnected`, served-row-exact retirement, full-rate reconcile |
|
||||
| UI/deploy | `e0e4b246` | Honest CLI HTTP timeouts, watermark-complete staleness (3 missed bump sites), phase-2 `PendingDeployment` staging, lock-safe cancellation |
|
||||
| Site persistence | `56c99c92` + `f689f495` | Required audit DB path on wonder; explicit `MaxBatchSize:64` LocalDb read-page cap; rate-limited observer drop logging; LocalDb 0.2.1 (HLC anchor flush on dereg, bounded 64-message sync inbox) |
|
||||
|
||||
## Gate results
|
||||
|
||||
- **Baseline:** 7587 tests green at Phase 0 entry; test count never reduced across any phase gate.
|
||||
- **Live probes (rig):**
|
||||
- Stream lifetime forced to 2 minutes — the alarm stream reconnected in **8.5s**, within one
|
||||
reconcile tick of the OK completion, `IsLive` correctly reflected the gap (WP1.1).
|
||||
- CDC conditional-registration: site-b booted clean on LocalDb 0.2.0 with **30 stale triggers
|
||||
dropped** at startup (WP1.3/WP3.3) — confirms the self-heal path fires, not just the steady
|
||||
state.
|
||||
- S&F due-sweep: `EXPLAIN QUERY PLAN` confirmed index-terminated (WP1.4).
|
||||
- Failover drill (`docker/failover-drill.sh`) unaffected by the actor/timeout changes.
|
||||
- **Adversarial review tally:** ~25 confirmed findings across the six areas, **4 High**, all
|
||||
fixed in the Phase 4 commits above. Zero findings deferred as won't-fix.
|
||||
- **Test-flake root causes (3, all test-side, not production bugs):**
|
||||
1. `cfa6acbf` — an assertion on `MarkForwarded` ran before the push it depended on was
|
||||
guaranteed to have landed; reordered behind the push.
|
||||
2. `c4caebe9` — two dispatcher audit-safety tests asserted an attempt count without
|
||||
synchronizing on the async write that produced it; the unsynchronized assertion was removed.
|
||||
3. `950c54c5` (embedded) — `ScriptDeadlineAtEnqueueTests`' "no started event" assertion went
|
||||
vacuous once WP3.2 flipped `PerRunScriptEvents` to off-by-default (fixed by opting the test
|
||||
back in); `ScriptRunLauncherParityTests` widened an `ExpectMsg` window that would have passed
|
||||
for any deadline from 1s to 300s, not just the intended one (sharpened to assert the reported
|
||||
timeout value AND a tight wall-clock range).
|
||||
- **Follow-up recommended, not done here:** a suite-wide sweep for the same
|
||||
`AwaitAssert(...)`-then-bare-`Assert` pattern — an `AwaitAssert` that only proves "eventually
|
||||
true," followed by a plain assertion that silently inherits its timing slack, is the shape
|
||||
behind all three; worth a grep-and-review pass rather than fixing on-demand as flakes surface.
|
||||
|
||||
## Residuals register
|
||||
|
||||
Deliberately not fixed in this program — each has a stated reason, not an oversight:
|
||||
|
||||
1. **DCL unsubscribe-during-reconnect count staleness.** The `37f13e2e` fix discards orphaned
|
||||
in-flight results but a per-connection counter can still drift under rapid
|
||||
subscribe/unsubscribe churn during a reconnect; needs a per-tag counted set. Low severity,
|
||||
cosmetic (a health-report number), deferred.
|
||||
2. **Per-table `needs_snapshot` in LocalDb.** Baselining one table currently re-streams every
|
||||
registered table in both directions. Narrowing it needs an on-disk schema change LocalDb 0.2.1
|
||||
deliberately avoided (wire/schema compatibility). Documented as a follow-up in the library's
|
||||
own README and `RegisterReplicated` remarks.
|
||||
3. **Event batching per proto message.** Individual `AttributeValueChanged`/`AlarmStateChanged`
|
||||
events still ride one gRPC message each; batching them is a new wire shape (proto + both
|
||||
client/server), deferred rather than folded into this program's additive-only changes.
|
||||
4. **Deployments page server-side paging + status counts.** Still client-materializes the full
|
||||
list; out of scope for this pass (WP2.5 touched the deploy pipeline, not this specific UI
|
||||
surface).
|
||||
5. **OtOpcUa still pins LocalDb 0.1.3.** A supported skew — 0.1.x peers sync with 0.2.x under the
|
||||
library's wire-compatibility guarantee — not a blocker for this program.
|
||||
6. **Fragile `SandboxTests` timing pin.** Pre-existing, unrelated to this remediation's changes;
|
||||
noted so it isn't mistaken for a regression if it flakes later.
|
||||
7. **Target-scale load test (deferred-work register #25).** This program's exit criterion is the
|
||||
live probes above, not #25 — #25 remains the follow-on validation that the moved ceilings hold
|
||||
under real load; schedule separately.
|
||||
8. **Playwright 14 pre-existing env failures.** Present on `main` too, rig-state related, not
|
||||
introduced by this branch.
|
||||
9. **`site_events` retention purge still oplog-visible.** WP3.2's sliced retention DELETE is a row
|
||||
change like any other and is captured by CDC on a replicated site (site-a) — correct per the
|
||||
"CDC does all three jobs" design (no separate resync path to gate), but means a purge burst is
|
||||
visible in the oplog/backlog metrics; not a correctness issue, just a metrics-reading note for
|
||||
operators watching `LocalDbOplogBacklog` during a purge window.
|
||||
|
||||
## Docs propagated
|
||||
|
||||
Component docs (`DataConnectionLayer`, `SiteRuntime`, `Communication`, `AuditLog`,
|
||||
`SiteEventLogging`, `StoreAndForward`, `NotificationOutbox`, `SiteCallAudit`, `TemplateEngine`,
|
||||
`DeploymentManager`, `CentralUI`, `ConfigurationDatabase`), CLAUDE.md Key Design Decisions
|
||||
(stream-completion reconnect, required site audit DB path, CLI HTTP timeout honesty, bulk
|
||||
`DeploySiteAsync`, LocalDb 0.2.1), and `docs/known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md`
|
||||
(amended for the `MaxBatchSize:64` read-page pin superseding the "retired, left at defaults" note).
|
||||
No components added/removed; README component table unchanged. Cross-reference sweep found no
|
||||
stale live references to `ScriptExecutionActor`/`AlarmExecutionActor` or `UX_AuditLog_EventId`
|
||||
outside historical plan/known-issue records, after correcting four residual mentions in
|
||||
`docs/requirements/Component-SiteRuntime.md` left over from the WP3.1 doc pass.
|
||||
@@ -139,6 +139,8 @@ Central cluster only. Sites have no user interface.
|
||||
- View diff between deployed and current template-derived configuration.
|
||||
- Deploy updated configuration to individual instances. **Pre-deployment validation** runs automatically before any deployment is sent — validation errors are displayed and block deployment.
|
||||
- Track deployment status (pending, in-progress, success, failed).
|
||||
- **Push-reload coalescing (arch-review WP2.4).** The page reloads its whole table (every deployment record + every instance) on each `DeploymentStatusChange` push, but the notifier fires per status *write* — a site-wide bulk deploy of N instances previously drove 2N+ back-to-back full reloads on the same circuit. Pushes are now leading-edge debounced (500ms): the first push after an idle gap reloads immediately (a single deployment stays as responsive as before), and every push inside the window collapses into one trailing reload.
|
||||
- **Known residual — no server-side paging.** The table still loads and filters every deployment record client-side; server-side paging plus precomputed status counts (deferred-work register item) is the follow-on for large fleets, not shipped in this remediation.
|
||||
|
||||
### System-Wide Artifact Deployment (Deployment Role)
|
||||
- Explicitly deploy shared scripts, external system definitions, database connection definitions, and data connection definitions to all sites or to an individual site. (Notification lists and SMTP configuration are central-only and are not deployed.)
|
||||
@@ -153,6 +155,7 @@ Central cluster only. Sites have no user interface.
|
||||
- The `DebugStreamService` creates a `DebugStreamBridgeActor` on the central side. The bridge actor opens a **gRPC server-streaming subscription** to the site's `SiteStreamGrpcServer` for the selected instance, then requests an initial `DebugViewSnapshot` over the central→site gRPC command channel (`SiteCommandService`).
|
||||
- Ongoing events (`AttributeValueChanged`, `AlarmStateChanged`) flow via the gRPC data stream directly to the bridge actor — they do not travel on the command channel.
|
||||
- Events are delivered to the Blazor component via callbacks, which call `InvokeAsync(StateHasChanged)` to push UI updates through the built-in SignalR circuit.
|
||||
- **Render coalescing (arch-review WP2.4).** Streamed events no longer trigger an individual dispatcher marshal + `StateHasChanged()` each — a chatty instance used to drive one full render (and two full tree rebuilds) per value change. Events now land in a thread-safe pending map keyed by attribute/alarm name (repeated updates to the same tag inside one window collapse to the latest), and exactly one dispatcher marshal per **250ms coalesce window** drains the map, bumps a version stamp, and renders once. This also fixes a latent thread-safety issue: the render dictionaries were plain (non-concurrent) `Dictionary`s enumerated by the render thread while written from the Akka/gRPC callback thread.
|
||||
- A pulsing "Live" indicator replaces the static "Connected" badge when streaming is active.
|
||||
- Subscribe-on-demand — stream starts when opened, stops when closed.
|
||||
- Read-only per-instance view (one instance per connection); no alarm acknowledgement is available from Debug View.
|
||||
@@ -192,8 +195,10 @@ Per-leaf alarm rendering (leaf nodes are individual conditions for native alarms
|
||||
- **Data path** — no new site-side code and no central alarm store. The page selects a site, queries its deployed instances, then fans out the existing per-instance `DebugViewSnapshot` Ask **concurrently** (capped with a `SemaphoreSlim`) and aggregates the returned `AlarmStates` client-side. The fan-out is **partial-results tolerant**: instances that time out are listed as "not reporting" while the rest still render. This snapshot fan-out now doubles as the **seed** for a near-real-time live feed (see **Live updates** below) rather than the sole refresh mechanism.
|
||||
- **Live updates** — the page is driven by a **transient, per-site central live alarm cache** (`ISiteAlarmLiveCache`, owned by the Communication component; see [Component-Communication](Component-Communication.md)). On site select the page subscribes to the cache; the cache runs one shared, reference-counted per-site aggregator that **seeds** from the snapshot fan-out and then stays warm on a single **site-wide, alarm-only** `SubscribeSite` gRPC stream (seed-then-stream, dedup by `(InstanceUniqueName, AlarmName, SourceReference)`). Applied deltas raise an in-process change event (mirroring `IDeploymentStatusNotifier`) that the Blazor circuit pushes to the browser via `StateHasChanged()` — no new SignalR hub. `AlarmSummaryService.BuildFromLiveAlarms` rebuilds the roll-up + rows from the cache's current alarm set. The cache is **purely in-memory on the active central node** — there is still **no persisted central alarm store**; on a NodeA↔NodeB failover the new active node re-seeds from scratch.
|
||||
- **View** — roll-up tiles (total active, worst severity, unacked count, per-`AlarmKind` counts) plus a flat, sortable, filterable table. Filters cover instance, `AlarmKind` (Computed / NativeOpcUa / NativeMxAccess), state, acked/unacked, severity threshold, and name search.
|
||||
- **Row virtualization (arch-review WP2.4).** Above `VirtualizeThreshold` (150) visible rows the table switches from a plain `foreach` to Blazor `Virtualize` (`ItemSize=37`, `<tr>` spacers), so a large-site alarm burst renders only the on-screen rows instead of every row in the DOM. Below the threshold the plain `foreach` stays — cheaper than the virtualization machinery and needs no JS interop. Both paths share one row-template, so the switch is invisible to styling/behavior.
|
||||
- **Read-only** — there are no ack / shelve / suppress controls (native alarms remain read-only by design).
|
||||
- **Refresh** — manual refresh button plus the 15s poll timer (mirroring the Health dashboard), now retained as a **fallback + `NotReporting` authority** behind the live cache: when the cache reports `IsLive`, the page renders live-cache state; when a stream is unhealthy, **the aggregator has died (deathwatch resets `IsLive`)**, or a site has not yet seeded, the poll keeps the page fresh so a stream failure never blanks it. Since WP2.3 `IsLive` also tracks the *stream* itself: a site-wide stream that faults or ends gracefully drops `IsLive` on the spot, so the page falls back to polling for the reopen window instead of rendering a snapshot that has quietly stopped updating. When live, the poll updates only the `NotReporting` list and leaves the row set to the delta path, so a slow fan-out can never momentarily revert a fresher live delta (R2 N5). (Aggregated live stream **delivered 2026-07-10** — see `docs/plans/2026-07-10-aggregated-live-alarm-stream-plan.md`.)
|
||||
- **Poll fan-out is deduplicated too (arch-review WP2.4).** The cold-cache/fallback poll runs through `SharedAlarmSummaryService`, a process-level memoizing façade over `AlarmSummaryService`: one memo slot per site, single-flight, just-under-15s window, so N operators watching the same site's fallback poll cost the node ONE per-instance debug-snapshot fan-out per window, not N. While the live cache is serving a site there is no poll fan-out at all — the façade instead answers straight from `ISiteAlarmLiveCache` (same `BuildFromLiveAlarmsCore` path the live subscription uses), so the aggregator's own seed/reconcile fan-out is the only one running.
|
||||
- **Reuse** — the alarm badge/formatter markup is factored out of Debug View into a shared `AlarmStateBadges` component consumed by both Debug View and this page.
|
||||
|
||||
### Parked Message Management (Deployment Role)
|
||||
@@ -232,6 +237,7 @@ Per-leaf alarm rendering (leaf nodes are individual conditions for native alarms
|
||||
- Headline **Notification Outbox KPI tiles** — queue depth, stuck count, and parked count. These are central-computed by the Notification Outbox from the central `Notifications` table (not part of any site health report). The full outbox view is on the dedicated Notification Outbox page.
|
||||
- Headline **Site Call Audit KPI tiles** — buffered count, parked count, and failed-last-interval. These are central-computed by the Site Call Audit component from the central `SiteCalls` table (not part of any site health report). The full cached-call view is on the dedicated Site Calls page.
|
||||
- Headline **Audit KPI tiles** — three tiles in a new "Audit" KPI group: **Audit volume**, **Audit error rate**, and **Audit backlog**. These are sourced from the Audit Log component (#23) and Health Monitoring per the metric definitions in Component-HealthMonitoring.md; the dashboard simply surfaces them. The full audit query view is on the dedicated Audit Log page.
|
||||
- **Shared KPI cache (arch-review WP2.4).** All four KPI families above — Notification Outbox, Site Call Audit, Audit, and their per-site/per-node breakdowns — are read through a process-level `IKpiSnapshotCache`, not queried per page load. Each accessor is independently memoized (8s TTL) with single-flight production, so N Blazor circuits polling the same KPI inside one window (e.g. ten operators with the Health dashboard open) cost the node ONE aggregate SQL round trip per KPI per window, not N. A failed query is never memoized — the next caller re-attempts it, and the calling page's existing per-tile "unavailable" degradation is unchanged. Every KPI-tile page (Health, Notification Outbox, Site Calls, Audit Log) shares the same cache instance.
|
||||
|
||||
### Site Event Log Viewer (Deployment Role)
|
||||
- Query site event logs remotely.
|
||||
|
||||
@@ -432,6 +432,7 @@ The `AuditLog` table is append-only and grows by every script-trust-boundary eve
|
||||
- Connection strings are provided via the Host's `DatabaseConfiguration` options (bound from `appsettings.json`).
|
||||
- EF Core manages connection pooling via the underlying ADO.NET SQL Server provider.
|
||||
- The DbContext is registered as a **scoped** service in the DI container, ensuring each request/operation gets its own instance.
|
||||
- **`DbContext` pooling (`AddDbContextPool`) is deliberately NOT used (arch-review WP2.2 — verified, not overlooked).** Pooling requires a context with a single public constructor taking only `DbContextOptions<TContext>`; `ScadaBridgeDbContext` has a second, runtime constructor taking `IDataProtectionProvider`, because the encrypting value converter for secret-bearing columns is built during `OnModelCreating` from that provider — and the model itself *differs* between the two constructors (no provider ⇒ no encrypting converter), so a pooled activator could silently produce a context that reads secret columns as ciphertext. Adopting pooling would mean moving the protector out of the constructor into a `DbContextOptions` extension — a change to the secrets-at-rest path, not a performance refactor — so it is deferred rather than folded into this pass.
|
||||
- No connection management for the Machine Data Database — that is handled separately by consumers (Inbound API scripts, external system gateway).
|
||||
|
||||
---
|
||||
|
||||
@@ -150,17 +150,22 @@ It runs the ordinary deployment pipeline in three phases, of which only the midd
|
||||
one is parallel:
|
||||
|
||||
1. **Prepare (serial).** Validate transition, take the operation lock, flatten +
|
||||
validate, run query-before-redeploy reconciliation, stage the
|
||||
`PendingDeployment`, insert the `InProgress` record. Every step here touches the
|
||||
scoped, non-thread-safe `DbContext`, so the phase is strictly serial. All
|
||||
instances share ONE `FlattenSession`, so a template chain common to N instances
|
||||
is walked once and the session-global queries (shared scripts, schema library,
|
||||
the site's data connections) run once for the batch.
|
||||
2. **Send (bounded parallel).** The `RefreshDeploymentCommand` round-trips run
|
||||
concurrently up to `SiteDeploymentMaxParallelism` (default 4), each under a
|
||||
`SiteDeploymentTimeoutPerInstance` deadline (default 120 s). This phase touches
|
||||
no repository — that is exactly why it is the only phase allowed to run in
|
||||
parallel. Shape mirrors `ArtifactDeploymentService.DeployCoreAsync`.
|
||||
validate, run query-before-redeploy reconciliation, insert the `InProgress`
|
||||
record. Every step here touches the scoped, non-thread-safe `DbContext`, so the
|
||||
phase is strictly serial. All instances share ONE `FlattenSession`, so a
|
||||
template chain common to N instances is walked once and the session-global
|
||||
queries (shared scripts, schema library, the site's data connections) run once
|
||||
for the batch.
|
||||
2. **Send (bounded parallel).** Concurrently up to `SiteDeploymentMaxParallelism`
|
||||
(default 4), each under a `SiteDeploymentTimeoutPerInstance` deadline
|
||||
(default 120 s): stage the `PendingDeployment` row **immediately before** that
|
||||
instance's `RefreshDeploymentCommand` round-trip, not upfront in phase 1 —
|
||||
`PendingDeployment` carries a 5-minute TTL, and staging every instance in the
|
||||
batch serially in phase 1 would burn a chunk of that TTL for the tail
|
||||
instances before their round-trip even starts (review fix, commit
|
||||
`e0e4b246`). Staging touches the repository but is scoped per-instance, so it
|
||||
does not reintroduce serialization. Shape mirrors
|
||||
`ArtifactDeploymentService.DeployCoreAsync`.
|
||||
3. **Finalize (serial).** Commit terminal statuses, apply post-success side
|
||||
effects, write audit rows, release each operation lock.
|
||||
|
||||
@@ -170,6 +175,15 @@ reported as a failed row while the rest proceed, and is individually retryable v
|
||||
the ordinary single-instance deploy. This matches the artifact-deployment policy:
|
||||
successful targets are never rolled back because another target failed.
|
||||
|
||||
**Cancellation is lock-safe (review fix, commit `e0e4b246`).** A cancelled bulk
|
||||
deploy no longer leaks the per-instance operation lock of any instance that had
|
||||
already taken one — a leaked lock is a wedged semaphore that never releases for
|
||||
the life of the process. Phase 2 does not throw on cancellation; each in-flight
|
||||
instance is instead recorded with a `Failed` outcome so phase 3 (finalize) still
|
||||
runs for it. An escape from phase 1 or phase 3 unwinds every not-yet-finalised
|
||||
entry the same way: `Failed` status plus lock release, so no instance touched by
|
||||
a cancelled bulk deploy can be left holding its lock.
|
||||
|
||||
`DeployInstanceAsync` is composed from the same three phase helpers with a batch of
|
||||
one, so the two entry points cannot drift on deployment identity, idempotency, lock
|
||||
coverage, or optimistic concurrency.
|
||||
|
||||
@@ -227,6 +227,7 @@ When the Instance Actor is stopped (due to disable, delete, or redeployment), Ak
|
||||
- **Pool sizing is instance-scaled and grow-only (WP3.1).** The pool was a fixed 8 threads regardless of load. It is now `clamp(max(ScriptExecutionThreadCount, ceil(enabledInstances / 8)), 1, ScriptExecutionMaxThreadCount)` — the existing `ScriptExecutionThreadCount` (default 8) becomes the **floor**, so configurations at or below 64 instances behave exactly as before, and the new `ScriptExecutionMaxThreadCount` (default 32) is the ceiling. Beyond the ceiling the per-script cap below is the real regulator. `DeploymentManagerActor.UpdateInstanceCounts` calls `EnsureCapacity` on every deploy / undeploy / enable / disable and once per staggered startup batch. Growth only: undeploying leaves idle threads, which cost nothing measurable and avoid drain/steal complexity.
|
||||
- **The deadline is armed at enqueue, not at dequeue (WP3.1).** The run's timeout `CancellationTokenSource` is created on the actor thread *before* the body is queued, so queue wait consumes the script's own budget. A body that dequeues past its deadline **skips execution entirely** and takes the existing timeout path (site event, script-error counter, error reply, completion message): a saturated pool sheds stale work instead of running it late with a fresh full budget.
|
||||
- **Concurrent runs per script are capped (WP3.1).** `MaxConcurrentRunsPerScript` (default 4) bounds runs in flight — queued or executing — for any one script or alarm on-trigger script. Over the cap the **newest** run is shed: the four already in flight are closest to their own deadlines and already charged against them, so nothing is ever reordered and no extra queue is needed (the scheduler's FIFO already is the queue). A shed increments `ISiteHealthCollector.IncrementScriptRunShed` (surfaced as `ScriptRunShedCount`), emits a `script`/`Warning` site event **rate-limited to one per script per minute** so a hot trigger cannot flood `site_events`, and — for an Ask-based `CallScript` — replies with an explicit error so a nested call or inbound-API route fails fast instead of hanging.
|
||||
- **The cap gates only depth-0 launches (review fix).** A trigger fire or a depth-0 `CallScript`/inbound-API route counts against `MaxConcurrentRunsPerScript`; a *nested* `Instance.CallScript` self-recursion (`callDepth > 0`) is exempt — it is already bounded by `MaxScriptCallDepth` instead, and the calling run is holding one of the four cap slots itself while it awaits the callee, so counting the nested launch too would spuriously shed legitimate self-recursion.
|
||||
|
||||
### Handling `Instance.CallScript`
|
||||
- When an external caller (another script run, an alarm on-trigger run, or a routed call from the Inbound API) sends a `CallScript` message to the Script Actor, it launches a run to handle the call.
|
||||
@@ -252,7 +253,7 @@ When the Instance Actor is stopped (due to disable, delete, or redeployment), Ak
|
||||
- **Expression** trigger evaluation runs on the shared .NET thread pool behind a process-wide concurrency gate (`TriggerEvalGate`, sized by `TriggerEvalMaxConcurrency`, default `max(2, ProcessorCount)`) — **not** on the bounded script-execution pool. This is the WP3.1 fix for arch-review finding #4 (High): when evaluation shared that pool, N script bodies blocked in synchronous I/O stalled *every* Expression trigger on the node — scripts and alarms alike — for an unbounded time, and the evaluation's own 2 s timeout was constructed inside the queued body, so it did not start ticking until dequeue. An alarm that should have raised in milliseconds simply never raised, with neither a raise nor a timeout visible to the operator. Trigger expressions are non-blocking **by construction** (`TriggerExpressionGlobals` exposes only reads over an in-memory snapshot, and the script trust gate has already denied I/O, network, threading, and reflection), so the shared pool is where they belong; a second dedicated pool was considered and rejected as adding threads, gauges, and a second starvation surface for no isolation gain. The evaluation deadline (`TriggerEvalTimeoutSeconds`, default 2, previously hardcoded) is now armed **at enqueue**, so gate-wait time burns the same budget and a saturated gate yields a timely `false` rather than an unbounded stall. Per-actor coalescing (one evaluation in flight, one pending) is unchanged and caps waiters at one per Expression trigger, so the gate queue is bounded by trigger count.
|
||||
- For binary trigger types (ValueMatch / RangeViolation / RateOfChange), when the condition is met and the alarm is currently in **normal** state, the alarm transitions to **active**:
|
||||
- Updates the alarm state on the parent Instance Actor (which publishes to the Akka stream).
|
||||
- If an on-trigger script is defined, spawns an Alarm Execution Actor to execute it.
|
||||
- If an on-trigger script is defined, launches its run via `ScriptRunLauncher` (see Alarm On-Trigger Run below) — no per-run child actor.
|
||||
- When the condition clears and the alarm is in **active** state, the alarm transitions to **normal**.
|
||||
- For HiLo triggers, the actor tracks the current `AlarmLevel` (None / Low / LowLow / High / HighHigh). Each level transition emits a fresh `AlarmStateChanged` with the new level and its priority; level escalations (e.g., High → HighHigh) and de-escalations (HighHigh → High) both produce events. The on-trigger script fires only on the Normal → non-None edge, not on escalations between alarm bands.
|
||||
- No script execution on clear in any trigger type.
|
||||
@@ -507,14 +508,14 @@ Per Akka.NET best practices, internal actor communication uses **Tell** (fire-an
|
||||
|
||||
**Ask** is reserved for system boundaries where a synchronous response is needed:
|
||||
|
||||
- **`Instance.CallScript()`**: Ask pattern from Script Execution Actor to sibling Script Actor. The caller needs the return value. Acceptable because script calls are infrequent relative to tag updates.
|
||||
- **`Instance.CallScript()`**: Ask pattern from the running script (via `ScriptRuntimeContext.CallScript`) to sibling Script Actor. The caller needs the return value. Acceptable because script calls are infrequent relative to tag updates.
|
||||
- **`Route.To().Call()`**: Ask from Inbound API to site Instance Actor via Communication Layer. External caller needs a response.
|
||||
- **Debug view snapshot**: Ask from Communication Layer to Instance Actor for initial state.
|
||||
|
||||
## Concurrency & Serialization
|
||||
|
||||
- The Instance Actor processes messages **sequentially** (standard Akka actor model). This means `SetAttribute` calls from concurrent Script Execution Actors are serialized at the Instance Actor, preventing race conditions on attribute state.
|
||||
- Script Execution Actors may run concurrently, but all state mutations (attribute reads/writes, alarm state updates) are mediated through the parent Instance Actor's message queue.
|
||||
- The Instance Actor processes messages **sequentially** (standard Akka actor model). This means `SetAttribute` calls from concurrent script runs are serialized at the Instance Actor, preventing race conditions on attribute state.
|
||||
- Script runs (launched via `ScriptRunLauncher`, no per-run child actor) may run concurrently, but all state mutations (attribute reads/writes, alarm state updates) are mediated through the parent Instance Actor's message queue.
|
||||
- External side effects (external system calls, notifications, database writes) are not serialized — concurrent scripts may produce interleaved side effects. This is acceptable because each side effect is independent.
|
||||
|
||||
## SiteStreamManager and gRPC Integration
|
||||
|
||||
@@ -108,6 +108,12 @@ Notifications are unaffected: they have no tracking table. Their `NotificationId
|
||||
|
||||
On every tracking-table status transition, the site emits a `CachedCallTelemetry` message to the central Site Call Audit component over the site→central channel. Emission is best-effort, at-least-once, and idempotent on `TrackedOperationId`. Because telemetry is best-effort, the site also responds to `CachedCallReconcileRequest` reconciliation pulls — cursor-based per-site reads of tracking rows changed since a cursor — so any missed telemetry self-heals. The site never depends on central; central converges to the site.
|
||||
|
||||
The telemetry-emitting `ICachedCallLifecycleObserver` hook is dispatched through a bounded, single-reader **observer queue** (`StoreAndForwardOptions.ObserverQueueCapacity`, default 10,000, `DropOldest`) — the one unbounded `Channel<T>` left in the system before this bound was added (arch-review WP2.6c). A slow or stuck observer (e.g. a SQLite audit write wedged behind disk contention) can no longer grow this queue without limit; it instead sheds the oldest unprocessed notification, incrementing an `ObserverQueueDroppedCount` counter that counts every drop. **Logging is separately rate-limited (adversarial review finding F3):** a Warning fires for the first drop of an episode, then at most one rollup Warning per minute while drops keep happening — never one Warning-per-dropped-item, which would itself have been a log-flood risk under the exact sustained-drop condition the bound exists to survive. This bounds memory, not delivery: telemetry loss here is covered by the reconciliation pull above, same as any other missed telemetry.
|
||||
|
||||
### Retry Sweep Indexing
|
||||
|
||||
`GetMessagesForRetryAsync` orders candidates `ORDER BY created_at ASC` within the due `status`. The existing `idx_sf_messages_status_due (status, last_attempt_at_ms)` matches the status filter and due-time predicate but not this ordering, forcing a sort/scan on a large sweep. A second covering index, `idx_sf_messages_status_created (status, created_at)`, lets the sweep walk matching rows already in `created_at` order and stop at the batch limit without re-sorting or touching non-pending rows (arch-review WP1.4). Both indexes are retained — `idx_sf_messages_status_due` still backs status+due-time lookups that don't order by `created_at`.
|
||||
|
||||
## Parked Message Management
|
||||
|
||||
- Parked messages remain stored at the site in SQLite.
|
||||
|
||||
@@ -152,11 +152,19 @@ which still collapses the repeated composed-chain loads inside a single instance
|
||||
flatten.
|
||||
|
||||
Cache validity is decided by **`ITemplateGraphWatermark`**, a process-wide set of
|
||||
monotonic counters bumped by the configuration-database unit of work — the only
|
||||
place every template-graph writer funnels through (`TemplateService`, the
|
||||
`ManagementActor` native-alarm-source handlers, and the Transport bundle importer
|
||||
all commit via the same `SaveChangesAsync`, and the change tracker is inspected
|
||||
pre-commit to attribute each change to its owning template or instance):
|
||||
monotonic counters. `TemplateService` and the `ManagementActor` native-alarm-source
|
||||
handlers bump it via `TemplateEngineRepository.SaveChangesAsync`'s change-tracker
|
||||
inspection, which attributes each pending change to its owning template or
|
||||
instance pre-commit. The Transport bundle importer (`BundleImporter`) commits
|
||||
through the raw `DbContext` instead — bypassing that repository — so it calls
|
||||
`ITemplateGraphWatermark.BumpAll()` itself, once per import: after commit on
|
||||
success, after rollback on failure (a bundle import can touch templates,
|
||||
instances, and connections in one transaction, so a scoped bump isn't
|
||||
worthwhile). A data connection edit (`SiteRepository.SaveChangesAsync`) also bumps — via `BumpAll()`,
|
||||
since a connection has no owning template — because `Protocol`/`PrimaryConfiguration`/
|
||||
`BackupConfiguration`/`FailoverRetryCount` are revision-hash inputs (review fix, commit
|
||||
`e0e4b246`, F2c). All three writers converge on the same watermark, so a cached flatten
|
||||
still invalidates correctly regardless of which one changed the graph:
|
||||
|
||||
- a memoised **template** is keyed on `(id, template version)`;
|
||||
- a memoised **chain** is valid only while BOTH the graph's `StructureVersion`
|
||||
@@ -225,6 +233,18 @@ A derived template stores `IsInherited` **placeholder rows** mirroring every mem
|
||||
|
||||
Both delegate to one **order-independent reconcile** that compares a template's stored inherited rows against the inheritance resolver's effective set (the same precedence + HiLo merge the editor preview and deploy use) and only ever touches `IsInherited` placeholder rows — never an authored override (`IsInherited == false`). Because the resolver ignores placeholder rows when picking winners, reconciling one template never changes what another resolves, so the operation needs no particular ordering. The effective value mirrored into each placeholder matches the staleness comparison key per member type, so after a reconcile the "base changed" banner clears. Reconcile is best-effort housekeeping for the *stored authoring rows*: a deploy always re-resolves the chain fresh regardless, so a not-yet-resynced template still deploys correctly.
|
||||
|
||||
### Slim List Projections (WP2.5)
|
||||
|
||||
The `ListTemplates` management command (backing the CLI `template list` and the Central UI
|
||||
template tree) previously materialised every template's full child graph — attributes, alarms,
|
||||
script bodies, compositions, native alarm sources — to page it in memory and discard all but one
|
||||
page. `ITemplateEngineRepository.GetTemplateSummariesAsync(skip, take)` instead pages **in the
|
||||
database** against `TemplateSummary`, a row-shaped projection (id, name, description, parent/folder
|
||||
ids, derived flag) with child collections reduced to **counts** — the only thing a list surface
|
||||
renders. `TemplateSummary.MaxPageSize` (1000) mirrors the existing in-memory page clamp so DB-side
|
||||
paging cannot be used to pull an unbounded result set. Additive-only message-contract evolution,
|
||||
same rule as other Commons message types.
|
||||
|
||||
## Diff Calculation
|
||||
|
||||
The Template Engine can compare:
|
||||
|
||||
Reference in New Issue
Block a user