Resolve all 33 medium-severity architecture review findings
Second slice of the 2026-09-09 whole-system review remediation
(docs/plans/2026-09-09-arch-review.md §10), on top of the High slice 146a1323.
Runtime/devices: SR-2 alarm on-trigger integration scope; SR-3 correlated NativeAlarmActor
subscribe retry; SR-4 immutable audited DB target; SR-5 ordered InstanceActor stop/flush;
SR-12 rehydration fence; SR-13 S&F gate recheck per delivery; DC-4 no OPC UA security
downgrade; DC-5 alarm snapshot for every new subscriber; ORCH-1 sync-dispose joins the pump.
Management/UI: MG-2 site scope on audit tree + Flow Debugger Replay; MG-3 stale-probe memo
removed; MG-4 server-owned bulk-deploy deadline; UI-5 sandbox runs in a killable child
process (new CentralUI.SandboxWorker project); UI-14 bounded sandbox output/IPC.
Edge/audit: EI-1 relay audit ownership; EI-2 SkipBodyCapture suppresses error text; EI-3/EI-11
error-field redaction + caps; EI-4 single relay deadline, pre-execution-only retry; EI-5
authenticated relay provenance; EI-10 body cap before audit read.
Persistence/contracts: DL-1 global Instance.UniqueName unique index + conflict translation;
DL-2 DataConnection protocol config encrypted at rest (migration
20260909203046_EnforceGlobalInstanceNamesAndProtectProtocolConfiguration — coordinated
central upgrade, old binaries cannot read protected rows); DL-3 bounded Replay paging;
DL-6 cross-site Overwrite of a deployed instance rejected; XC-4 proto descriptor contract test.
Config/hygiene: XC-1 central listener isolation by local port; XC-2/XC-3 inert SMTP + cron
keys removed from rig settings; CM-2 env2 bootstrap guard enabled; DC-6 deploy site exit code;
DC-7 notifier retries only explicit standby rejection; XC-5 ClusterClient purged from docs.
Gates recorded in the report: build 0/0, 9 397 passed / 0 failed, rig redeployed on all
eight nodes with the migration applied. 46 Low findings remain open.
This commit is contained in:
Joseph Doherty
@@ -1,6 +1,6 @@
|
||||
# ScadaBridge — architectural / code review (whole system)
|
||||
|
||||
**Date:** 2026-09-09 · **Status:** ALL 13 HIGH FINDINGS COMPLETED (see §9)
|
||||
**Date:** 2026-09-09 · **Status:** ALL 13 HIGH AND 33 MEDIUM FINDINGS COMPLETED (see §§9–10)
|
||||
**Reviewed revision:** `main` @ `c078bc8f`. Its only difference from the first review's
|
||||
`d8229916` is the original findings document; production source/configuration is identical.
|
||||
**Scope:** all 26 `src/` projects, 32 test/support projects, both Docker rig configurations,
|
||||
@@ -14,7 +14,7 @@ probes. Each area revalidated its existing IDs and searched for additional defec
|
||||
mechanisms, runtime probes, latent risks, and unproven operational impact are distinguished below.
|
||||
This is a focused whole-system review, not a claim that every line or production topology was
|
||||
exercised. The review phase ran no rig rebuild, device writes, failover drill, or remote QA/production inspection.
|
||||
Sections 1–8 preserve that reviewed baseline; §9 records the subsequent implementation and local rig verification.
|
||||
Sections 1–8 preserve that reviewed baseline; §§9–10 record subsequent implementation and local rig verification.
|
||||
|
||||
**Second-pass changes:** 11 new findings; UI-3 withdrawn as unsupported; XC-6 merged into CM-5;
|
||||
UI-4/6/7 reclassified as Low hardening gaps. Existing IDs are preserved for traceability. The
|
||||
@@ -38,8 +38,8 @@ contract changes since August are additive; all eight migrations are sound and t
|
||||
matches. The test suite is broad; fresh execution evidence and its limits are in §7.
|
||||
|
||||
The reviewed baseline contained **92 unique findings: 13 High, 33 Medium, 46 Low**.
|
||||
**Remediation status: all 13 High findings completed; 79 findings remain open (33 Medium, 46 Low).**
|
||||
Completion evidence and limits are recorded in §9.
|
||||
**Remediation status: all 13 High and 33 Medium findings completed; 46 Low findings remain open.**
|
||||
Completion evidence and limits are recorded in §§9–10.
|
||||
This includes low-priority test/documentation gaps and latent risks, not just reproduced failures.
|
||||
The reviewed High-priority findings were:
|
||||
|
||||
@@ -59,7 +59,7 @@ The reviewed High-priority findings were:
|
||||
| CM-10 | **High** | **Cross-node retry treats every `Unavailable` as unsent**, repeating non-idempotent writes after an ambiguous response loss | Communication |
|
||||
| UI-13 | **High** | **Open Blazor circuits retain revoked roles and site scopes**, even after the HTTP cookie has refreshed | Central UI / Security |
|
||||
|
||||
The remaining findings group into five themes: **audit completeness** (EI-1/2/3/5,
|
||||
The other reviewed findings grouped into five themes: **audit completeness** (EI-1/2/3/5,
|
||||
UI-1), **authorization scope** (MG-2, UI-6/7/8, XC-1, XC-8), **stale/duplicate-apply under
|
||||
failover** (MG-3/4, EI-4/9, DL-1/6, ORCH-1), **subscription/stream lifecycle** (DC-4/5, SR-3/5,
|
||||
CM-2), and **inert configuration** (XC-2/3/7). §5 groups everything into a remediation program.
|
||||
@@ -743,7 +743,7 @@ fetched production credentials.
|
||||
| CM-9 | Real central auth interceptor with site-A metadata/key accepted reconciliation payload `SiteIdentifier=site-b`; real mapper preserved B | Auth/request binding gap, followed in source to token staging; no cross-site secret fetch executed |
|
||||
| CM-10 | Provider callback applied a synthetic effect, threw Unavailable, then succeeded on peer: **applications=2**, returned `ok` | Fault-injected retry path, not physical network/device duplication |
|
||||
| SR-11 | In-memory SQLite: normal wrapped write audit count **1**; `tx.Connection` was original provider, not wrapper; raw write affected **1** row with **0** audit emissions, persisted value 42 | Actual ADO escape and missing audit; reflection only instantiated internal wrapper in harness, not used by escape; not SQL Server/Roslyn end-to-end |
|
||||
| EI-10 | Real Kestrel read-before-cap pattern: desired cap 8, listener cap 1000, chunked body 100 → **HTTP 200, IsReadOnly=True, acceptedBytes=100** | Real framework middleware-order mechanism, not full app/Traefik deployment |
|
||||
| EI-10 | Implemented; verification pending | Early bounded body reader applies even with auditing disabled; real Kestrel gate pending. |
|
||||
|
||||
Probe sources/output: `/tmp/scadabridge-review-probes/Program.cs` and
|
||||
`/tmp/scadabridge-review-probes.log`; `/tmp/scadabridge-sr11-probe/Program.cs`;
|
||||
@@ -759,7 +759,7 @@ Repeatable regression recipes are attached to each finding and the §5 acceptanc
|
||||
| UI-3 | Withdraw: caller awaits retain renderer synchronization context; no demonstrated off-dispatcher entry |
|
||||
| XC-6 / CM-5 | One defect, retain CM-5 Low; XC-6 is a historical alias |
|
||||
| UI-1 | Many bypass paths in 16 files, not every UI edit; template/shared-script service paths already audit |
|
||||
| MG-3 | Use peer-originated DB edit/local-watermark mismatch as trigger; activation-only invalidation is insufficient |
|
||||
| MG-3 | Implemented; verification pending | Removed stale deployment probe memoization; peer-change regressions passed. |
|
||||
| UI-4/6/7 | Low hardening/prevention gaps; no continuing-circuit refresh outage or unauthorized handler exploit demonstrated |
|
||||
| UI-2 / UI-13 | Passive cookie renewal and stale circuit authority are separate failures; both need server-side handling |
|
||||
| UI-5 / UI-14 | Timeout is not termination or a memory bound; never recycle an admission slot while runaway work remains |
|
||||
@@ -896,3 +896,91 @@ No device writes, failover drill, remote deployment, commit or push was performe
|
||||
Medium/Low register and baseline formatting debt remain. The local rig has an existing
|
||||
`zztest-conn-5d93ea81` test connection retrying unreachable `localhost:4840`; healthy cluster
|
||||
readiness does not assert that every configured device endpoint is reachable.
|
||||
|
||||
## 10. Medium-severity remediation completion
|
||||
|
||||
**All 33 Medium findings are completed and verified.** The High fixes were committed first as
|
||||
`146a1323` (`Resolve all high-severity architecture review findings`). Medium implementation,
|
||||
tests and documentation remain uncommitted on `codex/resolve-high-review-findings` in
|
||||
`/Users/dohertj2/Desktop/ScadaBridge-high-findings`; no push was requested for this phase.
|
||||
Parallel subagents handled runtime/devices, management/UI and edge integrations; the primary
|
||||
agent handled persistence, listener/contracts, configuration and final integration. Independent
|
||||
cross-reviews rechecked lifecycle, sandbox containment, relay provenance and migration safety.
|
||||
|
||||
| Finding | Status | Progress / evidence |
|
||||
|---|---|---|
|
||||
| SR-2 | Completed | Fresh alarm execution scope supplies database and tracking dependencies; runtime focused and full solution suites passed. |
|
||||
| SR-3 | Completed | Correlated bounded retry restores failed alarm subscriptions; full solution suite passed. |
|
||||
| SR-4 | Completed | Implemented; immutable audited target regression passed. |
|
||||
| SR-5 | Completed | Ordered persistence drain and same-name stop barriers; independent review passed; runtime suite 102 passed including locked delete/disable and queued cancellation. |
|
||||
| SR-12 | Completed | Implemented; delayed rehydration fencing regression passed. |
|
||||
| SR-13 | Completed | Authority rechecked after each lane admission and before every delivery; StoreAndForward suite 38 passed. |
|
||||
| CM-2 | Completed | Implemented; all four env2 node configurations enable deterministic bootstrap; parsed configuration validation passed. |
|
||||
| MG-2 | Completed | Mandatory fresh caller scope filters audit trees and replay; management regressions and full UI suite passed. |
|
||||
| MG-3 | Completed | Removed stale deployment probe memoization; peer-change regressions passed. |
|
||||
| MG-4 | Completed | Server-owned bulk deadline stops subsequent dispatch and returns unfinished failures; targeted deployment tests passed. |
|
||||
| UI-5 | Completed | Isolated child processes killed and joined before admission reuse; broker-deadline and runaway-worker tests included in 97 passing UI cases. Published Linux image contains worker DLL/deps/runtimeconfig; deployed browser termination/recovery check passed. |
|
||||
| UI-14 | Completed | Console, return serialization and IPC bounded; oversized-output regressions passed in the full UI suite and deployed browser. |
|
||||
| EI-1 | Completed | Relay ownership and uncertain-outcome audit completion implemented; InboundAPI suite passed. |
|
||||
| EI-2 | Completed | Body-capture opt-out also suppresses free-form failure details and companion events; InboundAPI suite passed. |
|
||||
| EI-3 | Completed | SQL exception details respect sensitive-parameter redaction policy; audit redaction regressions passed. |
|
||||
| EI-4 | Completed | Single relay deadline and explicit pre-execution-only retry; ambiguous outcomes are not replayed; InboundAPI suite passed. |
|
||||
| EI-5 | Completed | Authenticated request-bound relay provenance replaces trust in caller-supplied forwarding headers; independent review and InboundAPI suite passed. |
|
||||
| EI-10 | Completed | Early body cap applies with auditing disabled; real Kestrel oversized/chunked cases included in six passing Host regressions. |
|
||||
| EI-11 | Completed | All retained error-message/detail fields are capped and redacted; targeted audit suite passed. |
|
||||
| DL-1 | Completed | Global unique index, domain conflict translation and duplicate preflight migration; real SQL concurrency/upgrade guard tests passed. |
|
||||
| DL-2 | Completed | Protected primary/backup configuration with per-provider model cache and idempotent bounded startup upgrade; real SQL and SQLite regressions passed. |
|
||||
| DL-3 | Completed | Eight-row keyset paging, 500-event ceiling, UTF-8 budget and preserved root/truncation metadata; all six real SQL regressions passed. |
|
||||
| DL-6 | Completed | Deployed cross-site overwrite rejected; both real import cases passed after a valid failing baseline. |
|
||||
| DC-4 | Completed | Exact requested OPC UA security mode enforced during discovery/connect; focused device suite passed. |
|
||||
| DC-5 | Completed | Shared alarm cache merges live events with snapshots and fences stale epochs; independent review and 34 device regressions passed. |
|
||||
| DC-6 | Completed | CLI returns failure for nonzero bulk failureCount and malformed responses; all three actual command/socket regressions passed. |
|
||||
| DC-7 | Completed | Notifier retries only explicit standby rejection; notifier suite passed. |
|
||||
| XC-1 | Completed | Actual socket and gRPC metadata enforce endpoint ownership despite forged authority; six combined Host/Kestrel regressions passed. |
|
||||
| XC-2 | Completed | Implemented; removed four ignored SMTP keys from all four central rig settings; database configuration guidance replaces them. |
|
||||
| XC-3 | Completed | Implemented; removed ignored cron key from all eight site settings; documents interval-based purge. |
|
||||
| XC-4 | Completed | Actual protoc descriptors match vendored message/service graphs; four tests passed including three name/type/service drift controls. Implicit JSON names normalized using effective runtime descriptors. |
|
||||
| XC-5 | Completed | Implemented; README, high-level/component requirements and component skill now describe actual gRPC/HTTP transport. |
|
||||
| ORCH-1 | Completed | Synchronous disposal joins context-free persistence drain; runtime 102-case suite passed. |
|
||||
|
||||
### Final verification
|
||||
|
||||
| Gate | Result / evidence |
|
||||
|---|---|
|
||||
| Full solution build | **0 warnings, 0 errors**; `/tmp/medium-full-build2.log`. Final deployed-browser project build also passed 0/0 after its test-only update (`/tmp/medium-browser-build.log`). |
|
||||
| Selected full solution | `dotnet test ZB.MOM.WW.ScadaBridge.slnx --no-restore --filter 'FullyQualifiedName!~PlaywrightTests' --logger trx --results-directory /tmp/scadabridge-medium-full`: **9,397 passed, 0 failed, 0 skipped**, exit 0, 30 result files. `/tmp/medium-full-tests.log`. Includes 1,399 UI, 596 Host, 125 Transport integration and 117 cross-component integration cases plus 13 performance checks. |
|
||||
| Deployed browser | **2 passed, 0 failed, 0 skipped**: Linux sandbox normal execution, bounded console/return output, noncooperative infinite-loop timeout and subsequent successful execution; trusted/passive session input and antiforgery. `/tmp/medium-browser-final.log`. No child worker remained afterward on either central node. |
|
||||
| Real persistence | Global-name concurrency and duplicate-upgrade rejection, primary/backup encryption and idempotent conversion passed against SQL Server; key-ring isolation and guards passed against SQLite. `/tmp/medium-persistence-final.log`. Final replay keyset/Unicode/oversized-root SQL checks also passed and are included in the full solution run. |
|
||||
| Transport contract | Six real Kestrel listener/body-cap cases passed (`/tmp/medium-host-green.log`); actual protoc descriptor comparison and three drift controls passed (`/tmp/medium-proto-green2.log`). |
|
||||
| Local rig | Image `sha256:b8d158309b5d850fc01a3d32e7325e2238611b039e37f5fa0d51e7a488ddee36` on all **eight nodes**; all ready, one active per pair, two members per pair, zero unreachable members or restarts. Site-A replication connected, backlog 0. `/tmp/scadabridge-medium-postdeploy-health.json`. |
|
||||
| Upgrade | Coordinated central-pair stop, successful COPY_ONLY/CHECKSUM backup and RESTORE VERIFYONLY, then migration `20260909203046_EnforceGlobalInstanceNamesAndProtectProtocolConfiguration`. All 4 protocol rows retained; 3 non-null primary values protected, 0 unprotected values, existing 1-entry key ring preserved. `/tmp/medium-rig-backup.log`, `/tmp/medium-rig-before-counts.log`, `/tmp/medium-rig-after-counts.log`. Live Site-A OPC UA connection reconnected after upgrade. |
|
||||
| Configuration/docs | All 12 rig JSON files parse; four env2 bootstrap guards enabled; obsolete SMTP/cron keys removed; transport entry-point documentation corrected. Component requirements and umbrella project index updated. |
|
||||
|
||||
### Operational notes and verification limits
|
||||
|
||||
- The SQL backup remains inside the existing SQL Server data volume at
|
||||
`/var/opt/mssql/backup/ScadaBridgeConfig_before_medium_20260909_211100.bak`.
|
||||
Protocol encryption is a coordinated central upgrade: old binaries cannot read the new values.
|
||||
A rollback requires the coordinated pre-upgrade database/key-ring backup, as documented in
|
||||
`Component-ConfigurationDatabase.md`; the migration refuses a downgrade over protected rows.
|
||||
- The image was built from the implementation worktree and deployed using the original
|
||||
repository's canonical Compose file. Existing configuration/log bind mounts and named data
|
||||
volumes were preserved. Env2 bootstrap settings were verified statically; env2 containers
|
||||
remain untouched. No device writes, live failover drill or remote deployment was performed.
|
||||
- Sandbox isolation provides a process resource/lifetime boundary, not a separate OS identity.
|
||||
The worker is killed at the deadline. If a parent-side broker ignores cancellation, its slot
|
||||
and scope remain owned until that operation actually completes; no orphan broker operation
|
||||
or early capacity reuse is introduced. Already accepted remote work cannot be undone by
|
||||
killing its client. Console retains 32,768 characters plus a fixed truncation marker; return
|
||||
serialization is capped at 32 KiB bytes, IPC at 1 MiB, managed worker heap at 512 MiB.
|
||||
- Replay is capped at 500 events and a 1 MiB serialized UTF-8 budget, with eight-row keyset
|
||||
pages bounding EF buffering. Oversized events are omitted whole with truthful truncation
|
||||
and separately retained root metadata. Shared snapshot/lifecycle races are covered by
|
||||
deterministic tests, not a live raised-alarm scenario. The pre-existing unreachable
|
||||
`zztest-conn-5d93ea81` test connection remains outside this remediation.
|
||||
- The first proto comparison exposed implicit JSON-name representation differences, not
|
||||
wire drift. It now compares effective runtime names against compiler output while preserving
|
||||
explicit field/service metadata. Earlier fixture/build failures were corrected without
|
||||
weakening production guards: audit test doubles accept caller scope, and protocol-writing
|
||||
fixtures use stable ephemeral key rings. These are not claimed as before-fix reproductions.
|
||||
- The 46 Low findings remain open. Focused test counts overlap the full suite and are not additive.
|
||||
|
||||
@@ -898,3 +898,9 @@ the row remains durable locally. Good rows preceding it can still advance. Opera
|
||||
reduce that oversized payload or deploy coordinated larger limits to resume past it; repeated
|
||||
drain failures are logged. A lost response retains the existing at-least-once retry/cursor
|
||||
semantics. The separate Flow Debugger trace pull remains governed by its own contract.
|
||||
|
||||
### Error-field privacy
|
||||
|
||||
The canonical redactor applies global and per-target body patterns and UTF-8 byte caps to `ErrorMessage` as well as `ErrorDetail`, request, response, and extra fields. A secret cannot survive only because an emitter placed it in the short error message.
|
||||
|
||||
For `DbOutbound` targets with `RedactSqlParamsMatching`, request parameters are selected by parameter name. SQL exception messages often echo parameter values without names, so both free-text error fields are suppressed under that policy; the audit channel, failed status, operation target and other structured metadata remain available. Applying the name regex to exception text would not protect the values. This policy covers inbound helper and site raw-connection emitters through the shared canonical redactor.
|
||||
|
||||
@@ -464,3 +464,11 @@ Configuration is resolved in the following priority order (highest wins):
|
||||
- **Management Service (via HTTP)**: The primary runtime dependency. All operations except `debug stream` are sent as HTTP POST requests to the Management API endpoint on a central node, which dispatches to the ManagementActor.
|
||||
- **Central Host**: Serves the Management API at `POST /management` and the debug stream SignalR hub at `/hubs/debug-stream`. Handles LDAP authentication, role resolution, and ManagementActor dispatch.
|
||||
- **Debug Stream Hub (via SignalR WebSocket)**: The `debug stream` command connects to the `/hubs/debug-stream` hub on the central server for real-time event streaming. This is the only CLI command that uses a persistent connection rather than request/response.
|
||||
|
||||
|
||||
### Bulk deployment completion
|
||||
|
||||
deploy site preserves the returned summary in JSON/table output and exits 1 when failureCount
|
||||
is nonzero or the summary is malformed, even when HTTP returned 2xx. A zero failure count exits
|
||||
0; HTTP authorization failures retain exit 2. The server owns the bounded bulk operation deadline;
|
||||
the client does not impose a shorter total timer on this command.
|
||||
|
||||
@@ -338,3 +338,11 @@ Page configuration handlers use `ConfigurationCommandService` and the guarded ma
|
||||
Cookie circuits retain their initial principal without retaining `HttpContext`; authoritative reads re-map stored directory groups against current grants. Mutation handlers recheck role and persisted target site immediately before dispatch. Passive session pings do not count as user activity. Trusted browser input posts an antiforgery-protected activity signal; the server activity tracker also makes it visible to the existing circuit.
|
||||
|
||||
The explicit server-configured development auto-login mode preserves its synthetic all-role principal. It uses the Cookies scheme but has no LDAP group claims; it must not enter directory re-mapping. This exception follows validated `AuthDisableLoginOptions`, never a caller-supplied claim.
|
||||
|
||||
### Isolated Test Run workers
|
||||
|
||||
Test Run executes in a disposable `CentralUI.SandboxWorker` process. A process-wide admission gate permits at most four workers. Queueing, compilation, execution and gateway calls consume the requested deadline (1–10 seconds); cancellation kills the process tree and capacity is released only after confirmed exit. Console capture retains at most 32,000 characters with an explicit truncation marker; return JSON is limited to 32,000 bytes. Private IPC frames are limited to 1 MiB and stderr is drained without accumulation. Each worker has a 512 MiB managed-heap ceiling. The worker is a resource/lifetime boundary; the semantic script trust policy still gates compilation.
|
||||
|
||||
External calls, shared-script lookup and bound-instance operations use a parent broker. Configured SQL connections are recreated inside the worker using credentials passed only over its private pipe, preserving the existing SQL permissions and API. Termination prevents further calls and closes worker-owned sockets; it cannot undo SQL or remote work already accepted by another server. Timeout results do not retain a terminated worker's partial console buffer.
|
||||
|
||||
The deadline terminates the child independently of gateway cancellation. If a broker provider ignores cancellation, the response waits for that operation to finish and its admission slot remains charged; this preserves the owning request scope and prevents unbounded abandoned operations.
|
||||
|
||||
@@ -289,7 +289,7 @@ ZB.MOM.WW.ScadaBridge.Commons/
|
||||
│ │ # IngestCachedTelemetryCommand/Reply,
|
||||
│ │ # UpsertSiteCallCommand/Reply, SiteCallQueries,
|
||||
│ │ # SiteCallRelayMessages
|
||||
│ └── Management/ # HTTP/ClusterClient management commands + registry,
|
||||
│ └── Management/ # HTTP management commands + registry,
|
||||
│ # including TransportCommands (Export/Preview/Import bundle)
|
||||
├── Observability/ # ScadaBridgeTelemetry — shared Meter ("ZB.MOM.WW.ScadaBridge") +
|
||||
│ # counters/observable-gauges (deployments applied, inbound-API
|
||||
|
||||
@@ -482,3 +482,20 @@ Site-to-central notification submission and audit ingestion can retry because th
|
||||
identifiers are idempotent; status reads and sequence-guarded health reports can retry as well.
|
||||
Reconciliation mints fetch tokens and only retries a proven pre-dispatch rejection. Attempts
|
||||
share the original absolute deadline. No transport retries `DeadlineExceeded` across nodes.
|
||||
|
||||
|
||||
### Source and generated wire contract guard
|
||||
|
||||
ProtoDescriptorContractTests invokes the pinned Grpc.Tools protoc for the test host and compares
|
||||
the full compiled FileDescriptorProto against each vendored descriptor. This covers message/field
|
||||
names, numbers, types, maps, oneofs, enums, options and RPC service signatures, including streaming.
|
||||
Mutation controls prove name/type/service drift is detected. Updating a proto requires regenerating
|
||||
its vendored C# with docker/regen-proto.sh; changing a source hash alone cannot satisfy the guard.
|
||||
The guard requires the package's native compiler to execute on the test host; it does not run
|
||||
inside the Docker image build, which retains vendored sources for the documented ARM workaround.
|
||||
|
||||
Central listener isolation runs after endpoint routing and before authentication/dispatch. Only
|
||||
mapped gRPC methods can execute on the control-plane socket; they are rejected on HTTP/metrics
|
||||
sockets. Dedicated metrics sockets expose metrics and health only. Decisions use LocalPort and
|
||||
endpoint metadata, never the caller's Host/authority header. The env2 central/site pairs now enable
|
||||
the same deterministic bootstrap guard as the main rig for simultaneous container recreation.
|
||||
|
||||
@@ -435,7 +435,7 @@ The `AuditLog` table is append-only and grows by every script-trust-boundary eve
|
||||
- Connection strings are provided via the Host's `DatabaseConfiguration` options (bound from `appsettings.json`).
|
||||
- EF Core manages connection pooling via the underlying ADO.NET SQL Server provider.
|
||||
- The DbContext is registered as a **scoped** service in the DI container, ensuring each request/operation gets its own instance.
|
||||
- **`DbContext` pooling (`AddDbContextPool`) is deliberately NOT used (arch-review WP2.2 — verified, not overlooked).** Pooling requires a context with a single public constructor taking only `DbContextOptions<TContext>`; `ScadaBridgeDbContext` has a second, runtime constructor taking `IDataProtectionProvider`, because the encrypting value converter for secret-bearing columns is built during `OnModelCreating` from that provider — and the model itself *differs* between the two constructors (no provider ⇒ no encrypting converter), so a pooled activator could silently produce a context that reads secret columns as ciphertext. Adopting pooling would mean moving the protector out of the constructor into a `DbContextOptions` extension — a change to the secrets-at-rest path, not a performance refactor — so it is deferred rather than folded into this pass.
|
||||
- **`DbContext` pooling (`AddDbContextPool`) is deliberately NOT used (arch-review WP2.2 — verified, not overlooked).** Pooling requires a context with a single public constructor taking only `DbContextOptions<TContext>`; `ScadaBridgeDbContext` has a second, runtime constructor taking `IDataProtectionProvider`, because the encrypting value converter for secret-bearing columns is built during `OnModelCreating` from that provider — and the model itself *differs* between the two constructors (no provider ⇒ schema-only protector that rejects secret access), so a pooled activator could silently produce a context that reads secret columns as ciphertext. Adopting pooling would mean moving the protector out of the constructor into a `DbContextOptions` extension — a change to the secrets-at-rest path, not a performance refactor — so it is deferred rather than folded into this pass.
|
||||
- No connection management for the Machine Data Database — that is handled separately by consumers (Inbound API scripts, external system gateway).
|
||||
|
||||
---
|
||||
@@ -464,3 +464,35 @@ The `AuditLog` table is append-only and grows by every script-trust-boundary eve
|
||||
### Configuration command transactions (2026-09-09)
|
||||
|
||||
`ConfigurationCommandTransaction` creates a fresh dependency scope and EF context for each execution-strategy attempt. Entity changes and staged audit entries share one transaction, including intermediate saves needed for generated keys. A failure before commit rolls back both. An uncertain commit is not replayed, preventing duplicate creates; the caller must reconcile its result. Postcommit cache actions execute while their dependency scope remains alive; failures are logged without replaying the committed operation.
|
||||
|
||||
### Bounded trace retrieval
|
||||
|
||||
Execution-chain readers use a parameterized permitted-site JSON set throughout the up/down SQL walk and metadata projection. Replay caps chain identifiers at2048, reads at most501ordered candidates sequentially, and bounds DetailsJson in SQL before materialization. Whole events exceeding the remaining UTF-8 budget are omitted and `TraceEventBatch.Truncated` is set; `RootExecutionId` survives independently when an oversized first row cannot be included.
|
||||
|
||||
|
||||
### Global instance identity and protocol credential upgrade
|
||||
|
||||
Instance UniqueName is globally unique in SQL Server, matching name-based routing and lookup.
|
||||
The migration checks for existing duplicates before creating the unique index and fails with
|
||||
an explicit inventory/remediation message. It does not select a winning site or rename deployed
|
||||
instances. Concurrent creation/import conflicts become InstanceNameConflictException and a
|
||||
curated domain failure. SiteId retains a separate lookup index.
|
||||
|
||||
DataConnection.PrimaryConfiguration and BackupConfiguration are protected in full with a
|
||||
purpose-scoped Data Protection converter, using a versioned dp:protocol:v1: prefix. Both SQL
|
||||
columns are nvarchar(max) to accommodate ciphertext expansion. Schema-only contexts reject
|
||||
non-null protocol configuration writes without a real key ring.
|
||||
|
||||
After schema migration/validation and before repositories are used, DataConnectionSecretUpgrade
|
||||
reads legacy rows in batches of 256 and protects both columns with an atomic, binary-comparison
|
||||
compare-and-swap update. Concurrent central startup is safe; reruns validate existing ciphertext
|
||||
without double encryption. No plaintext fallback exists on normal entity reads. Protection failure
|
||||
prevents startup rather than silently using plaintext.
|
||||
|
||||
Upgrade central nodes together: old binaries cannot read the new protected values. SQL migration
|
||||
alone does not encrypt data; the first updated host performs the data upgrade with the shared
|
||||
persisted key ring. Keep a coordinated pre-upgrade database/key-ring backup. Downgrade is rejected
|
||||
while protected protocol rows exist. The existing same-database key-ring storage remains
|
||||
defence in depth; this change does not introduce external key management.
|
||||
|
||||
Trace replay reads keyset pages of eight rows, so EF retry buffering cannot materialize a whole trace before the consumer applies its byte budget. SQL omits an oversized DetailsJson value before transfer; all other projected strings have schema length limits. Canonical serialized UTF-8 size, including escaping and metadata, determines the final 1 MiB limit.
|
||||
|
||||
@@ -319,8 +319,8 @@ The registered `SourceReference` is a **NodeId** for OPC UA (the picker, CSV and
|
||||
- **Capability check**: if `_adapter is not IAlarmSubscribableConnection`, the actor replies `SubscribeAlarmsResponse(Success = false, ...)`.
|
||||
- **Reconnect handling**: on entering **Reconnecting**, the actor pushes a `NativeAlarmSourceUnavailable` to every alarm subscriber (consumers mark mirrored alarms uncertain rather than clearing them). On successful reconnection it re-subscribes the feed; the adapter re-emits a snapshot, reconciling state.
|
||||
- **Shared condition filter (last-subscriber-wins)**: because the feed is opened once per source, it carries a single condition filter. A second subscriber that registers a *different* filter for the same source overwrites it (last writer wins) and the actor logs a warning — co-subscribers to one source are expected to agree on the filter.
|
||||
- **Routing index**: subscribed sources are bucketed by the **first path segment** of their reference (everything before the first `.`), and a transition is matched only against the bucket for its own first segment, plus a residue list of sources that carry no separator at all (a prefix shorter than one segment, which can match other buckets). This is sound because a source reference containing a separator can only prefix a reference sharing its entire first segment. The per-source `StartsWith` test and the condition-type gate inside the bucket are unchanged, so routing decisions are identical to the previous linear scan over every subscribed source; snapshot boundaries bypass the condition filter, but OPC UA boundaries are sent only to the exact source binding. Only legacy connection-wide Mx boundaries broadcast to all subscribers.
|
||||
- **Gateway union filter (MxGateway)**: `StreamAlarms` carries a single `alarm_filter_prefix`, so the adapter opens the stream on the **longest common prefix** of the currently subscribed source references and restarts it only when a NEW source falls outside that prefix (the source then replays a fresh snapshot, which `NativeAlarmActor` already handles). An unsubscribe never restarts the stream — the prefix it leaves behind is at worst too broad, which costs bandwidth, not correctness. The prefix is a bandwidth optimisation only; the actor's per-source + condition-type gate remains authoritative, mirroring the OPC UA server-side WhereClause stance.
|
||||
- **Routing index**: subscribed sources are bucketed by the **first path segment** of their reference (everything before the first `.`), and a transition is matched only against the bucket for its own first segment, plus a residue list of sources that carry no separator at all (a prefix shorter than one segment, which can match other buckets). This is sound because a source reference containing a separator can only prefix a reference sharing its entire first segment. The per-source `StartsWith` test and the condition-type gate inside the bucket are unchanged, so routing decisions are identical to the previous linear scan over every subscribed source; snapshot boundaries bypass the condition filter, but OPC UA boundaries are sent only to the exact source binding. Connection-wide Mx boundaries broadcast to all subscribers while preserving their refresh epoch.
|
||||
- **Gateway union filter (MxGateway)**: `StreamAlarms` carries a single `alarm_filter_prefix`, so the adapter opens the stream on the **longest common prefix** of the currently subscribed source references and restarts it for each new source, including one covered by the existing prefix, so every newly bound source receives standing state. Each transport refresh has connection-wide start/end markers and a distinct epoch. An unsubscribe never restarts the stream — the prefix it leaves behind is at worst too broad, which costs bandwidth, not correctness. The prefix is a bandwidth optimisation only; the actor's per-source + condition-type gate remains authoritative, mirroring the OPC UA server-side WhereClause stance.
|
||||
|
||||
### Protocol-Neutral Types & Messages
|
||||
|
||||
@@ -459,3 +459,10 @@ The DCL reports the following metrics to the Health Monitoring component via the
|
||||
- **Site Runtime (NativeAlarmActor)**: Bidirectional — receives alarm subscribe/unsubscribe requests, delivers native alarm transitions and source-unavailable notifications (read-only; no ack-back to the source).
|
||||
- **Health Monitoring**: Reports connection health periodically.
|
||||
- **Site Event Logging**: Logs connection/disconnection events.
|
||||
|
||||
|
||||
### Medium architecture review connection guarantees (2026-09-09)
|
||||
|
||||
OPC UA connection and endpoint verification select exactly the configured security mode. If a protected mode is unavailable, or discovery fails while protection was requested, the operation fails with the requested mode in its diagnostic; it never falls back to an unsecured session.
|
||||
|
||||
DCL retains each subscribed alarm source's current standing state and its in-progress snapshot. A new subscriber sharing a feed receives a scoped replay, or the already-received prefix of an in-progress frame followed by the remaining frame. Cache state is discarded on reconnect and last-owner unsubscribe. A new Mx source reopens the shared stream even when the existing prefix covers it, obtaining a fresh standing-state snapshot; callbacks from the canceled stream are ignored. Mx refresh boundaries remain connection-wide but share a distinct per-stream epoch. Both replay cache and runtime mirror ignore stale-epoch rows and apply newer live raises/clears over a concurrent refresh before committing it.
|
||||
|
||||
@@ -180,8 +180,7 @@ to re-enable", and the single-instance deploy remains the explicit re-enable pat
|
||||
**Reconciliation is bounded per batch.** Phase 1 is serial, so an unreachable site
|
||||
would otherwise cost the batch one full communication-layer `QueryTimeout` (30 s)
|
||||
per instance before a single deploy was sent — hours for a large site, past the
|
||||
CLI's 5-minute bulk deadline, which cancels and re-arms the identical cost on the
|
||||
next attempt. Each reconcile in a bulk deploy therefore runs under the short
|
||||
server's bulk scheduling deadline. Repeated client timeouts must not re-arm the identical cost on another attempt. Each reconcile in a bulk deploy therefore runs under the short
|
||||
`SiteDeploymentReconcileTimeout` (default **5 s**), and the FIRST one to hit that
|
||||
deadline opens a once-per-batch circuit-breaker that skips the remaining
|
||||
reconciles (one summary log line, not N). Skipping costs nothing: a timed-out
|
||||
@@ -319,3 +318,9 @@ The Deployment Manager sends the following commands to sites via the Communicati
|
||||
- **Site Runtime**: Receives and applies configurations and lifecycle commands.
|
||||
- **Health Monitoring**: Deployment failures contribute to site health status.
|
||||
- **Transport (#24)**: Does not invoke Deployment Manager directly. Transport-driven template changes propagate to deployed instances through the existing revision-hash drift detection in `DeploymentService.CompareAsync`; the Deployments page surfaces affected instances as stale automatically.
|
||||
|
||||
### Authoritative staleness and bulk deadlines
|
||||
|
||||
`StaleInstanceProbe` no longer memoizes revision hashes against process-local watermarks. Every probe starts a fresh flatten session, so a configuration edit through another central is visible to the next deployment skip decision. This trades extra flattening work for correctness across shared-database writers.
|
||||
|
||||
Management bulk deployment uses `LongRunningCommandTimeout` as an overall scheduling/send deadline. Cancellation stops scheduling more instances, fails unfinished eligible rows, and releases prepared operation locks. Completed and deliberately skipped outcomes remain in the response. HTTP Ask reserves an additional60 seconds for cleanup and response; the bulk CLI defers its timeout to this server-owned deadline.
|
||||
|
||||
@@ -74,9 +74,9 @@ Each API method definition includes:
|
||||
|
||||
> **Authoring note — idempotent methods for at-least-once callers**: some external callers retry
|
||||
> on their own connect-failure/timeout failover loop and cannot tell "the first attempt actually
|
||||
> succeeded" from "the first attempt was lost" — e.g. the `DelmiaRecipeDownload` method backing
|
||||
> `DelmiaNotifier` (see `docs/plans/2026-06-26-delmia-recipe-notifier-design.md`, "Delivery
|
||||
> semantics: at-least-once"). Implementation scripts for such methods must be idempotent on their
|
||||
> succeeded" from "the first attempt was lost". The updated `DelmiaNotifier` returns an uncertain
|
||||
> result instead of automatically replaying such attempts. Scripts exposed to other retrying
|
||||
> clients should still be idempotent on their
|
||||
> natural key so a duplicate delivery returns the same result without double-applying side effects.
|
||||
|
||||
### Management
|
||||
@@ -148,7 +148,8 @@ The `sbk_<keyId>_<secret>` design already supports zero-downtime rotation — a
|
||||
| `VALIDATION_FAILED` | 400 | Parameter validation failed (missing required field, wrong type, undeclared field). |
|
||||
| `BODY_TOO_LARGE` | 413 | Request body exceeded `MaxRequestBodyBytes`. |
|
||||
| `STANDBY_NODE` | 503 | Request hit a standby central node that could not relay it: no `StandbyForwardPeerEndpoints` configured, or the request was already relayed once (loop guard — neither node is active, e.g. mid-failover). Carries the `X-ScadaBridge-Standby: 1` response header. |
|
||||
| `ACTIVE_NODE_UNREACHABLE` | 503 | A standby tried every configured peer and none served the relay (unreachable, timed out after `StandbyForwardTimeout`, or also standby). |
|
||||
| `ACTIVE_NODE_UNREACHABLE` | 503 | Every configured peer refused before execution or failed with proven name-resolution/connection-refusal errors. |
|
||||
| `OUTCOME_UNKNOWN` | 502 | A relay timed out or lost its response after possible execution; it does not replay the request. Verify the operation before retrying. |
|
||||
| `TIMEOUT` | 500 | The method script exceeded its execution timeout. |
|
||||
| `SITE_UNREACHABLE` | 500 | The target site did not respond to the routed request within the routing timeout (no contact / down / partitioned) — typed from the routed Ask expiring (`AskTimeoutException`), never inferred from an error-message substring. A `Success=false` response means the site answered and is classified `SCRIPT_ERROR`. |
|
||||
| `SCRIPT_COMPILE_FAILED` | 500 | The method's script failed to compile (returned consistently on every node until a compiling version is saved). |
|
||||
@@ -362,10 +363,10 @@ Both central nodes expose `POST /api/{methodName}`, but only the active node (th
|
||||
|
||||
A standby now **relays** the request to the active peer instead (`InboundApiEndpointFilter` → `IStandbyRequestForwarder` / `HttpStandbyRequestForwarder`):
|
||||
|
||||
- **Configuration.** `ScadaBridge:InboundApi:StandbyForwardPeerEndpoints` — the HTTP base URL(s) of the *other* central node(s), e.g. `["http://scadabridge-central-b:5000"]` on central-a. Empty (default) keeps the old 503 behaviour. `StandbyForwardTimeout` (default 60 s) is the per-peer attempt deadline and must exceed the longest method timeout in use. Validated at startup (absolute http/https URLs, positive timeout). Note the standby discovers the active peer from **appsettings**, not from cluster state — the Akka member address carries the remoting port, not the HTTP port — so this is a static, restart-to-change list, like `CentralGrpcEndpoints` on sites.
|
||||
- **Pure relay.** Method, path, query string, body and headers travel verbatim (hop-by-hop headers and `Host` excepted), so the **active node** authenticates the caller's own `Authorization`/`X-API-Key` credential, applies its own body cap and method timeout, and writes the audit row. The peer's status, `Content-Type` and body are replayed unchanged. The standby's `AuditWriteMiddleware` skips its own `ApiInbound` row for a relayed request (`HttpContext.Items[RelayedToActiveItemKey]`) so a call is audited once, where it ran.
|
||||
- **Order of checks.** The body-size cap (413 `BODY_TOO_LARGE`) runs *before* the active-node check, so a standby never buffers or relays a body the active node would refuse. The body is buffered once on the standby (bounded by `MaxRequestBodyBytes`) so it can be re-sent to a second peer.
|
||||
- **Peer selection.** The standby knows only that it is *not* active. It tries the peers in order and accepts the first response that is not a standby refusal; a peer answering 503 with `X-ScadaBridge-Standby: 1`, a connection failure, or a `StandbyForwardTimeout` expiry each move to the next peer. A 503 *without* the marker is the active node's own answer and is replayed, not retried. All peers failing → 503 `ACTIVE_NODE_UNREACHABLE`.
|
||||
- **Configuration.** `ScadaBridge:InboundApi:StandbyForwardPeerEndpoints` — the HTTP base URL(s) of the *other* central node(s), e.g. `["http://scadabridge-central-b:5000"]` on central-a. Empty (default) keeps the old 503 behaviour. `StandbyForwardTimeout` (default 60 s) is the total relay budget and should exceed the longest method timeout plus network overhead. Validated at startup (absolute http/https URLs, positive timeout). Note the standby discovers the active peer from **appsettings**, not from cluster state — the Akka member address carries the remoting port, not the HTTP port — the options snapshot is read per relay.
|
||||
- **Pure relay.** Method, path, query string, body and credentials travel unchanged; hop-by-hop/Host/forwarding-provenance headers are excluded and authenticated provenance is rebuilt, so the **active node** authenticates the caller's own `Authorization`/`X-API-Key` credential, applies its own body cap and method timeout, and writes the audit row. The peer's status, `Content-Type` and body are replayed unchanged. The serving node owns its `ApiInbound` row; the originating standby retains a failure row when no peer serves or the outcome is uncertain. Verified internal standby refusals are not duplicated.
|
||||
- **Order of checks.** The body-size cap (HTTP 413) runs before audit reads and the active-node check, so a standby never buffers or relays a body the active node would refuse. The body is buffered once on the standby (bounded by `MaxRequestBodyBytes`) so it can be re-sent to a second peer.
|
||||
- **Peer selection.** The standby knows only that it is *not* active. It tries the peers in order and accepts the first response that is not a standby refusal; a peer answering 503 with `X-ScadaBridge-Standby: 1`, proven name-resolution failure, or connection refusal permits trying the next peer. Timeout or ambiguous response loss returns 502 `OUTCOME_UNKNOWN` without replay. A 503 *without* the marker is the active node's own answer and is replayed, not retried. All peers failing → 503 `ACTIVE_NODE_UNREACHABLE`.
|
||||
- **One-hop loop guard.** Every relayed request carries `X-ScadaBridge-Forwarded: <relaying host>`. A standby never relays a request that already carries it — it answers 503 `STANDBY_NODE` with the marker header — so a pair in which neither node is active (a transient during failover) bounces at most one hop and never ping-pongs. The active node ignores the header and serves the request normally.
|
||||
- **Still not a retry layer.** The relay makes one HTTP attempt per peer and does not retry a peer that answered; at-least-once callers keep their own retry loop and the idempotent-method authoring note above still applies.
|
||||
|
||||
@@ -394,3 +395,15 @@ A standby now **relays** the request to the active peer instead (`InboundApiEndp
|
||||
- **Site Runtime (Instance Actors, Script Actors)**: Routed calls execute on site Instance Actors via their Script Actors.
|
||||
- **Central UI**: Administrator manages API keys; Designer manages method definitions.
|
||||
- **Configuration Database (via IAuditService)**: Configuration changes are audited.
|
||||
|
||||
### Relay audit ownership, privacy, and request limits
|
||||
|
||||
The inbound body limit is installed before audit buffering. Declared oversized bodies and unknown-length HTTP/1.1 or HTTP/2 bodies are rejected with HTTP 413; enabling audit capture cannot relax the route limit. The relay also bounds its buffered copy explicitly.
|
||||
|
||||
A serving peer owns the inbound audit for its success or method failure. The originating standby writes a correlated failure if no peer serves the request, including an uncertain lost response. Authenticated internal standby refusals do not create duplicate inbound rows. Audit writes remain best effort; ownership is not a durable audit receipt.
|
||||
|
||||
Relays overwrite caller-supplied provenance with an HMAC signed envelope using a domain-separated key derived from `ApiKeyPepper`. Both centrals must share that server secret. The envelope binds the original verified remote address, execution ID, timestamp, method, path/query, content type, caller credentials, and bounded body digest. The receiver accepts it for at most five minutes and only when the request binding matches. Unsigned loop markers and forwarded-address headers confer no identity trust; missing or invalid proof uses the receiver's verified remote address. Forwarding remains one hop. Use TLS between peers when network confidentiality is required.
|
||||
|
||||
`StandbyForwardTimeout` is one budget across permitted attempts and response consumption. Only explicit pre-execution standby rejection, name-resolution failure, or connection refusal permits another peer. Timeout or response loss returns HTTP 502 `OUTCOME_UNKNOWN`: verify the operation before manually retrying because it may already have applied. This avoids blindly replaying non-idempotent methods.
|
||||
|
||||
`SkipBodyCapture` suppresses request/response summaries and all exception message/detail text, including stashed script exceptions; retain status/category and request metadata for diagnosis. Error text can quote payload values and is subject to the same privacy boundary as the body.
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
## Purpose
|
||||
|
||||
The Management Service is an Akka.NET actor on the central cluster that provides programmatic access to all administrative operations. It exposes the same capabilities as the Central UI but through an actor-based interface, enabling the CLI (and potentially other tooling) to interact with the system without going through the web UI. The ManagementActor registers with ClusterClientReceptionist for cross-cluster access and is also exposed via an HTTP Management API endpoint (`POST /management`) for external tools like the CLI.
|
||||
The Management Service is an Akka.NET actor on the central cluster that provides programmatic access to all administrative operations. It exposes the same capabilities as the Central UI but through an actor-based interface, enabling the CLI (and potentially other tooling) to interact with the system without going through the web UI. The ManagementActor is reached in-process through `ManagementActorHolder` and is exposed via an authenticated HTTP Management API endpoint (`POST /management`) for external tools like the CLI.
|
||||
|
||||
## Location
|
||||
|
||||
@@ -13,18 +13,18 @@ Central cluster only. The ManagementActor runs as a plain actor on **every** cen
|
||||
## Responsibilities
|
||||
|
||||
- Provide an actor-based interface to all administrative operations available in the Central UI.
|
||||
- Register with Akka.NET ClusterClientReceptionist so external tools (CLI) can discover and communicate with it via ClusterClient.
|
||||
- Serve external tools through the authenticated HTTP Management API; actor references remain internal to the central host.
|
||||
- Expose an HTTP API endpoint (`POST /management`) that accepts JSON commands with Basic Auth, performs LDAP authentication and role resolution, and dispatches to the ManagementActor.
|
||||
- Validate and authorize all incoming commands using the authenticated user identity carried in message envelopes.
|
||||
- Delegate to the appropriate services and repositories for each operation.
|
||||
- Return structured response messages for all commands and queries.
|
||||
- Failover: The ManagementActor runs on all central nodes, so no actor-level failover is needed. If one node goes down, the ClusterClient transparently routes to the ManagementActor on the remaining node.
|
||||
- Failover: The ManagementActor runs on all central nodes, so no actor-level failover is needed. Clients address the HTTP management endpoint through the configured active-node proxy. Site-bound commands use the gRPC pair transport and its explicit retry-safety contract.
|
||||
|
||||
## Key Classes
|
||||
|
||||
### ManagementActor
|
||||
|
||||
The central actor that receives and processes all management commands. Created at the well-known actor path (`/user/management`) and handed to `ManagementActorHolder`, which is how every in-process caller (notably `ManagementEndpoints`) reaches it. It is **not** advertised via ClusterClientReceptionist — see the note under "ManagementActor and ClusterClient" in Component-Communication.
|
||||
The central actor that receives and processes all management commands. Created at the well-known actor path (`/user/management`) and handed to `ManagementActorHolder`, which is how every in-process caller (notably `ManagementEndpoints`) reaches it. There is no receptionist registration or cross-cluster actor discovery.
|
||||
|
||||
### ManagementEndpoints
|
||||
|
||||
@@ -45,10 +45,6 @@ All request/response messages are defined in **Commons** under `Messages/Managem
|
||||
- **CorrelationId**: Application-level correlation ID for request/response pairing.
|
||||
- **AuthenticatedUser**: The identity and roles of the user issuing the command (username, display name, roles, permitted sites). This is populated by the CLI from the user's authenticated session.
|
||||
|
||||
## ClusterClientReceptionist Registration
|
||||
|
||||
The ManagementActor registers itself with `ClusterClientReceptionist` at startup. This allows external processes using `ClusterClient` to send messages to the ManagementActor without joining the Akka.NET cluster as a full member. The receptionist advertises the actor under its well-known path.
|
||||
|
||||
## HTTP Management API
|
||||
|
||||
The Management Service also exposes a `POST /management` endpoint on the Central Host's web server. This provides an HTTP interface to the same ManagementActor, enabling the CLI (and other HTTP clients) to interact without Akka.NET dependencies.
|
||||
@@ -298,13 +294,13 @@ The ManagementActor receives the following services and repositories via DI (inj
|
||||
- **Configuration Database (via IAuditService)**: All mutating operations are audit logged through the existing transactional audit mechanism.
|
||||
- **Communication Layer**: Deployment commands and remote queries (parked messages, event logs) are routed to sites via Communication.
|
||||
- **Security & Auth**: Authorization rules are enforced on every command using the authenticated user identity from the message envelope.
|
||||
- **Cluster Infrastructure**: ManagementActor runs on all central nodes; ClusterClientReceptionist requires cluster membership.
|
||||
- **Cluster Infrastructure**: ManagementActor runs on all central nodes within the shared central ActorSystem.
|
||||
- **All service components**: The ManagementActor delegates to the same services used by the Central UI — Template Engine, Deployment Manager, etc.
|
||||
|
||||
## Interactions
|
||||
|
||||
- **CLI**: The primary consumer. Connects via the HTTP Management API (`POST /management`) and sends commands as JSON with Basic Auth credentials.
|
||||
- **Host**: Registers the ManagementActor and ClusterClientReceptionist on central nodes during startup.
|
||||
- **Host**: Creates the ManagementActor and its holder, and maps the authenticated HTTP endpoint on central nodes during startup.
|
||||
- **Central UI**: Shares the same underlying services and repositories. The ManagementActor and Central UI are parallel interfaces to the same operations.
|
||||
- **Communication Layer**: Deployment commands and remote site queries flow through communication actors.
|
||||
- **Configuration Database (via IAuditService)**: All configuration changes are audited.
|
||||
@@ -313,3 +309,7 @@ The ManagementActor receives the following services and repositories via DI (inj
|
||||
### Atomic configuration command boundary (2026-09-09)
|
||||
|
||||
Database-only configuration commands may execute through `ExecuteConfigurationCommandAsync` from the actor or Central UI. The boundary requires role authorization and `IConfigurationCommandTransaction`; missing transaction configuration fails closed. Repository mutations may flush generated keys within the transaction; a final flush persists the audit before commit. Existing service-owned audits are retained. Site cache refresh and inbound API handler publication run only after a successful commit.
|
||||
|
||||
### Scoped and bounded execution-chain reads
|
||||
|
||||
Audit tree HTTP reads and Flow Debugger Replay pass the authenticated caller's permitted source-site set into the repository. Administrator and system-wide grants omit the filter; scoped reads retain central rows with null SourceSiteId while excluding other sites from ancestors, descendants and aggregates. Replay retains root identity separately from payload rows and propagates truncation truthfully. It returns at most500 events and reserves envelope overhead within a1MiB UTF-8 budget. The Central UI direct tree-query facade applies the same current-scope policy.
|
||||
|
||||
@@ -28,7 +28,7 @@ SMTP and HTTP delivery is blocking I/O. A dispatch sweep runs **off the actor th
|
||||
%%{init: {'theme':'base', 'themeVariables': {'textColor':'#111111','lineColor':'#555555','edgeLabelBackground':'#ffffff','fontSize':'15px'}}}%%
|
||||
flowchart TD
|
||||
SCRIPT(["Site script: Notify.To('list').Send(subject, body)<br/>generate NotificationId (GUID) locally;<br/>return it to the script immediately"])
|
||||
SNF["Site Store-and-Forward Engine<br/>(notification category, target = central)<br/>durably forwards to central via Central-Site Communication<br/>(ClusterClient); buffers/retries if central is unreachable"]
|
||||
SNF["Site Store-and-Forward Engine<br/>(notification category, target = central)<br/>durably forwards to central via Central-Site Communication<br/>(authenticated CentralControlService gRPC); buffers/retries if central is unreachable"]
|
||||
INGEST[("Central ingest: insert-if-not-exists on NotificationId<br/>to Notifications table (Pending)<br/>ack the site, site S and F clears the message")]
|
||||
OUTBOX["Central Notification Outbox actor<br/>(singleton, active central node)<br/>polls due rows; resolves the list;<br/>delivers via the matching adapter"]
|
||||
D1{Delivery outcome}
|
||||
|
||||
@@ -196,7 +196,7 @@ All five call sites that previously maintained their own script trust enforcemen
|
||||
| **Central UI** `ScriptAnalysisService` | Semantic + full compile, lenient threading | Delegates forbidden-API verdict and sources editor-marker deny-list from `ScriptTrustPolicy`; retains Test-Run execution host |
|
||||
| **Transport** `BundleImporter.RunSemanticValidationAsync` / `DetectBlockersAsync` | No trust gate — a bundle's forbidden-API scripts imported clean, deferred to runtime (fifth write path) | Runs `ScriptTrustValidator.FindViolations` over every non-Skip template / shared / ApiMethod body, template script + alarm Expression-trigger bodies, **and instance alarm-override trigger expressions** at import review — hard error (apply) / Blocker row (preview) for all kinds |
|
||||
|
||||
The static enforcement is **defence-in-depth**, not a true runtime sandbox. Scripts execute in-process; the denied API list prevents obvious escapes at compile time but does not provide the isolation guarantees of an out-of-process sandbox or a restricted `AssemblyLoadContext`. This caveat applies to all consumers.
|
||||
The static enforcement is **defence-in-depth**, not a true runtime sandbox. The denied API list blocks disallowed APIs at compile time but does not establish an operating-system security boundary. Site-runtime consumers execute in-process. Central UI Test Run uses a bounded, killable worker process; that adds resource/lifetime containment, while the worker retains the same operating-system identity and permitted gateway access.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -213,3 +213,5 @@ Each event entry contains:
|
||||
- **Central UI**: Site Event Log Viewer displays queried events.
|
||||
- **Store-and-Forward Engine**: Its notification path (the site→central forward of script-generated notifications) reports forward failures and long-buffered notifications as Notification-category events. Routine enqueue and forward-success events are deliberately not logged — central's authoritative `Notifications` table (owned by the Notification Outbox component) is the audit record of record; site-side logging covers only the in-transit blind spot when central is unreachable.
|
||||
- **Health Monitoring**: Script error rates and alarm evaluation error rates can be derived from event log data; this component also reports `SiteEventLogWriteFailures`, and its purge is the expected cause of the transient `LocalDbOplogBacklog` spikes described in [Component-HealthMonitoring.md](Component-HealthMonitoring.md) → Monitored Metrics.
|
||||
|
||||
Rig settings no longer advertise PurgeScheduleCron. Retention uses PurgeInterval after the previous purge completes, so it is interval-based rather than a fixed wall-clock schedule.
|
||||
|
||||
@@ -506,7 +506,7 @@ Available to all script runs and alarm on-trigger runs:
|
||||
|
||||
### Operation Tracking
|
||||
- `Tracking.Status("trackedOperationId")` — Returns a status record (status, retry count, last error, key timestamps) for any tracked operation: a cached external system call, a cached database write, or a notification. For cached calls and writes the answer is always site-local and authoritative — the site owns the operation tracking table. (`Notify.Status(...)` is a thin alias scoped to the notification domain.)
|
||||
- **Cached-call telemetry is off the script thread (2026-09-06).** A `Database.CachedWrite` / `ExternalSystem.CachedCall` emits three `CachedCallTelemetry` packets (submit, attempted, resolve) through `CachedCallTelemetryForwarder`, each an audit row plus a tracking row. Both halves used to be awaited to commit on the script thread; the audit half was the real cost — `SqliteAuditWriter.WriteAsync` returns the batch completion, and the writer coalesces for `FlushIntervalMs` (50 ms) before committing — so a cached call paid ~160 ms of a 176 ms run for bookkeeping (measured on the rig; now 17 ms). The forwarder now **enqueues the audit row and observes its completion on a fault-logging continuation** (the same pattern every other script-path audit emission already used), and `OperationTrackingStore` is a **single-writer queue**: `RecordEnqueue/Attempt/TerminalAsync` return once the row is *queued*, and one background pump applies each drained batch in one SQLite transaction. Preserved: FIFO order across the three lifecycle rows; **read-your-writes** — `GetStatusAsync` (behind `Tracking.Status`, and the telemetry actor's status join) and `ReadChangedSinceAsync` (central's `PullSiteCalls`) wait for every write queued before them; timestamps are stamped when recorded, not when applied; `PurgeTerminalAsync` returns after the purge lands; `DisposeAsync` drains the queue (the sync `Dispose()` does not — host shutdown uses the async path). Accepted: a row queued but unapplied is lost on a hard crash (milliseconds), leaving the tracking row at its previous state until reconciled; the S&F buffer and audit rows are unaffected. The queue is bounded (8 192) with wait backpressure — it never drops. `FlushAsync()` is the explicit barrier for callers that go to the file directly. **Trap for new audit emitters on a script path:** never `await` `IAuditWriter.WriteAsync` there — it waits for the writer's flush, not the enqueue. Design + measurements: `docs/plans/2026-09-06-tracking-store-async-writes-design.md`.
|
||||
- **Cached-call telemetry is off the script thread (2026-09-06).** A `Database.CachedWrite` / `ExternalSystem.CachedCall` emits three `CachedCallTelemetry` packets (submit, attempted, resolve) through `CachedCallTelemetryForwarder`, each an audit row plus a tracking row. Both halves used to be awaited to commit on the script thread; the audit half was the real cost — `SqliteAuditWriter.WriteAsync` returns the batch completion, and the writer coalesces for `FlushIntervalMs` (50 ms) before committing — so a cached call paid ~160 ms of a 176 ms run for bookkeeping (measured on the rig; now 17 ms). The forwarder now **enqueues the audit row and observes its completion on a fault-logging continuation** (the same pattern every other script-path audit emission already used), and `OperationTrackingStore` is a **single-writer queue**: `RecordEnqueue/Attempt/TerminalAsync` return once the row is *queued*, and one background pump applies each drained batch in one SQLite transaction. Preserved: FIFO order across the three lifecycle rows; **read-your-writes** — `GetStatusAsync` (behind `Tracking.Status`, and the telemetry actor's status join) and `ReadChangedSinceAsync` (central's `PullSiteCalls`) wait for every write queued before them; timestamps are stamped when recorded, not when applied; `PurgeTerminalAsync` returns after the purge lands; Both `DisposeAsync` and sync `Dispose()` drain the queue before closing the writer; host shutdown uses the async path. Accepted: a row queued but unapplied is lost on a hard crash (milliseconds), leaving the tracking row at its previous state until reconciled; the S&F buffer and audit rows are unaffected. The queue is bounded (8 192) with wait backpressure — it never drops. `FlushAsync()` is the explicit barrier for callers that go to the file directly. **Trap for new audit emitters on a script path:** never `await` `IAuditWriter.WriteAsync` there — it waits for the writer's flush, not the enqueue. Design + measurements: `docs/plans/2026-09-06-tracking-store-async-writes-design.md`.
|
||||
|
||||
### Alarm State (read-only)
|
||||
- `Alarms.CurrentAsync()` — Returns a snapshot of the instance's **current** alarm conditions as `IReadOnlyList<ScriptAlarm>` (`Commons.Types.Scripts`), covering computed alarms and mirrored native (OPC UA A&C / MxAccess Gateway) conditions alike. Served by a **local Ask** (`GetAlarmSnapshotRequest`/`GetAlarmSnapshotResponse`) to the script's own Instance Actor — the same mechanism attribute reads use, never a cross-cluster hop.
|
||||
@@ -637,3 +637,12 @@ Per Akka.NET best practices, internal actor communication uses **Tell** (fire-an
|
||||
- **Communication Layer**: Receives commands from central, sends deployment results, serves debug view data.
|
||||
- **Store-and-Forward Engine**: Scripts route cached writes, notifications, and external system calls here.
|
||||
- **Health Monitoring**: Periodically reports error rate metrics.
|
||||
|
||||
|
||||
### Medium architecture review lifecycle guarantees (2026-09-09)
|
||||
|
||||
- Alarm on-trigger runs resolve the same scoped integration services as ordinary scripts, including external systems, databases, notifications, operation tracking, auditing, and site/node provenance. The scope remains alive for the run and is disposed on every exit.
|
||||
- Native alarm subscription retries retain the current correlation ID until its response deadline, reject stale replies, and back off to a bounded interval. Rehydration cannot overwrite a condition touched by a live event, including deletion, or a completed authoritative snapshot.
|
||||
- Audited raw ADO connections expose no connection string and reject `ConnectionString` assignment and `ChangeDatabase`; their configured database name remains the audit target.
|
||||
- Static override persistence runs in order off the actor dispatcher. Redeployment waits for both predecessor termination and successful final persistence acknowledgement before clearing old state; failed persistence fails the redeployment. At most four redeployment predecessor stops are in progress per deployment manager. Delete and disable retain the same-name stop barrier until the final flush and lifecycle persistence finish; an overlapping deploy/enable is rejected with a retryable busy diagnostic. Timer flushes keep coalescing while a previous SQLite write is pending.
|
||||
- Synchronous operation-tracking disposal drains and joins its thread-pool writer before disposing SQLite resources, including when invoked under a synchronization context.
|
||||
|
||||
@@ -167,3 +167,6 @@ Each buffered message stores:
|
||||
- **Notification Outbox**: The central destination for the notification category — central ingests each forwarded notification into the `Notifications` table and acks the site, on which the engine clears the buffered message.
|
||||
- **Site Call Audit**: The central observability sibling for the cached-call categories — this engine emits `CachedCallTelemetry` on every tracking-table transition, answers `CachedCallReconcileRequest` pulls, and executes the `RetryParkedOperation` / `DiscardParkedOperation` commands it relays.
|
||||
- **Health Monitoring**: Reports buffer depth metrics, including the notification backlog covering the site→central forward leg.
|
||||
|
||||
|
||||
The active-node gate is checked again before each new retry attempt, including after waiting for a target lane. A false or throwing gate abandons the remaining lane without changing retry counters. A delivery already sent at demotion can still finish; downstream idempotency remains necessary for ambiguous outcomes.
|
||||
|
||||
@@ -513,3 +513,12 @@ The `manifest.json` file is always present in the ZIP root and is never encrypte
|
||||
| `encryption.ivB64` | Base64-encoded 12-byte (GCM standard) random IV generated at export time. |
|
||||
| `summary` | Artifact count by type, for display in the import wizard's upload step without needing to decrypt content. Schema 1.1 adds `sites` / `dataConnections` / `instances` counts. |
|
||||
| `contents` | Ordered list of all artifacts in the bundle. Order is topological (base templates before derived; sites before their connections and instances). Each entry carries the artifact's name (connections use the site-qualified `{SiteIdentifier}/{Name}`), the schema `version` at export time, and its direct `dependsOn` edges for dependency display in the export wizard's Step 2. T18 adds the `Site`, `DataConnection`, and `Instance` entry `type`s. |
|
||||
|
||||
|
||||
### Instance ownership during overwrite
|
||||
|
||||
An instance Overwrite may change SiteId only when the existing instance is NotDeployed.
|
||||
Other states require an explicit undeploy first; import rejects the operation before replacing
|
||||
the instance fields/children and rolls back its database transaction. Same-site overwrite retains
|
||||
the existing behavior. Rename targets are checked for global name collisions, and the database
|
||||
global unique index also rejects concurrent add/rename collisions as domain conflicts.
|
||||
|
||||
@@ -46,10 +46,10 @@
|
||||
|
||||
### 2.2 Communication: Central ↔ Site
|
||||
- Two transport layers are used for central-site communication:
|
||||
- **Akka.NET ClusterClient/ClusterClientReceptionist**: Handles **command/control** messaging — deployments, instance lifecycle commands, subscribe/unsubscribe handshake, debug snapshots, health reports, remote queries, and integration routing. Provides automatic failover between contact points.
|
||||
- **gRPC server-streaming (site→central)**: Handles **real-time data streaming** — attribute value updates and alarm state changes. Each site node hosts a **SiteStreamGrpcServer** on a dedicated HTTP/2 port (Kestrel, default port 8083). Central creates per-site **SiteStreamGrpcClient** instances to subscribe to site streams. gRPC provides HTTP/2 flow control and per-stream backpressure that ClusterClient lacks.
|
||||
- **Site addressing**: Site Akka base addresses (NodeA and NodeB) and gRPC endpoints (GrpcNodeAAddress and GrpcNodeBAddress) are stored in the **Sites database table** and configured via the Central UI or CLI. Central creates a ClusterClient per site using both Akka addresses as contact points, and per-site gRPC clients using the gRPC addresses.
|
||||
- **Central contact points**: Sites configure **multiple central contact points** (both central node addresses) for redundancy. ClusterClient handles failover between central nodes automatically.
|
||||
- **Authenticated gRPC command/control**: `SiteCommandService` handles central-to-site lifecycle, snapshots, queries and integration routing. `CentralControlService` handles site-to-central notifications, telemetry, health and reconciliation. Per-site credentials bind the authenticated site to payload ownership. Deployment configuration is fetched over token-gated HTTP.
|
||||
- **gRPC server-streaming (site→central)**: Each site hosts `SiteStreamService` through `SiteStreamGrpcServer` on its HTTP/2 listener. Central subscribes through `SiteStreamGrpcClient` for attribute updates and computed/native alarm state, with flow control and bounded batching.
|
||||
- **Site addressing**: The Sites table stores node Akka addresses for cluster identity and gRPC endpoints for application transport. `SitePairChannelProvider` manages gRPC pair selection. Mutations are not replayed after ambiguous response loss; only explicit pre-dispatch rejection permits retry.
|
||||
- **Central endpoints**: Sites configure both central gRPC endpoints. Read/idempotent operations can fail over within their deadline; non-idempotent calls preserve outcome uncertainty.
|
||||
- **Central as integration hub**: Central brokers requests between external systems and sites. For example, a recipe manager sends a recipe to central, which routes it to the appropriate site. MES requests machine values from central, which routes the request to the site and returns the response.
|
||||
- **Real-time data streaming** is not continuous for all machine data. The only real-time stream is an **on-demand debug view** — an engineer in the central UI can open a live view of a specific instance's tag values and alarm states for troubleshooting purposes. This is session-based and temporary. The debug view subscribes via gRPC to the site's SiteStreamManager filtered by instance (see Section 8.1).
|
||||
|
||||
@@ -402,7 +402,7 @@ The central cluster hosts a **configuration and management UI** (no live machine
|
||||
- **Site Event Log Viewer**: Query and view operational event logs from site clusters (see Section 12).
|
||||
|
||||
### 8.1 Debug View
|
||||
- **Subscribe-on-demand**: When an engineer opens a debug view for an instance, central opens a **gRPC server-streaming subscription** to the site's `SiteStreamGrpcServer` for the instance, then requests a **snapshot** of all current attribute values and alarm states via ClusterClient. The gRPC stream delivers subsequent attribute value and alarm state changes directly from the site's `SiteStreamManager`.
|
||||
- **Subscribe-on-demand**: When an engineer opens a debug view for an instance, central opens a **gRPC server-streaming subscription** to the site's `SiteStreamGrpcServer` for the instance, then requests a **snapshot** of all current attribute values and alarm states via `SiteCommandService`. The gRPC stream delivers subsequent attribute value and alarm state changes directly from the site's `SiteStreamManager`.
|
||||
- Attribute value stream messages are structured as: `[InstanceUniqueName].[AttributePath].[AttributeName]`, attribute value, attribute quality, attribute change timestamp.
|
||||
- Alarm state stream messages are structured as: `[InstanceUniqueName].[AlarmName]`, alarm state (active/normal), priority, timestamp.
|
||||
- The stream continues until the engineer **closes the debug view**, at which point central unsubscribes and the site stops streaming.
|
||||
@@ -520,7 +520,7 @@ Sites log operational events locally, including:
|
||||
|
||||
### 13.1 Management Service
|
||||
- The central cluster exposes a **ManagementActor** that provides programmatic access to all administrative operations — the same operations available through the Central UI.
|
||||
- The ManagementActor registers with Akka.NET **ClusterClientReceptionist** for cross-cluster access, and is also exposed via an HTTP Management API endpoint (`POST /management`) with Basic Auth, LDAP authentication, and role resolution — enabling external tools like the CLI to interact without Akka.NET dependencies.
|
||||
- The ManagementActor is reached in-process through `ManagementActorHolder` and exposed via an HTTP Management API endpoint (`POST /management`) with Basic Auth, LDAP authentication, and role resolution — enabling external tools like the CLI to interact without Akka.NET dependencies.
|
||||
- The ManagementActor enforces the **same role-based authorization** as the Central UI. Every incoming message carries the authenticated user's identity and roles.
|
||||
- All mutating operations performed through the Management Service are **audit logged** via IAuditService, identical to operations performed through the Central UI.
|
||||
- The ManagementActor runs on **every central node** (stateless). For HTTP API access, any central node can handle any request without sticky sessions.
|
||||
|
||||
Reference in New Issue
Block a user