Files

T

Joseph Doherty 9668600c38 docs(inbound): correct bus-consumer claims in tracker + invalidation-contract doc + CLAUDE.md (plan R2-06 T2)

2026-07-13 09:49:53 -04:00

40 KiB

Raw Blame History

ScadaBridge Implementation Project

This is the full implementation project for ScadaBridge — a distributed SCADA system built on Akka.NET in a hub-and-spoke architecture (one central cluster + multiple site clusters). It contains source code, tests, deployable docker topology, and the design documentation that the code implements. The design docs are the spec; src/ is the binary.

When a change is requested, the default assumption is: update the design doc and the code and the tests and (if it ships) the docker deploy — together, in one session, with git diff review before committing.

Project Structure

Top-level directories

src/ — C#/.NET implementation, one project per component (e.g. ZB.MOM.WW.ScadaBridge.AuditLog, ZB.MOM.WW.ScadaBridge.NotificationOutbox, ZB.MOM.WW.ScadaBridge.SiteCallAudit, ZB.MOM.WW.ScadaBridge.CentralUI, ZB.MOM.WW.ScadaBridge.Host, …). Solution file: ZB.MOM.WW.ScadaBridge.slnx.
tests/ — Test projects (unit + integration).
docs/ — Design documentation: docs/requirements/ (high-level + per-component specs), docs/test_infra/ (test infrastructure), docs/plans/ (design-decision and implementation-plan docs). The spec the code implements.
docker/ — 8-node cluster topology (2 central + 3 sites), deploy.sh, per-node appsettings.*.json. See docker/README.md for setup, ports, and management commands. Rebuild + redeploy with bash docker/deploy.sh.
docker-env2/ — Minimal second cluster topology (2 central + 1 site × 2 nodes), runs concurrently with docker/ on host ports 91XX. Built specifically for testing the Transport (#24) feature with two real environments. See docker-env2/README.md. Rebuild + redeploy with bash docker-env2/deploy.sh.
infra/ — Docker Compose for local test services (MS SQL, OPC UA, SMTP, REST API, Traefik). LDAP is no longer started here — dev/test LDAP is the shared GLAuth on 10.100.0.35:3893 (source of truth: scadaproj/infra/glauth/).
deploy/ — Production/on-host deployment artifacts (e.g. deploy/wonder-app-vd03/: appsettings.Central.json, appsettings.Site.json, install.ps1/uninstall.ps1, RUNBOOK.md).
deployments/ — Deployment topology notes (docker-cluster.md, docker-cluster-env2.md, README.md).
code-reviews/ — Per-component code-review notes (one folder per component, plus _template).
tools/ — Repo maintenance/utility scripts (e.g. rename-to-scadabridge.sh).
AkkaDotNet/ — Akka.NET reference documentation and best-practices notes.
deprecated/ — Retired docs/notes kept for reference.
logs/ — Local runtime log output.
vendor/ — Vendored third-party assets (currently an empty placeholder).
.claude/ — Claude Code project config (settings, skills, agents).

Key documents

README.md — Master index with component table and architecture diagrams.
docs/requirements/HighLevelReqs.md — Complete high-level requirements covering all functional areas.
docs/requirements/Component-*.md — Individual component design documents (one per component) — the spec the code implements.
docs/test_infra/test_infra.md — Master test infrastructure doc (OPC UA, MS SQL, SMTP, REST API, Traefik). LDAP is the shared GLAuth on 10.100.0.35:3893 (not a local infra container; see scadaproj/infra/glauth/).
docs/plans/ — Design decision and implementation-plan documents from refinement sessions.

Sister Projects

This repo is indexed by the parent workspace ~/Desktop/scadaproj, whose CLAUDE.md is the canonical umbrella index of the app family (the scadaproj repo is the umbrella for shared ZB.MOM.WW.* libs, infra/GLAuth, and the design docs the peers implement). It carries a ScadaBridge entry plus the full cross-project relationship map — consult it rather than duplicating the family list here, as it stays current.

Repos cloned as sibling directories under ~/Desktop/ — referenced for context, not part of this solution. Directly referenced by ScadaBridge:

~/Desktop/MxAccessGateway — MxAccess Gateway (https://gitea.dohertylan.com/dohertj2/mxaccessgw).
~/Desktop/OtOpcUa — OtOpcUa (https://gitea.dohertylan.com/dohertj2/lmxopcua).

Other peers in the scadaproj family (see scadaproj/CLAUDE.md for details): ~/Desktop/HistorianGateway (historiangw) — AVEVA Historian gRPC sidecar, OtOpcUa's sole historian backend; its SDK source ~/Desktop/histsdk (histsdk); and ~/Desktop/scadaproj (scadaproj) itself — shared ZB.MOM.WW.* libs + the dev/test GLAuth on 10.100.0.35:3893.

Document Conventions

Requirements documents (high-level and component-level) live in docs/requirements/.
Test infrastructure documentation lives in docs/test_infra/.
Component documents are named Component-<Name>.md (PascalCase, hyphen-separated).
Each component document follows a consistent structure: Purpose, Location, Responsibilities, detailed design sections, Dependencies, and Interactions.
The README.md component table must stay in sync with actual component documents. When a component is added, removed, or renamed, update the table.
Cross-component references in Dependencies and Interactions sections must be kept accurate across all documents. When a component's role changes, update references in all affected documents.
Exception: DelmiaNotifier (#27) is an external client tool, not a cluster component; its spec is the project README (src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier/README.md) plus docs/plans/2026-06-26-delmia-recipe-notifier-design.md — there is intentionally no Component-DelmiaNotifier.md.

Refinement Process

When refining requirements, use the Socratic method — ask clarifying questions before making changes.
Probe for implications of decisions across components before updating documents.
When a decision is made, identify all affected documents and update them together for consistency.
After updates, verify no stale cross-references remain (e.g., references to removed or renamed components).

Editing Rules

Edit documents and code in place. Do not create copies or backup files.
When a change affects multiple documents or projects, update them all in the same session — design doc, entities/repos, actors/services, UI, tests, migrations, and deploy config travel together.
Use git diff to review changes before committing.
Commit related changes together with a descriptive message summarizing the design decision and the implementation slice.
After non-trivial code changes, build (dotnet build ZB.MOM.WW.ScadaBridge.slnx) and run relevant tests before declaring done; for cluster-runtime changes, rebuild the image with bash docker/deploy.sh.
Run tests with dotnet test ZB.MOM.WW.ScadaBridge.slnx. ZB.MOM.WW.ScadaBridge.CLI.Tests is now a member of the slnx (added 2026-07; arch-review 08 §2.4) — the old "silently skipped" gotcha no longer applies, so a solution-level test run exercises the CLI suite too.
Propagate cross-repo changes to the umbrella index. This repo is indexed by the parent workspace ~/Desktop/scadaproj. When a fact its index records changes here — remote/push status, stack/component summary, cross-project wire relationships (OPC UA → OtOpcUa, gRPC → mxaccessgw), or the solution/namespace shape — update the ScadaBridge entry in ../scadaproj/CLAUDE.md in the same change so the umbrella index never drifts from this repo. (Mirrors the same rule in the peer repos, e.g. MxAccessGateway/HistorianGateway.)

Current Component List (27 components)

Template Engine — Template modeling, inheritance, composition, validation, flattening, diffs.
Deployment Manager — Central-side deployment pipeline, system-wide artifact deployment, instance lifecycle.
Site Runtime — Site-side actor hierarchy (Deployment Manager singleton, Instance/Script/Alarm Actors), script compilation, Akka stream.
Data Connection Layer — Protocol abstraction (OPC UA, custom), subscription management, clean data pipe.
Central–Site Communication — Akka.NET ClusterClient (command/control) + gRPC server-streaming (real-time data), message patterns, debug streaming.
Store-and-Forward Engine — Buffering, fixed-interval retry, parking, SQLite persistence, replication.
External System Gateway — External system definitions, API method invocation, database connections.
Notification Service — Central-only notification-list and SMTP definitions, per-type delivery adapters (sites no longer deliver notifications).
Central UI — Web-based management interface, all workflows.
Security & Auth — LDAP/AD authentication, role-based authorization, site-scoped permissions.
Health Monitoring — Site health metrics collection and central reporting.
Site Event Logging — Local operational event logs at sites with central query access.
Cluster Infrastructure — Akka.NET cluster setup, active/standby failover, singleton support.
Inbound API — Web API for external systems, API key auth, script-based implementations.
Host — Single deployable binary, role-based component registration, Akka.NET bootstrap.
Commons — Shared types, POCO entity classes, repository interfaces, message contracts.
Configuration Database — EF Core data access layer, repositories, unit-of-work, audit logging (IAuditService), migrations.
Management Service — Akka.NET actor providing programmatic access to all admin operations, ClusterClientReceptionist registration.
CLI — Command-line tool using HTTP Management API, System.CommandLine, JSON/table output.
Traefik Proxy — Reverse proxy/load balancer fronting central cluster, active node routing via /health/active, automatic failover.
Notification Outbox — Central component ingesting store-and-forwarded notifications, Notifications audit table, dispatcher loop, retry/parking, delivery KPIs.
Site Call Audit — Central component auditing site cached calls (CachedCall/CachedWrite); SiteCalls audit table, telemetry ingest, reconciliation, KPIs, central→site Retry/Discard relay; sites remain the source of truth.
Audit Log — Central append-only AuditLog table spanning every script-trust-boundary action (outbound API sync+cached, outbound DB sync+cached, notifications, inbound API). Site SQLite hot-path + gRPC telemetry + reconciliation; combined telemetry with Site Call Audit; central direct-write for Notification Outbox dispatch + Inbound API; monthly partitioning, 365-day retention.
Transport — File-based, encrypted bundle export/import via Central UI. Templates (with all child collections — attributes/alarms/scripts/compositions/native-alarm-sources, field-complete incl. LockedInDerived + script cadence/timeout), system artifacts, central-only configuration, plus site/instance-scoped config (Sites, site DataConnections, Instances + real AreaName by name) reconciled across environments by a BundleNameMap name-mapping subsystem. Per-conflict resolution with a per-line Myers diff. Runs the script trust gate at import review (5th ScriptTrustValidator call site — forbidden-API scripts rejected pre-runtime; template-script name-resolution findings are advisory warnings, ApiMethod findings hard-block). Correlated audit via BundleImportId. Never touches site runtime nodes (imported instances land NotDeployed).
Script Analysis — Shared authoritative script-trust analyzer: unified forbidden-API deny-list (ScriptTrustPolicy), fused semantic+syntactic validator (ScriptTrustValidator), Roslyn compile wrapper (RoslynScriptCompiler), and compile-only globals stubs (ScriptCompileSurface/TriggerCompileSurface); consumed by Template Engine, Site Runtime, Inbound API, and Central UI.
KPI History — Reusable central KPI-history backbone: tall/EAV KpiSample store in central MS SQL, KpiHistoryRecorderActor cluster singleton (kpi-history-recorder, not readiness-gated) sampling DI-registered IKpiSampleSources every minute, bucketed query (GetRawSeriesAsync + KpiSeriesBucketer) + scoped KpiHistoryQueryService, and a reusable custom-SVG KpiTrendChart; ships trends for Notification Outbox, Site Call Audit, Audit Log, and Site Health.
DelmiaNotifier — Standalone external client tool (NOT a cluster component, NOT in the Host): a compact Native-AOT (win-x64) console app (WWNotifier.exe) that DELMIA Apriso shells out to per recipe download. POSTs to the Inbound API DelmiaRecipeDownload method (X-API-Key, key from SCADABRIDGE_API_KEY), with connect-failure-only failover across a comma-list of base URLs, and reports the legacy YES/NO + exit-code stdout contract — a drop-in replacement for the legacy WWNotifier (see docs/former-api-specs/dnc/). Zero-dependency BCL-only, System.Text.Json source-gen. Project README: src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier/README.md; design: docs/plans/2026-06-26-delmia-recipe-notifier-design.md.

Key Design Decisions (for context across sessions)

Architecture & Runtime

Instance modeled as Akka actor (Instance Actor) — single source of truth for runtime state.
Site Runtime actor hierarchy: Deployment Manager singleton → Instance Actors → Script Actors + Alarm Actors.
Script Actors spawn short-lived Script Execution Actors on a dedicated blocking I/O dispatcher.
Alarm Actors are separate peer subsystem from scripts (not inside Script Engine).
Shared scripts execute inline as compiled code (no separate actors).
Site-wide Akka stream for attribute value and alarm state changes with per-subscriber buffering.
Instance Actors serialize all state mutations (Akka actor model); concurrent scripts produce interleaved side effects.
Staggered Instance Actor startup on failover to prevent reconnection storms.
Explicit supervision strategies: Resume for coordinator actors, Stop for short-lived execution actors.

Data & Communication

Data Connection Layer is a clean data pipe — publishes to Instance Actors only.
DCL connection actor uses Become/Stash pattern for lifecycle state machine.
DCL auto-reconnect at fixed interval; immediate bad quality on disconnect; transparent re-subscribe.
DCL write failures returned synchronously to calling script.
Tag path resolution retried periodically for devices still booting.
Static attribute writes persisted to local SQLite (survive restart/failover, reset on redeployment).
All timestamps are UTC throughout the system.
Inter-cluster communication uses two transports: ClusterClient for command/control (deployments, lifecycle, subscribe/unsubscribe handshake, snapshots) and gRPC server-streaming for real-time data (attribute values, alarm states). Both CentralCommunicationActor and SiteCommunicationActor registered with receptionist. Central creates one ClusterClient per site using NodeA/NodeB as contact points. Sites configure multiple central contact points for failover. Addresses cached in CentralCommunicationActor, refreshed periodically (60s) and on admin changes. Heartbeats serve health monitoring only.
gRPC streaming channel: SiteStreamGrpcServer on each site node (Kestrel HTTP/2, port 8083); central creates per-site SiteStreamGrpcClient via SiteStreamGrpcClientFactory. Site entity has GrpcNodeAAddress/GrpcNodeBAddress fields. Proto: sitestream.proto with SiteStreamService, SiteStreamEvent (oneof: AttributeValueUpdate, AlarmStateUpdate). DebugStreamEvent message removed (no longer flows through ClusterClient).
Native alarms: a read-only mirror of native alarms from OPC UA Alarms & Conditions servers and the MxAccess Gateway, unified onto an A&C-style condition model (AlarmConditionState: orthogonal Active/Acked/Confirmed/Shelved/Suppressed + 0–1000 severity) plus an AlarmKind discriminator (Computed/NativeOpcUa/NativeMxAccess). New DCL capability seam IAlarmSubscribableConnection (implemented by the OPC UA and MxGateway adapters); the DataConnectionActor opens ONE alarm feed per connection and routes transitions to instances by source-object reference. A NativeAlarmActor (peer to the computed AlarmActor under InstanceActor) mirrors one source binding: snapshot atomic-swap on (re)subscribe, retention (drops once inactive+acked), per-source cap, and site SQLite persistence (native_alarm_state, survives failover, cleared on redeploy/undeploy — mirrors static overrides). State streams to central over the additively-enriched gRPC AlarmStateUpdate (the existing computed AlarmStateChanged was enriched additively) and seeds via the DebugView snapshot. Authoring: TemplateNativeAlarmSource / InstanceNativeAlarmSourceOverride entities flatten to ResolvedNativeAlarmSource (inherit/compose/override); management commands + ManagementActor handlers + CLI (template/instance native-alarm-source) + Central UI (template editor tab + instance override panel) + enriched DebugView alarm table. Read-only — no ack-back; no central tables.
OPC UA / MxGateway UX (M7): operator Alarm Summary page (/monitoring/alarms, RequireDeployment, read-only) fans out the existing per-instance DebugViewSnapshot Ask (SemaphoreSlim-capped, partial-results tolerant) and aggregates client-side — no central alarm store; shared AlarmStateBadges component. Aggregated live stream shipped 2026-07-10 (docs/plans/2026-07-10-aggregated-live-alarm-stream-plan.md): a transient in-memory per-site central live cache (ISiteAlarmLiveCache) fed by a site-wide, alarm-only SubscribeSite gRPC stream (seed-then-stream), pushing near-real-time deltas to the page over the Blazor circuit with the 15s poll kept as fallback + NotReporting authority — still no persisted central alarm store. OPC UA node browser gains BrowseNext continuation paging ("Load more"), a bounded recursive address-space search (IAddressSpaceSearchable seam; depth + result caps; substring on DisplayName/path), and type-info (DataType/ValueRank/Writable on BrowseNode for Variables). Attribute-override CSV bulk import (OverrideCsvParser, all-or-nothing) via InstanceConfigure InputFile + CLI instance import-overrides --file (native-alarm-source-override CSV deferred). Verify-endpoint probe (temporary RealOpcUaClient, short timeout, captures an untrusted server cert but NEVER trusts it) + site-local cert trust: per-node CertStoreActor (runs on every site node, not a singleton) writing the .der into the node's OPC UA trusted-peer PKI store; DeploymentManager broadcasts TrustServerCertCommand/RemoveServerCertCommand to BOTH site nodes so PKI stores stay consistent across failover; Admin-gated cert-management UI (/design/connections/{id}/certificates). No central persistence of cert trust (follow-up).

External Integrations

External System Gateway: HTTP/REST only, JSON serialization, API key + Basic Auth.
Dual call modes: ExternalSystem.Call() (synchronous) and ExternalSystem.CachedCall() (store-and-forward on transient failure).
Error classification: HTTP 5xx/408/429/connection errors = transient; other 4xx = permanent (returned to script).
Notification Service: SMTP with OAuth2 Client Credentials (Microsoft 365) or Basic Auth. BCC delivery, plain text.
Notification delivery is central-only: sites store-and-forward notifications to the central cluster (target = central, not SMTP); sites never talk to SMTP. Notification lists and SMTP config are no longer deployed to sites; recipient resolution happens at central, at delivery time.
Notification lists carry a Type discriminator (Email and Sms). Notify.To("list") is type-agnostic; delivery is via per-type INotificationDeliveryAdapter (Email via SMTP; Sms via Twilio REST — SmsNotificationDeliveryAdapter, no SDK, one POST per recipient, per-recipient rollup). List Type is fixed after creation.
Notify.Send is async and enqueue-only — it buffers the notification into the local SQLite S&F store and returns a NotificationId (GUID, idempotency key) status handle immediately; it never runs the forwarder's central Ask inline on the script thread (deferToSweep: true buffers due-immediately + kicks a background sweep), so its worst-case latency is the local insert whether central is up or down. It enqueues with maxRetries: 0 (the "no limit" escape hatch), so notifications retry until central acks and are never parked for retry exhaustion — only a corrupt payload parks them (arch-review 02, Tasks 13/14). Notify.Status(notificationId) returns a status record (status, retry count, last error, key timestamps); answered site-locally as Forwarding while still in the site S&F buffer, otherwise round-trips to central.
SMS delivery adapter (T9/T10, 2026-06-19): SmsNotificationDeliveryAdapter — Twilio REST, no SDK, HTTP Basic auth (AccountSid:AuthToken), one POST .../Messages.json per recipient. SmsConfiguration entity (AccountSid, AuthToken encrypted via Data Protection, FromNumber/MessagingServiceSid, ApiBaseUrl, timeout/retry) stored centrally; managed Admin-only via CLI notification sms list|update + Central UI /notifications/sms. Central-only (never deployed to sites). Per-recipient rollup: all-accepted → Delivered; any-transient → retry/park; mix → delivered-to-good + note; all-permanent → Park. SmsConfiguration (Auth Token in SecretsBlock) travels in Transport bundles via --sms-configs.
Inbound API: POST /api/{methodName}, X-API-Key header, flat JSON, extended type system (Object, List).

Templates & Deployment

Pre-deployment validation includes semantic checks (call targets, argument types, trigger operand types).
Composed member addressing uses path-qualified canonical names: [ModuleInstanceName].[MemberName].
Override granularity defined per entity type and per field.
Template graph acyclicity enforced on save.
Flattened configs include a revision hash for staleness detection.
Deployment identity: unique deployment ID + revision hash for idempotency.
Per-instance operation lock covers all mutating commands (deploy, disable, enable, delete).
Site-side apply is all-or-nothing per instance.
System-wide artifact version skew across sites is supported.
Last-write-wins for concurrent template editing (no optimistic concurrency on templates).
Optimistic concurrency on deployment status records.
Naming collisions in composed feature modules are design-time errors.
Transport (#24, M8): bundles now move site/instance config (Sites, site-scoped DataConnections, Instances + override children + Area by name), not just central-only config — but still never touch site runtime nodes (imported instances land NotDeployed). A BundleNameMap name-mapping subsystem (Commons: SiteMapping/ConnectionMapping/MappingAction) reconciles environment-specific identifiers (sites by SiteIdentifier, connections by site-qualified {SiteIdentifier}/{Name}, instances by UniqueName): auto-match → operator override via the import-wizard Map step / CLI --map-site/--map-connection/--create-missing-* → blocker rows for the unresolved. D3 "carry full config (encrypted secrets)": Site addresses travel; DataConnection config rides the encrypted SecretsBlock (presence-only in diffs). Per-line Myers diff for code fields via the pure LineDiffer (ArtifactDiff embeds a size-capped structured lineDiff; input capped at MaxInputLines=4000 → summary-only, so a crafted bundle can't OOM the active node); ImportResult.StaleInstanceIds is real via the IStaleInstanceProbe seam (Commons; implemented in DeploymentManager). schemaVersion 1.0→1.1 (additive); bundleFormatVersion stays 1; no new EF tables/columns. Arch-review 05 (plan 05): template child fidelity completed (round-trip-guard-verified: LockedInDerived, native-alarm-sources, script cadence/timeout, real AreaName, ES retry config + methods-on-Add, notification recipients, folder parent edges); import runs the script trust gate (template/shared/ApiMethod bodies and Expression-trigger bodies) as a hard pre-runtime gate + a name-resolution severity split (template=advisory ConflictKind.Warning+ImportResult.Warnings, ApiMethod=hard error); ArtifactDiff resolves folder/base/composition names (unchanged structured templates classify Identical again); MaxConcurrentImportSessions=8 session cap. Rename does NOT rewrite call sites (deploy gate on the target is the real check). Plan-06 handoff: ScriptArtifactsChanged/IScriptArtifactChangeBus seam + Host in-process bus + Transport post-commit publisher shipped (ApiMethod consumer = InboundAPI ScriptArtifactChangeSubscriber, wired in plan R2-06; SharedScript/Template publish with no consumer — nothing at central caches those kinds).

Store-and-Forward

Fixed retry interval, no max buffer size. Only transient failures buffered.
Async best-effort replication to standby (no ack wait).
Messages not cleared on instance deletion.
CachedCall idempotency is the caller's responsibility.
Notification Outbox: central NotificationOutboxActor singleton on the active central node — the first centrally-hosted store-and-forward component (the S&F Engine remains site-only).
Notifications table in central MS SQL is the single source of audit truth (one row per notification); type-agnostic via the Type discriminator.
Status lifecycle Pending → Retrying → Delivered / Parked / Discarded, plus site-local Forwarding (never persisted centrally).
Dispatcher loop polls due rows, resolves the list, delivers via the typed adapter; transient failures retry to Parked, permanent failures park immediately.
Site→central handoff is at-least-once: ack-after-persist plus insert-if-not-exists on NotificationId.
No Akka replication — MS SQL is the HA store; daily purge of terminal rows after a configurable window (default 365 days).
Notification Outbox retry reuses central SMTP max-retry-count and fixed interval for Email; Sms uses SmsConfiguration.MaxRetries/RetryDelay, falling back to the SMTP policy when unset.
Cached calls (ExternalSystem.CachedCall, Database.CachedWrite) return a TrackedOperationId tracking handle, unified with Notify.Send's existing tracking model (Notify.Status retained as a thin alias).
A site-local operation tracking table (SQLite, alongside the S&F buffer) is the source of truth for cached-call status; Tracking.Status(id) reads it site-locally and authoritatively; terminal rows purged after a configurable window (default 7 days).
Unified tracking status lifecycle Pending → Retrying → Delivered / Parked / Failed / Discarded; Failed = permanent failure (also returned synchronously to the calling script). No Forwarding state for cached calls.
Site Call Audit (#22): central SiteCallAuditActor singleton with a SiteCalls audit table (central MS SQL) fed by best-effort site telemetry plus periodic reconciliation pulls — an eventually-consistent mirror, NOT a dispatcher; cached-call delivery stays site-local. Ingest is insert-if-not-exists then upsert-on-newer-status.
Central UI Site Calls page + central→site RetryParkedOperation/DiscardParkedOperation relay for parked cached calls; central never mutates the SiteCalls row directly.

Centralized Audit Log

Layered design — append-only AuditLog (#23) sits alongside operational Notifications (#21) and SiteCalls (#22), not replacing them.
Scope = script trust boundary: outbound API (sync + cached), outbound DB (sync + cached), notifications, inbound API. Framework/internal traffic is explicitly excluded.
One row per lifecycle event; cached calls produce 4+ rows per operation (Submitted, Forwarded, Attempted, Delivered/Parked/Discarded).
ExecutionId (uniqueidentifier NULL) is the universal per-run correlation value — every audit row emitted by one script execution / inbound request shares it; CorrelationId remains the per-operation lifecycle id (NULL for sync one-shots).
ParentExecutionId (uniqueidentifier NULL) is the cross-execution spawn pointer — every row of a spawned run carries the spawner's ExecutionId; bridges inbound API → routed-site-script, alarm-triggered on-trigger scripts, and nested CallScript/CallShared invocations; IX_AuditLog_ParentExecution backs the filter + the recursive execution-tree walk. Tag-cascade coverage is complete as of M5.4 (T4) — no further spawn points are deferred.
Site SQLite hot-path first, then gRPC telemetry to central; ingest is idempotent on EventId; periodic reconciliation pull as fallback when telemetry is lost.
Cached operations: site emits a single additively-extended CachedCallTelemetry packet carrying both audit events and operational state; central writes AuditLog + SiteCalls in one transaction.
Payload cap 8 KB by default / 64 KB on error rows; auth headers redacted by default; SQL parameter values captured by default; per-target redaction opt-in. Inbound API: full verbatim capture up to InboundMaxBytes (default 1 MiB); request headers stored in Extra.requestHeaders (post-redaction); per-method SkipBodyCapture flag suppresses bodies while still recording headers + metadata; AuditInboundCeilingHits counter surfaced on health snapshot. (M5.3 T7)
Audit-write failure NEVER aborts the user-facing action — audit is best-effort, the action's own success/failure path is authoritative.
365-day central retention with monthly partition-switch purge; per-channel retention overrides (AuditLog:PerChannelRetentionDays) expire rows earlier than the global window via a bounded, batched row DELETE on the purge actor's maintenance path — values must be shorter than the global window (M5.5 T3); 7-day site SQLite retention with a hard ForwardState invariant (no row purged until forwarded or reconciled).
Append-only enforced via DB roles (writer role has INSERT only, no UPDATE/DELETE); hash-chain tamper evidence (T1) and Parquet archival (T2) are deferred to v1.x — not shipped in M5.
Node-of-origin is captured alongside site-of-origin: SourceNode (varchar(64) NULL) on AuditLog, Notifications, and SiteCalls — node-a/node-b for site rows (qualified by SourceSiteId/SourceSite), central-a/central-b for central direct-write rows. Stamped at the writing node, carried verbatim through telemetry + reconciliation, and indexed via IX_AuditLog_Node_Occurred (SourceNode, OccurredAtUtc) on AuditLog.
Per-node stuck KPIs (M5.3 T6): Notification Outbox and Site Call Audit expose PerNodeNotificationKpiRequest/PerNodeSiteCallKpiRequest messages that group stuck/parked/delivered counts by SourceNode, surfacing per-node breakdowns on the Health dashboard.
audit tree --execution-id <guid> CLI command (M5.3 T8) + GET /api/audit/tree endpoint — resolves any node to its chain root and renders the full execution tree; backed by IAuditLogRepository.GetExecutionTreeAsync.
Central UI: new top-level Audit nav group + Audit Log page, with drill-ins from Notifications, Site Calls, External Systems, Inbound API Keys, Sites, and Instances.

Security & Auth

Authentication: direct LDAP bind (username/password), no Kerberos/NTLM. LDAPS/StartTLS required.
Cookie+JWT hybrid sessions: HttpOnly/Secure cookie carries an embedded JWT (HMAC-SHA256 shared symmetric key), 15-minute expiry with sliding refresh, 30-minute idle timeout. Cookies are the correct transport for Blazor Server (SignalR circuits).
LDAP failure: new logins fail; active sessions continue with current roles.
Load balancer in front of central UI; cookie-embedded JWT + shared Data Protection keys for failover transparency.
Two-person MxGateway secured writes (M7): two new global roles — Operator (initiates) + Verifier (approves) — added alongside the canonical Administrator/Designer/Deployer/Viewer, with RequireOperator/RequireVerifier policies. An Operator submits a secured write from the Central UI Secured Writes page (/operations/secured-writes); it stays a Pending PendingSecuredWrite row until a distinct Verifier approves it (no-self-approval enforced server-side in the ManagementActor, plus a compare-and-swap race guard). Approval relays a WriteTagRequest to the site MxGateway; MxGateway-protocol connections only; each lifecycle event (submit/approve/reject/execute) emits a best-effort AuditChannel.SecuredWrite / AuditKind.SecuredWrite* central-direct-write row sharing the row id as CorrelationId. (SecuredWrite audit rows stamp SourceNode via ICentralAuditWriter/INodeIdentityProvider.) Pending secured writes expire server-side after a configurable TTL (ManagementServiceOptions.SecuredWritePendingTtl, default 24 h): an overdue Pending row is CAS'd to Expired (never relayed) — enforced at approve/reject and swept opportunistically on list (arch-review S2, AuditKind.SecuredWriteExpire).

Cluster & Failover

Keep-oldest split-brain resolver with down-if-alone = on, 15s stable-after.
Both nodes are seed nodes. min-nr-of-members = 1.
Failure detection: 2s heartbeat, 10s threshold. Total failover ~25s.
CoordinatedShutdown for graceful singleton handover.
Automatic dual-node recovery from persistent storage.

UI & Monitoring

Central UI: Blazor Server (ASP.NET Core + SignalR) with Bootstrap CSS. No third-party component frameworks (no Blazorise, MudBlazor, Radzen, etc.). Build custom Blazor components for tables, grids, forms, etc.
UI design: Clean, corporate, internal-use aesthetic. Not flashy. Use the frontend-design skill when designing UI pages/components.
Debug view: real-time streaming via DebugStreamBridgeActor + gRPC (events via SiteStreamGrpcClient, snapshot via ClusterClient). Health dashboard: 10s polling timer. Deployment status: real-time push via SignalR.
Health reports: 30s interval, 60s offline threshold, monotonic sequence numbers, raw error counts per interval.
Dead letter monitoring as a health metric.
Site Event Logging: 30-day retention, 1GB storage cap, daily purge, paginated queries with keyword search.
Notification Outbox KPIs are central-computed point-in-time from the Notifications table (global + per-source-site): queue depth, stuck count, parked count, delivered-last-interval, oldest-pending age.
Stuck = Pending/Retrying older than a configurable age threshold (default 10 min) — display-only (KPI count + row badge), no escalation/alerting.
Headline KPI tiles surface on the Health dashboard; a new Central UI Notification Outbox page offers a queryable list with Retry/Discard actions on parked notifications.
Site Call Audit KPIs are central-computed point-in-time from the SiteCalls table (global + per-site), mirroring the Notification Outbox KPI shape; tiles surface on the Health dashboard alongside a queryable Central UI Site Calls page with Retry/Discard on parked rows.
KPI History & Trends (#26, M6): a reusable central KPI-history backbone — supersedes the prior "point-in-time only, no time-series store" stance — backed by a tall/EAV KpiSample table in central MS SQL (no new infra). A KpiHistoryRecorderActor cluster singleton (kpi-history-recorder, not readiness-gated, best-effort with per-source isolation) samples every minute by enumerating DI-registered IKpiSampleSources (each lives with its owner, registered via TryAddEnumerable, reusing existing KPI/aggregator reads); daily purge after RetentionDays (default 90). Querying is IKpiHistoryRepository.GetRawSeriesAsync → KpiSeriesBucketer (last-value-per-bucket) → scoped dual-ctor KpiHistoryQueryService → a reusable custom-SVG KpiTrendChart (no third-party charting lib). Trends ship on four surfaces: Notification Outbox, Site Calls, Audit Log pages + a per-site Health-dashboard panel. KpiHistoryOptions (ScadaBridge:KpiHistory): SampleInterval 60s, RetentionDays 90, PurgeInterval 1d, DefaultMaxSeriesPoints 200; validated. Hourly rollups shipped 2026-07-10 (deferred #22): a second KpiRollupHourly table folded by a third recorder tick (kpi-rollup, RollupInterval 1h, re-folds RollupLookbackHours window via idempotent failover-self-healing upsert + one-shot backfill on start), per-metric gauge-vs-rate fold (KpiMetricAggregationCatalog), and raw-vs-rollup query routing by RollupThresholdHours (168h) unblock long-range 30d/90d trend windows (rollups retained RollupRetentionDays 365 ≥ raw 90, dual daily purge). M6's T9 and T10 were originally deferred. T9/T10 delivered 2026-06-19 as an SMS (Twilio) adapter — Teams was evaluated and dropped (outbound-only backends cannot send 1:1 chat bodies via Graph without a Bot Framework inbound endpoint; SMS is inherently per-person and outbound-only). NotificationType.Sms added; SmsNotificationDeliveryAdapter (Twilio REST, no SDK) registered alongside Email; Central UI Type selector (adapter-gated) + SMS recipient input; CLI --type sms --phones on list create/update; notification sms list|update commands.

Code Organization

Entity classes are persistence-ignorant POCOs in Commons; EF mappings in Configuration Database.
Repository interfaces defined in Commons; implementations in Configuration Database.
Commons namespace hierarchy: Types/, Interfaces/, Entities/, Messages/ with domain area subfolders.
Message contracts follow additive-only evolution rules for version compatibility.
Per-component configuration via appsettings.json sections bound to options classes (Options pattern).
Options classes owned by component projects, not Commons.
Host readiness gating: /health/ready endpoint, no traffic until operational.
EF Core migrations: auto-apply in dev, manual SQL scripts for production. Gotcha: dotnet ef migrations add … --no-build scaffolds off the previously-built DLL and can emit an empty migration — build first (drop the --no-build), and delete the empty Migrations/ files if it happens (migrations remove needs a live DB).
Audit logging absorbed into Configuration Database component (IAuditService).

Akka.NET Conventions

Tell for hot-path internal communication; Ask reserved for system boundaries.
ClusterClient for cross-cluster communication; ClusterClientReceptionist for service discovery across cluster boundaries.
Script trust model: forbidden APIs (System.IO, Process, Threading, Reflection, raw network). The trust boundary is centralized in the Script Analysis component (#25) — ScriptTrustPolicy is the single source of truth; all four call sites (Template Engine, Site Runtime, Inbound API, Central UI) delegate to ScriptTrustValidator. The design-time deploy gate in Template Engine is authoritative (real semantic compile), not advisory.
Application-level correlation IDs on all request/response messages.

Tool Usage

When consulting with the Codex MCP tool, use model gpt-5.4.
When a task requires setting up or controlling system state (sites, templates, instances, data connections, deployments, security, etc.) and the Central UI is not needed, prefer the ScadaBridge CLI over manual DB edits or UI navigation. See src/ZB.MOM.WW.ScadaBridge.CLI/README.md for the full command reference.

CLI Quick Reference (Docker / OrbStack)

Management URL: http://localhost:9000 — the CLI connects via the Traefik load balancer, which routes to the active central node. Direct access: central-a on port 9001, central-b on port 9002.
Test user: --username multi-role --password password — has Admin, Design, and Deployment roles. The admin user only has the Admin role and cannot create templates, data connections, or deploy.
Config file: ~/.scadabridge/config.json — stores managementUrl and default format. See docker/README.md for a ready-to-use test config.
Rebuild cluster: bash docker/deploy.sh — builds the scadabridge:latest image and recreates all containers. Run this after code changes to ManagementActor, Host, or any server-side component.
Infrastructure services: cd infra && docker compose up -d — starts MS SQL, OPC UA, SMTP, and REST API. These are separate from the cluster containers in docker/. LDAP is NOT started here — it is the shared GLAuth on 10.100.0.35:3893 (dc=zb,dc=local); source of truth and config: scadaproj/infra/glauth/.
All test LDAP passwords: password (see scadaproj/infra/glauth/config.toml for users and groups; canonical cross-app login: multi-role).

40 KiB Raw Blame History Unescape Escape