Joseph Doherty
e606b24a55
The site's deployed_configurations rows are the durable truth about what should be
running; the set of live Instance Actors is what IS running. Nothing kept the two in
agreement after boot, and nothing surfaced a disagreement.
Row 78 (availability, proven live 2026-08-15) -- deployed_configurations is one of
LocalDb's ten replicated tables, so on a replicated pair a config can appear in a
node's file with no local actor involved at all: the CDC sync applier writes the row
directly. Live sequence: site-a-b's reconcile fetched 4 configs at its solo boot; the
pair was then restarted with a fresh site-a-a file and snapshot replication seeded
site-a-a's table within ~5s -- AFTER its boot-time load had read an empty table, and
invisibly to its own reconcile (central rightly reported "0 fetched", because central's
view of the site was correct). Enabled instances sat dead with their configs present
until yet another restart. Row 67's EnsureInstanceStartedCommand only covers configs
the reconcile itself FETCHES; this is the same gap through a side door.
Row 72 (silent divergence) -- for a STALE ReconcileGapItem the pass writes the newer
config via StoreDeployedConfigIfNewerAsync, a pure file write, while the running
Instance Actor keeps executing the previously compiled config until the next restart
or redeploy. Nothing said "stored revision != running revision".
Both are closed by one periodic sweep owned by the Deployment Manager singleton
(ScadaBridge:SiteRuntime:InstanceSweepIntervalSeconds, default 60, validated >= 0;
0 disables). Each tick re-reads the durable rows off-thread and, on the actor thread,
for every ENABLED stored config either starts a missing Instance Actor (row 78) or --
when the running one's revision no longer matches the stored one -- re-applies the
stored config through the ordinary redeploy path (row 72).
A sweep was chosen over a LocalDb change hook deliberately: a hook covers only the
replication arrival path, and only on the half of the fleet running with replication
enabled, whereas the sweep is arrival-path-agnostic (replication seeding, post-boot
snapshot resync, a hand-restored file, any future path). It is also not available --
ILocalDb 0.3.0 exposes no change-notification or applied-rows callback, so a hook would
mean a new upstream capability in scadaproj plus an in-process bus into this singleton,
a cross-repo change to close a site-local gap.
Row 72's decision is to CONVERGE, not merely warn. The stored row is what central
successfully deployed; an actor serving the config central believes was replaced is the
actual fault, and a surface-only warning leaves the operator to issue by hand exactly
the redeploy the sweep would have performed. Reusing HandleDeploy means the compile
gate, warm-then-gate ordering, redeploy-supersede, the static-override / native-alarm
reset and all-or-nothing-per-instance are the production ones, not a second
implementation -- and a stop still does NOT cancel an in-flight script run, exactly as
for a central redeploy.
Guards, shared with row 67 via the extracted InstanceIsOwnedByADeployPipeline predicate
and the shared StartStoredInstance helper:
- a DISABLED instance is never started (config row, no actor is its correct state);
- the sweep defers to any deploy pipeline owning the instance (compile warm in flight,
predecessor still terminating, deployment awaiting its persistence/init join), so it
can never interleave with a real redeploy -- that window has the config stored and no
actor, which an unguarded sweep would answer by creating a child whose actor name is
still registered;
- inert until boot-time Instance Actor creation has finished, so a tick between
staggered batches does not "heal" instances the boot load is about to create;
- ONE repair attempt per (instance, revision) pair -- this is what stops a periodic
self-heal becoming a restart loop for an actor that dies during init, or for a stored
config the compile gate rejects (a rejection leaves the running instance untouched:
the gate returns before anything is stopped);
- reads never overlap, so a slow read cannot stack ticks into duplicate repairs.
A sweep that finds nothing is completely silent -- no log above Trace, no site event.
A repair emits the same Warning `deployment` site-event pattern as row 67 with the
trigger named ("deployment-manager sweep self-heal") so it is distinguishable from a
startup-reconcile heal; a stale-revision restart names BOTH revisions and a rejected
one adds an Error event. The synthetic redeploy replies to the singleton itself rather
than NoSender, because dead letters are a monitored site health metric.
CreateInstanceActor gains a revisionHash parameter and is the single creation funnel
that records it, so the stale check covers every path that can start an Instance Actor;
a missing entry reads as "unknown", never "stale", so a future path bypassing the funnel
degrades to no-op rather than to spurious restarts.
Fail-before by behavioral revert (sweep timer left unarmed): 5 of 10 new tests fail,
including both live symptoms -- the row-78 repro ("A replication-seeded config for an
enabled instance left no running Instance Actor") and the row-72 repro, whose snapshot
still shows the rev-1 value 111 while the store holds rev-2's 222. The 5 negative
guards (disabled / in-flight redeploy / matching revisions / silent-when-healthy /
interval-zero) are green in both states, as they must be.
SiteRuntime 626/626, Host 505/505, AuditLog 383/383.
Design note: docs/plans/2026-08-15-instance-sweep-self-heal-design.md
ScadaBridge
ScadaBridge is a centrally-managed, distributed SCADA configuration and deployment platform built on Akka.NET, running across a central cluster and multiple site clusters in a hub-and-spoke topology.
Overview
This repository is the full implementation project for ScadaBridge — the C#/.NET source (src/), tests (tests/), deployable Docker topology (docker/, docker-env2/, infra/), and the design documentation (docs/) that the code implements. This README is the master index: it links the per-component design specs (the spec the code in src/ implements), the per-component reference docs (how the shipped code works), and shows the system architecture. The solution file is ZB.MOM.WW.ScadaBridge.slnx.
Technology Stack
| Layer | Technology |
|---|---|
| Runtime | .NET, Akka.NET (actors, clustering, remoting, persistence, streams) |
| Central UI | Blazor Server (ASP.NET Core + SignalR) |
| Inbound API | ASP.NET Core Web API (REST/JSON) |
| Central Database | MS SQL Server, Entity Framework Core |
| Site Storage | SQLite (deployed configs, S&F buffer, event logs) |
| Authentication | Direct LDAP/AD bind (LDAPS/StartTLS), JWT sessions |
| Notifications | Delivered from the central cluster (Email via SMTP/OAuth2-M365 or on-prem Exchange EWS; SMS via Twilio REST); store-and-forwarded from sites |
| Hosting | Windows Server, Windows Service |
| Cluster | Akka.NET Cluster (active/standby, keep-oldest SBR) |
| Logging | Serilog (structured) |
Scale
- Central cluster: 2-node active/standby behind a load balancer.
- Site clusters: 2-node active/standby, headless (no UI).
Repository Layout
| Path | Contents |
|---|---|
src/ |
C#/.NET implementation — one project per component (ZB.MOM.WW.ScadaBridge.<Component>). Solution: ZB.MOM.WW.ScadaBridge.slnx. |
tests/ |
Unit and integration test projects. |
docs/ |
Documentation — docs/requirements/ (high-level + per-component specs, the spec the code implements), docs/components/ (per-component developer reference — how the code works), docs/test_infra/, docs/plans/. |
docker/ |
Primary 8-node cluster topology (2 central + 3 sites × 2 nodes + Traefik) + deploy.sh. |
docker-env2/ |
Minimal second cluster (2 central + 1 site) for exercising Transport (#24) against a real second environment. |
infra/ |
Local test services (MS SQL, LDAP, OPC UA, SMTP, REST API, Traefik). |
deploy/ |
Production/on-host deployment artifacts (e.g. wonder-app-vd03/). |
AkkaDotNet/ |
Akka.NET reference notes. |
Build, Test & Run
# Build the solution
dotnet build ZB.MOM.WW.ScadaBridge.slnx
# Run the tests
dotnet test ZB.MOM.WW.ScadaBridge.slnx
# Bring up the primary local cluster (builds the scadabridge:latest image + recreates containers)
bash docker/deploy.sh # central load balancer at http://localhost:9000
# Drive the system from the CLI (reads ~/.scadabridge/config.json; test user has all roles)
dotnet run --project src/ZB.MOM.WW.ScadaBridge.CLI -- \
--username multi-role --password password template list
See docker/README.md for ports and management commands, and src/ZB.MOM.WW.ScadaBridge.CLI/README.md for the full CLI reference.
Local Test Environments
Two Docker-based cluster topologies are available for local development and testing:
- Primary (
docker/) — Full topology (2 central + 3 sites × 2 nodes + Traefik). Default development target. - Env2 (
docker-env2/) — Minimal sibling stack (2 central + 1 site × 2 nodes + Traefik), runs concurrently with primary on host ports 91XX. Purpose: exercise the Transport (#24) bundle export/import feature against a real second environment.
Both stacks share the infrastructure services in infra/ (MS SQL, LDAP, SMTP, OPC UA, REST API).
Document Map
Requirements
- HighLevelReqs.md — Complete high-level requirements covering all functional areas.
Component Design Documents
| # | Component | Document | Description |
|---|---|---|---|
| 1 | Template Engine | docs/requirements/Component-TemplateEngine.md | Template modeling, inheritance, composition, path-qualified member addressing, override granularity, locking, alarms, native alarm source bindings, flattening, semantic validation, revision hashing, diff calculation, and folder organization (nested folders, drag-drop). |
| 2 | Deployment Manager | docs/requirements/Component-DeploymentManager.md | Central-side deployment pipeline with deployment ID/idempotency, per-instance operation lock, state transition matrix, all-or-nothing site apply, system-wide artifact deployment with per-site status. |
| 3 | Site Runtime | docs/requirements/Component-SiteRuntime.md | Site-side actor hierarchy with explicit supervision strategies, staggered startup, script trust model (constrained APIs), Tell/Ask conventions, concurrency serialization, site-wide Akka stream with per-subscriber backpressure, and a read-only Native Alarm Actor (peer to the computed Alarm Actor) mirroring native OPC UA A&C / MxAccess alarms with site SQLite persistence. |
| 4 | Data Connection Layer | docs/requirements/Component-DataConnectionLayer.md | Common data connection interface (OPC UA, MxGateway, custom), Become/Stash connection actor model, auto-reconnect, immediate bad quality on disconnect, transparent re-subscribe, synchronous write failures, tag path resolution retry, protocol-agnostic address-space browse (with BrowseNext paging, type-info, and the IAddressSpaceSearchable recursive-search seam), a never-trusting OPC UA verify-endpoint probe, and optional read-only native alarm mirroring (IAlarmSubscribableConnection, one alarm feed per connection with snapshot replay). |
| 5 | Central–Site Communication | docs/requirements/Component-Communication.md | Dual transport: Akka.NET ClusterClient (command/control) + gRPC server-streaming (real-time data). 9 message patterns with per-pattern timeouts, SiteStreamGrpcServer/Client, application-level correlation IDs, transport heartbeat config, gRPC keepalive, message ordering, connection failure behavior. The gRPC stream additively carries the read-only native alarm mirror (computed + native OPC UA / MxAccess) via the enriched AlarmStateUpdate. |
| 6 | Store-and-Forward Engine | docs/requirements/Component-StoreAndForward.md | Buffering (transient failures only), fixed-interval retry, parking, async best-effort replication, SQLite persistence at sites. |
| 7 | External System Gateway | docs/requirements/Component-ExternalSystemGateway.md | HTTP/REST + JSON, API key/Basic Auth, per-system timeout, dual call modes (Call/CachedCall), transient/permanent error classification, dedicated blocking I/O dispatcher, ADO.NET connection pooling. |
| 8 | Notification Service | docs/requirements/Component-NotificationService.md | Central-only — manages typed notification-list, SMTP, and SMS definitions; supplies per-type delivery adapters (Email via SMTP with OAuth2 (M365) or Basic Auth, or on-prem Exchange EWS, BCC, plain text; SMS via Twilio REST, per-recipient, outbound-only); delivery performed by the Notification Outbox. |
| 9 | Central UI | docs/requirements/Component-CentralUI.md | Blazor Server with SignalR real-time push, load balancer failover with JWT, all management workflows. Custom-content modal host (DialogService.ShowAsync<T>) with focus-trap/restore; dark-mode CSS-variable token layer ([data-bs-theme="dark"] overriding ZB.MOM.WW.Theme tokens in site.css, localStorage-persisted, SSR no-flash); reusable presentational components OffsetPager, KeysetPager, and DateTimeRangeFilter adopted across report/audit pages. |
| 10 | Security & Auth | docs/requirements/Component-Security.md | Direct LDAP bind (LDAPS/StartTLS), JWT sessions (HMAC-SHA256, 15-min refresh, 30-min idle), role-based authorization (incl. the Operator/Verifier two-person secured-write roles + policies), site-scoped permissions. |
| 11 | Health Monitoring | docs/requirements/Component-HealthMonitoring.md | 30s report interval, 60s offline threshold, monotonic sequence numbers, raw error counts, tag resolution counts, dead letter monitoring. |
| 12 | Site Event Logging | docs/requirements/Component-SiteEventLogging.md | SQLite storage, 30-day retention + 1GB cap, daily purge, paginated remote queries with keyword search. |
| 13 | Cluster Infrastructure | docs/requirements/Component-ClusterInfrastructure.md | Akka.NET cluster, keep-oldest SBR with down-if-alone, min-nr-of-members=1, 2s/10s/15s failure detection, CoordinatedShutdown, automatic dual-node recovery. The ClusterInfrastructure project owns the ClusterOptions config model; the Akka bootstrap/SBR/CoordinatedShutdown wiring lives in the Host. |
| 14 | Inbound API | docs/requirements/Component-InboundAPI.md | POST /api/{methodName}, X-API-Key header, flat JSON, extended type system (Object/List), script-based implementations, failures-only logging. |
| 15 | Host | docs/requirements/Component-Host.md | Single deployable binary, role-based component registration, per-component config binding (Options pattern), readiness gating, dead letter monitoring, Akka.NET bootstrap, ASP.NET Core hosting for central. |
| 16 | Commons | docs/requirements/Component-Commons.md | Namespace/folder convention (Types/Interfaces/Entities/Messages), shared data types, POCOs, repository interfaces, message contracts with additive-only versioning, UTC timestamp convention, the unified read-only alarm condition model (AlarmConditionState/AlarmKind), and native alarm source entities + the IAlarmSubscribableConnection capability seam. |
| 17 | Configuration Database | docs/requirements/Component-ConfigurationDatabase.md | EF Core data access, per-component repositories, unit-of-work, optimistic concurrency on deployment status, audit logging (IAuditService), migration management (incl. the AddNativeAlarmSources migration + native alarm source repository CRUD). |
| 18 | Management Service | docs/requirements/Component-ManagementService.md | Akka.NET ManagementActor on central, ClusterClientReceptionist registration, programmatic access to all admin operations (incl. the MxGateway two-person secured-write handlers — no-self-approval + CAS, MxGateway-only relay — and OPC UA browse/search/verify/cert-trust verbs), CLI interface. |
| 19 | CLI | docs/requirements/Component-CLI.md | Standalone command-line tool, System.CommandLine, HTTP transport via Management API, JSON/table output, mirrors all Management Service operations. |
| 20 | Traefik Proxy | docs/requirements/Component-TraefikProxy.md | Reverse proxy/load balancer fronting central cluster, active node routing via /health/active, automatic failover. |
| 21 | Notification Outbox | docs/requirements/Component-NotificationOutbox.md | Central component ingesting store-and-forwarded notifications into the Notifications audit table, with NotificationOutboxActor singleton dispatcher, per-type delivery adapters, retry/parking, status tracking, daily purge, and delivery KPIs. |
| 22 | Site Call Audit | docs/requirements/Component-SiteCallAudit.md | Central component auditing site cached calls (ExternalSystem.CachedCall/Database.CachedWrite) into the SiteCalls audit table, with SiteCallAuditActor singleton, telemetry ingest, periodic reconciliation, point-in-time KPIs, daily purge, and central→site Retry/Discard relay for parked calls. |
| 23 | Audit Log | docs/requirements/Component-AuditLog.md | New central append-only AuditLog spanning every script-trust-boundary action (outbound API sync+cached, outbound DB sync+cached, notifications, inbound API). Site-local SQLite hot-path append + gRPC telemetry + central reconciliation; combined telemetry packet with Site Call Audit; central direct-write for Notification Outbox dispatch + Inbound API middleware; monthly partitioning, 365-day default retention. |
| 24 | Transport | docs/requirements/Component-Transport.md | Bundle export/import for templates, shared scripts, external systems, central-only artifacts, plus site-scoped config — Sites, site DataConnections, and Instances — reconciled across environments by a BundleNameMap name-mapping subsystem (auto-match + Map wizard step / CLI flags). AES-256-GCM encryption (carry-full-config with encrypted connection secrets); per-conflict resolution with a per-line Myers diff for code fields; real stale-instance enumeration; correlated audit trail. Never touches site runtime nodes. |
| 25 | Script Analysis | docs/requirements/Component-ScriptAnalysis.md | Shared authoritative script-trust analyzer: unified forbidden-API deny-list (ScriptTrustPolicy), fused semantic+syntactic validator (ScriptTrustValidator), Roslyn compile wrapper (RoslynScriptCompiler), and compile-only globals stubs (ScriptCompileSurface/TriggerCompileSurface); consumed by Template Engine, Site Runtime, Inbound API, and Central UI. |
| 26 | KPI History | docs/requirements/Component-KpiHistory.md | Reusable central KPI-history backbone: tall/EAV KpiSample store (central MS SQL), KpiHistoryRecorderActor cluster singleton (kpi-history-recorder, not readiness-gated) sampling DI-registered IKpiSampleSources each minute, bucketed GetRawSeriesAsync + KpiSeriesBucketer query, and a reusable custom-SVG KpiTrendChart. Ships trends for Notification Outbox, Site Call Audit, Audit Log, and Site Health. |
| 27 | DelmiaNotifier | src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier/README.md | Standalone external client tool (not deployed to the cluster, not in the Host): a compact Native-AOT (win-x64) console app (WWNotifier.exe) that DELMIA Apriso shells out to per recipe download. POSTs to the Inbound API DelmiaRecipeDownload method (X-API-Key, key from SCADABRIDGE_API_KEY) with connect-failure-only failover across a comma-list of base URLs, and reports the legacy YES/NO + exit-code stdout contract. Drop-in replacement for the legacy WWNotifier (docs/former-api-specs/dnc/). Design: docs/plans/2026-06-26-delmia-recipe-notifier-design.md. |
Shared UI sub-component (not a top-level component): TreeView — reusable hierarchical tree/grid Blazor component used by the Central UI (#9) for the templates folder hierarchy, data-connection browse, and tag pickers.
Component Reference Documentation
The design documents above are the specs (what each component does and why). For developer reference docs that describe how the shipped code works — with real code examples, drawn from src/ and written to the Style Guide — see docs/components/. Reference docs exist for 24 of the 27 components (plus the shared TreeView); ScriptAnalysis (#25), KpiHistory (#26), and DelmiaNotifier (#27) are pending — tracked in docs/plans/2026-07-08-deferred-work-register.md. Indexed in docs/components/README.md.
Reference Documentation
- AkkaDotNet/ — Akka.NET reference notes covering actors, remoting, clustering, persistence, streams, serialization, hosting, testing, and best practices.
- docs/plans/ — Design decision documents from refinement sessions.
Architecture Diagram (Logical)
Site Runtime Actor Hierarchy
%%{init: {'theme':'base', 'themeVariables': {'textColor':'#111111','lineColor':'#555555','edgeLabelBackground':'#ffffff','fontSize':'15px'}}}%%
flowchart TD
DMS["Deployment Manager Singleton<br/>(Cluster Singleton)"]
IA["Instance Actor<br/>(one per deployed, enabled instance)"]
IA2["Instance Actor<br/>( … )"]
MOREIA["… more Instance Actors"]
DMS --> IA
DMS --> IA2
DMS -.-> MOREIA
SA["Script Actor<br/>(coordinator, one per instance script)"]
AA["Alarm Actor<br/>(coordinator, one per alarm definition)"]
MORE1["… more Script /<br/>Alarm Actors"]
IA --> SA
IA --> AA
IA -.-> MORE1
SEA["Script Execution Actor<br/>(short-lived, per invocation)"]
AEA["Alarm Execution Actor<br/>(short-lived, per on-trigger invocation)"]
IA2C["… (Script / Alarm Actors)"]
SA --> SEA
AA --> AEA
IA2 -.-> IA2C
subgraph STREAM["Site-Wide Akka Stream"]
PUB["All Instance Actors"]
STR["Site-Wide Akka Stream<br/>(attribute + alarm state changes)"]
DBG["Debug view<br/>(instance-level filtering)"]
PUB -->|publish| STR
STR -->|subscribe filtered| DBG
end
classDef start fill:#d5e8d4,stroke:#82b366,color:#111111;
classDef proc fill:#dae8fc,stroke:#6c8ebf,color:#111111;
classDef warn fill:#ffe6cc,stroke:#d79b00,color:#111111;
classDef alt fill:#e1d5e7,stroke:#9673a6,color:#111111;
classDef muted fill:#f5f5f5,stroke:#999999,color:#666666;
class DMS,STR alt
class IA,IA2,PUB proc
class SA,AA,DBG start
class SEA,AEA warn
class MOREIA,MORE1,IA2C muted