Joseph Doherty
8f2c2813ea
An Instance Actor started by a self-heal came up with NO DCL connection actors, so every tag it subscribed was answered "Unknown connection: <name>" by the DCL manager indefinitely. Proven live 2026-08-16 by the row-78 close-out probe: the sweep correctly started the replication-seeded instances, then every subscribe failed for 6+ min until a second pair restart. Root cause is narrower than the register assumed. A DCL connection actor is never created from data_connection_definitions -- that table has no production reader at all (GetAllDataConnectionDefinitionsAsync is called only by a test). It exists only as a side effect of starting an instance, from the Connections map inside that instance's own flattened config, via EnsureDclConnections. That ensure was the CALLER's job and only two of the four callers made it: CreateInstanceActorBatch (boot) ensured OK ApplyDeployment (deploy) ensured OK StartStoredInstance MISSING <- row-67 ensure-started + row-78 sweep HandleEnableResult MISSING <- enabling a disabled instance The two misses are the two paths added after the original pair -- the signature of a convention living at call sites instead of in the funnel. It was never replication-specific: the boot load reads only ENABLED configs, so a node booting with an instance disabled never creates its connections either, and enabling it afterwards hit the identical dead end on an ordinary unreplicated site. Fix: EnsureDclConnections moves into CreateInstanceActor, already documented as the single Instance Actor creation funnel (it is where the running revision is stamped for the same reason), and the two caller-side calls are removed. Placed before the already-exists guard so the two paths that already called it keep their exact previous semantics. No new message, option or DCL change -- it reuses EnsureDclConnection's existing per-name hash guard, so steady state is silent. Structural rather than another guard: a fifth start path cannot forget. Deliberately NOT done, with rationale in the memo: a sweep pass over data_connection_definitions. It would not have fixed row 85 (that table is empty on a site that has only ever received instance deploys), and re-hashing from it would fight the deploy path over the same per-name hash slot, recreating connection actors and dropping live subscriptions on a loop. Two residual sibling gaps (connection-only artifact deploy replicating to a peer that never adopts it, and the mirror-image removal case) are recorded in the memo and the component docs as open, not silently skipped. Tests: SelfHealCreatesDclConnectionTests, 7 tests asserting on the CreateConnectionCommand messages sent to a recording DCL stand-in. Verified fail-before against a behavioral revert (call removed from the funnel, restored at the two original sites): 4 fail / 3 pass, the 3 being steady-state silence plus two refactor guards that must pass on both sides. Docs: sweep design memo gains section 7; Component-SiteRuntime, Component-DataConnectionLayer, docs/components/SiteRuntime and docs/components/DataConnectionLayer updated (the deploy-flow step ordering in docs/components/SiteRuntime was stale as written).
Component Reference Documentation
Developer-reference docs for each ScadaBridge component, describing how the shipped
code in src/ actually works — with real code examples. These complement the
design specs in docs/requirements/: the specs say what a
component should do and why; these docs say how the code does it.
Reference docs exist for 24 of the 27 components (plus the shared TreeView, listed below). ScriptAnalysis (#25), KpiHistory (#26), and DelmiaNotifier (#27) are pending — tracked in ../plans/2026-07-08-deferred-work-register.md.
| # | Component | Description |
|---|---|---|
| 1 | Template Engine | Template modeling, inheritance, composition, flattening, diffs, and semantic validation as implemented by the engine's services. |
| 2 | Deployment Manager | The central deployment pipeline: deployment identity and idempotency, per-instance operation locks, state transitions, all-or-nothing site apply, and artifact deployment. |
| 3 | Site Runtime | The site actor hierarchy (Deployment Manager singleton, Instance, Script, and Alarm actors), script compilation and execution, the site-wide stream, native alarms, and supervision. |
| 4 | Data Connection Layer | Protocol adapters (OPC UA, MxGateway), the Become/Stash connection state machine, reconnect and re-subscribe, and the native-alarm subscription seam. |
| 5 | Central–Site Communication | ClusterClient command and control plus gRPC server-streaming, the central and site communication actors, receptionist registration, and per-site contact points. |
| 6 | Store-and-Forward Engine | Site-only buffering, fixed-interval retry, parking, SQLite persistence, standby replication, and the operation-tracking store. |
| 7 | External System Gateway | Script-facing HTTP/REST and database access, the Call and CachedCall modes, and transient versus permanent error classification. |
| 8 | Notification Service | Central notification-list and SMTP definitions, the per-type delivery adapters, and OAuth2 or Basic SMTP delivery. |
| 9 | Central UI | The Blazor Server app structure, authentication, real-time updates (SignalR and gRPC), and the management pages. |
| 10 | Security & Auth | LDAP bind authentication, cookie and JWT sessions, role-based and site-scoped authorization, and shared Data Protection keys. |
| 11 | Health Monitoring | Site metric collection and central aggregation, report and heartbeat intervals, offline detection, and dead-letter monitoring. |
| 12 | Site Event Logging | The site-local SQLite event log, retention and storage cap, daily purge, and paginated central query access. |
| 13 | Cluster Infrastructure | The ClusterOptions model and validation for active/standby clustering, split-brain resolution, and failover. |
| 14 | Inbound API | POST /api/{method} script invocation, API-key authentication, the extended type system, and inbound-request auditing. |
| 15 | Host | The deployable binary: role-based component registration, Akka bootstrap, ASP.NET Core hosting, and the readiness and active-node health endpoints. |
| 16 | Commons | Shared POCO entities, repository and service interfaces, message contracts, and the Types/Interfaces/Entities/Messages namespace layout. |
| 17 | Configuration Database | The EF Core DbContext, repository implementations, the audit service, secret encryption, and migration and partition maintenance. |
| 18 | Management Service | The ManagementActor admin command surface, its HTTP endpoints, and ClusterClientReceptionist registration. |
| 19 | CLI | The System.CommandLine tool over the HTTP Management API — its command groups, config file, and output formats. |
| 20 | Traefik Proxy | The reverse proxy fronting the central cluster, active-node routing via /health/active, and the Docker topology configuration. |
| 21 | Notification Outbox | The central NotificationOutboxActor, the Notifications table, the dispatcher loop, retry and parking, and KPIs. |
| 22 | Site Call Audit | The central SiteCallAuditActor, the SiteCalls mirror table, telemetry ingest, and the parked-call Retry/Discard relay. |
| 23 | Audit Log | The append-only audit store: site SQLite hot-path, gRPC telemetry, central ingest and reconciliation, redaction, retention, and KPIs. |
| 24 | Transport | Encrypted bundle export and import, dependency resolution, conflict detection, and BundleImportId-correlated audit. |
| 25 | Tree View | The reusable Blazor tree component — parameters, selection modes, and usage in the Central UI. |
Related Documentation
- Component design specs — the spec each component implements.
- Documentation Style Guide — the writing conventions these docs follow.
- README — the repository master index.