Joseph Doherty
c57e74e660
Closes deferred-work-register rows 89 and 90, and corrects their premise twice.
THE RULE
The `data_connection_definitions` store wins wherever it has an opinion; an
instance's flattened `Connections` entry is a deploy-time snapshot and is
authoritative only for names the store has never owned (the deploy-only site,
where that table is legitimately empty).
PREMISE CORRECTION 1 — the peer's DCL is EMPTY, not stale
`DeploymentManagerActor` is a `site-{SiteId}`-scoped singleton and the only
sender of Create/RemoveConnectionCommand, to its OWN node's `dcl-manager`
(which exists on every node but is populated only where the singleton runs).
So the standby serves nothing. The real defect is that `_createdConnections`
is in-memory per singleton incarnation: a restart or failover rebuilds every
connection from the flattened snapshots and silently discards a newer
connection-only artifact deploy. Row 89 is therefore create-time, not
message-delivery, and NOT replication-specific — the same loss happens on an
unreplicated single-node restart.
PREMISE CORRECTION 2 — a changed config never reached a LIVE connection either
`DataConnectionManagerActor.HandleCreateConnection` is create-only (warns and
returns on a name it holds), so `EnsureDclConnection`'s documented "re-issues
the create so the DCL adopts the new configuration" has never been true on any
node. Now surfaced with a Warning saying exactly that, once per changed config.
Not fixed here: stop-and-recreate drops every tag subscription and InstanceActor
cancels its subscribe-retry timer on first success, so nothing re-subscribes.
MECHANISM
`data_connection_definitions` gains its first production reader. It is read on
the SAME off-thread task as the deployed configs (`ReadStoreSnapshotAsync`), by
the boot load and every sweep tick, into an actor-thread-confined arbitration
overlay applied BEFORE the boot batch and BEFORE the sweep's repair loop. A
CDC-delivered definition change converges within one tick on whichever node runs
the singleton — no new cross-node message. A failed definitions read yields null
and leaves the overlay untouched (distinct from an empty list, which is a real
answer); clearing it would reintroduce the ping-pong through the error path.
THE PING-PONG, KILLED BY CONSTRUCTION
§7.4 of the sweep memo rejected a store-level pass because two sources would
flip the single per-name hash slot, recreating the actor every tick. That is a
property of WHERE the hash is computed: every ensure path now resolves the
winner BEFORE hashing, so only one configuration can ever occupy the slot. The
artifact-deploy path writes its rows into the overlay in the same pass, closing
the same flip arriving through the other door.
REMOVAL (row 90)
Both removal paths share `RemoveConnectionCentrally` and record an in-memory
tombstone, so a centrally removed connection is not resurrected by the next
instance start — a pre-existing bug. Tombstones are deliberately not persisted;
a definition row reappearing clears one.
TESTS
tests/.../Actors/DclConnectionArbitrationTests.cs — 8 tests. Fail-before
(behavioral revert: overlay inert + tombstone check disabled): 5 fail, 3 pass;
the 3 passers are the two deploy-only regression guards and the ping-pong guard.
DOCS
New decision memo docs/plans/2026-08-17-dcl-connection-arbitration-decision.md
(topology, rule, ping-pong mechanism, tombstones, residuals); Component-SiteRuntime,
Component-DataConnectionLayer + their docs/components mirrors; the §7.4 residual in
the sweep memo marked closed with its premise corrected.
Component Reference Documentation
Developer-reference docs for each ScadaBridge component, describing how the shipped
code in src/ actually works — with real code examples. These complement the
design specs in docs/requirements/: the specs say what a
component should do and why; these docs say how the code does it.
Reference docs exist for 24 of the 27 components (plus the shared TreeView, listed below). ScriptAnalysis (#25), KpiHistory (#26), and DelmiaNotifier (#27) are pending — tracked in ../plans/2026-07-08-deferred-work-register.md.
| # | Component | Description |
|---|---|---|
| 1 | Template Engine | Template modeling, inheritance, composition, flattening, diffs, and semantic validation as implemented by the engine's services. |
| 2 | Deployment Manager | The central deployment pipeline: deployment identity and idempotency, per-instance operation locks, state transitions, all-or-nothing site apply, and artifact deployment. |
| 3 | Site Runtime | The site actor hierarchy (Deployment Manager singleton, Instance, Script, and Alarm actors), script compilation and execution, the site-wide stream, native alarms, and supervision. |
| 4 | Data Connection Layer | Protocol adapters (OPC UA, MxGateway), the Become/Stash connection state machine, reconnect and re-subscribe, and the native-alarm subscription seam. |
| 5 | Central–Site Communication | ClusterClient command and control plus gRPC server-streaming, the central and site communication actors, receptionist registration, and per-site contact points. |
| 6 | Store-and-Forward Engine | Site-only buffering, fixed-interval retry, parking, SQLite persistence, standby replication, and the operation-tracking store. |
| 7 | External System Gateway | Script-facing HTTP/REST and database access, the Call and CachedCall modes, and transient versus permanent error classification. |
| 8 | Notification Service | Central notification-list and SMTP definitions, the per-type delivery adapters, and OAuth2 or Basic SMTP delivery. |
| 9 | Central UI | The Blazor Server app structure, authentication, real-time updates (SignalR and gRPC), and the management pages. |
| 10 | Security & Auth | LDAP bind authentication, cookie and JWT sessions, role-based and site-scoped authorization, and shared Data Protection keys. |
| 11 | Health Monitoring | Site metric collection and central aggregation, report and heartbeat intervals, offline detection, and dead-letter monitoring. |
| 12 | Site Event Logging | The site-local SQLite event log, retention and storage cap, daily purge, and paginated central query access. |
| 13 | Cluster Infrastructure | The ClusterOptions model and validation for active/standby clustering, split-brain resolution, and failover. |
| 14 | Inbound API | POST /api/{method} script invocation, API-key authentication, the extended type system, and inbound-request auditing. |
| 15 | Host | The deployable binary: role-based component registration, Akka bootstrap, ASP.NET Core hosting, and the readiness and active-node health endpoints. |
| 16 | Commons | Shared POCO entities, repository and service interfaces, message contracts, and the Types/Interfaces/Entities/Messages namespace layout. |
| 17 | Configuration Database | The EF Core DbContext, repository implementations, the audit service, secret encryption, and migration and partition maintenance. |
| 18 | Management Service | The ManagementActor admin command surface, its HTTP endpoints, and ClusterClientReceptionist registration. |
| 19 | CLI | The System.CommandLine tool over the HTTP Management API — its command groups, config file, and output formats. |
| 20 | Traefik Proxy | The reverse proxy fronting the central cluster, active-node routing via /health/active, and the Docker topology configuration. |
| 21 | Notification Outbox | The central NotificationOutboxActor, the Notifications table, the dispatcher loop, retry and parking, and KPIs. |
| 22 | Site Call Audit | The central SiteCallAuditActor, the SiteCalls mirror table, telemetry ingest, and the parked-call Retry/Discard relay. |
| 23 | Audit Log | The append-only audit store: site SQLite hot-path, gRPC telemetry, central ingest and reconciliation, redaction, retention, and KPIs. |
| 24 | Transport | Encrypted bundle export and import, dependency resolution, conflict detection, and BundleImportId-correlated audit. |
| 25 | Tree View | The reusable Blazor tree component — parameters, selection modes, and usage in the Central UI. |
Related Documentation
- Component design specs — the spec each component implements.
- Documentation Style Guide — the writing conventions these docs follow.
- README — the repository master index.