fix(siteruntime): arbitrate DCL connection config — definitions store wins (rows 89 + 90)
Closes deferred-work-register rows 89 and 90, and corrects their premise twice.
THE RULE
The `data_connection_definitions` store wins wherever it has an opinion; an
instance's flattened `Connections` entry is a deploy-time snapshot and is
authoritative only for names the store has never owned (the deploy-only site,
where that table is legitimately empty).
PREMISE CORRECTION 1 — the peer's DCL is EMPTY, not stale
`DeploymentManagerActor` is a `site-{SiteId}`-scoped singleton and the only
sender of Create/RemoveConnectionCommand, to its OWN node's `dcl-manager`
(which exists on every node but is populated only where the singleton runs).
So the standby serves nothing. The real defect is that `_createdConnections`
is in-memory per singleton incarnation: a restart or failover rebuilds every
connection from the flattened snapshots and silently discards a newer
connection-only artifact deploy. Row 89 is therefore create-time, not
message-delivery, and NOT replication-specific — the same loss happens on an
unreplicated single-node restart.
PREMISE CORRECTION 2 — a changed config never reached a LIVE connection either
`DataConnectionManagerActor.HandleCreateConnection` is create-only (warns and
returns on a name it holds), so `EnsureDclConnection`'s documented "re-issues
the create so the DCL adopts the new configuration" has never been true on any
node. Now surfaced with a Warning saying exactly that, once per changed config.
Not fixed here: stop-and-recreate drops every tag subscription and InstanceActor
cancels its subscribe-retry timer on first success, so nothing re-subscribes.
MECHANISM
`data_connection_definitions` gains its first production reader. It is read on
the SAME off-thread task as the deployed configs (`ReadStoreSnapshotAsync`), by
the boot load and every sweep tick, into an actor-thread-confined arbitration
overlay applied BEFORE the boot batch and BEFORE the sweep's repair loop. A
CDC-delivered definition change converges within one tick on whichever node runs
the singleton — no new cross-node message. A failed definitions read yields null
and leaves the overlay untouched (distinct from an empty list, which is a real
answer); clearing it would reintroduce the ping-pong through the error path.
THE PING-PONG, KILLED BY CONSTRUCTION
§7.4 of the sweep memo rejected a store-level pass because two sources would
flip the single per-name hash slot, recreating the actor every tick. That is a
property of WHERE the hash is computed: every ensure path now resolves the
winner BEFORE hashing, so only one configuration can ever occupy the slot. The
artifact-deploy path writes its rows into the overlay in the same pass, closing
the same flip arriving through the other door.
REMOVAL (row 90)
Both removal paths share `RemoveConnectionCentrally` and record an in-memory
tombstone, so a centrally removed connection is not resurrected by the next
instance start — a pre-existing bug. Tombstones are deliberately not persisted;
a definition row reappearing clears one.
TESTS
tests/.../Actors/DclConnectionArbitrationTests.cs — 8 tests. Fail-before
(behavioral revert: overlay inert + tombstone check disabled): 5 fail, 3 pass;
the 3 passers are the two deploy-only regression guards and the ping-pong guard.
DOCS
New decision memo docs/plans/2026-08-17-dcl-connection-arbitration-decision.md
(topology, rule, ping-pong mechanism, tombstones, residuals); Component-SiteRuntime,
Component-DataConnectionLayer + their docs/components mirrors; the §7.4 residual in
the sweep memo marked closed with its premise corrected.
This commit is contained in:
Joseph Doherty
@@ -161,7 +161,11 @@ A separate in-flight set `_subscribesInFlight` tracks the initial `SubscribeAsyn
|
||||
|
||||
### Creating a connection
|
||||
|
||||
The Site Runtime's `DeploymentManagerActor` sends `CreateConnectionCommand` to `DataConnectionManagerActor` whenever it creates an `InstanceActor` — the config comes from the `Connections` map inside that instance's flattened config, not from the site's `data_connection_definitions` table (which has no production reader). The manager never reads storage and has no independent view of which connections a site should have, so a start path that fails to push is indistinguishable here from a connection that genuinely does not exist: that was register row **85**, where two of the four Instance-Actor start paths omitted the push and every subscribe from the resulting instances was answered `Unknown connection: <name>` forever. The push now lives inside `CreateInstanceActor` on the Site Runtime side. Sends are hash-guarded there, so an unchanged connection produces no message at all:
|
||||
The Site Runtime's `DeploymentManagerActor` sends `CreateConnectionCommand` to `DataConnectionManagerActor` whenever it creates an `InstanceActor` — the config comes from the `Connections` map inside that instance's flattened config, not from the site's `data_connection_definitions` table (which the DCL never reads — and which had no production reader at all until rows 89/90 made it the Deployment Manager's arbitration overlay; see below). The manager never reads storage and has no independent view of which connections a site should have, so a start path that fails to push is indistinguishable here from a connection that genuinely does not exist: that was register row **85**, where two of the four Instance-Actor start paths omitted the push and every subscribe from the resulting instances was answered `Unknown connection: <name>` forever. The push now lives inside `CreateInstanceActor` on the Site Runtime side. Sends are hash-guarded there, so an unchanged connection produces no message at all:
|
||||
|
||||
> **Arbitration (register rows 89 + 90)**: the Deployment Manager *does* read `data_connection_definitions` as of 2026-08-17 — a name with a row there is configured from that row, and only a name with no row falls to the flattened snapshot. The resolution happens **before** the hash is computed, so the two sources can never occupy the same per-name hash slot alternately. Still nothing changes here: the DCL manager remains a router over actors it is told to create. See `SiteRuntime.md` and `docs/plans/2026-08-17-dcl-connection-arbitration-decision.md`.
|
||||
>
|
||||
> **`HandleCreateConnection` is create-only, and that is a live gap**: a create for a name it already holds warns and returns, so a live connection actor never adopts a changed endpoint. The change takes effect only on the next creation from scratch (node restart, singleton failover, remove-then-re-add). A stop-and-recreate would be destructive — subscriptions are held by `DataConnectionActor` and `InstanceActor` cancels its subscribe-retry timer on first success, so nothing re-subscribes. The non-destructive fix is a `ReconfigureConnectionCommand` inside `DataConnectionActor` reusing the primary→backup failover adapter swap.
|
||||
|
||||
```csharp
|
||||
// Commons/Messages/DataConnection/CreateConnectionCommand.cs
|
||||
|
||||
@@ -91,6 +91,8 @@ Central sends a `DeployInstanceCommand` carrying a JSON `FlattenedConfiguration`
|
||||
|
||||
> **`CreateInstanceActor` is the single funnel for the connection push too (register row 85)**: the `EnsureDclConnections` call moved *into* `CreateInstanceActor` on 2026-08-17. It used to sit at each caller, and only two of the four callers made it — the boot-time batch and this deploy path. `StartStoredInstance` (row-67 ensure-started and the row-78 sweep) and `HandleEnableResult` did not, so a self-healed or newly-enabled instance came up with no connection actors and every tag subscribe was answered `Unknown connection: <name>` indefinitely. See the *Instance reconcile sweep* section and `docs/plans/2026-08-15-instance-sweep-self-heal-design.md` §7.
|
||||
|
||||
> **Which config source wins (register rows 89 + 90, 2026-08-17)**: `EnsureDclConnections` no longer trusts the flattened snapshot blindly. `data_connection_definitions` is read on the same off-thread task as the deployed configs, at boot and on every sweep tick, into an actor-thread-confined **arbitration overlay**: a connection name with a row there is configured from that row, and only a name with no row uses the instance's flattened `Connections` entry (which is the correct source on a deploy-only site, where that table is legitimately empty). The winner is resolved **before** the per-name hash is computed, which is what makes the two-source ping-pong — the reason a store-level sweep was rejected in §7.4 — unrepresentable rather than merely avoided. Removals additionally leave an in-memory tombstone so a centrally deleted connection is not resurrected by the next instance start. **The register's premise was wrong in two ways** (the standby's DCL is empty rather than stale, so the real defect is create-time on the next singleton incarnation and is not replication-specific; and a re-issued create never reconfigured a *live* connection actor at all). See `docs/plans/2026-08-17-dcl-connection-arbitration-decision.md`.
|
||||
|
||||
For redeployment (instance already running), the existing actor is stopped and watched:
|
||||
|
||||
```csharp
|
||||
@@ -219,7 +221,7 @@ Both `AlarmActor` and `NativeAlarmActor` tell the `InstanceActor` an `AlarmState
|
||||
| `shared_scripts` | Shared script code from artifact deployments | No |
|
||||
| `external_systems` | External system definitions (incl. `timeout_seconds`, and `max_retries`/`retry_delay_ms` — the deployed store-and-forward retry policy, `0` = use the engine default) | No |
|
||||
| `database_connections` | Database connection strings | No |
|
||||
| `data_connection_definitions` | OPC UA / MxGateway endpoint definitions | No |
|
||||
| `data_connection_definitions` | OPC UA / MxGateway endpoint definitions — read at boot and on every sweep tick as the DCL **arbitration overlay** (rows 89/90) | No |
|
||||
| `notification_lists` | Notification list definitions | No |
|
||||
| `smtp_configurations` | SMTP configuration (from artifact deployment) | No |
|
||||
|
||||
|
||||
@@ -229,7 +229,8 @@ The register's suggested fix was "ensure a connection actor exists for every sto
|
||||
connection". Reading the code first changed the shape of the answer, because the premise is not
|
||||
how the runtime works:
|
||||
|
||||
- **`data_connection_definitions` has no production reader.**
|
||||
- **`data_connection_definitions` has no production reader** *(true when this was written; rows
|
||||
89/90 gave it one on 2026-08-17 — see the residual note at the end of this section)*.
|
||||
`SiteStorageService.GetAllDataConnectionDefinitionsAsync` is called from exactly one place in
|
||||
the repo, and it is a test. Nothing loads that table at boot.
|
||||
- A DCL connection actor is created **only as a side effect of starting an instance**, from the
|
||||
@@ -311,6 +312,19 @@ sibling of row 85, it is **not** closed here, and closing it needs the arbitrati
|
||||
artifact-deploy reconcile; a connection deleted on the peer via replication leaves a live actor on
|
||||
this node. Same root cause, same arbitration question, same register entry.
|
||||
|
||||
> **CLOSED 2026-08-17 (rows 89 + 90) — and the residual above is worded on a wrong premise.**
|
||||
> The arbitration decision was made and implemented: **the definitions store wins wherever it has
|
||||
> an opinion; the flattened config is authoritative only for names the store has never owned.** The
|
||||
> guarded-version-is-near-empty objection above does not apply to that rule, because the guard is
|
||||
> not "skip names already in `_createdConnections`" but "resolve the WINNER before hashing", which
|
||||
> leaves exactly one candidate per name and makes the flip unrepresentable rather than merely
|
||||
> avoided. Two premise corrections came out of validating it: the peer's DCL is **empty**, not
|
||||
> stale (only the singleton's node ever gets a `CreateConnectionCommand`), so the defect is
|
||||
> create-time on the next singleton incarnation and is **not replication-specific**; and
|
||||
> `HandleCreateConnection` is create-only, so a changed config never reached a *live* connection
|
||||
> actor on any node — that one is still open and is this file's successor's §6.1. See
|
||||
> `docs/plans/2026-08-17-dcl-connection-arbitration-decision.md`.
|
||||
|
||||
### 7.5 Tests
|
||||
|
||||
`tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/SelfHealCreatesDclConnectionTests.cs` —
|
||||
|
||||
@@ -0,0 +1,293 @@
|
||||
# DCL connection arbitration — which config source owns a live connection
|
||||
|
||||
**Date:** 2026-08-17
|
||||
**Scope:** Site Runtime (`DeploymentManagerActor`), deferred-work-register rows **89** and **90**
|
||||
**Status:** implemented
|
||||
**Predecessor:** `docs/plans/2026-08-15-instance-sweep-self-heal-design.md` §7.4, which designed a
|
||||
store-level connection sweep, **rejected** it, and recorded rows 89/90 as the residual — explicitly
|
||||
blocked on "the arbitration decision above (which source wins), not another sweep pass".
|
||||
|
||||
---
|
||||
|
||||
## 1. Verified topology — and two premise corrections
|
||||
|
||||
Both register rows are worded in terms of "the peer node's DCL". That framing is wrong, and the
|
||||
correction changes what the fix has to do, so it comes first.
|
||||
|
||||
### 1.1 What actually runs where
|
||||
|
||||
| Actor | Placement | Verified at |
|
||||
|---|---|---|
|
||||
| `DeploymentManagerActor` | **Cluster singleton**, role-scoped to `site-{SiteId}` (`SingletonRegistrar.Start(…, "deployment-manager", …, role: siteRole)`) | `Host/Actors/AkkaHostedService.cs` |
|
||||
| `DataConnectionManagerActor` (`/user/dcl-manager`) | **Every site node**, unconditional `ActorOf` — *not* a singleton | `Host/Actors/AkkaHostedService.cs` |
|
||||
| `CertStoreActor` | Every site node (documented; re-confirmed) | same file |
|
||||
|
||||
`CreateConnectionCommand` and `RemoveConnectionCommand` have exactly **one** sender in the whole
|
||||
repo — `DeploymentManagerActor` — and it sends them to the `_dclManager` ref it was constructed
|
||||
with, which is *its own node's* local `dcl-manager` (the Props are built per node, so whichever
|
||||
node materialises the singleton uses that node's manager).
|
||||
|
||||
### 1.2 Premise correction 1 — the standby's DCL is EMPTY, not stale
|
||||
|
||||
Because the singleton is the only sender, **the standby node's `dcl-manager` holds no connection
|
||||
actors at all**. It is not "serving the old endpoint"; it is serving nothing, and nothing subscribes
|
||||
through it. So there is no per-node divergence to reconcile with a cross-node message.
|
||||
|
||||
The real failure mode is one layer over:
|
||||
|
||||
> `_createdConnections` — the per-name hash cache that records what the DM last pushed — is
|
||||
> **in-memory and per singleton incarnation**. When the singleton restarts (failover, or an
|
||||
> ordinary node restart), the new incarnation starts with an empty cache, loads
|
||||
> `deployed_configurations`, and rebuilds every connection **from the `Connections` snapshot frozen
|
||||
> into each instance's flattened config at that instance's deploy time**. Nothing has ever read
|
||||
> `data_connection_definitions` — that table had **no production reader** — so a newer,
|
||||
> connection-only artifact deploy is silently discarded and the site goes back to the old endpoint.
|
||||
|
||||
This also means **row 89 is not replication-specific**. Replication only puts the definition row on
|
||||
both nodes, which is if anything helpful. The identical loss happens on an unreplicated single-node
|
||||
restart. The row-89 fix is therefore a *create-time arbitration* fix, not a *message-delivery* fix.
|
||||
|
||||
Row 90 is the same shape: the delete is durable in the store, but the flattened configs still name
|
||||
the connection, so the next instance start re-creates it.
|
||||
|
||||
### 1.3 Premise correction 2 — a changed config never reached a LIVE connection either
|
||||
|
||||
`EnsureDclConnection`'s contract, as documented in the code before this change, was that a changed
|
||||
endpoint/credential *"re-issues a `CreateConnectionCommand` so the DCL adopts the new configuration
|
||||
instead of keeping a stale connection"*. **That is false.**
|
||||
`DataConnectionManagerActor.HandleCreateConnection` is create-only:
|
||||
|
||||
```csharp
|
||||
if (_connectionActors.ContainsKey(command.ConnectionName))
|
||||
{
|
||||
_log.Warning("Connection {0} already exists", command.ConnectionName);
|
||||
return;
|
||||
}
|
||||
```
|
||||
|
||||
So the system-wide artifact deploy's "make it live immediately" push has never worked on the
|
||||
deploy-handling node, and the DM's hash cache recorded it as applied regardless. This is a
|
||||
**separate, pre-existing defect** discovered while validating rows 89/90; see §6.
|
||||
|
||||
---
|
||||
|
||||
## 2. The arbitration rule
|
||||
|
||||
> **The definitions store wins wherever it has an opinion. The flattened config is authoritative
|
||||
> only for names the store has never owned.**
|
||||
|
||||
1. **A name with a row in `data_connection_definitions`** — that row's config configures the live
|
||||
DCL connection actor. The identically-named entry in an instance's flattened `Connections` map
|
||||
is a deploy-time snapshot and never overrides it.
|
||||
2. **A name with no definition row** — the flattened config is the source, exactly as before. This
|
||||
is not a fallback; it is the *correct* answer for a deploy-only site, where the definitions
|
||||
table is legitimately empty because nothing has ever sent that site a system-wide artifact
|
||||
deploy while every connection that matters lives inline in the instance configs. §7.4 of the
|
||||
predecessor memo identified this as the reason a blanket store-level sweep is wrong, and it
|
||||
remains the reason rule 2 exists.
|
||||
3. **Removal** semantics on the deploy-handling node are unchanged: the actor is stopped even if
|
||||
deployed instances still reference it, and their attributes go bad quality until redeploy. What
|
||||
is *added* is a tombstone (§4).
|
||||
|
||||
### 2.1 The overlay is an arbiter, not a creation source
|
||||
|
||||
`_connectionDefinitions` decides **which configuration** a connection the runtime already wants is
|
||||
built from. It never creates a connection on its own: a definition row naming a connection no
|
||||
deployed instance references produces no actor, because none is wanted. This is what makes it safe
|
||||
to consult on every sweep tick, and it is why re-adding a centrally deleted connection brings the
|
||||
actor back at the *next instance start* rather than instantly (pinned by
|
||||
`ReAddedDefinition_ClearsTheTombstone_SoTheNextInstanceStartRecreatesIt`).
|
||||
|
||||
---
|
||||
|
||||
## 3. The ping-pong, and why keying the hash on the winner makes it unrepresentable
|
||||
|
||||
The predecessor memo rejected a store-level sweep partly because of this:
|
||||
|
||||
> `_createdConnections` holds one hash per name. The flattened-config hash and the
|
||||
> artifact-definition hash for the same connection can legitimately differ (the supported
|
||||
> version-skew case). A sweep re-hashing from the definitions table would flip the slot every tick
|
||||
> against the deploy path flipping it back, **recreating the connection actor and dropping its live
|
||||
> subscriptions on a loop.**
|
||||
|
||||
That analysis is correct, and it is a property of *where the hash is computed*, not of the sweep.
|
||||
The fix is one line of ordering discipline:
|
||||
|
||||
> **Every ensure path resolves the winning configuration BEFORE hashing or pushing.**
|
||||
|
||||
`EnsureDclConnections(configJson)` no longer hands the flattened `ConnectionConfig` straight to
|
||||
`EnsureDclConnection`. It first asks the overlay whether the store owns that name; if it does, the
|
||||
*definition's* protocol/config/backup/failover fields are what get hashed and pushed. The
|
||||
artifact-deploy path pushes those same fields. Therefore, for any given name, **only one
|
||||
configuration can ever be hashed**, no matter which path calls — the two sources cannot occupy the
|
||||
slot alternately because there is only ever one candidate. The flip is not guarded against; it is
|
||||
unrepresentable.
|
||||
|
||||
The guard test `DisagreeingSources_DoNotPingPongTheConnection_AcrossManySweeps` pins exactly this
|
||||
with the two sources deliberately disagreeing: one create, zero removes, across many ticks. It
|
||||
passes before *and* after the fix — before, because nothing read the definitions at all — which is
|
||||
the point: it is a guard against the **rejected naive design**, not a fail-before test.
|
||||
|
||||
### 3.1 A second flip source, closed in the same way
|
||||
|
||||
`HandleApplyArtifactDataConnectionsToDcl` now writes the artifact's rows into the overlay in the
|
||||
same pass it pushes them. Without that there is a window between an artifact deploy and the next
|
||||
sweep read in which an instance start would resolve the name to its flattened snapshot while the
|
||||
artifact path had already pushed the definition — the same flip, arriving through a different door.
|
||||
|
||||
---
|
||||
|
||||
## 4. Removal and the tombstone
|
||||
|
||||
`_centrallyRemovedConnections` is a `HashSet<string>` of names this singleton incarnation has seen
|
||||
removed centrally. It is populated by both removal paths, which share one helper
|
||||
(`RemoveConnectionCentrally`) so they cannot drift:
|
||||
|
||||
- the artifact reconcile's `RemovedConnectionNames` (the existing path); and
|
||||
- **the sweep's disappearance detection** (new — §5).
|
||||
|
||||
`EnsureDclConnections` skips a tombstoned name. Without the tombstone the removal does not survive
|
||||
one tick: the removal evicts the hash slot, and the next instance start whose flattened config still
|
||||
names the connection re-creates it. That is not hypothetical — the sweep starts instances — and it
|
||||
is a pre-existing bug this change also closes
|
||||
(`ArtifactReconcileRemoval_IsNotResurrectedByALaterInstanceStart` fails on the old code).
|
||||
|
||||
**Tombstones are deliberately in-memory only.** Persisting them was rejected: a durable "this name
|
||||
is gone" marker is a second divergent authority over the same names, and the flattened configs
|
||||
already resurrect a removed connection across a restart today. That pre-existing semantic is
|
||||
accepted and unchanged — see the residual in §6.
|
||||
|
||||
A tombstone is cleared the moment a definition row for the name reappears (a definition row *is*
|
||||
the store re-asserting ownership), so a centrally re-added connection is creatable again.
|
||||
|
||||
---
|
||||
|
||||
## 5. The mechanism: the definitions table as an overlay on the existing reads
|
||||
|
||||
No new message, no new actor, no new cross-node path. The two durable tables the DM arbitrates over
|
||||
are read in **one off-thread task**, `ReadStoreSnapshotAsync`, used by both existing reads:
|
||||
|
||||
```
|
||||
PreStart → LoadDeployedConfigs ─┐
|
||||
├─ ReadStoreSnapshotAsync (off-thread, one task)
|
||||
InstanceSweepTick ──────────────┘ deployed_configurations
|
||||
data_connection_definitions
|
||||
│
|
||||
▼ piped back
|
||||
StartupConfigsLoaded / InstanceSweepLoaded (actor thread)
|
||||
│
|
||||
ApplyConnectionDefinitionOverlay
|
||||
│
|
||||
┌───────────────────┴───────────────────┐
|
||||
row present row disappeared
|
||||
adopt + push if live tombstone + stop actor
|
||||
```
|
||||
|
||||
Nothing blocks the actor thread on SQLite; the overlay mutation, the hash cache and the tombstone
|
||||
set are all actor-thread-confined, matching the existing `StartupConfigsLoaded` /
|
||||
`ApplyArtifactDataConnectionsToDcl` patterns.
|
||||
|
||||
**Ordering is load-bearing in both handlers.** The overlay is applied *before* the boot batch
|
||||
creates any Instance Actor and *before* the sweep's repair loop, so a connection is created from the
|
||||
winning config on its **first** push rather than created stale and corrected a tick later. On
|
||||
failover that is the difference between a clean takeover and a needless recreate on every one.
|
||||
|
||||
### 5.1 The two reads fail differently, on purpose
|
||||
|
||||
A failed **config** read propagates and re-arms the whole load (the existing S5 path, unchanged). A
|
||||
failed **definitions** read is swallowed into a `null` result, and `null` means *keep the overlay
|
||||
exactly as it is* — deliberately distinct from an empty list, which is a real answer (a deploy-only
|
||||
site, or a site after central deletes every data connection). Clearing the overlay on a transient
|
||||
read failure would hand arbitration back to the flattened snapshots for one pass and take it away
|
||||
on the next successful read: the §3 ping-pong, reintroduced through the error path.
|
||||
|
||||
### 5.2 Disappearance detection, and precisely how far it reaches
|
||||
|
||||
A name that **was** in the overlay and now has no row is a removal this incarnation can *prove*: the
|
||||
store owned it a moment ago and does not now. It is tombstoned and its connection actor stopped.
|
||||
|
||||
A name that was **never** in the overlay is indistinguishable from one the store has never owned —
|
||||
a fresh incarnation cannot tell "central deleted this" from "this site keeps its connections
|
||||
inline". It is left to the flattened config. That asymmetry is deliberate; the alternative (infer
|
||||
deletion from "the definitions table is non-empty, therefore it is the complete set") would stop
|
||||
live connections on an inference, and is recorded as a candidate follow-up in §6.
|
||||
|
||||
### 5.3 Cost
|
||||
|
||||
One extra `SELECT` over `data_connection_definitions` per sweep tick (default 60 s) and once at
|
||||
boot, on the same task as the existing deployed-configs read. Steady state is otherwise silent: the
|
||||
hash guard means an unchanged connection produces no message and no log.
|
||||
|
||||
---
|
||||
|
||||
## 6. Residuals — what this does NOT close
|
||||
|
||||
These are precise, and each is a candidate register row.
|
||||
|
||||
1. **A live connection actor is still not reconfigured in place** (premise correction 1.3). The DCL
|
||||
manager's `HandleCreateConnection` is create-only, so a changed configuration for a name whose
|
||||
actor already exists takes effect only when the connection is next created from scratch — a node
|
||||
restart, a singleton failover, or the connection being removed and re-added. This change makes
|
||||
that **honest** rather than silent: `EnsureDclConnection` now logs a Warning naming exactly that,
|
||||
once per changed configuration (the hash guard suppresses repeats, so a healthy site stays
|
||||
silent). It is not *fixed* here because the obvious fix is destructive: stopping and re-creating
|
||||
the connection actor drops every Instance Actor's tag subscription, and `InstanceActor` cancels
|
||||
its per-connection subscribe-retry timer on the first success, so **nothing re-subscribes** —
|
||||
trading "stale endpoint, data flowing" for "correct endpoint, no data until redeploy". The
|
||||
non-destructive fix is a `ReconfigureConnectionCommand` handled *inside* `DataConnectionActor`,
|
||||
reusing the adapter-swap machinery the primary→backup failover path already has
|
||||
(`CountFailureAndMaybeFailover`: dispose adapter, `_factory.Create` the new one, bump
|
||||
`_adapterGeneration`, re-enter the connect cycle, which re-issues everything from
|
||||
`_subscriptionsByInstance`). That is a Data Connection Layer change and is out of rows 89/90's
|
||||
scope.
|
||||
2. **A removal is still not durable across a restart.** Tombstones are in-memory, so a connection
|
||||
deleted centrally and then resurrected by an instance's flattened config on the next node restart
|
||||
behaves as it always has. Closing this needs a durable marker or — better — flattened configs
|
||||
that stop carrying connection bodies at all.
|
||||
3. **A delete that arrives while this node has never adopted the row is undetectable** (§5.2). The
|
||||
narrow case that remains open: the singleton moves to a node whose file already has the row
|
||||
deleted, and some instance's flattened config still names the connection. That node re-creates
|
||||
it. A stronger rule — "a non-empty definitions table is the complete set, so absence means
|
||||
deleted" — would close it, at the cost of stopping live connections on an inference about what
|
||||
central shipped; deliberately not taken.
|
||||
4. **Dual-active (a real partition under `auto-down`)** is the only routine way a definition change
|
||||
reaches a singleton that did not process the deploy itself. It is handled correctly by §5, but it
|
||||
is worth naming that this is not the common path — the common path is failover/restart (§1.2).
|
||||
|
||||
---
|
||||
|
||||
## 7. Tests
|
||||
|
||||
`tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DclConnectionArbitrationTests.cs` — 8 tests,
|
||||
asserting on the `CreateConnectionCommand` / `RemoveConnectionCommand` messages the DM sends to a
|
||||
recording stand-in for the DCL manager, and on the **endpoint inside** those commands (which is what
|
||||
distinguishes the two config sources).
|
||||
|
||||
Every test gates on `DeploymentManagerStartupProbe` for the same reason the row-78/85 suites do: a
|
||||
row written before the boot load reads storage is consumed by the boot load, and an ungated test
|
||||
would silently prove the boot load works instead of the path it names.
|
||||
|
||||
Fail-before evidence (behavioral revert: `ApplyConnectionDefinitionOverlay` made inert and the
|
||||
tombstone check disabled — i.e. the exact pre-fix arbitration) — **5 fail, 3 pass**:
|
||||
|
||||
| Test | Pre-fix |
|
||||
|---|---|
|
||||
| `DefinitionRow_WinsOverAnOlderFlattenedSnapshot_WhenTheSingletonBoots` | **FAIL** — the row-89 symptom in its failover shape (create carries the old endpoint) |
|
||||
| `DefinitionArrivingAfterBoot_IsAdoptedByTheNextSweepTick` | **FAIL** — the CDC-delivered change is never read |
|
||||
| `DefinitionRowDisappearing_StopsTheConnection_AndItIsNotResurrected` | **FAIL** — the row-90 symptom (orphan actor stays live) |
|
||||
| `ArtifactReconcileRemoval_IsNotResurrectedByALaterInstanceStart` | **FAIL** — removal undone by the next instance start |
|
||||
| `ReAddedDefinition_ClearsTheTombstone_SoTheNextInstanceStartRecreatesIt` | **FAIL** |
|
||||
| `DisagreeingSources_DoNotPingPongTheConnection_AcrossManySweeps` | pass (guard against the rejected naive design) |
|
||||
| `NoDefinitionRow_LeavesTheFlattenedConfigAuthoritative_AtBoot` | pass (deploy-only regression guard) |
|
||||
| `NoDefinitionRow_LeavesTheFlattenedConfigAuthoritative_OnTheSweepPath` | pass (deploy-only regression guard) |
|
||||
|
||||
The three passers are the point of passing both before and after: rule 2 must be byte-for-byte
|
||||
unchanged for a deploy-only site, and the ping-pong must never appear.
|
||||
|
||||
---
|
||||
|
||||
## 8. Configuration
|
||||
|
||||
**None added.** The overlay refresh rides `ScadaBridge:SiteRuntime:InstanceSweepIntervalSeconds`
|
||||
(default 60; `0` disables the sweep and therefore the post-boot half of this convergence — the
|
||||
boot-time arbitration still applies, because it hangs off the startup load, not the sweep timer).
|
||||
@@ -355,9 +355,11 @@ Each value update delivered to an Instance Actor includes:
|
||||
|
||||
## Connection Actor Model
|
||||
|
||||
> **Where connection actors come from — the Site Runtime is the only creator (register row 85)**: the DCL manager creates and removes connection actors **only** when the Deployment Manager tells it to, via `CreateConnectionCommand` / `RemoveConnectionCommand`. It never reads storage; the site's `data_connection_definitions` table has no production reader at all, and the DCL has no independent notion of "the connections this site ought to have". A connection actor's config therefore comes from the `Connections` map inside a deployed instance's **flattened config** (pushed as part of starting that instance) or from the system-wide artifact-deploy path. `HandleCreateConnection` is idempotent on the name — it warns and returns if it already holds that connection — but the Deployment Manager is the side that keeps the traffic quiet, hash-guarding on the last config pushed per name so an unchanged connection sends nothing. Row 85 was the consequence of that ownership split being incompletely honoured on the Site Runtime side: two of the four Instance-Actor start paths omitted the connection push, so instances came up whose every tag subscribe was answered `Unknown connection: <name>` here, forever. The fix is in the Site Runtime (the ensure moved into its single creation funnel) — **not** here; giving the DCL manager its own config source would create a second, divergent authority with no arbiter when the two disagree. See `Component-SiteRuntime.md` and `docs/plans/2026-08-15-instance-sweep-self-heal-design.md` §7.
|
||||
> **Where connection actors come from — the Site Runtime is the only creator (register row 85)**: the DCL manager creates and removes connection actors **only** when the Deployment Manager tells it to, via `CreateConnectionCommand` / `RemoveConnectionCommand`. It never reads storage, and the DCL has no independent notion of "the connections this site ought to have" — the site's `data_connection_definitions` table is read by the **Deployment Manager**, never here (see the arbitration note below; it had no reader at all until register rows 89/90). A connection actor's config therefore comes from the `Connections` map inside a deployed instance's **flattened config** (pushed as part of starting that instance) or from the system-wide artifact-deploy path. `HandleCreateConnection` is idempotent on the name — it warns and returns if it already holds that connection — but the Deployment Manager is the side that keeps the traffic quiet, hash-guarding on the last config pushed per name so an unchanged connection sends nothing. Row 85 was the consequence of that ownership split being incompletely honoured on the Site Runtime side: two of the four Instance-Actor start paths omitted the connection push, so instances came up whose every tag subscribe was answered `Unknown connection: <name>` here, forever. The fix is in the Site Runtime (the ensure moved into its single creation funnel) — **not** here; giving the DCL manager its own config source would create a second, divergent authority with no arbiter when the two disagree. See `Component-SiteRuntime.md` and `docs/plans/2026-08-15-instance-sweep-self-heal-design.md` §7.
|
||||
>
|
||||
> Two known residual gaps of the same family, **not** closed by row 85's fix: `data_connection_definitions` is one of LocalDb's ten replicated tables, so a **connection-only** artifact deploy replicates to the peer node — but only the node that received the deploy pushes it to its DCL, so the peer keeps serving the old endpoint until an instance is redeployed or the node restarts. The removal side is the mirror image: a connection deleted on the peer leaves a live connection actor on this node.
|
||||
> **The two residual gaps of that family are now closed — but not where the register said they were (rows 89 + 90, 2026-08-17)**: the peer node's DCL was never "serving the old endpoint". Only the node running the `site-{SiteId}` Deployment Manager singleton ever receives a `CreateConnectionCommand`, so the standby's manager is **empty**. The divergence surfaces when the singleton's *next* incarnation (failover, or an ordinary restart — no replication required) rebuilds every connection from the `Connections` snapshots frozen into the instances' flattened configs, because the per-name hash cache that recorded the artifact deploy's change is in-memory and dies with the incarnation. The fix lives entirely in the Site Runtime, as an **arbitration rule** — the definitions store wins wherever it has an opinion, resolved *before* the hash is computed so the two sources cannot fight over the same per-name slot — plus in-memory tombstones so a centrally removed connection is not resurrected by a flattened config. Nothing changes here. See `Component-SiteRuntime.md` → *DCL connection arbitration* and `docs/plans/2026-08-17-dcl-connection-arbitration-decision.md`.
|
||||
>
|
||||
> **What DOES remain open here, and it is this component's:** `HandleCreateConnection` is **create-only**, so a re-issued create for a name it already holds is a no-op — a live connection actor never adopts a changed endpoint/credential/backup/failover-count, on any node. The Deployment Manager's documented "a changed config re-issues the create so the DCL adopts it" was therefore never true; it now logs a Warning saying so, once per changed configuration, and the change takes effect only when the connection is next created from scratch (node restart, singleton failover, or remove-then-re-add). The naive fix — stop the actor and re-create it — is destructive: every Instance Actor's tag subscriptions are lost and `InstanceActor` cancels its per-connection subscribe-retry timer on the first success, so **nothing re-subscribes**. The non-destructive fix is a `ReconfigureConnectionCommand` handled *inside* `DataConnectionActor`, reusing the adapter-swap machinery the primary→backup failover path already has (dispose adapter → `_factory.Create` the new one → bump `_adapterGeneration` → re-enter the connect cycle, which re-issues everything from `_subscriptionsByInstance`).
|
||||
|
||||
Each data connection is managed by a dedicated connection actor that uses the Akka.NET **Become/Stash** pattern to model its lifecycle as a state machine:
|
||||
|
||||
|
||||
@@ -108,7 +108,19 @@ flowchart TD
|
||||
>
|
||||
> Guards, all shared with the row-67 handler: a **disabled** instance is never started; the sweep **defers to any deploy pipeline that owns the instance** (compile warm in flight, predecessor still terminating, deployment awaiting its persistence/init join) so it can never interleave with a real redeploy; it is **inert until boot-time Instance Actor creation has finished**, so a tick between staggered batches does not "heal" instances the boot load is about to create; it makes **one repair attempt per (instance, revision) pair**, which is what stops a periodic self-heal becoming a restart loop for an actor that dies during init or a stored config the compile gate rejects (a rejection leaves the running instance untouched — the gate returns before anything is stopped); and reads never overlap. A sweep that finds nothing is **completely silent**. A repair emits the same Warning `deployment` site-event pattern as row 67, naming the trigger (`deployment-manager sweep self-heal`) so it is distinguishable from a startup-reconcile heal, and a stale-revision restart names **both** revisions; a rejected restart adds an Error event. The synthetic redeploy replies to the singleton itself rather than to `NoSender`, because dead letters are a monitored site health metric.
|
||||
|
||||
> **Starting an Instance Actor also creates its DCL connections — the ensure lives in the creation funnel, not at call sites (register row 85)**: a DCL connection actor is **never** created from the `data_connection_definitions` table (nothing in production reads it). It exists only as a side effect of starting an instance, built from the `Connections` map inside that instance's **own flattened config** via `EnsureDclConnections` — plus the separate system-wide artifact-deploy path. That ensure used to be each *caller's* responsibility, and only two of the four callers made it (the boot-time staggered batch and `ApplyDeployment`). The two that did not were **`StartStoredInstance`** — the path shared by the row-67 ensure-started handler and the row-78 sweep — and **`HandleEnableResult`**. Both therefore produced a running instance whose every tag subscribe was answered `Unknown connection: <name>` by the DCL manager, indefinitely, because nothing else on the node would ever create that connection actor. Proven live 2026-08-16: a replication-seeded node's instances were correctly started by the sweep and then failed every subscribe for 6+ minutes until a second pair restart let the boot load create the connections. **It was never replication-specific** — the boot-time load reads only *enabled* configs, so a node that boots with an instance disabled never creates its connections either, and enabling it afterwards hit the identical dead end on an ordinary unreplicated site. `EnsureDclConnections` now sits at the top of **`CreateInstanceActor`**, already the single Instance Actor creation funnel (it is where the running revision is stamped for the same reason), so all four paths get it and a fifth cannot forget. It is placed *before* the already-exists guard so the two paths that already called it keep their exact previous semantics, and it stays silent in steady state because `EnsureDclConnection` is hash-guarded on the last config pushed per connection name. Deliberately **not** done: a sweep pass over `data_connection_definitions` — it would not have fixed row 85 (that table is empty on a site that has only ever received instance deploys), and re-hashing from it would fight the deploy path over the same per-name hash slot, recreating connection actors and dropping live subscriptions on a loop. See `docs/plans/2026-08-15-instance-sweep-self-heal-design.md` §7.
|
||||
> **Starting an Instance Actor also creates its DCL connections — the ensure lives in the creation funnel, not at call sites (register row 85)**: a DCL connection actor is **never** created from the `data_connection_definitions` table — that stayed true when rows 89/90 gave the table a reader, because the overlay it feeds is an *arbiter*, not a creation source, and at the time of row 85 nothing read the table at all. A connection actor exists only as a side effect of starting an instance, built from the `Connections` map inside that instance's **own flattened config** via `EnsureDclConnections` — plus the separate system-wide artifact-deploy path. That ensure used to be each *caller's* responsibility, and only two of the four callers made it (the boot-time staggered batch and `ApplyDeployment`). The two that did not were **`StartStoredInstance`** — the path shared by the row-67 ensure-started handler and the row-78 sweep — and **`HandleEnableResult`**. Both therefore produced a running instance whose every tag subscribe was answered `Unknown connection: <name>` by the DCL manager, indefinitely, because nothing else on the node would ever create that connection actor. Proven live 2026-08-16: a replication-seeded node's instances were correctly started by the sweep and then failed every subscribe for 6+ minutes until a second pair restart let the boot load create the connections. **It was never replication-specific** — the boot-time load reads only *enabled* configs, so a node that boots with an instance disabled never creates its connections either, and enabling it afterwards hit the identical dead end on an ordinary unreplicated site. `EnsureDclConnections` now sits at the top of **`CreateInstanceActor`**, already the single Instance Actor creation funnel (it is where the running revision is stamped for the same reason), so all four paths get it and a fifth cannot forget. It is placed *before* the already-exists guard so the two paths that already called it keep their exact previous semantics, and it stays silent in steady state because `EnsureDclConnection` is hash-guarded on the last config pushed per connection name. Deliberately **not** done at the time: a sweep pass over `data_connection_definitions` — it would not have fixed row 85 (that table is empty on a site that has only ever received instance deploys), and re-hashing from it would fight the deploy path over the same per-name hash slot, recreating connection actors and dropping live subscriptions on a loop. See `docs/plans/2026-08-15-instance-sweep-self-heal-design.md` §7. That pass is now present, in a form that resolves the conflict rather than creating it — see **DCL connection arbitration** immediately below.
|
||||
|
||||
> **DCL connection arbitration — the definitions store wins wherever it has an opinion (register rows 89 + 90)**: `data_connection_definitions` now has a production reader. The rule is: **a connection name with a row in that table is configured from that row; a name with no row is configured from the instance's flattened `Connections` entry, exactly as before.** Rule 2 is not a fallback — it is the correct answer for a deploy-only site, where the table is legitimately empty while every connection that matters lives inline in the instance configs.
|
||||
>
|
||||
> **The register's premise needed two corrections, and they change what the fix is.** First, the peer node's DCL is not "serving the old endpoint" — it is **empty**. The Deployment Manager is a `site-{SiteId}`-scoped singleton and the *only* sender of `CreateConnectionCommand`/`RemoveConnectionCommand`, sending to its own node's `dcl-manager` (which exists on every node but is populated only where the singleton runs). The real defect is that `_createdConnections`, the per-name hash cache, is **in-memory and per singleton incarnation**: a restart or failover rebuilds every connection from the `Connections` snapshot frozen into each instance's flattened config at *that instance's* deploy time, and a newer connection-only artifact deploy is silently discarded. **That makes row 89 create-time, not message-delivery, and not replication-specific** — the identical loss happens on an unreplicated single-node restart. Second, `EnsureDclConnection`'s documented "a changed config re-issues the create so the DCL adopts it" was **false**: `DataConnectionManagerActor.HandleCreateConnection` is create-only (warns and returns for a name it already holds), so the artifact deploy's live push has never worked on the deploy-handling node either. That is now surfaced honestly rather than silently — see the residual below.
|
||||
>
|
||||
> **Mechanism.** `data_connection_definitions` is read in the *same* off-thread task as `deployed_configurations` (`ReadStoreSnapshotAsync`), by both the boot load and every sweep tick, and applied to an actor-thread-confined **arbitration overlay** *before* the boot batch creates any Instance Actor and *before* the sweep's repair loop — so a connection is built from the winning config on its **first** push rather than created stale and corrected a tick later. The overlay is an **arbiter, not a creation source**: a definition row naming a connection no deployed instance references creates nothing. A CDC-delivered definition change therefore converges within one sweep tick on whichever node runs the singleton, with **no new cross-node message**. A definitions read that *fails* yields `null` and leaves the overlay untouched — deliberately distinct from an empty list, which is a real answer; clearing on a transient failure would reintroduce the ping-pong through the error path.
|
||||
>
|
||||
> **The ping-pong is killed by keying the hash on the winner.** The conflict §7.4 identified is a property of *where the hash is computed*, not of the sweep: every ensure path now resolves the winning configuration **before** hashing or pushing, so for any name only one configuration can ever occupy the hash slot regardless of which path calls. The flip is not guarded against — it is unrepresentable. `HandleApplyArtifactDataConnectionsToDcl` writes its rows into the overlay in the same pass it pushes them, closing the same flip arriving through the artifact-deploy door.
|
||||
>
|
||||
> **Removal (row 90).** Both removal paths share one helper and now record a **tombstone** for the incarnation: the artifact reconcile's `RemovedConnectionNames`, and the sweep noticing that a name it had adopted no longer has a row (a removal it can *prove* — the store owned it a tick ago and does not now). `EnsureDclConnections` skips a tombstoned name, so a removal survives the next instance start instead of being undone by it — a pre-existing bug, since the removal evicted the hash slot but left no memory of why. Tombstones are **in-memory only** by decision: a durable marker would be a second divergent authority, and flattened configs already resurrect a removed connection across a restart. A definition row reappearing clears the tombstone.
|
||||
>
|
||||
> **Residuals**, all recorded precisely in the decision memo: a **live** connection actor is still not reconfigured in place (the non-destructive fix is a `ReconfigureConnectionCommand` inside `DataConnectionActor` reusing the failover adapter-swap, because stopping and re-creating the actor drops every tag subscription and `InstanceActor` cancels its subscribe-retry timer on first success, so nothing re-subscribes) — this is now announced with a Warning naming exactly that, once per changed configuration; a removal is not durable across a restart; and a delete is undetectable on a node that never adopted the row. See `docs/plans/2026-08-17-dcl-connection-arbitration-decision.md`.
|
||||
|
||||
> **Per-instance compilation during staggered startup — CLOSED (WP3.1)**: this was a deferred optimization (P6) affecting failover time-to-recover; each Instance Actor Roslyn-compiled its own scripts inside its own start, serialising a site's whole recovery behind compilation. Each staggered startup batch now runs a **pre-warm step**: before the batch's Instance Actors are created, the Deployment Manager compiles that batch's distinct script bodies and trigger expressions on a background task and pipes `BatchCompileWarmed` back, so every `PreStart` compile is a cache hit. Instance Actor start still calls the compilation service — it remains the correctness backstop — but in every warmed path those calls are memoised lookups. Cost: one extra message per batch.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user