fix(siteruntime): arbitrate DCL connection config — definitions store wins (rows 89 + 90)

Closes deferred-work-register rows 89 and 90, and corrects their premise twice.

THE RULE
The `data_connection_definitions` store wins wherever it has an opinion; an
instance's flattened `Connections` entry is a deploy-time snapshot and is
authoritative only for names the store has never owned (the deploy-only site,
where that table is legitimately empty).

PREMISE CORRECTION 1 — the peer's DCL is EMPTY, not stale
`DeploymentManagerActor` is a `site-{SiteId}`-scoped singleton and the only
sender of Create/RemoveConnectionCommand, to its OWN node's `dcl-manager`
(which exists on every node but is populated only where the singleton runs).
So the standby serves nothing. The real defect is that `_createdConnections`
is in-memory per singleton incarnation: a restart or failover rebuilds every
connection from the flattened snapshots and silently discards a newer
connection-only artifact deploy. Row 89 is therefore create-time, not
message-delivery, and NOT replication-specific — the same loss happens on an
unreplicated single-node restart.

PREMISE CORRECTION 2 — a changed config never reached a LIVE connection either
`DataConnectionManagerActor.HandleCreateConnection` is create-only (warns and
returns on a name it holds), so `EnsureDclConnection`'s documented "re-issues
the create so the DCL adopts the new configuration" has never been true on any
node. Now surfaced with a Warning saying exactly that, once per changed config.
Not fixed here: stop-and-recreate drops every tag subscription and InstanceActor
cancels its subscribe-retry timer on first success, so nothing re-subscribes.

MECHANISM
`data_connection_definitions` gains its first production reader. It is read on
the SAME off-thread task as the deployed configs (`ReadStoreSnapshotAsync`), by
the boot load and every sweep tick, into an actor-thread-confined arbitration
overlay applied BEFORE the boot batch and BEFORE the sweep's repair loop. A
CDC-delivered definition change converges within one tick on whichever node runs
the singleton — no new cross-node message. A failed definitions read yields null
and leaves the overlay untouched (distinct from an empty list, which is a real
answer); clearing it would reintroduce the ping-pong through the error path.

THE PING-PONG, KILLED BY CONSTRUCTION
§7.4 of the sweep memo rejected a store-level pass because two sources would
flip the single per-name hash slot, recreating the actor every tick. That is a
property of WHERE the hash is computed: every ensure path now resolves the
winner BEFORE hashing, so only one configuration can ever occupy the slot. The
artifact-deploy path writes its rows into the overlay in the same pass, closing
the same flip arriving through the other door.

REMOVAL (row 90)
Both removal paths share `RemoveConnectionCentrally` and record an in-memory
tombstone, so a centrally removed connection is not resurrected by the next
instance start — a pre-existing bug. Tombstones are deliberately not persisted;
a definition row reappearing clears one.

TESTS
tests/.../Actors/DclConnectionArbitrationTests.cs — 8 tests. Fail-before
(behavioral revert: overlay inert + tombstone check disabled): 5 fail, 3 pass;
the 3 passers are the two deploy-only regression guards and the ping-pong guard.

DOCS
New decision memo docs/plans/2026-08-17-dcl-connection-arbitration-decision.md
(topology, rule, ping-pong mechanism, tombstones, residuals); Component-SiteRuntime,
Component-DataConnectionLayer + their docs/components mirrors; the §7.4 residual in
the sweep memo marked closed with its premise corrected.

docs/requirements/Component-DataConnectionLayer.md

@@ -355,9 +355,11 @@ Each value update delivered to an Instance Actor includes:
 
 ## Connection Actor Model
 
-> **Where connection actors come from — the Site Runtime is the only creator (register row 85)**: the DCL manager creates and removes connection actors **only** when the Deployment Manager tells it to, via `CreateConnectionCommand` / `RemoveConnectionCommand`. It never reads storage; the site's `data_connection_definitions` table has no production reader at all, and the DCL has no independent notion of "the connections this site ought to have". A connection actor's config therefore comes from the `Connections` map inside a deployed instance's **flattened config** (pushed as part of starting that instance) or from the system-wide artifact-deploy path. `HandleCreateConnection` is idempotent on the name — it warns and returns if it already holds that connection — but the Deployment Manager is the side that keeps the traffic quiet, hash-guarding on the last config pushed per name so an unchanged connection sends nothing. Row 85 was the consequence of that ownership split being incompletely honoured on the Site Runtime side: two of the four Instance-Actor start paths omitted the connection push, so instances came up whose every tag subscribe was answered `Unknown connection: <name>` here, forever. The fix is in the Site Runtime (the ensure moved into its single creation funnel) — **not** here; giving the DCL manager its own config source would create a second, divergent authority with no arbiter when the two disagree. See `Component-SiteRuntime.md` and `docs/plans/2026-08-15-instance-sweep-self-heal-design.md` §7.
+> **Where connection actors come from — the Site Runtime is the only creator (register row 85)**: the DCL manager creates and removes connection actors **only** when the Deployment Manager tells it to, via `CreateConnectionCommand` / `RemoveConnectionCommand`. It never reads storage, and the DCL has no independent notion of "the connections this site ought to have" — the site's `data_connection_definitions` table is read by the **Deployment Manager**, never here (see the arbitration note below; it had no reader at all until register rows 89/90). A connection actor's config therefore comes from the `Connections` map inside a deployed instance's **flattened config** (pushed as part of starting that instance) or from the system-wide artifact-deploy path. `HandleCreateConnection` is idempotent on the name — it warns and returns if it already holds that connection — but the Deployment Manager is the side that keeps the traffic quiet, hash-guarding on the last config pushed per name so an unchanged connection sends nothing. Row 85 was the consequence of that ownership split being incompletely honoured on the Site Runtime side: two of the four Instance-Actor start paths omitted the connection push, so instances came up whose every tag subscribe was answered `Unknown connection: <name>` here, forever. The fix is in the Site Runtime (the ensure moved into its single creation funnel) — **not** here; giving the DCL manager its own config source would create a second, divergent authority with no arbiter when the two disagree. See `Component-SiteRuntime.md` and `docs/plans/2026-08-15-instance-sweep-self-heal-design.md` §7.
 >
-> Two known residual gaps of the same family, **not** closed by row 85's fix: `data_connection_definitions` is one of LocalDb's ten replicated tables, so a **connection-only** artifact deploy replicates to the peer node — but only the node that received the deploy pushes it to its DCL, so the peer keeps serving the old endpoint until an instance is redeployed or the node restarts. The removal side is the mirror image: a connection deleted on the peer leaves a live connection actor on this node.
+> **The two residual gaps of that family are now closed — but not where the register said they were (rows 89 + 90, 2026-08-17)**: the peer node's DCL was never "serving the old endpoint". Only the node running the `site-{SiteId}` Deployment Manager singleton ever receives a `CreateConnectionCommand`, so the standby's manager is **empty**. The divergence surfaces when the singleton's *next* incarnation (failover, or an ordinary restart — no replication required) rebuilds every connection from the `Connections` snapshots frozen into the instances' flattened configs, because the per-name hash cache that recorded the artifact deploy's change is in-memory and dies with the incarnation. The fix lives entirely in the Site Runtime, as an **arbitration rule** — the definitions store wins wherever it has an opinion, resolved *before* the hash is computed so the two sources cannot fight over the same per-name slot — plus in-memory tombstones so a centrally removed connection is not resurrected by a flattened config. Nothing changes here. See `Component-SiteRuntime.md` → *DCL connection arbitration* and `docs/plans/2026-08-17-dcl-connection-arbitration-decision.md`.
+>
+> **What DOES remain open here, and it is this component's:** `HandleCreateConnection` is **create-only**, so a re-issued create for a name it already holds is a no-op — a live connection actor never adopts a changed endpoint/credential/backup/failover-count, on any node. The Deployment Manager's documented "a changed config re-issues the create so the DCL adopts it" was therefore never true; it now logs a Warning saying so, once per changed configuration, and the change takes effect only when the connection is next created from scratch (node restart, singleton failover, or remove-then-re-add). The naive fix — stop the actor and re-create it — is destructive: every Instance Actor's tag subscriptions are lost and `InstanceActor` cancels its per-connection subscribe-retry timer on the first success, so **nothing re-subscribes**. The non-destructive fix is a `ReconfigureConnectionCommand` handled *inside* `DataConnectionActor`, reusing the adapter-swap machinery the primary→backup failover path already has (dispose adapter → `_factory.Create` the new one → bump `_adapterGeneration` → re-enter the connect cycle, which re-issues everything from `_subscriptionsByInstance`).
 
 Each data connection is managed by a dedicated connection actor that uses the Akka.NET **Become/Stash** pattern to model its lifecycle as a state machine:
 

docs/requirements/Component-SiteRuntime.md

@@ -108,7 +108,19 @@ flowchart TD
 >
 > Guards, all shared with the row-67 handler: a **disabled** instance is never started; the sweep **defers to any deploy pipeline that owns the instance** (compile warm in flight, predecessor still terminating, deployment awaiting its persistence/init join) so it can never interleave with a real redeploy; it is **inert until boot-time Instance Actor creation has finished**, so a tick between staggered batches does not "heal" instances the boot load is about to create; it makes **one repair attempt per (instance, revision) pair**, which is what stops a periodic self-heal becoming a restart loop for an actor that dies during init or a stored config the compile gate rejects (a rejection leaves the running instance untouched — the gate returns before anything is stopped); and reads never overlap. A sweep that finds nothing is **completely silent**. A repair emits the same Warning `deployment` site-event pattern as row 67, naming the trigger (`deployment-manager sweep self-heal`) so it is distinguishable from a startup-reconcile heal, and a stale-revision restart names **both** revisions; a rejected restart adds an Error event. The synthetic redeploy replies to the singleton itself rather than to `NoSender`, because dead letters are a monitored site health metric.
 
-> **Starting an Instance Actor also creates its DCL connections — the ensure lives in the creation funnel, not at call sites (register row 85)**: a DCL connection actor is **never** created from the `data_connection_definitions` table (nothing in production reads it). It exists only as a side effect of starting an instance, built from the `Connections` map inside that instance's **own flattened config** via `EnsureDclConnections` — plus the separate system-wide artifact-deploy path. That ensure used to be each *caller's* responsibility, and only two of the four callers made it (the boot-time staggered batch and `ApplyDeployment`). The two that did not were **`StartStoredInstance`** — the path shared by the row-67 ensure-started handler and the row-78 sweep — and **`HandleEnableResult`**. Both therefore produced a running instance whose every tag subscribe was answered `Unknown connection: <name>` by the DCL manager, indefinitely, because nothing else on the node would ever create that connection actor. Proven live 2026-08-16: a replication-seeded node's instances were correctly started by the sweep and then failed every subscribe for 6+ minutes until a second pair restart let the boot load create the connections. **It was never replication-specific** — the boot-time load reads only *enabled* configs, so a node that boots with an instance disabled never creates its connections either, and enabling it afterwards hit the identical dead end on an ordinary unreplicated site. `EnsureDclConnections` now sits at the top of **`CreateInstanceActor`**, already the single Instance Actor creation funnel (it is where the running revision is stamped for the same reason), so all four paths get it and a fifth cannot forget. It is placed *before* the already-exists guard so the two paths that already called it keep their exact previous semantics, and it stays silent in steady state because `EnsureDclConnection` is hash-guarded on the last config pushed per connection name. Deliberately **not** done: a sweep pass over `data_connection_definitions` — it would not have fixed row 85 (that table is empty on a site that has only ever received instance deploys), and re-hashing from it would fight the deploy path over the same per-name hash slot, recreating connection actors and dropping live subscriptions on a loop. See `docs/plans/2026-08-15-instance-sweep-self-heal-design.md` §7.
+> **Starting an Instance Actor also creates its DCL connections — the ensure lives in the creation funnel, not at call sites (register row 85)**: a DCL connection actor is **never** created from the `data_connection_definitions` table — that stayed true when rows 89/90 gave the table a reader, because the overlay it feeds is an *arbiter*, not a creation source, and at the time of row 85 nothing read the table at all. A connection actor exists only as a side effect of starting an instance, built from the `Connections` map inside that instance's **own flattened config** via `EnsureDclConnections` — plus the separate system-wide artifact-deploy path. That ensure used to be each *caller's* responsibility, and only two of the four callers made it (the boot-time staggered batch and `ApplyDeployment`). The two that did not were **`StartStoredInstance`** — the path shared by the row-67 ensure-started handler and the row-78 sweep — and **`HandleEnableResult`**. Both therefore produced a running instance whose every tag subscribe was answered `Unknown connection: <name>` by the DCL manager, indefinitely, because nothing else on the node would ever create that connection actor. Proven live 2026-08-16: a replication-seeded node's instances were correctly started by the sweep and then failed every subscribe for 6+ minutes until a second pair restart let the boot load create the connections. **It was never replication-specific** — the boot-time load reads only *enabled* configs, so a node that boots with an instance disabled never creates its connections either, and enabling it afterwards hit the identical dead end on an ordinary unreplicated site. `EnsureDclConnections` now sits at the top of **`CreateInstanceActor`**, already the single Instance Actor creation funnel (it is where the running revision is stamped for the same reason), so all four paths get it and a fifth cannot forget. It is placed *before* the already-exists guard so the two paths that already called it keep their exact previous semantics, and it stays silent in steady state because `EnsureDclConnection` is hash-guarded on the last config pushed per connection name. Deliberately **not** done at the time: a sweep pass over `data_connection_definitions` — it would not have fixed row 85 (that table is empty on a site that has only ever received instance deploys), and re-hashing from it would fight the deploy path over the same per-name hash slot, recreating connection actors and dropping live subscriptions on a loop. See `docs/plans/2026-08-15-instance-sweep-self-heal-design.md` §7. That pass is now present, in a form that resolves the conflict rather than creating it — see **DCL connection arbitration** immediately below.
+
+> **DCL connection arbitration — the definitions store wins wherever it has an opinion (register rows 89 + 90)**: `data_connection_definitions` now has a production reader. The rule is: **a connection name with a row in that table is configured from that row; a name with no row is configured from the instance's flattened `Connections` entry, exactly as before.** Rule 2 is not a fallback — it is the correct answer for a deploy-only site, where the table is legitimately empty while every connection that matters lives inline in the instance configs.
+>
+> **The register's premise needed two corrections, and they change what the fix is.** First, the peer node's DCL is not "serving the old endpoint" — it is **empty**. The Deployment Manager is a `site-{SiteId}`-scoped singleton and the *only* sender of `CreateConnectionCommand`/`RemoveConnectionCommand`, sending to its own node's `dcl-manager` (which exists on every node but is populated only where the singleton runs). The real defect is that `_createdConnections`, the per-name hash cache, is **in-memory and per singleton incarnation**: a restart or failover rebuilds every connection from the `Connections` snapshot frozen into each instance's flattened config at *that instance's* deploy time, and a newer connection-only artifact deploy is silently discarded. **That makes row 89 create-time, not message-delivery, and not replication-specific** — the identical loss happens on an unreplicated single-node restart. Second, `EnsureDclConnection`'s documented "a changed config re-issues the create so the DCL adopts it" was **false**: `DataConnectionManagerActor.HandleCreateConnection` is create-only (warns and returns for a name it already holds), so the artifact deploy's live push has never worked on the deploy-handling node either. That is now surfaced honestly rather than silently — see the residual below.
+>
+> **Mechanism.** `data_connection_definitions` is read in the *same* off-thread task as `deployed_configurations` (`ReadStoreSnapshotAsync`), by both the boot load and every sweep tick, and applied to an actor-thread-confined **arbitration overlay** *before* the boot batch creates any Instance Actor and *before* the sweep's repair loop — so a connection is built from the winning config on its **first** push rather than created stale and corrected a tick later. The overlay is an **arbiter, not a creation source**: a definition row naming a connection no deployed instance references creates nothing. A CDC-delivered definition change therefore converges within one sweep tick on whichever node runs the singleton, with **no new cross-node message**. A definitions read that *fails* yields `null` and leaves the overlay untouched — deliberately distinct from an empty list, which is a real answer; clearing on a transient failure would reintroduce the ping-pong through the error path.
+>
+> **The ping-pong is killed by keying the hash on the winner.** The conflict §7.4 identified is a property of *where the hash is computed*, not of the sweep: every ensure path now resolves the winning configuration **before** hashing or pushing, so for any name only one configuration can ever occupy the hash slot regardless of which path calls. The flip is not guarded against — it is unrepresentable. `HandleApplyArtifactDataConnectionsToDcl` writes its rows into the overlay in the same pass it pushes them, closing the same flip arriving through the artifact-deploy door.
+>
+> **Removal (row 90).** Both removal paths share one helper and now record a **tombstone** for the incarnation: the artifact reconcile's `RemovedConnectionNames`, and the sweep noticing that a name it had adopted no longer has a row (a removal it can *prove* — the store owned it a tick ago and does not now). `EnsureDclConnections` skips a tombstoned name, so a removal survives the next instance start instead of being undone by it — a pre-existing bug, since the removal evicted the hash slot but left no memory of why. Tombstones are **in-memory only** by decision: a durable marker would be a second divergent authority, and flattened configs already resurrect a removed connection across a restart. A definition row reappearing clears the tombstone.
+>
+> **Residuals**, all recorded precisely in the decision memo: a **live** connection actor is still not reconfigured in place (the non-destructive fix is a `ReconfigureConnectionCommand` inside `DataConnectionActor` reusing the failover adapter-swap, because stopping and re-creating the actor drops every tag subscription and `InstanceActor` cancels its subscribe-retry timer on first success, so nothing re-subscribes) — this is now announced with a Warning naming exactly that, once per changed configuration; a removal is not durable across a restart; and a delete is undetectable on a node that never adopted the row. See `docs/plans/2026-08-17-dcl-connection-arbitration-decision.md`.
 
 > **Per-instance compilation during staggered startup — CLOSED (WP3.1)**: this was a deferred optimization (P6) affecting failover time-to-recover; each Instance Actor Roslyn-compiled its own scripts inside its own start, serialising a site's whole recovery behind compilation. Each staggered startup batch now runs a **pre-warm step**: before the batch's Instance Actors are created, the Deployment Manager compiles that batch's distinct script bodies and trigger expressions on a background task and pipes `BatchCompileWarmed` back, so every `PreStart` compile is a cache hit. Instance Actor start still calls the compilation service — it remains the correctness backstop — but in every warmed path those calls are memoised lookups. Cost: one extra message per batch.