feat(dcl+siteruntime): reconfigure a LIVE data connection in place — register row 97

`DataConnectionManagerActor.HandleCreateConnection` is create-only (warns and
returns for a name it already holds), so `EnsureDclConnection`'s documented
"a changed config re-issues the create so the DCL adopts it" was never true on
ANY node: a corrected endpoint/credential/backup/failover-count took effect only
when the connection was next built from scratch (node restart, singleton
failover, remove-then-re-add), while the DM's hash cache recorded it as applied.
Rows 89/90 made that honest with a Warning; this closes it.

New `ReconfigureConnectionCommand` (Commons, same shape as the create):

- `DeploymentManagerActor.EnsureDclConnection` now has three outcomes off its one
  per-name hash slot — unchanged -> nothing at all, new -> create, CHANGED ->
  reconfigure — and logs "Reconfigured DCL connection {Connection}
  (protocol={Protocol}) in place" at Information in place of the old Warning. The
  winner-resolution / arbitration / tombstone logic is untouched; only what
  happens after a change is detected changed.
- `DataConnectionManagerActor` FORWARDS the command to the owning connection
  actor, falling back to a create for a name it does not hold, so the command is
  safe whatever the manager's state.
- `DataConnectionActor` handles it in ALL THREE states and never stashes it — a
  connection pointed at a wrong or dead endpoint lives in Connecting/Reconnecting,
  which is exactly when a correction arrives; stashed it would wait on the broken
  configuration forever. Handling replaces protocol/primary/backup/failover-count,
  re-arms endpoint selection to Primary (the new config's primary is the intent,
  whatever the old config had failed over to), and calls the new shared
  `SwapAdapter` helper — now used by both failover sites too, so the swap exists
  once: unhook + fire-and-forget dispose -> _factory.Create -> rewire -> bump
  _adapterGeneration LAST. `BecomeReconnecting(afterReconfigure: true)` then
  reconnects immediately (StartSingleTimer replaces any armed reconnect, so no
  racing attempts) with the unstable-disconnect accounting and failover check
  suppressed — the link was dropped on purpose, and counting it as an unstable
  cycle could trip a failover onto the NEW backup before the new primary had been
  tried once. ReSubscribeAll/ReSubscribeAllAlarms then re-issue everything from
  the untouched subscription registries. Idempotent: a reconfigure carrying the
  configuration already in force is a no-op.

Stop-and-recreate was rejected as destructive: subscriptions live inside
DataConnectionActor and InstanceActor cancels its subscribe-retry timer on first
success, so a rebuilt actor would have no subscribers and nothing would ever
re-subscribe.

Tests: new DataConnectionReconfigureTests (7; 5 fail / 2 pass against a
behavioural revert). DclConnectionArbitrationTests gains two row-97 tests and its
CDC-arrival test now asserts the second push is a reconfigure; its ping-pong
guard also asserts zero reconfigures, since a flip would now show up there.
SiteRuntime-010 renamed to ..._ReissuesReconfigureCommand. 3 fail against a
reverted EnsureDclConnection. DataConnectionLayer.Tests 327/327,
SiteRuntime.Tests 643/643, solution build 0 warnings.

Docs: Component-DataConnectionLayer.md gains an "In-place reconfiguration"
section under Endpoint Redundancy; Component-SiteRuntime.md, and the
docs/components/ mirrors, updated; the arbitration decision memo gains an
appended §9 postscript resolving its residual 1 (residuals 2 and 3 stay open).
This commit is contained in:
Joseph Doherty
2026-08-17 06:19:13 -04:00
parent dcfcf822af
commit 7c1268d85c
12 changed files with 943 additions and 126 deletions
@@ -147,6 +147,20 @@ flowchart TD
See [`2026-03-22-primary-backup-data-connections-design.md`](../plans/2026-03-22-primary-backup-data-connections-design.md) for the full design.
### In-place reconfiguration (register row 97)
A `ReconfigureConnectionCommand` replaces a **live** connection's configuration without stopping the connection actor. It is the same adapter swap the failover path above performs — the two share one `SwapAdapter` helper — so the diagram applies verbatim from *dispose adapter* onwards:
1. The command lands on the actor and **is never stashed**, in *any* of the three states. That is load-bearing, not tidiness: a connection whose configured endpoint is wrong or dead spends its life looping in `Connecting` or `Reconnecting`, which is precisely when a corrected configuration is sent. Stashed, it would wait for the *old* configuration to succeed — i.e. forever.
2. Protocol, primary/backup details and failover retry count are replaced wholesale, and **endpoint selection is re-armed from the top**: whatever endpoint the previous configuration had failed over to, the new configuration's primary is the operator's intent, so `ActiveEndpoint` and both failure counters reset.
3. `SwapAdapter` unhooks and disposes the outgoing adapter (fire-and-forget, never awaited on the actor thread), builds the replacement via `IDataConnectionFactory.Create` — which is why a **protocol change** is supported too — and bumps `_adapterGeneration` **last**, so every callback and in-flight background task belonging to the outgoing adapter is recognised as stale and dropped rather than delivered as if it came from the new endpoint.
4. The actor re-enters the reconnect cycle immediately (no `ReconnectInterval` wait — there is nothing to wait for; `StartSingleTimer` replaces any reconnect already armed, so a reconfigure landing mid-backoff cannot leave two attempts racing). The unstable-disconnect accounting and the failover check are **suppressed** for this transition: the link was dropped on purpose, and counting it as an unstable cycle could trip a failover onto the new backup before the new primary had been tried once.
5. On connect, `ReSubscribeAll` / `ReSubscribeAllAlarms` re-issue everything from `_subscriptionsByInstance` and `_alarmSourceSubscribers`, which the swap never touched. **Subscribers do nothing and know nothing** — the tags go bad quality across the swap, exactly as they do on a failover, and come back Good.
**Accepted consequences**, all identical to a failover: in-flight writes against the outgoing adapter may fail (returned synchronously to the calling script, as always), and tag quality dips for the duration of the reconnect.
**Idempotent.** A reconfigure carrying the configuration already in force is a no-op. The sender is hash-guarded and does not send one, but a swap is destructive enough that a duplicate must never cost a reconnect.
## Connection Configuration Reference
All settings are parsed from the data connection's configuration JSON dictionaries (`PrimaryConfiguration` and optional `BackupConfiguration`, stored as `IDictionary<string, string>` connection details). Both endpoints use the same protocol-specific keys. Invalid numeric values fall back to defaults silently.
@@ -359,7 +373,9 @@ Each value update delivered to an Instance Actor includes:
>
> **The two residual gaps of that family are now closed — but not where the register said they were (rows 89 + 90, 2026-08-17)**: the peer node's DCL was never "serving the old endpoint". Only the node running the `site-{SiteId}` Deployment Manager singleton ever receives a `CreateConnectionCommand`, so the standby's manager is **empty**. The divergence surfaces when the singleton's *next* incarnation (failover, or an ordinary restart — no replication required) rebuilds every connection from the `Connections` snapshots frozen into the instances' flattened configs, because the per-name hash cache that recorded the artifact deploy's change is in-memory and dies with the incarnation. The fix lives entirely in the Site Runtime, as an **arbitration rule** — the definitions store wins wherever it has an opinion, resolved *before* the hash is computed so the two sources cannot fight over the same per-name slot — plus in-memory tombstones so a centrally removed connection is not resurrected by a flattened config. Nothing changes here. See `Component-SiteRuntime.md` → *DCL connection arbitration* and `docs/plans/2026-08-17-dcl-connection-arbitration-decision.md`.
>
> **What DOES remain open here, and it is this component's:** `HandleCreateConnection` is **create-only**, so a re-issued create for a name it already holds is a no-op — a live connection actor never adopts a changed endpoint/credential/backup/failover-count, on any node. The Deployment Manager's documented "a changed config re-issues the create so the DCL adopts it" was therefore never true; it now logs a Warning saying so, once per changed configuration, and the change takes effect only when the connection is next created from scratch (node restart, singleton failover, or remove-then-re-add). The naive fix — stop the actor and re-create it — is destructive: every Instance Actor's tag subscriptions are lost and `InstanceActor` cancels its per-connection subscribe-retry timer on the first success, so **nothing re-subscribes**. The non-destructive fix is a `ReconfigureConnectionCommand` handled *inside* `DataConnectionActor`, reusing the adapter-swap machinery the primary→backup failover path already has (dispose adapter → `_factory.Create` the new one → bump `_adapterGeneration` → re-enter the connect cycle, which re-issues everything from `_subscriptionsByInstance`).
> **A live connection IS reconfigured in place, via a third command (register row 97, 2026-08-17).** `HandleCreateConnection` remains **create-only** — it warns and returns for a name it already holds — so the Deployment Manager's documented "a changed config re-issues the create so the DCL adopts it" was never true, on any node: a corrected endpoint/credential/backup/failover-count took effect only when the connection was next built from scratch (node restart, singleton failover, remove-then-re-add), while the DM's hash cache recorded it as applied. The Site Runtime now sends a **`ReconfigureConnectionCommand`** on the change edge instead. The manager **forwards** it to the connection actor that owns the name, and falls back to a create for a name it does not hold — so the command is safe whatever the manager's state and a sender never has to know whether the actor exists.
>
> **Stop-and-recreate was rejected, and the reason is worth keeping.** Subscriptions are held inside `DataConnectionActor`, and `InstanceActor` cancels its per-connection subscribe-retry timer on the first success — so a rebuilt connection actor would have *no* subscribers and nothing would ever re-subscribe. That trades "stale endpoint, data flowing" for "correct endpoint, no data until redeploy". See *In-place reconfiguration* below for what the actor does instead.
Each data connection is managed by a dedicated connection actor that uses the Akka.NET **Become/Stash** pattern to model its lifecycle as a state machine:
@@ -434,6 +450,7 @@ The DCL reports the following metrics to the Health Monitoring component via the
## Interactions
- **Site Runtime (Deployment Manager)**: Receives `CreateConnectionCommand` / `ReconfigureConnectionCommand` / `RemoveConnectionCommand`. It is the sole creator, reconfigurer and remover of connection actors; the DCL never reads storage.
- **Site Runtime (Instance Actors)**: Bidirectional — delivers value updates, receives subscription registrations and write-back commands.
- **Site Runtime (NativeAlarmActor)**: Bidirectional — receives alarm subscribe/unsubscribe requests, delivers native alarm transitions and source-unavailable notifications (read-only; no ack-back to the source).
- **Health Monitoring**: Reports connection health periodically.