feat(dcl+siteruntime): reconfigure a LIVE data connection in place — register row 97

`DataConnectionManagerActor.HandleCreateConnection` is create-only (warns and
returns for a name it already holds), so `EnsureDclConnection`'s documented
"a changed config re-issues the create so the DCL adopts it" was never true on
ANY node: a corrected endpoint/credential/backup/failover-count took effect only
when the connection was next built from scratch (node restart, singleton
failover, remove-then-re-add), while the DM's hash cache recorded it as applied.
Rows 89/90 made that honest with a Warning; this closes it.

New `ReconfigureConnectionCommand` (Commons, same shape as the create):

- `DeploymentManagerActor.EnsureDclConnection` now has three outcomes off its one
  per-name hash slot — unchanged -> nothing at all, new -> create, CHANGED ->
  reconfigure — and logs "Reconfigured DCL connection {Connection}
  (protocol={Protocol}) in place" at Information in place of the old Warning. The
  winner-resolution / arbitration / tombstone logic is untouched; only what
  happens after a change is detected changed.
- `DataConnectionManagerActor` FORWARDS the command to the owning connection
  actor, falling back to a create for a name it does not hold, so the command is
  safe whatever the manager's state.
- `DataConnectionActor` handles it in ALL THREE states and never stashes it — a
  connection pointed at a wrong or dead endpoint lives in Connecting/Reconnecting,
  which is exactly when a correction arrives; stashed it would wait on the broken
  configuration forever. Handling replaces protocol/primary/backup/failover-count,
  re-arms endpoint selection to Primary (the new config's primary is the intent,
  whatever the old config had failed over to), and calls the new shared
  `SwapAdapter` helper — now used by both failover sites too, so the swap exists
  once: unhook + fire-and-forget dispose -> _factory.Create -> rewire -> bump
  _adapterGeneration LAST. `BecomeReconnecting(afterReconfigure: true)` then
  reconnects immediately (StartSingleTimer replaces any armed reconnect, so no
  racing attempts) with the unstable-disconnect accounting and failover check
  suppressed — the link was dropped on purpose, and counting it as an unstable
  cycle could trip a failover onto the NEW backup before the new primary had been
  tried once. ReSubscribeAll/ReSubscribeAllAlarms then re-issue everything from
  the untouched subscription registries. Idempotent: a reconfigure carrying the
  configuration already in force is a no-op.

Stop-and-recreate was rejected as destructive: subscriptions live inside
DataConnectionActor and InstanceActor cancels its subscribe-retry timer on first
success, so a rebuilt actor would have no subscribers and nothing would ever
re-subscribe.

Tests: new DataConnectionReconfigureTests (7; 5 fail / 2 pass against a
behavioural revert). DclConnectionArbitrationTests gains two row-97 tests and its
CDC-arrival test now asserts the second push is a reconfigure; its ping-pong
guard also asserts zero reconfigures, since a flip would now show up there.
SiteRuntime-010 renamed to ..._ReissuesReconfigureCommand. 3 fail against a
reverted EnsureDclConnection. DataConnectionLayer.Tests 327/327,
SiteRuntime.Tests 643/643, solution build 0 warnings.

Docs: Component-DataConnectionLayer.md gains an "In-place reconfiguration"
section under Endpoint Redundancy; Component-SiteRuntime.md, and the
docs/components/ mirrors, updated; the arbitration decision memo gains an
appended §9 postscript resolving its residual 1 (residuals 2 and 3 stay open).
This commit is contained in:
Joseph Doherty
2026-08-17 06:19:13 -04:00
parent dcfcf822af
commit 7c1268d85c
12 changed files with 943 additions and 126 deletions
@@ -291,3 +291,99 @@ unchanged for a deploy-only site, and the ping-pong must never appear.
**None added.** The overlay refresh rides `ScadaBridge:SiteRuntime:InstanceSweepIntervalSeconds`
(default 60; `0` disables the sweep and therefore the post-boot half of this convergence — the
boot-time arbitration still applies, because it hangs off the startup load, not the sweep timer).
---
## 9. Postscript — residual 1 RESOLVED (register row 97, 2026-08-17)
*Appended; §6 above is left exactly as it was written.*
§6 residual 1 — "a live connection actor is still not reconfigured in place" — is closed, along
the sketch §6 recorded, in a Data Connection Layer change that leaves everything above untouched.
**The arbitration rule, the winner-resolution ordering and the tombstones are unchanged. Only what
happens AFTER a change is detected changed.**
### 9.1 The third command
`EnsureDclConnection` had two outcomes off its single per-name hash slot; it now has three:
| Hash slot | Command sent | Log |
|---|---|---|
| equal | **nothing** | none — what keeps a healthy site silent every sweep tick |
| absent | `CreateConnectionCommand` | `Created DCL connection {Connection} …` (Information) |
| **different** | **`ReconfigureConnectionCommand`** | `Reconfigured DCL connection {Connection} (protocol={Protocol}) in place …` (Information) |
The Warning §6 added to make the gap honest ("…is not reconfigured in place — it keeps serving its
current configuration until it is recreated") is **gone**, because the sentence is no longer true.
`DataConnectionManagerActor.HandleReconfigureConnection` **forwards** the command to the connection
actor that owns the name, and **falls back to a create** for a name it does not hold — so the
command is safe whatever the manager's state, and a sender whose view of which connections exist is
stale (a fresh incarnation, a node that has just taken over) still converges in one message.
`HandleCreateConnection` itself is untouched and stays create-only.
### 9.2 Inside the connection actor
`DataConnectionActor` handles the command in **all three** states, and **never stashes it**. That is
load-bearing rather than tidy: a connection pointed at a wrong or dead endpoint spends its life in
`Connecting` or `Reconnecting` — exactly when a corrected configuration is sent — so a stashed
reconfigure would wait for the *broken* configuration to succeed first.
The swap itself is the failover machinery, not a copy of it: the two failover sites and the
reconfigure now share one `SwapAdapter(newConfig)` helper (unhook + fire-and-forget dispose →
`_factory.Create` → rewire → bump `_adapterGeneration` **last**, so everything in flight on the
outgoing adapter is recognised as stale). Then `BecomeReconnecting(afterReconfigure: true)`:
- reconnects **immediately** rather than after `ReconnectInterval` — there is nothing to wait for,
and `StartSingleTimer` replaces any reconnect already armed, so a reconfigure landing mid-backoff
cannot leave two attempts racing;
- **suppresses the unstable-disconnect accounting and the failover check** — the link was dropped on
purpose, and counting it as an unstable cycle could trip a failover onto the *new* backup before
the new primary had been tried once;
- logs the site event at Info ("Connection reconfigured — reconnecting on the new configuration")
instead of the Warning "Connection lost".
`ReSubscribeAll` / `ReSubscribeAllAlarms` then re-issue everything from `_subscriptionsByInstance`
and `_alarmSourceSubscribers`, which the swap never touched. **Subscribers do nothing and know
nothing**: tags go bad quality across the swap and come back Good, exactly as on a failover.
Two edge cases §6 did not name:
1. **Endpoint selection is re-armed from the top.** Whatever endpoint the previous configuration had
failed over to, the new configuration's *primary* is the operator's intent — so `ActiveEndpoint`
and both failure counters reset. Without this, a connection sitting on its old backup would apply
a new configuration and immediately connect to the new *backup*.
2. **The command is idempotent.** A reconfigure carrying the configuration already in force is a
no-op inside the actor. The sender is hash-gated and does not send one, but an adapter swap drops
the link and re-subscribes every tag, so a duplicate must never be able to cost that.
A **protocol change** is supported for free: `_protocolType` became mutable and the factory keys off
it.
### 9.3 What this cost the ping-pong guard
§3's guard test asserted "one create, zero removes, across many ticks". A flip would now show up as
a stream of **reconfigures**, not creates — each one dropping the link and re-subscribing every tag.
The guard asserts on both counts now; asserting only on creates would have left it silently
guarding the wrong thing.
### 9.4 Tests
`tests/…/DataConnectionLayer.Tests/Actors/DataConnectionReconfigureTests.cs` — 7 tests. Fail-before
evidence (behavioral revert: the three `ReconfigureConnectionCommand` cases removed from the state
machine and the manager's forward replaced by the create-only path — i.e. the exact pre-fix
behaviour) — **5 fail, 2 pass**: the swap-and-re-subscribe from `Connected`, the generation bump,
the honoured-not-stashed cases in `Connecting` and in `Reconnecting`, and the manager's forward all
fail; the identical-config no-op and the unknown-name fallback-to-create pass, as guards.
`DclConnectionArbitrationTests` gained two (`ChangedConfigurationForALiveConnection_IsSentAsA
Reconfigure_NotASecondCreate`, `UnchangedConfiguration_SendsNeitherACreateNorAReconfigure`) and
`DefinitionArrivingAfterBoot_IsAdoptedByTheNextSweepTick` now asserts the second push is a
reconfigure. With `EnsureDclConnection` reverted to sending a create on the change edge, **3 fail**
across `DclConnectionArbitrationTests` + `DeploymentManagerMediumFindingsTests` (the two changed-edge
tests plus SiteRuntime-010, renamed `…_ReissuesReconfigureCommand`).
### 9.5 Residuals after this
§6 residuals **2** (a removal is not durable across a restart) and **3** (a delete is undetectable on
a node that never adopted the row) are unchanged and still open. §6 residual **4** was never a gap.