feat(dcl+siteruntime): reconfigure a LIVE data connection in place — register row 97
`DataConnectionManagerActor.HandleCreateConnection` is create-only (warns and
returns for a name it already holds), so `EnsureDclConnection`'s documented
"a changed config re-issues the create so the DCL adopts it" was never true on
ANY node: a corrected endpoint/credential/backup/failover-count took effect only
when the connection was next built from scratch (node restart, singleton
failover, remove-then-re-add), while the DM's hash cache recorded it as applied.
Rows 89/90 made that honest with a Warning; this closes it.
New `ReconfigureConnectionCommand` (Commons, same shape as the create):
- `DeploymentManagerActor.EnsureDclConnection` now has three outcomes off its one
per-name hash slot — unchanged -> nothing at all, new -> create, CHANGED ->
reconfigure — and logs "Reconfigured DCL connection {Connection}
(protocol={Protocol}) in place" at Information in place of the old Warning. The
winner-resolution / arbitration / tombstone logic is untouched; only what
happens after a change is detected changed.
- `DataConnectionManagerActor` FORWARDS the command to the owning connection
actor, falling back to a create for a name it does not hold, so the command is
safe whatever the manager's state.
- `DataConnectionActor` handles it in ALL THREE states and never stashes it — a
connection pointed at a wrong or dead endpoint lives in Connecting/Reconnecting,
which is exactly when a correction arrives; stashed it would wait on the broken
configuration forever. Handling replaces protocol/primary/backup/failover-count,
re-arms endpoint selection to Primary (the new config's primary is the intent,
whatever the old config had failed over to), and calls the new shared
`SwapAdapter` helper — now used by both failover sites too, so the swap exists
once: unhook + fire-and-forget dispose -> _factory.Create -> rewire -> bump
_adapterGeneration LAST. `BecomeReconnecting(afterReconfigure: true)` then
reconnects immediately (StartSingleTimer replaces any armed reconnect, so no
racing attempts) with the unstable-disconnect accounting and failover check
suppressed — the link was dropped on purpose, and counting it as an unstable
cycle could trip a failover onto the NEW backup before the new primary had been
tried once. ReSubscribeAll/ReSubscribeAllAlarms then re-issue everything from
the untouched subscription registries. Idempotent: a reconfigure carrying the
configuration already in force is a no-op.
Stop-and-recreate was rejected as destructive: subscriptions live inside
DataConnectionActor and InstanceActor cancels its subscribe-retry timer on first
success, so a rebuilt actor would have no subscribers and nothing would ever
re-subscribe.
Tests: new DataConnectionReconfigureTests (7; 5 fail / 2 pass against a
behavioural revert). DclConnectionArbitrationTests gains two row-97 tests and its
CDC-arrival test now asserts the second push is a reconfigure; its ping-pong
guard also asserts zero reconfigures, since a flip would now show up there.
SiteRuntime-010 renamed to ..._ReissuesReconfigureCommand. 3 fail against a
reverted EnsureDclConnection. DataConnectionLayer.Tests 327/327,
SiteRuntime.Tests 643/643, solution build 0 warnings.
Docs: Component-DataConnectionLayer.md gains an "In-place reconfiguration"
section under Endpoint Redundancy; Component-SiteRuntime.md, and the
docs/components/ mirrors, updated; the arbitration decision memo gains an
appended §9 postscript resolving its residual 1 (residuals 2 and 3 stay open).
This commit is contained in:
@@ -291,3 +291,99 @@ unchanged for a deploy-only site, and the ping-pong must never appear.
|
||||
**None added.** The overlay refresh rides `ScadaBridge:SiteRuntime:InstanceSweepIntervalSeconds`
|
||||
(default 60; `0` disables the sweep and therefore the post-boot half of this convergence — the
|
||||
boot-time arbitration still applies, because it hangs off the startup load, not the sweep timer).
|
||||
|
||||
---
|
||||
|
||||
## 9. Postscript — residual 1 RESOLVED (register row 97, 2026-08-17)
|
||||
|
||||
*Appended; §6 above is left exactly as it was written.*
|
||||
|
||||
§6 residual 1 — "a live connection actor is still not reconfigured in place" — is closed, along
|
||||
the sketch §6 recorded, in a Data Connection Layer change that leaves everything above untouched.
|
||||
**The arbitration rule, the winner-resolution ordering and the tombstones are unchanged. Only what
|
||||
happens AFTER a change is detected changed.**
|
||||
|
||||
### 9.1 The third command
|
||||
|
||||
`EnsureDclConnection` had two outcomes off its single per-name hash slot; it now has three:
|
||||
|
||||
| Hash slot | Command sent | Log |
|
||||
|---|---|---|
|
||||
| equal | **nothing** | none — what keeps a healthy site silent every sweep tick |
|
||||
| absent | `CreateConnectionCommand` | `Created DCL connection {Connection} …` (Information) |
|
||||
| **different** | **`ReconfigureConnectionCommand`** | `Reconfigured DCL connection {Connection} (protocol={Protocol}) in place …` (Information) |
|
||||
|
||||
The Warning §6 added to make the gap honest ("…is not reconfigured in place — it keeps serving its
|
||||
current configuration until it is recreated") is **gone**, because the sentence is no longer true.
|
||||
|
||||
`DataConnectionManagerActor.HandleReconfigureConnection` **forwards** the command to the connection
|
||||
actor that owns the name, and **falls back to a create** for a name it does not hold — so the
|
||||
command is safe whatever the manager's state, and a sender whose view of which connections exist is
|
||||
stale (a fresh incarnation, a node that has just taken over) still converges in one message.
|
||||
`HandleCreateConnection` itself is untouched and stays create-only.
|
||||
|
||||
### 9.2 Inside the connection actor
|
||||
|
||||
`DataConnectionActor` handles the command in **all three** states, and **never stashes it**. That is
|
||||
load-bearing rather than tidy: a connection pointed at a wrong or dead endpoint spends its life in
|
||||
`Connecting` or `Reconnecting` — exactly when a corrected configuration is sent — so a stashed
|
||||
reconfigure would wait for the *broken* configuration to succeed first.
|
||||
|
||||
The swap itself is the failover machinery, not a copy of it: the two failover sites and the
|
||||
reconfigure now share one `SwapAdapter(newConfig)` helper (unhook + fire-and-forget dispose →
|
||||
`_factory.Create` → rewire → bump `_adapterGeneration` **last**, so everything in flight on the
|
||||
outgoing adapter is recognised as stale). Then `BecomeReconnecting(afterReconfigure: true)`:
|
||||
|
||||
- reconnects **immediately** rather than after `ReconnectInterval` — there is nothing to wait for,
|
||||
and `StartSingleTimer` replaces any reconnect already armed, so a reconfigure landing mid-backoff
|
||||
cannot leave two attempts racing;
|
||||
- **suppresses the unstable-disconnect accounting and the failover check** — the link was dropped on
|
||||
purpose, and counting it as an unstable cycle could trip a failover onto the *new* backup before
|
||||
the new primary had been tried once;
|
||||
- logs the site event at Info ("Connection reconfigured — reconnecting on the new configuration")
|
||||
instead of the Warning "Connection lost".
|
||||
|
||||
`ReSubscribeAll` / `ReSubscribeAllAlarms` then re-issue everything from `_subscriptionsByInstance`
|
||||
and `_alarmSourceSubscribers`, which the swap never touched. **Subscribers do nothing and know
|
||||
nothing**: tags go bad quality across the swap and come back Good, exactly as on a failover.
|
||||
|
||||
Two edge cases §6 did not name:
|
||||
|
||||
1. **Endpoint selection is re-armed from the top.** Whatever endpoint the previous configuration had
|
||||
failed over to, the new configuration's *primary* is the operator's intent — so `ActiveEndpoint`
|
||||
and both failure counters reset. Without this, a connection sitting on its old backup would apply
|
||||
a new configuration and immediately connect to the new *backup*.
|
||||
2. **The command is idempotent.** A reconfigure carrying the configuration already in force is a
|
||||
no-op inside the actor. The sender is hash-gated and does not send one, but an adapter swap drops
|
||||
the link and re-subscribes every tag, so a duplicate must never be able to cost that.
|
||||
|
||||
A **protocol change** is supported for free: `_protocolType` became mutable and the factory keys off
|
||||
it.
|
||||
|
||||
### 9.3 What this cost the ping-pong guard
|
||||
|
||||
§3's guard test asserted "one create, zero removes, across many ticks". A flip would now show up as
|
||||
a stream of **reconfigures**, not creates — each one dropping the link and re-subscribing every tag.
|
||||
The guard asserts on both counts now; asserting only on creates would have left it silently
|
||||
guarding the wrong thing.
|
||||
|
||||
### 9.4 Tests
|
||||
|
||||
`tests/…/DataConnectionLayer.Tests/Actors/DataConnectionReconfigureTests.cs` — 7 tests. Fail-before
|
||||
evidence (behavioral revert: the three `ReconfigureConnectionCommand` cases removed from the state
|
||||
machine and the manager's forward replaced by the create-only path — i.e. the exact pre-fix
|
||||
behaviour) — **5 fail, 2 pass**: the swap-and-re-subscribe from `Connected`, the generation bump,
|
||||
the honoured-not-stashed cases in `Connecting` and in `Reconnecting`, and the manager's forward all
|
||||
fail; the identical-config no-op and the unknown-name fallback-to-create pass, as guards.
|
||||
|
||||
`DclConnectionArbitrationTests` gained two (`ChangedConfigurationForALiveConnection_IsSentAsA
|
||||
Reconfigure_NotASecondCreate`, `UnchangedConfiguration_SendsNeitherACreateNorAReconfigure`) and
|
||||
`DefinitionArrivingAfterBoot_IsAdoptedByTheNextSweepTick` now asserts the second push is a
|
||||
reconfigure. With `EnsureDclConnection` reverted to sending a create on the change edge, **3 fail**
|
||||
across `DclConnectionArbitrationTests` + `DeploymentManagerMediumFindingsTests` (the two changed-edge
|
||||
tests plus SiteRuntime-010, renamed `…_ReissuesReconfigureCommand`).
|
||||
|
||||
### 9.5 Residuals after this
|
||||
|
||||
§6 residuals **2** (a removal is not durable across a restart) and **3** (a delete is undetectable on
|
||||
a node that never adopted the row) are unchanged and still open. §6 residual **4** was never a gap.
|
||||
|
||||
Reference in New Issue
Block a user