dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(dcl+siteruntime): reconfigure a LIVE data connection in place — register row 97

Browse Source 
`DataConnectionManagerActor.HandleCreateConnection` is create-only (warns and
returns for a name it already holds), so `EnsureDclConnection`'s documented
"a changed config re-issues the create so the DCL adopts it" was never true on
ANY node: a corrected endpoint/credential/backup/failover-count took effect only
when the connection was next built from scratch (node restart, singleton
failover, remove-then-re-add), while the DM's hash cache recorded it as applied.
Rows 89/90 made that honest with a Warning; this closes it.

New `ReconfigureConnectionCommand` (Commons, same shape as the create):

- `DeploymentManagerActor.EnsureDclConnection` now has three outcomes off its one
  per-name hash slot — unchanged -> nothing at all, new -> create, CHANGED ->
  reconfigure — and logs "Reconfigured DCL connection {Connection}
  (protocol={Protocol}) in place" at Information in place of the old Warning. The
  winner-resolution / arbitration / tombstone logic is untouched; only what
  happens after a change is detected changed.
- `DataConnectionManagerActor` FORWARDS the command to the owning connection
  actor, falling back to a create for a name it does not hold, so the command is
  safe whatever the manager's state.
- `DataConnectionActor` handles it in ALL THREE states and never stashes it — a
  connection pointed at a wrong or dead endpoint lives in Connecting/Reconnecting,
  which is exactly when a correction arrives; stashed it would wait on the broken
  configuration forever. Handling replaces protocol/primary/backup/failover-count,
  re-arms endpoint selection to Primary (the new config's primary is the intent,
  whatever the old config had failed over to), and calls the new shared
  `SwapAdapter` helper — now used by both failover sites too, so the swap exists
  once: unhook + fire-and-forget dispose -> _factory.Create -> rewire -> bump
  _adapterGeneration LAST. `BecomeReconnecting(afterReconfigure: true)` then
  reconnects immediately (StartSingleTimer replaces any armed reconnect, so no
  racing attempts) with the unstable-disconnect accounting and failover check
  suppressed — the link was dropped on purpose, and counting it as an unstable
  cycle could trip a failover onto the NEW backup before the new primary had been
  tried once. ReSubscribeAll/ReSubscribeAllAlarms then re-issue everything from
  the untouched subscription registries. Idempotent: a reconfigure carrying the
  configuration already in force is a no-op.

Stop-and-recreate was rejected as destructive: subscriptions live inside
DataConnectionActor and InstanceActor cancels its subscribe-retry timer on first
success, so a rebuilt actor would have no subscribers and nothing would ever
re-subscribe.

Tests: new DataConnectionReconfigureTests (7; 5 fail / 2 pass against a
behavioural revert). DclConnectionArbitrationTests gains two row-97 tests and its
CDC-arrival test now asserts the second push is a reconfigure; its ping-pong
guard also asserts zero reconfigures, since a flip would now show up there.
SiteRuntime-010 renamed to ..._ReissuesReconfigureCommand. 3 fail against a
reverted EnsureDclConnection. DataConnectionLayer.Tests 327/327,
SiteRuntime.Tests 643/643, solution build 0 warnings.

Docs: Component-DataConnectionLayer.md gains an "In-place reconfiguration"
section under Endpoint Redundancy; Component-SiteRuntime.md, and the
docs/components/ mirrors, updated; the arbitration decision memo gains an
appended §9 postscript resolving its residual 1 (residuals 2 and 3 stay open).
This commit is contained in:
Joseph Doherty
2026-08-17 06:19:13 -04:00
parent dcfcf822af
commit 7c1268d85c
12 changed files with 943 additions and 126 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs
+38 -26
View File
@@ -2155,9 +2155,10 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
/// <summary>
/// Tracks the configuration last sent to the DCL for each connection name, keyed by
/// a hash of the connection's protocol/endpoints/credentials/failover count.
/// A name whose hash is unchanged is skipped; a name whose config
/// changed re-issues a <c>CreateConnectionCommand</c> so the DCL adopts the new
/// configuration instead of keeping a stale connection after a redeployment.
/// A name whose hash is unchanged is skipped; a name that is new gets a
/// <c>CreateConnectionCommand</c>, and a name whose config CHANGED gets a
/// <c>ReconfigureConnectionCommand</c> so the live connection actor adopts the new
/// configuration in place instead of serving a stale one until it is next rebuilt.
/// </summary>
private readonly Dictionary<string, string> _createdConnections = new();
@@ -2365,13 +2366,29 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
/// Hash-guarded DCL connection push shared by the inline
/// per-instance path (<see cref="EnsureDclConnections(string)"/>) and the
/// system-wide artifact-deploy path (<see cref="HandleDeployArtifacts"/>).
/// Unchanged config is a no-op; a changed endpoint/credentials/backup/
/// failover-count re-issues a <c>CreateConnectionCommand</c> so a system-
/// wide artifact-deploy makes its data-connection change live immediately
/// (the artifact-deploy path previously only persisted to SQLite — the
/// DCL didn't see the change until next instance redeploy or node
/// restart, contradicting the "site is self-contained after artifact
/// deployment" intent).
///
/// <para>
/// <b>Three outcomes, one hash slot.</b> An unchanged configuration is a no-op (no message,
/// no log — this is what keeps a healthy site silent across every sweep tick). A name the
/// hash cache has never held gets a <c>CreateConnectionCommand</c>. A name whose
/// endpoint/credentials/backup/failover-count CHANGED gets a
/// <c>ReconfigureConnectionCommand</c>, which the DCL applies to the live connection actor
/// <b>in place</b> — it swaps its adapter through the same machinery the primary→backup
/// failover uses and re-issues every subscription it holds.
/// </para>
///
/// <para>
/// <b>Why the changed case is not a re-issued create</b> (deferred register row 97).
/// <c>DataConnectionManagerActor.HandleCreateConnection</c> is CREATE-ONLY — it warns and
/// returns for a name it already holds — so the re-issued create this path used to send
/// never reached the live actor at all, while the hash cache recorded it as applied. A
/// changed endpoint therefore took effect only when the connection was next built from
/// scratch (node restart, singleton failover, or a remove-then-add). Stopping and
/// re-creating the actor is not the fix either: subscriptions live inside the connection
/// actor and <c>InstanceActor</c> cancels its subscribe-retry timer on first success, so
/// nothing would re-subscribe. See
/// <c>docs/plans/2026-08-17-dcl-connection-arbitration-decision.md</c> §9.
/// </para>
/// </summary>
private void EnsureDclConnection(
string name,
@@ -2392,32 +2409,27 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
? null
: FlattenConnectionConfig(protocol, backupConfigurationJson);
_dclManager.Tell(new Commons.Messages.DataConnection.CreateConnectionCommand(
name, protocol, primaryDetails, backupDetails, failoverRetryCount));
var changed = _createdConnections.ContainsKey(name);
_createdConnections[name] = configHash;
if (changed)
{
// Be honest about what a re-issued create does. DataConnectionManagerActor's
// HandleCreateConnection is CREATE-ONLY: it warns and returns for a name it already
// holds, so a live connection actor does NOT adopt the new configuration here. The
// change becomes effective the next time the connection is created from scratch —
// the connection is removed and re-added, or this node restarts / the singleton fails
// over, at which point the arbitration overlay hands out the current configuration.
_dclManager.Tell(new Commons.Messages.DataConnection.ReconfigureConnectionCommand(
name, protocol, primaryDetails, backupDetails, failoverRetryCount));
// Emitted once per changed configuration (the hash guard above suppresses repeats),
// so a healthy site stays silent. See the "live reconfiguration" residual in
// docs/plans/2026-08-17-dcl-connection-arbitration-decision.md.
_logger.LogWarning(
"DCL connection {Connection} (protocol={Protocol}) has a new configuration, but a "
+ "live connection actor is not reconfigured in place — it keeps serving its "
+ "current configuration until it is recreated (node restart, singleton failover, "
+ "or the connection being removed and re-added)",
// so a healthy site stays silent.
_logger.LogInformation(
"Reconfigured DCL connection {Connection} (protocol={Protocol}) in place — the "
+ "live connection actor swaps its adapter onto the new configuration and "
+ "re-issues its subscriptions; no redeployment or restart is needed",
name, protocol);
return;
}
_dclManager.Tell(new Commons.Messages.DataConnection.CreateConnectionCommand(
name, protocol, primaryDetails, backupDetails, failoverRetryCount));
_logger.LogInformation(
"Created DCL connection {Connection} (protocol={Protocol})", name, protocol);
}