fix(siteruntime): self-healed and newly-enabled instances now get their DCL connections (register row 85)
An Instance Actor started by a self-heal came up with NO DCL connection actors, so every tag it subscribed was answered "Unknown connection: <name>" by the DCL manager indefinitely. Proven live 2026-08-16 by the row-78 close-out probe: the sweep correctly started the replication-seeded instances, then every subscribe failed for 6+ min until a second pair restart. Root cause is narrower than the register assumed. A DCL connection actor is never created from data_connection_definitions -- that table has no production reader at all (GetAllDataConnectionDefinitionsAsync is called only by a test). It exists only as a side effect of starting an instance, from the Connections map inside that instance's own flattened config, via EnsureDclConnections. That ensure was the CALLER's job and only two of the four callers made it: CreateInstanceActorBatch (boot) ensured OK ApplyDeployment (deploy) ensured OK StartStoredInstance MISSING <- row-67 ensure-started + row-78 sweep HandleEnableResult MISSING <- enabling a disabled instance The two misses are the two paths added after the original pair -- the signature of a convention living at call sites instead of in the funnel. It was never replication-specific: the boot load reads only ENABLED configs, so a node booting with an instance disabled never creates its connections either, and enabling it afterwards hit the identical dead end on an ordinary unreplicated site. Fix: EnsureDclConnections moves into CreateInstanceActor, already documented as the single Instance Actor creation funnel (it is where the running revision is stamped for the same reason), and the two caller-side calls are removed. Placed before the already-exists guard so the two paths that already called it keep their exact previous semantics. No new message, option or DCL change -- it reuses EnsureDclConnection's existing per-name hash guard, so steady state is silent. Structural rather than another guard: a fifth start path cannot forget. Deliberately NOT done, with rationale in the memo: a sweep pass over data_connection_definitions. It would not have fixed row 85 (that table is empty on a site that has only ever received instance deploys), and re-hashing from it would fight the deploy path over the same per-name hash slot, recreating connection actors and dropping live subscriptions on a loop. Two residual sibling gaps (connection-only artifact deploy replicating to a peer that never adopts it, and the mirror-image removal case) are recorded in the memo and the component docs as open, not silently skipped. Tests: SelfHealCreatesDclConnectionTests, 7 tests asserting on the CreateConnectionCommand messages sent to a recording DCL stand-in. Verified fail-before against a behavioral revert (call removed from the funnel, restored at the two original sites): 4 fail / 3 pass, the 3 being steady-state silence plus two refactor guards that must pass on both sides. Docs: sweep design memo gains section 7; Component-SiteRuntime, Component-DataConnectionLayer, docs/components/SiteRuntime and docs/components/DataConnectionLayer updated (the deploy-flow step ordering in docs/components/SiteRuntime was stale as written).
This commit is contained in:
@@ -646,7 +646,7 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
|
||||
for (var i = startIdx; i < endIdx; i++)
|
||||
{
|
||||
var config = state.Configs[i];
|
||||
EnsureDclConnections(config.ConfigJson);
|
||||
// CreateInstanceActor ensures this config's DCL connections (register row 85).
|
||||
CreateInstanceActor(config.InstanceUniqueName, config.ConfigJson, config.RevisionHash);
|
||||
}
|
||||
|
||||
@@ -1076,10 +1076,8 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
|
||||
{
|
||||
var instanceName = command.InstanceUniqueName;
|
||||
|
||||
// Ensure DCL connections exist for any data-sourced attributes
|
||||
EnsureDclConnections(command.FlattenedConfigurationJson);
|
||||
|
||||
// Create the Instance Actor immediately
|
||||
// Create the Instance Actor immediately. CreateInstanceActor first ensures the DCL
|
||||
// connections this config's data-sourced attributes subscribe through (register row 85).
|
||||
CreateInstanceActor(instanceName, command.FlattenedConfigurationJson, command.RevisionHash);
|
||||
if (!isRedeploy)
|
||||
_deployedInstanceNames.Add(instanceName);
|
||||
@@ -2725,6 +2723,31 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
|
||||
/// </param>
|
||||
internal void CreateInstanceActor(string instanceName, string configJson, string revisionHash)
|
||||
{
|
||||
// Deferred register row 85 — an Instance Actor is useless without the DCL connection
|
||||
// actors its data-sourced attributes subscribe through, so ensuring them belongs in the
|
||||
// single creation funnel rather than at each caller.
|
||||
//
|
||||
// This used to be the caller's job, and only TWO of the four callers did it: the
|
||||
// boot-time batch and ApplyDeployment. The two self-heal starts did not —
|
||||
// StartStoredInstance (the shared row-67 ensure-started / row-78 sweep path) and
|
||||
// HandleEnableResult. Both therefore created an instance whose every tag subscribe was
|
||||
// answered "Unknown connection: X" by the DCL manager, forever, because nothing else on
|
||||
// the node would ever create that connection actor:
|
||||
//
|
||||
// - Row 85's live symptom (2026-08-16): a replication-seeded deployed_configurations row
|
||||
// is started by the row-78 sweep, but the connection actor is only ever created as a
|
||||
// side effect of a deploy or the boot load, so tag subscribes failed for 6+ minutes
|
||||
// until a second pair restart let the boot load create it.
|
||||
// - The same defect is reachable with no replication at all: the boot load reads only
|
||||
// ENABLED configs, so a node that boots with an instance DISABLED never creates that
|
||||
// instance's connections, and enabling it afterwards hit the identical dead end.
|
||||
//
|
||||
// Placed BEFORE the already-exists guard so the two paths that already called it keep
|
||||
// their exact previous semantics (they called it unconditionally). It is cheap and
|
||||
// silent when there is nothing to do: EnsureDclConnection is hash-guarded on
|
||||
// _createdConnections, so an unchanged connection is a no-op with no message and no log.
|
||||
EnsureDclConnections(configJson);
|
||||
|
||||
if (_instanceActors.ContainsKey(instanceName))
|
||||
{
|
||||
_logger.LogWarning("Instance Actor {Instance} already exists, skipping creation", instanceName);
|
||||
|
||||
Reference in New Issue
Block a user