Joseph Doherty
8f2c2813ea
An Instance Actor started by a self-heal came up with NO DCL connection actors, so every tag it subscribed was answered "Unknown connection: <name>" by the DCL manager indefinitely. Proven live 2026-08-16 by the row-78 close-out probe: the sweep correctly started the replication-seeded instances, then every subscribe failed for 6+ min until a second pair restart. Root cause is narrower than the register assumed. A DCL connection actor is never created from data_connection_definitions -- that table has no production reader at all (GetAllDataConnectionDefinitionsAsync is called only by a test). It exists only as a side effect of starting an instance, from the Connections map inside that instance's own flattened config, via EnsureDclConnections. That ensure was the CALLER's job and only two of the four callers made it: CreateInstanceActorBatch (boot) ensured OK ApplyDeployment (deploy) ensured OK StartStoredInstance MISSING <- row-67 ensure-started + row-78 sweep HandleEnableResult MISSING <- enabling a disabled instance The two misses are the two paths added after the original pair -- the signature of a convention living at call sites instead of in the funnel. It was never replication-specific: the boot load reads only ENABLED configs, so a node booting with an instance disabled never creates its connections either, and enabling it afterwards hit the identical dead end on an ordinary unreplicated site. Fix: EnsureDclConnections moves into CreateInstanceActor, already documented as the single Instance Actor creation funnel (it is where the running revision is stamped for the same reason), and the two caller-side calls are removed. Placed before the already-exists guard so the two paths that already called it keep their exact previous semantics. No new message, option or DCL change -- it reuses EnsureDclConnection's existing per-name hash guard, so steady state is silent. Structural rather than another guard: a fifth start path cannot forget. Deliberately NOT done, with rationale in the memo: a sweep pass over data_connection_definitions. It would not have fixed row 85 (that table is empty on a site that has only ever received instance deploys), and re-hashing from it would fight the deploy path over the same per-name hash slot, recreating connection actors and dropping live subscriptions on a loop. Two residual sibling gaps (connection-only artifact deploy replicating to a peer that never adopts it, and the mirror-image removal case) are recorded in the memo and the component docs as open, not silently skipped. Tests: SelfHealCreatesDclConnectionTests, 7 tests asserting on the CreateConnectionCommand messages sent to a recording DCL stand-in. Verified fail-before against a behavioral revert (call removed from the funnel, restored at the two original sites): 4 fail / 3 pass, the 3 being steady-state silence plus two refactor guards that must pass on both sides. Docs: sweep design memo gains section 7; Component-SiteRuntime, Component-DataConnectionLayer, docs/components/SiteRuntime and docs/components/DataConnectionLayer updated (the deploy-flow step ordering in docs/components/SiteRuntime was stale as written).
91 lines
3.4 KiB
C#
91 lines
3.4 KiB
C#
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
|
|
using ZB.MOM.WW.ScadaBridge.HealthMonitoring;
|
|
|
|
namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.TestSupport;
|
|
|
|
/// <summary>
|
|
/// Minimal <see cref="ISiteHealthCollector"/> whose only job is to expose the observable edge
|
|
/// "the Deployment Manager's boot-time config load has PROVABLY read storage". The manager calls
|
|
/// <see cref="SetInstanceCounts"/> unconditionally the moment that load finishes — including for
|
|
/// an empty site — so a non-zero <see cref="ReportCount"/> replaces a wall-clock sleep.
|
|
///
|
|
/// <para>
|
|
/// <b>Why every post-boot start-path test must gate on this (deferred register rows 83 + 78).</b>
|
|
/// A test that writes a <c>deployed_configurations</c> row BEFORE that read lands has its row
|
|
/// picked up by the boot load itself — so the test passes while proving the boot load works,
|
|
/// not the post-boot path (startup reconcile, sweep, enable) it claims to cover. A bare
|
|
/// <c>Task.Delay</c> only makes that race unlikely, not impossible: on a loaded machine the load
|
|
/// can still be in flight when the delay expires, and the test silently changes what it proves
|
|
/// instead of failing. Gating on the first instance-count report removes the race entirely.
|
|
/// </para>
|
|
/// </summary>
|
|
public sealed class DeploymentManagerStartupProbe : ISiteHealthCollector
|
|
{
|
|
private int _reports;
|
|
|
|
/// <summary>
|
|
/// Number of <see cref="SetInstanceCounts"/> calls seen. Greater than zero means the
|
|
/// boot-time config load has completed and read storage.
|
|
/// </summary>
|
|
public int ReportCount => Volatile.Read(ref _reports);
|
|
|
|
/// <inheritdoc />
|
|
public void SetInstanceCounts(int deployed, int enabled, int disabled) =>
|
|
Interlocked.Increment(ref _reports);
|
|
|
|
/// <inheritdoc />
|
|
public void IncrementScriptError() { }
|
|
|
|
/// <inheritdoc />
|
|
public void IncrementAlarmError() { }
|
|
|
|
/// <inheritdoc />
|
|
public void IncrementDeadLetter() { }
|
|
|
|
/// <inheritdoc />
|
|
public void IncrementSiteAuditWriteFailures() { }
|
|
|
|
/// <inheritdoc />
|
|
public void IncrementAuditRedactionFailure() { }
|
|
|
|
/// <inheritdoc />
|
|
public void UpdateSiteAuditBacklog(Commons.Types.SiteAuditBacklogSnapshot snapshot) { }
|
|
|
|
/// <inheritdoc />
|
|
public void UpdateConnectionHealth(string connectionName, ConnectionHealth health) { }
|
|
|
|
/// <inheritdoc />
|
|
public void RemoveConnection(string connectionName) { }
|
|
|
|
/// <inheritdoc />
|
|
public void UpdateTagResolution(string connectionName, int totalSubscribed, int successfullyResolved) { }
|
|
|
|
/// <inheritdoc />
|
|
public void UpdateConnectionEndpoint(string connectionName, string endpoint) { }
|
|
|
|
/// <inheritdoc />
|
|
public void UpdateTagQuality(string connectionName, int good, int bad, int uncertain) { }
|
|
|
|
/// <inheritdoc />
|
|
public void SetStoreAndForwardDepths(IReadOnlyDictionary<string, int> depths) { }
|
|
|
|
/// <inheritdoc />
|
|
public void SetParkedMessageCount(int count) { }
|
|
|
|
/// <inheritdoc />
|
|
public void SetNodeHostname(string hostname) { }
|
|
|
|
/// <inheritdoc />
|
|
public void SetClusterNodes(IReadOnlyList<Commons.Messages.Health.NodeStatus> nodes) { }
|
|
|
|
/// <inheritdoc />
|
|
public void SetActiveNode(bool isActive) { }
|
|
|
|
/// <inheritdoc />
|
|
public bool IsActiveNode => true;
|
|
|
|
/// <inheritdoc />
|
|
public Commons.Messages.Health.SiteHealthReport CollectReport(string siteId) =>
|
|
throw new NotSupportedException("Startup probe collects no report.");
|
|
}
|