fix(drivers): stop silently discarding driver config edits (§8.3, #516)

Five drivers ignored the config handed to them on reinitialize, serving the
options their constructor captured — and the deployment sealed green anyway.
Fixed on both halves, as chosen.

Seam (load-bearing): DriverSpawnPlanner routes a changed DriverConfig to
ToStop + ToSpawn instead of an in-place delta, making the factory the single
parse authority. This REVERSES the deliberate decision documented at
DriverSpawnPlan.cs:49-50 ("a pure DriverConfig change stays an in-place delta
— no reconnect"). That reasoning was correct about the resilience pipeline and
wrong about the driver. The accepted price is a reconnect per config edit.

Per-driver, and this split was not anticipated:
- Re-parse in place — Modbus, AbLegacy, OpcUaClient. ParseOptions extracted
  from each factory, called from InitializeAsync behind a HasConfigBody guard
  so "{}" still keeps the constructor options.
- Respawn-only, deliberately NOT re-parsed — Sql and FOCAS. Each builds more
  than options from config (Sql's dialect + resolved connection string,
  FOCAS's client-factory backend, both injected at construction), so adopting
  new options alone would run a NEW tag set against an OLD connection. Half a
  re-parse is worse than none. SqlDriver's doc-comment asserted the opposite
  premise — "config parsing belongs to the factory, which builds a fresh
  instance" — which was false when written and is true only now.

Two green seals removed from ApplyChildDelta: it overwrote the cached Spec
synchronously BEFORE the child dequeued the message, so the host believed the
new config was live and the next reconcile computed no delta, sealing the
drift permanently; and it Tell'd with no Receive<ApplyResult> registered, so a
failed reinit — including Galaxy's deliberate NotSupportedException —
dead-lettered.

Exposed (not created) by the change: a factory throw is a CONFIG error, and
SpawnChild catches it and silently substitutes a stub. Only a brand-new driver
could reach that before; an ordinary config edit can now. Raised Warning ->
Error with an actionable message. It still does not fail the deployment —
doing so would let one malformed driver block a fleet deploy, so that is a
follow-up rather than a drive-by.

Tests: every pre-existing reinit test in these suites passes "{}", the exact
input a guarded re-parser treats as "keep constructor options" — blind to this
defect by construction. New tests pass CHANGED json; the Modbus one was
verified falsifiable by deleting the re-parse (goes red). Two tests asserted
the old behaviour and were rewritten, including the positive control in
DriverHostActorUnreadableArtifactTests, whose observable moved from
UnsubscribeAsync to ShutdownAsync now that teardown happens by stopping the
child rather than emptying its desired set.

Remaining full-suite failures are pre-existing and environmental, verified
identical on the pre-change tree: Host.IntegrationTests (3) and
Driver.AbLegacy.IntegrationTests (4, docker fixture-gated).
This commit is contained in:
Joseph Doherty
2026-07-27 19:32:46 -04:00
parent adce27e7fa
commit d32d89c340
17 changed files with 500 additions and 57 deletions
@@ -244,6 +244,11 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
/// value maps so stale condition state never leaks across redeploys.</summary>
private readonly NativeAlarmProjector _nativeAlarmProjector = new();
/// <summary>In-flight <see cref="DriverInstanceActor.ApplyDelta"/> sends, keyed by correlation, so
/// <see cref="HandleApplyResult"/> can advance the cached spec only once the child confirms it adopted
/// the config (#516). Bounded by the number of driver children with a delta in flight.</summary>
private readonly Dictionary<CorrelationId, DriverInstanceSpec> _pendingDelta = new();
/// <summary>The composition from the most-recent apply (set at the END of
/// <see cref="PushDesiredSubscriptions"/>). Null until the first apply.</summary>
private AddressSpaceComposition? _lastComposition;
@@ -868,6 +873,7 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
Receive<DriverInstanceActor.AttributeAlarmPublished>(ForwardNativeAlarm);
Receive<DriverInstanceActor.ConnectivityChanged>(OnDriverConnectivityChanged);
Receive<DriverInstanceActor.DeltaApplied>(HandleDeltaApplied);
Receive<DriverInstanceActor.ApplyResult>(HandleApplyResult);
Receive<RestartDriver>(HandleRestartDriver);
Receive<ReconnectDriver>(HandleReconnectDriver);
Receive<RouteNodeWrite>(HandleRouteNodeWrite);
@@ -901,6 +907,7 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
Receive<DriverInstanceActor.AttributeAlarmPublished>(ForwardNativeAlarm);
Receive<DriverInstanceActor.ConnectivityChanged>(OnDriverConnectivityChanged);
Receive<DriverInstanceActor.DeltaApplied>(HandleDeltaApplied);
Receive<DriverInstanceActor.ApplyResult>(HandleApplyResult);
Receive<RestartDriver>(HandleRestartDriver);
Receive<ReconnectDriver>(HandleReconnectDriver);
Receive<RouteNodeWrite>(HandleRouteNodeWrite);
@@ -1353,6 +1360,7 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
// A late DeltaApplied (an apply completed just before the DB went Stale) — re-register the driver's
// mux adapter anyway; it simply re-reads the driver's current refs (harmless, no DB access).
Receive<DriverInstanceActor.DeltaApplied>(HandleDeltaApplied);
Receive<DriverInstanceActor.ApplyResult>(HandleApplyResult);
Receive<SubscribeAck>(_ => { /* PubSub ack */ });
Timers.StartPeriodicTimer("retry-db", RetryConfigDbConnection.Instance, ReconnectInterval);
}
@@ -2007,7 +2015,15 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
try { driver = _driverFactory.TryCreate(spec.DriverType, spec.DriverInstanceId, spec.DriverConfig); }
catch (Exception ex)
{
_log.Warning(ex, "DriverHost {Node}: factory for {Type} threw on {Id}; stubbing",
// A factory throw is a CONFIG error, not a connectivity one — TryCreate is pure parsing;
// device I/O happens later in InitializeAsync. Logged at Error because the node silently
// degrades to a stub that answers nothing, and since #516 routed DriverConfig changes
// through a respawn this path is now reachable by an ordinary operator edit rather than
// only by a brand-new driver. It still does NOT fail the deployment — making it do so is
// a deliberate follow-up, since it would let one malformed driver block a fleet deploy.
_log.Error(ex,
"DriverHost {Node}: factory for {Type} REJECTED the config for {Id} — the driver is " +
"stubbed and will serve nothing until the config is fixed and redeployed",
_localNode, spec.DriverType, spec.DriverInstanceId);
}
if (driver is null)
@@ -2087,15 +2103,54 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
Context.Stop(adapter);
}
/// <summary>
/// Sends an in-place config delta to a running child.
/// <para><b>Unreachable from the reconcile path since #516</b> — a DriverConfig change is now a
/// stop + respawn (<see cref="DriverSpawnPlanner"/>), so <c>ToApplyDelta</c> is always empty.
/// Kept because the seam is still exercised by <c>DriverInstanceActor</c>'s own mid-connect config
/// adoption.</para>
/// <para><b>Two seals were removed here.</b> This used to overwrite the cached
/// <c>Spec</c> SYNCHRONOUSLY, before the child had even dequeued the message — so the host
/// immediately believed the new config was live, the NEXT reconcile computed no delta against it,
/// and any drift was sealed permanently. It also <c>Tell</c>d with no <c>Receive&lt;ApplyResult&gt;</c>
/// handler registered, so a FAILED reinit — including Galaxy's deliberate
/// <c>NotSupportedException</c> — dead-lettered and was never surfaced.</para>
/// </summary>
private void ApplyChildDelta(DriverInstanceSpec spec)
{
if (!_children.TryGetValue(spec.DriverInstanceId, out var entry)) return;
entry.Actor.Tell(new DriverInstanceActor.ApplyDelta(spec.DriverConfig, CorrelationId.NewId()));
// Store the full new spec — a delta can change Name, Enabled, ClusterId, etc. in addition to config.
_children[spec.DriverInstanceId] = entry with { Spec = spec };
var correlation = CorrelationId.NewId();
_pendingDelta[correlation] = spec;
entry.Actor.Tell(new DriverInstanceActor.ApplyDelta(spec.DriverConfig, correlation), Self);
_log.Debug("DriverHost {Node}: ApplyDelta queued for {Id}", _localNode, spec.DriverInstanceId);
}
/// <summary>
/// A child's reply to <see cref="ApplyChildDelta"/>. On success the cached <c>Spec</c> is advanced —
/// only now, when the child has actually adopted the config, so a failed reinit leaves the host
/// believing the OLD config is live and the next reconcile re-attempts the change instead of
/// silently treating the drift as applied. A failure is logged at Error rather than swallowed.
/// </summary>
private void HandleApplyResult(DriverInstanceActor.ApplyResult msg)
{
if (msg.Success)
{
if (_pendingDelta.Remove(msg.Correlation, out var spec)
&& _children.TryGetValue(spec.DriverInstanceId, out var entry))
{
// A delta can change Name, Enabled, ClusterId etc. alongside the config.
_children[spec.DriverInstanceId] = entry with { Spec = spec };
}
return;
}
_pendingDelta.Remove(msg.Correlation, out var failed);
_log.Error(
"DriverHost {Node}: driver {Id} REJECTED an in-place config change ({Reason}) — the host keeps " +
"the previous config so the next reconcile re-attempts it",
_localNode, failed?.DriverInstanceId ?? "<unknown>", msg.Reason ?? "no reason given");
}
/// <summary>
/// A driver child finished applying an in-place <see cref="DriverInstanceActor.ApplyDelta"/> (its
/// <see cref="IDriver.ReinitializeAsync"/> completed). Re-register THAT driver's dependency-consumer