lmxopcua/docs/reviews/2026-10-21/02-runtime-cluster.md

Review 02 — Runtime + Cluster (Akka actors, deploy apply/acks, redundancy gates, mesh/config/telemetry transports, LocalDb)

Scope: src/Server/ZB.MOM.WW.OtOpcUa.Runtime/** and src/Core/ZB.MOM.WW.OtOpcUa.Cluster/** at master d51d35bd. Production code only. Every finding below was verified by reading the cited lines; drivers / Core / ControlPlane were read only where a Runtime behaviour depends on them.

Severity scale: Critical / High / Medium / Low. Confidence: High = mechanism verified in code; Medium = mechanism verified, magnitude or trigger frequency estimated.

---

1. Memory / resource leaks

L1 — IDriver instances are never disposed; the runtime only ever calls ShutdownAsync (Calculation leaks compiled-script ALCs; OpcUaClient leaks a LoggerFactory + gate + CTS per respawn)

• Severity: High (Calculation) / Low (OpcUaClient) · Confidence: High (never-disposed), Medium (growth rate)
• Location:
  • src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverInstanceActor.cs:1527-1555 (PostStop)
  • src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs:3759-3775 (StopChild), :3811-3827 (HandleRestartDriver)
  • src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/IDriver.cs:11 (no IDisposable/IAsyncDisposable)
  • src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Calculation/CalculationDriver.cs:323-330 vs :719-727
  • src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient/OpcUaClientDriver.cs:3326-3367
• Evidence: the only teardown the runtime performs on a stopped child is

  try { _driver.ShutdownAsync(CancellationToken.None).GetAwaiter().GetResult(); }   // PostStop
  

  No code path in Runtime/Host/Core.Resilience ever does (driver as IDisposable)?.Dispose() (grep over those trees found none). CalculationDriver.ShutdownAsync only stops timers and clears _pending; _evaluator.Dispose() lives exclusively in CalculationDriver.Dispose() (line 726). CalculationEvaluator._cache is a CompiledScriptCache that owns "a collectible AssemblyLoadContext per evaluator" and unloads them only from its Dispose()/Clear() (CompiledScriptCache.cs:130-135). OpcUaClientDriver.ShutdownAsync does detach the cert-validation handler, but the per-instance SdkTelemetry.ForLogger(...) LoggerFactory (SdkTelemetry.Release, line 3367/3395), _gate and _sessionWorkCts are released only in Dispose/DisposeAsync.
• Failure scenario: DriverSpawnPlanner respawns a driver on ANY DriverConfig change (DriverSpawnPlan.cs:62-68), and a Calculation driver's merged config folds in its tag/script set, so every edit to a calculation tag or script = one abandoned CalculationEvaluator holding every compiled script assembly it ever loaded. Same on operator Restart and on driver removal. On a calc-heavy node a day of authoring leaves N loaded script ALCs pinned by an unreachable-but-unloaded context; JIT code + metadata are never reclaimed. OpcUaClient leaks one LoggerFactory (with its ForwardingLoggerProvider) + semaphore + CTS per respawn.
• Fix: in DriverInstanceActor.PostStop, after ShutdownAsync, do if (_driver is IAsyncDisposable ad) await-ish ad.DisposeAsync(); else if (_driver is IDisposable d) d.Dispose(); — but ONLY when the actor is being stopped for good, not on a supervision restart (the Props closure re-uses the same driver object; see H2). Override PreRestart to skip the dispose, or have the host pass an "owned" flag. Alternatively make IDriver : IAsyncDisposable and dispose in StopChild/HandleRestartDriver after Context.Stop via a watch.

L2 — DriverHostActor._lastReprovisionUtc is never pruned

• Severity: Low · Confidence: High
• Location: DriverHostActor.cs:531 (declaration), :3897 (write); StopChild (:3759) never removes.
• Evidence: _lastReprovisionUtc[msg.DriverInstanceId] = now; with no Remove anywhere. The comment says "bounded by the number of driver children" but it is bounded by the number of distinct ids ever reprovisioned, which grows with every driver that was removed after crashing. ~16 bytes/entry; cosmetic unless ids are generated per deploy (they are stable GUID-ish ids).
• Fix: _lastReprovisionUtc.Remove(driverInstanceId) in StopChild.

L3 — DependencyMuxTagUpstreamSource._cache retains every RawPath ever pushed, across deploys

• Severity: Low · Confidence: High
• Location: src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ScriptedAlarms/DependencyMuxTagUpstreamSource.cs:45-48
• Evidence: // Intentionally never pruned — _cache[path] = snapshot with no eviction. The engine re-registers mux interest per apply, so renamed/removed tags stop being pushed, but their last snapshot (boxed value + two DateTimes) stays forever.
• Failure scenario: a site that renames raw paths often (RawTree rename is a surgical remove+add) accumulates one stale snapshot per retired path for the process lifetime. Bounded by churn, not by live config.
• Fix: on ScriptedAlarmHostActor.OnAlarmsLoaded, prune _cache to the loaded dep-ref union (add a RetainOnly(IReadOnlySet<string>)).

L4 — TelemetryLocalHub._evicted grows per removed driver id

• Severity: Low · Confidence: High
• Location: src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Telemetry/TelemetryLocalHub.cs:47, 77-89, 92-98
• Evidence: Evict adds to _evicted; only Readmit (re-spawn of the SAME id) removes. A driver removed for good keeps its id in the set forever. Bounded by distinct removed ids; a few bytes each. Noted because the tombstone is load-bearing (#609) and cannot simply be cleared on a timer without re-opening the late-write resurrection.
• Fix: acceptable as is; if desired, stamp eviction time and sweep entries older than the longest plausible in-flight capability call (minutes), since the tombstone only needs to outlive the stopping child.

---

2. Unimplemented / stubbed / dormant

U1 — GetDiagnostics → NodeDiagnosticsSnapshot is a stub payload AND the only client has no caller

• Severity: Medium · Confidence: High
• Location: DriverHostActor.cs:2499-2517; src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Clients/FleetDiagnosticsClient.cs:35; AdminUI/Clients/ServiceCollectionExtensions.cs:17
• Evidence:

  new DriverInstanceDiagnostics(DriverInstanceId: Guid.Empty, Name: kv.Key,
      State: kv.Value.Stubbed ? "Stubbed" : "Spawned", ConnectedDevices: 0, FaultedDevices: 0, LastChangeUtc: DateTime.UtcNow)
  

  Connectivity, device counts and change time are placeholders regardless of the child's actual state. IFleetDiagnosticsClient is registered in AdminUI DI but no .razor/.cs resolves it (grep: only the registration + the client itself). The message is handled in all three behaviours so it looks live, but the data is fake and nobody reads it. RunningFromCache is the one real field.
• Fix: either delete GetDiagnostics/NodeDiagnosticsSnapshot/IFleetDiagnosticsClient (the /hosts page already gets real state over the telemetry channels) or populate it from the child health snapshots. Leaving a plausible-looking stub is the "dormant DI registration" class the review asked for.

U2 — TelemetryDialOptions.CallTimeoutSeconds is bound and defaulted but consumed nowhere

• Severity: Low · Confidence: High
• Location: src/Core/ZB.MOM.WW.OtOpcUa.Cluster/TelemetryOptions.cs:75-76
• Evidence: public int CallTimeoutSeconds { get; set; } = 30; — zero references outside the options file (validators included; the only other CallTimeout* hits are the unrelated Galaxy gateway option). The telemetry dialer therefore runs its server-streaming call with no deadline from this knob, and an operator setting TelemetryDial:CallTimeoutSeconds changes nothing.
• Fix: wire it into the dialer's CallOptions/keep-alive, or delete it and the doc row in docs/Configuration.md.

U3 — HistorianAdapterActor.Receive<AlarmHistorianEvent> has no production sender

• Severity: Low · Confidence: High
• Location: src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Historian/HistorianAdapterActor.cs:63-66
• Evidence: comment says "kept for a future engine→historian path"; grep "new AlarmHistorianEvent(" over src finds no producer (the adapter's own Translate is target-typed new(). The handler carries a WARNING about double-writes if anyone ever uses it.
• Fix: delete the receive (and the warning) — a latent double-historize trap with no current benefit.

U4 — DependencyMuxActor.OnAttributeValuePublished is a documented test-only seam kept in the production actor

• Severity: Low · Confidence: High
• Location: src/Server/ZB.MOM.WW.OtOpcUa.Runtime/VirtualTags/DependencyMuxActor.cs:55, 106-123; DriverInstanceActor.cs:110-117
• Evidence: "No production sender remains" / "treat it as dead in production". Allocates a per-subscriber single-item batch. Harmless, but it is a message with a handler and no sender, which the review asked to flag.
• Fix: move the per-value shape to a test helper that wraps it in a one-item AttributeValueBatchPublished.

U5 — DriverHostActor constructor keeps an unused ILoggerFactory? loggerFactory positional parameter

• Severity: Low (nit) · Confidence: High
• Location: DriverHostActor.cs:639-640, 705, 840
• Evidence: "Retained for constructor-signature stability; unused". Given the file's own warning that the positional Props list is a mis-binding hazard (:727-731), dead positional slots raise that risk for no benefit.

---

3. Performance limiters

P1 — Per-tag drivers pay a 3-hop actor chain and one SDK sink lock per changed tag; only 4 of 12 drivers batch

• Severity: Medium · Confidence: High
• Location: DriverInstanceActor.cs:975-990 (handler choice), :1384-1390 (OnDataChangeForward), DriverHostActor.cs:1602-1644 (ForwardBatchToMux), OpcUaPublishActor.cs:402-423; src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/PollGroupEngine.cs:404-416
• Evidence: IBatchSubscribable is implemented by Modbus, FOCAS, S7 and AbCip only (grep). TwinCAT, AbLegacy and Sql use the same PollGroupEngine but expose only ISubscribable, so the engine's per-tag _onChange(...) fires self.Tell(new DataChangeForward(..)) once per changed tag; each becomes a 1-item AttributeValueBatchPublished → host → _dependencyMux.Tell + 1-item AttributeValueBatchUpdate(List) → _sink.WriteValues (one lock acquisition each). Galaxy, MQTT, MTConnect and OpcUaClient are per-tag by nature too.
• Failure scenario: a 2k-tag TwinCAT/Sql device whose values all move per 1 s cycle = ~6k actor messages/s + 2k node-manager lock round-trips/s on the pinned OPC UA thread, versus 3 messages + 1 lock for a Modbus device of the same size. That thread is also every client's Publish path.
• Fix: implement IBatchSubscribable on the three remaining PollGroupEngine drivers (the engine already builds the batch; it is a one-line event forward), and consider a time-boxed coalescer (e.g. 10–20 ms) in DriverInstanceActor for the push drivers.

P2 — DriverInstanceActor suspends its mailbox for the whole device round-trip on writes/subscribes (serialized writes, head-of-line blocking of data changes)

• Severity: Medium · Confidence: High
• Location: DriverInstanceActor.cs:520-523, 542 (ReceiveAsync<WriteAttribute>, <Subscribe>, <Unsubscribe>, <SubscribeAlarms>), :858-901 (5 s write bound), :948-1011 (10 s subscribe + 5 s unsubscribe), :1296-1355 (10 s alarm unsub + 10 s sub)
• Evidence: Akka ReceiveAsync suspends the actor until the task completes. A write holds the actor up to 5 s; a data (re)subscribe up to 15 s; an alarm re-subscribe up to 20 s. Meanwhile every DataChangeForward/DataChangeBatchForward and every further WriteAttribute queues behind it. The host's Ask for a write is 8 s (DriverHostActor.cs:2176-2177), so the SECOND of two concurrent operator writes to a slow device can time out at the host ("write timeout") while the device actually accepted it later.
• Failure scenario: operators issuing several writes to a Galaxy/OPC UA upstream with ~2 s latency get spurious "write timeout" reverts (the optimistic-write self-correction then reverts a value the device DID take); value publishes stall for the duration.
• Fix: run writes via PipeTo (capture Sender, don't suspend the mailbox) with a per-driver in-flight cap, keeping ReceiveAsync only for the subscribe/unsubscribe pair that must serialize against _subscriptionHandle.

P3 — ContinuousHistorizationRecorder performs one awaited fsync per ingested burst on the actor thread (serial fsync cap; unbounded mailbox backlog)

• Severity: Medium · Confidence: High
• Location: src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Historian/ContinuousHistorizationRecorder.cs:308-309, 463-466; src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway/Recorder/FasterLogHistorizationOutbox.cs:158-169
• Evidence: ReceiveAsync<DependencyValueBatch>(OnValueBatchAsync) → await _outbox.AppendBatchAsync(entries, ...); under the default CommitMode = PerEntry that is _log.CommitAsync (fsync) before the handler returns. Ingest throughput is therefore 1 / fsync_latency bursts/s regardless of core count, and the mux keeps Telling into an unbounded mailbox.
• Failure scenario: 50 drivers × 1 Hz poll on a disk with 10 ms fsync = 500 ms/s busy; at 100+ bursts/s (many drivers, or MQTT push) the mailbox grows without bound and heap climbs until the input stops. Nothing meters mailbox depth.
• Fix: default CommitMode to Periodic (100 ms) for production, or move the append off the mailbox (pipe the commit outcome back, like the drain) with a bounded in-flight count; expose mailbox depth via GetStatus.

P4 — DriverInstanceActor.PostStop blocks a dispatcher thread on an unbounded ShutdownAsync

• Severity: Medium · Confidence: High
• Location: DriverInstanceActor.cs:1550
• Evidence: _driver.ShutdownAsync(CancellationToken.None).GetAwaiter().GetResult(); — no token, no WaitAsync(timeout). Runs on the default dispatcher for every Context.Stop(child) (deploy removal, Restart, respawn-on-config-change) and inside DriverHostActor's PreRestart child-stop cascade.
• Failure scenario: a driver whose shutdown waits on a dead socket/COM proxy (OpcUaClient subscription deletes against a half-dead session, Galaxy gateway stream close, FOCAS handle release) pins a fork-join thread for its whole wait; a few such stops starve the default dispatcher, and CoordinatedShutdown / a DriverHostActor restart (which waits for all children to terminate before PostRestart) hangs for the sum of them.
• Fix: ShutdownAsync(cts.Token).WaitAsync(TimeSpan.FromSeconds(5)) with a logged timeout; or stop children via a two-phase "ShutdownRequested → PipeTo → PoisonPill" protocol so no actor thread blocks.

P5 — OpcUaPublishActor runs the entire deploy apply (parse, plan, materialise) on the single pinned OPC UA thread; legacy path also does a SQL read there

• Severity: Medium (by design, worth a note) · Confidence: High
• Location: OpcUaPublishActor.cs:483-725 (HandleRebuild/HandleRebuildCore), :942-977 (LoadArtifact/LoadLatestArtifact)
• Evidence: the actor is pinned (opcua-synchronized-dispatcher) so a rebuild of a 50k-node composition serialises every live AttributeValueBatchUpdate and (because the node manager lock is the SDK's) every client Publish behind it. The surgical-deploy work bounds the common case, but MaterialiseHierarchy still runs in full each apply and a classifier miss is a full rebuild. The _dbFactory-backed LoadArtifact branch (only reachable for a RebuildAddressSpace without bytes — no production caller today) would also block that thread on central SQL.
• Fix: parse + plan (DeploymentArtifact.Open, ParseComposition, AddressSpacePlanner.Compute) off-thread via PipeTo, keeping only the sink materialisation on the pinned thread; delete the bytes-less LoadArtifact branch or route it through the host.

P6 — VirtualTagActor publishes a cluster-wide DPS ScriptLogEntry on EVERY failed evaluation (no rate limit)

• Severity: Medium · Confidence: High
• Location: src/Server/ZB.MOM.WW.OtOpcUa.Runtime/VirtualTags/VirtualTagActor.cs:353-356, 375-399
• Evidence: OnEvaluationFailed → PublishLog(level, reason) unconditionally ("Always meters + publishes the per-failure script-log entry"), then DistributedPubSub...Publish(ScriptLogsTopic, entry) + _telemetryHub.Emit. The Bad-status publish is deduped once per transition; the log line is not.
• Failure scenario: a script that throws on every input (e.g. a null-dereferencing cast after a device started sending nulls) on a 1 Hz dependency emits 1 DPS message/s per virtual tag across the mesh and into every telemetry subscriber; 200 such tags = 200 msg/s of identical text into the AdminUI /script-log bridges and the gRPC telemetry stream (bounded channels drop-oldest, so real entries are shed).
• Fix: rate-limit per vtag (first N per minute, then a summary count), mirroring SdkTelemetry's 20/min window in the OpcUaClient driver.

P7 — DbHealthProbeActor runs a synchronous EF query on its actor thread with no explicit command timeout

• Severity: Low · Confidence: High
• Location: src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Health/DbHealthProbeActor.cs:52-65
• Evidence: db.Deployments.AsNoTracking().Take(1).ToList() every 5 s; relies on SqlClient's implicit 15 s connect / 30 s command timeouts. OpcUaPublishActor Asks it with a 1 s timeout and fail-safe-demotes, so the consumer is fine; the cost is a dispatcher thread held for up to 30 s per tick during a SQL hang, and ticks queue behind each other.
• Fix: Task.Run + PipeTo, or SetCommandTimeout(2s) like DriverHostActor.LoadArtifactBlob does.

---

4. Other hazards (concurrency / lifecycle)

H1 — A supervision restart of ScriptedAlarmHostActor disposes the shared ScriptedAlarmEngine; the restarted actor then fails every LoadAsync forever (scripted alarms on the node silently die)

• Severity: High · Confidence: High (mechanism), Medium (trigger frequency)
• Location: src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ScriptedAlarms/ScriptedAlarmHostActor.cs:155-163 (Props captures the engine), :676-687 (PostStop disposes it), :406-429 (documented "restarts the actor" policy), DriverHostActor.cs:1176-1183 (engine built once); src/Core/ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms/ScriptedAlarmEngine.cs:239-241, 1153-1158
• Evidence:

  // PostStop
  _cts.Cancel(); _engine.OnEvent -= _onEngineEvent; _engine.Dispose(); _cts.Dispose();
  

  Akka's default PreRestart calls PostStop(), then re-creates the actor from the SAME Props — i.e. around the same, now-disposed, engine and upstream. ScriptedAlarmEngine.Dispose sets _disposed, kills the shelving timer and UnsubscribeFromUpstream(); LoadAsync then throws ObjectDisposedException (:241) and every reactive path early-returns on _disposed. The host's own doc (:406-416) states that an exception escaping OnDependencyChanged "restarts the actor" and calls that acceptable because "the next delivery re-converges" — it cannot, because the engine is dead. The new incarnation re-subscribes OnEvent (ctor) and keeps piping Status.Failure → OnLoadFailed ("engine LoadAsync failed — alarms not (re)loaded") on every deploy until DriverHostActor itself restarts.
• Failure scenario: any throw inside a Receive of this actor — _upstream.Push argument validation, a ToSnapshot/ToDefinition mapping surprise, ReassertConditionNodes → _engine.GetProjections() — and the node's scripted alarms stop evaluating with only a Warning per deploy as evidence; condition nodes stay at their last state (no Bad quality, no /alerts row), which reads as "no alarms".
• Fix: either (a) override PreRestart to NOT dispose (detach OnEvent only; dispose in a real stop via a flag) and re-subscribe in PostRestart, or (b) make the engine owned per incarnation: have DriverHostActor pass a factory Func<(upstream, engine)> and build them in the actor ctor, so a restart gets a fresh engine. Add a test: throw from a DependencyValueBatch handler, then assert a later ApplyScriptedAlarms loads cleanly.

H2 — DriverInstanceActor.InitializeAsync lets reconnect attempts overlap on the same driver object (concurrent ShutdownAsync + InitializeAsync)

• Severity: Medium · Confidence: High
• Location: DriverInstanceActor.cs:743, 791 (RetryConnect every _reconnectInterval = 10 s), :807-856 (Task.Run, generation token), src/Core/ZB.MOM.WW.OtOpcUa.Core/Resilience/DriverResilienceOptions.cs:88, 100, 112 (Initialize timeouts 5 / 8 / 15 s)
• Evidence: the periodic timer fires regardless of whether the previous attempt is still running; _initGeneration only discards the RESULT of an older attempt. Each attempt first runs driver.ShutdownAsync(CancellationToken.None) (unbounded, unwrapped) and then the invoker-bounded InitializeAsync. With Tier C's 15 s Initialize timeout (> 10 s cadence), or NullDriverCapabilityInvoker (no timeout — tests/harness graphs), attempt N+1's ShutdownAsync runs while attempt N's InitializeAsync is still inside the driver.
• Failure scenario: exactly the dispose-before-overwrite race the drivers' rule file warns about (.claude/rules/drivers.md, "MUST tear down before re-initialising"): a transport field is overwritten by attempt N while attempt N+1's teardown is disposing the one it read a moment earlier — a leaked socket or an ObjectDisposedException inside the driver per overlap, for as long as the device is down.
• Fix: gate the retry on "no attempt in flight" (a _initInFlight bool cleared by the piped InitializeSucceeded/Failed), and bound the teardown (ShutdownAsync(ct).WaitAsync(5 s)).

H3 — ScriptedAlarmHostActor.OnAlarmCommand and ContinuousHistorizationRecorder.GetStatus suspend their mailbox on unbounded awaits

• Severity: Low · Confidence: High
• Location: ScriptedAlarmHostActor.cs:581-648 (CancellationToken.None on every engine op), ContinuousHistorizationRecorder.cs:313-320
• Evidence: both are ReceiveAsync; the engine ops end in IAlarmStateStore.SaveAsync (LocalDb/SQLite) and GetStatus in _outbox.CountAsync. A hung SQLite file lock (replication snapshot, disk full) stalls dependency-value ingestion / status replies for the duration with no timeout.
• Fix: linked token with a timeout (10 s) and log on cancel.

H4 — HistorianAdapterActor fans every alert into an unbounded number of concurrent SQLite inserts

• Severity: Low · Confidence: High
• Location: HistorianAdapterActor.cs:66, 79, 159-170
• Evidence: _ = EnqueueAsync(Translate(t)) per transition — no in-flight cap. Each catches and logs, so nothing is lost silently, but an alarm storm (a native source re-raising hundreds of conditions on reconnect) spawns hundreds of concurrent ExecuteAsync against one SQLite connection pool, where they serialise on the file lock and hold pool threads.
• Fix: a Channel<AlarmHistorianEvent> drained by one loop, or a bounded SemaphoreSlim.

H5 — DriverHostActor Stale behaviour has no receive for driver value/alarm bursts or a late ArtifactForApply

• Severity: Low · Confidence: High (unreachable in practice today)
• Location: DriverHostActor.cs:2448-2497
• Evidence: Stale() registers neither AttributeValueBatchPublished, AttributeAlarmPublished, ArtifactForApply nor ApplyWedged, so they would hit Unhandled (dead-letter log noise, dropped). It is only entered from Bootstrap's catch before any child is spawned, so no child exists to send them — but BeginDirectApply's orphan replay in the same try block has already Become(Applying) + started a Task.Run pipe-back before the catch could run; nothing between them throws today, which is the same "by accident of ordering" argument the file itself rejects for the cache pipe-back (:3159-3166).
• Fix: register the four receives in Stale as drops, like the others.

---

5. Checked and fine (looked like leaks/hazards, are bounded)

• DriverHostActor._pendingAcks / apply-outcome timers — every entry is removed by RebuildCompleted, ApplyOutcomeTimedOut or DropPendingAck; timer keyed per correlation (:2778-2800). Bounded by in-flight applies (one at a time).
• Stash growth in Applying — bounded by distinct deployment ids via _deferredDispatches (:1529-1546), cleared in BecomeSteady.
• Routing maps (_nodeIdByDriverRef, _driverRefByNodeId, alarm maps, _desiredByDriver, _valueRefsByDriver) — clear-and-rebuild per apply (:3485-3565); NativeAlarmProjector.RetainOnly prunes removed conditions (:3550); _commsLatched prunes to empty (:1839-1844).
• DPS / cluster subscriptions — the DPS mediator watches subscribers and removes them on Terminated; DriverHostActor.PostStop unsubscribes its cluster subscriptions (:1080-1088); ClusterRoleInfo.SubscriberActor.PostStop unsubscribes (ClusterRoleInfo.cs:276). EventStream subscriptions are keyed by actor ref and dropped on termination.
• IDriver event handlers — data/batch/alarm/rediscovery/host-status/connection-loss handlers are all detached in PostStop (DriverInstanceActor.cs:1544-1549), so a respawn around the same driver object does not accumulate dead-Self closures.
• Child actor lifecycle — VirtualTagHostActor watches + stops children and prunes on Terminated; PeerProbeSupervisor reconciles against the snapshot with generation-suffixed names; DependencyMuxActor watches subscribers and unregisters on Terminated; DriverHostActor uses generation-suffixed child names so stop+respawn never collides.
• DriverInstanceActor._faultTimestamps — pruned to the 5-minute window on every health publish (:1425-1431).
• gRPC channels — one GrpcChannel per endpoint, cached, disposed by DI at host shutdown (GrpcApplyAckSender, GrpcDeploymentArtifactFetcher); per-endpoint deadlines (AckTimeoutSeconds 10 s, FetchTimeoutSeconds 30 s) are positive by default and validated.
• LocalDb artifact cache — newest-2 retention per cluster inside the store transaction, idempotent re-store skip (LocalDbDeploymentArtifactCache.cs:139-166, 204-235).
• ParsedDeploymentArtifact / JsonDocument — opened once per apply, disposed in HandleArtifactForApply's finally and in OpcUaPublishActor.HandleRebuildCore's using; the cache continuation captures only scalars (DriverHostActor.cs:3207-3231).
• TelemetryLocalHub channels — bounded DropOldest; Detach removes the subscriber and completes the writer; Emit never blocks.
• ContinuousHistorizationRecorder drain — single-flight (_draining), bounded parallelism (SemaphoreSlim(_drainConcurrency)), outbox capacity 500k drop-oldest; _tagWriteFailures entries are cleared on the tag's next success and bounded by tag count.
• ConfigureAwait(false) + Self/Sender — every such site captures Self/Sender before the await (ContinuousHistorizationRecorder.IngestAsync:410, GetStatus:317, DriverHostActor.HandleRouteNodeWrite:2175, HandleSubscribeAsync:955-956); OpcUaPublishActor.OnHealthTick's ContinueWith only constructs a record. No regression of the #491 class found.
• DriverHostActor restart — default PreRestart stops all children and Akka waits for their termination before PostRestart, so the re-run PreStart (SpawnVirtualTagHost/SpawnScriptedAlarmHost with fixed names) cannot hit an actor-name collision; a NEW ScriptedAlarmEngine is built, so H1 applies only to the alarm host's own restart.
• DriverHostActor.DriverMemberCount() on the per-write / per-transition gate path — Cluster.State is the cached read-view snapshot and the LINQ count is over a 2–6 member set; negligible.
• Unreachable-member set (_unreachableNodes) — removed members deliberately retained (documented #672 reasoning); bounded by addresses ever seen in a pair.