fix(historian): the recorder died on its first value — live gate for continuous historization (#491)
v2-ci / build (push) Successful in 4m54s
v2-ci / unit-tests (push) Failing after 16m42s

Ran #491's live gate against a real ZB.MOM.WW.HistorianGateway talking to a real AVEVA
Historian, and it found a production-fatal defect on the first value it ever saw.

ContinuousHistorizationRecorder.OnValueChangedAsync touched `Self` after awaiting
IHistorizationOutbox.AppendAsync(...).ConfigureAwait(false). ConfigureAwait(false) discards
Akka's ActorTaskScheduler synchronization context, so `Self` — which reads ActorBase.Context —
threw NotSupportedException. The throw escaped into supervision, which RESTARTED the actor;
because production seeds an EMPTY ref set (the deployed set arrives later via
UpdateHistorizedRefs), the restart silently discarded every historized ref. Net effect: one
value captured, then nothing, with no further error to show for it. Fixed by capturing
`IActorRef self = Self` before the await — the idiom the GetStatus handler already applied to
Sender.

Thirteen unit tests missed it because InMemoryOutbox.AppendAsync returned
ValueTask.CompletedTask: a completed ValueTask never yields, so the continuation ran inline
with the context still installed. The production FasterLog outbox fsyncs per entry and does
yield. The first regression test was itself vacuous — `await Task.Yield()` posts back to the
CAPTURED context, so the append still completed on the actor thread and reverting the fix left
the test green. The fake must complete on a POOL thread (Task.Delay(1).ConfigureAwait(false)),
as a real fsync does; with that, reverting the fix reddens exactly one test.

The rig could not run this gate at all: docker-dev carried ServerHistorian__* but no
ContinuousHistorization__* keys, so the recorder could never spawn. Both centrals now carry
them (default OFF — the recorder needs ContinuousHistorization:Enabled AND
ServerHistorian:Enabled), with the outbox on the per-node named volume so it survives a
container recreate. ServerHistorian__UseTls was hard-coded true and is now parameterized; a
plaintext h2c gateway was unreachable without it.

Gate results: outbox append, drain, WriteLiveValues accepted by a real historian, and restart
convergence (capture resumes after a restart with no new deployment) all pass. The read-back
hop is NOT proven — the LAN historian accepts writes but serves no reads under a "License
violation: Database tag count exceeds license tag count", and the licensed corporate one needs
the VPN. EnsureTags provisioning is likewise unexercised there (RemoteGrpc-only, and that CAP
is not listening). #491 stays open for the read-back hop alone.

Also recorded, not fixed: the recorder has NO Primary gate, unlike the alarm store-and-forward
sink, so both nodes of a redundant pair send duplicate live values. Gating it would introduce a
failover window with no capture — that is a design decision, not a cleanup.

Evidence: docs/plans/2026-08-01-continuous-historization-live-gate.md

Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
This commit is contained in:
Joseph Doherty
2026-08-01 19:57:37 -04:00
parent 6b3da93151
commit ad74473452
6 changed files with 369 additions and 21 deletions
@@ -247,6 +247,21 @@ public sealed class ContinuousHistorizationRecorder : ReceiveActor, IWithTimers
var timestampUtc = DateTime.SpecifyKind(msg.TimestampUtc, DateTimeKind.Utc);
// Capture Self BEFORE the first await. Self reads ActorBase.Context, and the awaits below use
// ConfigureAwait(false) — which drops Akka's ActorTaskScheduler synchronization context, so any
// continuation that actually resumes off the actor thread finds NO ActorContext and throws
// NotSupportedException. This is the same idiom the GetStatus handler already applies to Sender.
//
// It only bites when the append genuinely YIELDS: the production FasterLog outbox fsyncs per entry
// (PerEntry commit mode), so it does; a fake outbox returning ValueTask.CompletedTask completes
// synchronously, the continuation runs inline with the context still installed, and the bug is
// invisible. That is exactly how it reached production — see the yielding fake in
// ContinuousHistorizationRecorderTests. Live-caught on the docker-dev rig (Gitea #491), where the
// throw escaped the handler into supervision: the actor restarted, and because production seeds an
// EMPTY ref set (the deployed set arrives later via UpdateHistorizedRefs), the restart silently
// dropped every historized ref — so historization stopped dead after the FIRST value.
IActorRef self = Self;
// One outbox entry PER historian name: a single mux fan-out must reach every aliased historized tag
// (each its own historian name), not just one — collapsing to one name here would silently drop the
// rest. Snapshot the names first: although the mailbox is suspended across the awaits below (so the
@@ -290,8 +305,9 @@ public sealed class ContinuousHistorizationRecorder : ReceiveActor, IWithTimers
}
// Nudge a prompt drain attempt; the DrainTick handler de-dups (already draining) and honours
// any active backoff window, so this never bypasses a failure cooldown.
Self.Tell(DrainTick.Instance);
// any active backoff window, so this never bypasses a failure cooldown. Uses the pre-await
// capture, never `Self` — see the note above.
self.Tell(DrainTick.Instance);
}
/// <summary>