Review follow-ups to 34829ea0.
IMPORTANT - the leak filter had its own flake vector. This test saturates the
thread pool by design, and TimedScriptEvaluator starts its 5s clock at WaitAsync
rather than when the inner Task.Run gets a thread, so a legitimately-enqueued
evaluation can blow the budget while still queued and surface "script timed out
after 5.0s". The reason != exhaustion filter classified that as a raw disposed
escape and hard-failed on it - handing the test back the load-fragility the
reshape removed. Timeouts are now their own bucket, tolerated and rate-bounded
exactly as exhaustions are; a raw disposed escape still fails on first occurrence.
Doc-comment names this second residual vector alongside the exhaustion calibration.
MINOR (scope extension authorised by review) - "evaluator disposed during
evaluation" existed as three unshared literals. Promoted to
ScriptEvalFailureReasons alongside VirtualTagEvalResult in Commons.Engines (the
assembly all three already reference) and referenced from both evaluators and the
test. The timeout prefix joins it, since the test now classifies on it too. String
values are byte-identical - no behaviour change.
Falsified: with a 12ms script budget the race produced real timeouts (3/2695).
Classified as leaks (the pre-review filter) they hard-failed the test 6/6 items;
bucketed as starvation they sit 0.11% under the 10% bound and the test is green.
OnDependencyValue ran every change-triggered dependent's script INLINE, under _lock,
via TimedScriptEvaluator.RunAsync(...).GetAwaiter().GetResult() — on the Akka dispatcher
thread of Runtime's DependencyConsumerMuxAdapter, for up to RunTimeout (2 s default) per
arrival, with the state lock held for the whole run.
The callback now records the value, PREPARES the affected tags under the lock (definition
+ warm-up gate + dependency snapshot taken at trigger time) and queues them for a single
serialized evaluation pump. _lock is a short-held state lock again; a new _evalGate
serializes the pump against the timer ticks and is taken only by evaluation workers, so
no host thread can wait on a script. Lock order is always _evalGate -> _lock.
Semantics preserved: per-tag ordering (one FIFO pump), the warm-up gate BEFORE the
evaluator runs, dedup/Bad-transition/recovery publishes, and — because the dependency
snapshot is taken when the trigger fires rather than when the pump catches up — a burst
of arrivals still evaluates against the values that triggered it.
CalculationDriverProbe.Opts (Test-Connect) omitted AllowTrailingCommas and
ReadCommentHandling that CalculationDriverFactoryExtensions.JsonOptions
(spawn) and CalculationDriver.ReinitJsonOptions (in-place ApplyDelta) both
set, so a config blob the real deploy path tolerates could fail Test-Connect
as "invalid JSON" even though deploying it would succeed. Aligned the three
option sets. Also adds CalculationDriverProbeTests.cs, which previously did
not exist at all for this driver type's Test-Connect surface.
Part of code-reviews/Driver.Calculation/findings.md (-003, -004).
Code review of Driver.Calculation found two real defects, both TDD-fixed:
- CalculationDriver's ctor tried to reuse its ILogger<CalculationDriver> as the
evaluator's ILogger<CalculationEvaluator> via an `is` check that can never
succeed (unrelated closed generics), so CalculationEvaluator always ran with
a NullLogger regardless of what the driver was constructed with. Fixed by
having CalculationEvaluator accept the non-generic ILogger instead.
- EvaluateTag called the evaluator BEFORE checking whether every declared
dependency had arrived, so a multi-dependency change-triggered calc tag
evaluated its script against a partial snapshot on every warm-up tick.
VirtualTagContext.GetTag returns a Bad snapshot (.Value == null) for a
missing dep, so the idiomatic `(double)ctx.GetTag("x").Value` cast throws
NullReferenceException — caught and logged as a spurious "script threw"
script-log Warning on ordinary deploy/restart warm-up, for the standard
multi-input calc-tag shape. Fixed by gating evaluation itself on
"all deps arrived", not just the publish.
See code-reviews/Driver.Calculation/findings.md for full detail (5 findings:
001-004 Resolved, 005 Open/cross-module).
Deferment-register G-7: these four were validated by the AdminUI's TagConfigValidator at
authoring but by nothing at deploy, so config arriving by CSV import or API met no gate at
all. The deploy gate dispatches to a driver's *TagDefinitionFactory.Inspect in its Contracts
assembly, and none of the four had one.
Rather than mirror the AdminUI models (a mirror of an accept/reject boundary drifts silently,
with the gate describing a shape the driver no longer reads), each Inspect is built on the
driver's own runtime parse:
- Sql wraps the existing SqlEquipmentTagParser, which already lived in Sql.Contracts. The
P3-deferred Query model stays refused; Inspect surfaces the refusal rather than weakening it.
- MTConnect's blob reader MOVED out of MTConnectDriver (it was private, with a private DTO)
into MTConnectTagDefinitionFactory; the driver now calls it. Behaviour unchanged.
- OpcUaClient's nodeId read MOVED off NamespaceMap into OpcUaClientTagDefinitionFactory;
NamespaceMap now calls it. Behaviour unchanged.
- Calculation gains a Contracts assembly it never had. Only the JSON reading moved there
(CalculationTagDefinitionFactory.TryReadTagConfig); building a CalculationTagDefinition
still needs EquipmentScriptPaths from Commons and stays in the driver, so the split is at
the blob boundary. The new project is deliberately reference-free — a calc blob is strings,
a bool and an int, with no Core.Abstractions type in it.
Each Inspect runs the parse FIRST and returns empty on success, so "never stricter than the
runtime" is structural rather than a promise: a warning is only reachable for a blob the
driver would have refused and then skipped silently. Calculation additionally reports two
present-but-silently-discarded fields (a non-boolean changeTriggered reads as true; a
non-positive-integer timerIntervalMs is dropped), and deliberately does NOT warn about a
trigger-less tag or a sub-50ms timer — the editor refuses both, the runtime honours both, and
the deploy gate follows the runtime.
Found on the docker-dev live gate, not in tests. Two drivers that faulted
during initialise (rtu-gate, opcua1) each rendered "0 tags" on /hosts against
two authored tags apiece, because a ledger that had never committed returned
SkippedTagReport.Empty.
A zero report is a positive claim -- "this driver skipped nothing" -- and
making it on behalf of a driver that never got as far as looking is exactly
the quiet inaccuracy #541 exists to remove. GetSkippedTags() and
SkippedTagLedger.Snapshot() now return null until the first committed build,
so the column renders the same em-dash it already used for a driver that
cannot report at all. Both are 'no answer'; neither is 'nothing skipped'.
Knock-on: a FOCAS driver whose only init threw now reports null rather than a
zero report, which is strictly more honest than the assertion the test
originally made.
A driver builds its RawPath -> definition table at init and tolerates a
per-tag miss: an unmappable TagConfig, an unparseable address, a
capability-matrix rejection. That tolerance is right -- one bad tag must not
take down the other few thousand. What was missing is any statement of the
fact afterwards. A skip is not an error and happens before any read, so the
driver stays Healthy, errors/5 min stays 0, Last error stays empty, and Last
read advances live off the tags that did map. On the wonder Makino in #540
that reading held while focas-z-34184 ignored 20 of its 22 tags.
New optional capability ISkippedTagReporter, backed by a shared
SkippedTagLedger so the reset/commit/cap rules are one implementation rather
than one per driver. The report -- authored count, exact skipped count, and a
capped sample of RawPath + reason -- rides DriverHealthChanged to a new Tags
column on /hosts, over the Phase 5 telemetry stream. No table: a driver-only
node has no ConfigDb to write to, the lesson from #521.
The issue named eight drivers. It is twelve -- every driver in
DriverTypeNames.All has an authored-tag skip path. Calculation (no scriptId),
MTConnect (no readable DataItem id), Galaxy (no attributeRef) and OpcUaClient
(no nodeId) were unlisted, and OpcUaClient's was the quietest of the twelve:
NamespaceMap drops such an entry with a bare `continue` and not even a log
line. SkippedTagReporterFleetTests pins the invariant reflectively over every
IDriver in src/Drivers/, so a thirteenth cannot re-introduce the silence.
Sql drops a tag at TWO points and its report is the union of both: a blob
that does not map, AND an identifier the 8.1 catalog gate rejects against the
live catalog. The second is what schema drift produces -- a renamed column, a
dropped view -- months after a deployment sealed green, and reporting only
the first would have that driver claim "0 skipped" while polling half its
tags, which is worse than reporting nothing because the column would then be
actively reassuring.
Deliberately additive to driver state, never a change of it. Latching
non-Healthy for a config problem would satisfy both halves of the #527
comms-loss conjunction -- 90 s without a successful read AND non-Healthy --
and a driver whose tags were ALL skipped has no successful reads by
construction, so it would reconnect-loop forever against a healthy device.
Three details that are load-bearing rather than incidental:
- The skip digest is part of the publish fingerprint. A skipped tag leaves
every other fingerprint component identical to a clean driver's, so without
it PublishHealthSnapshot's dedup swallows exactly the publish carrying the
signal -- the trap that already bit the rediscovery and host-status
channels. The digest covers the sample, not just the counts: a redeploy
that fixes one tag and breaks another leaves both counts unchanged.
- The count is exact; only the sample is capped, and the tooltip says so when
it is showing fewer than it counted. A truncated list that looked complete
would have an operator fix twenty tags and believe the job done.
- The ledger stages and commits, so an init that throws part-way leaves the
previous report standing. A partial snapshot would read as "we now skip
fewer tags", the opposite of the truth.
The proto field needs no companion presence flag, unlike host_statuses: a
singular message field carries real presence in proto3, and a has_ flag would
collide with protoc's generated HasSkippedTags.
Verified red-first three ways: dropping SkippedTagDigest from the fingerprint
fails the two dedup tests, removing the seam from one driver fails the fleet
guard, and dropping the catalog-gate leg fails the Sql union test. Full
solution builds with zero new warnings against 4e22f024.
Both `IDriver` members had no production call site. Their only consumer was
`MemoryTracking`, deleted with the driver-tier recycle machinery in #522, and
`GetMemoryFootprint`'s doc-comment claimed "polled every 30s by Core" — which
had not been true since v3, even before #522 removed the poller.
Owner decision on #525 was (a) delete rather than (b) build the process-wide
watchdog that would consume them. The driver-side implementations are in git
history and were mostly reusable; what was always missing is the trigger, and a
member nobody calls is how the misleading doc-comment survived this long.
Removed from `IDriver`, all 12 driver implementations, and every test stub —
plus the support code that existed only to feed them, which is the part a
member-only deletion would have left behind as dead weight:
- MTConnect's three per-entry byte-weight constants, its cached
`ProbeModelDataItemCount`, and `CountDataItems` — a full device-tree walk run
on every cache fill to populate a number with no remaining reader.
- OpcUaClient's `_discoveredNodeCount` field and the discovery-pass write to it.
- MQTT's `ApproxBytesPerAuthoredTag`.
Two behavioural notes, neither a change in what the software does:
- `AbCipTemplateCache` now has no production reset path. Its only caller was
AB CIP's flush, which itself had no caller — so nothing changes except how
visible the fact is. Recorded on the type, along with the correction that its
comment's claim about a device `ReinitializeAsync` also flushing it was never
true. A UDT redefined on a live controller keeps its stale shape until the
driver instance is rebuilt.
- MTConnect's `/probe` cache is still dropped, by `InvalidateProbeModel()` on an
Agent restart. That is now `internal` rather than `private` so the tests which
used the flush purely to empty the cache drive the real path instead — they
now cover a case that actually occurs in production. Several unrelated doc
comments cited `FlushOptionalCachesAsync` as the canonical CAS example and are
repointed to it.
Tests: the ones that existed only to assert on the removed members are deleted
(3 OpcUaClient, 3 Galaxy, 2 MQTT, 1 MTConnect, 1 AbCip). Worth noting one was
vacuous — `FlushOptionalCachesAsync_resets_footprint_counter` asserted a
footprint of 0 before the flush and 0 after with no discovery in between, so it
would have passed against a flush that did nothing at all.
Full suite green except 7 failures verified identical on master before and
after (4 AbLegacy integration with the compose profile down, 3
Host.IntegrationTests), checked by name rather than assumed.
Docs corrected where they described this as live: `DriverLifecycle.md` (which
carried the #525 placeholder), `drivers/AbCip.md`, `drivers/AbLegacy.md`, and
`v2/driver-stability.md` — whose "polls this every 30 s" + per-driver
`DriverConfig` memory budget were never built as written and are now banner-ed
as design intent rather than description.
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
#519 — a driver's OPC UA status code now reaches a client with its sub-code intact.
DriverInstanceActor projected the driver's uint onto a 3-state Good/Uncertain/Bad enum
and OtOpcUaNodeManager re-expanded Bad to a bare 0x80000000, so BadNoCommunication,
BadTypeMismatch and BadNotSupported were indistinguishable on the wire and every
driver's status constants (incl. the 16 #497 had just corrected) were diagnostic in
logs only. Resolved by option (b): the raw uint rides AttributeValuePublished ->
AttributeValueUpdate -> IOpcUaAddressSpaceSink.WriteValue, OpcUaQuality is deleted, and
severity classification is a helper (OpcUaStatus.IsGood/IsUncertain/IsBad).
The surface was wider than the issue costed (41 files, not "the sink and 3
implementations"): OpcUaQuality had non-driver producers with no uint to carry.
Each got its natural code — BadInternalError for a failed virtual-tag script,
BadNoCommunication for driver comms loss — and ScriptedAlarmHostActor's pair of
mutually-inverse uint<->enum mappers is deleted, having round-tripped a code the
engine already held. AddressSpaceApplier's removal writes deliberately keep the
generic Bad: not losing a code the driver had is this issue; inventing one for the
deploy path is a separate call.
Behaviour change: WriteAlarmQuality fires its Part 9 event on a status-code change
rather than a severity-bucket change. While the annotation was 3-state those were the
same predicate; now a move between two Bad reasons changes what a subscriber reads.
New SDK-free named constants live on Commons.OpcUa.OpcUaStatus, and
StatusCodeParityTests is widened to cover them — that guard is reflection-driven, so
it can only see a named constant, never an inline literal.
#517 — the symptom is real but the issue named the wrong path and its fix was unsafe.
ShutdownAsync cannot race: DriverInstanceActor calls it from PostStop, after the
mailbox drains. (Core/Hosting/DriverHost, the one class with a concurrent GetHealth,
has no production call site.) InitializeAsync can: it runs on a fire-and-forget
Task.Run, and MTConnectDriver writes Healthy then starts loops that can throw into a
catch which disposes the client before writing Faulted.
Moving the write earlier, as suggested, inverts the terminal state — teardown drains
in-flight loops that write health as they unwind, so a late loop could leave Healthy at
rest with every connection closed. The fix is a bracket (conservative before, terminal
after), applied to MTConnect's init-failure path only; shutdown keeps its single
trailing write, and the rule plus the reachability table are documented on
IDriver.ShutdownAsync.
CalculationDriver.GetHealth returned a hard-coded Healthy that consulted nothing —
Healthy before initialize, after shutdown and after dispose. Now derived from _running.
Its test, named Health_is_always_Connected, asserted the defect and is replaced.
Every new guard was falsified by breaking the production code and watching it go red.
The two sub-code tests each assert two different Bad codes, since a single-code
assertion could pass against a hard-coded constant under the old projection.
Pre-existing failures, verified unchanged on a stashed tree and not chased: 4 in
Driver.AbLegacy.IntegrationTests (slc500/micrologix/plc5 fixtures not up) and 3 in
Host.IntegrationTests.
Closes#517Closes#519
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
A calc-tag add/remove/edit or a script edit changes the merged DriverConfig,
which DriverSpawnPlanner routes to an in-place ApplyDelta (not a respawn), i.e.
CalculationDriver.ReinitializeAsync. The old ReinitializeAsync ignored the new
config: the tag table + dependency-ref set were built once at construction and
never rebuilt, so after a redeploy added tags never computed, removed tags kept
publishing, and edited scripts served stale results (deploy reported success).
Part 1 (HIGH): extract RebuildTagTable(config) — shared by the ctor and reinit
so they cannot drift — and have ReinitializeAsync re-bind the new config
(ParseOptions) and rebuild _tagsByRawPath/_changeDependents/_dependencyRefs/
_stateByRawPath under _lock, preserving last-known state for surviving tags,
dropping removed tags, and starting new tags fresh. _dependencyRefs now reflects
the new authored tags. Corrected the now-false ReinitializeAsync comment.
Part 2 (MEDIUM ordering hazard): the host's ReRegister was Tell-ed synchronously
alongside the ApplyDelta Tell, so the adapter re-read DependencyRefs before the
async ReinitializeAsync rebuilt them (stale set, no self-correction). Now
DriverInstanceActor sends DeltaApplied(driverInstanceId) to DriverHostActor
AFTER ReinitializeAsync completes, and HandleDeltaApplied re-registers only that
driver's adapter — sequenced after the rebuild. Removed the inline loop.
Test: added a redeploy test to CalculationDependencyFlowTests exercising the
real DependencyMuxActor + adapter + driver — a new tag reading a NEW dep flows +
computes, a removed tag stops publishing, an edited script serves the new
result, and DependencyRefs re-registers the new set. Fails on pre-fix code
(DependencyRefs stays stale ["src/a"]), passes after the fix.
Claude-Session: https://claude.ai/code/session_01LVneM3eh1UtJxEisFXgmox
Adds the Calculation pseudo-driver (tags computed by C# scripts over other tags' live
values) plus the host seam that feeds it:
- IDependencyConsumer capability interface (Core.Abstractions) + DriverTypeNames.Calculation.
- CalculationDriver : IDriver, ISubscribable, IReadable, IDependencyConsumer — change-gate +
dedupe (VirtualTagActor parity), timer-group scheduler, Bad-transition + script-log emission,
Good recovery; reuses the T0-4 CalculationEvaluator.
- DriverHostActor spawns a DependencyConsumerMuxAdapter per dependency-consuming driver,
registering its refs on the per-node DependencyMuxActor and forwarding DependencyValueChanged
→ OnDependencyValue; re-registers on every apply, torn down with the driver.
- Factory + probe registered in DriverFactoryBootstrap (keeps the T0-3 guard green); guard test
csproj references the driver so the bin-scan discovers the factory.
- DeploymentArtifact injects the resolved scriptSource (scriptId → SourceCode) into a calc tag's
delivered TagConfig so the host-blind driver can compile it.
- DraftValidator deploy gates: CalculationScriptMissing / CalculationScriptNotFound (scriptId
existence) + CalculationDependencyCycle (DependencyGraph.DetectCycles over calc→calc edges).
- Tests: driver units (dep-ref extraction, change-gate, dedupe, Bad+script-log+recovery, timer,
read), tag-config parsing, DraftValidator gates (self/2-cycle/cross-driver-terminal), and a
Runtime integration test proving values FLOW mux → adapter → driver → calc-of-calc end-to-end.
Claude-Session: https://claude.ai/code/session_01LVneM3eh1UtJxEisFXgmox
New Driver.Calculation project + CalculationEvaluator, mirroring the
VirtualTag RoslynVirtualTagEvaluator: source-hash compile cache
(CompiledScriptCache), 2 s TimedScriptEvaluator, passthrough fast-path,
single-tag mode (ctx.SetVirtualTag dropped + logged), sandbox/compile/
timeout/throw all surfaced as VirtualTagEvalResult.Failure for WP7 to map
to Bad quality. IScriptCacheOwner.ClearCompiledScripts for the deploy
generation boundary. 11 parity unit tests. Both projects added to the
solution. Evaluator + tests only — driver shell / registration is WP7.
Claude-Session: https://claude.ai/code/session_01LVneM3eh1UtJxEisFXgmox
Joseph Doherty