Follow-up to a94eb8a3, addressing CHANGES REQUESTED on code review. Production
logic was confirmed correct on all spec points; this commit fixes the review's
test/comment findings only.
IMPORTANT-1: A_permanently_unresolvable_pskProvider_is_reported_once_per_incarnation_not_once_per_refresh
did not prove the second refresh ran — its Expect(0) window gated on a dialer
dispatch landing, but LoadContactsFromDb is fire-and-forget and the first
block's AwaitCondition already re-Tells MeshCommand per poll, so the dialer
call count could reach the threshold before PublishKnownClusters was ever
re-invoked by the second load. Fixed by having the PSK factory count its own
invocations and gating both AwaitConditions on that count directly (pskCalls
>= 2 proves the first retry; pskCalls > pskCallsAfterFirstLoad, awaited INSIDE
the Expect(0) window, proves the second one). Bite-checked: with
_pskProviderAbsenceReported's guard temporarily removed, the test now
correctly fails ("Received 1 message too many... received 1 that matched
filter"); guard restored and reverified green.
MINOR-2: reworded four comments (CentralCommunicationActor.cs ~109, ~259,
~305-309, ~853) from narrating the change ("this fix removes", "already
existed, unchanged", "added by this fix", "the old two-call-site shape") to
stating the timeless invariant.
MINOR-3: added A_supplied_dialer_factory_that_always_throws_still_drops_each_command_loudly_while_PreStart_stays_quiet
— the symmetric case to the existing never-supplied-dialer test, exercising
Resolve's catch + firstUse:true path (a factory that WAS supplied and always
throws) rather than the null-factory path. Asserts PreStart plus the first
contact load stay silent and DispatchOverGrpc's existing per-dispatch Error
still fires on every dispatch, unguarded.
MINOR-4: "Check the ControlPlane registration wires IMeshPskProvider" ->
"Check that the ControlPlane registration wires up IMeshPskProvider"; no test
pinned the old substring, so no test update was needed for this one.
Skipped item 5 (restart-re-arms pin) per instruction — out of scope.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
PreStart's eager resolve of IMeshCommandDialer/IMeshPskProvider loses a
millisecond-wide race against the Akka builder's remaining starters on
essentially every admin boot (live gates: 1A 2/3 boots, 1B 4/4 boots) and
self-heals at the memoized Dialer/PskProvider properties' first use. Before
this change that produced two FALSE Errors per healthy boot, which made a
soak error-grep on a healthy rig impossible.
Resolve() now takes a `firstUse` flag and logs at Debug on BOTH call sites
(PreStart and the memoized properties' retry) — from inside Resolve a lost
race and a genuinely broken factory are indistinguishable, so severity is
decided by the CONSUMER of a still-null result after the retry, not by
Resolve itself:
- Dialer: DispatchOverGrpc's existing per-dispatch Error ("no command
dialer is wired") is unchanged and is now the ONLY loud signal for that
dependency.
- PskProvider: PublishKnownClusters gained a new Error carrying the old
PreStart guard's diagnostic content (gate never published, cached keys
never dropped, rotated key needs restart), guarded to fire ONCE per
actor incarnation via `_pskProviderAbsenceReported` — that handler runs
on the contact-refresh timer for the actor's whole life, so an
unguarded Error there would repeat every ContactRefreshSeconds forever.
Design choice for the "factory never supplied" case (Dps/ClusterClient test
constructions that pass no factories at all): kept silent at BOTH call
sites for both dependencies, same as before. For the dialer this was
already true (DispatchOverGrpc's check doesn't care why the dialer is
null). For the PSK provider the new once-guarded Error is gated on
`_pskProviderFactory is not null` specifically so the many existing
ClusterClient/Dps facts in CentralCommunicationActorTests that never wire a
PSK provider (using the bare Spawn helper) keep passing unmodified — a
never-supplied factory is a deliberate wiring/test choice, not a resolve
failure, and Resolve() already stayed silent about it for the same reason.
Two new tests pin the net behavior: a healthy boot (factories that throw
once then succeed) produces zero Warning/Error end-to-end through PreStart,
the first contact load and the first dispatch; a permanently-broken PSK
provider logs the absence exactly once across two contact loads, not once
per refresh.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
CRITICAL. The ack deadline was one absolute DateTime computed once for the whole
endpoint walk. Against the characteristic WAN failure — a central that HANGS
(SYN black-hole, frozen host) rather than refusing — endpoint 1 burned the whole
AckTimeoutSeconds and endpoint 2 was then dialled with a deadline already in the
past: instant DeadlineExceeded, no packet leaving the node, the surviving central
never really tried. Failover looked like it ran and did not. It also contradicted
MeshTransportOptions' own documented "applied per endpoint, not per ack". The
deadline is now computed fresh inside the loop; headers stay hoisted (read-only,
sequential walk).
Two test gaps that let that survive are closed: nothing pinned the ackRouter gate
(narrowing it back to ClusterClient-only compiled and passed every test while
dead-ending every Grpc ack to DPS), and the deadline fact's lower bound accepted
any positive timeout at all. Both new pins are mutation-verified.
Docs corrected where they overstated the case: duplicate acks CANNOT mis-seal a
deployment — ConfigPublishCoordinator.HandleAck keys _acks by NodeId behind an
expected-set guard, so a duplicate overwrites its own entry. The single-endpoint
rule is defence in depth and a bound on wasted work, not a correctness
dependency; and the walk is at-least-once on the deadline path, not at-most-once.
Task 1B.2 — the node→central half of the ClusterClient→gRPC migration.
GrpcApplyAckSender walks MeshTransport:CentralGrpcEndpoints in order and
STOPS AT THE FIRST SUCCESS. That single `return` is the whole single-count
guarantee: the transport it replaces was ClusterClient.Send, deliberately not
SendToAll, because both central nodes forward to the SAME
ConfigPublishCoordinator singleton proxy — an ack posted to both would have the
coordinator count one node twice and could seal a deployment on a doubled count.
The list is an availability list, not a fan-out list.
No retry beyond that one ordered pass, no queue: the receptionist buffered
nothing and retried nothing, and this is like-for-like, not an upgrade to
delivery guarantees. All endpoints failing is still a DROP — but a loud one,
naming the node, deployment, correlation id and the status each endpoint
answered with, which is what separates "central is down" from "central rejected
this node's key". The bearer never reaches a log; only endpoints and status
codes do. Per-endpoint GrpcChannel cache and the endpoint loop clone
GrpcDeploymentArtifactFetcher; the cluster id in x-otopcua-cluster comes from
this node's own cluster-<id> role via RoleParser, off the same bound
Cluster:Roles ClusterRoleInfo reads.
SendAsync never throws: the caller discards the task, so an escaping exception
would become an unobserved fault — a drop with no log line, the exact failure
mode this migration removes.
NodeCommunicationActor gains an OPTIONAL ackSender; every existing test compiles
and passes unedited. It is checked before the ClusterClient so the two can never
both fire. The registration arm creates no ClusterClient under Grpc.
NOTE — one wiring change beyond the actor: DriverHostActor's ackRouter is now
`useClusterClient || useGrpc ? nodeComm : null`. Left at ClusterClient-only, a
Grpc node's ack would have gone out on the DPS deployment-acks topic and never
reached the sender at all.
Two review items from Task 1B.1, both diagnostic:
1. CentralCommandGrpcService's "actor not ready" Warning omitted the peer and
the correlation id while the decode-failure Warning six lines above carried
both. These are the only two ways an ack fails to reach the coordinator, and
an operator has to correlate either against the node's own log. Decoding has
already succeeded on this path, so the DECODED correlation id is free.
2. A fused driver,admin,cluster-MAIN node under Mode=Grpc owes BOTH validator
surfaces at once (driver keys AND ConfigServe:GrpcListenPort). The two blocks
are independent `if`s; chained as `else if` a fused central node would be
graded on one surface only and boot with nowhere for any ack to land. Pinned
from both sides plus the fully-configured pass.
Phase 1A live gate defect 4. Neither central node booted on this branch,
in ANY transport mode:
Akka.Hosting.MissingActorRegistryEntryException: No actor registered
for key ...ControlPlane.AuditWriterActorKey
at AdminUI.Audit.ActorAuditWriter..ctor(ActorRegistry registry)
at ControlPlane.ServiceCollectionExtensions.<WithOtOpcUaControlPlaneSingletons>
WithOtOpcUaControlPlaneSingletons' first WithActors starter resolved
IMeshCommandDialer and IMeshPskProvider eagerly for the Props. Their
graph reaches back into the registry — dialer -> IMeshPskProvider ->
ISecretResolver -> AddZbSecrets -> IAuditWriter -> ActorAuditWriter,
whose ctor calls registry.Get<AuditWriterActorKey>() — and
WithSingleton<AuditWriterActorKey> is registered ~60 lines LATER in the
same builder. Boot died before the entry existed. Mode-independent: the
resolve happened before anything consulted MeshTransport:Mode, so Dps
(the default, and the rollback target) died exactly like Grpc.
Both dependencies now arrive as Func<> and are invoked in PreStart, by
which point the host is fully built — the same shape, for the same
class of reason, as the coordinator proxy Func one argument earlier.
PreStart re-resolves on restart rather than caching across one; both
are container singletons, so a restart gets the same instances. A
resolution throw is caught and logged rather than propagated: a throw
out of PreStart is an actor failure, the supervisor answers it with a
restart, and a broken DI graph is not transient — that is an unbounded
restart loop that would take the transport-independent ack path down
with it.
Regression pin: ControlPlaneStarterOrderTests boots a REAL admin host
through WithOtOpcUaClusterBootstrap + WithOtOpcUaControlPlaneSingletons
in production starter order, with the production graph from
IMeshCommandDialer down to a registry-bound IAuditWriter, and asserts
StartAsync itself — the failure occurs while the builder's starters
execute, so building a ServiceProvider and resolving by hand cannot
reproduce it. Verified RED against the eager resolve: 3/3 modes fail
with MissingActorRegistryEntryException. A second fact pins that the
deferral did not become 'never resolved' — the actor must resolve the
provider from the container and drive it through the contact refresh,
signalled by a TaskCompletionSource (PreStart runs on the actor thread
and the contact load is a piped DB read, so no synchronous assertion
after StartAsync is correct).
Why nothing caught this: the dialer's container-activation test resolves
from a collection with no secrets registration, so its graph stops one
hop in; the actor facts inject fakes and never touch DI; and the
existing whole-host boots register no mesh provider, so the eager
lookups returned null and walked nothing.
Two aggravators recorded by the gate, no code here: the failed process
did not exit (docker Up while /health/ready 503s — Phase 2 readiness
concern, tracked), and this is the third instance of the same
eager-resolve-during-AddAkka class after scadaproj#1 and mesh Phase 6's
IClusterRoleInfo boot crash.
ControlPlane.Tests 236 -> 240, Host.Tests 92, Cluster.Tests 184,
0 warnings in every touched project.
The known-cluster gate published only under Mode=Grpc, but the
interceptor it feeds is registered on every admin node and 1B maps
CentralCommandService by LISTENER PORT — neither consults the mode. A
dark-switch rollback to ClusterClient or Dps would therefore have left
that endpoint serving with the gate permanently inactive, silently
restoring both the timing oracle and the pre-auth secret-store/audit
flood, as a side effect of a rollback nobody would connect to
authentication. The gate is central's cluster inventory, not transport
state, so it is now decoupled from the mode entirely:
- SetKnownClusters + InvalidateAll run on EVERY successful contact load
in EVERY mode, and the contact-refresh timer starts in all three.
Under Dps this is a ConfigDb query every ContactRefreshSeconds that
did not run before; central always has a ConfigDb, and one indexed
read a minute is the cheaper half of the trade.
- The GATE set is built from ALL ClusterNode rows. The eligibility
filter moved out of the query's WHERE and into the loop, so the FAN
set is byte-for-byte what it was (enabled && !maintenance) while the
gate widens. This also dissolves the maintenance coupling flagged in
1A.2: a cluster whose nodes are all disabled or in maintenance can
still ack, and refusing it would make maintenance mode an
authentication switch. Central still refuses clusters it has never
heard of.
- HandleContactsLoaded keeps a documented order — targets adopted, gate
published, cache dropped — now pinned by test.
Guards: PreStart mirrors the null-dialer error for a null PSK provider
(silent absence is the worst outcome there — nothing malfunctions, so
nothing is investigated, while the gate never publishes and keys never
rotate); a failed contact load now says out loud that the gate is stale
and, before any load has succeeded, inactive; MeshGrpcCommandDialer
clamps a non-positive DispatchTimeoutSeconds to 10 with a construction
Warning, because the validator for that key is driver-role-scoped and a
pure-admin central reads it unvalidated; the dialer's local PSK refusal
was shortened so its remediation survives the actor's 200-char
truncation of remote-authored status detail; the portless-node warn-once
set is rebuilt each refresh, so a node that regains then loses a port
warns again — the regression is the occurrence worth seeing.
Docs/pins: dialer disposal comment no longer claims one-way; the two
interceptor-test comments claiming Matches has a default implementation
corrected (it deliberately has none); new pins for per-refresh
invalidation, the SetKnownClusters-before-InvalidateAll order, and one
channel per endpoint via a new CachedChannelCount.
The_known_cluster_set_is_not_published_by_the_other_two_modes is
replaced — its premise is obsolete — by three facts: every mode
publishes, gate set is a strict superset of the fan set, and the handler
ordering.
ControlPlane.Tests 230 -> 236, Host.Tests 92, Cluster.Tests 184,
0 warnings.
MeshGrpcCommandDialer places one NodeCommandService.Execute call on one
node: a GrpcChannel cached per endpoint (the GrpcDeploymentArtifactFetcher
shape), the TARGET node's cluster's bearer plus x-otopcua-cluster naming
that cluster, and a per-call deadline linked to the caller's token. A
cluster with no resolvable key is refused locally with Unauthenticated —
there is no arm on which a command leaves central unauthenticated, even
one the far side would have rejected.
CentralCommunicationActor gains the third mode branch. The contact load
now also projects GrpcPort and, from the SAME rows in the SAME pass,
builds the dispatch set — which is what makes 'the gRPC fan reaches
exactly the SendToAll recipient set' structural rather than two queries
that have to keep agreeing. A row with a null GrpcPort is skipped
warn-once (the TelemetryNodeSource precedent). Per-node results are
piped back for LOGGING ONLY: named node, gRPC status, dropped. No retry,
no queue.
Every dial is scheduled on the thread pool rather than called from the
receive. Found by review and pinned red-first: an async method runs
synchronously to its first await, and the dialer's pre-await work is
TryGetPsk, which blocks on the secret store — bounded but seconds, and
repeated after every refresh since the refresh drops the cache. Dialing
inline parked the actor's mailbox, so deployment acks queued behind a
slow node; the regression test tells an ApplyAck while a dial is held
and fails without the hop.
The contact refresh also wires the Phase 1A register: it publishes the
loaded cluster ids as the PSK provider's known-cluster set and drops the
key cache, so a rotated key takes effect within one ContactRefreshSeconds.
Neither happens under Dps or ClusterClient.
Both existing mode branches are behaviourally untouched and every
existing fact passes unmodified. The one edit to shared text: the
Status.Failure warning drops the word 'ClusterClient', since that handler
now serves all three transports.
ControlPlane.Tests 205 -> 230, Host.Tests 92, Cluster.Tests 184, 0 warnings.
Phase 1A security register items 1 and 2, landed ahead of the command
fan-out because they are independently testable and the fan-out is the
first caller of TryGetPsk.
Item 1 — known-cluster gate. MeshPskProvider gains SetKnownClusters, an
atomic whole-set swap consulted by TryGetPsk BEFORE the config map and
before the secret store. A cluster id outside the set is refused in O(1)
with no store read and no audit write, which closes two things at once:
the pre-auth store/audit flood an unauthenticated caller could drive with
a forged x-otopcua-cluster header, and the timing oracle that defeated
CentralCommandAuthInterceptor's uniform denial text (unknown clusters are
never cached, so they paid a store read on every probe while known ones
answered from memory). Empty or never-published leaves the gate inactive
— central publishes the set from its DB contact refresh, and 'the DB told
me nothing' must not become a fleet-wide authentication outage during
admin boot.
Item 2 — bool Matches(clusterId, presented) on IMeshPskProvider, with
FixedTimeEquals inside the provider. CentralCommandAuthInterceptor now
calls it and never holds key material; its two provider-side denial logs
collapse into one (the provider answers both as false) and the operator
hint is restated there.
No production caller publishes a cluster set yet, so behaviour is
unchanged until the fan-out commit wires the contact refresh.
ControlPlane.Tests 190 -> 205, Host.Tests 91 -> 92.
Found on the docker-dev live gate, not in tests. Two drivers that faulted
during initialise (rtu-gate, opcua1) each rendered "0 tags" on /hosts against
two authored tags apiece, because a ledger that had never committed returned
SkippedTagReport.Empty.
A zero report is a positive claim -- "this driver skipped nothing" -- and
making it on behalf of a driver that never got as far as looking is exactly
the quiet inaccuracy #541 exists to remove. GetSkippedTags() and
SkippedTagLedger.Snapshot() now return null until the first committed build,
so the column renders the same em-dash it already used for a driver that
cannot report at all. Both are 'no answer'; neither is 'nothing skipped'.
Knock-on: a FOCAS driver whose only init threw now reports null rather than a
zero report, which is strictly more honest than the assertion the test
originally made.
A driver builds its RawPath -> definition table at init and tolerates a
per-tag miss: an unmappable TagConfig, an unparseable address, a
capability-matrix rejection. That tolerance is right -- one bad tag must not
take down the other few thousand. What was missing is any statement of the
fact afterwards. A skip is not an error and happens before any read, so the
driver stays Healthy, errors/5 min stays 0, Last error stays empty, and Last
read advances live off the tags that did map. On the wonder Makino in #540
that reading held while focas-z-34184 ignored 20 of its 22 tags.
New optional capability ISkippedTagReporter, backed by a shared
SkippedTagLedger so the reset/commit/cap rules are one implementation rather
than one per driver. The report -- authored count, exact skipped count, and a
capped sample of RawPath + reason -- rides DriverHealthChanged to a new Tags
column on /hosts, over the Phase 5 telemetry stream. No table: a driver-only
node has no ConfigDb to write to, the lesson from #521.
The issue named eight drivers. It is twelve -- every driver in
DriverTypeNames.All has an authored-tag skip path. Calculation (no scriptId),
MTConnect (no readable DataItem id), Galaxy (no attributeRef) and OpcUaClient
(no nodeId) were unlisted, and OpcUaClient's was the quietest of the twelve:
NamespaceMap drops such an entry with a bare `continue` and not even a log
line. SkippedTagReporterFleetTests pins the invariant reflectively over every
IDriver in src/Drivers/, so a thirteenth cannot re-introduce the silence.
Sql drops a tag at TWO points and its report is the union of both: a blob
that does not map, AND an identifier the 8.1 catalog gate rejects against the
live catalog. The second is what schema drift produces -- a renamed column, a
dropped view -- months after a deployment sealed green, and reporting only
the first would have that driver claim "0 skipped" while polling half its
tags, which is worse than reporting nothing because the column would then be
actively reassuring.
Deliberately additive to driver state, never a change of it. Latching
non-Healthy for a config problem would satisfy both halves of the #527
comms-loss conjunction -- 90 s without a successful read AND non-Healthy --
and a driver whose tags were ALL skipped has no successful reads by
construction, so it would reconnect-loop forever against a healthy device.
Three details that are load-bearing rather than incidental:
- The skip digest is part of the publish fingerprint. A skipped tag leaves
every other fingerprint component identical to a clean driver's, so without
it PublishHealthSnapshot's dedup swallows exactly the publish carrying the
signal -- the trap that already bit the rediscovery and host-status
channels. The digest covers the sample, not just the counts: a redeploy
that fixes one tag and breaks another leaves both counts unchanged.
- The count is exact; only the sample is capped, and the tooltip says so when
it is showing fewer than it counted. A truncated list that looked complete
would have an operator fix twenty tags and believe the job done.
- The ledger stages and commits, so an init that throws part-way leaves the
previous report standing. A partial snapshot would read as "we now skip
fewer tags", the opposite of the truth.
The proto field needs no companion presence flag, unlike host_statuses: a
singular message field carries real presence in proto3, and a has_ flag would
collide with protoc's generated HasSkippedTags.
Verified red-first three ways: dropping SkippedTagDigest from the fingerprint
fails the two dedup tests, removing the seam from one driver fails the fleet
guard, and dropping the catalog-gate leg fails the Sql union test. Full
solution builds with zero new warnings against 4e22f024.
Authoring FOCAS tags through /raw -> Browse device... produced tags the driver
could never read. Every committed tag read BadNodeIdUnknown forever, and the
failure was silent end to end: the AdminUI accepted it, the deploy gate sealed
green, and the driver reported Healthy with 0 errors while ignoring the tags.
Net effect: the whole fixed-tree surface could not be published through any v3
authoring path. Found cutting wonder-app-vd03 over to v3 against a real Makino
Pro 5 / Fanuc 30i, where 0 of 20 committed tags read while two hand-authored
MACRO: tags on the same device read Good.
A captured browse leaf's id IS its DriverAttributeInfo.FullName, and the fixed
tree emitted "{deviceHost}/{path}" there -- an internal cache key, not an
address. Both authoring paths wrote it verbatim into TagConfig.address, and
FocasAddress.TryParse accepts no such form, so the driver skipped the tags at
init.
Fixed at the emitter rather than by stripping the host at commit, which fixes
the browse-commit flow and the manual picker's tree-select together and leaves
one canonical address form:
- FocasAddress gains a fourth kind, FIXED:<path>, and moves to Contracts so the
AdminUI and ControlPlane can parse without referencing the runtime driver
(#526). A FIXED: read is served from the fixed-tree snapshot the poll loop
already maintains -- no area read, no connection. Writes refuse: nothing in
that tree is a writable location on the CNC.
- The address carries no device. A tag's device comes from its Device row, and
a second routing channel for one fact is what created this defect.
- An authored FIXED: tag arms the fixed-tree poll loop by itself. Enabled
defaults false and the browser force-enables it only for a browse, so the
operator would otherwise browse, commit, deploy, and land on a driver whose
loop never starts -- every tag Bad with a correct address.
- Fixed-tree values are now dated from the last answered poll and go
BadCommunicationError once it is stale (8 cadences, floored at 30s). The poll
loop swallows a failed tick by design, so a dead CNC would otherwise serve its
last snapshot as Good indefinitely -- harmless while nothing could bind to the
tree, not harmless now.
Guardrails, so this class cannot recur silently:
- FocasTagConfigModel.Validate() runs the real parser instead of a non-empty
check, and FocasTagDefinitionFactory.Inspect() warns at the deploy gate, which
covers CSV/API config that never touches the modal.
- The manual picker offered axis/spindle/program/status "parameter groups" --
none of which is a FOCAS addressing space -- and its own default value,
axis:5, was unusable. It now composes PMC / PARAM: / MACRO: through
FocasAddress, so what it emits is parseable by construction.
Tests assert addresses ROUND-TRIP through the parser rather than matching an
expected string: the previous FocasAddressBuilderTests pinned "axis:5" exactly,
which is how a green suite coexisted with a picker nobody could use. Verified
red-first against the old emitter. FOCAS 308, AdminUI 1095, ControlPlane 169,
Core 581 -- all pass, 0 warnings.
The remaining half of #540 -- a driver reporting Healthy/0 errors while
skipping tags at init -- is fleet-wide (eight drivers do this) and is filed
separately rather than patched per-driver.
SqlDriverProbe.DefaultDialectFor was left SqlServer-only when the provider
gates were widened, so AdminUI Test Connect failed for both new providers
with InvalidOperationException while Deploy and browse worked. Found by the
live chrome/AdminUI gate the plan deferred, not by any test: every case in
SqlDriverProbeTests builds its probe through ForTest(), which injects the
dialect and never reaches the production switch.
Widen the switch arm-for-arm with SqlDriverFactoryExtensions.CreateDialect,
and cover the gate directly via an internal seam — plus a reserved-provider
falsification control and an end-to-end production-constructor case.
Two disposable docker fixtures (postgres:17-alpine on 15432, gvenzl/oracle-free:
23-slim on 11521) shaped like the existing mssql one — project label, distinct
container name, healthcheck, one-shot seed service — plus a live suite per
provider, gated on SQL_PG_* / SQL_ORACLE_* and skipping clean offline.
Each suite covers what a DIALECT can get wrong, rather than re-running the
provider-agnostic read semantics already pinned against SQL Server and SQLite:
liveness + catalog load, a KeyValue read, the provider's own row-limit syntax via
topByTimestamp (LIMIT 1 / FETCH FIRST 1 ROWS ONLY, where T-SQL emits TOP 1),
identifier case-folding through the design §8.1 catalog gate, and the AdminUI
browse walk driven through SqlDriverBrowser.
Two seed decisions are load-bearing rather than scaffolding:
- The Postgres schema is owned by a role the suite is NOT; the connecting login
holds USAGE + SELECT only. That is the exact provisioning PostgresDialect's
ListSchemasSql exists for (information_schema.schemata lists only schemas the
role OWNS, so a granted-not-owning poller would see none and every tag would be
rejected). A fixture that owned its own schema would pass either way.
- Every identifier in both seeds is created UNQUOTED, so the catalogs hold
lower-case (Postgres) and UPPER-case (Oracle) spellings. The case-folding legs
author the opposite case and pin SqlCatalog.TryMatch's REAL contract — exact
ordinal first, then a UNIQUE case-insensitive match whose catalog spelling is
substituted — which on both providers is what makes mixed-case authoring work
at all, since both dialects emit case-sensitive double-quoted identifiers.
Connection strings are composed through dialect.Factory.CreateConnectionString-
Builder() and the abstract DbConnectionStringBuilder indexer, so the csproj keeps
its no-direct-provider-reference discipline; every key used was verified accepted
through that indexer. The csproj gains only Driver.Sql.Browser, for the browse leg.
Live-run evidence (local OrbStack, both images arm64): Postgres 5/5 green;
Oracle 6/6 green after the ORA-01745 dialect fix this suite found. Offline the
whole project is 33 skipped / 0 failed.
Found by the new OracleReadTests against a real Oracle 23 Free: EVERY Oracle
driver instance failed its Initialize-time catalog load, and every AdminUI browse
of a table's columns threw, with
ORA-01745: invalid host/bind variable name
Oracle rejects a bind variable named after a reserved word, and the catalog
readers bind a parameter whose base name is `table` (SqlCatalogLoader /
SqlBrowseSession bind `schema` + `table`), so `ListColumnsSql` could not execute
at all. The provider was structurally non-functional.
OracleDialect now namespaces every marker/bind name through one `p_` constant
(`:p_k0` / `p_k0`), which a reserved word can never collide with, and its catalog
SQL is respelled to match. Prefixing everything rather than carrying a
reserved-word list is deliberate: the list is long and version-dependent, and a
future base name (`date`, `user`, `comment`, `level`…) would silently re-open the
same hole. Marker and bind name are derived from the same constant, so they
cannot drift apart.
No offline test could have caught this — only a server parses the SQL — so the
regression test asserts the invariant that made it possible (a bind name is never
the bare base name), with `table` called out explicitly.
Live evidence: 6/6 OracleReadTests red with ORA-01745 before, 6/6 green after,
against gvenzl/oracle-free:23-slim.
The fixture Agent flips to AllowPut = true (its comment rewritten: writes are in
scope now that the driver implements IWritable) and the suite grows three
env-gated write legs, all still Assert.Skip-clean with no Agent reachable.
Devices.xml gains a DataItem, and that is the deliberate part. The write target
has to be one the SHDR adapter NEVER feeds, or the assertion races the stream --
but it must NOT be the existing never-fed fixture_block: the first successful
write takes an item off UNAVAILABLE for the life of the container, which would
silently rob Read_maps_an_unavailable_observation_to_BadNoCommunication of its
subject. So fixture_writeback (free-text EVENT, PROGRAM_COMMENT) is a separate,
dedicated target, overridable via MTCONNECT_WRITE_TARGET for a run pointed at a
real machine tool.
The three legs:
* Round trip -- driver write accepted, then the injected value read back off
/current over plain HTTP (not through the parser under test) AND through the
driver's own ReadAsync. The value carries a TickCount64 suffix: the Agent
keeps an injected observation for the container's life, so a constant would
be satisfied by a previous run's leftover.
* Agent refusal -> BadCommunicationError. The design doc doubted this leg was
constructible ("an id in the probe but refused by the agent"); it is, and the
mechanism is BuildWriteTargets. Write targets are derived from the tags the
BINDING serves, with the device resolved from the probe model and falling
back to the configured deviceName -- so an authored tag naming an id the
Agent never declared, on a driver whose deviceName IS set, resolves to a real
URL and is POSTed. That is exactly the shape a stale tag has after a DataItem
is renamed. The second half of the test is the falsifiability control: the
SAME unknown id with NO deviceName has no URL and must be refused locally as
BadNodeIdUnknown. Two different codes is what proves the first write actually
reached the Agent rather than being short-circuited into a look-alike.
* Disabled gate -- write.enabled absent returns BadNotWritable AND /current is
unchanged from the snapshot taken just before, which is what distinguishes
"refused" from "reported refused and sent anyway".
RAN LIVE, not just compiled: the fixture was brought up locally (OrbStack,
MTCONNECT_AGENT_HOST_PORT=5555 -- macOS AirPlay squats :5000) and the whole
suite passed 15/15, 0 skipped. Evidence beyond the green: the Agent reports
<ProgramComment dataItemId="fixture_writeback">OTOPCUA-WRITE-2CBCB530</...> and
its log carries "Error processing request:
/OtFixtureCnc?otopcua_no_such_data_item_2CBCB3D9=refuse-me" -- the refusal leg
demonstrably went to the wire. Offline (no Agent) the suite is 15 skipped, 0
failed; build 0 warnings / 0 errors.
Note for anyone reading agent.cfg: BufferSize = 14 is an EXPONENT (2^14 = 16384
observations), not a 14-deep ring, and /current is answered from the Agent's
per-DataItem checkpoint anyway -- so eviction is not a hazard for a write
read-back even on this busy fixture stream.
LiveOptions/ConfigJson/LiveDriverAsync gained writeEnabled + deviceName
parameters, both defaulted off/null so every pre-existing test still runs
against the shipped read-only driver whatever the Agent's AllowPut says. The
config JSON sources write.enabled FROM the options object, so the parsed config
can never disagree with the options a test thinks it built.
Widens the two provider gates so an authored `{"provider":"Postgres"}` or
`{"provider":"Oracle"}` binds that provider's own dialect instead of failing
"not available in this build":
- `SqlDriverFactoryExtensions.CreateDialect` gains a Postgres and an Oracle
arm. There is deliberately no fallback arm — a default resolving to
SqlServerDialect would construct happily and then emit T-SQL at a Postgres
or Oracle server. The rejection message now names the three supported and
the two still-reserved providers.
- `SqlDriverBrowser.DefaultDialectSelector` widens identically, so a provider
the runtime constructs is also browsable; MySql/Odbc keep resolving to null
and the caller keeps reporting them as unavailable.
Tests: the existing rejected-provider controls flip from Postgres to MySql
(factory) and Odbc (browser) so the reserved half stays falsifiable, and the
new legs assert the bound dialect TYPE rather than mere construction —
"no throw" alone would pass against a SqlServer fallback.
Doc-comment sweep (contract addition, one line each): the "v1 constructs
SqlServer only" / "reserved for P2" claims in SqlProvider, SqlDriverConfigDto,
ISqlDialect.Provider and SqlServerDialect are now false and were corrected in
the same change.
TryFormatValue's catch filter named only FormatException / InvalidCastException
/ OverflowException, and the call sits OUTSIDE WriteAsync's per-slot try. But
Convert.ToString falls back to the value's OWN ToString/IFormattable - arbitrary
caller code that may throw anything - so any other exception escaped WriteAsync
uncaught and failed the entire batch, breaking the documented never-throws
per-slot contract. Broadened to a catch-all; the slot gets BadTypeMismatch and
the exception is logged at Warning (TryFormatValue became an instance method to
reach _logger). One unformattable value now costs its own slot and nothing more.
Falsification checked, not assumed: restoring the narrow filter makes the new
test fail with a raw "System.InvalidOperationException : this formatter is
broken" propagating out of WriteAsync - i.e. it fails for the contract-breach
reason, not on an assertion.
Two tests added:
- a value whose ToString throws InvalidOperationException is refused
BadTypeMismatch while the *following* good slot still writes (the load-bearing
half - it proves the batch survived).
- mid-batch cancellation: cancellation injected from inside slot 1's own
formatting, so slot 0 has provably already reached the Agent. It keeps its
real Good outcome while slots 1-2 fill BadCommunicationError. Reporting a
landed write as failed is worse than reporting a failure, and a fill that
started at 0 instead of i would do exactly that.
Tests: 534 passed / 0 failed (was 532). Solution build: 0 errors, no warnings
in any MTConnect project.
MTConnectDriver now implements IWritable over the Task-1 agent-client seam
(WriteObservationAsync -> the Agent's AllowPut POST surface). A write is
OBSERVATION INJECTION, not machine command: it sets what the Agent reports for
a DataItem, is visible to every consumer of that Agent, and is overwritten by
the next adapter/SHDR update. Off at two independent gates, both off by default
- the Agent's own AllowPut and the driver's write.enabled (Task 0).
Per-slot refusal order (never throws, strict N-in/N-out batch shape):
write disabled -> BadNotWritable
no live client -> BadNotConnected (ReadAsync's existing convention)
reference unresolved -> BadNodeIdUnknown
no resolvable device -> BadNodeIdUnknown
CONDITION category -> BadNotWritable
null / non-scalar value -> BadTypeMismatch
agent refusal OR throw -> BadCommunicationError
OperationCanceledException fills the remaining slots and returns. The Agent's
refusal text is logged at Warning and never enters DriverHealth.LastError
(type-only discipline); a refused write leaves health untouched.
Device-name resolution: TagBinding gains WriteTargetsByDataItemId, built in
BuildBinding from a probe-model walk that remembers which device each DataItem
was declared under (name, or id when nameless) plus its category. Falls back to
options.DeviceName; unresolvable -> BadNodeIdUnknown rather than a guess.
Values render via Convert.ToString(..., InvariantCulture) - so a Boolean writes
the invariant string "True"/"False", NOT "1"/"0". Arrays are refused rather
than rendered: Convert.ToString would emit the literal "System.Double[]", which
the Agent stores happily and which is invisible until read-back.
FINDING - the CurrentWrite bit does NOT key on SecurityClassification, and the
discovery stamping below is therefore cosmetic. Traced end to end: a browsed
leaf's SecurityClass survives into the browse session's AttributeInfo
(CapturedTreeBrowseSession.cs:53) and stops there - RawBrowseModal never reads
it, RawBrowseCommitMapper.MapLeaf hardcodes TagAccessLevel.Read, and Tag has no
SecurityClass column. AddressSpaceComposer:746/:809 sets Writable purely from
Tag.AccessLevel == ReadWrite; AddressSpaceApplier ANDs !IsArray;
OtOpcUaNodeManager.ComposeAccessLevel:2002 composes CurrentWrite from that
alone. Driver capability is never consulted at compose time either - IWritable
is checked only at runtime in DriverInstanceActor:755. Consequences: (a) raw
tags already committed keep their authored AccessLevel, so nothing this commit
does makes an existing node writable; (b) an operator must still set the tag to
ReadWrite for an MTConnect write to be reachable over OPC UA; (c) conversely a
ReadWrite tag on a non-IWritable driver has always advertised CurrentWrite and
failed only at the actor - pre-existing, not touched here.
Discovery stamps CanWrite(writeEnabled, dataItem) = write.enabled && !CONDITION
-> Operate, else ViewOnly - kept honest so the browse never advertises a tier
the write path would refuse (MQTT's CanWrite principle), with the caveat above
recorded verbatim at the call site.
Status-code constants are spelled WITHOUT MqttDriver's "Status" prefix, to
match the two this file already declares (BadCommunicationError,
BadNotConnected). StatusCodeParityTests strips either spelling and passes.
TDD: WriteAsync landed first as a NotImplementedException scaffold - 19 of 20
new tests red on behaviour, 1 (the capability assertion) green on the scaffold
- then implemented. MTConnectFactoryTests' "not_writable" characterization test
is rewritten to pin the new truth, and is the guard for the concrete-return
contract the factory remark explains.
Tests: 532 passed / 0 failed (was ~509). Full solution build: 0 errors,
0 warnings; OTOPCUA0001 clean.
ISqlDialect gains ParameterMarker (the token in the SQL text), ParameterBindName
(the DbParameter.ParameterName) and ConfigureCommand (per-command provider setup,
where Oracle will set BindByName). SQL Server and the test SQLite dialect answer
"@"+name for both spellings and no-op the hook, so SQL Server behaviour is
byte-identical — every existing expected-SQL assertion passes untouched.
SqlGroupPlanner drops its hardcoded @k/@w consts and mints base names (k0..kN, w)
that the dialect spells; SqlQueryPlan carries the emitted markers AND the matching
bind names, so SqlPollReader keeps copying the plan's pairing rather than
re-deriving a provider convention (the two diverge on ODP.NET: ":k0" in the text,
bare "k0" on the parameter). SqlCatalogLoader and SqlBrowseSession bind
schema/table through the dialect, so their bind names cannot drift from the
markers in the dialect-owned catalog SQL. Every parameter-carrying command site
calls ConfigureCommand before its first parameter.
Controlled by a new test-only ColonMarkerDialect (":name" markers, bare bind
names) that really executes against the SQLite fixture. Each leg was falsified
against the pre-change behaviour: the planner emitted "@k0", the reader bound
":k0", and a hardcoded "@schema" in the loader failed catalog load outright.
Both defects were found by #491's live gate against a real HistorianGateway and filed rather than
fixed at the time, because each one's fix is a design decision rather than a patch.
#537 — both nodes of a redundant pair captured every value, so the historian received duplicate
live samples for every historized tag (measured: the gateway's write counter advanced twice per
value and halved when one central was stopped).
The recorder now reads a capture gate per value, fed from the new
PrimaryGatePolicy.ShouldRecordHistorizedValues through IHistorizationRoleView — a third verdict on
the same RedundancyRoleView object, registered unconditionally on driver nodes rather than
alongside the alarm section's IRedundancyRoleView (which only exists when that section is enabled;
piggybacking is the #511 registration accident).
Two things are deliberate and easy to get backwards:
* It gates the CAPTURE, not the drain — the opposite of the alarm sink. That queue is in LocalDb
and replicates, so the Secondary already holds the Primary's rows. The historization outbox is
a node-local FasterLog replicating to nobody, so a drain-only gate would leave the standby
appending forever to a queue no one empties: unbounded disk, then a flood of stale duplicates
the moment it is promoted. Entries a node already captured are still drained by that node.
* The peer it stands down for must RECORD THE SAME REFS, not merely hold the same rows — a
Primary in this node's own per-cluster mesh. A node carrying no cluster-scoped role cannot
establish that and never stands down: under a fleet-wide election the elected Primary is
usually running a different cluster's configuration, and deferring to it would end this
cluster's historization silently, with the drain reporting success on an empty outbox forever.
That is the scope mismatch that once suspended the alarm drain on every node of the rig.
An unknown role captures, matching the alarm drain — a duplicate sample is recoverable, a hole in
the trend is not.
#538 — the drain writes every per-tag group and then acks all-or-nothing (deliberate: RemoveAsync
truncates THROUGH an id, so partial acking could drop older unwritten entries). One permanently
failing tag therefore pinned the FIFO head while every healthy tag in the batch was re-submitted
to the historian on every pass, indefinitely — duplicate samples and an outbox that never
truncated, observed live at 8.3 MB from a single unprovisionable tag.
The drain now tracks consecutive failures per historian tag and abandons that tag's entries once
it exceeds MaxWriteAttempts (new, default 5), so the head advances. The counter only advances on a
pass where some OTHER tag wrote successfully: that is what separates "this tag is poison" from
"the backend is down". A count-only quarantine looks identical on the poison-tag case and silently
discards the whole backlog during an ordinary gateway restart, turning a duplicate-samples bug
into a lost-history one. Abandonment is also deferred until every failing tag is over budget, so
nothing is discarded on a pass that could not have advanced the head anyway. Each abandonment logs
a Warning naming the tag; the total rides RecorderStatus.AbandonedEntries.
Capacity also gains a non-zero default (500,000 entries) as the disk backstop for the one case the
quarantine deliberately cannot resolve — a historian refusing every write, where retaining is
correct. Set 0 to restore unbounded retention.
Every new test was falsified by reverting the production change it guards, and each one reddens
alone:
* drop the backend-reachable evidence -> only A_backend_that_fails_every_tag_abandons_nothing
* never abandon -> only A_permanently_failing_tag_is_abandoned_so_healthy_tags_stop_being_rewritten
* latch the capture gate at construction -> only The_capture_gate_is_re_read_per_value...
* treat any Primary as a scope peer -> only the two scope-mismatch cases
Full suite green. (Four OpcUaClient.Browser failures during the run were the opc-plc fixture being
down on the shared Docker host; brought it back up and they pass.)
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
Ran #491's live gate against a real ZB.MOM.WW.HistorianGateway talking to a real AVEVA
Historian, and it found a production-fatal defect on the first value it ever saw.
ContinuousHistorizationRecorder.OnValueChangedAsync touched `Self` after awaiting
IHistorizationOutbox.AppendAsync(...).ConfigureAwait(false). ConfigureAwait(false) discards
Akka's ActorTaskScheduler synchronization context, so `Self` — which reads ActorBase.Context —
threw NotSupportedException. The throw escaped into supervision, which RESTARTED the actor;
because production seeds an EMPTY ref set (the deployed set arrives later via
UpdateHistorizedRefs), the restart silently discarded every historized ref. Net effect: one
value captured, then nothing, with no further error to show for it. Fixed by capturing
`IActorRef self = Self` before the await — the idiom the GetStatus handler already applied to
Sender.
Thirteen unit tests missed it because InMemoryOutbox.AppendAsync returned
ValueTask.CompletedTask: a completed ValueTask never yields, so the continuation ran inline
with the context still installed. The production FasterLog outbox fsyncs per entry and does
yield. The first regression test was itself vacuous — `await Task.Yield()` posts back to the
CAPTURED context, so the append still completed on the actor thread and reverting the fix left
the test green. The fake must complete on a POOL thread (Task.Delay(1).ConfigureAwait(false)),
as a real fsync does; with that, reverting the fix reddens exactly one test.
The rig could not run this gate at all: docker-dev carried ServerHistorian__* but no
ContinuousHistorization__* keys, so the recorder could never spawn. Both centrals now carry
them (default OFF — the recorder needs ContinuousHistorization:Enabled AND
ServerHistorian:Enabled), with the outbox on the per-node named volume so it survives a
container recreate. ServerHistorian__UseTls was hard-coded true and is now parameterized; a
plaintext h2c gateway was unreachable without it.
Gate results: outbox append, drain, WriteLiveValues accepted by a real historian, and restart
convergence (capture resumes after a restart with no new deployment) all pass. The read-back
hop is NOT proven — the LAN historian accepts writes but serves no reads under a "License
violation: Database tag count exceeds license tag count", and the licensed corporate one needs
the VPN. EnsureTags provisioning is likewise unexercised there (RemoteGrpc-only, and that CAP
is not listening). #491 stays open for the read-back hop alone.
Also recorded, not fixed: the recorder has NO Primary gate, unlike the alarm store-and-forward
sink, so both nodes of a redundant pair send duplicate live values. Gating it would introduce a
failover window with no capture — that is a design decision, not a cleanup.
Evidence: docs/plans/2026-08-01-continuous-historization-live-gate.md
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
A browse-committed AB CIP / TwinCAT / FOCAS tag bound with its model's DEFAULT
data type, not the one the device reported. A browsed REAL committed as
`{"tagPath":"Pump_1_Speed","dataType":"DInt"}` — and those three drivers decode
the wire bytes by that field, not by the raw row's OPC UA DataType — so the tag
read four bytes of a float through an integer decoder and published a plausible
wrong number with no error raised anywhere. FOCAS was missed when #536 was filed.
The type is CARRIED, not derived. Reverse-mapping the leaf's driver-agnostic
DriverDataType looks like the smaller fix and is wrong: SInt/Int/DInt/USInt/UInt
all collapse into Int32 on the way out of the driver, so that version would bind
a 16-bit INT as a 32-bit DINT while appearing to fix the REAL. Drivers now state
their own type name on DriverAttributeInfo.NativeDataType, which reaches the
picker as AttributeInfo.AddressFields["dataType"] and the commit mapper from
there. A leaf that states nothing keeps the model default — the pre-#536
behaviour, kept deliberately as the fallback rather than a guess.
Numeric strings are refused before parsing: Enum.TryParse accepts "9" for any
enum and Enum.IsDefined then confirms it, so an unguarded parse would bind a tag
to a type nobody named. A test caught this in the first version of the guard.
--- and the other half, which the fixture was hiding ---
Test Connect on a device with NO authored tags — every AB CIP device at the
moment an operator first presses the button — reported "CIP handshake failed"
against a healthy controller. That was written down as a fixture limitation
("ab_server doesn't implement @raw_cpu_type"). Both halves of that were wrong:
* @raw_cpu_type is not a libplctag special tag at all (only @raw, @tags and
@udt/ are), so it goes out as an ordinary symbolic name that no controller
defines, and the controller's "tag does not exist" IS the successful answer.
* ab_server answered an unresolvable tag path with CIP 0x03 where a Logix
controller answers 0x04 — ErrorBadParam vs ErrorNotFound, which the probe
classifies oppositely. Fixed in vendored patch 0002.
* With the fixture then telling the truth, the real defect surfaced:
AbCipDriverProbe.IsReachableException matched the native C status spellings
("STATUS_NOT_FOUND"), but libplctag.NET throws with the MANAGED enum name
("ErrorNotFound"). Those substrings can never match, so the reachable
classification could not fire on the exception path — which is the path every
no-authored-tags probe takes. This was failing on real hardware too, not just
against the sim. It now recovers the Status and defers to IsReachableStatus,
so there is one classification rather than two that can drift.
Rejected: switching the fallback to @tags. libplctag builds a tag-listing tag
with NO network I/O, so it reports OK against any TCP listener — measured green
against a Mosquitto broker.
Verification: full solution suite green. Falsification runs confirm exactly the
intended tests go red — 9 for the type carry, 3 (the reachable statuses only) for
the probe classification. Live on docker-dev: browse-committed TestString/
Motor_Run/TestSINT now store String/Bool/SInt, alongside last session's
Pump_1_Speed still holding the wrong DInt. TestSINT is the falsifier — its row
DataType is Int32, identical to TestDINT's, so no reverse map could have produced
"SInt".
Known consequence: pre-existing browse-committed tags keep their wrong dataType;
the fix applies to new commits only.
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
Joseph Doherty