PreStart's eager resolve of IMeshCommandDialer/IMeshPskProvider loses a
millisecond-wide race against the Akka builder's remaining starters on
essentially every admin boot (live gates: 1A 2/3 boots, 1B 4/4 boots) and
self-heals at the memoized Dialer/PskProvider properties' first use. Before
this change that produced two FALSE Errors per healthy boot, which made a
soak error-grep on a healthy rig impossible.
Resolve() now takes a `firstUse` flag and logs at Debug on BOTH call sites
(PreStart and the memoized properties' retry) — from inside Resolve a lost
race and a genuinely broken factory are indistinguishable, so severity is
decided by the CONSUMER of a still-null result after the retry, not by
Resolve itself:
- Dialer: DispatchOverGrpc's existing per-dispatch Error ("no command
dialer is wired") is unchanged and is now the ONLY loud signal for that
dependency.
- PskProvider: PublishKnownClusters gained a new Error carrying the old
PreStart guard's diagnostic content (gate never published, cached keys
never dropped, rotated key needs restart), guarded to fire ONCE per
actor incarnation via `_pskProviderAbsenceReported` — that handler runs
on the contact-refresh timer for the actor's whole life, so an
unguarded Error there would repeat every ContactRefreshSeconds forever.
Design choice for the "factory never supplied" case (Dps/ClusterClient test
constructions that pass no factories at all): kept silent at BOTH call
sites for both dependencies, same as before. For the dialer this was
already true (DispatchOverGrpc's check doesn't care why the dialer is
null). For the PSK provider the new once-guarded Error is gated on
`_pskProviderFactory is not null` specifically so the many existing
ClusterClient/Dps facts in CentralCommunicationActorTests that never wire a
PSK provider (using the bare Spawn helper) keep passing unmodified — a
never-supplied factory is a deliberate wiring/test choice, not a resolve
failure, and Resolve() already stayed silent about it for the same reason.
Two new tests pin the net behavior: a healthy boot (factories that throw
once then succeed) produces zero Warning/Error end-to-end through PreStart,
the first contact load and the first dispatch; a permanently-broken PSK
provider logs the absence exactly once across two contact loads, not once
per refresh.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Node→central ApplyAck over gRPC behind MeshTransport:Mode=Grpc: CentralCommandGrpcService
on the AdminUI listener, GrpcApplyAckSender with per-endpoint ack deadlines (Critical fix
e54711a2 proven live in the 4b hang drill), NodeCommunicationActor wiring, plus the
phases 2-4 plan update (targeted-test policy + Task 2.0 must-fix).
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Node→central ApplyAcks over gRPC, gated live on the docker-dev rig at
4a2a3d3a. All 7 checks PASS (the 6 planned plus the mandated 4b hang
drill).
The 4b drill is the one that earns its place: with central-1 PAUSED
rather than stopped, both site-a nodes burned a full AckTimeoutSeconds
on it as DeadlineExceeded and still reached central-2 with a live
deadline — the per-endpoint FRESH deadline from 667639e6, proven on the
failure shape a hoisted deadline could not survive. Observed failover
latency ~10 s, exactly one AckTimeoutSeconds.
It also produced the run's only duplicate ack, and that turned into
evidence rather than a defect: the frozen central had in fact received
both acks and replayed them on unpause, and the coordinator reported
2/6 acks landed, not 4/6 — the idempotent absorption the sender's
at-least-once remark asserts, observed instead of argued.
Coexistence is per pair in one deployment: site-a's rows are Applied
with matching ReportApply lines on central and zero receptionist
associations, site-b's are Applied with zero ReportApply lines and a
live receptionist association. Neither pair could have used the other's
transport.
Carry-forwards recorded: the I-4 cold-PSK-cache latency is structurally
unmeasurable on docker-dev (ClusterPsks is read ahead of the store, so
InvalidateAll clears an empty cache) and moves to a store-backed rig;
a restarted central's cached channel can stay in backoff ~2 min on the
node that watched it die, absorbed by the walk with no ack lost. The
1A finding-5 wording fix gains a second confirmation — 4 of 4 admin
boots lost the PreStart race, and every ack central-1 accepted proves
the provider it said would NEVER be published was.
Docs only; the rig flip and the four Serilog Debug overrides were
reverted and are not in this diff. Rig restored to six healthy,
all-ClusterClient, sealing 6/6.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
CRITICAL. The ack deadline was one absolute DateTime computed once for the whole
endpoint walk. Against the characteristic WAN failure — a central that HANGS
(SYN black-hole, frozen host) rather than refusing — endpoint 1 burned the whole
AckTimeoutSeconds and endpoint 2 was then dialled with a deadline already in the
past: instant DeadlineExceeded, no packet leaving the node, the surviving central
never really tried. Failover looked like it ran and did not. It also contradicted
MeshTransportOptions' own documented "applied per endpoint, not per ack". The
deadline is now computed fresh inside the loop; headers stay hoisted (read-only,
sequential walk).
Two test gaps that let that survive are closed: nothing pinned the ackRouter gate
(narrowing it back to ClusterClient-only compiled and passed every test while
dead-ending every Grpc ack to DPS), and the deadline fact's lower bound accepted
any positive timeout at all. Both new pins are mutation-verified.
Docs corrected where they overstated the case: duplicate acks CANNOT mis-seal a
deployment — ConfigPublishCoordinator.HandleAck keys _acks by NodeId behind an
expected-set guard, so a duplicate overwrites its own entry. The single-endpoint
rule is defence in depth and a bound on wasted work, not a correctness
dependency; and the walk is at-least-once on the deadline path, not at-most-once.
Task 1B.2 — the node→central half of the ClusterClient→gRPC migration.
GrpcApplyAckSender walks MeshTransport:CentralGrpcEndpoints in order and
STOPS AT THE FIRST SUCCESS. That single `return` is the whole single-count
guarantee: the transport it replaces was ClusterClient.Send, deliberately not
SendToAll, because both central nodes forward to the SAME
ConfigPublishCoordinator singleton proxy — an ack posted to both would have the
coordinator count one node twice and could seal a deployment on a doubled count.
The list is an availability list, not a fan-out list.
No retry beyond that one ordered pass, no queue: the receptionist buffered
nothing and retried nothing, and this is like-for-like, not an upgrade to
delivery guarantees. All endpoints failing is still a DROP — but a loud one,
naming the node, deployment, correlation id and the status each endpoint
answered with, which is what separates "central is down" from "central rejected
this node's key". The bearer never reaches a log; only endpoints and status
codes do. Per-endpoint GrpcChannel cache and the endpoint loop clone
GrpcDeploymentArtifactFetcher; the cluster id in x-otopcua-cluster comes from
this node's own cluster-<id> role via RoleParser, off the same bound
Cluster:Roles ClusterRoleInfo reads.
SendAsync never throws: the caller discards the task, so an escaping exception
would become an unobserved fault — a drop with no log line, the exact failure
mode this migration removes.
NodeCommunicationActor gains an OPTIONAL ackSender; every existing test compiles
and passes unedited. It is checked before the ClusterClient so the two can never
both fire. The registration arm creates no ClusterClient under Grpc.
NOTE — one wiring change beyond the actor: DriverHostActor's ackRouter is now
`useClusterClient || useGrpc ? nodeComm : null`. Left at ClusterClient-only, a
Grpc node's ack would have gone out on the DPS deployment-acks topic and never
reached the sender at all.
Two review items from Task 1B.1, both diagnostic:
1. CentralCommandGrpcService's "actor not ready" Warning omitted the peer and
the correlation id while the decode-failure Warning six lines above carried
both. These are the only two ways an ack fails to reach the coordinator, and
an operator has to correlate either against the node's own log. Decoding has
already succeeded on this path, so the DECODED correlation id is free.
2. A fused driver,admin,cluster-MAIN node under Mode=Grpc owes BOTH validator
surfaces at once (driver keys AND ConfigServe:GrpcListenPort). The two blocks
are independent `if`s; chained as `else if` a fused central node would be
graded on one surface only and boot with nowhere for any ack to land. Pinned
from both sides plus the fully-configured pass.
Re-ran the 1A gate on the docker-dev rig at e54883a8 after the defect-4 boot
fix. Image rebuilt (otopcua-host:dev -> 1d8452a29018); six host nodes plus
both one-shots recreated; run 1's preserved image was stale and deliberately
not reused. VERDICT: PASS.
Boot: 6/6 nodes healthy INCLUDING the central pair, 0 Hosting failed to start,
/health/ready 200 on every node, MAIN mesh 2/2.
Check 1 PASS: one DispatchDeployment, one correlation id, accepted by all six
nodes; central logs 6 accepted / 0 failed; zero ClusterClient lines on either
central node with Akka.Cluster.Tools.Client at Debug.
Check 2 PASS: deployment Sealed with all six NodeDeploymentState rows Applied
inside ~90 ms, while the four site nodes hold receptionist associations — the
only ack path they have in 1A. Mixed transport proven.
Check 4 PASS: Ack clicked on the real /alerts page against a live ACTIVATED
row; central fanned one AlarmCommand to all six over gRPC and the condition
acknowledged 0.54 s later carrying the operator identity.
Check 7 PASS: with site-b-2's GrpcPort NULLed, both central nodes warned once
and the next dispatch reached exactly five nodes; the deployment then honestly
ended TimedOut with site-b-2 alone unapplied. Row restored, 6/6 sealed after.
Check 8 PASS and no longer vacuous: 0 denials on all four interceptors, and
this time config-serve (28 KB verified artifact fetches from :4055) and
telemetry (streams open on all six, live driver health on /hosts) both carried
real traffic.
Check 3 closed by composition, with the reason now established rather than
assumed: <DriverStatusPanel has 0 usages repo-wide, no rendered page carries a
Restart/Reconnect control, and the only non-script-analysis POST is
/api/deployments. Run 1 drove the node-side leg for both commands; check 4
drives the identical AdminOperationsActor.Dispatch -> DispatchOverGrpc origin
path from the AdminUI.
Checks 5 + 6 stand from run 1 — nothing in the fix touches the interceptor.
New finding 5, recorded because it is load-bearing for the fix's design: the
PreStart resolve LOSES its race on real boots — 2 of 3 observed admin boots
logged Failed to resolve for both dependencies with the same
AuditWriterActorKey cause. The memoize-on-first-use hardening is therefore not
theoretical, and its heal was proven live by moving the coordinator singleton
onto the node that lost the race (stop central-2) and watching central-1 fan a
dispatch to all five live nodes with no dialer error. Residual is cosmetic and
carried to Phase 2: the Error text reads as permanent, and it will trip a soak
error-grep on a healthy node.
Defect 4 marked CLOSED; run 1's record kept in full underneath the combined
table. Rig restored: flip reverted, six nodes healthy on all-ClusterClient on
the committed image, three mesh ClusterClients back on each central node.
Phase 1A live gate defect 4. Neither central node booted on this branch,
in ANY transport mode:
Akka.Hosting.MissingActorRegistryEntryException: No actor registered
for key ...ControlPlane.AuditWriterActorKey
at AdminUI.Audit.ActorAuditWriter..ctor(ActorRegistry registry)
at ControlPlane.ServiceCollectionExtensions.<WithOtOpcUaControlPlaneSingletons>
WithOtOpcUaControlPlaneSingletons' first WithActors starter resolved
IMeshCommandDialer and IMeshPskProvider eagerly for the Props. Their
graph reaches back into the registry — dialer -> IMeshPskProvider ->
ISecretResolver -> AddZbSecrets -> IAuditWriter -> ActorAuditWriter,
whose ctor calls registry.Get<AuditWriterActorKey>() — and
WithSingleton<AuditWriterActorKey> is registered ~60 lines LATER in the
same builder. Boot died before the entry existed. Mode-independent: the
resolve happened before anything consulted MeshTransport:Mode, so Dps
(the default, and the rollback target) died exactly like Grpc.
Both dependencies now arrive as Func<> and are invoked in PreStart, by
which point the host is fully built — the same shape, for the same
class of reason, as the coordinator proxy Func one argument earlier.
PreStart re-resolves on restart rather than caching across one; both
are container singletons, so a restart gets the same instances. A
resolution throw is caught and logged rather than propagated: a throw
out of PreStart is an actor failure, the supervisor answers it with a
restart, and a broken DI graph is not transient — that is an unbounded
restart loop that would take the transport-independent ack path down
with it.
Regression pin: ControlPlaneStarterOrderTests boots a REAL admin host
through WithOtOpcUaClusterBootstrap + WithOtOpcUaControlPlaneSingletons
in production starter order, with the production graph from
IMeshCommandDialer down to a registry-bound IAuditWriter, and asserts
StartAsync itself — the failure occurs while the builder's starters
execute, so building a ServiceProvider and resolving by hand cannot
reproduce it. Verified RED against the eager resolve: 3/3 modes fail
with MissingActorRegistryEntryException. A second fact pins that the
deferral did not become 'never resolved' — the actor must resolve the
provider from the container and drive it through the contact refresh,
signalled by a TaskCompletionSource (PreStart runs on the actor thread
and the contact load is a piped DB read, so no synchronous assertion
after StartAsync is correct).
Why nothing caught this: the dialer's container-activation test resolves
from a collection with no secrets registration, so its graph stops one
hop in; the actor facts inject fakes and never touch DI; and the
existing whole-host boots register no mesh provider, so the eager
lookups returned null and walked nothing.
Two aggravators recorded by the gate, no code here: the failed process
did not exit (docker Up while /health/ready 503s — Phase 2 readiness
concern, tracked), and this is the third instance of the same
eager-resolve-during-AddAkka class after scadaproj#1 and mesh Phase 6's
IClusterRoleInfo boot crash.
ControlPlane.Tests 236 -> 240, Host.Tests 92, Cluster.Tests 184,
0 warnings in every touched project.
Ran the 1A live gate on the docker-dev rig at fbf7b708 (image rebuilt from
this worktree, all eight containers recreated). VERDICT: FAIL.
Both central nodes fail to boot on this branch with
MissingActorRegistryEntryException: AuditWriterActorKey — the first WithActors
starter eagerly resolves IMeshCommandDialer/IMeshPskProvider, whose graph
reaches ActorAuditWriter, whose ctor reads an ActorRegistry key registered ~60
lines later in the same builder. Reproduced on the same image with
MeshTransport__Mode=ClusterClient, so it is mode-independent and not caused by
the gate's rig-only flip. Recorded as defect 4; no product change made.
Checks 5 (auth negative/positive on real Kestrel h2c) and 6 (wrong-key
isolation, both directions, with a positive control) PASS. Checks 3 and 4 pass
their node-side leg only, driven with real RestartDriver/ReconnectDriver/
AlarmCommand envelopes that restarted and reconnected SITEA-modbus. Checks 1,
2 and 7 are blocked outright — no central, no dispatch, no contact refresh —
and check 8 is vacuous for telemetry + config-serve because central dials both
and never booted. All five gaps are named in the carry-forward register and
must be closed by a 1A gate re-run, not carried past the phase.
Issues filed by this task: lmxopcua#544 (ConfigPublishCoordinatorTimeoutTests
deflake) and lmxopcua#545 (mesh planes are h2c-only — accept the risk or add a
UseTls option); the tracking table's two placeholder rows now carry them.
Also recorded: DriverStatusPanel.razor is referenced by no page or component,
so the AdminUI Reconnect/Restart controls are unreachable on this branch and on
master — pre-existing, out of program scope, carried forward.
Rig restored: compose flip reverted, six nodes healthy on all-ClusterClient.
The rig runs the PRE-gate image (the 1A build is kept as otopcua-host:grpc-1a);
a re-run must rebuild.
The known-cluster gate published only under Mode=Grpc, but the
interceptor it feeds is registered on every admin node and 1B maps
CentralCommandService by LISTENER PORT — neither consults the mode. A
dark-switch rollback to ClusterClient or Dps would therefore have left
that endpoint serving with the gate permanently inactive, silently
restoring both the timing oracle and the pre-auth secret-store/audit
flood, as a side effect of a rollback nobody would connect to
authentication. The gate is central's cluster inventory, not transport
state, so it is now decoupled from the mode entirely:
- SetKnownClusters + InvalidateAll run on EVERY successful contact load
in EVERY mode, and the contact-refresh timer starts in all three.
Under Dps this is a ConfigDb query every ContactRefreshSeconds that
did not run before; central always has a ConfigDb, and one indexed
read a minute is the cheaper half of the trade.
- The GATE set is built from ALL ClusterNode rows. The eligibility
filter moved out of the query's WHERE and into the loop, so the FAN
set is byte-for-byte what it was (enabled && !maintenance) while the
gate widens. This also dissolves the maintenance coupling flagged in
1A.2: a cluster whose nodes are all disabled or in maintenance can
still ack, and refusing it would make maintenance mode an
authentication switch. Central still refuses clusters it has never
heard of.
- HandleContactsLoaded keeps a documented order — targets adopted, gate
published, cache dropped — now pinned by test.
Guards: PreStart mirrors the null-dialer error for a null PSK provider
(silent absence is the worst outcome there — nothing malfunctions, so
nothing is investigated, while the gate never publishes and keys never
rotate); a failed contact load now says out loud that the gate is stale
and, before any load has succeeded, inactive; MeshGrpcCommandDialer
clamps a non-positive DispatchTimeoutSeconds to 10 with a construction
Warning, because the validator for that key is driver-role-scoped and a
pure-admin central reads it unvalidated; the dialer's local PSK refusal
was shortened so its remediation survives the actor's 200-char
truncation of remote-authored status detail; the portless-node warn-once
set is rebuilt each refresh, so a node that regains then loses a port
warns again — the regression is the occurrence worth seeing.
Docs/pins: dialer disposal comment no longer claims one-way; the two
interceptor-test comments claiming Matches has a default implementation
corrected (it deliberately has none); new pins for per-refresh
invalidation, the SetKnownClusters-before-InvalidateAll order, and one
channel per endpoint via a new CachedChannelCount.
The_known_cluster_set_is_not_published_by_the_other_two_modes is
replaced — its premise is obsolete — by three facts: every mode
publishes, gate set is a strict superset of the fan set, and the handler
ordering.
ControlPlane.Tests 230 -> 236, Host.Tests 92, Cluster.Tests 184,
0 warnings.
MeshGrpcCommandDialer places one NodeCommandService.Execute call on one
node: a GrpcChannel cached per endpoint (the GrpcDeploymentArtifactFetcher
shape), the TARGET node's cluster's bearer plus x-otopcua-cluster naming
that cluster, and a per-call deadline linked to the caller's token. A
cluster with no resolvable key is refused locally with Unauthenticated —
there is no arm on which a command leaves central unauthenticated, even
one the far side would have rejected.
CentralCommunicationActor gains the third mode branch. The contact load
now also projects GrpcPort and, from the SAME rows in the SAME pass,
builds the dispatch set — which is what makes 'the gRPC fan reaches
exactly the SendToAll recipient set' structural rather than two queries
that have to keep agreeing. A row with a null GrpcPort is skipped
warn-once (the TelemetryNodeSource precedent). Per-node results are
piped back for LOGGING ONLY: named node, gRPC status, dropped. No retry,
no queue.
Every dial is scheduled on the thread pool rather than called from the
receive. Found by review and pinned red-first: an async method runs
synchronously to its first await, and the dialer's pre-await work is
TryGetPsk, which blocks on the secret store — bounded but seconds, and
repeated after every refresh since the refresh drops the cache. Dialing
inline parked the actor's mailbox, so deployment acks queued behind a
slow node; the regression test tells an ApplyAck while a dial is held
and fails without the hop.
The contact refresh also wires the Phase 1A register: it publishes the
loaded cluster ids as the PSK provider's known-cluster set and drops the
key cache, so a rotated key takes effect within one ContactRefreshSeconds.
Neither happens under Dps or ClusterClient.
Both existing mode branches are behaviourally untouched and every
existing fact passes unmodified. The one edit to shared text: the
Status.Failure warning drops the word 'ClusterClient', since that handler
now serves all three transports.
ControlPlane.Tests 205 -> 230, Host.Tests 92, Cluster.Tests 184, 0 warnings.
Phase 1A security register items 1 and 2, landed ahead of the command
fan-out because they are independently testable and the fan-out is the
first caller of TryGetPsk.
Item 1 — known-cluster gate. MeshPskProvider gains SetKnownClusters, an
atomic whole-set swap consulted by TryGetPsk BEFORE the config map and
before the secret store. A cluster id outside the set is refused in O(1)
with no store read and no audit write, which closes two things at once:
the pre-auth store/audit flood an unauthenticated caller could drive with
a forged x-otopcua-cluster header, and the timing oracle that defeated
CentralCommandAuthInterceptor's uniform denial text (unknown clusters are
never cached, so they paid a store read on every probe while known ones
answered from memory). Empty or never-published leaves the gate inactive
— central publishes the set from its DB contact refresh, and 'the DB told
me nothing' must not become a fleet-wide authentication outage during
admin boot.
Item 2 — bool Matches(clusterId, presented) on IMeshPskProvider, with
FixedTimeEquals inside the provider. CentralCommandAuthInterceptor now
calls it and never holds key material; its two provider-side denial logs
collapse into one (the provider answers both as false) and the operator
hint is restated there.
No production caller publishes a cluster set yet, so behaviour is
unchanged until the fan-out commit wires the contact refresh.
ControlPlane.Tests 190 -> 205, Host.Tests 91 -> 92.
Phase 0 task 0.6 of the ClusterClient->gRPC migration: wire the dark
MeshTransport Grpc-mode keys into the docker-dev rig (GrpcPsk +
CentralGrpcEndpoints on all six nodes, ClusterPsks on the central pair
only), document the new keys in docs/Configuration.md as shipped
dark/cutover pending, and record the Phase 0 live-gate evidence
(rig boot with no validator fires, two deliberate-misconfig refusals,
full suite green).
Files gitea.dohertylan.com/dohertj2/lmxopcua#542 (FleetStatusBroadcaster
mesh-blind post-Phase-6) and #543 (three pre-existing interceptors accept
a whitespace-only key as set) as follow-ups out of Phase 0's scope.
Closes the half of #540 that made 20 dead Makino tags invisible rather than
merely wrong. A skip is not an error and happens before any read, so a driver
ignoring part or all of its configuration reported Healthy / 0 errors / no
last error / a live-advancing last read, and no surface anywhere said
otherwise.
New optional capability ISkippedTagReporter over a shared SkippedTagLedger,
riding DriverHealthChanged to a Tags column on /hosts via the Phase 5
telemetry stream (no table -- a driver-only node has no ConfigDb, the #521
lesson). Additive to driver state, never Degraded, because latching
non-Healthy would satisfy both halves of #527's comms-loss conjunction and
reconnect-loop a healthy device whose tags were all skipped.
The issue named eight drivers; it is twelve. Calculation, MTConnect, Galaxy
and OpcUaClient were unlisted, and OpcUaClient's skip path was the quietest
of the twelve -- a bare 'continue' in NamespaceMap with not even a log line.
A reflective fleet guard now pins the invariant so a thirteenth cannot
re-introduce the silence. Sql reports the union of its two drop points, the
blob that does not map and the 8.1 catalog gate, because schema drift
produces the second and reporting only the first would have it claim
'0 skipped' while polling half its tags.
Live-gated on docker-dev over two rounds (docs/plans/
2026-08-05-skipped-tag-visibility-live-gate.md). Round one found a defect the
unit tests could not: two faulted drivers rendered '0 tags' against two
authored tags apiece, because a never-committed ledger returned a zero
report -- a positive claim made on behalf of a driver that never looked.
GetSkippedTags() now returns null until the first committed build. Round two
closed the truncation note (25 skipped -> sample capped at 20 with '+5 more
not shown'; trimming to exactly 20 correctly shows no note) and the central
proto path, using a SITE-A driver whose own Akka mesh shares no gossip with
central, so TelemetryProtoMapNode -> gRPC -> TelemetryProtoMapCentral is its
only route to the page.
Unplanned finding: three of the four drivers showing skips on the rig were
not planted. They had been silently ignoring their tags for an unknown
length of time, each reporting Healthy with zero errors.
25 unmappable tags on SITE-A's sa-modbus2 closed both gaps in one setup: chip
25/26 SKIPPED, sample capped at exactly MaxSample=20, tooltip '(+5 more not
shown)'. Trimming to 20 skipped gave 20/21 with NO truncation note, pinning
the off-by-one boundary.
SITE-A is the discriminator for the proto path. Phase 6 partitioned it into
its own Akka mesh -- central's MEMBERS panel lists only central-1/central-2,
and DistributedPubSub does not federate across meshes. The single site-a
reference in central's log is the ClusterClient receptionist, a
point-to-point command path, not gossip. So the report's only route to the
page is TelemetryProtoMapNode -> gRPC -> TelemetryProtoMapCentral, and
central's view tracking a change made solely on the site node is conclusive.
Also corrects round one's claim that the fused central pair meant telemetry
never crossed gRPC -- wrong about this rig. All six ClusterNode rows carry
GrpcPort 4056 and Telemetry__Mode=Grpc; the two 'Telemetry stream to node'
lines that suggested a two-node dial set were startup failure warnings.
Rig restored: test tags deleted, SITE-A redeployed, sa-modbus2 back to 1 tag.
Found on the docker-dev live gate, not in tests. Two drivers that faulted
during initialise (rtu-gate, opcua1) each rendered "0 tags" on /hosts against
two authored tags apiece, because a ledger that had never committed returned
SkippedTagReport.Empty.
A zero report is a positive claim -- "this driver skipped nothing" -- and
making it on behalf of a driver that never got as far as looking is exactly
the quiet inaccuracy #541 exists to remove. GetSkippedTags() and
SkippedTagLedger.Snapshot() now return null until the first committed build,
so the column renders the same em-dash it already used for a driver that
cannot report at all. Both are 'no answer'; neither is 'nothing skipped'.
Knock-on: a FOCAS driver whose only init threw now reports null rather than a
zero report, which is strictly more honest than the assertion the test
originally made.
A driver builds its RawPath -> definition table at init and tolerates a
per-tag miss: an unmappable TagConfig, an unparseable address, a
capability-matrix rejection. That tolerance is right -- one bad tag must not
take down the other few thousand. What was missing is any statement of the
fact afterwards. A skip is not an error and happens before any read, so the
driver stays Healthy, errors/5 min stays 0, Last error stays empty, and Last
read advances live off the tags that did map. On the wonder Makino in #540
that reading held while focas-z-34184 ignored 20 of its 22 tags.
New optional capability ISkippedTagReporter, backed by a shared
SkippedTagLedger so the reset/commit/cap rules are one implementation rather
than one per driver. The report -- authored count, exact skipped count, and a
capped sample of RawPath + reason -- rides DriverHealthChanged to a new Tags
column on /hosts, over the Phase 5 telemetry stream. No table: a driver-only
node has no ConfigDb to write to, the lesson from #521.
The issue named eight drivers. It is twelve -- every driver in
DriverTypeNames.All has an authored-tag skip path. Calculation (no scriptId),
MTConnect (no readable DataItem id), Galaxy (no attributeRef) and OpcUaClient
(no nodeId) were unlisted, and OpcUaClient's was the quietest of the twelve:
NamespaceMap drops such an entry with a bare `continue` and not even a log
line. SkippedTagReporterFleetTests pins the invariant reflectively over every
IDriver in src/Drivers/, so a thirteenth cannot re-introduce the silence.
Sql drops a tag at TWO points and its report is the union of both: a blob
that does not map, AND an identifier the 8.1 catalog gate rejects against the
live catalog. The second is what schema drift produces -- a renamed column, a
dropped view -- months after a deployment sealed green, and reporting only
the first would have that driver claim "0 skipped" while polling half its
tags, which is worse than reporting nothing because the column would then be
actively reassuring.
Deliberately additive to driver state, never a change of it. Latching
non-Healthy for a config problem would satisfy both halves of the #527
comms-loss conjunction -- 90 s without a successful read AND non-Healthy --
and a driver whose tags were ALL skipped has no successful reads by
construction, so it would reconnect-loop forever against a healthy device.
Three details that are load-bearing rather than incidental:
- The skip digest is part of the publish fingerprint. A skipped tag leaves
every other fingerprint component identical to a clean driver's, so without
it PublishHealthSnapshot's dedup swallows exactly the publish carrying the
signal -- the trap that already bit the rediscovery and host-status
channels. The digest covers the sample, not just the counts: a redeploy
that fixes one tag and breaks another leaves both counts unchanged.
- The count is exact; only the sample is capped, and the tooltip says so when
it is showing fewer than it counted. A truncated list that looked complete
would have an operator fix twenty tags and believe the job done.
- The ledger stages and commits, so an init that throws part-way leaves the
previous report standing. A partial snapshot would read as "we now skip
fewer tags", the opposite of the truth.
The proto field needs no companion presence flag, unlike host_statuses: a
singular message field carries real presence in proto3, and a has_ flag would
collide with protoc's generated HasSkippedTags.
Verified red-first three ways: dropping SkippedTagDigest from the fingerprint
fails the two dedup tests, removing the seam from one driver fails the fleet
guard, and dropping the catalog-gate leg fails the Sql union test. Full
solution builds with zero new warnings against 4e22f024.
Joseph Doherty