Re-ran the 1A gate on the docker-dev rig at e54883a8 after the defect-4 boot
fix. Image rebuilt (otopcua-host:dev -> 1d8452a29018); six host nodes plus
both one-shots recreated; run 1's preserved image was stale and deliberately
not reused. VERDICT: PASS.
Boot: 6/6 nodes healthy INCLUDING the central pair, 0 Hosting failed to start,
/health/ready 200 on every node, MAIN mesh 2/2.
Check 1 PASS: one DispatchDeployment, one correlation id, accepted by all six
nodes; central logs 6 accepted / 0 failed; zero ClusterClient lines on either
central node with Akka.Cluster.Tools.Client at Debug.
Check 2 PASS: deployment Sealed with all six NodeDeploymentState rows Applied
inside ~90 ms, while the four site nodes hold receptionist associations — the
only ack path they have in 1A. Mixed transport proven.
Check 4 PASS: Ack clicked on the real /alerts page against a live ACTIVATED
row; central fanned one AlarmCommand to all six over gRPC and the condition
acknowledged 0.54 s later carrying the operator identity.
Check 7 PASS: with site-b-2's GrpcPort NULLed, both central nodes warned once
and the next dispatch reached exactly five nodes; the deployment then honestly
ended TimedOut with site-b-2 alone unapplied. Row restored, 6/6 sealed after.
Check 8 PASS and no longer vacuous: 0 denials on all four interceptors, and
this time config-serve (28 KB verified artifact fetches from :4055) and
telemetry (streams open on all six, live driver health on /hosts) both carried
real traffic.
Check 3 closed by composition, with the reason now established rather than
assumed: <DriverStatusPanel has 0 usages repo-wide, no rendered page carries a
Restart/Reconnect control, and the only non-script-analysis POST is
/api/deployments. Run 1 drove the node-side leg for both commands; check 4
drives the identical AdminOperationsActor.Dispatch -> DispatchOverGrpc origin
path from the AdminUI.
Checks 5 + 6 stand from run 1 — nothing in the fix touches the interceptor.
New finding 5, recorded because it is load-bearing for the fix's design: the
PreStart resolve LOSES its race on real boots — 2 of 3 observed admin boots
logged Failed to resolve for both dependencies with the same
AuditWriterActorKey cause. The memoize-on-first-use hardening is therefore not
theoretical, and its heal was proven live by moving the coordinator singleton
onto the node that lost the race (stop central-2) and watching central-1 fan a
dispatch to all five live nodes with no dialer error. Residual is cosmetic and
carried to Phase 2: the Error text reads as permanent, and it will trip a soak
error-grep on a healthy node.
Defect 4 marked CLOSED; run 1's record kept in full underneath the combined
table. Rig restored: flip reverted, six nodes healthy on all-ClusterClient on
the committed image, three mesh ClusterClients back on each central node.
Ran the 1A live gate on the docker-dev rig at fbf7b708 (image rebuilt from
this worktree, all eight containers recreated). VERDICT: FAIL.
Both central nodes fail to boot on this branch with
MissingActorRegistryEntryException: AuditWriterActorKey — the first WithActors
starter eagerly resolves IMeshCommandDialer/IMeshPskProvider, whose graph
reaches ActorAuditWriter, whose ctor reads an ActorRegistry key registered ~60
lines later in the same builder. Reproduced on the same image with
MeshTransport__Mode=ClusterClient, so it is mode-independent and not caused by
the gate's rig-only flip. Recorded as defect 4; no product change made.
Checks 5 (auth negative/positive on real Kestrel h2c) and 6 (wrong-key
isolation, both directions, with a positive control) PASS. Checks 3 and 4 pass
their node-side leg only, driven with real RestartDriver/ReconnectDriver/
AlarmCommand envelopes that restarted and reconnected SITEA-modbus. Checks 1,
2 and 7 are blocked outright — no central, no dispatch, no contact refresh —
and check 8 is vacuous for telemetry + config-serve because central dials both
and never booted. All five gaps are named in the carry-forward register and
must be closed by a 1A gate re-run, not carried past the phase.
Issues filed by this task: lmxopcua#544 (ConfigPublishCoordinatorTimeoutTests
deflake) and lmxopcua#545 (mesh planes are h2c-only — accept the risk or add a
UseTls option); the tracking table's two placeholder rows now carry them.
Also recorded: DriverStatusPanel.razor is referenced by no page or component,
so the AdminUI Reconnect/Restart controls are unreachable on this branch and on
master — pre-existing, out of program scope, carried forward.
Rig restored: compose flip reverted, six nodes healthy on all-ClusterClient.
The rig runs the PRE-gate image (the 1A build is kept as otopcua-host:grpc-1a);
a re-run must rebuild.
Phase 0 task 0.6 of the ClusterClient->gRPC migration: wire the dark
MeshTransport Grpc-mode keys into the docker-dev rig (GrpcPsk +
CentralGrpcEndpoints on all six nodes, ClusterPsks on the central pair
only), document the new keys in docs/Configuration.md as shipped
dark/cutover pending, and record the Phase 0 live-gate evidence
(rig boot with no validator fires, two deliberate-misconfig refusals,
full suite green).
Files gitea.dohertylan.com/dohertj2/lmxopcua#542 (FleetStatusBroadcaster
mesh-blind post-Phase-6) and #543 (three pre-existing interceptors accept
a whitespace-only key as set) as follow-ups out of Phase 0's scope.
25 unmappable tags on SITE-A's sa-modbus2 closed both gaps in one setup: chip
25/26 SKIPPED, sample capped at exactly MaxSample=20, tooltip '(+5 more not
shown)'. Trimming to 20 skipped gave 20/21 with NO truncation note, pinning
the off-by-one boundary.
SITE-A is the discriminator for the proto path. Phase 6 partitioned it into
its own Akka mesh -- central's MEMBERS panel lists only central-1/central-2,
and DistributedPubSub does not federate across meshes. The single site-a
reference in central's log is the ClusterClient receptionist, a
point-to-point command path, not gossip. So the report's only route to the
page is TelemetryProtoMapNode -> gRPC -> TelemetryProtoMapCentral, and
central's view tracking a change made solely on the site node is conclusive.
Also corrects round one's claim that the fused central pair meant telemetry
never crossed gRPC -- wrong about this rig. All six ClusterNode rows carry
GrpcPort 4056 and Telemetry__Mode=Grpc; the two 'Telemetry stream to node'
lines that suggested a two-node dial set were startup failure warnings.
Rig restored: test tags deleted, SITE-A redeployed, sa-modbus2 back to 1 tag.
Found on the docker-dev live gate, not in tests. Two drivers that faulted
during initialise (rtu-gate, opcua1) each rendered "0 tags" on /hosts against
two authored tags apiece, because a ledger that had never committed returned
SkippedTagReport.Empty.
A zero report is a positive claim -- "this driver skipped nothing" -- and
making it on behalf of a driver that never got as far as looking is exactly
the quiet inaccuracy #541 exists to remove. GetSkippedTags() and
SkippedTagLedger.Snapshot() now return null until the first committed build,
so the column renders the same em-dash it already used for a driver that
cannot report at all. Both are 'no answer'; neither is 'nothing skipped'.
Knock-on: a FOCAS driver whose only init threw now reports null rather than a
zero report, which is strictly more honest than the assertion the test
originally made.
A driver builds its RawPath -> definition table at init and tolerates a
per-tag miss: an unmappable TagConfig, an unparseable address, a
capability-matrix rejection. That tolerance is right -- one bad tag must not
take down the other few thousand. What was missing is any statement of the
fact afterwards. A skip is not an error and happens before any read, so the
driver stays Healthy, errors/5 min stays 0, Last error stays empty, and Last
read advances live off the tags that did map. On the wonder Makino in #540
that reading held while focas-z-34184 ignored 20 of its 22 tags.
New optional capability ISkippedTagReporter, backed by a shared
SkippedTagLedger so the reset/commit/cap rules are one implementation rather
than one per driver. The report -- authored count, exact skipped count, and a
capped sample of RawPath + reason -- rides DriverHealthChanged to a new Tags
column on /hosts, over the Phase 5 telemetry stream. No table: a driver-only
node has no ConfigDb to write to, the lesson from #521.
The issue named eight drivers. It is twelve -- every driver in
DriverTypeNames.All has an authored-tag skip path. Calculation (no scriptId),
MTConnect (no readable DataItem id), Galaxy (no attributeRef) and OpcUaClient
(no nodeId) were unlisted, and OpcUaClient's was the quietest of the twelve:
NamespaceMap drops such an entry with a bare `continue` and not even a log
line. SkippedTagReporterFleetTests pins the invariant reflectively over every
IDriver in src/Drivers/, so a thirteenth cannot re-introduce the silence.
Sql drops a tag at TWO points and its report is the union of both: a blob
that does not map, AND an identifier the 8.1 catalog gate rejects against the
live catalog. The second is what schema drift produces -- a renamed column, a
dropped view -- months after a deployment sealed green, and reporting only
the first would have that driver claim "0 skipped" while polling half its
tags, which is worse than reporting nothing because the column would then be
actively reassuring.
Deliberately additive to driver state, never a change of it. Latching
non-Healthy for a config problem would satisfy both halves of the #527
comms-loss conjunction -- 90 s without a successful read AND non-Healthy --
and a driver whose tags were ALL skipped has no successful reads by
construction, so it would reconnect-loop forever against a healthy device.
Three details that are load-bearing rather than incidental:
- The skip digest is part of the publish fingerprint. A skipped tag leaves
every other fingerprint component identical to a clean driver's, so without
it PublishHealthSnapshot's dedup swallows exactly the publish carrying the
signal -- the trap that already bit the rediscovery and host-status
channels. The digest covers the sample, not just the counts: a redeploy
that fixes one tag and breaks another leaves both counts unchanged.
- The count is exact; only the sample is capped, and the tooltip says so when
it is showing fewer than it counted. A truncated list that looked complete
would have an operator fix twenty tags and believe the job done.
- The ledger stages and commits, so an init that throws part-way leaves the
previous report standing. A partial snapshot would read as "we now skip
fewer tags", the opposite of the truth.
The proto field needs no companion presence flag, unlike host_statuses: a
singular message field carries real presence in proto3, and a has_ flag would
collide with protoc's generated HasSkippedTags.
Verified red-first three ways: dropping SkippedTagDigest from the fingerprint
fails the two dedup tests, removing the seam from one driver fails the fleet
guard, and dropping the catalog-gate leg fails the Sql union test. Full
solution builds with zero new warnings against 4e22f024.
Authoring FOCAS tags through /raw -> Browse device... produced tags the driver
could never read. Every committed tag read BadNodeIdUnknown forever, and the
failure was silent end to end: the AdminUI accepted it, the deploy gate sealed
green, and the driver reported Healthy with 0 errors while ignoring the tags.
Net effect: the whole fixed-tree surface could not be published through any v3
authoring path. Found cutting wonder-app-vd03 over to v3 against a real Makino
Pro 5 / Fanuc 30i, where 0 of 20 committed tags read while two hand-authored
MACRO: tags on the same device read Good.
A captured browse leaf's id IS its DriverAttributeInfo.FullName, and the fixed
tree emitted "{deviceHost}/{path}" there -- an internal cache key, not an
address. Both authoring paths wrote it verbatim into TagConfig.address, and
FocasAddress.TryParse accepts no such form, so the driver skipped the tags at
init.
Fixed at the emitter rather than by stripping the host at commit, which fixes
the browse-commit flow and the manual picker's tree-select together and leaves
one canonical address form:
- FocasAddress gains a fourth kind, FIXED:<path>, and moves to Contracts so the
AdminUI and ControlPlane can parse without referencing the runtime driver
(#526). A FIXED: read is served from the fixed-tree snapshot the poll loop
already maintains -- no area read, no connection. Writes refuse: nothing in
that tree is a writable location on the CNC.
- The address carries no device. A tag's device comes from its Device row, and
a second routing channel for one fact is what created this defect.
- An authored FIXED: tag arms the fixed-tree poll loop by itself. Enabled
defaults false and the browser force-enables it only for a browse, so the
operator would otherwise browse, commit, deploy, and land on a driver whose
loop never starts -- every tag Bad with a correct address.
- Fixed-tree values are now dated from the last answered poll and go
BadCommunicationError once it is stale (8 cadences, floored at 30s). The poll
loop swallows a failed tick by design, so a dead CNC would otherwise serve its
last snapshot as Good indefinitely -- harmless while nothing could bind to the
tree, not harmless now.
Guardrails, so this class cannot recur silently:
- FocasTagConfigModel.Validate() runs the real parser instead of a non-empty
check, and FocasTagDefinitionFactory.Inspect() warns at the deploy gate, which
covers CSV/API config that never touches the modal.
- The manual picker offered axis/spindle/program/status "parameter groups" --
none of which is a FOCAS addressing space -- and its own default value,
axis:5, was unusable. It now composes PMC / PARAM: / MACRO: through
FocasAddress, so what it emits is parseable by construction.
Tests assert addresses ROUND-TRIP through the parser rather than matching an
expected string: the previous FocasAddressBuilderTests pinned "axis:5" exactly,
which is how a green suite coexisted with a picker nobody could use. Verified
red-first against the old emitter. FOCAS 308, AdminUI 1095, ControlPlane 169,
Core 581 -- all pass, 0 warnings.
The remaining half of #540 -- a driver reporting Healthy/0 errors while
skipping tags at init -- is fleet-wide (eight drivers do this) and is filed
separately rather than patched per-driver.
Also adds the two fixture connection strings to both docker-dev centrals, so
the rig can author + browse the Postgres and Oracle fixtures on 10.100.0.35.
All three legs pass against the patched AbCip fixture: the edit modal and the
manual-entry grid both browse the live nine-tag symbol table, and an unreachable
device degrades to the graceful error chip while the modal still saves.
The gate also found a defect unrelated to this change and filed it as #539: the
AbCip discovery path (LibplctagTagEnumerator, LibplctagTemplateReader) passes
libplctag the bare host, so an authored non-default EtherNet/IP port is dropped
and the browse reaches whatever answers on 44818 — #534's defect surviving in the
discovery half. Discovered because the first negative-leg endpoint differed from
the fixture only by port, which never reached the wire.
Records the outcome in the design doc §6 (including the stale-open race the
review found and fixed in both modals), closes the follow-up in the tracking
doc and the universal-browser plan, and registers RawBrowseModal's shared
race shape in deferment.md §0.1 as the deliberate leftover.
The tracking doc and program-design §9 still showed Wave 0's live gate open —
it closed 2026-08-01 (c2c3c522). Both now record Waves 0-2 complete.
New design doc for the one residual item: the raw modals' typed tag editors
pass a '{}' stub as GetDriverConfigJson, so the AbCip/TwinCAT/FOCAS pickers'
Browse button captures against no endpoint and can never succeed. Decision:
wire the merged Driver+Device config through (the RawBrowseModal/Test-connect
seam already exists), not hide the button.
Ran on the docker-dev central pair against the local HistorianGateway container talking to the real
AVEVA Historian on wonder-sql-vd03 (VPN up), with the host image rebuilt at e7fbe977.
Two things about the setup matter more than the results:
The value source had to be DRIVEN. Both the project Modbus fixture and oee-modbus-sim hold static
registers, and a static source produces no dependency-mux fan-out at all — the recorder captures
once at deploy and then goes quiet, which reads exactly like a broken recorder. Hand-written
Modbus FC6 writes put the value stream under the gate's control, which is what makes the rate
measurements below mean anything. The project fixture was also brought back up on the docker host
and MAIN-modbus-dev repointed to it, undoing the temporary :1502 repoint the read-back leg made.
The poison tag is the real shape, not a synthetic fault: GATE491-B's historianTagname was CHANGED
on an EXISTING tag, which Issue 5 guarantees is never provisioned, so its writes fail permanently
against a historian that has no such tag. That is how #538 was reached in the first place.
#537 — both nodes up: the Secondary logged the suppression line once and its outbox file froze
while the Primary's kept growing; 6 value changes x 2 tags gave WriteLiveValues{ok} +12 exactly,
where the pre-fix measurement was twice that. Failover: stopping the Primary made the survivor log
"now the recording Primary ... resuming value capture" and pick up every subsequent change. Return:
the restarted node was demoted, logged suppression, and the rate stayed at one node.
#538 — the quarantine fired with the tag named; 5 value changes over 30s gave ok +5 and error +5,
exactly one write per tag per change against a 2s drain interval.
The decisive leg is the IDLE one: 40s with no value changes left both counters unchanged and the
outbox byte size flat. The driven-rate legs only show a rate that happens to look right; a pre-fix
build re-writes its queued healthy entries on every drain pass whether or not anything new arrives,
so its counters climb during a quiet window with no input at all. Flat counters over idle is the
observation the bug cannot produce.
Incidental finding, recorded against Issue 5 rather than treated as a fix: a node RESTART cures an
unprovisioned tag. central-2's boot logged provisioning (dispatched=2, requested=2, ensured=2,
failed=0) and the poison tag started succeeding, because a full address-space rebuild treats every
historized tag as added and re-runs EnsureTags — where the incremental deploy that introduced the
renamed tag did not. So an operator who restarts a node while diagnosing "my tag isn't historizing"
makes the symptom disappear without learning the cause.
Docs only; no production code changed.
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
Both defects were found by #491's live gate against a real HistorianGateway and filed rather than
fixed at the time, because each one's fix is a design decision rather than a patch.
#537 — both nodes of a redundant pair captured every value, so the historian received duplicate
live samples for every historized tag (measured: the gateway's write counter advanced twice per
value and halved when one central was stopped).
The recorder now reads a capture gate per value, fed from the new
PrimaryGatePolicy.ShouldRecordHistorizedValues through IHistorizationRoleView — a third verdict on
the same RedundancyRoleView object, registered unconditionally on driver nodes rather than
alongside the alarm section's IRedundancyRoleView (which only exists when that section is enabled;
piggybacking is the #511 registration accident).
Two things are deliberate and easy to get backwards:
* It gates the CAPTURE, not the drain — the opposite of the alarm sink. That queue is in LocalDb
and replicates, so the Secondary already holds the Primary's rows. The historization outbox is
a node-local FasterLog replicating to nobody, so a drain-only gate would leave the standby
appending forever to a queue no one empties: unbounded disk, then a flood of stale duplicates
the moment it is promoted. Entries a node already captured are still drained by that node.
* The peer it stands down for must RECORD THE SAME REFS, not merely hold the same rows — a
Primary in this node's own per-cluster mesh. A node carrying no cluster-scoped role cannot
establish that and never stands down: under a fleet-wide election the elected Primary is
usually running a different cluster's configuration, and deferring to it would end this
cluster's historization silently, with the drain reporting success on an empty outbox forever.
That is the scope mismatch that once suspended the alarm drain on every node of the rig.
An unknown role captures, matching the alarm drain — a duplicate sample is recoverable, a hole in
the trend is not.
#538 — the drain writes every per-tag group and then acks all-or-nothing (deliberate: RemoveAsync
truncates THROUGH an id, so partial acking could drop older unwritten entries). One permanently
failing tag therefore pinned the FIFO head while every healthy tag in the batch was re-submitted
to the historian on every pass, indefinitely — duplicate samples and an outbox that never
truncated, observed live at 8.3 MB from a single unprovisionable tag.
The drain now tracks consecutive failures per historian tag and abandons that tag's entries once
it exceeds MaxWriteAttempts (new, default 5), so the head advances. The counter only advances on a
pass where some OTHER tag wrote successfully: that is what separates "this tag is poison" from
"the backend is down". A count-only quarantine looks identical on the poison-tag case and silently
discards the whole backlog during an ordinary gateway restart, turning a duplicate-samples bug
into a lost-history one. Abandonment is also deferred until every failing tag is over budget, so
nothing is discarded on a pass that could not have advanced the head anyway. Each abandonment logs
a Warning naming the tag; the total rides RecorderStatus.AbandonedEntries.
Capacity also gains a non-zero default (500,000 entries) as the disk backstop for the one case the
quarantine deliberately cannot resolve — a historian refusing every write, where retaining is
correct. Set 0 to restore unbounded retention.
Every new test was falsified by reverting the production change it guards, and each one reddens
alone:
* drop the backend-reachable evidence -> only A_backend_that_fails_every_tag_abandons_nothing
* never abandon -> only A_permanently_failing_tag_is_abandoned_so_healthy_tags_stop_being_rewritten
* latch the capture gate at construction -> only The_capture_gate_is_re_read_per_value...
* treat any Primary as a scope peer -> only the two scope-mismatch cases
Full suite green. (Four OpcUaClient.Browser failures during the run were the opc-plc fixture being
down on the shared Docker host; brought it back up and they pass.)
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
The 2026-08-01 run closed every leg except read-back, which no reachable historian could
serve that day (the LAN instance accepts writes but serves no reads under a licence tag-count
violation; the licensed one needed the VPN). With the VPN up the run was repeated against
wonder-sql-vd03 and the remaining legs pass:
- EnsureTags provisioning: ensured=2, failed=0 — the path the LAN box could not exercise at
all (it is RemoteGrpc-only and that CAP was not listening).
- Read-back: two Modbus registers carrying DISTINCT NON-ZERO constants were historized, and
each read back as its own value (HR20 -> 8224, HR3 -> 120, quality 192). A first attempt
read back 0, which proves nothing — the same trap #536 recorded, so the proof was rebuilt
on values a crossed or fabricated answer could not fake.
Found during the run and FILED rather than fixed:
- #538 — one permanently-failing tag re-writes every healthy tag on every drain pass, forever.
DrainOnceAsync writes EVERY per-tag group and then acks all-or-nothing, so a tag whose
historian tag does not exist stops the FIFO head advancing while the healthy tags' values
are re-submitted every drain interval; the outbox never truncates (Capacity defaults to 0 =
unbounded, observed at 8.3 MB). The all-or-nothing ack is deliberate — RemoveAsync truncates
THROUGH an id, so partial acking could drop older unwritten entries — so the fix changes the
outbox contract and does not belong in this commit.
Also re-confirmed live: EnsureTags provisioning fires only for newly-ADDED historized tags, so
flipping an existing tag's historianTagname provisions nothing (Issue 5 of
2026-06-27-historian-gateway-integration-issues.md).
No production code changes — the recorder fix shipped in ad744734.
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
Ran #491's live gate against a real ZB.MOM.WW.HistorianGateway talking to a real AVEVA
Historian, and it found a production-fatal defect on the first value it ever saw.
ContinuousHistorizationRecorder.OnValueChangedAsync touched `Self` after awaiting
IHistorizationOutbox.AppendAsync(...).ConfigureAwait(false). ConfigureAwait(false) discards
Akka's ActorTaskScheduler synchronization context, so `Self` — which reads ActorBase.Context —
threw NotSupportedException. The throw escaped into supervision, which RESTARTED the actor;
because production seeds an EMPTY ref set (the deployed set arrives later via
UpdateHistorizedRefs), the restart silently discarded every historized ref. Net effect: one
value captured, then nothing, with no further error to show for it. Fixed by capturing
`IActorRef self = Self` before the await — the idiom the GetStatus handler already applied to
Sender.
Thirteen unit tests missed it because InMemoryOutbox.AppendAsync returned
ValueTask.CompletedTask: a completed ValueTask never yields, so the continuation ran inline
with the context still installed. The production FasterLog outbox fsyncs per entry and does
yield. The first regression test was itself vacuous — `await Task.Yield()` posts back to the
CAPTURED context, so the append still completed on the actor thread and reverting the fix left
the test green. The fake must complete on a POOL thread (Task.Delay(1).ConfigureAwait(false)),
as a real fsync does; with that, reverting the fix reddens exactly one test.
The rig could not run this gate at all: docker-dev carried ServerHistorian__* but no
ContinuousHistorization__* keys, so the recorder could never spawn. Both centrals now carry
them (default OFF — the recorder needs ContinuousHistorization:Enabled AND
ServerHistorian:Enabled), with the outbox on the per-node named volume so it survives a
container recreate. ServerHistorian__UseTls was hard-coded true and is now parameterized; a
plaintext h2c gateway was unreachable without it.
Gate results: outbox append, drain, WriteLiveValues accepted by a real historian, and restart
convergence (capture resumes after a restart with no new deployment) all pass. The read-back
hop is NOT proven — the LAN historian accepts writes but serves no reads under a "License
violation: Database tag count exceeds license tag count", and the licensed corporate one needs
the VPN. EnsureTags provisioning is likewise unexercised there (RemoteGrpc-only, and that CAP
is not listening). #491 stays open for the read-back hop alone.
Also recorded, not fixed: the recorder has NO Primary gate, unlike the alarm store-and-forward
sink, so both nodes of a redundant pair send duplicate live values. Gating it would introduce a
failover window with no capture — that is a design decision, not a cleanup.
Evidence: docs/plans/2026-08-01-continuous-historization-live-gate.md
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
A browse-committed AB CIP / TwinCAT / FOCAS tag bound with its model's DEFAULT
data type, not the one the device reported. A browsed REAL committed as
`{"tagPath":"Pump_1_Speed","dataType":"DInt"}` — and those three drivers decode
the wire bytes by that field, not by the raw row's OPC UA DataType — so the tag
read four bytes of a float through an integer decoder and published a plausible
wrong number with no error raised anywhere. FOCAS was missed when #536 was filed.
The type is CARRIED, not derived. Reverse-mapping the leaf's driver-agnostic
DriverDataType looks like the smaller fix and is wrong: SInt/Int/DInt/USInt/UInt
all collapse into Int32 on the way out of the driver, so that version would bind
a 16-bit INT as a 32-bit DINT while appearing to fix the REAL. Drivers now state
their own type name on DriverAttributeInfo.NativeDataType, which reaches the
picker as AttributeInfo.AddressFields["dataType"] and the commit mapper from
there. A leaf that states nothing keeps the model default — the pre-#536
behaviour, kept deliberately as the fallback rather than a guess.
Numeric strings are refused before parsing: Enum.TryParse accepts "9" for any
enum and Enum.IsDefined then confirms it, so an unguarded parse would bind a tag
to a type nobody named. A test caught this in the first version of the guard.
--- and the other half, which the fixture was hiding ---
Test Connect on a device with NO authored tags — every AB CIP device at the
moment an operator first presses the button — reported "CIP handshake failed"
against a healthy controller. That was written down as a fixture limitation
("ab_server doesn't implement @raw_cpu_type"). Both halves of that were wrong:
* @raw_cpu_type is not a libplctag special tag at all (only @raw, @tags and
@udt/ are), so it goes out as an ordinary symbolic name that no controller
defines, and the controller's "tag does not exist" IS the successful answer.
* ab_server answered an unresolvable tag path with CIP 0x03 where a Logix
controller answers 0x04 — ErrorBadParam vs ErrorNotFound, which the probe
classifies oppositely. Fixed in vendored patch 0002.
* With the fixture then telling the truth, the real defect surfaced:
AbCipDriverProbe.IsReachableException matched the native C status spellings
("STATUS_NOT_FOUND"), but libplctag.NET throws with the MANAGED enum name
("ErrorNotFound"). Those substrings can never match, so the reachable
classification could not fire on the exception path — which is the path every
no-authored-tags probe takes. This was failing on real hardware too, not just
against the sim. It now recovers the Status and defers to IsReachableStatus,
so there is one classification rather than two that can drift.
Rejected: switching the fallback to @tags. libplctag builds a tag-listing tag
with NO network I/O, so it reports OK against any TCP listener — measured green
against a Mosquitto broker.
Verification: full solution suite green. Falsification runs confirm exactly the
intended tests go red — 9 for the type carry, 3 (the reachable statuses only) for
the probe classification. Live on docker-dev: browse-committed TestString/
Motor_Run/TestSINT now store String/Bool/SInt, alongside last session's
Pump_1_Speed still holding the wrong DInt. TestSINT is the falsifier — its row
DataType is Int32, identical to TestDINT's, so no reverse map could have produced
"SInt".
Known consequence: pre-existing browse-committed tags keep their wrong dataType;
the fix applies to new commits only.
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
The AbCip controller browse could never be verified end to end: libplctag's
ab_server answers CIP service 0x55 (Get Instance Attribute List) on the Symbol
Object class with "unsupported", so the `@tags` walk had no backend. #468's own
definition of done offered a sim that serves `@tags` as an alternative to real
hardware; this is that.
The fixture now applies a vendored patch teaching ab_server to serve the Symbol
Object listing, with partial-transfer paging. Correctness is checked against
libplctag's own `list_tags_logix` rather than against our decoder — our decoder
is the thing under test, so its agreement would prove nothing. The clone is also
pinned to a concrete tag: the Dockerfile called `--branch release` "pinned" while
tracking a branch that moves.
That immediately exposed a real defect. CipSymbolObjectDecoder advanced to an
even byte boundary after each symbol name, but entries are packed with no
padding — libplctag walks them by sizeof(entry) + string_len, and a captured
reply confirms it. Any odd-length name therefore desynchronised every entry
behind it, and odd-length names are ordinary. In the live browse below,
`Motor_Run` is 9 bytes and first on the wire, so the old decoder would have
shown that one tag and silently dropped the other eight.
Every existing multi-entry test passed throughout, because the test builder
padded exactly as the decoder did: the two agreed with each other and neither
agreed with a controller. The builder no longer pads, and the layout is pinned
by a captured reply (CipSymbolObjectWireCaptureTests) instead of by a second
statement of the assumption. Restoring the alignment step reddens exactly the
three new tests and none of the 29 others.
Live gate on docker-dev, all three of #468's criteria:
- populated tree — all nine seeded tags render as leaves, in wire order
- refresh re-captures
- leaf commit — Pump_1_Speed + TestDINT saved with their CIP tag paths
The gate also surfaced two things it was not asked about: browse-commit drops
the discovered data type, so a browsed REAL commits as a DINT and the driver
decodes it as one (filed as #536), and Test Connect fails against this fixture
regardless of the device because it probes `@raw_cpu_type` (documented).
TwinCAT and FOCAS still have no browsable backend; #468's AbCip half is closed,
theirs is not.
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
An area or line could be named "Packaging" or "Line 1", save cleanly, and then
fail the NEXT deployment with UnsSegmentInvalid. Because a deployment snapshots
the whole database, one badly-named area blocked EVERY cluster's deploy - and
whoever hit that was usually not whoever typed the name. Only the equipment form
carried the rule; the deploy gate held its own private copy.
The rule now lives once, on Configuration.Validation.UnsSegment, and both ends
call it. DraftValidator no longer restates it.
Enforced in UnsTreeService, not just on the forms:
- Create/Update Area, Line and Equipment reject an invalid name before touching
the row.
- The CSV import reports a bad name as an ERROR, not a skip - a skip means "you
already have this row", which would tell the operator the equipment exists
when it does not. Good rows in the same batch still insert.
A form attribute could never have covered the import, and with no bUnit in this
repo a rule only a form enforces is also a rule no test can reach. The forms
keep an attribute for the immediate message, but it now references the shared
pattern rather than restating it - a second copy is how these drifted apart.
One behaviour change beyond the fix: UnsSegment.AttributePattern admits
"_default", which the equipment form's hand-written literal rejected. The form
was stricter than the gate it was mirroring, so a _default row was uneditable
through the AdminUI.
Tests: 41, falsified with three production mutations - dropping the CreateArea
guard reds exactly the CreateArea rejections, turning the import error back into
a skip reds only the import test, and narrowing AttributePattern reds only the
attribute/IsValid agreement. Every rejection is paired with a name that must
still be accepted, or a guard that refused everything would pass.
Live-verified on docker-dev: the area modal now refuses "Packaging" in place
with the rule spelled out, "packaging-b" saves, and a CSV import of two rows
reports Inserted 1 / Skipped 0 / Errors 1 naming the offending row - the path no
form attribute reaches. Rig cleaned up and a further deployment sealed.
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
Family-wide docs correction (paired with scadaproj 6d222ea4). This plan's superseded
banner was right but its task body still read as if the watchdog were merely patched.
- Dated 2026-08-01 correction section (history preserved): the timer calling
Cluster.Join(SelfAddress) was implemented and REJECTED in both repos after live
testing — outside Akka's join handshake, it fires mid-join on a routine standby
restart (peer alive, join stalled behind stale-incarnation removal) and forms a
permanent second cluster at the same address. ScadaBridge measured a split unhealed
after 90s; Task 8 hit the same shape live here.
- The 'a booting node only self-forms when NO seed answers InitJoin' claim in the Task 2
doc comment is marked FALSE for the watchdog: that guarantee belongs to Akka's native
FirstSeedNodeProcess, which runs inside the handshake; the timer does not, and fires
while a seed is answering.
- Task 8's TCP reachability guard is recorded as an INTERIM patch, not the fix — the
race survived it, which is why the watchdog was deleted rather than patched.
- Shipped design restated: self-first seed ordering (AkkaClusterOptionsValidator +
SelfFirstSeedBootstrapTests) plus the opt-in, default-OFF bootstrap guard (d1dac87f).
Source re-verified 2026-08-01: the defective watchdog is NOT present in this repo — no
ClusterBootstrapFallback.cs, no SelfFormAfter option, no Cluster.Join(SelfAddress) in
src/. Documentation-only change; no code touched.
The first pass could not exercise UNS-realm scope selection because SITE-A had
no UNS. Authored one on docker-dev (area -> line -> equipment) and re-ran that
part: the picker's UNS column offers UnsArea/UnsLine/Equipment, selecting the
equipment resolves, and the grant deploys Sealed.
Leg 15 is the one worth keeping. Re-pointing the SITE-A grant at MAIN's
equipment is the only rejection whose id is REAL, so a check that asked only
"does a row with this id exist?" would accept it and confer authority across a
cluster boundary. The list page and the deploy gate both catch it and both name
the owning cluster.
Also found and filed (Gitea #535, not fixed here): the /uns area and line modals
do not validate the UNS segment format, so "Packaging" and "Line 1" saved
happily and then failed the deploy with UnsSegmentInvalid. Only the equipment
form carries the rule. A deployment snapshots the whole database, so one bad
area name blocks every cluster's deploy - the same authoring-vs-gate gap stage 7
just closed for NodeAcl rows.
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
The last stage of #520. Enforcement landed in stages 5-6; this closes the
authoring end, where a grant could be saved, deployed and sealed green while
resolving to nothing.
Scope kind and scope target are now chosen TOGETHER from a picker over the
cluster's raw and UNS trees (AclScopePickerModal -> AclScopeTree, backed by
IAclScopeService). They were previously an unrelated dropdown and a free-text
box, so "RawDevice + a TagId" was a well-formed row. Because the trees are
rooted at the grant's own cluster and each node supplies its own kind,
NodeAclScopeCrossCluster, NodeAclScopeIdUnexpected, NodeAclScopeIdMissing and
NodeAclScopeKindRetired all become unauthorable by construction.
CheckScopeAsync mirrors DraftValidator.ValidateNodeAclScopes rule for rule and
blocks the save when a scope resolves to nothing - re-checked on submit, not
trusted from form load, because a row whose entity was deleted since is exactly
where the operator never touches the picker. The list page marks such rows
unresolved with the reason. The principal field suggests known principals and
reports how many live keys reach the one typed; reaching nobody warns rather
than blocks, since authoring ahead of minting is legitimate.
Deliberate departures from the plan line, all recorded in the design doc:
- The banners are reshaped, not removed. "Not enforced" stopped being true at
stage 6, but enforcement is still a per-node flag the AdminUI cannot see, so
a narrow grant really does confer nothing on a default-configured node.
- A Cluster row produces a NULL ScopeId. The deploy gate rejects one that
carries an id, so filling in the cluster id would author undeployable rows.
- AclScopeTree is a new component rather than a third mode on RawTree, which
already carries the /raw editing tree and the equipment reference picker.
- UNS tags are not offered: a UNS signal is a UnsTagReference, and a Tag grant
already covers the raw node and every UNS projection of it.
Also fixed: the ACL editor had a DataAnnotationsValidator and no
ValidationSummary, so an invalid NodeAclId made Create a silent no-op.
Tests: 30 in AdminUI.Tests/Acls, falsified with six production mutations -
each turned exactly the intended tests red and no others.
Live gate PASSED on docker-dev, 11 legs; both central containers asserted on
the same image by container id. The pair that matters is legs 8 and 11: the
page and the deploy gate reject the same row with the same reason. Legs 5 and
9 are the positive controls - without them the gate passes against a page that
refuses everything. Not covered: UNS-realm scope selection (SITE-A has no UNS
areas). Rig restored to zero ACL rows.
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
site-a-1 (driver-only, no ConfigDb) with Security:Acl:Enabled=true; central-1
on the same artifact as the A/B control. 11 legs.
The two that only a live rig could settle, as a transcript:
13:49:39 SiteAOnly = 53 (0x00000000) streaming ~1/s
13:49:40 SiteAOnly = (0x801F0000) revoking deploy lands; client TOLD
… 25 seconds of silence, ~25 updates that never arrived …
13:50:05 SiteAOnly = 157 (0x00000000) restoring deploy; feed resumes
53 -> 157 across the gap: the device counter ran throughout, so delivery
stopped, not the source. The 0x801F0000 frame is exactly what the
MonitoringMode.Disabled implementation would have swallowed.
Also recorded: the leg that distinguishes working enforcement from a trie that
denies everything is resolvedScopes=1 in the install log, not any denial.
Stated plainly as NOT covered: a UNS-realm grant (the rig's SITE-A has no UNS
equipment) and alarm-command scoping (no alarms there). Both covered offline,
neither seen live.
Two rig facts worth keeping: :9200 round-robins central-1/central-2, and a
stale central-2 rejected the deploy outright because its image predated the
stage-3 scope kinds — image parity must be asserted per container, not assumed
from one build. And SITEA-modbus had an empty DriverConfig, so it ran as a stub
serving nothing; until that was fixed every read returned
BadWaitingForInitialData and no allow leg could be told from a denial.
Rig left as found.
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
Completes data-plane enforcement behind Security:Acl:Enabled. Read (every
attribute, not just Value), Browse, TranslateBrowsePath, CreateMonitoredItems,
SubscribeToEvents and SetMonitoringMode now consult scoped NodeAcl grants.
With the flag off nothing changes.
Four things went beyond the stage as written, each because leaving it out
would have made the feature look like it worked:
- Browse needed ancestor visibility. The trie inherits downward, and a client
reaches a deep node by browsing the nodes above it, so without this every
grant narrower than the cluster was authored, shipped and unreachable. It is
a separate non-inheriting marker, NOT a synthesized Browse grant on each
ancestor: the additive downward walk would carry such a grant from Plant onto
Plant/Siemens/dev9 and every other sibling subtree. Pinned by a test that
goes through the evaluator — probing the trie helper directly passes against
the broken design, because a sibling is absent from the trie either way.
- Revocation had to reach LIVE subscriptions. Every other operation re-asks per
request; a subscription is authorized once and then streams, and streaming is
how machine consumers actually read this server — so stopping only the
re-asking operations would have made revocation close to cosmetic.
IDataPlaneAuthoritySource gained a Changed event.
A denied item is queued BadUserAccessDenied and then DETACHED from its
MonitoredNode2 fan-out, not set MonitoringMode.Disabled. Disabling discards
the item's queued notifications, so the client is silenced without ever being
told why. The over-the-wire test caught this; the unit-level path reports the
item as stood down either way. Detaching is also reversible, so a restored
grant resumes the feed with no client action.
- The decision cache holds the grant WALK, not the decision. Grants and the
trie are fixed for a generation; a key's ExpiresUtc passes with the clock, so
caching the whole decision would let an expired key keep working until the
next deploy — the stage-5 open-session hole in a subtler form.
- TranslateBrowsePath is gated, without which browse filtering is decorative: a
client refused a browse asks TranslateBrowsePathsToNodeIds for the same path.
Found en route: UserAuthorizationState's staleness ceiling couples two clocks.
MembershipResolvedUtc is stamped from DataPlaneAuthorizer's TimeProvider while
IsStale is evaluated against TriePermissionEvaluator's; they are the same
instance in production and diverge the moment anything injects a test clock, at
which point the evaluator denies EVERYTHING with a verdict indistinguishable
from "no grant matched". Stage 5's claim that these predicates cannot fire was
accidentally true; they are now disabled explicitly.
Also fixed: ContinuationPoint.NodeToBrowse is the manager HANDLE, not a
NodeState. The declared type is object, so the wrong cast compiled cleanly and
silently yielded null — skipping the browse source gate entirely.
Falsified: disabling the transit marking reddens exactly the three tests that
assert transit works; implementing transit as an ancestor grant reddens the
sibling-leak test; skipping the fan-out detach reddens the revocation test.
Full solution suite green.
The stage 8 live gate has NOT run yet.
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
`Security:Acl:Enabled` (default false) makes scoped `NodeAcl` grants decide
data-plane authorization at the three seams that have one, and gives HistoryRead
a check it never had. Read, Browse and Subscribe stay ungated — stage 6.
The ACL decision REPLACES the server-wide role check rather than chaining with
it. Chained, a grant would have to confer the role as well as pass the scope
check, and only Cluster-scoped grants confer a role — so every narrow grant
would have stayed inert and the flag would have looked inert too. The
load-bearing test is an ALLOW: a session carrying no roles at all writing
successfully on the strength of its grant. A deny-only test passes against a
build where the ACL is never consulted, because the role gate denies too.
Found en route: the principal never reached the session. The authenticator
passed it as OpcUaUserAuthResult.DisplayName, but RoleCarryingUserIdentity
derives DisplayName from the presented token — for a key-bearing client that is
the static username `apikey`, identical fleet-wide. So the trie's matching key
was dropped at the identity boundary, and the audit ClientUserId on a rejected
write recorded `apikey` for every consuming system. Principal and ApiKeyId are
now carried explicitly.
Beyond the stage as sketched:
- Revocation reaches an OPEN session. Every decision re-checks the
authenticating key against the current snapshot; without it a key revoked by a
deployment kept its authority until the client chose to reconnect, which for a
machine-to-machine consumer can be indefinitely.
- UserAuthorizationState is built per CALL, not per session. The per-session
shape exists so an LDAP session can re-resolve membership that changed out of
band; nothing can change out of band under API keys, so the generation stamp is
the whole invalidation signal and a per-call record picks it up for free.
IsStale/NeedsRefresh therefore cannot fire here and are documented as vestigial.
- The trie is built inside DataPlaneAuthorityState.Apply, under the same
generation as the snapshot, rather than by a second wiring at the publish
actor. An uninstalled trie denies everything — indistinguishable from working
enforcement. Deleting the install turns 13 of the 22 new tests red.
- The deploy log reports what the BUILD dropped instead of re-deriving it. The
old count agreed with the builder on one of its four drop reasons; a rule with
a blank ScopeId, a retired scope kind, or a path not starting at a realm
vanished with the log reading zero.
Both IApiKeyStore implementations widened to the full grant set. The narrowing
was redundant — ApiKeyInterimRoleMapper applies the same filter itself — and
became a trap once scope was enforced. Pinned by asserting the roles are
unchanged, with the two grants carrying different flags so it cannot pass by
accident.
Falsified by mutation: enforcement never engaging, the write seam ignoring the
ACL, the realm dropped from the scope, the revocation check removed, and the
trie never installed each turn the intended tests red and no others.
Full solution suite green.
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
Two independent defects surfaced in review of #520 stage 3.
#532 — Cluster:Roles had no OTOPCUA_ROLES fallback, despite two doc sites
promising one. A node configured only through the env var — which drives ALL of
Program.cs's conditional wiring, so it reads as the authoritative knob — bound an
empty Roles array and joined the cluster carrying no Akka roles at all. It ran,
served, and logged nothing wrong, while central reported "no driver-role nodes
are Up" and every deployment to it timed out. Not hypothetical: the field fix on
wonder-app-vd03 was to hand-add Cluster__Roles__0/1 to a Windows service's
registry environment. Anyone reading the code to diagnose that box would have
concluded from the XML doc that the fallback was already applied.
AddOtOpcUaCluster now post-configures AkkaClusterOptions through the shared
RoleParser.ResolveEffectiveRoles, so BuildClusterOptions (Akka's own member
roles), ClusterRoleInfo, the ControlPlane singleton scope and
ConfigDbApiKeyStore's ClusterId all see one answer. Two sibling validators
(ConfigSource, Telemetry) read Cluster:Roles directly and were exempting the
exact node they exist for — an env-only driver-only node has no ConfigDb and
could sit on ConfigSource:Mode=Direct unchallenged. They resolve the same way now.
Also collapsed the four open-coded `Roles.FirstOrDefault(IsClusterRole)` +
ClusterIdFromRole copies into RoleParser.ClusterIdFromRoles. They had drifted:
three classified the cluster role ordinally, so a verbatim-bound `Cluster-SITE-A`
yielded a null ClusterId — and a null ClusterId applies NO cluster filter, which
made #528's cross-cluster over-grant guard fail OPEN. IsClusterRole is now
case-insensitive; the ClusterId suffix keeps its authored spelling, because
normalizing it would silently rename the Akka role of a running deployment.
#533 — three string-identity rules in one authorization path. AclScopeRef used
default (ordinal) equality while the trie builder filtered ClusterId and hashed
principals OrdinalIgnoreCase; the deploy gate's cross-cluster check was ordinal
too, which was the worse half — it could reject a whole deploy over a ClusterId
case difference the runtime would have accepted. It only worked because resolver
and builder were handed the same NodeAcl instances.
The rule is now stated once, on AclScopeRef, and applied throughout:
identifiers (ScopeId, ClusterId, Principal) compare OrdinalIgnoreCase; path
segments compare Ordinal. AclScopePathResolver re-keys its own maps rather than
documenting a comparer contract for callers, so the two independent producers —
EF at the gate, artifact JSON on the node — cannot diverge from the outside.
PermissionTrieNode.Children stays Ordinal: those are browse names, and Speed /
speed are two nodes that can coexist under one device.
Split PermissionTrieBuildDiagnosticReason.RetiredScopeKind out of
UnresolvedScope. A legacy FolderSegment row and a typo'd id had the same message
and different remedies.
Falsified: five mutations, each reddening only its own guard — remove the
PostConfigure (3 tests), revert IsClusterRole to ordinal (3), revert AclScopeRef
equality (1), revert the resolver's map comparer (6), revert the gate's
comparisons (2), remove the FolderSegment short-circuit (1).
Test isolation: OTOPCUA_ROLES is process-global and xUnit runs classes in
parallel, so the three validator suites that now resolve effective roles share a
non-parallel collection. Without it, cases passing no roles would intermittently
observe another class's env var.
Full solution build clean; full suite green apart from two pre-existing failures
confirmed identical on 3a6b2a1d (PrimaryGateFailoverTests, and
DriverProbeRegistrationTests expecting 10 probe types where 12 now register)
plus the AbLegacy integration suite, which needs a docker fixture.
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
Joseph Doherty