The two items deferment.md §0.1 named as the remaining untracked engineering.
adminui-followups (19 pending / 0 completed) — CLOSED by the overlap audit the plan
itself asks for first, with no code written. All 19 tasks are shipped, superseded by
v3, or moot. WS1's premise had also expired: it assumed driver-embedded tag lists in
DriverConfig are the runtime source of truth, which v3 ended (the artifact populates
RawTags). Its capability shipped in the v3 shape — TagConfigEditorMap + DeviceFormMap,
both verified reachable from RawTagModal/DeviceModal rather than merely present.
Live gates on docker-dev (image rebuilt: the rig was running a build 3 h older than the
last code commit; parity asserted per container plus a positive check that the symbols
under test are in the shipped DLLs):
- R2-03 T11 PASSED — build 0 errors, Runtime.Tests 555/555, RoslynVirtualTagEvaluator 20/20.
- R2-03 T12 PASSED — VT 42 -> 0x80020000 -> 42, staged WITHOUT a deployment (a Modbus
register flipped on the fixture) so the VT child survived; a deploy-driven trip creates
a fresh child with no last value, which is correct-by-design and masked the result twice.
P7 confirmed: two byte-identical redeploys, zero disturbance.
- driver-reconfigure Task 3 PASSED — a faulted driver adopted a corrected config and
connected with the node's startedAt unchanged and RestartCount=0.
- R2-10 T10 still OPEN, but the recorded cause was wrong. Not ConnectionBackoff
throttling (the breaker opens on 2 failures / 30 s): the dead-endpoint failure mode
never enters the resilience pipeline. IDriver.InitializeAsync is not invoker-wrapped
(DriverInstanceActor.cs:707) and ModbusDriver swallows per-tag failures in BOTH
directions, so even the plan's fallback trigger cannot work. Live-proven: a driver
timing out every read against a blackhole reported ConsecutiveFailures=0 while the
panel read "All targets healthy."
Finding from T12: the Bad publish's documented "carrying the last-known value" is
unobservable by any client — subscription and a raw Read service call both return NULL.
The actor is correct; every existing failure assertion checked StatusCode alone, so the
value half was asserted nowhere. Adds
VirtualTagActorTests.The_Bad_result_carries_the_last_known_value, which proves the actor
emits it. The documentation is what overstates.
Re-gates Akka peer-to-peer secret replication on the current three-mesh docker-dev
rig and supersedes the 2026-07-18 six-node gate (lmxopcua#482, c878fbbd) as evidence.
That gate proved convergence to all five peers on the old single fleet-wide mesh;
after mesh Phase 6 (2839deb5) partitioned the rig into three independent 2-node
meshes, six-node convergence would be a FAILURE, not a pass. Phase 6's own leg 6
recorded "PASS (by fallback criterion)" — it inferred pair-locality structurally and
never seeded a secret. This replaces that inference with direct evidence.
All four checks PASS, on master @ 21009874 with one image (OtOpcUa.Host.dll
a434b04f byte-identical on all six nodes), Secrets libs 0.2.3:
1. Pair convergence — seeded on site-a-1 alone, on site-a-2 in <=7s (budget 60s),
byte-identical ciphertext + kek_id + updated_utc, decrypt-verified on the partner.
2. No cross-pair propagation — absent from site-b and central across 8 samples /
240s / 48 announce intervals, and clean in the reverse direction too. Isolation
controls seeded on site-b-1 and central-1 converged within their OWN pairs, so
the absence is isolation rather than a dead replicator.
3. Delete propagates within the pair — issued on site-a-2 (the receiving node, so
this also covers the reverse intra-pair direction); tombstone on site-a-1 in <1s.
4. Clean pair restart (both together) — mesh re-formed in 7s, no resurrection of the
tombstone, live secret survived and decrypted correctly on both nodes.
Zero secret-related warnings or errors fleet-wide across the whole run. Satisfies the
scadaproj#3 precondition for the OtOpcUa production topology (Akka pair-local).
Secrets:Replication:Enabled remains default-false in the product; the rig KEK is dev-only.
Rig config: SqlitePath moved onto each node's existing per-node /app/data volume. The
appsettings default resolves to /app, inside the image's writable container layer, so
the store was destroyed by any `up -d --build` recreate and was unreachable from the
host — seeding or inspecting a node meant stopping the container to docker cp the file
out. Still one private store per node; replication is still what syncs them.
Residuals in the gate doc: the live-push path is not covered (anti-entropy is what a
driver-only node can be gated on), and appsettings.json still carries a stale
"KNOWN NON-FUNCTIONAL against 0.2.0 / do not enable" comment that this gate disproves
— left uncorrected here because this run is config-only.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Ran on master @ 0b194003, post-deletion image d2a290fc5711 (provenance-guarded).
PSK matrix 7/7 incl. unset-key boot refusal; command matrix with Restart/Reconnect
driven from the real /hosts panel (#546 closed the 1A composition carry-forward by
direct evidence); exactly-once ack matrix sealed 6/6; central-kill + site-kill drills
honest (one loud WRN, +2:00 deadlines, boot-replay, no retro-ack); mid-deploy
coordinator kill: recovered coordinator fired the deadline at exactly +2:00, all
landed acks preserved, zero dupes; boundary: 3 established :4053 associations, all
pair-internal, zero site↔central; full-rig restart transport grep = 0 on all six
(one non-mesh observation: central pair reproduced the documented guard-off
cold-start split-brain, healed by single-node rejoin); MeshGrpcBoundaryTests 3/3;
terminal full suite RC=0 direct capture, 50/50 suites, 9,333/0.
I-4 cold-PSK-cache re-carried out of the program (unmeasurable on docker-dev).
No new defects.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Task 3.1: 7 ordered deletion steps + MeshPaths review fix, spec+code reviews closed.
Task 3.2: docs sweep + accuracy review, scadaproj index pushed on its main.
DoD full-suite: run 3 exit 0 (50 suites, 9326 passed); runs 1-2 each hit one
pre-existing unrelated timing flake (3 distinct tests, all untouched by the
branch, 9/9 green in isolation). Also records the exit-code-capture trap found
in run 1 and its retroactive caveat on Phase 2's check-8 wording.
Merge to master is user-owned — STOPPED at the Phase 3 DoD.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Node→central ApplyAcks over gRPC, gated live on the docker-dev rig at
4a2a3d3a. All 7 checks PASS (the 6 planned plus the mandated 4b hang
drill).
The 4b drill is the one that earns its place: with central-1 PAUSED
rather than stopped, both site-a nodes burned a full AckTimeoutSeconds
on it as DeadlineExceeded and still reached central-2 with a live
deadline — the per-endpoint FRESH deadline from 667639e6, proven on the
failure shape a hoisted deadline could not survive. Observed failover
latency ~10 s, exactly one AckTimeoutSeconds.
It also produced the run's only duplicate ack, and that turned into
evidence rather than a defect: the frozen central had in fact received
both acks and replayed them on unpause, and the coordinator reported
2/6 acks landed, not 4/6 — the idempotent absorption the sender's
at-least-once remark asserts, observed instead of argued.
Coexistence is per pair in one deployment: site-a's rows are Applied
with matching ReportApply lines on central and zero receptionist
associations, site-b's are Applied with zero ReportApply lines and a
live receptionist association. Neither pair could have used the other's
transport.
Carry-forwards recorded: the I-4 cold-PSK-cache latency is structurally
unmeasurable on docker-dev (ClusterPsks is read ahead of the store, so
InvalidateAll clears an empty cache) and moves to a store-backed rig;
a restarted central's cached channel can stay in backoff ~2 min on the
node that watched it die, absorbed by the walk with no ack lost. The
1A finding-5 wording fix gains a second confirmation — 4 of 4 admin
boots lost the PreStart race, and every ack central-1 accepted proves
the provider it said would NEVER be published was.
Docs only; the rig flip and the four Serilog Debug overrides were
reverted and are not in this diff. Rig restored to six healthy,
all-ClusterClient, sealing 6/6.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Re-ran the 1A gate on the docker-dev rig at e54883a8 after the defect-4 boot
fix. Image rebuilt (otopcua-host:dev -> 1d8452a29018); six host nodes plus
both one-shots recreated; run 1's preserved image was stale and deliberately
not reused. VERDICT: PASS.
Boot: 6/6 nodes healthy INCLUDING the central pair, 0 Hosting failed to start,
/health/ready 200 on every node, MAIN mesh 2/2.
Check 1 PASS: one DispatchDeployment, one correlation id, accepted by all six
nodes; central logs 6 accepted / 0 failed; zero ClusterClient lines on either
central node with Akka.Cluster.Tools.Client at Debug.
Check 2 PASS: deployment Sealed with all six NodeDeploymentState rows Applied
inside ~90 ms, while the four site nodes hold receptionist associations — the
only ack path they have in 1A. Mixed transport proven.
Check 4 PASS: Ack clicked on the real /alerts page against a live ACTIVATED
row; central fanned one AlarmCommand to all six over gRPC and the condition
acknowledged 0.54 s later carrying the operator identity.
Check 7 PASS: with site-b-2's GrpcPort NULLed, both central nodes warned once
and the next dispatch reached exactly five nodes; the deployment then honestly
ended TimedOut with site-b-2 alone unapplied. Row restored, 6/6 sealed after.
Check 8 PASS and no longer vacuous: 0 denials on all four interceptors, and
this time config-serve (28 KB verified artifact fetches from :4055) and
telemetry (streams open on all six, live driver health on /hosts) both carried
real traffic.
Check 3 closed by composition, with the reason now established rather than
assumed: <DriverStatusPanel has 0 usages repo-wide, no rendered page carries a
Restart/Reconnect control, and the only non-script-analysis POST is
/api/deployments. Run 1 drove the node-side leg for both commands; check 4
drives the identical AdminOperationsActor.Dispatch -> DispatchOverGrpc origin
path from the AdminUI.
Checks 5 + 6 stand from run 1 — nothing in the fix touches the interceptor.
New finding 5, recorded because it is load-bearing for the fix's design: the
PreStart resolve LOSES its race on real boots — 2 of 3 observed admin boots
logged Failed to resolve for both dependencies with the same
AuditWriterActorKey cause. The memoize-on-first-use hardening is therefore not
theoretical, and its heal was proven live by moving the coordinator singleton
onto the node that lost the race (stop central-2) and watching central-1 fan a
dispatch to all five live nodes with no dialer error. Residual is cosmetic and
carried to Phase 2: the Error text reads as permanent, and it will trip a soak
error-grep on a healthy node.
Defect 4 marked CLOSED; run 1's record kept in full underneath the combined
table. Rig restored: flip reverted, six nodes healthy on all-ClusterClient on
the committed image, three mesh ClusterClients back on each central node.
Ran the 1A live gate on the docker-dev rig at fbf7b708 (image rebuilt from
this worktree, all eight containers recreated). VERDICT: FAIL.
Both central nodes fail to boot on this branch with
MissingActorRegistryEntryException: AuditWriterActorKey — the first WithActors
starter eagerly resolves IMeshCommandDialer/IMeshPskProvider, whose graph
reaches ActorAuditWriter, whose ctor reads an ActorRegistry key registered ~60
lines later in the same builder. Reproduced on the same image with
MeshTransport__Mode=ClusterClient, so it is mode-independent and not caused by
the gate's rig-only flip. Recorded as defect 4; no product change made.
Checks 5 (auth negative/positive on real Kestrel h2c) and 6 (wrong-key
isolation, both directions, with a positive control) PASS. Checks 3 and 4 pass
their node-side leg only, driven with real RestartDriver/ReconnectDriver/
AlarmCommand envelopes that restarted and reconnected SITEA-modbus. Checks 1,
2 and 7 are blocked outright — no central, no dispatch, no contact refresh —
and check 8 is vacuous for telemetry + config-serve because central dials both
and never booted. All five gaps are named in the carry-forward register and
must be closed by a 1A gate re-run, not carried past the phase.
Issues filed by this task: lmxopcua#544 (ConfigPublishCoordinatorTimeoutTests
deflake) and lmxopcua#545 (mesh planes are h2c-only — accept the risk or add a
UseTls option); the tracking table's two placeholder rows now carry them.
Also recorded: DriverStatusPanel.razor is referenced by no page or component,
so the AdminUI Reconnect/Restart controls are unreachable on this branch and on
master — pre-existing, out of program scope, carried forward.
Rig restored: compose flip reverted, six nodes healthy on all-ClusterClient.
The rig runs the PRE-gate image (the 1A build is kept as otopcua-host:grpc-1a);
a re-run must rebuild.
Phase 0 task 0.6 of the ClusterClient->gRPC migration: wire the dark
MeshTransport Grpc-mode keys into the docker-dev rig (GrpcPsk +
CentralGrpcEndpoints on all six nodes, ClusterPsks on the central pair
only), document the new keys in docs/Configuration.md as shipped
dark/cutover pending, and record the Phase 0 live-gate evidence
(rig boot with no validator fires, two deliberate-misconfig refusals,
full suite green).
Files gitea.dohertylan.com/dohertj2/lmxopcua#542 (FleetStatusBroadcaster
mesh-blind post-Phase-6) and #543 (three pre-existing interceptors accept
a whitespace-only key as set) as follow-ups out of Phase 0's scope.
25 unmappable tags on SITE-A's sa-modbus2 closed both gaps in one setup: chip
25/26 SKIPPED, sample capped at exactly MaxSample=20, tooltip '(+5 more not
shown)'. Trimming to 20 skipped gave 20/21 with NO truncation note, pinning
the off-by-one boundary.
SITE-A is the discriminator for the proto path. Phase 6 partitioned it into
its own Akka mesh -- central's MEMBERS panel lists only central-1/central-2,
and DistributedPubSub does not federate across meshes. The single site-a
reference in central's log is the ClusterClient receptionist, a
point-to-point command path, not gossip. So the report's only route to the
page is TelemetryProtoMapNode -> gRPC -> TelemetryProtoMapCentral, and
central's view tracking a change made solely on the site node is conclusive.
Also corrects round one's claim that the fused central pair meant telemetry
never crossed gRPC -- wrong about this rig. All six ClusterNode rows carry
GrpcPort 4056 and Telemetry__Mode=Grpc; the two 'Telemetry stream to node'
lines that suggested a two-node dial set were startup failure warnings.
Rig restored: test tags deleted, SITE-A redeployed, sa-modbus2 back to 1 tag.
Also adds the two fixture connection strings to both docker-dev centrals, so
the rig can author + browse the Postgres and Oracle fixtures on 10.100.0.35.
All three legs pass against the patched AbCip fixture: the edit modal and the
manual-entry grid both browse the live nine-tag symbol table, and an unreachable
device degrades to the graceful error chip while the modal still saves.
The gate also found a defect unrelated to this change and filed it as #539: the
AbCip discovery path (LibplctagTagEnumerator, LibplctagTemplateReader) passes
libplctag the bare host, so an authored non-default EtherNet/IP port is dropped
and the browse reaches whatever answers on 44818 — #534's defect surviving in the
discovery half. Discovered because the first negative-leg endpoint differed from
the fixture only by port, which never reached the wire.
Records the outcome in the design doc §6 (including the stale-open race the
review found and fixed in both modals), closes the follow-up in the tracking
doc and the universal-browser plan, and registers RawBrowseModal's shared
race shape in deferment.md §0.1 as the deliberate leftover.
The tracking doc and program-design §9 still showed Wave 0's live gate open —
it closed 2026-08-01 (c2c3c522). Both now record Waves 0-2 complete.
New design doc for the one residual item: the raw modals' typed tag editors
pass a '{}' stub as GetDriverConfigJson, so the AbCip/TwinCAT/FOCAS pickers'
Browse button captures against no endpoint and can never succeed. Decision:
wire the merged Driver+Device config through (the RawBrowseModal/Test-connect
seam already exists), not hide the button.
Ran on the docker-dev central pair against the local HistorianGateway container talking to the real
AVEVA Historian on wonder-sql-vd03 (VPN up), with the host image rebuilt at e7fbe977.
Two things about the setup matter more than the results:
The value source had to be DRIVEN. Both the project Modbus fixture and oee-modbus-sim hold static
registers, and a static source produces no dependency-mux fan-out at all — the recorder captures
once at deploy and then goes quiet, which reads exactly like a broken recorder. Hand-written
Modbus FC6 writes put the value stream under the gate's control, which is what makes the rate
measurements below mean anything. The project fixture was also brought back up on the docker host
and MAIN-modbus-dev repointed to it, undoing the temporary :1502 repoint the read-back leg made.
The poison tag is the real shape, not a synthetic fault: GATE491-B's historianTagname was CHANGED
on an EXISTING tag, which Issue 5 guarantees is never provisioned, so its writes fail permanently
against a historian that has no such tag. That is how #538 was reached in the first place.
#537 — both nodes up: the Secondary logged the suppression line once and its outbox file froze
while the Primary's kept growing; 6 value changes x 2 tags gave WriteLiveValues{ok} +12 exactly,
where the pre-fix measurement was twice that. Failover: stopping the Primary made the survivor log
"now the recording Primary ... resuming value capture" and pick up every subsequent change. Return:
the restarted node was demoted, logged suppression, and the rate stayed at one node.
#538 — the quarantine fired with the tag named; 5 value changes over 30s gave ok +5 and error +5,
exactly one write per tag per change against a 2s drain interval.
The decisive leg is the IDLE one: 40s with no value changes left both counters unchanged and the
outbox byte size flat. The driven-rate legs only show a rate that happens to look right; a pre-fix
build re-writes its queued healthy entries on every drain pass whether or not anything new arrives,
so its counters climb during a quiet window with no input at all. Flat counters over idle is the
observation the bug cannot produce.
Incidental finding, recorded against Issue 5 rather than treated as a fix: a node RESTART cures an
unprovisioned tag. central-2's boot logged provisioning (dispatched=2, requested=2, ensured=2,
failed=0) and the poison tag started succeeding, because a full address-space rebuild treats every
historized tag as added and re-runs EnsureTags — where the incremental deploy that introduced the
renamed tag did not. So an operator who restarts a node while diagnosing "my tag isn't historizing"
makes the symptom disappear without learning the cause.
Docs only; no production code changed.
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
Both defects were found by #491's live gate against a real HistorianGateway and filed rather than
fixed at the time, because each one's fix is a design decision rather than a patch.
#537 — both nodes of a redundant pair captured every value, so the historian received duplicate
live samples for every historized tag (measured: the gateway's write counter advanced twice per
value and halved when one central was stopped).
The recorder now reads a capture gate per value, fed from the new
PrimaryGatePolicy.ShouldRecordHistorizedValues through IHistorizationRoleView — a third verdict on
the same RedundancyRoleView object, registered unconditionally on driver nodes rather than
alongside the alarm section's IRedundancyRoleView (which only exists when that section is enabled;
piggybacking is the #511 registration accident).
Two things are deliberate and easy to get backwards:
* It gates the CAPTURE, not the drain — the opposite of the alarm sink. That queue is in LocalDb
and replicates, so the Secondary already holds the Primary's rows. The historization outbox is
a node-local FasterLog replicating to nobody, so a drain-only gate would leave the standby
appending forever to a queue no one empties: unbounded disk, then a flood of stale duplicates
the moment it is promoted. Entries a node already captured are still drained by that node.
* The peer it stands down for must RECORD THE SAME REFS, not merely hold the same rows — a
Primary in this node's own per-cluster mesh. A node carrying no cluster-scoped role cannot
establish that and never stands down: under a fleet-wide election the elected Primary is
usually running a different cluster's configuration, and deferring to it would end this
cluster's historization silently, with the drain reporting success on an empty outbox forever.
That is the scope mismatch that once suspended the alarm drain on every node of the rig.
An unknown role captures, matching the alarm drain — a duplicate sample is recoverable, a hole in
the trend is not.
#538 — the drain writes every per-tag group and then acks all-or-nothing (deliberate: RemoveAsync
truncates THROUGH an id, so partial acking could drop older unwritten entries). One permanently
failing tag therefore pinned the FIFO head while every healthy tag in the batch was re-submitted
to the historian on every pass, indefinitely — duplicate samples and an outbox that never
truncated, observed live at 8.3 MB from a single unprovisionable tag.
The drain now tracks consecutive failures per historian tag and abandons that tag's entries once
it exceeds MaxWriteAttempts (new, default 5), so the head advances. The counter only advances on a
pass where some OTHER tag wrote successfully: that is what separates "this tag is poison" from
"the backend is down". A count-only quarantine looks identical on the poison-tag case and silently
discards the whole backlog during an ordinary gateway restart, turning a duplicate-samples bug
into a lost-history one. Abandonment is also deferred until every failing tag is over budget, so
nothing is discarded on a pass that could not have advanced the head anyway. Each abandonment logs
a Warning naming the tag; the total rides RecorderStatus.AbandonedEntries.
Capacity also gains a non-zero default (500,000 entries) as the disk backstop for the one case the
quarantine deliberately cannot resolve — a historian refusing every write, where retaining is
correct. Set 0 to restore unbounded retention.
Every new test was falsified by reverting the production change it guards, and each one reddens
alone:
* drop the backend-reachable evidence -> only A_backend_that_fails_every_tag_abandons_nothing
* never abandon -> only A_permanently_failing_tag_is_abandoned_so_healthy_tags_stop_being_rewritten
* latch the capture gate at construction -> only The_capture_gate_is_re_read_per_value...
* treat any Primary as a scope peer -> only the two scope-mismatch cases
Full suite green. (Four OpcUaClient.Browser failures during the run were the opc-plc fixture being
down on the shared Docker host; brought it back up and they pass.)
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
Joseph Doherty