Live read-only check against a production CompactLogix press: with one nonexistent tag
first in the read list, libplctag's create threw LibPlcTagException("ErrorNotFound"); the
driver counted it as a failed Forward-Open, opened the device's 05/STAB-8 connect-backoff
window, and every other cold tag on the device - all real (WW_InCycle, WW_ActualPress,
WW_WorkOrderNum) - read BadCommunicationError. In steady polling the bad tag re-opens
the window each attempt, so siblings ordered after it may never connect.
- A create that fails with a "controller answered" status (NotFound/NoMatch/BadDevice,
the probe's existing reachable classification) records backoff SUCCESS, not failure.
Genuine connection failures still open the window (falsification control test).
- Read exceptions map through the libplctag Status they carry: unknown tag ->
BadNodeIdUnknown, decode past the buffer -> BadOutOfRange, timeout -> BadTimeout,
instead of a blanket BadCommunicationError. One status-name parser
(AbCipStatusMapper.TryGetStatus) now serves both the read path and the probe.
Live read-only check: a production CompactLogix lists the module tag ABB_OmniCore:I in
@tags and libplctag reads ABB_OmniCore:I.ConnectionFaulted without complaint, but the
driver refused the path as malformed (identifier rule had no colon) and reported the
tag BadCommunicationError. Colon-separated identifier/slot parts are now legal in the
controller-scope base segment only (never past it, never under Program: scope).
Live read-only check against production CompactLogix 5069-L320ERS2 presses:
- The controller @tags listing names each program (Program:UN01_Press, a system symbol)
but none of its tags; those are listed only by Program:<name>.@tags. The enumerator read
@tags alone, so every program-scoped tag was unbrowseable — including the PackML state
(Program:UN01_Press.Out_eState) and all 51 alarm bits the Ignition project reads. The
enumerator now reads each program's listing and decodes it with that program as scope;
an unreadable program listing is skipped, not fatal.
- The Symbol Object decoder ignored the dimension bits + dim words, so every discovered
array (DINT[32], REAL[32], STRING[50], UDT[23] on the two controllers checked) browsed as
a scalar. IsArray/ElementCount are now populated (multi-dim flattened). A 1-D single
element entry stays scalar: ab_server reports every scalar that way, a real controller
sets no dimension bits on scalars.
Pinned by entries captured off the controllers (CipSymbolObjectRealControllerCaptureTests,
AbCipProgramScopeEnumerationTests).
Live read-only check against production CompactLogix 5069-L320ERS2 rev 34.11 presses
(Z-35162 / 10.209.70.160): every @udt/<id> template the controller returned decoded to
null, so controller-discovered UDTs never fanned out. The decoder assumed a 12-byte
header led by the member count and [info][array][offset] blocks with ';'-terminated
names; libplctag actually returns a 14-byte header (u16 id, u32 def words, u32 instance
size, u16 member count, u16 handle), [info][type][offset] blocks, and NUL-terminated
names after a 'Name;<tail>' type name. All three test builders restated the old
assumption, so the unit suite agreed with itself (the #468 trap again); they now share a
TemplateBufferBuilder pinned by real captures (CipTemplateObjectWireCaptureTests).
Discovery follow-ons that the working decoder exposed:
- STRING (and user string types, LEN DINT + DATA SINT[n]) browse as one String variable,
top-level and nested, not as a LEN/DATA folder.
- Controller-generated members (ZZZZZZZZZZ* BOOL hosts, __* AOI backing) are not browsed.
- A UDT array is not fanned out into element-free Tag.Member paths.
Additive/wire-compatible releases across all three families; Directory.Packages.props
is the only pin surface (no csproj-level overrides, no lock files). Auth 0.3.0 adds
IApiKeyStoreMigrator and the optional Auth.ApiKeys.SqlServer package, neither needed
here. LocalDb 0.3.4's on-disk bookkeeping schema goes v2->v3 (upgrades in place on
open); 0.3.0 made the snapshot-resync flag per-table instead of per-database; 0.3.2
added validated Replication:KeepAlivePingDelay/KeepAlivePingTimeout config (defaults
60s/20s, left unset here) and a public LocalDbOplogGapException; 0.3.3 makes
RegisterReplicated throw LocalDbRegistrationException for a table that is the parent
of a row-mutating ON DELETE FK, which does not apply to this repo's four replicated
tables (no FKs). MxGateway.Client/Contracts 0.2.1 publishes the alarm snapshot_status
truncation frame and live alarm watch-list refresh; vendored the new nupkgs into
nuget-packages/ (the local-mxgw source MxGateway.* is exclusively package-source-mapped
to), matching the existing pattern from the 0.1.0->0.1.1 and 0.1.1->0.2.0 bumps.
Full solution builds clean (0 errors, 908 pre-existing analyzer warnings, unchanged
count). Full test suite: one failure, ClusterRoleInfoRoleLeaderCaseTests.
RoleLeaderChanged_for_driver_then_Driver_keeps_distinct_entries, reproduces
identically on the unbumped 0.2.1/0.2.1/0.2.0 tree under the same full-suite parallel
run (and passes standalone in isolation in 2s on the bumped tree) - a pre-existing
timing/parallelism flake, not a bump regression, so left as-is.
Updated CLAUDE.md and two docs (Configuration.md, operations/2026-07-20-localdb-pair-
replication.md) where they stated the old pin numbers and the now-superseded
per-database snapshot-resync behavior as current fact; dated historical records
(archreview snapshots, dated plan docs) were left untouched.
Claude-Session: https://claude.ai/code/session_01Wzg6NUCWHipSexLs2cgRFv
The R3-05 gate's own recipe correction, actioned. Security__Acl__Enabled:
"${OTOPCUA_ACL_ENABLED:-false}" was wired to site-a-1/site-a-2 only, so
`OTOPCUA_ACL_ENABLED=true docker compose up -d central-1 central-2` was a silent
no-op ("Running", no recreate, no env) — on the one node kind that carries
BoundedApiKeyStore at all, since only a fused admin+driver node wraps
ConfigDbApiKeyStore. The gate had to proceed via a scratch compose override.
The key is now on central-1 and central-2 too, same env var and same default
false, so today's rig behaviour is byte-identical and the plan's step 1 does what
it says. site-b stays hardcoded OFF as the A/B control.
Plan gains a "## S18 hard-deadline follow-up (2026-08-20)" section: the
cooperative-vs-hard root cause, the red-before-green observation (3/18 failing
with TimeoutException, and why the 15 existing deadline tests could not see it),
the fix, and the still-owed live re-gate table (expected vs. what the gate saw).
BoundedApiKeyStore bounded ConfigDbApiKeyStore with a
CancellationTokenSource(TimeSpan, TimeProvider) linked into the inner call. That
bounds the call only if the callee chooses to observe the token, and
Microsoft.Data.SqlClient does not while a connect/login is in flight on a
black-holed socket. The 2026-08-20 live gate paused the SQL container and watched
a keyed activation hang 42.9 s past its 2000 ms bound — the "did not answer
within 2000 ms" Warning fired only at unpause — and because this runs inside the
SDK's synchronous SessionManager-lock-holding impersonation callback, an
unrelated already-open subscription stalled 57 s behind it. That is precisely the
hazard the bound exists to prevent, and the TTL cache cannot help a miss.
BoundAsync now races the inner call against Task.Delay on the injected clock with
Task.WhenAny. Inner wins: stop the timer, await it, faults and genuine caller
cancellation propagate unchanged. Delay wins: cancel the linked token anyway (a
cooperative inner store should still stop), hand the abandoned task to
ObserveAbandoned, meter outcome=timeout, log the now-honest Warning, and throw
OperationCanceledException — the same exception the authenticator's catch-all
already maps to the standard opaque denial. Fail-closed is unchanged: nothing
here can admit.
ObserveAbandoned disposes the CTS and TOUCHES finished.Exception (the observation
itself), so a late SqlClient fault can never become an UnobservedTaskException.
Debug, not Warning — the timeout is already reported at Warning, and an outage
must not produce two lines per activation for one event.
An abandoned answer is NEVER cached: it was taken before the deadline and would
be served after it, so a key revoked or expired in the meantime could be admitted
from a lookup nobody was still waiting for. StoreTimeoutMs <= 0 still
short-circuits to the unbounded call byte-for-byte (the documented rollback).
RED BEFORE GREEN. Every pre-existing deadline test drives GatedApiKeyStore, whose
waits take the token, so a cooperative CancelAfter unblocks them and all 15
passed against the defective build — the suite could not see the gate's failure
at all. The new UncooperativeApiKeyStore ignores ct entirely and completes only
when the test releases it. Against the unmodified store the three new tests fail
3/18 with "should throw OperationCanceledException but threw TimeoutException"
under a test-side WaitAsync(2 s), so the red is a failure and not a hang — and
that exception swap IS the finding restated as an assertion.
Host.Tests 137/137.
options.Roles is unconditionally overwritten by AddOtOpcUaCluster's PostConfigure to equal
EffectiveRoles.Gossiped (T2's own structural-identity fix), so comparing options.Roles against
Gossiped could never observe a divergence — an overlay setting Cluster:Roles was silently
ignored, not fail-start. The validator now injects IConfiguration and re-reads the raw
Cluster:Roles section LIVE (reflecting any overlay applied after EffectiveRoles was first
resolved), normalized through EffectiveRoles.Resolve for reuse of its own validation.
AddOtOpcUaCluster now TryAddSingleton(configuration) so a bare ServiceCollection test harness
(not just WebApplicationBuilder) can supply IConfiguration to the validator.
Both guard tests rewritten to go through the real AddOtOpcUaCluster + PostConfigure pipeline
over a live ConfigurationManager, simulating an overlay as a second provider added AFTER
EffectiveRoles was resolved off the same object — exactly Program.cs's own sequencing. Confirmed
red beforehand: the "changes" test failed to throw against the pre-fix validator.
The default decider restarts a faulting child immediately and without limit, and
the hook makes every restart do ShutdownAsync+InitializeAsync+SubscribeAsync, so a
driver faulting on every value became a reconnect storm. Per-driver last-reprovision
stamp + DefaultReprovisionBackoff (10s, matched to DefaultReconnectInterval);
a held request is re-delivered by a single per-driver timer carrying the original
incarnation ref, so the F2 identity check still applies.
Seen RED with the backoff disabled: InitializeCount == 3 (second applied
immediately) where the test requires 2-then-3.
Also: the _children.Keys union-arm test (seeder gained taglessDrivers) and the
_desiredByDriver xmldoc note that it lingers on last-known-good after a failed apply.
T11: site-b-2 drops OTOPCUA_ROLES entirely — Cluster:Roles is its sole role source, the rig's
only node that can exhibit the two-role-authority split (01/S-12). Expected to fail every
deploy on a pre-R3-04 build.
T12: site-a-2 pins Cluster__Roles__0="Driver" (capital D) as the standing 01/C-9 case-mismatch
pin, keeping cluster-SITE-A verbatim. Pre-fix this un-elects the SITE-A Primary.
Combined into one commit (deviation from the plan's two separate T11/T12 commits): the header
comment block documenting all six nodes' role sources interleaves both nodes on adjacent lines,
so a clean split was not practical. Recorded in the plan's Execution notes.
Compose edits only — the rig is NOT brought up as part of this change.
New docs/security.md section "Mesh gRPC ingress is AllowAnonymous by design"
(archreview 03/S14): what the cookie FallbackPolicy covers, why it can block a
correctly-PSK'd call before the interceptor's verdict matters, why the
AllowAnonymous opt-out is safe (all five interceptors fail-closed), the
affected-endpoint table (fused-vs-driver-only distinction), and the two
standing guards (MeshGrpcAnonymousMetadataTests, MeshGrpcMapConventionTests)
plus the auth-stack/driver-only boundary tests.
One cross-reference line each in docs/Configuration.md (§MeshTransport,
§ConfigSource/ConfigServe) and docs/Telemetry.md, so an operator reading
about the listener ports meets the constraint where they'd look for it.
A ReprovisionRequested in flight from an old incarnation could reach the NEW child
after HandleRestartDriver swapped it in. In Connecting that child DOES handle
InitializeRequested, so it started a second concurrent InitializeAsync, racing the
#527 dispose-before-overwrite.
Seen RED: InitializeCount == 2 on a child held in Connecting by a gated init
(an already-connected child swallows InitializeRequested and passes vacuously).
Joseph Doherty