Seen RED against a T3-reverted build: all 4 failed, Created[1].SubscribeCount == 0
on the headline case. 20x isolation run green after the fix was restored.
Every round-2 finding re-verified against code (73 closed / 31 partial / 97 carried);
matrix 19/28 cells up, one down. New Critical 03/S14 (AdminUI FallbackPolicy 401s every
mesh gRPC map on a login-enabled central; masked on every rig gate by DisableLogin=true).
New Highs 03/S15, 05/STAB-19, 05/STAB-18, 03/S16 — the stale-Good family at four seams.
The connect path was the one capability call the resilience pipeline never
saw, so a driver that could not connect contributed zero breaker input and
the /hosts resilience panel read 'All targets healthy' against a dead
endpoint. InitializeAsync now routes through the invoker as
DriverCapability.Initialize, keyed on the driver-instance id.
- No pipeline retry: the actor's RetryConnect timer is the retry loop
(parser forces retryCount overrides to 0, same funnel as write-shaped).
- Breaker threshold 2, not the Read-sized 5: the breaker samples 30 s and
the reconnect timer fires every 10 s, so at most three attempts fit one
window — threshold 5 could never open. Pinned by its own test.
- An open breaker fast-fails into the same InitializeFailed path as a real
connect failure; the 15 s break window costs at most one extra tick.
The best-effort teardown ShutdownAsync stays unwrapped.
- Companion fix: DriverCapabilityInvokerFactory.Create now invalidates the
instance's tracker snapshots with its pipelines — a respawn during an
outage discarded the only breaker whose OnClosed could clear the OPEN
flag, stranding a false 'Breaker OPEN' chip forever.
Live gate PASSED on docker-dev (image symbol-verified per container):
rtu-gate against dead 10.100.0.35:5021 opened its Initialize breaker on
both centrals with the red 'BREAKER OPEN' chip on /hosts; a socat
5021→5020 forwarder recovered it (both breakers closed within 1 s, panel
back to 'All targets healthy'); healthy MAIN-modbus recorded zero breaker
events. Full suite 50/50 green. Evidence:
docs/plans/2026-08-07-adminui-followups-and-live-gates.md §7.
The two items deferment.md §0.1 named as the remaining untracked engineering.
adminui-followups (19 pending / 0 completed) — CLOSED by the overlap audit the plan
itself asks for first, with no code written. All 19 tasks are shipped, superseded by
v3, or moot. WS1's premise had also expired: it assumed driver-embedded tag lists in
DriverConfig are the runtime source of truth, which v3 ended (the artifact populates
RawTags). Its capability shipped in the v3 shape — TagConfigEditorMap + DeviceFormMap,
both verified reachable from RawTagModal/DeviceModal rather than merely present.
Live gates on docker-dev (image rebuilt: the rig was running a build 3 h older than the
last code commit; parity asserted per container plus a positive check that the symbols
under test are in the shipped DLLs):
- R2-03 T11 PASSED — build 0 errors, Runtime.Tests 555/555, RoslynVirtualTagEvaluator 20/20.
- R2-03 T12 PASSED — VT 42 -> 0x80020000 -> 42, staged WITHOUT a deployment (a Modbus
register flipped on the fixture) so the VT child survived; a deploy-driven trip creates
a fresh child with no last value, which is correct-by-design and masked the result twice.
P7 confirmed: two byte-identical redeploys, zero disturbance.
- driver-reconfigure Task 3 PASSED — a faulted driver adopted a corrected config and
connected with the node's startedAt unchanged and RestartCount=0.
- R2-10 T10 still OPEN, but the recorded cause was wrong. Not ConnectionBackoff
throttling (the breaker opens on 2 failures / 30 s): the dead-endpoint failure mode
never enters the resilience pipeline. IDriver.InitializeAsync is not invoker-wrapped
(DriverInstanceActor.cs:707) and ModbusDriver swallows per-tag failures in BOTH
directions, so even the plan's fallback trigger cannot work. Live-proven: a driver
timing out every read against a blackhole reported ConsecutiveFailures=0 while the
panel read "All targets healthy."
Finding from T12: the Bad publish's documented "carrying the last-known value" is
unobservable by any client — subscription and a raw Read service call both return NULL.
The actor is correct; every existing failure assertion checked StatusCode alone, so the
value half was asserted nowhere. Adds
VirtualTagActorTests.The_Bad_result_carries_the_last_known_value, which proves the actor
emits it. The documentation is what overstates.
Bookkeeping: 13 .tasks.json files closed — the 8 the audit named, plus
mesh-phase4 (Task 9 landed 3a590a0c), the 4th stillpending phase file, and
historian-tcp-transport closed as OBSOLETE (it targets the retired Wonderware
sidecar; there is no Wonderware backend in the tree). Each carries a
closureNote naming the evidence rather than just a status flip.
11 archreview live gates written back from STATUS.md, which had recorded them
passed since 2026-07-13/15 without ever updating the task files: R2-01 #11,
R2-02 #15/#18, R2-05 T15, R2-06 T12, R2-07 T5/T6/T12/T14, R2-11 T22/T24.
R2-03 and R2-10 were deliberately NOT closed. STATUS.md's headline says "every
Round-2 live gate is now GREEN", but its own per-item detail says R2-03 is
"live-blocked on this data-less rig" and R2-10 verified the pipeline with
"breaker-OPEN state not forced". Both now carry an openNote recording the
actual blocker.
The point of the sweep: ~140 phantom pending tasks are gone, so the genuinely
unexecuted 19-task AdminUI follow-ups plan is now the largest open item in
docs/plans/ instead of being buried under work that shipped months ago.
Tier truth: docs/v2/driver-stability.md now states that its tier table is
aspirational — every driver runs Tier A because no factory passes a tier, so
MemoryRecycle and ScheduledRecycleScheduler have never engaged, and the config
parser validates a RecycleIntervalSeconds that does nothing. It also corrects a
staleness the audit missed: the separate-Windows-service hosting it describes
for Tier C no longer exists (Galaxy moved to the mxaccessgw sidecar in PR 7.2,
FOCAS went in-process with its managed wire client). docs/drivers/README.md
already said Tier A for both, so the two no longer contradict.
No runtime change — enabling recycle on two live drivers wants its own live
gate, not a docs-pass side effect. Keep-or-delete tracked as Gitea #522.
Ran the full item-1 grind (authored a live Modbus equipment tag, connected
to the sim, LDAP-authed writes, docker pause the peer). 4 failing wrapped
writes produced 0 retry/breaker lines on either central.
Root cause (corrects the issue's hypothesis): ModbusDriver.WriteAsync swallows
all exceptions and returns WriteResult(StatusBadInternalError) — never throws;
CapabilityInvoker feeds only exceptions to Polly; breaker ShouldHandle is
Handle<Exception>; Write retry pinned to 0 (R2-02/S-8). So a failing wrapped
write emits no line by construction, for any polling driver. The line is
reachable ONLY for a session driver (OpcUaClient) faulting mid-Discover/Subscribe
(30s Polly timeout throws) — a production timing race, not deterministically
forcible on the rig. Behaviour stays unit-proven by the pipeline-builder test.
- New: archreview/plans/artifacts/456-retry-breaker-live-finding-2026-07-15.md
- FOLLOWUP-10 updated with the structural finding + recommendation to close item 1.
R2-11 Phase C. All six equipment-tag parsers (Modbus/S7/AbCip/AbLegacy/
TwinCAT/Focas) now read enum fields via the new TagConfigJson.TryReadEnumStrict:
absent -> fallback, valid -> parsed, present-but-invalid (typo) -> TryParse
returns false -> EquipmentTagRefResolver.TryResolve false -> driver surfaces
BadNodeIdUnknown, instead of the old lenient path that silently defaulted a
typo to a wrong-width Good.
Modbus flips all three enum fields (region/dataType/byteOrder); the other five
flip dataType. The deploy-time Deployment:TagConfigValidationMode=Error gate is
unchanged and remains the operator pre-flight.
Coverage:
- Six *EquipmentTagParserStrictnessTests inverted from Freeze_typo_* (lenient)
to Typo_*_rejects_the_tag + Valid_*_still_parses.
- TagConfigJsonTests.TryReadEnumStrict_rejects_only_invalid matrix.
- Driver-level end-to-end proof:
AbCipEquipmentTagTests.Driver_read_of_a_typod_dataType_ref_surfaces_BadNodeIdUnknown
drives the real AbCipDriver.ReadAsync through resolve->status.
Golden parity corpus has no typo'd enums, so the flip is a no-op there.
Full solution builds clean; all six driver suites + core + parity + gate green.
2-container docker-dev rig: hard-killing the oldest node makes the younger
survivor down ITSELF (SBR DownReachable including myself -> CoordinatedShutdown)
-> no failover, total outage. Reproduced twice. Confirms #459 is REAL and
disproves Option A (keep keep-oldest). Baseline ServiceLevel 250(leader)/240(follower)
captured before the kill. Documents why (textbook 2-node keep-oldest limitation;
down-if-alone doesn't rescue a crashed oldest) and why the in-process
HardKillFailoverTests gave false confidence (Transport.Shutdown keeps the
ActorSystem alive, not a real crash). Fix needs Option B (witness+keep-majority)
or Option C (DB-lease arbiter).
Captures the docker-dev live verification for follow-up #10 acceptance item 2:
a non-default per-instance ResilienceConfig (Subscribe.retryCount:999) authored on
MAIN rode the deploy artifact into BOTH central-1/central-2 runtimes, which parsed
and clamped it — the authored value appears verbatim in the spawn-time diagnostic.
Confirms the artifact bytes carry ResilienceConfig and the runtime pipeline applies
authored (not tier-default) policy. Item 1 (raw retry/breaker-on-fault log line)
remains, with the connect-then-idempotent-write recipe documented.
The 4 non-blocking residuals (#10 live-gate/ResilienceConfig sub-gap, R2-11 Phase C
strict parsers, CVE-2025-6965 suppression removal, SBR oldest-node failover gap) are
now tracked as Gitea issues, cross-referenced in the STATUS.md capstone.
Ran Category=LiveIntegration 6/6 against the running local 0.2.0 HistorianGateway
(localhost:5221 h2c) + the real AVEVA historian over VPN. Confirms R2-08 S3's
GatewayHistorianDataSource read path and re-confirms R2-06. With R2-01 + R2-08
GLAuth already self-serviced, every Round-2 live gate is now GREEN.
LdapGlauthOutageLiveTests drives the S2 directory-outage circuit through the
production auth path against a real GLAuth (docker pause/unpause) on 10.100.0.35.
Only R2-06 VPN + R2-08 S3 LiveIntegration gateway soak remain (both need VPN).
Running S7_1500ConnectTimeoutOutageTests (docker-pause blackhole, self-serviceable
over SSH) uncovered + fixed the read-leg wall-clock gap STAB-14 left. Updates the
R2-01 STATUS row, the needs-user/still-open lists, and the STAB-14 remediation note.
Integration-sweep follow-up #6 (SQL leg). The sweep flagged 7 heavy 2-node
deploy/failover/reconnect E2E tests as 'time out under amd64-emulated SQL' and
suggested raising deadlines. That diagnosis was WRONG: run against real
(native-amd64) SQL on the Docker host, they still hang — even at 4x deadline.
Root cause is that these tests had never actually run against real SQL, only the
EF in-memory provider, which does NOT enforce foreign keys.
Two distinct defects, both hidden by in-memory:
1. Missing FK-parent seed (5 tests). The deploy records a NodeDeploymentState row
per node, FK'd to ClusterNode (FK_NodeDeploymentState_ClusterNode_NodeId). The
tests never seeded ServerCluster + ClusterNode, so on SQL each node-state INSERT
throws and the deploy never seals (waits the full deadline regardless of size).
Added TwoNodeClusterHarness.SeedDefaultClusterAsync (seeds a ServerCluster +
a ClusterNode per node; Warm/NodeCount=2 to satisfy the SQL CHECK constraint
CK_ServerCluster_RedundancyMode_NodeCount AND the ClusterEnabledNodeCountMismatch
validator). Called it in DeployHappyPath x2 / Failover / FleetDiagnostics /
EquipmentNamespace; fixed DriverReconnect to seed node B + NodeCount=2.
2. Stale EquipmentId (EquipmentNamespace test; pre-existing on BOTH providers).
Seeded EquipmentId='eq-1', but the later-added EquipmentIdNotDerived validator
requires EquipmentId == DeriveEquipmentId(EquipmentUuid) ('EQ-'+first 12 hex).
Fixed the seed to a canonical id + matching UUID. Only surfaced once the FK fix
let the deploy reach equipment validation.
Also added OTOPCUA_HARNESS_SQL_HOST / OTOPCUA_HARNESS_LDAP_HOST overrides so the
harness fixtures can run on the native-amd64 Docker host (avoids arm64 mssql
emulation entirely).
Verified: the 5 SQL-backed classes go 11/11 green vs native SQL; in-memory default
unregressed (12/12). RoslynVirtualTagEvaluatorTests racing test is a pre-existing
flaky race (not SQL-backed; passes in isolation) — left as-is.
The Host.IntegrationTests opt-in real-LDAP mode (OTOPCUA_HARNESS_USE_LDAP=1)
used bitnami/openldap:2.6, gone since Bitnami deprecated their free image
catalog in 2025. Replaced it with GLAuth — the same LDAP server the rest of
the project already uses (docker-dev + the live OPC UA data-plane auth, both
against the shared GLAuth on :3893) — removing the lone OpenLDAP outlier and
the gone-image breakage.
- Added tests/.../Host.IntegrationTests/glauth/config.toml: baseDN dc=zb,dc=local,
a search-capable serviceaccount, dev users alice (full access) / bob (read-only),
and role groups with the same gidnumbers as scadaproj/infra/glauth so GroupToRole
maps identically.
- Compose ldap service -> glauth/glauth:latest, mounting the config, host :3894 -> :3893.
- Repointed the harness real-LDAP override from the OpenLDAP cn=admin/ldapadmin to
GLAuth's cn=serviceaccount/serviceaccount123.
Live-verified against a deployed container on :3894: the serviceaccount bind
searches and returns alice + her 5 memberOf groups; alice/bob bind with the
correct password; a wrong password yields Invalid credentials (49) — exactly the
OtOpcUaLdapAuthService flow (proven identical to the data-plane's shared-GLAuth path).
NB: real-LDAP mode is opt-in; the default Host run uses StubLdapAuthService, so
this never blocked the default suite. Integration-sweep follow-up #6 (LDAP leg);
the amd64-emulated-SQL deadline leg remains open.
Two test/fixture bugs; NO OtOpcUa driver change.
1. Topology: FocasSimFixture reads an env-overridable endpoint
(OTOPCUA_FOCAS_SIM_ENDPOINT, default localhost:8193) and exposes Host/Port,
but WireBackendTests + WireBackendCoverageTests hardcoded
focas://127.0.0.1:8193. Against a remote fixture the skip-gate passed
(remote reachable) but the driver dialed localhost -> KeyNotFound. Added a
DeviceUri property to the fixture; each test now derives DeviceHost from it.
Recovered 8 of 9.
2. Mock timer endianness (surfaced once #1 let the tests reach the mock): the
last failure read 60397977600 for a 3600 s power-on timer (= 0x3C000000 x 60).
The focas-mock's _wire_timer encoded the minute/msec fields big-endian (_u32),
but cnc_rdtimer's timer fields are LITTLE-endian on real hardware — a
documented, live-31i-B-validated quirk that FocasWireClient.ParseTimer decodes
LE (docs/plans/2026-06-25-focas-pdu-v3-30i-b-support.md). The driver was
correct; the mock was wrong. _wire_timer now emits little-endian (_u32_le).
Suite 9F/1P -> 10/10 against the remote fixture at ~/otopcua-focas.
Integration-sweep follow-up #3.
Joseph Doherty