Ran the full item-1 grind (authored a live Modbus equipment tag, connected
to the sim, LDAP-authed writes, docker pause the peer). 4 failing wrapped
writes produced 0 retry/breaker lines on either central.
Root cause (corrects the issue's hypothesis): ModbusDriver.WriteAsync swallows
all exceptions and returns WriteResult(StatusBadInternalError) — never throws;
CapabilityInvoker feeds only exceptions to Polly; breaker ShouldHandle is
Handle<Exception>; Write retry pinned to 0 (R2-02/S-8). So a failing wrapped
write emits no line by construction, for any polling driver. The line is
reachable ONLY for a session driver (OpcUaClient) faulting mid-Discover/Subscribe
(30s Polly timeout throws) — a production timing race, not deterministically
forcible on the rig. Behaviour stays unit-proven by the pipeline-builder test.
- New: archreview/plans/artifacts/456-retry-breaker-live-finding-2026-07-15.md
- FOLLOWUP-10 updated with the structural finding + recommendation to close item 1.
R2-11 Phase C. All six equipment-tag parsers (Modbus/S7/AbCip/AbLegacy/
TwinCAT/Focas) now read enum fields via the new TagConfigJson.TryReadEnumStrict:
absent -> fallback, valid -> parsed, present-but-invalid (typo) -> TryParse
returns false -> EquipmentTagRefResolver.TryResolve false -> driver surfaces
BadNodeIdUnknown, instead of the old lenient path that silently defaulted a
typo to a wrong-width Good.
Modbus flips all three enum fields (region/dataType/byteOrder); the other five
flip dataType. The deploy-time Deployment:TagConfigValidationMode=Error gate is
unchanged and remains the operator pre-flight.
Coverage:
- Six *EquipmentTagParserStrictnessTests inverted from Freeze_typo_* (lenient)
to Typo_*_rejects_the_tag + Valid_*_still_parses.
- TagConfigJsonTests.TryReadEnumStrict_rejects_only_invalid matrix.
- Driver-level end-to-end proof:
AbCipEquipmentTagTests.Driver_read_of_a_typod_dataType_ref_surfaces_BadNodeIdUnknown
drives the real AbCipDriver.ReadAsync through resolve->status.
Golden parity corpus has no typo'd enums, so the flip is a no-op there.
Full solution builds clean; all six driver suites + core + parity + gate green.
2-container docker-dev rig: hard-killing the oldest node makes the younger
survivor down ITSELF (SBR DownReachable including myself -> CoordinatedShutdown)
-> no failover, total outage. Reproduced twice. Confirms #459 is REAL and
disproves Option A (keep keep-oldest). Baseline ServiceLevel 250(leader)/240(follower)
captured before the kill. Documents why (textbook 2-node keep-oldest limitation;
down-if-alone doesn't rescue a crashed oldest) and why the in-process
HardKillFailoverTests gave false confidence (Transport.Shutdown keeps the
ActorSystem alive, not a real crash). Fix needs Option B (witness+keep-majority)
or Option C (DB-lease arbiter).
Captures the docker-dev live verification for follow-up #10 acceptance item 2:
a non-default per-instance ResilienceConfig (Subscribe.retryCount:999) authored on
MAIN rode the deploy artifact into BOTH central-1/central-2 runtimes, which parsed
and clamped it — the authored value appears verbatim in the spawn-time diagnostic.
Confirms the artifact bytes carry ResilienceConfig and the runtime pipeline applies
authored (not tier-default) policy. Item 1 (raw retry/breaker-on-fault log line)
remains, with the connect-then-idempotent-write recipe documented.
The 4 non-blocking residuals (#10 live-gate/ResilienceConfig sub-gap, R2-11 Phase C
strict parsers, CVE-2025-6965 suppression removal, SBR oldest-node failover gap) are
now tracked as Gitea issues, cross-referenced in the STATUS.md capstone.
Ran Category=LiveIntegration 6/6 against the running local 0.2.0 HistorianGateway
(localhost:5221 h2c) + the real AVEVA historian over VPN. Confirms R2-08 S3's
GatewayHistorianDataSource read path and re-confirms R2-06. With R2-01 + R2-08
GLAuth already self-serviced, every Round-2 live gate is now GREEN.
LdapGlauthOutageLiveTests drives the S2 directory-outage circuit through the
production auth path against a real GLAuth (docker pause/unpause) on 10.100.0.35.
Only R2-06 VPN + R2-08 S3 LiveIntegration gateway soak remain (both need VPN).
Running S7_1500ConnectTimeoutOutageTests (docker-pause blackhole, self-serviceable
over SSH) uncovered + fixed the read-leg wall-clock gap STAB-14 left. Updates the
R2-01 STATUS row, the needs-user/still-open lists, and the STAB-14 remediation note.
Integration-sweep follow-up #6 (SQL leg). The sweep flagged 7 heavy 2-node
deploy/failover/reconnect E2E tests as 'time out under amd64-emulated SQL' and
suggested raising deadlines. That diagnosis was WRONG: run against real
(native-amd64) SQL on the Docker host, they still hang — even at 4x deadline.
Root cause is that these tests had never actually run against real SQL, only the
EF in-memory provider, which does NOT enforce foreign keys.
Two distinct defects, both hidden by in-memory:
1. Missing FK-parent seed (5 tests). The deploy records a NodeDeploymentState row
per node, FK'd to ClusterNode (FK_NodeDeploymentState_ClusterNode_NodeId). The
tests never seeded ServerCluster + ClusterNode, so on SQL each node-state INSERT
throws and the deploy never seals (waits the full deadline regardless of size).
Added TwoNodeClusterHarness.SeedDefaultClusterAsync (seeds a ServerCluster +
a ClusterNode per node; Warm/NodeCount=2 to satisfy the SQL CHECK constraint
CK_ServerCluster_RedundancyMode_NodeCount AND the ClusterEnabledNodeCountMismatch
validator). Called it in DeployHappyPath x2 / Failover / FleetDiagnostics /
EquipmentNamespace; fixed DriverReconnect to seed node B + NodeCount=2.
2. Stale EquipmentId (EquipmentNamespace test; pre-existing on BOTH providers).
Seeded EquipmentId='eq-1', but the later-added EquipmentIdNotDerived validator
requires EquipmentId == DeriveEquipmentId(EquipmentUuid) ('EQ-'+first 12 hex).
Fixed the seed to a canonical id + matching UUID. Only surfaced once the FK fix
let the deploy reach equipment validation.
Also added OTOPCUA_HARNESS_SQL_HOST / OTOPCUA_HARNESS_LDAP_HOST overrides so the
harness fixtures can run on the native-amd64 Docker host (avoids arm64 mssql
emulation entirely).
Verified: the 5 SQL-backed classes go 11/11 green vs native SQL; in-memory default
unregressed (12/12). RoslynVirtualTagEvaluatorTests racing test is a pre-existing
flaky race (not SQL-backed; passes in isolation) — left as-is.
The Host.IntegrationTests opt-in real-LDAP mode (OTOPCUA_HARNESS_USE_LDAP=1)
used bitnami/openldap:2.6, gone since Bitnami deprecated their free image
catalog in 2025. Replaced it with GLAuth — the same LDAP server the rest of
the project already uses (docker-dev + the live OPC UA data-plane auth, both
against the shared GLAuth on :3893) — removing the lone OpenLDAP outlier and
the gone-image breakage.
- Added tests/.../Host.IntegrationTests/glauth/config.toml: baseDN dc=zb,dc=local,
a search-capable serviceaccount, dev users alice (full access) / bob (read-only),
and role groups with the same gidnumbers as scadaproj/infra/glauth so GroupToRole
maps identically.
- Compose ldap service -> glauth/glauth:latest, mounting the config, host :3894 -> :3893.
- Repointed the harness real-LDAP override from the OpenLDAP cn=admin/ldapadmin to
GLAuth's cn=serviceaccount/serviceaccount123.
Live-verified against a deployed container on :3894: the serviceaccount bind
searches and returns alice + her 5 memberOf groups; alice/bob bind with the
correct password; a wrong password yields Invalid credentials (49) — exactly the
OtOpcUaLdapAuthService flow (proven identical to the data-plane's shared-GLAuth path).
NB: real-LDAP mode is opt-in; the default Host run uses StubLdapAuthService, so
this never blocked the default suite. Integration-sweep follow-up #6 (LDAP leg);
the amd64-emulated-SQL deadline leg remains open.
Two test/fixture bugs; NO OtOpcUa driver change.
1. Topology: FocasSimFixture reads an env-overridable endpoint
(OTOPCUA_FOCAS_SIM_ENDPOINT, default localhost:8193) and exposes Host/Port,
but WireBackendTests + WireBackendCoverageTests hardcoded
focas://127.0.0.1:8193. Against a remote fixture the skip-gate passed
(remote reachable) but the driver dialed localhost -> KeyNotFound. Added a
DeviceUri property to the fixture; each test now derives DeviceHost from it.
Recovered 8 of 9.
2. Mock timer endianness (surfaced once #1 let the tests reach the mock): the
last failure read 60397977600 for a 3600 s power-on timer (= 0x3C000000 x 60).
The focas-mock's _wire_timer encoded the minute/msec fields big-endian (_u32),
but cnc_rdtimer's timer fields are LITTLE-endian on real hardware — a
documented, live-31i-B-validated quirk that FocasWireClient.ParseTimer decodes
LE (docs/plans/2026-06-25-focas-pdu-v3-30i-b-support.md). The driver was
correct; the mock was wrong. _wire_timer now emits little-endian (_u32_le).
Suite 9F/1P -> 10/10 against the remote fixture at ~/otopcua-focas.
Integration-sweep follow-up #3.
The sweep's AbLegacy 4F was a fixture-deployment gap, not a driver/test defect.
The suite needs ab_server in PCCC mode (--plc=SLC500|Micrologix|PLC/5), which
reuses the AbCip image (already on the host) and binds :44818. No PCCC container
was deployed, and the 4F-not-skip means :44818 was answering CIP (a leftover
AbCip container) so the PCCC reads returned Bad — same shared-port contamination
class as the Modbus false alarm.
Deployed the PCCC compose to ~/otopcua-ablegacy and ran each family with
AB_LEGACY_COMPOSE_PROFILE set to match the running container (required — without
it the [Theory] parametrises over all 3 families and 2 fail against a single-
family container): slc500 2/2, micrologix 1/1 (+1 skip), plc5 1/1 (+1 skip).
Skips are family-inapplicable file types. No code change.
Corrected the sweep record + added the AbLegacy bring-up + run recipe.
Integration-sweep follow-up #4.
The 2026-07 integration sweep flagged the Modbus non-standard profiles as
8F/5F/3F/8F and guessed 'pymodbus 4.x drift'. Re-investigation shows that was
wrong on both counts:
- The image runs the PINNED pymodbus 3.13.0 (verified via import pymodbus;
__version__). The '...removed in v4' log is 3.13.0's forward-deprecation
warning, misread as running 4.x.
- The failures were a fixture-cycling artifact: each profile is a separate
profile-gated compose service sharing :5020, and a plain 'docker compose down'
(no --profile) does NOT stop the running profile container. It kept holding
:5020; the next '--profile up' container silently failed to bind (compose
still says 'Started'), so every later profile run hit the STALE sim. Proven by
a raw Modbus probe: the 'mitsubishi' sim served DL205's 0xCAFE at HR0.
After force-removing all modbus containers and bringing up each profile cleanly
(verifying it actually published :5020), all four pass: dl205 16/16,
mitsubishi 13/13, s7_1500 10/10, exception_injection 17/17. No OtOpcUa or test
code change needed.
Added a profile-cycling gotcha + reliable-cycle recipe to Docker/README.md and
corrected the sweep record. Integration-sweep follow-up #2.
OpcUaServer.IntegrationTests was 4F. Two distinct, previously-masked issues:
1. Cert-store gap: the in-test client SecurityConfiguration used a bare
new SecurityConfiguration()/CertificateIdentifier(), so ValidateAsync threw
'TrustedIssuerCertificates StorePath must be specified' before any connect.
Added TestClientSecurity helper building Directory own/issuer/trusted/rejected
stores under a throwaway temp PKI dir (mirrors DefaultApplicationConfigurationFactory);
wired into DualEndpointTests + SubscriptionSurvivalTests.
2. Fixing #1 unmasked an SDK-version drift: the 1.5.378 node-cache read path
throws ServiceResultException(BadNodeIdUnknown) for a removed node instead of
returning a bad DataValue. The two subscription-survival assertions now expect
the throw via Should.ThrowAsync. Behavior under test (removed node -> unknown)
is unchanged; only the delivery mechanism differs.
Suite 4F -> 4/4 green. Integration-sweep follow-up #5.
The historian's Int1 analog-tag creation path is server-degenerate:
EnsureTags(Int1) stores an unusable stub (live-probed on the deployed
2023 R2 historian; root-caused + documented gateway-side in
HistorianGateway #11 / PR #16). HistorianTypeMapper mapped
DriverDataType.Boolean -> HistorianDataType.Int1, so Boolean historized
tags failed auto-provisioning.
Map Boolean -> Int2 instead (store 0/1). Int2 is a proven-creatable
analog type, and the value-write path already sends every value as a
double on WriteLiveValues regardless of tag type, so Boolean 0.0/1.0
round-trips cleanly. Both Float and Int2 are supported, so a
pre-existing Float Boolean tag retypes cleanly on its next deploy.
- HistorianTypeMapper: Boolean => Int2 (+ remarks)
- Unit pins: HistorianTypeMapperTests, GatewayTagProvisionerTests
(Boolean_definition_carries_explicit_Int2_presence)
- Live gate: EnsureTags_boolean_sandbox_provisions_as_Int2 (was _Int1);
retype probe reframed Float->Int2 (drops the dead Int1 leg)
- Docs: docs/Historian.md 'Boolean historization maps to Int2', CLAUDE.md,
archreview STATUS.md (R2-06 live gate -> 6/6)
Resolves#441.
- StorageRateMs bug found+fixed (PR #439); live suite 3/6 -> 5/6
- U-7: EnsureTags is a real upsert; supported-type retype ACCEPTED (Float->Double);
Int1 not creatable on this histsdk (ProtocolEvidenceMissingException, Int1-only)
- record the observed retype policy in docs/Historian.md (fills the placeholder)
R2-10: driver failures propagate through the full Polly pipeline (CircuitBreaker->
Retry->Timeout->CapabilityInvoker.cs:84) live — the #10 dispatch-gap fix + tracker
source confirmed in prod; breaker-OPEN not forced (backoff-throttled, per FOLLOWUP-10).
R2-03: rig has no driver data (all nodes Bad_WaitingForInitialData), so VT Good->Bad
can't be staged; unit-verified. 5 gates live-confirmed total; remaining items need
external infra (integration fixtures, VPN gateway, live LDAP, S7 blackhole).
All 12 R2 plans executed + merged (PRs #426-#437) in 3 waves. Consolidated unit
sweep green on merged master; 3 failures confirmed pre-existing/environmental (not
R2). Records the PR/wave map, unit-sweep tallies, and the deferred integration /
docker-dev / 2-node-rig / VPN heavy-pass checklist.
Joseph Doherty