AbCip/TwinCAT/FOCAS driver-config pages host the same *AddressPickerBody as the /uns TagModal but
did not pass GetConfigJson, so their universal Browse button would capture against an empty {} (no
device). Thread SerializeCurrentConfig through. DriverType is left to the picker-body default
(canonical registered type) rather than the page's DriverTypeKey const, whose TwinCAT/FOCAS values
("TwinCat"/"Focas") are non-canonical casing that could miss the factory TryCreate lookup.
18 tasks / 9 batch commits. One generic DiscoveryDriverBrowser captures any discovery-capable
driver's ITagDiscovery.DiscoverAsync into a browse tree and serves it via the existing picker
plumbing, lighting up AbCip/TwinCAT/FOCAS pickers with zero per-driver browser code. Live-gated
on docker-dev (binding + PatchForBrowse verified in prod; full tree-render fixture-blocked by
ab_server's missing @tags walk). See docs/plans/2026-07-15-universal-discovery-browser-*.
Live-verified on docker-dev: AbCip picker renders Browse button (full CanBrowse chain), clicking
it runs the universal-browser capture in production (central-2 logs) with PatchForBrowse
demonstrably applied (@tags walk ran without the authored config setting EnableControllerBrowse);
error surfaces gracefully. Modbus picker shows NO Browse button (negative). Populated tree-render
is fixture-blocked — ab_server returns ErrorUnsupported for the controller @tags walk (test sim
limitation, not a code defect). Two follow-ups recorded: (1) full tree-render needs a symbol-browse
capable AB backend; (2) driver-page pickers don't pass live config to the picker body.
Task 14/15/16: each picker body keeps its manual builder and appends a DriverOperator-gated
browse affordance (mirrors OpcUaClientAddressPickerBody) — Browse/Refresh/Close, DriverBrowseTree
over the universal DiscoveryDriverBrowser session, leaf-only commit (OnNodeSelected fires for
folders too), snapshot-at-open label, fire-and-forget close on dispose. _canBrowse evaluated once
in OnInitializedAsync (CanBrowse constructs a throwaway driver). FOCAS commits the leaf FullName
directly (the group/id builder can't reconstruct it). Editors pass DriverType + GetConfigJson
through. Razor-live-verified at Task 18.
Task 9: DiscoverTreeAsync UntilStable settle loop (FOCAS) — re-capture on a 1s interval until
the node set is non-empty and stable across two passes, bounded by open-timeout; on
timeout with a prior non-empty capture, return it (best-effort) instead of failing.
Task 11: BrowserSessionService resolves bespoke-first, falls back to IUniversalDriverBrowser
when no bespoke browser matches and CanBrowse is true; new CanBrowse on the service.
16 unit tests green (4 settle + 12 service).
Task 3/4/5: SupportsOnlineDiscovery => true on AbCipDriver, TwinCATDriver, FocasDriver
(their device enumeration is real, config-gated on EnableControllerBrowse / FixedTree.Enabled
which PatchForBrowse guarantees at open). One gate fact per driver test project.
Task 1: ITagDiscovery.SupportsOnlineDiscovery default member (=> false)
Task 2: Commons -> Core.Abstractions ProjectReference
Task 13: TagModal BuildEditorParameters passes DriverType + GetDriverConfigJson;
all 7 typed tag editors declare the two new [Parameter]s (DynamicComponent
throws on unmatched params, so every editor must accept them).
Every concern the 7-agent review parked is now decided and integrated:
- universal browser: in-flight capture coalescing keyed (driverType,
config-hash) + global cap of 4 concurrent captures; CanBrowse catches a
throwing TryCreate and defensively shuts down the throwaway instance;
cleanup ShutdownAsync bounded at 10s (R2-01); picker Refresh = close +
re-capture. Coalesced sessions share an immutable tree, so DisposeAsync
drops the session's reference, not the tree.
- mtconnect: UNAVAILABLE pinned to BadNoCommunication (fleet's
BadCommunicationError stays reserved for the driver's own transport
failures); TIME_SERIES sampleCount flows into ArrayDim; Subscribe timeout
bounds only the stream-start handshake; library version/TFM folded into
the license checklist.
- mqtt: browse rebirth is an explicit DriverOperator-gated "Request rebirth"
button (RequestRebirthAsync(scope)) - never fired by open/root/expand;
hand-rolled reconnect loop committed for P1 (MQTTnet v5 dropped
ManagedMqttClient); Wave-2-start library re-verification checkbox;
implement from the Sparkplug v3.0 spec text.
- bacnet: P1 opens with a first-spike checklist (package TFM, BBMD/
segmented-RPM/COV API smoke, unicast-I-Am fixture behavior); AdminUI-node
browse registers a second foreign device - BBMD FD-table sizing note.
- sql-poll: split-node browse needs env parity for Sql__ConnectionStrings__
refs on admin nodes (actionable error + session-only pasted literal);
one-row-per-key query contract (last-wins + rate-limited warning);
absent key -> BadNoData; operationTimeout > commandTimeout authoring rule;
Browser->Driver.Sql SqlClient transitive on AdminUI accepted on record.
- omron: FINS framer gated on W227 + live golden vectors; CIP string layout
is the first P1 live-gate item; AbCip harness static-init anti-pattern
note; writable-defaults-true operator warning.
- modbus-rtu: first P1 step confirms pymodbus exposes framer=rtu on a TCP
server (custom-script fallback otherwise).
- program doc: new cross-cutting rule - driver ctors must be connection-free
(the universal browser's CanBrowse throwaway instances depend on it).
Per user decision 2026-07-15: the direct-serial transport (ModbusRtuTransport,
System.IO.Ports, serial config fields, socat live rig) is not being built.
Serial RS-485 buses are reached exclusively via serial->Ethernet gateways
(Moxa NPort etc.) using the new ModbusRtuOverTcpTransport — zero new package
deps, no container device mapping, reuses the hardened socket lifecycle.
ModbusTransportMode shrinks to Tcp|RtuOverTcp (Rtu member reserved). The
direct-serial design is kept in the RTU doc's §2b as a marked record,
including the R2-01 SerialPort-BaseStream-ignores-ReadTimeout trap.
Adds the driver-expansion program design (umbrella: universal Discover-backed
browser + MTConnect, MQTT/Sparkplug B, BACnet/IP, SQL poll, Omron, Modbus RTU;
MELSEC deferred) plus the per-driver research reports.
All docs went through a 7-agent parallel review against the codebase before
this commit. Highlights fixed in review:
- universal browser: FOCAS FixedTree fills post-connect -> UntilStable settle
+ FixedTree.Enabled patch; MQTT reconciled to bespoke (was contradicting the
program doc's SupportsOnlineDiscovery=false verdict)
- modbus-rtu: SerialPort.ReadTimeout doesn't bound async BaseStream reads ->
linked-CTS per-op deadline (R2-01 class); BCL enum reuse would leak
System.IO.Ports into Contracts
- bacnet: DiscoveryRediscoverPolicy enum name; UDP 47808 contention; live
suite rewritten around unicast Who-Is + BBMD (broadcast doesn't cross VMs)
- sql-poll: real tier registration via DriverFactoryRegistry.Register;
blackhole gate must not docker-pause the shared central SQL Server
- mqtt: Sparkplug v3.0 STATE topic form; first-in-repo proto codegen noted
- omron: host hardcodes isIdempotent:false today (retry seam unshipped);
v1 scopes UDTs to dotted-leaf access
- mtconnect: SecurityClassification.ViewOnly; factory ParseEnum<T> pattern
- program doc: both valid enum-serialization patterns; IRediscoverable is
change-signal-gated; RTU P2 adds System.IO.Ports; label is host-side
#420 (filed 2026-05-01 as '4 inert scaffolds + D.1 smoke') is stale. Current
master + the mxaccessgw sibling repo show the feature landed on both sides:
- mxaccessgw A.2/A.4: native alarms served via the wnwrap consumer
(WnWrapAlarmConsumer over WNWRAPCONSUMERLib.wwAlarmConsumerClass) +
FailoverAlarmConsumer + AlarmDispatcher; StreamAlarms active-alarm snapshot
(20 alarms verified live). The WM_APP-pump premise this issue was written
against was ruled out by the 2026-05-01 probes; the design pivoted to wnwrap.
- mxaccessgw A.3: NotWiredAlarmRpcDispatcher gone (only in generated proto);
ack routes MxGatewayClient.AcknowledgeAlarmAsync -> AlarmClient.AlarmAckByGUID.
- lmxopcua: GalaxyDriver : IAlarmSource wires GatewayGalaxyAlarmFeed
(-> OnAlarmEvent, native metadata incl. operator-comment) + inbound ack via
GatewayGalaxyAlarmAcknowledger; native feed live-proven by
GatewayGalaxyAlarmFeedLiveTests.
- C.1 (SdkAlarmHistorianWriteBackend / aahClientManaged): SUPERSEDED. The whole
Wonderware sidecar was retired; alarm-history writes cut over to the
HistorianGateway (GatewayAlarmHistorianWriter, SendEvent path), live-verified
2026-07-15 (R2-06/R2-08, 6/6). The target no longer exists in the tree.
Updated the alarms-over-gateway.md banner (final reconciliation) + the
alarms-d1-smoke-artifact.md status note. Sole residual = a Windows-parity-rig
running-server OPC UA A&C round-trip (needs live Galaxy + human IDE).
Ran the full item-1 grind (authored a live Modbus equipment tag, connected
to the sim, LDAP-authed writes, docker pause the peer). 4 failing wrapped
writes produced 0 retry/breaker lines on either central.
Root cause (corrects the issue's hypothesis): ModbusDriver.WriteAsync swallows
all exceptions and returns WriteResult(StatusBadInternalError) — never throws;
CapabilityInvoker feeds only exceptions to Polly; breaker ShouldHandle is
Handle<Exception>; Write retry pinned to 0 (R2-02/S-8). So a failing wrapped
write emits no line by construction, for any polling driver. The line is
reachable ONLY for a session driver (OpcUaClient) faulting mid-Discover/Subscribe
(30s Polly timeout throws) — a production timing race, not deterministically
forcible on the rig. Behaviour stays unit-proven by the pipeline-builder test.
- New: archreview/plans/artifacts/456-retry-breaker-live-finding-2026-07-15.md
- FOLLOWUP-10 updated with the structural finding + recommendation to close item 1.
R2-11 Phase C. All six equipment-tag parsers (Modbus/S7/AbCip/AbLegacy/
TwinCAT/Focas) now read enum fields via the new TagConfigJson.TryReadEnumStrict:
absent -> fallback, valid -> parsed, present-but-invalid (typo) -> TryParse
returns false -> EquipmentTagRefResolver.TryResolve false -> driver surfaces
BadNodeIdUnknown, instead of the old lenient path that silently defaulted a
typo to a wrong-width Good.
Modbus flips all three enum fields (region/dataType/byteOrder); the other five
flip dataType. The deploy-time Deployment:TagConfigValidationMode=Error gate is
unchanged and remains the operator pre-flight.
Coverage:
- Six *EquipmentTagParserStrictnessTests inverted from Freeze_typo_* (lenient)
to Typo_*_rejects_the_tag + Valid_*_still_parses.
- TagConfigJsonTests.TryReadEnumStrict_rejects_only_invalid matrix.
- Driver-level end-to-end proof:
AbCipEquipmentTagTests.Driver_read_of_a_typod_dataType_ref_surfaces_BadNodeIdUnknown
drives the real AbCipDriver.ReadAsync through resolve->status.
Golden parity corpus has no typo'd enums, so the flip is a no-op there.
Full solution builds clean; all six driver suites + core + parity + gate green.
Corrects the #459 finding. 2-node keep-oldest recovery works fine (the ScadaBridge
sister project proves it); OtOpcUa was missing the supervision pieces that make it
automatic, and docs/Redundancy.md wrongly claimed in-place oldest-crash failover.
Mechanism (confirmed on a 2-container rig + by decompiling Akka KeepOldest.OldestDecision):
on an OLDEST-node crash keep-oldest downs the LONE survivor (DownReachable including
myself) — down-if-alone can't rescue a lone survivor (its branch needs >=2 survivors).
Recovery is exit-and-rejoin: run-coordinated-shutdown-when-down terminates the node and
the service supervisor restarts it. My earlier 'total outage' was a docker-dev artifact
(no restart policy); production Install-Services.ps1 already has sc.exe failure restart.
Changes (ScadaBridge parity):
- ActorSystemTerminationWatchdog (Host, registered after AddAkka): watches
ActorSystem.WhenTerminated and on an unexpected self-down calls StopApplication so the
process exits (supervisor restarts it) instead of idling with a dead actor system.
Distinguishes graceful shutdown via _stopRequested + ApplicationStopping. 3 unit tests.
- docker-dev: restart: unless-stopped on the host anchor (models production supervision) +
both redundancy peers in SeedNodes so a restarted node re-forms via either peer.
- docs/Redundancy.md: rewrote the split-brain recovery section — younger-loss = in-place
fast failover; oldest-loss = exit-and-rejoin under supervision (not in-place); the three
requirements (supervisor + watchdog + both-node seeds); flagged HardKillFailoverTests as
non-representative (Transport.Shutdown, not a real crash). Instant in-place takeover on
ANY single loss needs 3+ members.
Cluster.Tests 29/29 (SBR guards), watchdog tests 3/3, full solution builds.
Live re-verify of the watchdog image pending (host docker disk full).
2-container docker-dev rig: hard-killing the oldest node makes the younger
survivor down ITSELF (SBR DownReachable including myself -> CoordinatedShutdown)
-> no failover, total outage. Reproduced twice. Confirms #459 is REAL and
disproves Option A (keep keep-oldest). Baseline ServiceLevel 250(leader)/240(follower)
captured before the kill. Documents why (textbook 2-node keep-oldest limitation;
down-if-alone doesn't rescue a crashed oldest) and why the in-process
HardKillFailoverTests gave false confidence (Transport.Shutdown keeps the
ActorSystem alive, not a real crash). Fix needs Option B (witness+keep-majority)
or Option C (DB-lease arbiter).
Captures the docker-dev live verification for follow-up #10 acceptance item 2:
a non-default per-instance ResilienceConfig (Subscribe.retryCount:999) authored on
MAIN rode the deploy artifact into BOTH central-1/central-2 runtimes, which parsed
and clamped it — the authored value appears verbatim in the spawn-time diagnostic.
Confirms the artifact bytes carry ResilienceConfig and the runtime pipeline applies
authored (not tier-default) policy. Item 1 (raw retry/breaker-on-fault log line)
remains, with the connect-then-idempotent-write recipe documented.
The arch-review #10 sub-gap worried that per-instance ResilienceConfig never
reaches the runtime pipeline. The threading is in fact wired end-to-end (task #13):
ConfigComposer serialises the whole DriverInstance entity, so ResilienceConfig
rides the artifact into DriverInstanceSpec, which DriverHostActor layers onto the
driver's Polly pipeline. The only leg with no test was the real composer->artifact
serialization — the existing DeploymentArtifactTests hand-build the JSON.
Adds ResilienceConfig_survives_ConfigComposer_to_ParseDriverInstances_round_trip
(mirrors the DeviceHost round-trip guard): seeds a DriverInstance WITH a non-default
override + one WITHOUT, runs the real SnapshotAndFlattenAsync, and asserts
ParseDriverInstances recovers the override byte-for-byte (and null stays null).
Guards against a future projection / [JsonIgnore] silently reverting authored
resilience policy to tier defaults while hand-built artifact tests stay green.
ControlPlane.Tests ConfigComposerTests 6/6 green.
The patched native SQLite bundle shipped: SQLitePCLRaw.bundle_e_sqlite3 2.1.12
is the first version outside the CVE-2025-6965 / GHSA-2m69-gcr7-jv3q vulnerable
range (<= 2.1.11), embedding the SQLite 3.50.2+ fix.
- Bump Microsoft.Data.Sqlite 9.0.0 -> 10.0.7 (aligns with the EF Core 10.0.7 family).
This alone still pulls the native bundle 2.1.11, so:
- Add a surgical direct pin of SQLitePCLRaw.bundle_e_sqlite3 2.1.12 in Core.AlarmHistorian
(the sole consumer), overriding the transitive 2.1.11 without enabling
CentralPackageTransitivePinningEnabled (which breaks the Roslyn 5.0/4.12 split).
- Remove the temporary NuGetAuditSuppress from Directory.Build.props.
Verified: dotnet restore/build clean with audit active (no GHSA-2m69), every
in-solution project resolves lib.e_sqlite3 2.1.12, AlarmHistorian tests 29/29 green
against the real native bundle.
Closes#458
The [reference_*] pointers target agent memory (not repo files) and rendered as
undefined markdown shortcut-reference links; converted to plain code spans. All
repo-relative links + paths in infra/README.md verified to resolve.
The 4 non-blocking residuals (#10 live-gate/ResilienceConfig sub-gap, R2-11 Phase C
strict parsers, CVE-2025-6965 suppression removal, SBR oldest-node failover gap) are
now tracked as Gitea issues, cross-referenced in the STATUS.md capstone.
Consolidates the scattered infra facts (docker host layout, driver fixtures, the
three env-gated live-gate recipes, the Host.IntegrationTests harness, the HistGW
local deploy, docker-dev rig, shared GLAuth, deploy) into infra/README.md as the
single entry point, cross-linked to dev-environment.md + the deep docs. CLAUDE.md
points at it from the Docker Workflow section.
Ran Category=LiveIntegration 6/6 against the running local 0.2.0 HistorianGateway
(localhost:5221 h2c) + the real AVEVA historian over VPN. Confirms R2-08 S3's
GatewayHistorianDataSource read path and re-confirms R2-06. With R2-01 + R2-08
GLAuth already self-serviced, every Round-2 live gate is now GREEN.
LdapGlauthOutageLiveTests drives the S2 directory-outage circuit through the
production auth path against a real GLAuth (docker pause/unpause) on 10.100.0.35.
Only R2-06 VPN + R2-08 S3 LiveIntegration gateway soak remain (both need VPN).
Closes the R2-08 S2 live leg. The offline LdapAuthResilienceTests prove the
directory-outage circuit with fakes + an unroutable-host blackhole; this drives
the SAME circuit through the production auth path (OtOpcUaLdapAuthService +
LdapOpcUaUserAuthenticator) against a REAL GLAuth that is paused mid-run and
unpaused — and doubles as the first end-to-end verification that the PR #451
GLAuth swap (retired bitnami/openldap) binds correctly.
Triple-gated (GLAUTH_LIVE_HOST + GLAUTH_OUTAGE_START_CMD/STOP_CMD), skips clean
offline (verified: 1 skipped, 8 ms). try/finally always unpauses GLAuth.
Verified GREEN against 10.100.0.35:3894 (12 s): healthy binds (alice+bob, mapped
roles) -> docker pause -> 3 bounded failures -> circuit opens -> sub-second
"unavailable" fast-deny -> docker unpause -> half-open probe closes -> binds
succeed again. A boundary AuthTimeout feeds the circuit (authenticator :123-130),
so the docker-pause timeout shape is a faithful outage.
Follow-ups from the fleet-wide read-timeout audit that the S7 R2-01 read-leg
fix (PR #453) prompted. The audit confirmed S7 was the ONLY driver with the
async-read-ignores-socket-timeout hang; these are the two adjacent (non-hang)
findings it surfaced.
1. FOCAS TimeoutMs:0 footgun (the risky one): a non-positive Timeout made
SynchronizedFocasClient DISABLE its per-call wall-clock ceiling, reverting to
the caller's long-lived poll token — reintroducing exactly the frozen-peer
wedge S7 just eliminated, under misconfig. Clamp non-positive TimeoutMs to the
2s default at the config boundary so the deadline can never be authored away.
2. TimeoutMs validation symmetry: apply the same clamp in the AbCip + AbLegacy
factories. libplctag's Tag.Timeout setter throws on <=0, faulting tag creation
on every read/write; clamping keeps a misconfigured TimeoutMs:0 running on the
default bound instead. (Shared PositiveTimeoutOrDefault helper per factory.)
3. AbLegacy reconnect parity with AbCip: AbLegacy evicted the cached libplctag
runtime on neither the non-zero-status nor transport-exception read/write path
(AbCip evicts on both), so a data-path fault recovered only via the probe loop
/ libplctag internals. Added EvictRuntime + wired it into both read and write
failure paths so a fresh handle is created on the next call.
Tests: FOCAS 265->269 (clamp theory + positive), AbCip 336->339 (clamp theory +
positive), AbLegacy 209->212 (read-nonzero / read-exception / write-nonzero evict).
No production regressions; all three driver suites green.
Running S7_1500ConnectTimeoutOutageTests (docker-pause blackhole, self-serviceable
over SSH) uncovered + fixed the read-leg wall-clock gap STAB-14 left. Updates the
R2-01 STATUS row, the needs-user/still-open lists, and the STAB-14 remediation note.
The R2-01 live gate (S7_1500ConnectTimeoutOutageTests, docker-pause blackhole)
surfaced a real gap the offline fakes couldn't: S7.Net's ReadTimeout/WriteTimeout
map to the TcpClient's ReceiveTimeout/SendTimeout, which govern only SYNCHRONOUS
socket calls — the async read/write paths S7.Net uses ignore them. On an
established-but-frozen peer (frozen PLC / firewall DROP / cable pulled mid-flow,
TCP session still open) a read blocked until the OS TCP stack gave up (minutes),
silently wedging the poll loop: no Bad tick, no reconnect. STAB-14 fixed only the
CONNECT leg (EnsureConnectedAsync CancelAfter); this is its READ-leg sibling.
- New S7OperationDeadline: bounds every data-plane wire op with a wall-clock
ceiling (= _options.Timeout), surfacing an overrun as TimeoutException. Applied
in S7PlcAdapter to Read/ReadBytes/Write/WriteBytes/ReadStatus. OpenAsync is left
to EnsureConnectedAsync's own CancelAfter (not double-bounded).
- IsS7ConnectionFatal now classifies TimeoutException fatal → handle marked dead →
next EnsureConnectedAsync reopens (connect-timeout fix takes over from there).
- Tests: 5 S7OperationDeadline unit tests (deadline / token-honouring / caller-
cancel-passthrough / resultless), 1 driver-reaction test (read TimeoutException
→ reopen). Driver.S7.Tests 260/260.
- Live gate S7_1500ConnectTimeoutOutageTests now GREEN against the real snap7 sim
(8s: baseline Good -> pause blackhole -> Bad tick -> unpause -> recovered Good).
Joseph Doherty
dohertj2