Every round-2 finding re-verified against code (73 closed / 31 partial / 97 carried);
matrix 19/28 cells up, one down. New Critical 03/S14 (AdminUI FallbackPolicy 401s every
mesh gRPC map on a login-enabled central; masked on every rig gate by DisableLogin=true).
New Highs 03/S15, 05/STAB-19, 05/STAB-18, 03/S16 — the stale-Good family at four seams.
The connect path was the one capability call the resilience pipeline never
saw, so a driver that could not connect contributed zero breaker input and
the /hosts resilience panel read 'All targets healthy' against a dead
endpoint. InitializeAsync now routes through the invoker as
DriverCapability.Initialize, keyed on the driver-instance id.
- No pipeline retry: the actor's RetryConnect timer is the retry loop
(parser forces retryCount overrides to 0, same funnel as write-shaped).
- Breaker threshold 2, not the Read-sized 5: the breaker samples 30 s and
the reconnect timer fires every 10 s, so at most three attempts fit one
window — threshold 5 could never open. Pinned by its own test.
- An open breaker fast-fails into the same InitializeFailed path as a real
connect failure; the 15 s break window costs at most one extra tick.
The best-effort teardown ShutdownAsync stays unwrapped.
- Companion fix: DriverCapabilityInvokerFactory.Create now invalidates the
instance's tracker snapshots with its pipelines — a respawn during an
outage discarded the only breaker whose OnClosed could clear the OPEN
flag, stranding a false 'Breaker OPEN' chip forever.
Live gate PASSED on docker-dev (image symbol-verified per container):
rtu-gate against dead 10.100.0.35:5021 opened its Initialize breaker on
both centrals with the red 'BREAKER OPEN' chip on /hosts; a socat
5021→5020 forwarder recovered it (both breakers closed within 1 s, panel
back to 'All targets healthy'); healthy MAIN-modbus recorded zero breaker
events. Full suite 50/50 green. Evidence:
docs/plans/2026-08-07-adminui-followups-and-live-gates.md §7.
The two items deferment.md §0.1 named as the remaining untracked engineering.
adminui-followups (19 pending / 0 completed) — CLOSED by the overlap audit the plan
itself asks for first, with no code written. All 19 tasks are shipped, superseded by
v3, or moot. WS1's premise had also expired: it assumed driver-embedded tag lists in
DriverConfig are the runtime source of truth, which v3 ended (the artifact populates
RawTags). Its capability shipped in the v3 shape — TagConfigEditorMap + DeviceFormMap,
both verified reachable from RawTagModal/DeviceModal rather than merely present.
Live gates on docker-dev (image rebuilt: the rig was running a build 3 h older than the
last code commit; parity asserted per container plus a positive check that the symbols
under test are in the shipped DLLs):
- R2-03 T11 PASSED — build 0 errors, Runtime.Tests 555/555, RoslynVirtualTagEvaluator 20/20.
- R2-03 T12 PASSED — VT 42 -> 0x80020000 -> 42, staged WITHOUT a deployment (a Modbus
register flipped on the fixture) so the VT child survived; a deploy-driven trip creates
a fresh child with no last value, which is correct-by-design and masked the result twice.
P7 confirmed: two byte-identical redeploys, zero disturbance.
- driver-reconfigure Task 3 PASSED — a faulted driver adopted a corrected config and
connected with the node's startedAt unchanged and RestartCount=0.
- R2-10 T10 still OPEN, but the recorded cause was wrong. Not ConnectionBackoff
throttling (the breaker opens on 2 failures / 30 s): the dead-endpoint failure mode
never enters the resilience pipeline. IDriver.InitializeAsync is not invoker-wrapped
(DriverInstanceActor.cs:707) and ModbusDriver swallows per-tag failures in BOTH
directions, so even the plan's fallback trigger cannot work. Live-proven: a driver
timing out every read against a blackhole reported ConsecutiveFailures=0 while the
panel read "All targets healthy."
Finding from T12: the Bad publish's documented "carrying the last-known value" is
unobservable by any client — subscription and a raw Read service call both return NULL.
The actor is correct; every existing failure assertion checked StatusCode alone, so the
value half was asserted nowhere. Adds
VirtualTagActorTests.The_Bad_result_carries_the_last_known_value, which proves the actor
emits it. The documentation is what overstates.
R2-11 Phase C. All six equipment-tag parsers (Modbus/S7/AbCip/AbLegacy/
TwinCAT/Focas) now read enum fields via the new TagConfigJson.TryReadEnumStrict:
absent -> fallback, valid -> parsed, present-but-invalid (typo) -> TryParse
returns false -> EquipmentTagRefResolver.TryResolve false -> driver surfaces
BadNodeIdUnknown, instead of the old lenient path that silently defaulted a
typo to a wrong-width Good.
Modbus flips all three enum fields (region/dataType/byteOrder); the other five
flip dataType. The deploy-time Deployment:TagConfigValidationMode=Error gate is
unchanged and remains the operator pre-flight.
Coverage:
- Six *EquipmentTagParserStrictnessTests inverted from Freeze_typo_* (lenient)
to Typo_*_rejects_the_tag + Valid_*_still_parses.
- TagConfigJsonTests.TryReadEnumStrict_rejects_only_invalid matrix.
- Driver-level end-to-end proof:
AbCipEquipmentTagTests.Driver_read_of_a_typod_dataType_ref_surfaces_BadNodeIdUnknown
drives the real AbCipDriver.ReadAsync through resolve->status.
Golden parity corpus has no typo'd enums, so the flip is a no-op there.
Full solution builds clean; all six driver suites + core + parity + gate green.
The 4 non-blocking residuals (#10 live-gate/ResilienceConfig sub-gap, R2-11 Phase C
strict parsers, CVE-2025-6965 suppression removal, SBR oldest-node failover gap) are
now tracked as Gitea issues, cross-referenced in the STATUS.md capstone.
Ran Category=LiveIntegration 6/6 against the running local 0.2.0 HistorianGateway
(localhost:5221 h2c) + the real AVEVA historian over VPN. Confirms R2-08 S3's
GatewayHistorianDataSource read path and re-confirms R2-06. With R2-01 + R2-08
GLAuth already self-serviced, every Round-2 live gate is now GREEN.
LdapGlauthOutageLiveTests drives the S2 directory-outage circuit through the
production auth path against a real GLAuth (docker pause/unpause) on 10.100.0.35.
Only R2-06 VPN + R2-08 S3 LiveIntegration gateway soak remain (both need VPN).
Running S7_1500ConnectTimeoutOutageTests (docker-pause blackhole, self-serviceable
over SSH) uncovered + fixed the read-leg wall-clock gap STAB-14 left. Updates the
R2-01 STATUS row, the needs-user/still-open lists, and the STAB-14 remediation note.
The historian's Int1 analog-tag creation path is server-degenerate:
EnsureTags(Int1) stores an unusable stub (live-probed on the deployed
2023 R2 historian; root-caused + documented gateway-side in
HistorianGateway #11 / PR #16). HistorianTypeMapper mapped
DriverDataType.Boolean -> HistorianDataType.Int1, so Boolean historized
tags failed auto-provisioning.
Map Boolean -> Int2 instead (store 0/1). Int2 is a proven-creatable
analog type, and the value-write path already sends every value as a
double on WriteLiveValues regardless of tag type, so Boolean 0.0/1.0
round-trips cleanly. Both Float and Int2 are supported, so a
pre-existing Float Boolean tag retypes cleanly on its next deploy.
- HistorianTypeMapper: Boolean => Int2 (+ remarks)
- Unit pins: HistorianTypeMapperTests, GatewayTagProvisionerTests
(Boolean_definition_carries_explicit_Int2_presence)
- Live gate: EnsureTags_boolean_sandbox_provisions_as_Int2 (was _Int1);
retype probe reframed Float->Int2 (drops the dead Int1 leg)
- Docs: docs/Historian.md 'Boolean historization maps to Int2', CLAUDE.md,
archreview STATUS.md (R2-06 live gate -> 6/6)
Resolves#441.
- StorageRateMs bug found+fixed (PR #439); live suite 3/6 -> 5/6
- U-7: EnsureTags is a real upsert; supported-type retype ACCEPTED (Float->Double);
Int1 not creatable on this histsdk (ProtocolEvidenceMissingException, Int1-only)
- record the observed retype policy in docs/Historian.md (fills the placeholder)
R2-10: driver failures propagate through the full Polly pipeline (CircuitBreaker->
Retry->Timeout->CapabilityInvoker.cs:84) live — the #10 dispatch-gap fix + tracker
source confirmed in prod; breaker-OPEN not forced (backoff-throttled, per FOLLOWUP-10).
R2-03: rig has no driver data (all nodes Bad_WaitingForInitialData), so VT Good->Bad
can't be staged; unit-verified. 5 gates live-confirmed total; remaining items need
external infra (integration fixtures, VPN gateway, live LDAP, S7 blackhole).
All 12 R2 plans executed + merged (PRs #426-#437) in 3 waves. Consolidated unit
sweep green on merged master; 3 failures confirmed pre-existing/environmental (not
R2). Records the PR/wave map, unit-sweep tallies, and the deferred integration /
docker-dev / 2-node-rig / VPN heavy-pass checklist.
Re-ran all seven domain reviews at master f6eaa267 (reports rewritten in
place, each with a prior-finding status table): all 4 round-1 Criticals
verified closed; new top findings are the S7 connect-timeout OCE regression
(05/STAB-14), the ResilienceConfig operator-authorable brick (01/S-6), and
a batch of resilience-seam Mediums. 00-OVERALL.md carries the updated
maturity matrix + 12-item action list.
Adds R2-01..R2-12 design/implementation plans (one per action item, house
format + bite-sized TDD task breakdowns + co-located .tasks.json; 193 tasks,
~18-24 dev-days). STATUS.md updated: round-1 topology marked historical
(all merged+pushed), re-review findings table + plan pointers added.
Flip the #12 rig-bug note from 'recommend fixing' to fixed: FOLLOWUP-10 + STATUS task #12
+ new branch topology row. central-1 verified booting clean with the committed compose.
Ran the #10 live gate: built the Host image from source (crit13 tip; Gitea feed answers
200 anonymously so restore worked), stood up the docker-dev rig, central-1 booted healthy
on the new invoker wiring. Prod DI binds the REAL DriverCapabilityInvokerFactory —
confirmed live + by a new deterministic guard (ResilienceInvokerFactoryRegistrationTests,
crit13 8a8b9ec5). The behavioral retry/breaker log-line was NOT observed: reading the
Modbus driver showed its cheap paths don't throw a wrapped exception (Discover offline,
Subscribe lazy, reads non-wrapped) — only an idempotent write-to-dead-endpoint or S7 does,
not worth the multi-dependency chase since the retry behavior is deterministically proven.
Also found a rig bug: docker-dev compose's retired Wonderware ServerHistorian keys
(Host/Port/SharedSecret, Enabled=true) crash the Host at bring-up (historian-gateway cutover
now reads Endpoint -> empty UriFormatException). Worked around with an override; recommend
fixing the committed compose. FOLLOWUP-10 + STATUS updated; crit13 topology head -> 8a8b9ec5.
Updates FOLLOWUP-10 + STATUS.md now that #10 is wired:
- The filed "thread CapabilityInvoker into the actor" was infeasible — Runtime is
deliberately Polly-free (refs Core.Abstractions, not Core). Shipped a seam mirroring
IDriverFactory (branch fix/archreview-crit10-wire-capability-invoker @ 62556c24).
- Triage corrections: only 6 of 7 sites are real (GenericDriverNodeManager is test-only
scaffolding, re-annotated); tier-defaults only (ResilienceConfig not in the artifact).
- Observability finding: the resilience-status tracker has no reader (Stream E.2/E.3
never shipped) → added retry/breaker logging as the interim surface + live-verify enabler.
- Verification: negative-control, recording-invoker routing, real-invoker actor-context,
analyzer + logging tests; clean solution build; Runtime 358 / Core 238 / Analyzers 32.
- Residual: live behavioral gate (rig recipe recorded) + the ResilienceConfig-in-artifact
sub-gap follow-up.
Joseph Doherty