Merge R2-04 Failure-visibility trio (arch-review round 2) [PR #433]
Findings 01/S-1 (AddressSpaceApplyOutcome failure field + apply.failed logging, no optimistic success), 06/S-1 (Galaxy write fails closed -> BadCommunicationError + #5 revert, no knowingly-lost raw Write), 03/S4 (PrimaryGatePolicy default-deny unknown-role-multi-driver on all gates + scripted-alarm emit gate). T13/T15 (2-node live gates) deferred -- T15 is the behavior-affecting S4 live gate, must run in heavy pass. Clean merge, build clean.
This commit is contained in:
Joseph Doherty
@@ -537,3 +537,11 @@ T14 anytime; T15 last (after T10–T13 merge-ready)
|
||||
```
|
||||
|
||||
Overall effort: **~1.5–2 dev-days code+unit, plus the rig session**. The two HIGH-RISK slices (T1 record change, T10 gate change) each land behind their full regression suites before anything else stacks on them.
|
||||
|
||||
## Execution deviations (R2-04)
|
||||
|
||||
- **T1–T3 test increments built incrementally.** The three tasks edit the same two files (`AddressSpaceApplier.cs` + a new test file). The `OpcUaServer.Tests` project sets `TreatWarningsAsErrors=true`, so an unused test helper would break the build. To keep every per-task commit green, the new `AddressSpaceApplierFailureSurfaceTests.cs` grew one task's tests at a time (T1 RebuildFailed, T2 FailedNodes removal-pass, T3 Materialise* int returns) rather than landing all tests up-front.
|
||||
- **T11 write-path assertions made retry-tolerant.** `Unknown_role_single_driver_services_write` and the Primary arm of `Known_role_wins_over_member_count` transiently hit a pre-existing driver-connect race ("driver not connected" reply while the `RecordingDriver` child's async init lags the `ApplyAck`) — orthogonal to the gate. The gate correctly ALLOWED (reason was never "not primary"). Wrapped the write in an `AwaitAssert` retry (as a real client retries), asserting the invariant under test: the gate never denies. Verified deterministic across two full-class runs.
|
||||
- **T13 (in-process 2-node delivery) implemented as a gate-reason poll, not a tag write.** Rather than deploy a known tag and drive its NodeId (brittle: the harness's fixed deploy config + the DriverHostActor path), the test resolves both nodes' `DriverHostActor` via the `ActorRegistry` (`DriverHostActorKey`) and Asks `RouteNodeWrite` for an intentionally-unmapped node, polling until the DELIVERED snapshot promotes exactly one node to Primary (the Secondary rejects with `"not primary"`; the Primary passes the gate and rejects only with a MAPPING reason). This is a faithful delivery-path assertion with a built-in negative control (a broken redundancy-topic subscribe leaves both stuck `"not primary (role unknown)"` and times out). Build-verified; marked `deferred-live` (Host.IntegrationTests memory leak + needs SQL/2-node rig — controller runs it in the serialized heavy pass).
|
||||
- **T8 / T13 / T15 marked `deferred-live`.** T8 Galaxy live smoke skips cleanly offline (its live run is the operator's, matching the plan's acceptance) — kept `completed` (authored + skip-verified). T13 (in-process 2-node integration) and T15 (2-node docker-dev rig gate) require the live rig / heavy suites the plan reserves for the serialized pass.
|
||||
- **`ServiceCollectionExtensions.DriverRole` reused for the driver-member-count read** in both `DriverHostActor.DriverMemberCount()` and `ScriptedAlarmHostActor.DriverMemberCount()` (the canonical `"driver"` role constant), rather than a new literal.
|
||||
|
||||
@@ -2,20 +2,20 @@
|
||||
"planPath": "archreview/plans/R2-04-failure-visibility-plan.md",
|
||||
"lastUpdated": "2026-07-12",
|
||||
"tasks": [
|
||||
{ "id": "T1", "subject": "Extend AddressSpaceApplyOutcome with RebuildFailed/FailedNodes + SafeRebuild returns bool (01/S-1)", "status": "pending", "blockedBy": [] },
|
||||
{ "id": "T2", "subject": "Safe* helpers return bool; Apply tallies removal-pass failures into FailedNodes", "status": "pending", "blockedBy": ["T1"] },
|
||||
{ "id": "T3", "subject": "Materialise* passes return swallowed-failure counts (void -> int, five methods)", "status": "pending", "blockedBy": ["T2"] },
|
||||
{ "id": "T4", "subject": "OpcUaApplyFailed counter + OpcUaPublishActor Error/meter surfacing of degraded applies", "status": "pending", "blockedBy": ["T3"] },
|
||||
{ "id": "T5", "subject": "Galaxy internal IMxWriteOps seam under GatewayGalaxyDataWriter (sealed SDK session untestable) (06/S-1)", "status": "pending", "blockedBy": [] },
|
||||
{ "id": "T6", "subject": "Galaxy fail-closed write on AdviseSupervisory failure -> BadCommunicationError, no raw write", "status": "pending", "blockedBy": ["T5"] },
|
||||
{ "id": "T7", "subject": "Galaxy counters galaxy.writes.advise_failed + galaxy.writes.unconfirmed (empty-statuses Good metered)", "status": "pending", "blockedBy": ["T6"] },
|
||||
{ "id": "T8", "subject": "Galaxy skip-gated live smoke: normal advised write still returns Good post-change", "status": "pending", "blockedBy": ["T6"] },
|
||||
{ "id": "T9", "subject": "PrimaryGatePolicy pure class + truth-table tests + otopcua.redundancy.primary_gate_denied counter (03/S4)", "status": "pending", "blockedBy": [] },
|
||||
{ "id": "T10", "subject": "Wire DriverHostActor gates (:977/:1039/:1095) through PrimaryGatePolicy with member-count provider + S5 log-once", "status": "pending", "blockedBy": ["T9"] },
|
||||
{ "id": "T11", "subject": "DriverHostActorPrimaryGateTests: deny-on-unknown multi-node, allow single-node, ack/emit variants, meter tags", "status": "pending", "blockedBy": ["T10"] },
|
||||
{ "id": "T12", "subject": "ScriptedAlarmHostActor alerts-emit gate adopts PrimaryGatePolicy (consistency extension)", "status": "pending", "blockedBy": ["T9"] },
|
||||
{ "id": "T13", "subject": "In-process 2-node delivery-path test (TwoNodeClusterHarness): DPS-delivered snapshot drives the gate", "status": "pending", "blockedBy": ["T11"] },
|
||||
{ "id": "T14", "subject": "docs/Redundancy.md: gate policy table, boot-window client experience, new meters, Galaxy fail-closed semantics", "status": "pending", "blockedBy": [] },
|
||||
{ "id": "T15", "subject": "LIVE GATE: 2-node docker-dev rig verify (ServiceLevel 240/100, boot-window write rejected+reverted, single /alerts row)", "status": "pending", "blockedBy": ["T10", "T11", "T12", "T13"] }
|
||||
{ "id": "T1", "subject": "Extend AddressSpaceApplyOutcome with RebuildFailed/FailedNodes + SafeRebuild returns bool (01/S-1)", "status": "completed", "blockedBy": [] },
|
||||
{ "id": "T2", "subject": "Safe* helpers return bool; Apply tallies removal-pass failures into FailedNodes", "status": "completed", "blockedBy": ["T1"] },
|
||||
{ "id": "T3", "subject": "Materialise* passes return swallowed-failure counts (void -> int, five methods)", "status": "completed", "blockedBy": ["T2"] },
|
||||
{ "id": "T4", "subject": "OpcUaApplyFailed counter + OpcUaPublishActor Error/meter surfacing of degraded applies", "status": "completed", "blockedBy": ["T3"] },
|
||||
{ "id": "T5", "subject": "Galaxy internal IMxWriteOps seam under GatewayGalaxyDataWriter (sealed SDK session untestable) (06/S-1)", "status": "completed", "blockedBy": [] },
|
||||
{ "id": "T6", "subject": "Galaxy fail-closed write on AdviseSupervisory failure -> BadCommunicationError, no raw write", "status": "completed", "blockedBy": ["T5"] },
|
||||
{ "id": "T7", "subject": "Galaxy counters galaxy.writes.advise_failed + galaxy.writes.unconfirmed (empty-statuses Good metered)", "status": "completed", "blockedBy": ["T6"] },
|
||||
{ "id": "T8", "subject": "Galaxy skip-gated live smoke: normal advised write still returns Good post-change", "status": "completed", "blockedBy": ["T6"] },
|
||||
{ "id": "T9", "subject": "PrimaryGatePolicy pure class + truth-table tests + otopcua.redundancy.primary_gate_denied counter (03/S4)", "status": "completed", "blockedBy": [] },
|
||||
{ "id": "T10", "subject": "Wire DriverHostActor gates (:977/:1039/:1095) through PrimaryGatePolicy with member-count provider + S5 log-once", "status": "completed", "blockedBy": ["T9"] },
|
||||
{ "id": "T11", "subject": "DriverHostActorPrimaryGateTests: deny-on-unknown multi-node, allow single-node, ack/emit variants, meter tags", "status": "completed", "blockedBy": ["T10"] },
|
||||
{ "id": "T12", "subject": "ScriptedAlarmHostActor alerts-emit gate adopts PrimaryGatePolicy (consistency extension)", "status": "completed", "blockedBy": ["T9"] },
|
||||
{ "id": "T13", "subject": "In-process 2-node delivery-path test (TwoNodeClusterHarness): DPS-delivered snapshot drives the gate", "status": "deferred-live", "note": "rig/live; Host.IntegrationTests (~16GB/run leak + needs SQL/2-node rig) — implemented + build-verified, controller runs it in the serialized heavy pass", "blockedBy": ["T11"] },
|
||||
{ "id": "T14", "subject": "docs/Redundancy.md: gate policy table, boot-window client experience, new meters, Galaxy fail-closed semantics", "status": "completed", "blockedBy": [] },
|
||||
{ "id": "T15", "subject": "LIVE GATE: 2-node docker-dev rig verify (ServiceLevel 240/100, boot-window write rejected+reverted, single /alerts row)", "status": "deferred-live", "note": "rig/live; 2-node docker-dev rig session (rebuild both centrals) — merge-blocking gate the controller runs in the serialized heavy pass", "blockedBy": ["T10", "T11", "T12", "T13"] }
|
||||
]
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user