Merge branch 'master' into fix/r3-02-stale-good-family
# Conflicts: # src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs # tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Harness/V3ArtifactSeeder.cs
This commit is contained in:
@@ -352,3 +352,87 @@ T10 last — and its step-0 control runs on master BEFORE the branch is merged
|
||||
```
|
||||
|
||||
Overall effort: **~half a dev-day code+tests, plus the two-pass rig session.** The Critical itself is five lines; everything else in this plan exists because the five lines were invisible to a seven-phase live-gate program.
|
||||
|
||||
---
|
||||
|
||||
## Execution notes (T1–T9, 2026-08-20)
|
||||
|
||||
Branch `fix/r3-01-mesh-grpc-allowanonymous`, one commit per task as specified. T1–T9 all landed;
|
||||
T10 (live gate) left `pending` for the orchestrator per the run brief. Every code-touching task
|
||||
(T1–T7) was verified red-before-fix by hand — for T2–T5 by asserting against the un-fixed
|
||||
`MeshGrpcEndpoints.cs`/`Program.cs` (via a temporary `git show <parent-commit>:<path>` swap,
|
||||
rebuild, run, then restore — confirmed byte-identical restore via `git diff --stat` each time)
|
||||
and observing the expected failure before reapplying the fix. Two things surfaced during that
|
||||
process that are worth recording:
|
||||
|
||||
1. **The census's endpoint filter could not stay a "route pattern starts with `/`" heuristic once
|
||||
T3's empty-prefix `MapGroup` landed.** `RealMethodEndpoints` in `MeshGrpcAnonymousMetadataTests`
|
||||
originally filtered on `RoutePattern.RawText?.StartsWith('/')` to exclude grpc-dotnet's own
|
||||
per-app "Unimplemented service" catch-all and per-service "Unimplemented method for X"
|
||||
fallback (neither carries a route pattern starting with `/` in the ungrouped case). Once
|
||||
LocalDbSync moved behind `app.MapGroup(string.Empty).AllowAnonymous().MapZbLocalDbSync()`,
|
||||
those SAME fallback patterns *also* gained a leading `/` inside the group, so the heuristic
|
||||
started over-counting (`DriverOnly_WithASyncPort_AlsoMapsTheLocalDbSyncEndpoint` and
|
||||
`LocalDbSync_AllowAnonymousGroup_LeavesTheRoutePatternByteIdentical` both went red for the
|
||||
wrong reason — 3 endpoints found where 1 was expected). Fixed by matching against the exact
|
||||
known set of five method route-pattern literals instead of a shape heuristic. Not a plan defect
|
||||
— the plan didn't specify the filter mechanism — but a genuine gotcha for anyone extending this
|
||||
census: **grpc-dotnet's route-pattern-shape for its own fallback endpoints is not stable across
|
||||
a plain `MapGrpcService` vs. a grouped one**, so don't filter on shape once any mesh map moves
|
||||
behind a group.
|
||||
|
||||
2. **T4's fixture does not map or wire `LocalDbSyncService`'s real dependencies.** The plan's
|
||||
fixture description says "maps the mesh services through `MapOtOpcUaMeshGrpcServices`"; T4's
|
||||
`MeshGrpcAuthStackTests` calls it with `syncListenPort: 0` (LocalDbSync NOT mapped by the
|
||||
helper) and instead hand-maps the library's `LocalDbSyncService` directly, with no
|
||||
`.AllowAnonymous()`, purely as the negative control. This is a minimal, deliberate narrowing:
|
||||
no assertion in T4 or T5 exercises LocalDbSync's real RPC (the plan's own positive/negative
|
||||
lists never name it), and standing up its DI graph (`AddZbLocalDb`/`AddZbLocalDbReplication`,
|
||||
a temp-file SQLite `ILocalDb`) would have added fixture weight with nothing to verify — the
|
||||
authorization middleware refuses an unauthorized call before the service is ever constructed,
|
||||
so the DI graph doesn't need to exist for the negative control to be meaningful. Recorded here
|
||||
because it's a case where the fixture is deliberately narrower than a literal reading of the
|
||||
plan's prose, even though every named assertion is still covered.
|
||||
|
||||
3. **The positive/negative service lists T4 and T5 assert are asymmetric, and that asymmetry is
|
||||
the plan's own, not introduced here.** T4's positive list is {`DeploymentArtifactService.Fetch`,
|
||||
`CentralCommandService.ReportApply`, `NodeCommandService.Execute`} — Telemetry is not asserted
|
||||
positively there. T5's no-PSK detail-string list is {Config-serve, Telemetry stream, Node
|
||||
command} — `CentralCommandService`'s own detail string
|
||||
(`"Central command authentication failed."`) is never asserted anywhere in T4/T5. Implemented
|
||||
exactly as specified (no scope widening per the run brief); flagging the gap in case a
|
||||
follow-up wants a `NoOrWrongPsk_CentralCommandReportApply_FailsWithTheInterceptorsDetail` case
|
||||
for full symmetry — trivial to add given the fixture already maps and keys that service.
|
||||
|
||||
4. **Observed cookie-challenge signature (T5 step 2), captured directly against this repo's
|
||||
fixture stack (macOS, Kestrel h2c, `Grpc.Net.Client` over `HttpClient`/`SocketsHttpHandler`):**
|
||||
every blocked positive-test call in the red state failed as
|
||||
`RpcException(StatusCode.Internal, "Error starting gRPC call. HttpRequestException: The HTTP/2
|
||||
server reset the stream. HTTP/2 error code 'PROTOCOL_ERROR' (0x1). (HttpProtocolError)
|
||||
HttpProtocolException: The HTTP/2 server reset the stream. HTTP/2 error code 'PROTOCOL_ERROR'
|
||||
(0x1). (HttpProtocolError)")` — never `Unauthenticated`/`PermissionDenied`. The cookie handler's
|
||||
redirect/401 breaks the prior-knowledge h2c stream before any gRPC trailers are written, so the
|
||||
client sees a transport-level reset rather than a clean gRPC status. **The live gate (T10)
|
||||
should grep node/central logs and browser network traces for `PROTOCOL_ERROR`, `/login`, `302`
|
||||
and `401` together** — not for `401` alone, and not for a gRPC `Unauthenticated`/
|
||||
`PermissionDenied` status, which this environment never actually produces for the blocked case.
|
||||
(A real reverse-proxy in front of a production deployment could still surface a clean HTTP 401
|
||||
instead of a stream reset, depending on how it terminates HTTP/2 — the plan's own multi-signal
|
||||
grep advice already covers that; this note is the empirical confirmation of one concrete shape.)
|
||||
|
||||
5. **T6 passes identically on both the pre-fix and post-fix mapper (confirmed by the same
|
||||
temporary-swap procedure).** Driver-only nodes never register `AddOtOpcUaAuth` at all, so
|
||||
`.AllowAnonymous()` presence or absence on the mesh maps has zero effect on that shape — T6 is
|
||||
a structural pin/regression-control test, not a red-then-green TDD test for a behaviour T1–T3
|
||||
changed. This matches the plan's own "standard (test-only)" classification; recorded because
|
||||
the run brief asked for red-before-fix confirmation to be checked (and reported) for every task,
|
||||
and this is the one task where "red" never existed to begin with.
|
||||
|
||||
6. **T8's `docker compose config` verification ran locally on macOS (OrbStack-backed Docker
|
||||
Compose v5.1.2)** — no Windows-VM `lmxopcua-fix` helper needed, consistent with the Mac-path
|
||||
guidance in CLAUDE.md. Confirmed both branches: unset `OTOPCUA_DISABLE_LOGIN` resolves
|
||||
`Security__Auth__DisableLogin: "true"` on both central-1/central-2; `OTOPCUA_DISABLE_LOGIN=false`
|
||||
resolves `"false"` on both.
|
||||
|
||||
No other deviations from the plan's design or task breakdown. T10 was not run — left `pending`
|
||||
per the run brief, for the orchestrator.
|
||||
|
||||
@@ -5,13 +5,13 @@
|
||||
{
|
||||
"id": "T1",
|
||||
"subject": "Extract MapOtOpcUaMeshGrpcServices from Program.cs:743-784 (behaviour-preserving) + endpoint census scaffold (03/S14)",
|
||||
"status": "pending",
|
||||
"status": "done",
|
||||
"blockedBy": []
|
||||
},
|
||||
{
|
||||
"id": "T2",
|
||||
"subject": "Census asserts IAllowAnonymous + scope/count guard-of-the-guard; .AllowAnonymous() on the four MapGrpcService maps",
|
||||
"status": "pending",
|
||||
"status": "done",
|
||||
"blockedBy": [
|
||||
"T1"
|
||||
]
|
||||
@@ -19,7 +19,7 @@
|
||||
{
|
||||
"id": "T3",
|
||||
"subject": "LocalDb sync endpoint anonymised via an empty-prefix MapGroup (MapZbLocalDbSync returns IEndpointRouteBuilder, not chainable)",
|
||||
"status": "pending",
|
||||
"status": "done",
|
||||
"blockedBy": [
|
||||
"T2"
|
||||
]
|
||||
@@ -27,7 +27,7 @@
|
||||
{
|
||||
"id": "T4",
|
||||
"subject": "MeshGrpcAuthStackTests: PSK-bearing calls succeed through the REAL AddOtOpcUaAuth + UseAuthentication/UseAuthorization stack, with an un-anonymised negative control",
|
||||
"status": "pending",
|
||||
"status": "done",
|
||||
"blockedBy": [
|
||||
"T3"
|
||||
]
|
||||
@@ -35,7 +35,7 @@
|
||||
{
|
||||
"id": "T5",
|
||||
"subject": "Status fidelity: a no-PSK call reports the interceptor's DETAIL string, not a cookie 401/302 (ConfigServe Unauthenticated collision makes status-only assertions vacuous)",
|
||||
"status": "pending",
|
||||
"status": "done",
|
||||
"blockedBy": [
|
||||
"T4"
|
||||
]
|
||||
@@ -43,7 +43,7 @@
|
||||
{
|
||||
"id": "T6",
|
||||
"subject": "MeshGrpcDriverOnlyPipelineTests: driver-only nodes carry no authorization middleware — pins the 03/S14 exemption",
|
||||
"status": "pending",
|
||||
"status": "done",
|
||||
"blockedBy": [
|
||||
"T3"
|
||||
]
|
||||
@@ -51,7 +51,7 @@
|
||||
{
|
||||
"id": "T7",
|
||||
"subject": "MeshGrpcMapConventionTests: Program.cs source scan — zero direct MapGrpcService/MapZbLocalDbSync sites, exactly one mapper call",
|
||||
"status": "pending",
|
||||
"status": "done",
|
||||
"blockedBy": [
|
||||
"T1"
|
||||
]
|
||||
@@ -59,13 +59,13 @@
|
||||
{
|
||||
"id": "T8",
|
||||
"subject": "docker-dev: Security__Auth__DisableLogin parameterised as ${OTOPCUA_DISABLE_LOGIN:-true} on central-1/central-2 + header docs",
|
||||
"status": "pending",
|
||||
"status": "done",
|
||||
"blockedBy": []
|
||||
},
|
||||
{
|
||||
"id": "T9",
|
||||
"subject": "docs/security.md \"Mesh gRPC ingress is AllowAnonymous by design\" + cross-refs in docs/Configuration.md and docs/Telemetry.md",
|
||||
"status": "pending",
|
||||
"status": "done",
|
||||
"blockedBy": []
|
||||
},
|
||||
{
|
||||
|
||||
@@ -452,3 +452,321 @@ T8 anytime; T9 last (after T3/T5/T6/T7 merge-ready)
|
||||
```
|
||||
|
||||
Overall effort: **~0.75–1 dev-day code+unit, plus the rig session.** The one HIGH-RISK production slice (T3, which changes what every child receives at spawn) lands behind the full 25-suite `DriverHostActor` regression before any new test stacks on it, and behind T2's explicitly behaviour-preserving refactor.
|
||||
|
||||
---
|
||||
|
||||
## Execution notes
|
||||
|
||||
Branch `fix/r3-03-restart-resubscribe-watchdog`, worktree
|
||||
`.claude/worktrees/agent-a5bcdd61bfba22e10`, off master `5fd65d19`. TDD throughout: every
|
||||
behaviour task's test was written first, RUN, and observed red for the predicted reason before the
|
||||
production edit landed. Red observations are recorded per task below.
|
||||
|
||||
### T1 — 03/S17 watchdog grace (`??` → max)
|
||||
|
||||
**RED observed (as predicted).** With the two new tests in place against unmodified
|
||||
`DriverInstanceActor`:
|
||||
|
||||
```
|
||||
DriverInstanceActorCommsLossWatchdogTests.A_reconnected_driver_gets_a_full_grace_window_before_the_watchdog_trips_again [FAIL]
|
||||
Failed: Expected no messages during 00:00:00.4000000, instead we received
|
||||
<ConnectivityChanged { DriverInstanceId = stub-driver-1, Connected = False }> after 00:00:00.0496590
|
||||
Failed! - Failed: 1, Passed: 9, Total: 10
|
||||
```
|
||||
|
||||
The re-trip landed **49.7 ms** after the reconnect — one health tick (`FastPoll` = 50 ms) — against a
|
||||
`Grace` of 1500 ms, i.e. exactly the "churn at health-tick cadence instead of the designed grace"
|
||||
the finding describes. `A_half_broken_device_is_still_declared_lost_once_the_grace_elapses` passed
|
||||
in the same run, which is what makes it the anti-vacuity control: it cannot distinguish the fix from
|
||||
the bug, so the delay assertion is the one carrying the finding.
|
||||
|
||||
**GREEN after the one-line max at `:574`** (plus the "grace is per connection" remarks paragraph and
|
||||
the reciprocal falsification-partner xmldoc on the existing
|
||||
`A_degraded_driver_that_is_still_reading_successfully_is_left_alone`): 10/10 passed, 20 s.
|
||||
`A_reconnect_attempt_tears_the_dead_transport_down_before_re_initialising` stayed green — as the plan
|
||||
anticipated, each of its trip cycles now costs a full `FastTimeout` (250 ms) instead of one 50 ms
|
||||
tick, and its `AwaitCondition(InitializeCount >= 2, 5 s)` still has ample headroom. No budget change
|
||||
was needed.
|
||||
|
||||
### T2 — desired-subscription payload authority
|
||||
|
||||
Behaviour-preserving refactor, so there is no red observation to record by design — the gate is that
|
||||
**no test moves**. `--filter "FullyQualifiedName~DriverHostActor"` → **114/114 passed**, unchanged
|
||||
from the base commit.
|
||||
|
||||
⚠️ One run of that filter reported `DriverHostActorNativeAlarmAckRoutingTests.RouteNativeAlarmAck_on_non_primary_is_dropped`
|
||||
failing on an `AwaitAssert`. Re-run in isolation: **3/3 passed**; re-run of the whole 114-test filter:
|
||||
**114/114 passed**. Treated as the known #500 flake set for this assembly (~13 %/run), not a
|
||||
regression — it touches none of the code this task edits (the alarm-ack Primary gate is a different
|
||||
method entirely).
|
||||
|
||||
Deviation from the plan's literal step text (cosmetic, recorded for the reviewer): the plan's Finding-1
|
||||
step 1 says `_desiredByDriver` *replaces* `_lastComposition` at `:347-349` while T3 step 2 says T3
|
||||
deletes `_lastComposition`. Keeping T2 compilable means the two fields coexist for one commit;
|
||||
`_lastComposition` and its assignment are deleted in T3 as T3's own step text specifies.
|
||||
|
||||
### T3 — `SpawnChild` always sends the desired set; `_lastComposition` deleted
|
||||
|
||||
The falsification for this production slice is **T5**, written and observed red afterwards (below) —
|
||||
T3 lands the fix first because the plan sequences it that way and the ordering guard needs the whole
|
||||
25-suite regression before any new test is trusted on top of it.
|
||||
|
||||
**HIGH-RISK regression gate (every child's spawn-time state changes, deploy path included):** the
|
||||
FULL `Runtime.Tests` project, not just the `DriverHostActor` filter —
|
||||
**639 passed, 3 skipped, 0 failed, 55 s**. The 3 skips are pre-existing (`DeploymentArtifact*` parity
|
||||
tests, skipped on the base commit too).
|
||||
|
||||
`_lastComposition` is gone: `grep -c _lastComposition DriverHostActor.cs` → **0**. Its xmldoc had also
|
||||
drifted (it claimed the field was set at the end of `PushDesiredSubscriptions`; the assignment was in
|
||||
`ApplyDesiredSubscriptions`) — deleted with the field rather than corrected. The stale
|
||||
"dropped-driver fallback" comment on the old `SendAuthoredOnly` local function was rewritten as the
|
||||
`SendDesiredSubscriptions` remarks naming the three real consumers.
|
||||
|
||||
### T4 — test harness (seeder `enabled` flag + recording driver factory)
|
||||
|
||||
Test-only, additive, source-compatible: every pre-existing `SeedV3Deployment` / `SpawnHostAndApply`
|
||||
call site compiles and behaves identically (the new parameters are trailing and default to today's
|
||||
values). Gate: `--filter "…~DriverHostActor|…~DriverInstanceActor"` → **179/179 passed** (the plan
|
||||
named `--filter "…~V3"`, which matches nothing — the seeder's four consumers are all `DriverHostActor*`
|
||||
classes; the wider filter is the same set plus more).
|
||||
|
||||
**Deviation (recorded).** The plan lists one new seeder parameter (`enabled`); a second was needed and
|
||||
added: `IReadOnlyDictionary<string, string>? tagConfigByTag = null`. The seeder hard-codes
|
||||
`TagConfig = "{}"` on every tag, and a raw tag's alarm is derived *only* from `TagConfig.alarm` via
|
||||
`TagConfigIntent.Parse` (`AddressSpaceComposer.cs:769-772`) — so without it there is no way to seed the
|
||||
alarm-bearing raw tag that `Operator_restart_re_subscribes_alarm_refs_too` (a test the plan itself
|
||||
specifies) requires. It is trailing and optional, so it is the same source-compatible shape as
|
||||
`enabled`.
|
||||
|
||||
### T5 — `DriverHostActorRestartTests` (operator-restart falsification, 03/U6's first half)
|
||||
|
||||
**RED observed against a T3-reverted build** (the `SendDesiredSubscriptions(child, …)` call in
|
||||
`SpawnChild` commented out, everything else on the branch): **all 4 FAILED, 0 passed**, each at the
|
||||
bar the plan predicts —
|
||||
|
||||
```
|
||||
Operator_restart_re_subscribes_the_respawned_child [FAIL]
|
||||
factory.Created[1].SubscribeCount should be greater than or equal to 1 but was 0
|
||||
Operator_restart_re_subscribes_alarm_refs_too [FAIL]
|
||||
factory.Created[1].SubscribeAlarmsCount should be greater than or equal to 1 but was 0
|
||||
Restart_of_a_driver_with_an_empty_value_set_still_sends_the_set [FAIL]
|
||||
respawned.SubscribeAlarmsCount should be greater than or equal to 1 but was 0
|
||||
A_child_spawned_by_a_fresh_deploy_ends_with_the_NEW_refs [FAIL]
|
||||
factory.Created[2].LastSubscribedRefs should be ["Plant/Modbus/dev1/temp"] but was null
|
||||
```
|
||||
|
||||
The call was then restored; **4/4 pass**, and the class was run **20× in isolation, 20/20 green** (#500
|
||||
discipline for this assembly).
|
||||
|
||||
**Two deviations, both found by running the tests rather than by reading:**
|
||||
|
||||
1. ⚠️ **The plan's claim that the transient spawn-time set "is never acted on" is FALSE, and the
|
||||
production comment that repeated it has been corrected.** The plan reasons that the child is still
|
||||
`Connecting` when the stale set arrives, so it is only stored. But the child runs **concurrently**
|
||||
with the rest of `HandleArtifactForApply` — `Context.ActorOf` returns and the child starts
|
||||
processing while the host is still building `refsByDriver` — so a driver whose `InitializeAsync`
|
||||
completes instantly reaches Connected and subscribes the stale set *before* the bulk push arrives.
|
||||
Measured directly: `SubscribeCount == 2` on the respawned child, first with revision A's ref and
|
||||
then with B's. What Akka's per-pair ordering does guarantee is **convergence**, and
|
||||
`HandleSubscribeAsync` drops the prior subscription before establishing the new one, so the cost is
|
||||
one extra subscribe round-trip on a *respawning* deploy — a real device driver, whose initialise
|
||||
opens a socket, rarely wins that race. Accepted rather than "fixed": withholding the spawn-time send
|
||||
is the 03/S16 defect, and the operator-Restart and supervision paths have no bulk push coming at
|
||||
all. The `SpawnChild` comment now states this instead of the plan's claim, and the test asserts
|
||||
convergence (the plan's own stated acceptance — "the child's live subscription carries the NEW
|
||||
refs") rather than pinning the race.
|
||||
2. **`A_child_spawned_by_a_fresh_deploy_ends_with_the_NEW_refs` had to be re-shaped.** The plan
|
||||
describes revision B as one "which respawns nothing but re-pushes". It respawns: the raw tags are
|
||||
folded into the merged `DriverConfig`, so a tag-set edit changes the `DriverInstanceSpec` and trips
|
||||
the #516 config respawn. That is the *stronger* arrangement for this test's subject (a brand-new
|
||||
child spawned while the authority still holds A), so the test keeps the same name and asserts the
|
||||
respawned child converges on B — plus a subsequent operator Restart, which pins that the authority
|
||||
is repopulated **whole** per apply rather than accumulated.
|
||||
3. **`Restart_of_a_driver_with_no_authored_tags_sends_an_empty_set_not_nothing` was renamed
|
||||
`Restart_of_a_driver_with_an_empty_value_set_still_sends_the_set`, with a real positive control.**
|
||||
The seeder derives `DriverInstances` from the raw-tag rows, so a driver with *zero* tags is not
|
||||
expressible, and — more importantly — an all-empty set has **no driver-side observable at all**
|
||||
(the child stores it and, having no handle, does nothing), which would have made the test vacuous:
|
||||
"it did not subscribe" passes equally against a build that sends the respawned child nothing, i.e.
|
||||
against the defect itself. The subject is therefore a driver whose only raw tag is alarm-bearing:
|
||||
its value half is genuinely empty while its alarm half is not, and **both halves ride in one
|
||||
message** — so `SubscribeAlarmsCount >= 1` proves the message ARRIVED and `SubscribeCount == 0`
|
||||
then proves the empty value half was honoured. The falsification run above confirms the control
|
||||
works: with T3 reverted this test fails on the *positive control*, not on the absence.
|
||||
**Not covered, and worth a reviewer's eye:** the `_desiredByDriver` union's `_children.Keys` arm
|
||||
(a running child absent from BOTH ref maps) is not reachable through this seeder and is untested.
|
||||
|
||||
### T6 — supervision reprovision hook
|
||||
|
||||
`DriverInstanceActor.PostRestart` (calling `base.PostRestart(reason)` first, so `PreStart` re-attaches
|
||||
the rediscovery / host-status sources and restarts the health timer) → `ReprovisionRequested` →
|
||||
`DriverHostActor.HandleReprovisionRequested`, registered in **all three** `Receive` blocks that already
|
||||
accept `RestartDriver`. The falsification is T7, below.
|
||||
|
||||
Gate: `--filter "…~DriverHostActor|…~DriverInstanceActor"` → **183/183 passed** (179 before this task
|
||||
plus T5's four), no test moved.
|
||||
|
||||
### T7 — first supervision-restart test + the force-reconnect verified negative (03/U6)
|
||||
|
||||
**Crash seam that landed: the planned one** — `RecordingSubscribableDriver.FireMalformedDataChange`
|
||||
raises `OnDataChange` with a null `Snapshot`; `DriverInstanceActor.OnDataChangeForward` dereferences
|
||||
`msg.Snapshot.SourceTimestampUtc` on the actor thread and NREs; Akka's default decider Restarts. No
|
||||
fallback (`OneForOneStrategy` + an internal test message) was needed, so no production surface was
|
||||
added for the test.
|
||||
|
||||
**RED observed against a T6-reverted build** (the `Context.Parent.Tell(new ReprovisionRequested(…))`
|
||||
in `PostRestart` commented out):
|
||||
|
||||
```
|
||||
Supervision_restart_reinitialises_and_re_subscribes_the_child [FAIL]
|
||||
driver.InitializeCount should be 2 but was 1
|
||||
"the restarted child was never re-initialised — it is sitting in Connecting around a driver
|
||||
its own PostStop already shut down (03/S6, the worse half of 03/S16)"
|
||||
```
|
||||
|
||||
**Deviation, and the reason for it — the first red run was AMBIGUOUS and the test was strengthened.**
|
||||
`InitializeCount == 1` is equally consistent with "the child restarted and was not re-provisioned"
|
||||
and with "the child never crashed at all", so a seam that silently stopped working would read as a
|
||||
passing regression test forever. A preceding `AwaitAssert(ShutdownCount >= 1)` was added — `PreRestart`
|
||||
runs `PostStop`, which calls `ShutdownAsync`, so that is the evidence a restart genuinely happened.
|
||||
On the re-run of the falsification that assertion **passed** and the failure landed squarely on the
|
||||
re-initialise, which is what makes the red observation above meaningful.
|
||||
|
||||
`Force_reconnect_keeps_the_desired_set_and_re_subscribes` passed in **both** the red and green runs —
|
||||
correct, and the point: it is the verified negative (`ForceReconnect` reaches the same live actor, so
|
||||
`_desiredRefs` survive and the reconnect's Connected entry re-applies them). It carries a 30 s budget
|
||||
rather than `PresenceBudget`, because `SpawnChild` gives the child the production
|
||||
`DefaultReconnectInterval` (10 s) and the retry is a periodic timer, so the reconnect cannot land
|
||||
sooner; a budget is only spent on failure.
|
||||
|
||||
After restoring the hook: **6/6 pass**, and the class was run **20× in isolation, 20/20 green**.
|
||||
|
||||
### T8 — docs
|
||||
|
||||
`docs/DriverLifecycle.md`:
|
||||
|
||||
- New §"What a (re)spawn re-provisions" (placed before §IDriverHealthPublisher): `_desiredByDriver` as
|
||||
the single authority, a table of the three incarnation routes plus the verified-negative Force
|
||||
Reconnect, the supervision route's invisibility, why replacing the default decider is out of scope,
|
||||
the respawning-deploy ordering **with the measured caveat that the transient is not inert**, and the
|
||||
assert-a-function-not-existence rule naming the weaker existing test.
|
||||
- §"The comms-loss watchdog": conjunct 2 restated as the max, plus a new paragraph on the
|
||||
per-connection grace — why `??` was wrong, that it **paces** rather than ends the half-broken-device
|
||||
cycle, that the watchdog is not disabled by the change, and the two tests that pin each half.
|
||||
|
||||
### Final verification
|
||||
|
||||
- `dotnet build ZB.MOM.WW.OtOpcUa.slnx` → **0 errors** (904 warnings, all pre-existing analyzer
|
||||
`OTOPCUA0001` noise in unrelated driver test projects).
|
||||
- **Full `Runtime.Tests`: 645 passed, 3 skipped, 0 failed** (1 m 9 s). Baseline on the branch point was
|
||||
637 + 3 skipped; the +8 are this plan's new tests (T1 ×2, T5 ×4, T7 ×2). The 3 skips are pre-existing
|
||||
`DeploymentArtifact*` parity tests.
|
||||
- The two other test projects that reference `ZB.MOM.WW.OtOpcUa.Runtime` were run as well, since T3
|
||||
changes what every driver child receives at spawn: `ControlPlane.Tests` **262/262**, `Host.Tests`
|
||||
**106/106**.
|
||||
- T9 (live gate) deliberately left **pending** — run by the orchestrator.
|
||||
|
||||
### Summary of deviations (for the reviewer)
|
||||
|
||||
1. **The plan's "the transient spawn-time set is never acted on" is false** — measured, corrected in
|
||||
both the production comment and the docs, and accepted with its cost stated (T5, deviation 1). This
|
||||
is the one behavioural claim in the plan that did not survive contact with the code.
|
||||
2. `A_child_spawned_by_a_fresh_deploy_ends_with_the_NEW_refs` re-shaped — revision B *does* respawn the
|
||||
child (raw tags fold into the merged `DriverConfig`) (T5, deviation 2).
|
||||
3. `Restart_of_a_driver_with_no_authored_tags_sends_an_empty_set_not_nothing` renamed +
|
||||
re-based on an alarm-only driver, because an all-empty set has no driver-side observable and the
|
||||
planned test would have been vacuous; the `_children.Keys` union arm remains untested (T5,
|
||||
deviation 3).
|
||||
4. Seeder gained a second optional parameter (`tagConfigByTag`) — the only way to seed an alarm-bearing
|
||||
raw tag, which one of the plan's own tests needs (T4).
|
||||
5. The supervision test gained a preceding `ShutdownCount >= 1` assertion, because the plan's red
|
||||
observation was ambiguous with "the child never crashed" (T7).
|
||||
6. `Force_reconnect_keeps_the_desired_set_and_re_subscribes` uses a 30 s budget, not `PresenceBudget` —
|
||||
`SpawnChild` gives the child the production 10 s `DefaultReconnectInterval` (T7).
|
||||
7. `_lastComposition` and `_desiredByDriver` coexist for one commit (T2) so T2 stays compilable; the
|
||||
deletion lands in T3, as T3's own step text specifies.
|
||||
|
||||
---
|
||||
|
||||
## Review outcome
|
||||
|
||||
Verdict **FIX-FIRST**, 3 items. All three confirmed against the code, fixed TDD (test first, seen red
|
||||
for the predicted reason, then fixed), one commit each.
|
||||
|
||||
### F1 [HIGH regression] — a changed alarm set latched (`edd01fcc`)
|
||||
|
||||
**Confirmed.** `SubscribeDesiredAlarms` and `HandleSubscribeAlarmsAsync` both gated on
|
||||
`_alarmSubscriptionHandle is not null`, which is the right guard for an idempotent re-push and the wrong
|
||||
one for a deploy that changes the set. The value side converges by construction
|
||||
(`StoreDesiredSubscriptions` is a full replace; `HandleSubscribeAsync` drops the prior subscription
|
||||
first) — the alarm side did not. **T3 is what made it reachable**: a tag-set edit forces a respawn, and
|
||||
a fast-initialising driver subscribes the transient spawn-time set before the bulk push, so the
|
||||
PREVIOUS revision's alarm refs latched for the connection's life. `sourceNodeIds`-filtering drivers
|
||||
(OPC UA Client, AB CIP, FOCAS) would silently never deliver the newly-added alarm.
|
||||
|
||||
**RED:** `A_deploy_that_widens_the_alarm_set_re_subscribes_the_new_refs` — after deploying
|
||||
alarm `{alarmX}` then `{alarmX, alarmY}`, `LastSubscribedAlarmRefs` was still `["Plant/Modbus/dev1/alarmX"]`.
|
||||
|
||||
**Fix:** new `_subscribedAlarmRefs` records what the live handle was established with;
|
||||
`AlarmSubscriptionIsStale()` (no handle **or** the desired refs differ) replaces the bare null check in
|
||||
both places; a stale handle is **released before** re-subscribing so a filtering driver cannot
|
||||
accumulate two overlapping subscriptions. The release clears the handle *first*, so a throwing
|
||||
unsubscribe cannot leave the staleness check treating a dead handle as live, and it is best-effort
|
||||
(logged, non-fatal — the re-subscribe is what matters). Unsubscribe-on-**empty** is still deliberately
|
||||
absent, unchanged. Two notes: the release had to be wrapped in `_invoker.ExecuteAsync` or the repo's
|
||||
`OTOPCUA0001` analyzer fails the build; and the Connected-handler comment that stated the
|
||||
value-only convergence claim was corrected, as was the matching bullet in `docs/DriverLifecycle.md`.
|
||||
|
||||
### F2 [MEDIUM] — reprovision routed by id, not incarnation (`2b7100cf`)
|
||||
|
||||
**Confirmed.** Fixed with `if (!Equals(Sender, entry.Actor)) return;` (plus a Debug line naming the
|
||||
superseded ref).
|
||||
|
||||
⚠️ **The first version of the test PASSED against the unfixed code — the arrangement mattered.** With
|
||||
the child already **Connected**, `InitializeRequested` is unhandled, so the stale reprovision was
|
||||
swallowed for the wrong reason and the test was vacuous. The defect only bites a child in
|
||||
`Connecting`, which is exactly where `InitializeRequested` *is* handled. `RecordingSubscribableDriver`
|
||||
therefore gained an `InitializeGate` (a `TaskCompletionSource` its `InitializeAsync` awaits) so the
|
||||
test can hold the child in `Connecting`. **RED with that arrangement:** `InitializeCount == 2` — the
|
||||
second concurrent `InitializeAsync` racing the #527 dispose-before-overwrite. The absence assertion
|
||||
carries a positive control (the child is proven alive at `InitializeCount == 1` first) and a
|
||||
`GetDiagnostics` round-trip so the dropped message is provably processed before the absence is read.
|
||||
|
||||
### F3 [IMPORTANT] — no backoff on the reprovision (`0793b0cb`)
|
||||
|
||||
**Confirmed.** Added `DefaultReprovisionBackoff` (10 s, matched to
|
||||
`DriverInstanceActor.DefaultReconnectInterval` because the reprovision drives the same connect work),
|
||||
a per-driver `_lastReprovisionUtc` stamp, and a `reprovisionBackoff` Props parameter following the
|
||||
existing `applyWedgeTimeout` precedent so a test need not wait the production window.
|
||||
|
||||
A held request is **deferred, not dropped**: one `StartSingleTimer` per driver re-delivers a
|
||||
`ReprovisionDeferred` carrying the **original incarnation ref**, so F2's identity check still applies
|
||||
on re-delivery, and a burst collapses to a single pending reprovision rather than a queue. Each
|
||||
deferral logs a Warning naming the crash type, so a crash loop is visible rather than quiet.
|
||||
|
||||
**RED** (backoff disabled): `InitializeCount == 3` where the test requires 2-then-3 — i.e. the second
|
||||
reprovision applied immediately. The test drives **two genuine supervision restarts** (not synthetic
|
||||
messages), waits for the re-subscribe before arming the second crash, and asserts `ShutdownCount >= 2`
|
||||
so "the second crash tripped supervision" is proven before the backoff is read.
|
||||
|
||||
### Optional adds (both done)
|
||||
|
||||
- `A_running_driver_with_no_authored_tags_is_handed_an_empty_set` — the `_children.Keys` union arm,
|
||||
previously untested and called out as a gap in the T5 notes. It needed a driver with no tags at all,
|
||||
which the seeder could not express, so `SeedV3Deployment` gained `taglessDrivers`. The tagless child
|
||||
is asserted initialised-but-never-subscribed, with its tag-owning sibling's subscribe as the positive
|
||||
control that the push pass ran.
|
||||
- `_desiredByDriver` xmldoc now records that it is rebuilt only by a **successful** apply, so it
|
||||
lingers on the last-known-good composition after a failed one — consistent with the address space,
|
||||
drivers and subscriptions that also keep serving (#485/#486).
|
||||
|
||||
### Verification after the three fixes
|
||||
|
||||
- `dotnet build ZB.MOM.WW.OtOpcUa.slnx` → **0 errors**.
|
||||
- **Full `Runtime.Tests`: 649 passed, 3 skipped, 0 failed** (up 4 from 645 — F1, F2, F3 and the
|
||||
union-arm test).
|
||||
- **`ControlPlane.Tests`: 262/262.**
|
||||
- `DriverHostActorRestartTests` (now 10 cases) run **10× in isolation, 10/10 green**.
|
||||
- T9 (live gate) still **pending** — the orchestrator runs it. Worth noting for that session: F1 is
|
||||
directly observable on the rig, by deploying a second alarm-bearing raw tag to a driver that already
|
||||
has one and confirming a transition on the NEW tag reaches `/alerts`.
|
||||
|
||||
@@ -5,19 +5,19 @@
|
||||
{
|
||||
"id": "T1",
|
||||
"subject": "Comms-loss grace is per connection: ?? -> max(LastSuccessfulRead, _connectedSinceUtc) + falsification pair (03/S17)",
|
||||
"status": "pending",
|
||||
"status": "done",
|
||||
"blockedBy": []
|
||||
},
|
||||
{
|
||||
"id": "T2",
|
||||
"subject": "Desired-subscription payload authority: _desiredByDriver + SendDesiredSubscriptions (behaviour-preserving refactor)",
|
||||
"status": "pending",
|
||||
"status": "done",
|
||||
"blockedBy": []
|
||||
},
|
||||
{
|
||||
"id": "T3",
|
||||
"subject": "SpawnChild always sends the desired set; delete the write-only _lastComposition (03/S16)",
|
||||
"status": "pending",
|
||||
"status": "done",
|
||||
"blockedBy": [
|
||||
"T2"
|
||||
]
|
||||
@@ -25,13 +25,13 @@
|
||||
{
|
||||
"id": "T4",
|
||||
"subject": "Test harness: V3ArtifactSeeder enabled flag + RecordingSubscribableDriver(Factory) with a crash seam",
|
||||
"status": "pending",
|
||||
"status": "done",
|
||||
"blockedBy": []
|
||||
},
|
||||
{
|
||||
"id": "T5",
|
||||
"subject": "DriverHostActorRestartTests: operator restart re-subscribes the respawned child (falsifiable; 03/S16, U6)",
|
||||
"status": "pending",
|
||||
"status": "done",
|
||||
"blockedBy": [
|
||||
"T3",
|
||||
"T4"
|
||||
@@ -40,7 +40,7 @@
|
||||
{
|
||||
"id": "T6",
|
||||
"subject": "Supervision reprovision hook: DriverInstanceActor.PostRestart -> ReprovisionRequested -> host re-init + re-subscribe (03/S6, S16)",
|
||||
"status": "pending",
|
||||
"status": "done",
|
||||
"blockedBy": [
|
||||
"T3"
|
||||
]
|
||||
@@ -48,7 +48,7 @@
|
||||
{
|
||||
"id": "T7",
|
||||
"subject": "First real supervision-restart test + force-reconnect verified-negative regression (03/U6)",
|
||||
"status": "pending",
|
||||
"status": "done",
|
||||
"blockedBy": [
|
||||
"T4",
|
||||
"T6"
|
||||
@@ -57,7 +57,7 @@
|
||||
{
|
||||
"id": "T8",
|
||||
"subject": "docs/DriverLifecycle.md: what a (re)spawn re-provisions + per-connection watchdog grace",
|
||||
"status": "pending",
|
||||
"status": "done",
|
||||
"blockedBy": []
|
||||
},
|
||||
{
|
||||
|
||||
@@ -578,3 +578,242 @@ T10 anytime after T5/T8; T11 last (after T5 + T9 merge-ready); T12 anytime
|
||||
```
|
||||
|
||||
Overall effort: **~1 dev-day code+unit, plus the rig session.** The two HIGH-RISK slices are T2/T3 (authentication path — every new branch must land on deny) and T11 (the gate that proves the clock axis actually reaches a real client). T9 is the largest single task and is the only one that can falsify T8 end to end.
|
||||
|
||||
---
|
||||
|
||||
## Execution notes
|
||||
|
||||
Branch `fix/r3-05-acl-hot-path-expiry`, executed 2026-08-20 off master `dd7f4954`. TDD per task:
|
||||
failing test first, red observed and recorded, then green. T11 (live gate) deliberately left `pending`
|
||||
for the orchestrator.
|
||||
|
||||
### T1 — TimeProvider registration + option knobs
|
||||
|
||||
- **RED observed:** `error CS1061: 'ServiceCollection' does not contain a definition for
|
||||
'AddOtOpcUaDataPlaneAuth'` and `'DataPlaneAuthorizer' does not contain a definition for 'Clock'` —
|
||||
the seam and the assertion surface both absent, as predicted.
|
||||
- **DEVIATION (structural, required by the plan's own test):** the plan says the guard test "builds the
|
||||
service collection for an admin node". `Program.cs` is **top-level statements in an executable**, so no
|
||||
test can drive its wiring. The `DataPlaneAuthorizer` registration and the three-branch key-store /
|
||||
authenticator conditional were therefore **moved verbatim** into a new
|
||||
`Host/Configuration/DataPlaneAuthRegistration.AddOtOpcUaDataPlaneAuth(hasAdmin, hasDriver)` — the same
|
||||
shape `LocalDbRegistration` already uses — and `Program.cs` now calls it. No branch changed.
|
||||
- **DEVIATION (small):** `DataPlaneAuthorizer` gained a public `Clock` property. The two-clock hazard the
|
||||
class already documents can only be pinned by reference, and `_time` was private.
|
||||
- **Collateral:** `MeshRegistrationRoleScopingTests.HasAdminMarkerScan_MatchesTheKnownShapeOfProgramCs`
|
||||
pins the `if (hasAdmin)` marker count in `Program.cs`; the move took one braced marker with it, so the
|
||||
pin went 6 → 5 (the test's own message asks for exactly that when a marker moves).
|
||||
|
||||
### T2 — `BoundedApiKeyStore` deadline
|
||||
|
||||
- **RED observed:** `error CS0246: The type or namespace name 'BoundedApiKeyStore' could not be found` —
|
||||
type absent, as predicted.
|
||||
- **DEVIATION (API):** the plan's snippet calls `cts.CancelAfter(TimeSpan, TimeProvider)`. **No such
|
||||
overload exists** (`error CS1501: No overload for method 'CancelAfter' takes 2 arguments`). The
|
||||
clock-aware equivalent is the `CancellationTokenSource(TimeSpan, TimeProvider)` **constructor**, linked
|
||||
with the caller's token — same behaviour, same injected clock, and the linked source still lets the
|
||||
`when (!ct.IsCancellationRequested)` filter tell a deadline from a caller cancellation.
|
||||
- Added beyond the plan's list: `A_zero_timeout_disables_the_deadline` (the documented rollback, and the
|
||||
in-class control proving the *bound* is what denies rather than the fake) and
|
||||
`Anonymous_grant_resolution_is_bounded_too` (moved forward from the plan's T3 list — the deadline half
|
||||
reaches `FindGrantsAsync` in the same change, so it is asserted in the same change).
|
||||
- 5 tests, green.
|
||||
|
||||
### T3 — TTL read-through cache + the falsification control
|
||||
|
||||
- **RED observed:** `error CS0117: 'BoundedApiKeyStore' does not contain a definition for
|
||||
'MaxCacheEntries'` — the cache and its ceiling absent.
|
||||
- **DEVIATION (fail-closed refinement, 2 lines):** the plan keys `FindAsync`'s cache on `apiKeyId` alone.
|
||||
Taken literally that lets the cache **resurrect a key that expired by the clock** for up to a TTL, which
|
||||
is exactly the axis 03/S19 exists to close. A cached hit is therefore re-checked against the caller's
|
||||
`utcNow` before it is served (`IsExpired`) — so the TTL bounds how stale an `Enabled` **revocation** may
|
||||
be, and nothing else. `utcNow` is still not part of the key (it changes every call and would defeat the
|
||||
cache entirely).
|
||||
- The counter's `cache_hit` / `timeout` `Add` calls are deliberately NOT here — they are T4, and writing
|
||||
them early would have made T4's red vacuous.
|
||||
- 11 tests in the class, green.
|
||||
|
||||
### T4 — `otopcua.opcua.apikey_store.bounded` counter
|
||||
|
||||
- **RED observed (behavioural, not compile):** `A_timeout_meters_outcome_timeout` and
|
||||
`A_cache_hit_meters_outcome_cache_hit` both `[FAIL]` with `should be` — the instrument existed nowhere,
|
||||
so the listener saw nothing. `A_plain_read_through_meters_neither` passed vacuously at red, by design:
|
||||
its job starts once the counter exists.
|
||||
- **NOTE on the metrics-leak memory:** the recorder filters on **instrument name only**. That is normally
|
||||
the leak (a parallel test's measurements land in it), and it is safe here for a checkable reason
|
||||
recorded on the helper: `BoundedApiKeyStore` is the only writer of this instrument, only this class
|
||||
constructs one, and xUnit never runs two tests of one class in parallel. Adding a second class that
|
||||
exercises the decorator means adding a discriminating tag.
|
||||
- 14 tests in the class, green.
|
||||
|
||||
### T5 — wire the decorator + registration guard
|
||||
|
||||
- **RED observed:** `error CS1061: 'BoundedApiKeyStore' does not contain a definition for 'Inner'` /
|
||||
`… for 'Clock'` — the guard could not see inside the wrapper because nothing was wrapping yet.
|
||||
- **DEVIATION (small):** `BoundedApiKeyStore` gained public `Inner` and `Clock` properties, for the same
|
||||
reason `DataPlaneAuthorizer.Clock` did in T1 — a guard that cannot see what is inside the wrapper cannot
|
||||
tell a correctly-wired node from one wrapping the wrong store.
|
||||
- Doc-drift half of the finding landed here: the authenticator's "no network I/O on this path at all"
|
||||
remark is now qualified by node role and points at the bound, and `ConfigDbApiKeyStore` records both
|
||||
that it is wrapped in production and that "revocation on the next request" is **activation**-scoped.
|
||||
- Host.Tests: 123 green (was 106 at base).
|
||||
|
||||
### T6 — `AclRevalidationTimer`
|
||||
|
||||
- **RED observed:** `error CS0246: The type or namespace name 'AclRevalidationTimer' could not be found`.
|
||||
- **DEVIATION (visibility):** the plan declares the class `internal sealed`. `ZB.MOM.WW.OtOpcUa.Host` has
|
||||
**no `InternalsVisibleTo`**, so the tests could not reach it. Made `public sealed`, matching
|
||||
`BoundedApiKeyStore` and every other type in `Host/OpcUa/`.
|
||||
- Added beyond the plan's list: `A_tick_runs_the_pass` — the control for the two "nothing happened" cases
|
||||
(a `Tick()` that silently did nothing would satisfy the throwing-pass and dispose cases on its own).
|
||||
- 5 tests, green. Also corrected the T4 `MeterRecorder` remark: the assembly declares
|
||||
`CollectionBehavior(DisableTestParallelization = true)`, which is a stronger and simpler reason the
|
||||
name-only filter is safe than the one first written.
|
||||
|
||||
### T7 — node-manager expiry tests + xmldoc correction
|
||||
|
||||
- **RED: the three new cases PASSED on first run — as the plan predicted** ("the expiry case may already
|
||||
PASS … that is the point: it pins the property the timer will rely on before the timer exists"). The
|
||||
pass already reads the authorizer's `TimeProvider`, so no production change was needed.
|
||||
- **Falsification run (recorded, since "it passed immediately" proves nothing on its own):** shortening the
|
||||
clock advance in `An_expired_key_stands_down_a_live_item_with_no_deploy` from 20 min to 5 min (i.e.
|
||||
short of `ExpiresUtc`) turned it **`[FAIL]`**, then restored. So the assertion tracks the clock rather
|
||||
than a blanket deny.
|
||||
- `A_renewed_key_re_attaches_the_item_on_the_next_pass` observes the restore **indirectly and
|
||||
falsifiably**: the pass skips an already-suppressed item, so standing the item down a second time can
|
||||
only succeed if it was genuinely re-attached in between. (`NodeHandle.MonitoredNode` is null on a
|
||||
hand-tracked stub item, so attach/detach is not directly observable at this seam — it is proven over the
|
||||
wire in T9.)
|
||||
- Two stale remarks corrected, not one: `OnAuthorityChanged`'s "this is the only trigger" **and**
|
||||
`IsSubscriptionStillPermitted`'s "a subscription is re-checked at a deployment, never on a timer".
|
||||
- OpcUaServer.Tests: 493 passed / 1 skipped.
|
||||
|
||||
### T8 — wire the timer into `OtOpcUaServerHostedService`
|
||||
|
||||
- **RED observed:** `error CS1061: 'OtOpcUaServerHostedService' does not contain a definition for
|
||||
'StartRevalidationTimer'` and `error CS1729: … does not contain a constructor that takes 13 arguments`
|
||||
(the two new ctor parameters).
|
||||
- **DEVIATION (test seam, and the reason for it):** the plan says "extend the hosted-service tests". **There
|
||||
are none — `grep -rln OtOpcUaServerHostedService tests/` matched nothing**, because `StartAsync` boots
|
||||
the real OPC UA SDK (PKI store, listening socket). A new suite,
|
||||
`OtOpcUaServerHostedServiceRevalidationTests`, drives the timer's lifecycle through an **`internal`**
|
||||
`StartRevalidationTimer()` (new `InternalsVisibleTo` on the Host csproj, the convention six driver
|
||||
projects already use). Everything the timer's lifecycle touches — the options, the clock, the disposal —
|
||||
is exercised exactly as production reaches it; only the SDK boot is skipped.
|
||||
- **DEVIATION (assertion not made):** the plan asks the test to assert disposal happens **before**
|
||||
`SetDataPlaneAuthorizer(null)`. That is **not observable**: both detaches go through `_server?.…`, and
|
||||
`OtOpcUaSdkServer.NodeManager` is created only by the SDK's own `CreateMasterNodeManager` during a real
|
||||
boot, so with no server both calls are no-ops. The ordering is instead guaranteed structurally — the
|
||||
dispose is the **first statement** of `StopAsync`, with a comment saying why — and what IS asserted is
|
||||
that `StopAsync` (and `DisposeAsync`) dispose the timer, with a non-vacuity check that it was live
|
||||
beforehand. **Reviewer: this is the one plan assertion downgraded rather than met.**
|
||||
- The node manager is resolved **per tick** (`() => _server?.NodeManager?.Revalidate…`) rather than
|
||||
captured, so a restart cannot leave the tick walking a dead manager's item map.
|
||||
- Host.Tests: 132 green.
|
||||
|
||||
### T9 — over-the-wire expiry test (key-carrying session)
|
||||
|
||||
- **RED observed (build):** `error CS1503: Argument 2: cannot convert from 'string' to 'byte[]'` — the
|
||||
1.5.378 `UserNameIdentityToken.DecryptedPassword` is raw bytes, not a string (the host already decodes
|
||||
UTF-8; the token had to be built the same way).
|
||||
- **FALSIFICATION (the plan's step 2, run and recorded):** deleting the single `rig.TickRevalidation()`
|
||||
call from `An_expired_key_stops_a_live_subscription_with_no_deploy` makes it **`[FAIL]`** — the expired
|
||||
key goes on streaming. So the denial is produced by the T8 tick and by nothing else. Restored, both
|
||||
cases green.
|
||||
- **DEVIATION (fake choice):** the plan calls for "an `ApiKeyOpcUaUserAuthenticator` over a fake
|
||||
`IApiKeyStore`". The rig instead uses a 6-line `KeyPresentingAuthenticator` returning
|
||||
`OpcUaUserAuthResult.AllowKey(principal, [], keyId)`. The credential-verification path (parse, peppered
|
||||
HMAC, store lookup, deny shapes) is already covered in the Host suite and changes nothing about what
|
||||
this file asks — **whether the SDK hands a monitored item back an identity that still carries the key
|
||||
id**. Grants come from the deployed snapshot, so the authenticator returns no roles.
|
||||
- The tick is driven through the **real `AclRevalidationTimer`**, which required a new `ProjectReference`
|
||||
from the integration test project to `ZB.MOM.WW.OtOpcUa.Host` (the timer lives there). Without it the
|
||||
test would have called the node manager's pass directly and could not have falsified T8.
|
||||
- OpcUaServer.IntegrationTests: 20 green (was 18).
|
||||
|
||||
### T10 — docs
|
||||
|
||||
- `docs/security.md` store table: the "Revocation takes effect" column is now **two** columns — new
|
||||
activation vs already-open session — with an explicit ⚠️ paragraph saying the old single column was
|
||||
wrong and why (an open session re-checks the artifact snapshot, never `IApiKeyStore`), and that expiry
|
||||
is the exception on both node kinds.
|
||||
- `docs/security.md` new subsection **"Bounding the key store"** — the lock hazard, both knobs, what a
|
||||
timeout looks like in the log, the counter to watch, why a timeout denies, and why faults are never
|
||||
cached.
|
||||
- `docs/security.md` revocation bullets: split into the **deployment axis** and the **clock axis**; the
|
||||
live-subscription bullet now names **two triggers**, the timer's key and default, the silenced-not-closed
|
||||
semantics with re-activation recovery, and the event-monitored-item residual.
|
||||
- `CLAUDE.md`: the "asymmetric revocation" sentence corrected to activation-scoped (naming the old wording
|
||||
as wrong), a new bullet for the S18 bound, and the stage-6 "Revocation reaches live subscriptions" bullet
|
||||
extended with the second trigger and the pass-local-memo constraint.
|
||||
- **Beyond the plan's file list:** `docs/Configuration.md` gained rows for all three new keys. It carries
|
||||
the field-by-field tables for `Security:ApiKeys` and `Security:Acl`, so shipping the knobs without them
|
||||
would have been the exact doc drift this plan's cross-cutting requirement is about.
|
||||
|
||||
### T12 — residuals filed
|
||||
|
||||
- **R1** → Gitea **#562** — *ACL: event/alarm monitored items are revalidated by neither trigger*
|
||||
(`https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/562`).
|
||||
- **R2** → Gitea **#563** — *ACL: an expired API key keeps its session's roles while
|
||||
`Security:Acl:Enabled` is false* (`https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/563`).
|
||||
- **R3** (activation-time revocation now bounded by `StoreCacheSeconds`) needs no issue — it is documented
|
||||
behaviour after T10, in both `docs/security.md` and `docs/Configuration.md`.
|
||||
|
||||
### T11 — LIVE GATE
|
||||
|
||||
**Left `pending` for the orchestrator**, per the execution brief. Nothing in this branch has been run
|
||||
against the docker-dev rig; the nine-step recipe in Finding 2 → Live gate is unchanged and still applies.
|
||||
Two notes for whoever runs it:
|
||||
|
||||
- `Security:Acl:RevalidationIntervalSeconds` defaults to 60, so step 6's "within one revalidation
|
||||
interval" is ≤ 60 s unless the rig overrides it. Setting it to `15` shortens the wall-clock wait without
|
||||
changing what is proven.
|
||||
- Step 8's cache-hit leg reads `otopcua.opcua.apikey_store.bounded{outcome=cache_hit}`; the instrument only
|
||||
exists on an **admin/fused** node, since the artifact store is not wrapped.
|
||||
|
||||
### Summary of what a reviewer should look at
|
||||
|
||||
1. **`Program.cs` → `DataPlaneAuthRegistration`** (T1): a verbatim move of the authorizer + key-store
|
||||
branches out of top-level statements so they could be guarded. Confirm no branch changed.
|
||||
2. **`OtOpcUaServerHostedServiceRevalidationTests`** (T8): the plan's dispose-**before**-`SetDataPlaneAuthorizer(null)`
|
||||
ordering is guaranteed structurally but **not asserted** — see T8's notes for why it is unobservable.
|
||||
3. **`BoundedApiKeyStore.IsExpired`** (T3): a two-line fail-closed addition beyond the plan, so the TTL
|
||||
cannot resurrect a clock-expired key.
|
||||
4. New public members added purely so wiring could be pinned by reference: `DataPlaneAuthorizer.Clock`,
|
||||
`BoundedApiKeyStore.Inner` / `.Clock`.
|
||||
|
||||
---
|
||||
|
||||
## Review outcome — FIX-FIRST, applied 2026-08-20
|
||||
|
||||
Coordinator verdict on the branch was **FIX-FIRST** with four items. All four applied; Host.Tests and
|
||||
OpcUaServer.Tests rerun green.
|
||||
|
||||
1. **[Important] `IsExpired` had no test — the review was right, and the gap was mine.** T3's execution
|
||||
note called the guard a "fail-closed refinement" and shipped it with *no case that fails without it*:
|
||||
deleting `&& !IsExpired(cached, utcNow)` left all 14 `BoundedApiKeyStoreTests` green. That is the
|
||||
unchanged-observable trap — the guard's whole effect is invisible unless a cached entry outlives its
|
||||
key's `ExpiresUtc` **inside** the TTL, which no existing case constructed.
|
||||
Added **`A_cached_key_that_expires_by_the_clock_is_not_served`**: key expiring at `start + 2 s`, TTL
|
||||
60 s, first `FindAsync` non-null (and now cached); flip the inner store to `Revoked`; advance the fake
|
||||
clock 5 s — past `ExpiresUtc`, far inside the TTL; assert `null` and `inner.Calls == 2`. The
|
||||
`Revoked` flip is deliberate: it makes "served from cache" and "re-read" differ in the **answer**, not
|
||||
only in a call count, so a build that re-read and ignored the result cannot pass either.
|
||||
**RED confirmed by falsification:** with the guard removed, exactly **1 failed / 14 passed** — only the
|
||||
new case — then restored, 15/15 green.
|
||||
2. **[Minor] `cache_hit` was metered before the expiry guard could reject.** `TryServe` counted a hit for
|
||||
an entry that then fell through to a store read, so the counter over-reported. The `Add` moved to the
|
||||
two call sites, **after** the guard, behind a `MeterCacheHit()` helper; `TryServe` is now purely a
|
||||
lookup. This matters operationally — `cache_hit` is the ratio §"Bounding the key store" tells an
|
||||
operator to watch.
|
||||
3. **[Minor] Added `A_node_with_neither_role_registers_the_deny_all_authenticator`** — asserts the
|
||||
`IOpcUaUserAuthenticator` singleton **instance** is `NullOpcUaUserAuthenticator.Instance` and that no
|
||||
`IApiKeyStore` is registered. It closes the third branch of the role conditional, which the guard suite
|
||||
had left unpinned even though that branch is the one whose absence stops a node booting.
|
||||
4. **[Minor] `StartRevalidationTimer` now disposes before overwriting.** `StartAsync` can run again when a
|
||||
host restarts the service around the same instance, and the overwritten field would have left the
|
||||
previous timer armed with nothing left to dispose it — the dispose-before-overwrite rule the driver
|
||||
reconnect paths already carry.
|
||||
|
||||
**Verification after the fixes:** solution build 0 errors · Host.Tests **134 passed / 0 failed** (was 132)
|
||||
· OpcUaServer.Tests **493 passed / 1 skipped**. T11 remains `pending` for the orchestrator.
|
||||
|
||||
@@ -5,13 +5,13 @@
|
||||
{
|
||||
"id": "T1",
|
||||
"subject": "Register TimeProvider in DI (shared by authorizer, key store and revalidation timer) + ApiKeyOptions.StoreTimeoutMs/StoreCacheSeconds + AclOptions.RevalidationIntervalSeconds (03/S18, 03/S19)",
|
||||
"status": "pending",
|
||||
"status": "done",
|
||||
"blockedBy": []
|
||||
},
|
||||
{
|
||||
"id": "T2",
|
||||
"subject": "BoundedApiKeyStore: hard per-call deadline on the key store under the SessionManager lock, fail-closed (03/S18)",
|
||||
"status": "pending",
|
||||
"status": "done",
|
||||
"blockedBy": [
|
||||
"T1"
|
||||
]
|
||||
@@ -19,7 +19,7 @@
|
||||
{
|
||||
"id": "T3",
|
||||
"subject": "BoundedApiKeyStore: short TTL read-through cache + falsification control (a revoked key is denied once the TTL has passed)",
|
||||
"status": "pending",
|
||||
"status": "done",
|
||||
"blockedBy": [
|
||||
"T2"
|
||||
]
|
||||
@@ -27,7 +27,7 @@
|
||||
{
|
||||
"id": "T4",
|
||||
"subject": "otopcua.opcua.apikey_store.bounded counter (outcome=timeout|cache_hit) + MeterListener assertions",
|
||||
"status": "pending",
|
||||
"status": "done",
|
||||
"blockedBy": [
|
||||
"T3"
|
||||
]
|
||||
@@ -35,7 +35,7 @@
|
||||
{
|
||||
"id": "T5",
|
||||
"subject": "Wire BoundedApiKeyStore on admin/fused nodes in Program.cs + registration guard test; correct the authenticator's 'no network I/O at all' remark",
|
||||
"status": "pending",
|
||||
"status": "done",
|
||||
"blockedBy": [
|
||||
"T4"
|
||||
]
|
||||
@@ -43,7 +43,7 @@
|
||||
{
|
||||
"id": "T6",
|
||||
"subject": "AclRevalidationTimer (pure, TimeProvider-driven, non-throwing tick) + unit tests (03/S19)",
|
||||
"status": "pending",
|
||||
"status": "done",
|
||||
"blockedBy": [
|
||||
"T1"
|
||||
]
|
||||
@@ -51,7 +51,7 @@
|
||||
{
|
||||
"id": "T7",
|
||||
"subject": "Node-manager expiry tests (FakeTimeProvider advanced past ExpiresUtc, no deploy) + correct the ':316-328' only-trigger xmldoc",
|
||||
"status": "pending",
|
||||
"status": "done",
|
||||
"blockedBy": [
|
||||
"T1"
|
||||
]
|
||||
@@ -59,7 +59,7 @@
|
||||
{
|
||||
"id": "T8",
|
||||
"subject": "Wire the revalidation timer into OtOpcUaServerHostedService (create after StartAsync, dispose FIRST in StopAsync) + ordering test",
|
||||
"status": "pending",
|
||||
"status": "done",
|
||||
"blockedBy": [
|
||||
"T6",
|
||||
"T7"
|
||||
@@ -68,7 +68,7 @@
|
||||
{
|
||||
"id": "T9",
|
||||
"subject": "Over-the-wire AclKeyExpiryEnforcementTests: key-carrying session, clock advanced past expiry, item goes BadUserAccessDenied with no deploy (+ negative control)",
|
||||
"status": "pending",
|
||||
"status": "done",
|
||||
"blockedBy": [
|
||||
"T8"
|
||||
]
|
||||
@@ -76,7 +76,7 @@
|
||||
{
|
||||
"id": "T10",
|
||||
"subject": "Docs: docs/security.md store table + revocation bullets, and the CLAUDE.md 'asymmetric revocation' sentence (activation vs already-open session)",
|
||||
"status": "pending",
|
||||
"status": "done",
|
||||
"blockedBy": [
|
||||
"T5",
|
||||
"T8"
|
||||
@@ -94,7 +94,7 @@
|
||||
{
|
||||
"id": "T12",
|
||||
"subject": "File residuals R1 (event monitored items revalidated by nothing) and R2 (expired key retains roles while Security:Acl:Enabled=false)",
|
||||
"status": "pending",
|
||||
"status": "done",
|
||||
"blockedBy": []
|
||||
}
|
||||
]
|
||||
|
||||
Reference in New Issue
Block a user