lmxopcua/archreview/plans/R2-02-resilience-config-hardening-plan.md

Design + Implementation Plan — R2-02 ResilienceConfig Hardening Pass

• Source reports: archreview/01-core-composition.md (S-6, U-6, S-7, S-8), archreview/03-server-runtime.md (S12 = 01/S-8, S13 = 01/S-7), archreview/04-adminui.md (C-7)
• Review commit: f6eaa267 (2026-07-12 re-review) · Plan verified against tree at: f6eaa267 (master, working-tree archreview updates)
• Scope: src/Core/ZB.MOM.WW.OtOpcUa.Core/Resilience/ (parser, pipeline builder, invoker, factory), src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/ (seam docs, new ranges class), src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverInstanceActor.cs (write dispatch), src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Components/Shared/Drivers/ (form model + section), plus their test suites.
• This is action item #2 in archreview/00-OVERALL.md: "ResilienceConfig hardening pass: range-validate in parser (clamp + diagnostic, never brick) and AdminUI form; wire or delete the bulkhead knob; read WriteIdempotentAttribute (or default writes non-idempotent); fix lossy form round-trip." Effort Small-Medium, Impact High — closes six of the nine notable new findings from the 2026-07-12 round (theme #6: "the resilience seam is the new second-class citizen").
• Design history honored: FOLLOWUP-10 + FOLLOWUP-13. The Runtime is deliberately Polly-free (references Core.Abstractions, never Core); the invoker seam (IDriverCapabilityInvoker/IDriverCapabilityInvokerFactory) lives in Core.Abstractions; the null objects are genuine pass-throughs that keep dispatch tests byte-identical. Nothing in this plan touches the seam's interface shapes, the pass-through nulls, the per-instance Invalidate pairing, or the respawn-on-change semantics — they are extended, not reworked. The parser's contract stays never throw: clamp + diagnostic, don't reject.

Verification summary

Every anchor in the tasking was opened at f6eaa267 and checked against the current source. All five findings CONFIRMED. Two anchors carry minor line drift; two findings were sharpened by verification beyond what the reports state:

Finding	Anchor check	Status
01/S-6	DriverResilienceOptionsParser.cs:91-95 (merge without range checks — actual merge :90-94), DriverResiliencePipelineBuilder.cs:111-114 (AddTimeout(TimeSpan.FromSeconds(policy.TimeoutSeconds))), ResilienceFormModel.cs:32-37 (bare int?, no validation anywhere in the model)	CONFIRMED — and sharpened: two brick vectors, not one (below)
01/U-6	DriverResiliencePipelineBuilder.cs:99-177 (Build actually spans :99-176 — off by one), DriverResilienceOptions.cs:23-25 (drift: BulkheadMaxConcurrent at :23 but BulkheadMaxQueue at :29; doc lines between)	CONFIRMED — no AddConcurrencyLimiter/rate-limiter anywhere in src/; Polly.RateLimiting not in Directory.Packages.props; and strengthened for the delete option (below)
01/S-8 = 03/S12	DriverInstanceActor.cs:597-601 (isIdempotent: true at :599), CapabilityInvoker.cs:117-131 (the !isIdempotent arm)	CONFIRMED — grep -rn WriteIdempotentAttribute hits only its own file + two doc refs in DriverCapability.cs; zero readers. Tier Write/AlarmAcknowledge defaults are RetryCount: 0 on all three tiers (DriverResilienceOptions.cs:83,88,94,99,105,110), so behavior-neutral today. Sharpened: the !isIdempotent arm also skips the tracker's RecordCallStart/Complete accounting (compare :70-81 vs :117-135)
04/C-7	ResilienceFormModel.cs:52-74 (FromJson catches JsonException → empty form at :59, with the "raw view (next task)" comment), DriverResilienceSection.razor:37-40 (raw pane renders _m.ToJson(), not stored text), :60-66 (EmitAsync replaces the bound value)	CONFIRMED — first keystroke after a malformed/hand-edited stored config silently overwrites it. The house preserve-unknown pattern exists next door (Uns/TagEditors/ModbusTagConfigModel.cs — JsonObject _bag + TagConfigJson helpers)
01/S-7 = 03/S13	DriverCapabilityInvokerFactory.cs:71 (Invalidate inside Create, before old child terminates), DriverResiliencePipelineBuilder.cs:75-79 (GetOrAdd — options-blind on a hit; actual :76-79), DriverHostActor.cs:1278-1280 (ToStop → Context.Stop (async) then ToSpawn in one handler)	CONFIRMED — an in-flight old-child handler can re-cache a stale-options pipeline after the new spawn's Invalidate, and the cache serves it indefinitely

Sharpening 1 — S-6 has two brick vectors, verified against Polly.Core 8.6.6 (the pinned version):

• TimeoutStrategyOptions.Timeout is range-validated to [10 ms, 24 h] at pipeline build. "timeoutSeconds": 0 (→ TimeSpan.Zero), any negative, or any value > 86 400 throws ValidationException inside the GetOrAdd factory — on the first call and every subsequent call (a throwing factory caches nothing).
• CircuitBreakerStrategyOptions.MinimumThroughput (which Build sets to policy.BreakerFailureThreshold, :146) is validated to ≥ 2. So "breakerFailureThreshold": 1 — a perfectly plausible operator intent ("open on first failure") — bricks the same way. The report names only the timeout vector; the fix must cover both.
• RetryStrategyOptions.MaxRetryAttempts accepts [1, int.MaxValue] and Build guards RetryCount > 0 (:116), so retry cannot brick — but an absurd retryCount (e.g. 2000000000) is an operational hazard (each attempt backs off up to 5 s) and gets clamped anyway.

Sharpening 2 — U-6's "wire" option is provably inert in production dispatch. DriverInstanceActor is a ReceiveActor whose capability handlers are ReceiveAsync (:406-457) — the mailbox suspends for the whole handler (the source says so itself at :859, :881). At most one capability call is ever in flight per driver child, so a concurrency limiter with any permit ≥ 1 can never queue or reject there. The only fan-out surface that could drive concurrency through one invoker, AlarmSurfaceInvoker, has zero production references (grep: only its own file + analyzer doc remarks). Wiring the bulkhead would add a new package (Polly.RateLimiting) for a strategy that cannot engage — manufacturing a fresh instance of the exact dead-knob genre this pass exists to close. This decides wire-vs-delete (below).

Drift corrections for the reports (cosmetic; fold into the next report refresh):

• 01-core-composition.md U-6 anchor DriverResilienceOptions.cs:23-25 → the two properties sit at :23 and :29.
• 01-core-composition.md U-6 anchor DriverResiliencePipelineBuilder.cs:99-177 → Build is :99-176.
• All other cited lines are exact at f6eaa267.

---

Priority ordering

1. S-6 (High) — the operator-authorable permanent driver-brick; the S-6 brick-repro test is the pass's first commit and must fail on current code.
2. S-8 = 03/S12 (Medium) — the write-retry footgun #13 armed; do second because its parser change shares files with S-6's.
3. U-6 (Medium) — delete the dead bulkhead knob; after S-6/S-8 so the parser churn is serialized.
4. C-7 (Medium) — non-lossy form round-trip; after U-6 so the form's field set is final.
5. S-7 = 03/S13 (Low) — the stale-pipeline race; last, builds on the S-6 builder changes.

---

1. S-6 — HIGH — Out-of-range ResilienceConfig values brick every capability call, violating the parser's never-brick contract

Restatement: DriverResilienceOptionsParser.ParseOrDefaults defends against malformed JSON, unknown capability names, and a misapplied Tier-C recycle interval — but merges TimeoutSeconds/RetryCount/BreakerFailureThreshold overrides with no range validation (DriverResilienceOptionsParser.cs:90-94). Polly 8.6.6 range-validates at pipeline build, so "timeoutSeconds": 0 (or "breakerFailureThreshold": 1) makes DriverResiliencePipelineBuilder.Build throw ValidationException inside the GetOrAdd factory on every capability call for that (instance, host, capability). Subscribe, discover, write, and alarm-ack all fail persistently while the driver is healthy — precisely the outcome the parser's own remarks (:30-33, "a misconfigured ResilienceConfig should never brick a working driver") promise to prevent. Operator-reachable: the AdminUI ResilienceFormModel binds all three as bare int? with no validation (ResilienceFormModel.cs:32-37), and #13's respawn-on-change makes the bad config take effect on the next deploy.

Verification: Confirmed + sharpened (see summary): two brick vectors — timeout out of [1 s, 24 h] and breaker threshold == 1 (Polly MinimumThroughput floor is 2). Retry cannot brick (builder guards > 0; Polly accepts up to int.MaxValue) but is clamped for sanity. The throw site is _pipelines.GetOrAdd (DriverResiliencePipelineBuilder.cs:77-79) invoked from CapabilityInvoker.ResolvePipeline (CapabilityInvoker.cs:140-141) — i.e. inside the actor's try/catch arms, so each call fails "cleanly" (WriteAttributeResult(false, …), etc.) but forever.

Root cause: The parser validates shape (JSON well-formedness, enum membership) but not domain (Polly's documented ranges); nothing between the JSON column and ResiliencePipelineBuilder.Build() knows the ranges, and the one component that enforces them (Polly) does so by throwing at the worst possible seam — inside a memoizing factory on the dispatch path.

Proposed design — clamp + diagnostic at the parser (primary), belt-and-braces clamp at the builder (secondary), mirrored validation messaging in the AdminUI form (tertiary).

• Clamp-vs-reject: reject (refuse the whole override, or fail the deploy) was considered and rejected — the parser's contract is explicit ("callers … should NOT fail driver startup"), draft-time validation can't cover the non-AdminUI write paths (the JSON column is writable by SQL, import, future APIs), and rejecting a whole policy for one bad field discards the operator's valid fields. Clamping each field independently to the nearest legal value (or the tier default when the value is nonsense), with the existing parseDiagnostic out-parameter naming exactly what was clamped, preserves maximum operator intent and can never brick. The factory already logs the diagnostic as a warning (DriverCapabilityInvokerFactory.cs:61-66) — the observability channel exists.

• Single source of truth for the bounds: a new ResiliencePolicyRanges static class in Core.Abstractions (not Core) so all three layers can share it: the parser and builder (Core references Abstractions) and ResilienceFormModel (AdminUI references Runtime → transitively Core.Abstractions; AdminUI deliberately does not reference Core — verified in ZB.MOM.WW.OtOpcUa.AdminUI.csproj:20-26). Constants:

  namespace ZB.MOM.WW.OtOpcUa.Core.Abstractions;
  
  /// <summary>
  ///     Legal ranges for per-capability resilience policy values. Single source of truth shared by
  ///     DriverResilienceOptionsParser (clamp + diagnostic), DriverResiliencePipelineBuilder (last-resort
  ///     clamp so a pipeline build can never throw Polly's ValidationException), and the AdminUI
  ///     ResilienceFormModel (authoring-time validation messages). Derived from Polly.Core 8.6.6's
  ///     option validation: TimeoutStrategyOptions.Timeout ∈ [10 ms, 24 h];
  ///     CircuitBreakerStrategyOptions.MinimumThroughput ≥ 2.
  /// </summary>
  public static class ResiliencePolicyRanges
  {
      /// <summary>Minimum per-call timeout. Whole-second granularity puts the floor at 1 s (Polly's is 10 ms).</summary>
      public const int MinTimeoutSeconds = 1;
      /// <summary>Maximum per-call timeout — Polly's TimeoutStrategyOptions ceiling (24 h).</summary>
      public const int MaxTimeoutSeconds = 86_400;
      /// <summary>Retry floor; 0 = no retry strategy is added to the pipeline.</summary>
      public const int MinRetryCount = 0;
      /// <summary>Operational retry cap (Polly itself allows int.MaxValue; 100 exponential-backoff attempts ≈ minutes).</summary>
      public const int MaxRetryCount = 100;
      /// <summary>Breaker floor; 0 = no breaker strategy is added (the Tier C posture).</summary>
      public const int MinBreakerFailureThreshold = 0;
      /// <summary>Polly's MinimumThroughput floor — an enabled breaker needs a threshold of at least 2.</summary>
      public const int MinEnabledBreakerFailureThreshold = 2;
      /// <summary>Sanity cap on the breaker threshold (Polly has no ceiling; beyond this it can never trip in the 30 s window).</summary>
      public const int MaxBreakerFailureThreshold = 10_000;
  }
  

• Parser clamp semantics (per merged override field, in ParseOrDefaults after the null-coalesce at :91-94; diagnostics use the existing ??=-first-wins channel — upgrade it to append with "; " so multiple clamps all surface, matching the "one diagnostic string" signature):

  • timeoutSeconds ≤ 0 → that capability's tier default + diagnostic (a non-positive timeout is nonsense, not an extreme — snapping to 1 s could be a 30× surprise on a Discover policy). > MaxTimeoutSeconds → clamp to max + diagnostic.
  • retryCount < 0 → 0 + diagnostic. > MaxRetryCount → 100 + diagnostic.
  • breakerFailureThreshold < 0 → 0 (breaker off) + diagnostic. == 1 → MinEnabledBreakerFailureThreshold (2) + diagnostic (closest to the operator's "open fast" intent). > MaxBreakerFailureThreshold → clamp + diagnostic.

• Builder belt-and-braces (in Build, DriverResiliencePipelineBuilder.cs:108): clamp the resolved policy before constructing strategy options — Math.Clamp(policy.TimeoutSeconds, 1, 86_400), Math.Min(policy.RetryCount, 100) under the existing > 0 guard, Math.Max(2, …) for MinimumThroughput under the existing > 0 guard. Rationale: Build is the throw site, and DriverResilienceOptions is a public record constructible by paths that never traverse the parser (tests today; any future caller). This makes "a pipeline build never throws ValidationException" a builder invariant, directly testable. Cost: three integer ops once per pipeline build (not per call). Alternative rejected: catching ValidationException around GetOrAdd and falling back to a tier-default pipeline — heavier, hides the misconfiguration deeper, and still leaves the invariant unproven for direct Build callers.

• AdminUI form (ResilienceFormModel): add a Validate() returning IReadOnlyList<string> of range-violation messages (using ResiliencePolicyRanges, naming the capability + field + legal range — mirroring the driver tag editors' Validate() idiom), and render the messages in DriverResilienceSection.razor as an inline warning block under the table. The form still emits (the parser clamp is the authoritative guard; blocking save would require touching all 8 driver pages' save flows — out of scope, noted for 04/C-1's authz/gating pass). This is authoring-time feedback, not the enforcement layer — defense in depth stays in the parser, exactly as the report recommends.

Implementation steps:

1. Brick-repro test first (must fail at f6eaa267): new tests/Core/ZB.MOM.WW.OtOpcUa.Core.Tests/Resilience/ResilienceConfigBrickTests.cs — parse {"capabilityPolicies":{"Read":{"timeoutSeconds":0}}} (Tier A), GetOrCreate + execute a trivial callSite through the pipeline; assert it succeeds and repeat for {"breakerFailureThreshold":1}. On current code both throw ValidationException.
2. Add ResiliencePolicyRanges.cs to src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/.
3. Parser: extract a private static CapabilityPolicy ClampPolicy(DriverCapability cap, CapabilityPolicy merged, CapabilityPolicy tierDefault, ref string? diagnostic) applied inside the merge loop; switch the diagnostic channel from ??= to an append helper.
4. Builder: clamp in Build as above; keep the RetryCount > 0 / BreakerFailureThreshold > 0 enable-guards evaluating the clamped values.
5. AdminUI: ResilienceFormModel.Validate() + warning block in DriverResilienceSection.razor.

Tests:

• ResilienceConfigBrickTests (step 1) goes green — the load-bearing regression guard.
• DriverResilienceOptionsParserTests: one test per clamp rule (TimeoutSeconds_Zero_FallsBackToTierDefault_WithDiagnostic, TimeoutSeconds_AbovePollyMax_Clamped, RetryCount_Negative_ClampedToZero, RetryCount_AboveCap_ClampedTo100, BreakerThreshold_One_ClampedToTwo, BreakerThreshold_Negative_DisablesBreaker, MultipleClamps_AllSurfaceInDiagnostic); regression: in-range overrides pass through untouched with null diagnostic.
• DriverResiliencePipelineBuilderTests: Build_NeverThrows_ForHostileDirectOptions — loop a matrix of hostile CapabilityPolicy values (int.MinValue, -1, 0, 1, int.MaxValue) through GetOrCreate + execute; asserts the builder invariant independently of the parser.
• Production-wiring assertion (unit-green ≠ wired): DriverCapabilityInvokerFactoryTests.Create_with_out_of_range_config_yields_working_invoker — the exact prod path DriverHostActor.SpawnChild exercises (factory.Create(id, type, hostileJson) → invoker.ExecuteAsync succeeds + the clamp diagnostic is logged). Prod DI binding of this factory is already pinned by ResilienceInvokerFactoryRegistrationTests (Host.IntegrationTests) — unchanged, re-run.
• AdminUI live-/run: on docker-dev (http://localhost:9200, login disabled — rebuild both central-1 and central-2, the rig round-robins), open a driver's edit page, type 0 into a Timeout cell → the warning renders; deploy → grep the central logs for the clamp diagnostic (Driver resilience config for instance=…: … clamped) and confirm the driver's subscribe/read still works.

Effort: S. Risk: Low — clamping only ever widens the set of configs that work; in-range configs are byte-identical. Blast radius: parser + builder + one AdminUI component.

---

2. S-8 = 03/S12 — MEDIUM — Dispatch hardcodes isIdempotent: true; the non-idempotent write safeguard is unreachable

Restatement: The only production caller of ExecuteWriteAsync is DriverInstanceActor.HandleWriteAsync passing isIdempotent: true unconditionally (DriverInstanceActor.cs:597-601). WriteIdempotentAttribute — whose entire purpose is per-tag opt-in to write retries — has zero readers, and its own xmldoc lies ("The CapabilityInvoker … reads this attribute via reflection once at driver-init time", WriteIdempotentAttribute.cs:12-14). Behavior-neutral today (every tier's Write default is RetryCount: 0), but #13 made per-instance Write retries operator-authorable — the moment one is configured, all of that instance's writes retry, including command-shaped ones (pulse coils, counter increments, Galaxy supervisory writes) where a duplicate is not harmless.

Verification: Confirmed (see summary). Additional facts that shape the design: (a) the Phase 6.1 spec itself says retries are "skipped on Write and AlarmAcknowledge regardless of tier" (DriverResilienceOptions.cs:72-73 — currently a default, not an invariant: a parsed override defeats it); (b) the alarm-ack dispatch site (DriverInstanceActor.cs:~650, "Ack is a write-shaped operation — the AlarmAcknowledge tier policy never retries") routes through plain ExecuteAsync(DriverCapability.AlarmAcknowledge, …), so a configured AlarmAcknowledge.retryCount > 0 retries acks by the same mechanism — the finding as filed covers Write only, but a coherent fix must treat both write-shaped capabilities; (c) the !isIdempotent arm skips the tracker's RecordCallStart/Complete (CapabilityInvoker.cs:117-135 vs :70-81) and allocates a fresh options record + Dictionary per call (= 01/P-4).

Root cause: The per-tag idempotence signal was designed (attribute + invoker arm) but never threaded: the dispatch site has only a TagId string, no tag-definition metadata, so true was hardcoded with a comment rationalizing plain value-writes — silently converting a per-tag safety contract into an instance-wide one.

Proposed design — enforce the spec's write-shaped no-retry invariant at the parser, AND default the dispatch to non-idempotent (two independent layers), deferring per-tag opt-in.

Alternatives weighed (attribute-read vs config-flag vs default-flip):

• (a) Read WriteIdempotentAttribute per tag at the seam — the designed mechanism, but threading tag-definition metadata from the driver's typed tag rows through WriteAttribute/the host routing table to the dispatch site is a feature-sized change across composer/artifact/actor contracts, and no tag-definition POCO in the tree carries the attribute today, so it would still resolve to non-idempotent everywhere. Deferred as the documented forward path, not built now.
• (b) Delete the isIdempotent parameter and document all-writes-idempotent — codifies the unsafe posture the finding exists to prevent; rejected.
• (c — chosen, two layers) Layer 1 (parser, spec enforcement): ParseOrDefaults clamps retryCount overrides for Write and AlarmAcknowledge to 0 with a diagnostic ("Write/AlarmAcknowledge never retry (writes are treated as non-idempotent until per-tag opt-in ships); retryCount override ignored"). This turns the spec sentence into an enforced invariant on the only JSON→options funnel, and it honestly surfaces the knob's inertness instead of silently ignoring it (the house dead-knob lesson). When the per-tag opt-in ships later, this clamp is the one line to relax. Layer 2 (dispatch, belt): flip the dispatch site to isIdempotent: false (03/S12's own recommendation) so the invoker's no-retry arm is authoritative even if the parser layer is bypassed. Fix the arm while making it the live path: cache the no-retry options snapshot once per invoker (lazy field computed from the ctor snapshot — the options are immutable for the invoker's lifetime per #13's respawn-on-change design), and add the missing RecordCallStart/Complete tracker accounting — resolving 01/P-4 (per-write allocation) in the same motion instead of shipping a newly-hot allocating path.
• Docs + form: correct WriteIdempotentAttribute's false claim ("reserved for the per-tag opt-in; not yet consulted — dispatch currently treats every write as non-idempotent"); correct the HandleWriteAsync comment; annotate/disable the Write + AlarmAcknowledge "Retries" cells in DriverResilienceSection.razor (disabled input with a title/tooltip "writes never retry — per-tag opt-in not yet available") so the operator can't author a value the parser will discard. Keep the attribute type — it is the documented forward contract referenced by DriverCapability's xmldoc; deleting it would orphan the design history for a 20-line file.

Implementation steps:

1. Parser: in the merge loop's clamp helper (from S-6), special-case DriverCapability.Write and DriverCapability.AlarmAcknowledge: any retryCount override > 0 → 0 + diagnostic. (Tier defaults are already 0; only overrides can differ.)
2. DriverInstanceActor.HandleWriteAsync (:597-601): isIdempotent: true → isIdempotent: false; rewrite the comment block (:592-596) to state the non-idempotent default + the attribute-based future opt-in.
3. CapabilityInvoker: add private DriverResilienceOptions? _noRetryWriteOptions; built lazily from _optionsAccessor() (one allocation per invoker lifetime); wrap the !isIdempotent arm in the same RecordCallStart/Complete try/finally the other overloads use (host key: pass the caller's hostName to the tracker — the ::non-idempotent suffix is a pipeline-cache key, not an operator-facing host).
4. Docs: WriteIdempotentAttribute.cs remarks; IDriverCapabilityInvoker.ExecuteWriteAsync xmldoc (unchanged semantics, but note the current production default is false).
5. Razor: disable the two Retries inputs + tooltip.

Tests:

• DriverResilienceOptionsParserTests: WriteRetryOverride_IsForcedToZero_WithDiagnostic, AlarmAcknowledgeRetryOverride_IsForcedToZero_WithDiagnostic; regression: Read/Subscribe retry overrides still honored.
• Production-wiring assertion: DriverInstanceActorResilienceWiringTests (Runtime.Tests) — the existing recording-invoker write test records ExecuteWriteAsync args; first change its expectation to isIdempotent == false (fails on current code), then flip the dispatch (goes green). This is the exact "a transposed flag would compile and ship" class 03/U6 warns about.
• CapabilityInvokerTests: NonIdempotentWrite_DoesNotRetry_EvenWhenWritePolicyHasRetries (exists in spirit — extend to assert the cached snapshot: two calls, assert no per-call options divergence via the builder's CachedPipelineCount staying flat); NonIdempotentWrite_RecordsTrackerStartAndComplete (new — pins the accounting fix).
• FlakeyDriverIntegrationTests (Core.Tests): re-run — the end-to-end retry/breaker behavior tests must stay green (they exercise Read paths).

Effort: S. Risk: Low-Medium — behavior-changing on the write dispatch path, but the observable change is only "an operator-configured Write retry no longer fires", which is the fix. The pipeline key for writes changes ({host}::non-idempotent), preserving per-host breaker isolation; the write breaker now samples independently of the read breaker for the same host — acceptable (write failures opening the read path's breaker was never a stated contract) but called out for review.

---

3. U-6 — MEDIUM — Bulkhead is documented, defaulted, parsed, authored — and never built: DELETE the knob

Restatement: DriverResilienceOptions.BulkheadMaxConcurrent/BulkheadMaxQueue (:23, :29) are defaulted (32/64), merged by the parser (DriverResilienceOptionsParser.cs:116-117), authored by the AdminUI form (ResilienceFormModel.cs:18-20 + two razor inputs), and advertised in the seam docs (IDriverCapabilityInvoker.cs:5,18), the analyzer's diagnostic message (UnwrappedCapabilityCallAnalyzer.cs:105), and the builder's own composition remark (DriverResiliencePipelineBuilder.cs:18) — but Build (:99-176) adds only timeout → retry → breaker. No concurrency/rate-limiting strategy exists anywhere in src/. An operator can set it, the deploy respawns the driver, and nothing changes.

Verification: Confirmed + strengthened (see summary sharpening 2): beyond "not wired", the wire option is provably unable to engage — actor-serialized dispatch bounds in-flight calls at 1 per driver child, and the lone concurrent fan-out consumer (AlarmSurfaceInvoker) has zero production references. Wiring also requires a new package (Polly.RateLimiting — absent from Directory.Packages.props).

Root cause: Phase 6.1 Stream A specified the four-strategy composition; the builder shipped three and the surrounding surface (options, parser, form, docs) was built to the spec rather than to the builder.

Proposed design — DELETE (recommended), not wire.

• Wire (rejected): AddConcurrencyLimiter(permitLimit, queueLimit) after the breaker is ~15 lines + one package — but per the verification it cannot engage under the actor-serialized dispatch model (≤1 in-flight per child), so it would ship as a second-generation authored-but-inert knob: configured, built, and still doing nothing. If a genuinely concurrent dispatch surface ever lands (Tier C out-of-process hosts, a wired AlarmSurfaceInvoker fan-out, batched WriteValues), the strategy can be added then, against a consumer that exercises it.
• Delete (chosen): strip the two fields end-to-end. Migration-free: the parser deserializes into a private shape class, so bulkheadMaxConcurrent/bulkheadMaxQueue keys in stored ResilienceConfig JSON simply become ignored unknown keys (System.Text.Json default) — no config rewrite, no deploy failure. After C-7's fix the form likewise preserves those keys as unknowns instead of stripping them.
• Delete scope (code): DriverResilienceOptions two properties + doc example; parser shape properties + the two merge lines + the doc-example JSON (:14-23); ResilienceFormModel fields + Shape properties + hasAny/emit lines; DriverResilienceSection.razor two inputs (:10-13); tests DriverResilienceOptionsParserTests.BulkheadOverrides_AreHonored (replace with BulkheadKeys_InStoredJson_AreIgnored — pins the migration-free contract), ResilienceFormModelTests bulkhead usages.
• Doc scope (reword "bulkhead" on the live seam only): IDriverCapabilityInvoker.cs:5,18 and IPerCallHostResolver.cs:31 (→ "in-flight accounting"), UnwrappedCapabilityCallAnalyzer.cs:15,105 (analyzer message change — its own tests assert the text, update them), DriverResiliencePipelineBuilder.cs:18,36 composition remark, CapabilityInvoker.cs:42 + AlarmSurfaceInvoker.cs:55, DriverInstanceActor.cs:142 + DriverHostActor.cs:1630 comments, Configuration/Entities/DriverInstance.cs:38-39 doc example, and the operator docs that enumerate the pipeline (docs/OpcUaServer.md:22, docs/ReadWriteOperations.md:3, docs/security.md:323, docs/v1/HistoricalDataAccess.md:33, docs/v2/driver-stability.md if it lists the knob). Leave untouched: the DriverInstanceResilienceStatus.CurrentBulkheadDepth DB column + all EF migrations (renaming a column is migration churn for a reader-less entity — 03/U10's status-reader work owns that surface; add one doc line on the entity: "in-flight call depth; the Polly bulkhead strategy was removed, R2-02"), WedgeDetector's BulkheadDepth parameter (fed by in-flight counts; same deferral), and historical plan documents (docs/plans/, docs/v2/implementation/, docs/v2/plan.md — they are records, not contracts).
• Guard for the genre (from OVERALL theme #1): the reviews note the existing guards catch interface-forwarding and unwrapped-dispatch inertness but not option-parsed-but-unapplied inertness. Add the cheap version: DriverResilienceOptionsTests.Options_properties_are_exactly_the_pipeline_wired_set — reflection-asserts DriverResilienceOptions's public property set is exactly {Tier, CapabilityPolicies, RecycleIntervalSeconds} with a failure message requiring any new knob to (a) be added to the expected list and (b) cite its behavior test. (RecycleIntervalSeconds is itself Tier-C-dormant — 01/U-2's documented open question, explicitly out of this pass's scope; the guard's expected-list comment records that.)

Implementation steps: as scoped above; strip code first (compile errors enumerate every consumer — the safest sweep), then tests, then the doc pass.

Tests: BulkheadKeys_InStoredJson_AreIgnored (parser); updated ResilienceFormModelTests (a stored config carrying bulkhead keys round-trips them as unknown keys — lands with C-7); the property-set guard; analyzer message-text tests updated (tests/Tooling/…Analyzers.Tests — the whole-solution CI leg runs them). Production-wiring assertion: none needed beyond compile — deletion is self-proving; the guard test is the standing protection.

Effort: S-M (mechanical, wide but shallow). Risk: Low — no runtime behavior exists to change; the only hazards are missed doc mentions (grep bulkhead at the end, excluding bin/obj/Migrations/docs/plans) and the analyzer message tests.

---

4. C-7 — MEDIUM — ResilienceFormModel round-trip is lossy: malformed stored JSON silently overwritten, unknown keys stripped, raw pane shows re-serialized model

Restatement: FromJson maps only known top-level fields and known capability keys (ResilienceFormModel.cs:52-74; the TryGetValue at :67 drops unknown capabilities), and catches JsonException by returning an empty form (:59 — with a comment promising a raw-text view "next task" that never shipped). ToJson (:78-101) emits only the model's own fields. The raw pane (DriverResilienceSection.razor:37-40) renders _m.ToJson() — the re-serialized model, not the stored text. So a malformed or unknown-key-bearing stored DriverInstance.ResilienceConfig renders as blank/partial, and the first keystroke in any resilience field (EmitAsync, :60-66) silently overwrites the stored override — data loss with no indication the original existed. This violates the preserve-unknown-keys convention the sibling tag editors established (Uns/TagEditors/ModbusTagConfigModel.cs — JsonObject _bag).

Verification: Confirmed at all anchors. ResilienceFormModelTests.cs exists (4 tests) with no unknown-key coverage. The TagConfigJson helper class (Uns/TagEditors/TagConfigJson.cs) provides the house ParseOrNew/GetInt/Set/Serialize overlay idiom this fix reuses.

Root cause: The form was built from the 2026-05-29 AdminUI-followups plan's typed-shape sketch (which predates the preserve-unknown discipline) and its malformed-JSON story depended on a follow-up raw-view task that was never implemented — the :59 comment is the fossil.

Proposed design — adopt the tag-editor JsonObject-bag round-trip + a parse-failure fail-safe that never overwrites what it couldn't read.

• Round-trip: FromJson parses to a JsonObject bag (JsonNode.Parse); known top-level fields are read out; capabilityPolicies is walked — known capability names populate rows (and known per-policy fields; unknown per-policy fields stay in the bag's policy object), unknown capability entries stay untouched in the bag. ToJson overlays the model's non-null known values onto the bag (removing a known key when its model value is null — "blank = tier default" is preserved for known fields) and serializes the bag; returns null only when the bag is empty and every model field is null (current blank-form contract preserved: Blank_form_serializes_to_null stays green).
• Malformed stored JSON: FromJson sets ParseFailed = true and stores the original text in RawStoredJson. Two sub-options: (i) warn-then-replace (report's minimum: banner "stored JSON could not be parsed — editing will replace it", edits allowed) vs (ii) warn-and-lock (banner + inputs disabled + EmitAsync no-ops while ParseFailed). Choose (ii) — it is the only option with zero data-loss paths; the operator fixes the column via the same path that corrupted it (hand-edit/SQL), or clears it deliberately. Add an explicit "Discard stored JSON and start blank" button inside the warning for the deliberate-replace case (one click sets ParseFailed = false, clears the bag, emits null → intentional, visible destruction instead of a silent side effect of an unrelated keystroke).
• Raw pane: render the bound stored parameter (ResilienceConfig), not _m.ToJson() — it then shows the truth in both the healthy and the malformed case. Delete the :59 fossil comment.

Implementation steps:

1. Red tests first in ResilienceFormModelTests.cs: Unknown_top_level_key_survives_round_trip, Unknown_capability_entry_survives_round_trip, Unknown_per_policy_field_survives_round_trip, Malformed_json_sets_ParseFailed_and_ToJson_returns_original_text — all fail at f6eaa267.
2. Rework ResilienceFormModel per the design (drop the private Shape/PolicyShape classes in favor of the bag + TagConfigJson-style helpers — reuse TagConfigJson directly if its namespace/accessibility allows (it's AdminUI.Uns.TagEditors; referencing it from Components.Shared.Drivers is same-assembly — fine), else lift the three needed helpers).
3. DriverResilienceSection.razor: warning banner + disabled inputs + discard button under ParseFailed; raw pane → @(ResilienceConfig ?? "(null — all tier defaults)"); EmitAsync guard.
4. Keep Emitted_json_is_consumable_by_the_runtime_parser green — the bag must still serialize camelCase keys the parser reads (the parser is PropertyNameCaseInsensitive, and unknown keys are ignored — verified).

Tests: step 1's four tests + updated existing four; interop regression (parser consumes bag-emitted JSON, unknown keys don't produce diagnostics). Production-wiring assertion / live-/run: on docker-dev (:9200, login disabled, rebuild both centrals), hand-mangle a driver's ResilienceConfig in SQL (UPDATE … SET ResilienceConfig = '{ not json' — remember SET QUOTED_IDENTIFIER ON), open the driver page → warning renders, inputs disabled, raw pane shows the mangled text, save-without-touching leaves the column byte-identical; then a healthy config with an unknown key ({"futureKnob":1,"capabilityPolicies":{"Read":{"timeoutSeconds":5}}}) → edit a field, save, re-read the column: futureKnob survived. (This is the AdminUI-has-no-bUnit posture: binding behavior is proven live, model behavior in units.)

Effort: S-M. Risk: Low — the model is fully unit-tested and the emit contract to the parser is pinned by the interop test; the razor change is small but live-verified (the house lesson: Razor binding bugs pass all unit tests).

---

5. S-7 = 03/S13 — LOW — Respawn Invalidate race can permanently re-cache a stale-options pipeline

Restatement: DriverCapabilityInvokerFactory.Create calls _builder.Invalidate(driverInstanceId) (DriverCapabilityInvokerFactory.cs:71) before the old child has terminated (StopChild is an async Context.Stop; DriverHostActor.cs:1278-1280 runs ToStop→ToSpawn in one handler). An old-child handler already mid-await can reach GetOrCreate after the invalidate, re-caching a pipeline built from the old options snapshot under the same (instance, host, capability) key — and the cache ignores options on a hit (DriverResiliencePipelineBuilder.cs:76-79), so the new invoker serves the stale pipeline indefinitely. Millisecond window, permanent + invisible consequence: the operator's ResilienceConfig change silently doesn't apply for the affected keys.

Verification: Confirmed at all anchors. Note the actor-name generation counter (DriverHostActor.cs:1609-1615) shows the respawn machinery already tolerates the old child lingering — the design intent is "old child may still be draining", which is exactly the window.

Root cause: The cache key identifies the instance, not the options epoch; Invalidate is a point-in-time sweep racing a writer that can still hold the old epoch.

Proposed design — make staleness impossible: an options-generation in the pipeline key.

• Chosen: Create stamps each invoker with a monotonically increasing long generation (Interlocked.Increment on a factory field); CapabilityInvoker passes it to GetOrCreate; PipelineKey gains long Generation. An old invoker's post-invalidate re-cache lands under its old generation — a key the new invoker (new generation) never reads. Invalidate(instanceId) still sweeps by instance (all generations), so the next respawn's Create clears any lingering old-generation entries: stale entries are bounded (≤ one epoch's hosts×capabilities) and never served. Per-instance invalidate semantics and respawn-on-change are preserved exactly; the seam interfaces are untouched (the generation is a Core-internal ctor param on the concrete CapabilityInvoker + a defaulted GetOrCreate parameter, so every existing caller/test compiles unchanged).
• Alternative (a) — options-aware cache (store the options reference beside the pipeline; rebuild on reference mismatch): self-healing and no key growth, but adds a per-call reference comparison on the hot path and can thrash-rebuild while the old child drains. Rejected as strictly worse than epoch keying.
• Alternative (b) — defer the new spawn until the old child's Terminated: closes the race at the actor protocol (watch + stash in DriverHostActor) — the largest blast radius of the three for a milliseconds-wide window, and it would change respawn latency semantics. Rejected; noted as the fallback if a second options-epoch consumer ever appears.

Implementation steps:

1. DriverResiliencePipelineBuilder: PipelineKey gains long Generation; GetOrCreate(…, long optionsGeneration = 0); Invalidate unchanged (matches on DriverInstanceId only).
2. CapabilityInvoker: new optional ctor param long optionsGeneration = 0, threaded into both ResolvePipeline and the non-idempotent arm's GetOrCreate.
3. DriverCapabilityInvokerFactory: private long _generation; → var gen = Interlocked.Increment(ref _generation); per Create, passed to the invoker; keep the Invalidate call (it is now the cleanup, not the correctness mechanism — update its comment to say so).

Tests:

• DriverResiliencePipelineBuilderTests.StaleGeneration_ReCache_IsNotServed_ToNewGeneration — deterministic replay of the race: GetOrCreate(key, gen1, optsA); Invalidate(instance); GetOrCreate(key, gen1, optsA) (the old child's late call re-caches); GetOrCreate(key, gen2, optsB) must return a pipeline honoring optsB (behavioral assert: optsB's retry fires / optsA's timeout doesn't). No timing dependence — generations make the interleaving expressible sequentially.
• Production-wiring assertion: DriverCapabilityInvokerFactoryTests.Respawn_interleaved_with_old_invoker_call_still_applies_new_options — Create(id, …, oldJson) → execute once (cache warm) → Create(id, …, newJson) interleaved with an old-invoker call between the new Create's invalidate and the new invoker's first call (drive the old invoker directly — it's a captured reference, exactly what the old child holds) → assert the new invoker's behavior matches newJson. Fails on current code; green with epochs.

Effort: S. Risk: Low — additive keying; default parameter keeps every existing call site identical; the only semantic change is that a stale epoch's re-cache becomes unreachable garbage instead of a live poisoning.

---

Task breakdown

Bite-sized TDD tasks. Each carries classification, estimate, parallelism, files, steps with exact test commands, and the commit message to use at green. Run commands from the repo root. House verification bar: every behavior change lands with (a) a deterministic unit guard and (b) a production-wiring assertion — unit-green ≠ wired; AdminUI changes additionally get a live-/run pass on docker-dev (http://localhost:9200, login disabled — the rig round-robins central-1/central-2, so rebuild BOTH before verifying).

Task 0 — S-6 brick-repro test (RED — must fail on current code)

Classification: test-first (regression repro) · Estimated implement time: 5 min · Parallelizable with: 1 · Files: tests/Core/ZB.MOM.WW.OtOpcUa.Core.Tests/Resilience/ResilienceConfigBrickTests.cs (new)

1. Write ParsedConfig_WithZeroTimeout_DoesNotBrickCapabilityCalls: ParseOrDefaults(DriverTier.A, "{\"capabilityPolicies\":{\"Read\":{\"timeoutSeconds\":0}}}", out _) → new DriverResiliencePipelineBuilder().GetOrCreate("i1","h1",DriverCapability.Read, opts) → await pipeline.ExecuteAsync(ct => ValueTask.FromResult(42), CancellationToken.None) succeeds. Second fact ParsedConfig_WithBreakerThresholdOne_DoesNotBrickCapabilityCalls with {"breakerFailureThreshold":1}.
2. dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Core.Tests --filter "FullyQualifiedName~ResilienceConfigBrickTests" → expect 2 FAILURES (Polly ValidationException) — this proves the brick on f6eaa267. Do not commit red; the commit lands with Task 2/3 green.

Task 1 — ResiliencePolicyRanges in Core.Abstractions

Classification: implementation (new type) · Estimated implement time: 4 min · Parallelizable with: 0 · Files: src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/ResiliencePolicyRanges.cs (new)

1. Add the constants class exactly as specified in §1 (values: 1 / 86 400 / 0 / 100 / 0 / 2 / 10 000), xmldoc citing Polly.Core 8.6.6's validation as the derivation.
2. dotnet build src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions → clean. Commit with Task 2.

Task 2 — Parser range clamps + diagnostics (turns Task 0 half-green)

Classification: bugfix (TDD) · Estimated implement time: 5 min · Parallelizable with: 3 · Files: src/Core/ZB.MOM.WW.OtOpcUa.Core/Resilience/DriverResilienceOptionsParser.cs, tests/Core/…Core.Tests/Resilience/DriverResilienceOptionsParserTests.cs

1. RED: add the seven clamp tests from §1 (TimeoutSeconds_Zero_FallsBackToTierDefault_WithDiagnostic, TimeoutSeconds_AbovePollyMax_Clamped, RetryCount_Negative_ClampedToZero, RetryCount_AboveCap_ClampedTo100, BreakerThreshold_One_ClampedToTwo, BreakerThreshold_Negative_DisablesBreaker, MultipleClamps_AllSurfaceInDiagnostic) + regression InRangeOverrides_PassThrough_NoDiagnostic. dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Core.Tests --filter "FullyQualifiedName~DriverResilienceOptionsParserTests" → new tests fail.
2. GREEN: implement ClampPolicy in the merge loop using ResiliencePolicyRanges (semantics per §1: timeout ≤0 → per-capability tier default; others clamp) + switch the diagnostic to an appending helper (AppendDiagnostic(ref string?, string) joining with "; "). Re-run the filter → all green (verify the pre-existing 15 parser tests still pass — the ??= → append change must not break UnknownCapability_Surfaces_InDiagnostic_ButDoesNotFail).
3. Commit: fix(resilience): range-clamp ResilienceConfig overrides in the parser (01/S-6) — timeout/retry/breaker clamped with diagnostics, never brick

Task 3 — Builder belt-and-braces clamp (turns Task 0 fully green)

Classification: bugfix (TDD) · Estimated implement time: 5 min · Parallelizable with: 2 · Files: src/Core/ZB.MOM.WW.OtOpcUa.Core/Resilience/DriverResiliencePipelineBuilder.cs, tests/Core/…Core.Tests/Resilience/DriverResiliencePipelineBuilderTests.cs

1. RED: add Build_NeverThrows_ForHostileDirectOptions — matrix {int.MinValue, -1, 0, 1, int.MaxValue} across the three CapabilityPolicy fields, constructed directly (bypassing the parser), each GetOrCreate + trivial execute must succeed. Fails on current code.
2. GREEN: clamp in Build (Math.Clamp timeout to [MinTimeoutSeconds, MaxTimeoutSeconds]; Math.Min(retry, MaxRetryCount) under the > 0 guard; Math.Max(MinEnabledBreakerFailureThreshold, threshold) for MinimumThroughput under the > 0 guard). Update the class remark to state the never-throws invariant.
3. dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Core.Tests --filter "FullyQualifiedName~DriverResiliencePipelineBuilderTests|FullyQualifiedName~ResilienceConfigBrickTests" → all green including Task 0's repro.
4. Commit: fix(resilience): pipeline Build clamps policy values — building a pipeline can never throw Polly ValidationException (01/S-6 belt-and-braces)

Task 4 — Factory-seam wiring assertion for S-6

Classification: production-wiring guard · Estimated implement time: 4 min · Parallelizable with: 5, 6 · Files: tests/Core/…Core.Tests/Resilience/DriverCapabilityInvokerFactoryTests.cs

1. Add Create_with_out_of_range_config_yields_working_invoker — factory.Create("i1","Modbus", "{\"capabilityPolicies\":{\"Subscribe\":{\"timeoutSeconds\":0,\"breakerFailureThreshold\":1}}}") → ExecuteAsync(Subscribe, …) succeeds AND the recording logger (the suite's existing Log<TState> fake) captured a clamp diagnostic warning. This is the exact DriverHostActor.SpawnChild path (DriverHostActor.cs:1633).
2. dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Core.Tests --filter "FullyQualifiedName~DriverCapabilityInvokerFactoryTests" → green (should pass immediately post-Tasks 2/3 — it is the wiring proof, not new behavior). Also re-run the DI-binding guard unchanged: dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests --filter "FullyQualifiedName~ResilienceInvokerFactoryRegistrationTests".
3. Commit: test(resilience): factory-seam guard — hostile ResilienceConfig cannot brick the spawn path (01/S-6 wiring proof)

Task 5 — AdminUI form range validation + section warnings

Classification: feature (authoring feedback) · Estimated implement time: 5 min · Parallelizable with: 4, 6 · Files: src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Components/Shared/Drivers/ResilienceFormModel.cs, …/DriverResilienceSection.razor, tests/Server/ZB.MOM.WW.OtOpcUa.AdminUI.Tests/ResilienceFormModelTests.cs

1. RED: Validate_flags_out_of_range_values_with_capability_and_range (timeout 0, retry −1, breaker 1 → 3 messages naming capability/field/legal range) + Validate_passes_in_range_and_blank. dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.AdminUI.Tests --filter "FullyQualifiedName~ResilienceFormModelTests" → fail.
2. GREEN: Validate() using ResiliencePolicyRanges (Core.Abstractions — reachable transitively via the Runtime project reference; do NOT add a Core reference); render _m.Validate() results as a warning list under the table in the razor (emit still proceeds — parser clamp is authoritative).
3. Commit: feat(adminui): range-validation warnings on the resilience override form (01/S-6 authoring layer)

Task 6 — S-8 parser layer: write-shaped capabilities never retry

Classification: bugfix (TDD) · Estimated implement time: 4 min · Parallelizable with: 4, 5 (same file as Task 2 — rebase, don't parallel-edit) · Files: src/Core/…/Resilience/DriverResilienceOptionsParser.cs, tests/Core/…/Resilience/DriverResilienceOptionsParserTests.cs

1. RED: WriteRetryOverride_IsForcedToZero_WithDiagnostic + AlarmAcknowledgeRetryOverride_IsForcedToZero_WithDiagnostic + regression ReadRetryOverride_StillHonored. Filter run → fail.
2. GREEN: in ClampPolicy, for DriverCapability.Write/AlarmAcknowledge force RetryCount overrides > 0 to 0 with the diagnostic text from §2. Update the GetTierDefaults xmldoc (DriverResilienceOptions.cs:72-73) from "defaults skip retries" to "the parser enforces no-retry on these capabilities (R2-02/S-8); per-tag opt-in via WriteIdempotentAttribute is the future relaxation".
3. Commit: fix(resilience): enforce the spec's no-retry invariant for Write/AlarmAcknowledge overrides (01/S-8, 03/S12 layer 1)

Task 7 — S-8 dispatch layer: isIdempotent: false

Classification: bugfix (TDD, behavior-changing) · Estimated implement time: 5 min · Parallelizable with: — (Runtime wiring test is the gate) · Files: src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverInstanceActor.cs, tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Drivers/DriverInstanceActorResilienceWiringTests.cs

1. RED: in the wiring test's recorded-write assertion, change the expected flag to isIdempotent == false. dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests --filter "FullyQualifiedName~DriverInstanceActorResilienceWiringTests" → fails (records true).
2. GREEN: flip :599 to isIdempotent: false; rewrite the :592-596 comment (non-idempotent default; command-shaped writes must never replay; WriteIdempotentAttribute is the unshipped per-tag opt-in). Re-run filter → green. Run the full Runtime suite: dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests (write-path behavior tests must hold — the null invoker passes through identically).
3. Commit: fix(runtime): write dispatch defaults to non-idempotent — no-retry arm authoritative (03/S12 layer 2)

Task 8 — S-8/P-4 invoker: cached no-retry snapshot + tracker accounting

Classification: bugfix + perf (TDD) · Estimated implement time: 5 min · Parallelizable with: 9 · Files: src/Core/…/Resilience/CapabilityInvoker.cs, tests/Core/…/Resilience/CapabilityInvokerTests.cs

1. RED: NonIdempotentWrite_RecordsTrackerStartAndComplete (real DriverResilienceStatusTracker, assert in-flight returns to 0 and the call was counted) + NonIdempotentWrite_ReusesCachedNoRetryOptions (two calls; assert builder.CachedPipelineCount grows by exactly 1 and, via a counting _optionsAccessor, that the accessor was invoked at most once). Filter: dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Core.Tests --filter "FullyQualifiedName~CapabilityInvokerTests" → fail.
2. GREEN: lazy _noRetryWriteOptions field (compute once from _optionsAccessor(); safe — options are immutable per invoker lifetime by #13's respawn-on-change design; benign race tolerated with a simple null-check-assign); wrap the arm in the same RecordCallStart/Complete try/finally as ExecuteAsync, keyed on the caller's hostName (not the ::non-idempotent cache suffix). Update the arm's comment (it currently justifies once-per-call snapshotting — now it justifies once-per-lifetime).
3. Commit: fix(resilience): non-idempotent write arm — cache the no-retry snapshot (01/P-4) and record tracker in-flight accounting (S-8 residual)

Task 9 — S-8 docs + form annotation

Classification: docs + small UI · Estimated implement time: 4 min · Parallelizable with: 8 · Files: src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/WriteIdempotentAttribute.cs, …/IDriverCapabilityInvoker.cs, src/Server/…/AdminUI/Components/Shared/Drivers/DriverResilienceSection.razor

1. Fix WriteIdempotentAttribute's false "reads this attribute via reflection" remark → "reserved for per-tag opt-in; not yet consulted — dispatch treats every write as non-idempotent (R2-02/S-8)". Note the production default on ExecuteWriteAsync's xmldoc.
2. Razor: disable the Write and AlarmAcknowledge Retries inputs (disabled + title="Writes/acks never retry — per-tag opt-in not yet available").
3. dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.AdminUI.Tests --filter "FullyQualifiedName~ResilienceFormModelTests" (unchanged model → green). Commit: docs(resilience): correct WriteIdempotentAttribute's unwired claim; disable Write/Ack retry authoring cells (S-8)

Task 10 — U-6 delete: options + parser + Core tests

Classification: deletion (dead knob) · Estimated implement time: 5 min · Parallelizable with: — (parser file; after 2/6) · Files: src/Core/…/Resilience/DriverResilienceOptions.cs, …/DriverResilienceOptionsParser.cs, tests/Core/…/Resilience/DriverResilienceOptionsParserTests.cs, …/DriverResilienceOptionsTests.cs

1. Remove BulkheadMaxConcurrent/BulkheadMaxQueue from the options record, the parser shape + merge lines + the doc-example JSON. Compile errors enumerate stragglers.
2. Replace BulkheadOverrides_AreHonored with BulkheadKeys_InStoredJson_AreIgnored (JSON carrying both keys parses clean, no diagnostic — pins the migration-free contract).
3. dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Core.Tests --filter "FullyQualifiedName~Resilience" → green. Commit: refactor(resilience): delete the dead bulkhead knob — options + parser (01/U-6); stored JSON keys become ignored unknowns

Task 11 — U-6 delete: form + razor + AdminUI tests

Classification: deletion · Estimated implement time: 4 min · Parallelizable with: 12 · Files: src/Server/…/AdminUI/Components/Shared/Drivers/ResilienceFormModel.cs, …/DriverResilienceSection.razor, tests/Server/…AdminUI.Tests/ResilienceFormModelTests.cs

1. Strip the two model fields + shape props + hasAny terms + the two razor inputs (:10-13; keep RecycleIntervalSeconds); update the form tests (drop bulkhead usages; the C-7 tasks will pin that stored bulkhead keys survive as unknowns).
2. dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.AdminUI.Tests --filter "FullyQualifiedName~ResilienceFormModelTests" → green. Commit: refactor(adminui): remove bulkhead fields from the resilience form (01/U-6)

Task 12 — U-6 doc sweep + knob-inertness guard

Classification: docs + guard test · Estimated implement time: 5 min · Parallelizable with: 11 · Files: seam xmldocs + analyzer + operator docs per §3 scope; tests/Core/…/Resilience/DriverResilienceOptionsTests.cs; tests/Tooling/ZB.MOM.WW.OtOpcUa.Analyzers.Tests/ (message-text assertions)

1. Reword "bulkhead" on the live seam (§3 list: IDriverCapabilityInvoker.cs:5,18, IPerCallHostResolver.cs:31, UnwrappedCapabilityCallAnalyzer.cs:15,105, DriverResiliencePipelineBuilder.cs:18,36, CapabilityInvoker.cs:42, AlarmSurfaceInvoker.cs:55, DriverInstanceActor.cs:142, DriverHostActor.cs:1630, Configuration/Entities/DriverInstance.cs:38-39, one annotation line on DriverInstanceResilienceStatus.CurrentBulkheadDepth, and docs/OpcUaServer.md:22 / docs/ReadWriteOperations.md:3 / docs/security.md:323 / docs/v1/HistoricalDataAccess.md:33 / docs/v2/driver-stability.md). Leave migrations + WedgeDetector + historical plan docs. Final check: grep -rn -i bulkhead src docs --include='*.cs' --include='*.razor' --include='*.md' | grep -v 'bin/\|obj/\|Migrations/\|docs/plans/\|docs/v2/implementation/\|docs/v2/plan.md\|CurrentBulkheadDepth\|WedgeDetector' → empty.
2. Add Options_properties_are_exactly_the_pipeline_wired_set (reflection guard, expected set {Tier, CapabilityPolicies, RecycleIntervalSeconds}, failure message demanding a behavior test per new knob; comment noting RecycleIntervalSeconds is Tier-C-dormant per 01/U-2, out of scope here).
3. Analyzer message change: dotnet test tests/Tooling/ZB.MOM.WW.OtOpcUa.Analyzers.Tests → green after updating any message-text assertions.
4. Commit: docs+test(resilience): bulkhead doc sweep + parsed-knob-must-be-wired guard (01/U-6, OVERALL theme 1)

Task 13 — C-7 red tests: non-lossy round-trip

Classification: test-first · Estimated implement time: 5 min · Parallelizable with: — · Files: tests/Server/…AdminUI.Tests/ResilienceFormModelTests.cs

1. Add the four red tests from §4 step 1 (Unknown_top_level_key_survives_round_trip — use bulkheadMaxConcurrent as the unknown key, doubling as the U-6 continuity proof; Unknown_capability_entry_survives_round_trip; Unknown_per_policy_field_survives_round_trip; Malformed_json_sets_ParseFailed_and_ToJson_returns_original_text).
2. dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.AdminUI.Tests --filter "FullyQualifiedName~ResilienceFormModelTests" → 4 failures. No commit (lands with Task 14).

Task 14 — C-7 implement: JsonObject-bag round-trip + ParseFailed

Classification: bugfix (TDD) · Estimated implement time: 5 min · Parallelizable with: — · Files: src/Server/…/AdminUI/Components/Shared/Drivers/ResilienceFormModel.cs

1. Rework per §4: bag-based FromJson/ToJson (reuse AdminUI.Uns.TagEditors.TagConfigJson helpers — same assembly), ParseFailed + RawStoredJson, null-when-empty contract preserved, delete the :59 fossil comment and the private Shape/PolicyShape classes.
2. Filter run → all ResilienceFormModelTests green including Task 13's four and the parser-interop test.
3. Commit: fix(adminui): non-lossy ResilienceFormModel round-trip — preserve unknown keys, never overwrite unparseable stored JSON (04/C-7)

Task 15 — C-7 razor: warning, lock-on-ParseFailed, truthful raw pane (+ live-/run)

Classification: UI + live verification · Estimated implement time: 5 min (+ live pass) · Parallelizable with: 16 · Files: src/Server/…/AdminUI/Components/Shared/Drivers/DriverResilienceSection.razor

1. ParseFailed ⇒ warning banner + all inputs disabled + EmitAsync no-op + "Discard stored JSON and start blank" button (explicit destructive action → clears bag, emits null); raw pane renders the bound ResilienceConfig parameter, not _m.ToJson().
2. Live-/run (AdminUI has no bUnit — binding must be driven): rebuild BOTH docker-dev centrals; SQL-mangle one driver's ResilienceConfig (SET QUOTED_IDENTIFIER ON; first) → page shows warning/locked/raw truth and save-untouched leaves the column byte-identical; then the unknown-key survival check from §4. Also drive the S-6 (Task 5) warning and the S-8 (Task 9) disabled cells in the same session.
3. Commit: fix(adminui): resilience section — parse-failure lockout + truthful raw pane, live-verified (04/C-7)

Task 16 — S-7 builder: generation-keyed pipeline cache

Classification: bugfix (TDD) · Estimated implement time: 5 min · Parallelizable with: 15 · Files: src/Core/…/Resilience/DriverResiliencePipelineBuilder.cs, src/Core/…/Resilience/CapabilityInvoker.cs, tests/Core/…/Resilience/DriverResiliencePipelineBuilderTests.cs

1. RED: StaleGeneration_ReCache_IsNotServed_ToNewGeneration (deterministic interleave per §5 — behavioral assert on the new options). Fails: without generations both calls share one key.
2. GREEN: PipelineKey + GetOrCreate(…, long optionsGeneration = 0); CapabilityInvoker optional ctor param threaded to both resolve sites. All existing builder/invoker tests compile unchanged (defaulted param).
3. dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Core.Tests --filter "FullyQualifiedName~DriverResiliencePipelineBuilderTests|FullyQualifiedName~CapabilityInvokerTests" → green. Commit: fix(resilience): options-generation in the pipeline cache key — stale respawn re-cache unreachable (01/S-7, 03/S13)

Task 17 — S-7 factory: per-Create generation + interleave wiring proof

Classification: bugfix + production-wiring guard · Estimated implement time: 4 min · Parallelizable with: — · Files: src/Core/…/Resilience/DriverCapabilityInvokerFactory.cs, tests/Core/…/Resilience/DriverCapabilityInvokerFactoryTests.cs

1. RED: Respawn_interleaved_with_old_invoker_call_still_applies_new_options (per §5 — old invoker reference re-caches between the new Create and the new invoker's first call; new invoker must behave per newJson). Fails pre-fix.
2. GREEN: Interlocked.Increment generation per Create, passed to the invoker; update the Invalidate comment (now cleanup, correctness is the epoch key).
3. dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Core.Tests --filter "FullyQualifiedName~DriverCapabilityInvokerFactoryTests" → green. Commit: fix(resilience): factory stamps a per-Create options generation — closes the respawn stale-pipeline race (01/S-7)

Task 18 — Wrap-up: full-solution gate + bookkeeping

Classification: verification + docs · Estimated implement time: 5 min (+ suite runtime) · Parallelizable with: — · Files: archreview/plans/STATUS.md, archreview/plans/00-INDEX.md (if it tracks R2 items)

1. dotnet build ZB.MOM.WW.OtOpcUa.slnx (0 warnings — OTOPCUA0001 must stay silent: no dispatch site was unwrapped) then dotnet test ZB.MOM.WW.OtOpcUa.slnx — the whole-solution leg is the CI gate; the fail-on-skip discipline applies.
2. Record the pass in STATUS.md (completed table + re-review findings table: 01/S-6, 01/U-6, 01/S-8=03/S12, 04/C-7, 01/S-7=03/S13 → done, with commits) and note the two report anchor drifts from this plan's Verification summary for the next report refresh.
3. Commit: docs(archreview): record R2-02 resilience hardening pass in STATUS.md

---

Suggested PR sequencing

1. PR-1 (S-6, the High): Tasks 0–5 — parser/builder clamps + factory guard + form warnings. Ships the brick fix alone; smallest reviewable close of the round's #2-ranked finding.
2. PR-2 (S-8): Tasks 6–9 — no-retry invariant + dispatch flip + invoker arm hardening + docs.
3. PR-3 (U-6): Tasks 10–12 — bulkhead deletion + guard.
4. PR-4 (C-7): Tasks 13–15 — non-lossy form + live-verify (one docker-dev session covers PR-1/2/4's UI checks if landed together).
5. PR-5 (S-7): Tasks 16–17 — generation-keyed cache.
6. Task 18 rides the last PR.

Overall effort: Small-Medium (matches 00-OVERALL action #2) — ≈ 19 bite-sized tasks, roughly one focused day including the live-/run session. Overall risk: Low — every behavior change is either strictly widening (clamps), strictly safening (write no-retry), or deletion of provably-inert surface; the two hot-path touches (write arm, pipeline key) are covered by behavioral guards and the OTOPCUA0001 analyzer keeps every dispatch site wrapped.

---

Execution deviations (R2-02)

• Task 15 (AdminUI live-/run) marked deferred-live. All razor/code shipped and offline unit/bUnit-substitute suites are green; the docker-dev :9200 /run pass (warning banner, input lockout, unknown-key survival) is a single-tenant serial live pass the controller runs one-at-a-time. Not blocking.
• Task 18 whole-solution dotnet test sweep marked deferred-live. Per a mid-execution controller constraint, the *.IntegrationTests / whole-solution test run has a ~16 GB/run memory leak and cannot run concurrently with other plans. I ran the impacted UNIT suites filtered (Core resilience 121/121, AdminUI form 10/10, Analyzers 32/32, Runtime wiring 2/2, full Runtime 364/364 at task 7) and the full-solution dotnet build (0 errors, 0 production OTOPCUA0001). The whole-solution dotnet test gate is left for the controller's serialized heavy pass.
• DriverInstanceResilienceStatus.CurrentBulkheadDepth doc reworded to avoid the literal word "bulkhead" in free text (Task 12). The plan permitted "one doc line on the entity" mentioning the bulkhead removal, but the Task-12 grep-must-be-empty check treats any "bulkhead" hit as a miss. Reworded to "Formerly fed a Polly concurrency-limiter strategy that was removed (R2-02/01/U-6)…" so the explanatory annotation stays while the grep is clean; the column NAME CurrentBulkheadDepth (excluded by the grep) still carries the historical term. The EF migration + WedgeDetector.BulkheadDepth were left untouched as scoped.
• Baseline OTOPCUA0001 warnings in AbLegacy.Tests are pre-existing (test code invoking the driver directly to assert wire-level behavior — the documented-acceptable case). My branch never touched those files and introduced zero new dispatch-site unwraps; production code has 0 OTOPCUA0001.
• The worktree's local master ref is behind this branch's base (1676c8f4), so git diff master..HEAD shows unrelated base files (R2-01/03/05 plans, S7Driver, etc.). Only the 18 fix(r2-02):/docs(r2-02): commits are this plan's work.