Merge R2-10 Resilience observability (arch-review round 2) [PR #432]
Finding 03/U10=04/U-5: DPS mirror of the driver-health flow surfaces retry/breaker/staleness on AdminUI /hosts (tracker had no reader). Core truthfulness fixes (RecordBreakerClose, RecordSuccess on every success path). T10 live gate deferred. Clean merge, build clean.
This commit is contained in:
Joseph Doherty
@@ -175,9 +175,8 @@ Closed since 9cad9ed0: **hard-kill failover** (`HardKillFailoverTests`, in-proce
|
||||
|
||||
**U9 — LOW: `EnsureVariable` silently ignores changed historize-intent on an existing node** (`OtOpcUaNodeManager.cs:1345-1349`) — invariant still lives in two files' comments rather than an assert.
|
||||
|
||||
**U10 — LOW (NEW): The `DriverResilienceStatusTracker` has no reader — pipeline observability is log-only.**
|
||||
The tracker is registered, fed by every retry/breaker event, and consumed by nothing (Admin `/hosts` Stream E.2/E.3 never shipped). The interim mitigation (retry/breaker-open/breaker-close logging in `DriverResiliencePipelineBuilder.cs:124-170`, added with #10) is real and correctly gated, but breaker state — now a production behavior that can reject calls (`BrokenCircuitException`) — is invisible to operators except in logs. Known residual from FOLLOWUP-10; recorded here so the review tracks it.
|
||||
*Recommendation:* ship the `/hosts` resilience column, or at minimum a metric per breaker-open.
|
||||
**U10 — LOW (NEW): The `DriverResilienceStatusTracker` has no reader — pipeline observability is log-only.** — ✅ **REMEDIATED** (R2-10, branch `r2/10-resilience-observability`). The tracker now has a production reader: a periodic (5 s) full-snapshot publisher (`DriverResilienceStatusPublisherService`, driver nodes, registered next to the tracker with a wiring guard `ResilienceStatusReaderRegistrationTests`) → `driver-resilience-status` DPS topic → per-admin-node `DriverResilienceStatusBridge` → `IDriverResilienceStatusStore` → a resilience section on `DriverStatusPanel.razor` (breaker-OPEN / consecutive-failures / in-flight / staleness chips, read in-process per the self-HubConnection ban). Two tracker truthfulness gaps fixed along the way: `RecordBreakerClose` + `IsBreakerOpen` (breaker-open state is now derivable) and `RecordSuccess` wired onto the invoker success paths (`ConsecutiveFailures` is now a true consecutive counter). Mirrors the driver-health flow one-for-one. *(Original finding:)* The tracker was registered, fed by every retry/breaker event, and consumed by nothing (Admin `/hosts` Stream E.2/E.3 never shipped); the interim mitigation was retry/breaker logging in `DriverResiliencePipelineBuilder.cs`.
|
||||
*Live gate (T10) pending:* docker-dev breaker-open observation on both centrals + staleness dim.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -240,7 +240,9 @@ There is **no bUnit** — 77 razor components have no render/binding coverage, w
|
||||
|
||||
~31 `aria-*`/`role=` attributes across 77 components, concentrated in the modals (`role="dialog"`, `btn-close` aria-labels). Gaps: text-glyph expander buttons (`▼`/`▶` in `UnsTree.razor:49`) with no `aria-expanded`/label; modals have no focus trap or Escape handling; live tails (`Alerts`) have no `aria-live` region; tables lack captions/scope. Form labeling is decent (most inputs have `for`/`id` pairs). Acceptable for an internal ops tool; worth a targeted pass on the tree and the modals if operator diversity matters.
|
||||
|
||||
### U-5 (NEW) — Resilience runtime status has no AdminUI surface; overrides are authored blind
|
||||
### U-5 (NEW) — Resilience runtime status has no AdminUI surface; overrides are authored blind — ✅ REMEDIATED (R2-10)
|
||||
|
||||
**Remediated** on branch `r2/10-resilience-observability`: `DriverResilienceStatusTracker` now flows to the AdminUI through exactly the recommended idiom — a per-node DPS bridge (`DriverResilienceStatusBridge`) → in-process store (`IDriverResilienceStatusStore`, registered beside `IDriverStatusSnapshotStore` in `AddOtOpcUaDriverStatusServices`) → a **resilience section on `DriverStatusPanel.razor`** (embedded on all 8 driver pages, right where the `ResilienceConfig` overrides are authored — closing the feedback loop). It renders breaker-OPEN / consecutive-failures / in-flight / staleness chips, read in-process (self-HubConnection ban honored). The driver-node feed is a periodic 5 s full-snapshot publisher (`DriverResilienceStatusPublisherService`) with a wiring guard. Live gate (T10) — docker-dev breaker-open on both centrals + staleness — pending. *(Original finding below.)*
|
||||
|
||||
The remediation merges made per-driver resilience live in production: #10 (`bacea1a4`) wired `CapabilityInvoker` into every `DriverInstanceActor` dispatch site, and #13 (`75403caa`) applies the AdminUI-authored `ResilienceConfig` overrides from the deploy artifact. But `DriverResilienceStatusTracker` (`src/Core/ZB.MOM.WW.OtOpcUa.Core/Resilience/DriverResilienceStatusTracker.cs`) still has **no reader** — no AdminUI component, hub, or endpoint consumes it (`grep -rn Resilience src/Server/ZB.MOM.WW.OtOpcUa.AdminUI` matches only the authoring form + driver pages). The planned `/hosts` Stream E.2/E.3 status panels never shipped; the interim observability surface is retry/breaker **log lines** added during #10. `archreview/plans/STATUS.md` explicitly records this as "still open as an Admin-UI follow-on."
|
||||
|
||||
|
||||
@@ -39,11 +39,19 @@ the concrete invoker in would reverse that architectural decision.
|
||||
ResilienceConfig change. Per-instance overrides now reach the invoker. See
|
||||
[`FOLLOWUP-13`](FOLLOWUP-13-resilience-config-artifact.md).
|
||||
|
||||
**Observability finding + fix.** The resilience-status tracker has **no reader** (Admin `/hosts`, Phase 6.1 Stream
|
||||
**Observability finding + fix.** The resilience-status tracker had **no reader** (Admin `/hosts`, Phase 6.1 Stream
|
||||
E.2/E.3, was never built) and the pipeline builder logged nothing — so the pipeline ran **invisibly**. Added
|
||||
retry / breaker-open / breaker-close **structured logging** to `DriverResiliencePipelineBuilder` (optional ILogger,
|
||||
wired in the Host DI). This is the operator-facing observability surface AND the prerequisite for any live-verify.
|
||||
|
||||
> **UPDATE (R2-10, branch `r2/10-resilience-observability`):** the tracker now HAS a proper reader. A periodic
|
||||
> 5 s DPS publisher (`DriverResilienceStatusPublisherService`, driver nodes) mirrors each snapshot →
|
||||
> `driver-resilience-status` topic → per-admin `DriverResilienceStatusBridge` → `IDriverResilienceStatusStore` →
|
||||
> breaker/retry/in-flight/staleness chips on `DriverStatusPanel.razor` (mirrors the driver-health flow). Tracker
|
||||
> truthfulness fixed too: `IsBreakerOpen`/`RecordBreakerClose` + `RecordSuccess` on the invoker success paths. The
|
||||
> log-line remains the drill-down; the panel is the primary surface. **The live behavioral gate below (observing the
|
||||
> `circuit-breaker OPENED` line on a dead endpoint) is now folded into R2-10's T10 live-`/run` — still pending.**
|
||||
|
||||
**Verification done (deterministic + local):**
|
||||
- Negative control: unwrapping any site fails the Runtime build with OTOPCUA0001.
|
||||
- Recording-invoker Runtime test: write routes via `ExecuteWriteAsync` with the `IPerCallHostResolver` host;
|
||||
|
||||
@@ -191,3 +191,14 @@ Subscription lifecycle copies the existing pattern exactly: inject the store, su
|
||||
1. Mark U10/U-5 remediated with the branch/commit; update STATUS.md; annotate FOLLOWUP-10 that the reader shipped and (if observed in T10) the live behavioral log-line gate closed. Commit: `docs(archreview): R2-10 resilience observability shipped`.
|
||||
|
||||
**Full-suite gate before merge:** `dotnet test ZB.MOM.WW.OtOpcUa.slnx` green (integration suites need their fixtures per CLAUDE.md; the new tests themselves are fixture-free).
|
||||
|
||||
---
|
||||
|
||||
## Execution deviations (R2-10)
|
||||
|
||||
Executed on branch `r2/10-resilience-observability` (off master `46fedda3`). T1–T9 + T11 completed; T10 deferred-live.
|
||||
|
||||
- **T2 breaker-close test uses a controllable `TimeProvider` (`ManualTimeProvider`), not wall-clock waits.** Driving the real Polly breaker to `OnClosed` requires elapsing the 15 s `BreakDuration` then a successful half-open probe. Rather than a 15 s `Task.Delay`, the test injects a `TimeProvider` that advances both `GetUtcNow` and `GetTimestamp` off one tick counter (`Advance(16 s)`), making the close deterministic and instant. Same self-contained-fake style the repo already uses (`TriePermissionEvaluatorTests`); no new package. This is the one deviation from the plan's "extend the existing breaker test" wording — the existing test only reaches breaker-open, and open→close needs time control.
|
||||
- **T9 panel renders healthy hosts quietly** rather than one always-visible row per host. Per the plan's "healthy is quiet" intent, the resilience section shows: `No resilience activity recorded yet.` when the store has no entries for the instance; `All targets healthy.` when entries exist but none has an active signal; and a chip row **only** for hosts with breaker-OPEN / consecutive-failures>0 / in-flight>0 / stale. Keeps the panel silent under normal operation (the tracker populates an entry for every host that has ever had a successful call, so an always-on per-host row would be noisy).
|
||||
- **T10 (live-`/run` gate) deferred-live.** Code-complete + offline-verified (unit + full-solution build). The docker-dev breaker-open observation on both centrals + recovery-clear + staleness-dim is a heavy serial pass left for the consolidated live session (also the first live sighting of the #10 `circuit-breaker OPENED` log line, closing FOLLOWUP-10's behavioral gate). Marked `deferred-live` in `.tasks.json`.
|
||||
- **T6/T8 registration:** the publisher's `Func<ActorSystem>` is registered via `TryAddSingleton` (idempotent with the one `Program.cs` registers for the alarm-command router, per plan). The resilience bridge intentionally has **no SignalR hub** (the panel reads the store in-process; resilience has no browser-JS consumer) — a deliberate simplification the plan calls for.
|
||||
|
||||
@@ -2,16 +2,16 @@
|
||||
"planPath": "archreview/plans/R2-10-resilience-observability-plan.md",
|
||||
"lastUpdated": "2026-07-12",
|
||||
"tasks": [
|
||||
{ "id": "T1", "subject": "Tracker: LastBreakerClosedUtc + IsBreakerOpen + RecordBreakerClose (Core, TDD in DriverResilienceStatusTrackerTests)", "status": "pending", "blockedBy": [] },
|
||||
{ "id": "T2", "subject": "Builder: OnClosed records breaker-close on the tracker (TDD in DriverResiliencePipelineBuilderTests)", "status": "pending", "blockedBy": ["T1"] },
|
||||
{ "id": "T3", "subject": "Invoker: successful Execute* resets ConsecutiveFailures (TDD in CapabilityInvokerTests)", "status": "pending", "blockedBy": ["T1"] },
|
||||
{ "id": "T4", "subject": "Commons: DriverResilienceStatusChanged record + driver-resilience-status TopicName", "status": "pending", "blockedBy": [] },
|
||||
{ "id": "T5", "subject": "Host: DriverResilienceStatusPublisherService (BuildMessages mapping + 5s DPS timer shell; TDD mapping in Host.IntegrationTests)", "status": "pending", "blockedBy": ["T1", "T4"] },
|
||||
{ "id": "T6", "subject": "Host DI: register the publisher in AddOtOpcUaDriverFactories + ResilienceStatusReaderRegistrationTests wiring guard (tracker HAS a production reader)", "status": "pending", "blockedBy": ["T5"] },
|
||||
{ "id": "T7", "subject": "AdminUI: IDriverResilienceStatusStore + InMemory impl + AddOtOpcUaDriverStatusServices registration (TDD mirroring DriverStatusSnapshotStoreTests)", "status": "pending", "blockedBy": ["T4"] },
|
||||
{ "id": "T8", "subject": "AdminUI: DriverResilienceStatusBridge DPS actor + spawn in WithOtOpcUaSignalRBridges", "status": "pending", "blockedBy": ["T7"] },
|
||||
{ "id": "T9", "subject": "AdminUI: resilience chip section in DriverStatusPanel.razor (in-process store read; no bUnit — verified by T10)", "status": "pending", "blockedBy": ["T7"] },
|
||||
{ "id": "T10", "subject": "Live-/run gate on docker-dev: rebuild BOTH centrals, S7 dead-endpoint breaker-open, both-replica chips, recovery clear, staleness dim", "status": "pending", "blockedBy": ["T6", "T8", "T9"] },
|
||||
{ "id": "T11", "subject": "Docs + review ledger: mark 03/U10 + 04/U-5 remediated; update STATUS.md + FOLLOWUP-10", "status": "pending", "blockedBy": ["T10"] }
|
||||
{ "id": "T1", "subject": "Tracker: LastBreakerClosedUtc + IsBreakerOpen + RecordBreakerClose (Core, TDD in DriverResilienceStatusTrackerTests)", "status": "completed", "blockedBy": [] },
|
||||
{ "id": "T2", "subject": "Builder: OnClosed records breaker-close on the tracker (TDD in DriverResiliencePipelineBuilderTests)", "status": "completed", "blockedBy": ["T1"] },
|
||||
{ "id": "T3", "subject": "Invoker: successful Execute* resets ConsecutiveFailures (TDD in CapabilityInvokerTests)", "status": "completed", "blockedBy": ["T1"] },
|
||||
{ "id": "T4", "subject": "Commons: DriverResilienceStatusChanged record + driver-resilience-status TopicName", "status": "completed", "blockedBy": [] },
|
||||
{ "id": "T5", "subject": "Host: DriverResilienceStatusPublisherService (BuildMessages mapping + 5s DPS timer shell; TDD mapping in Host.IntegrationTests)", "status": "completed", "blockedBy": ["T1", "T4"] },
|
||||
{ "id": "T6", "subject": "Host DI: register the publisher in AddOtOpcUaDriverFactories + ResilienceStatusReaderRegistrationTests wiring guard (tracker HAS a production reader)", "status": "completed", "blockedBy": ["T5"] },
|
||||
{ "id": "T7", "subject": "AdminUI: IDriverResilienceStatusStore + InMemory impl + AddOtOpcUaDriverStatusServices registration (TDD mirroring DriverStatusSnapshotStoreTests)", "status": "completed", "blockedBy": ["T4"] },
|
||||
{ "id": "T8", "subject": "AdminUI: DriverResilienceStatusBridge DPS actor + spawn in WithOtOpcUaSignalRBridges", "status": "completed", "blockedBy": ["T7"] },
|
||||
{ "id": "T9", "subject": "AdminUI: resilience chip section in DriverStatusPanel.razor (in-process store read; no bUnit — verified by T10)", "status": "completed", "blockedBy": ["T7"] },
|
||||
{ "id": "T10", "subject": "Live-/run gate on docker-dev: rebuild BOTH centrals, S7 dead-endpoint breaker-open, both-replica chips, recovery clear, staleness dim", "status": "deferred-live", "blockedBy": ["T6", "T8", "T9"] },
|
||||
{ "id": "T11", "subject": "Docs + review ledger: mark 03/U10 + 04/U-5 remediated; update STATUS.md + FOLLOWUP-10", "status": "completed", "blockedBy": ["T10"] }
|
||||
]
|
||||
}
|
||||
|
||||
@@ -174,6 +174,7 @@ what the unit leg cannot; (c) migrate the 3 xunit v2 holdouts once Akka ships an
|
||||
| **R2-01** | 05/STAB-14 (High regression, rank 1), 05/STAB-15 (rank 3), 05/STAB-8 seam | `r2/01-s7-fault-hardening` (off `1676c8f4`) | `fcdcf0e2`…`bdb7ed41` (13 task commits) | **Code-complete, offline-verified.** All prod changes in `S7Driver.cs`. STAB-14: connect-timeout surfaces as `TimeoutException` (not an escaping OCE) at the `EnsureConnectedAsync`/`InitializeAsync` source + `when(token.IsCancellationRequested)` filters on the read/write ensure-wrappers and all three poll-loop OCE catches — subscription poll loops survive an unreachable-host outage and recover (T2 regression RED→GREEN). STAB-15: `IsS7ConnectionFatal` broadened to the ISO-on-TCP framing surface (TPKT/TPDU/WrongNumberOfBytes/`PlcException{WrongNumberReceivedBytes}`, NOT `InvalidDataException`); mid-PDU cancel marks handle dead; probe classifies+marks-dead under the gate. STAB-8 (S7 part): `EnsureConnectedAsync` documented as the R2-09 connect-throttle seam (note-only). Driver.S7.Tests 246/246. **Live outage gate deferred (T11)** — `S7_1500ConnectTimeoutOutageTests` (env `S7_TIMEOUT_OUTAGE_START_CMD`/`STOP_CMD`, SYN-blackhole) skips clean offline; the connect-*timeout* class the `docker restart` bounce can't produce. (Pre-existing unrelated CLI docs-scan test fails at base — out of scope.) |
|
||||
| **R2-03** | 02/S13 (rank 5), 02/S12, 02/P7 | `r2/03-vt-failure-quality` (off `1676c8f4`) | `b9775485`…`6bb91e8d` (12 task commits) | **Code-complete, offline-verified.** S13: VT script failure/timeout degrades the node to `OpcUaQuality.Bad` (carrying last-known value) instead of freezing on last-Good forever — additive `EvaluationResult.Quality`, bridge historizes `0x80020000u` BadInternalError, published once per Good→Bad transition gated on all deps arrived (inputs-ready, no cold-start flash), recovery force-publishes next Good. S12: apply-boundary `ClearCompiledScripts` racing an in-flight eval into ObjectDisposedException retried once in `RoslynVirtualTagEvaluator` (re-fetch→recompile into current generation); stress guard reproduces pre-fix, deterministically green post-fix. P7: apply-boundary clear gated on a changed deployed expression set (`HashSet.SetEquals`, ordinal) — identical redeploy / rename-with-same-expressions no longer forces full VT recompile. Runtime.Tests 372/372, RoslynVirtualTagEvaluatorTests 20/20. **Deferred (T11/T12):** full cross-suite sweep + docker-dev live-`/run` (fail→Bad→recover + P7 no-recompile spot-check). |
|
||||
| **R2-02** | 01/S-6 (High, rank 2), 01/U-6, 01/S-8=03/S12 (rank 4), 04/C-7, 01/S-7=03/S13 | `r2/02-resilience-config-hardening` (off `1676c8f4`) | `4be13429`…`dd82e740` (17 task commits) | **Code-complete, offline-verified.** **S-6:** parser clamps timeout/retry/breaker overrides (timeout≤0→tier default, breaker==1→2, caps) with diagnostics + belt-and-braces `Build` clamp so pipeline construction can never throw Polly `ValidationException` (new `ResiliencePolicyRanges` in Core.Abstractions); brick-repro + `Build_NeverThrows_ForHostileDirectOptions` + factory-seam wiring proof; AdminUI `Validate()` range warnings. **S-8:** parser forces Write/AlarmAck retry→0 (spec invariant) **and** dispatch flips to `isIdempotent:false` (RED-first on the Runtime wiring test); non-idempotent arm caches the no-retry snapshot once/invoker (01/P-4) + tracker in-flight accounting; `WriteIdempotentAttribute` false claim corrected; retry cells disabled. **U-6:** bulkhead knob deleted end-to-end (migration-free; stored keys ignored) + `Options_properties_are_exactly_the_pipeline_wired_set` inertness guard. **C-7:** `ResilienceFormModel` JsonObject-bag round-trip preserves unknown keys; malformed JSON → `ParseFailed` lock + verbatim `ToJson` + discard button. **S-7:** per-`Create` `Interlocked` options-generation in the pipeline-cache key kills the respawn stale-pipeline race. Core resilience 121/121, AdminUI form 10/10, Analyzers 32/32, Runtime 364/364; 0 production OTOPCUA0001. **Deferred (T15/T18):** docker-dev `:9200` `/run` (warning banner, input lockout, unknown-key survival) + whole-solution sweep. |
|
||||
| **R2-10** | 03/U10=04/U-5 (Medium/Low, action #10) | `r2/10-resilience-observability` (off `46fedda3`) | `205d3980`…`<docs>` (11 task commits) | **Code-complete, offline-verified.** The resilience status tracker now has a production reader, mirroring the driver-health flow. Core truthfulness: `RecordBreakerClose` + derived `IsBreakerOpen` (breaker-open state derivable), `OnClosed` records the close, and `RecordSuccess` wired onto the invoker success paths (`ConsecutiveFailures` is a true consecutive counter, not stuck-forever). Transport: `DriverResilienceStatusChanged` Commons contract → periodic 5 s full-snapshot publisher `DriverResilienceStatusPublisherService` (driver nodes, registered in `AddOtOpcUaDriverFactories` with wiring guard `ResilienceStatusReaderRegistrationTests` — closes the built-but-never-wired class) → `driver-resilience-status` DPS topic → per-admin-node `DriverResilienceStatusBridge` → `IDriverResilienceStatusStore` → resilience chips (breaker-OPEN / consecutive-failures / in-flight / staleness) on `DriverStatusPanel.razor`, read in-process (self-HubConnection ban honored). Core.Tests resilience 31/31 (incl. deterministic breaker-close via a controllable TimeProvider), Host.IntegrationTests publisher+guard 6/6, AdminUI.Tests store 5/5; full-solution build clean. **Deferred (T10):** docker-dev live-`/run` — S7 dead-endpoint breaker-open observed on BOTH centrals + recovery-clear + staleness-dim (also the first live sighting of the #10 `circuit-breaker OPENED` log line, closing FOLLOWUP-10's behavioral gate). |
|
||||
|
||||
**Report anchor drifts to fold into the next report refresh** (from the R2-02 verification pass): (1)
|
||||
`01-core-composition.md` U-6 anchor `DriverResilienceOptions.cs:23-25` → the two bulkhead properties sat at
|
||||
|
||||
@@ -0,0 +1,35 @@
|
||||
namespace ZB.MOM.WW.OtOpcUa.Commons.Messages.Drivers;
|
||||
|
||||
/// <summary>
|
||||
/// Snapshot of one driver instance's live resilience counters for a single target host,
|
||||
/// published periodically to the <c>driver-resilience-status</c> DistributedPubSub topic by
|
||||
/// <c>DriverResilienceStatusPublisherService</c> (Host, driver-role nodes) and consumed by the
|
||||
/// AdminUI <c>DriverResilienceStatusBridge</c> → <c>IDriverResilienceStatusStore</c> →
|
||||
/// resilience section of the driver status panel. Mirrors the <see cref="DriverHealthChanged"/>
|
||||
/// flow one-for-one; a separate topic keeps the two feeds (event-driven health vs. periodic
|
||||
/// resilience) decoupled.
|
||||
/// </summary>
|
||||
/// <param name="DriverInstanceId">Globally-unique driver instance ID (the panel filters on this alone).</param>
|
||||
/// <param name="HostName">Target host the counters aggregate for (per-device breaker isolation key).</param>
|
||||
/// <param name="BreakerOpen">Whether the circuit breaker is currently open for this (instance, host).</param>
|
||||
/// <param name="ConsecutiveFailures">Consecutive pipeline failures — nonzero only during an active retry storm.</param>
|
||||
/// <param name="CurrentInFlight">In-flight capability-call depth against this (instance, host).</param>
|
||||
/// <param name="LastBreakerOpenUtc">UTC of the most recent breaker-open event; null if never.</param>
|
||||
/// <param name="LastSampledUtc">UTC the tracker counters were last updated.</param>
|
||||
/// <param name="PublishedUtc">UTC this snapshot was published (drives staleness detection in the panel).</param>
|
||||
public sealed record DriverResilienceStatusChanged(
|
||||
string DriverInstanceId,
|
||||
string HostName,
|
||||
bool BreakerOpen,
|
||||
int ConsecutiveFailures,
|
||||
int CurrentInFlight,
|
||||
DateTime? LastBreakerOpenUtc,
|
||||
DateTime LastSampledUtc,
|
||||
DateTime PublishedUtc)
|
||||
{
|
||||
/// <summary>
|
||||
/// DPS topic name. Both the Host publisher and the AdminUI bridge reference this single
|
||||
/// constant on the contract so a rename can't silently desynchronise publisher and subscriber.
|
||||
/// </summary>
|
||||
public const string TopicName = "driver-resilience-status";
|
||||
}
|
||||
@@ -41,7 +41,7 @@ public sealed class CapabilityInvoker : IDriverCapabilityInvoker
|
||||
/// pipeline-invalidate can take effect without restarting the invoker.
|
||||
/// </param>
|
||||
/// <param name="driverType">Driver type name for structured-log enrichment (e.g. <c>"Modbus"</c>).</param>
|
||||
/// <param name="statusTracker">Optional resilience-status tracker. When wired, every capability call records start/complete so Admin <c>/hosts</c> can surface <see cref="ResilienceStatusSnapshot.CurrentInFlight"/> as the in-flight-call-depth proxy.</param>
|
||||
/// <param name="statusTracker">Optional resilience-status tracker. When wired, every capability call records start/complete so the AdminUI resilience panel can surface <see cref="ResilienceStatusSnapshot.CurrentInFlight"/> as the in-flight-call-depth proxy, and each successful call resets <see cref="ResilienceStatusSnapshot.ConsecutiveFailures"/> so the counter reflects only an active retry storm.</param>
|
||||
/// <param name="optionsGeneration">
|
||||
/// Options epoch for this invoker (01/S-7). Threaded into every pipeline-cache lookup so a lingering
|
||||
/// old child's late re-cache can never poison a newer invoker's pipeline. Defaults to 0 so existing
|
||||
@@ -81,7 +81,11 @@ public sealed class CapabilityInvoker : IDriverCapabilityInvoker
|
||||
{
|
||||
using (LogContextEnricher.Push(_driverInstanceId, _driverType, capability, LogContextEnricher.NewCorrelationId()))
|
||||
{
|
||||
return await pipeline.ExecuteAsync(callSite, cancellationToken).ConfigureAwait(false);
|
||||
var result = await pipeline.ExecuteAsync(callSite, cancellationToken).ConfigureAwait(false);
|
||||
// Success resets ConsecutiveFailures so the counter is a true "consecutive" gauge
|
||||
// (nonzero only during an active retry storm), not stuck at the last blip's count.
|
||||
_statusTracker?.RecordSuccess(_driverInstanceId, hostName, DateTime.UtcNow);
|
||||
return result;
|
||||
}
|
||||
}
|
||||
finally
|
||||
@@ -106,6 +110,7 @@ public sealed class CapabilityInvoker : IDriverCapabilityInvoker
|
||||
using (LogContextEnricher.Push(_driverInstanceId, _driverType, capability, LogContextEnricher.NewCorrelationId()))
|
||||
{
|
||||
await pipeline.ExecuteAsync(callSite, cancellationToken).ConfigureAwait(false);
|
||||
_statusTracker?.RecordSuccess(_driverInstanceId, hostName, DateTime.UtcNow);
|
||||
}
|
||||
}
|
||||
finally
|
||||
@@ -139,7 +144,9 @@ public sealed class CapabilityInvoker : IDriverCapabilityInvoker
|
||||
{
|
||||
using (LogContextEnricher.Push(_driverInstanceId, _driverType, DriverCapability.Write, LogContextEnricher.NewCorrelationId()))
|
||||
{
|
||||
return await pipeline.ExecuteAsync(callSite, cancellationToken).ConfigureAwait(false);
|
||||
var result = await pipeline.ExecuteAsync(callSite, cancellationToken).ConfigureAwait(false);
|
||||
_statusTracker?.RecordSuccess(_driverInstanceId, hostName, DateTime.UtcNow);
|
||||
return result;
|
||||
}
|
||||
}
|
||||
finally
|
||||
|
||||
@@ -38,14 +38,16 @@ public sealed class DriverResiliencePipelineBuilder
|
||||
/// <param name="timeProvider">Clock source for pipeline timeouts + breaker sampling. Defaults to system.</param>
|
||||
/// <param name="statusTracker">When non-null, every built pipeline wires Polly telemetry into
|
||||
/// the tracker — retries increment <c>ConsecutiveFailures</c>, breaker-open stamps
|
||||
/// <c>LastBreakerOpenUtc</c>, breaker-close resets failures. Feeds Admin <c>/hosts</c> +
|
||||
/// <c>LastBreakerOpenUtc</c>, breaker-close resets failures and stamps
|
||||
/// <c>LastBreakerClosedUtc</c> (so <c>IsBreakerOpen</c> reads false again). Feeds the AdminUI
|
||||
/// resilience panel via the <c>driver-resilience-status</c> DPS publisher +
|
||||
/// the in-flight-call-depth column. Absent tracker means no telemetry (unit tests +
|
||||
/// deployments that don't care about resilience observability).</param>
|
||||
/// <param name="logger">When non-null, retry / circuit-breaker-open / breaker-close events are
|
||||
/// logged (instance + host + capability + attempt/exception). This is the operator-facing
|
||||
/// observability surface for the resilience pipeline until the Admin <c>/hosts</c> reader
|
||||
/// (Phase 6.1 Stream E.2/E.3) ships — without it the pipeline runs silently. Absent logger =
|
||||
/// no resilience logging (unit tests).</param>
|
||||
/// logged (instance + host + capability + attempt/exception). This is the structured-log
|
||||
/// drill-down for the resilience pipeline, complementing the AdminUI resilience panel fed by
|
||||
/// the tracker — without it the pipeline runs silently. Absent logger = no resilience logging
|
||||
/// (unit tests).</param>
|
||||
public DriverResiliencePipelineBuilder(
|
||||
TimeProvider? timeProvider = null,
|
||||
DriverResilienceStatusTracker? statusTracker = null,
|
||||
@@ -186,8 +188,11 @@ public sealed class DriverResiliencePipelineBuilder
|
||||
breakerOptions.OnClosed = args =>
|
||||
{
|
||||
// Closing the breaker means the target recovered — reset the consecutive-
|
||||
// failure counter so Admin UI stops flashing red for this host.
|
||||
tracker?.RecordSuccess(driverInstanceId, hostName, timeProvider.GetUtcNow().UtcDateTime);
|
||||
// failure counter AND stamp the close time so IsBreakerOpen reads false again
|
||||
// (the resilience panel stops showing "Breaker OPEN" for this host).
|
||||
var closedUtc = timeProvider.GetUtcNow().UtcDateTime;
|
||||
tracker?.RecordSuccess(driverInstanceId, hostName, closedUtc);
|
||||
tracker?.RecordBreakerClose(driverInstanceId, hostName, closedUtc);
|
||||
logger?.LogInformation(
|
||||
"Driver resilience circuit-breaker closed (recovered): instance={DriverInstanceId} host={HostName} capability={Capability}",
|
||||
driverInstanceId, hostName, capability);
|
||||
|
||||
@@ -4,15 +4,19 @@ namespace ZB.MOM.WW.OtOpcUa.Core.Resilience;
|
||||
|
||||
/// <summary>
|
||||
/// Process-singleton tracker of live resilience counters per
|
||||
/// <c>(DriverInstanceId, HostName)</c>. Populated by the CapabilityInvoker and the
|
||||
/// MemoryTracking layer; consumed by a HostedService that periodically persists a
|
||||
/// snapshot to the <c>DriverInstanceResilienceStatus</c> table for Admin <c>/hosts</c>.
|
||||
/// <c>(DriverInstanceId, HostName)</c>. Populated by the CapabilityInvoker (call
|
||||
/// start/complete + success reset), the DriverResiliencePipelineBuilder's Polly
|
||||
/// telemetry (retry failure, breaker open/close), and the MemoryTracking layer
|
||||
/// (footprint). Read by <c>DriverResilienceStatusPublisherService</c> (Host, driver
|
||||
/// nodes) which periodically publishes each snapshot to the
|
||||
/// <c>driver-resilience-status</c> DistributedPubSub topic for the AdminUI
|
||||
/// resilience panel.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Per Phase 6.1 Stream E. No DB dependency here — the tracker is pure in-memory so
|
||||
/// tests can exercise it without EF Core or SQL Server. The HostedService that writes
|
||||
/// snapshots lives in the Server project (Stream E.2); the actual SignalR push + Blazor
|
||||
/// page refresh (E.3) lands in a follow-up visual-review PR.
|
||||
/// Pure in-memory (no EF Core / SQL Server dependency) so tests can exercise it directly.
|
||||
/// The operator-facing reader is the DPS publisher named above, mirroring the driver-health
|
||||
/// flow — not the DB-table/HostedService shape originally sketched for Phase 6.1 Stream E
|
||||
/// (R2-10 built the DPS reader instead).
|
||||
/// </remarks>
|
||||
public sealed class DriverResilienceStatusTracker
|
||||
{
|
||||
@@ -62,6 +66,23 @@ public sealed class DriverResilienceStatusTracker
|
||||
(_, existing) => existing with { LastBreakerOpenUtc = utcNow, LastSampledUtc = utcNow });
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Record a circuit-breaker close (recovery) event — the paired counterpart to
|
||||
/// <see cref="RecordBreakerOpen"/>. Stamps <see cref="ResilienceStatusSnapshot.LastBreakerClosedUtc"/>
|
||||
/// so <see cref="ResilienceStatusSnapshot.IsBreakerOpen"/> can report whether the breaker
|
||||
/// is currently open. Called from the pipeline builder's <c>OnClosed</c> hook.
|
||||
/// </summary>
|
||||
/// <param name="driverInstanceId">The driver instance identifier.</param>
|
||||
/// <param name="hostName">The host name.</param>
|
||||
/// <param name="utcNow">The UTC timestamp of the breaker close event.</param>
|
||||
public void RecordBreakerClose(string driverInstanceId, string hostName, DateTime utcNow)
|
||||
{
|
||||
var key = new StatusKey(driverInstanceId, hostName);
|
||||
_status.AddOrUpdate(key,
|
||||
_ => new ResilienceStatusSnapshot { LastBreakerClosedUtc = utcNow, LastSampledUtc = utcNow },
|
||||
(_, existing) => existing with { LastBreakerClosedUtc = utcNow, LastSampledUtc = utcNow });
|
||||
}
|
||||
|
||||
/// <summary>Record a process recycle event (Tier C only).</summary>
|
||||
/// <param name="driverInstanceId">The driver instance identifier.</param>
|
||||
/// <param name="hostName">The host name.</param>
|
||||
@@ -147,8 +168,21 @@ public sealed record ResilienceStatusSnapshot
|
||||
public int ConsecutiveFailures { get; init; }
|
||||
/// <summary>Gets the UTC timestamp of the last circuit-breaker open event, if any.</summary>
|
||||
public DateTime? LastBreakerOpenUtc { get; init; }
|
||||
/// <summary>Gets the UTC timestamp of the last circuit-breaker close (recovery) event, if any.</summary>
|
||||
public DateTime? LastBreakerClosedUtc { get; init; }
|
||||
/// <summary>Gets the UTC timestamp of the last recycle event, if any.</summary>
|
||||
public DateTime? LastRecycleUtc { get; init; }
|
||||
|
||||
/// <summary>
|
||||
/// Gets whether the circuit breaker is currently open for this (instance, host):
|
||||
/// an open event has been recorded and no later close event has superseded it.
|
||||
/// Note: Polly half-opens after <c>BreakDuration</c> without firing <c>OnClosed</c>
|
||||
/// (which fires only on a successful probe), so an idle driver can read "open" until
|
||||
/// the next call self-corrects it.
|
||||
/// </summary>
|
||||
public bool IsBreakerOpen =>
|
||||
LastBreakerOpenUtc is { } open &&
|
||||
(LastBreakerClosedUtc is not { } closed || open > closed);
|
||||
/// <summary>Gets the baseline process footprint in bytes.</summary>
|
||||
public long BaselineFootprintBytes { get; init; }
|
||||
/// <summary>Gets the current process footprint in bytes.</summary>
|
||||
|
||||
+80
-1
@@ -12,6 +12,7 @@
|
||||
@inject IAuthorizationService AuthorizationService
|
||||
@inject IAdminOperationsClient AdminOps
|
||||
@inject IDriverStatusSnapshotStore StatusStore
|
||||
@inject IDriverResilienceStatusStore ResilienceStore
|
||||
|
||||
<section class="panel rise mt-3" style="animation-delay:.04s; @(_stale ? "opacity:0.5;" : "")">
|
||||
<div class="panel-head d-flex align-items-center gap-2">
|
||||
@@ -73,6 +74,58 @@
|
||||
}
|
||||
}
|
||||
|
||||
@* --- Resilience (retry storm / circuit-breaker) — fed by the driver-resilience-status DPS store --- *@
|
||||
@if (Enabled)
|
||||
{
|
||||
@if (_resilience.Count == 0)
|
||||
{
|
||||
<p class="mb-0 mt-3" style="color:var(--ink-faint); font-size:0.85rem">No resilience activity recorded yet.</p>
|
||||
}
|
||||
else
|
||||
{
|
||||
var active = _resilience.Values
|
||||
.Where(h => h.BreakerOpen || h.ConsecutiveFailures > 0 || h.CurrentInFlight > 0 || IsStale(h))
|
||||
.OrderBy(h => h.HostName, StringComparer.Ordinal)
|
||||
.ToList();
|
||||
|
||||
<div class="mt-3">
|
||||
<div style="color:var(--ink-soft); font-size:0.8rem; font-weight:600; text-transform:uppercase; letter-spacing:.03em">Resilience</div>
|
||||
@if (active.Count == 0)
|
||||
{
|
||||
<p class="mb-0 mt-1" style="color:var(--ink-faint); font-size:0.85rem">All targets healthy.</p>
|
||||
}
|
||||
else
|
||||
{
|
||||
@foreach (var host in active)
|
||||
{
|
||||
var stale = IsStale(host);
|
||||
<div class="d-flex flex-wrap gap-2 align-items-baseline mt-1" style="@(stale ? "opacity:0.5;" : "")">
|
||||
<span style="color:var(--ink-soft); font-size:0.85rem"><strong>@host.HostName</strong></span>
|
||||
@if (host.BreakerOpen)
|
||||
{
|
||||
<span class="chip chip-bad">
|
||||
Breaker OPEN@(host.LastBreakerOpenUtc is { } opened ? $" · opened {HumanizeAge(opened)} ago" : "")
|
||||
</span>
|
||||
}
|
||||
@if (host.ConsecutiveFailures > 0)
|
||||
{
|
||||
<span class="chip chip-warn">@host.ConsecutiveFailures consecutive failure@(host.ConsecutiveFailures == 1 ? "" : "s")</span>
|
||||
}
|
||||
@if (host.CurrentInFlight > 0)
|
||||
{
|
||||
<span class="chip chip-idle">in-flight @host.CurrentInFlight</span>
|
||||
}
|
||||
@if (stale)
|
||||
{
|
||||
<span class="chip chip-idle">stale</span>
|
||||
}
|
||||
</div>
|
||||
}
|
||||
}
|
||||
</div>
|
||||
}
|
||||
}
|
||||
|
||||
@* --- Reconnect / Restart action buttons (DriverOperator-gated) --- *@
|
||||
@if (_canOperate && Enabled)
|
||||
{
|
||||
@@ -142,6 +195,10 @@
|
||||
private DriverHealthChanged? _snapshot;
|
||||
private DateTime _lastUpdateUtc = DateTime.MinValue;
|
||||
private bool _stale;
|
||||
|
||||
// Per-host resilience snapshots for THIS instance, keyed by HostName. Fed by the
|
||||
// driver-resilience-status DPS store (read in-process, same house pattern as the health store).
|
||||
private readonly Dictionary<string, DriverResilienceStatusChanged> _resilience = new(StringComparer.Ordinal);
|
||||
private bool _connecting;
|
||||
private string? _error;
|
||||
private System.Threading.Timer? _timer;
|
||||
@@ -195,6 +252,11 @@
|
||||
_snapshot = snap;
|
||||
_lastUpdateUtc = DateTime.UtcNow;
|
||||
}
|
||||
|
||||
// Prime + subscribe the resilience store (same in-process read pattern as the health store).
|
||||
ResilienceStore.SnapshotChanged += OnResilienceChanged;
|
||||
foreach (var r in ResilienceStore.GetForInstance(DriverInstanceId))
|
||||
_resilience[r.HostName] = r;
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
@@ -219,6 +281,22 @@
|
||||
InvokeAsync(StateHasChanged);
|
||||
}
|
||||
|
||||
// Invoked by the resilience store (on the bridge actor's thread) for every (instance, host);
|
||||
// ignore snapshots for other instances and marshal onto the render sync context.
|
||||
private void OnResilienceChanged(DriverResilienceStatusChanged msg)
|
||||
{
|
||||
if (!string.Equals(msg.DriverInstanceId, DriverInstanceId, StringComparison.Ordinal))
|
||||
return;
|
||||
|
||||
_resilience[msg.HostName] = msg;
|
||||
InvokeAsync(StateHasChanged);
|
||||
}
|
||||
|
||||
// A host row is stale once its last publish is older than 15 s (3× the 5 s publish interval) —
|
||||
// covers driver-node death, where entries freeze. Re-evaluated by the existing 5 s render timer.
|
||||
private static bool IsStale(DriverResilienceStatusChanged h) =>
|
||||
(DateTime.UtcNow - h.PublishedUtc).TotalSeconds > 15;
|
||||
|
||||
private async Task ReconnectAsync()
|
||||
{
|
||||
_busyReconnect = true;
|
||||
@@ -300,8 +378,9 @@
|
||||
|
||||
public async ValueTask DisposeAsync()
|
||||
{
|
||||
// Unsubscribe first so the singleton store can't invoke a handler on a disposed component.
|
||||
// Unsubscribe first so the singleton stores can't invoke a handler on a disposed component.
|
||||
StatusStore.SnapshotChanged -= OnSnapshotChanged;
|
||||
ResilienceStore.SnapshotChanged -= OnResilienceChanged;
|
||||
// Drain BOTH timers so an in-flight callback can't invoke StateHasChanged on a component
|
||||
// that's already gone. System.Threading.Timer's async dispose awaits any in-flight
|
||||
// callback (.NET 6+).
|
||||
|
||||
@@ -0,0 +1,39 @@
|
||||
using Akka.Actor;
|
||||
using Akka.Cluster.Tools.PublishSubscribe;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Drivers;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.AdminUI.Hubs;
|
||||
|
||||
/// <summary>
|
||||
/// Akka actor that subscribes to the <c>driver-resilience-status</c> DistributedPubSub topic and
|
||||
/// upserts every <see cref="DriverResilienceStatusChanged"/> snapshot into the in-process
|
||||
/// <see cref="IDriverResilienceStatusStore"/>. The driver status panel's resilience section reads
|
||||
/// that store directly (per the house ban on server-side self-targeted SignalR connections), so —
|
||||
/// unlike <see cref="DriverStatusSignalRBridge"/> — there is no SignalR hub here; resilience has no
|
||||
/// browser-JS consumer. Spawned per admin-role node by <c>WithOtOpcUaSignalRBridges</c>.
|
||||
/// </summary>
|
||||
public sealed class DriverResilienceStatusBridge : ReceiveActor
|
||||
{
|
||||
public const string TopicName = DriverResilienceStatusChanged.TopicName;
|
||||
|
||||
private readonly IDriverResilienceStatusStore _store;
|
||||
|
||||
/// <summary>Creates actor props for a <see cref="DriverResilienceStatusBridge"/>.</summary>
|
||||
/// <param name="store">Resilience status store updated on every forwarded snapshot.</param>
|
||||
/// <returns>An Akka.NET <see cref="Props"/> instance for spawning the bridge actor.</returns>
|
||||
public static Props Props(IDriverResilienceStatusStore store) =>
|
||||
Akka.Actor.Props.Create(() => new DriverResilienceStatusBridge(store));
|
||||
|
||||
/// <summary>Initializes a new instance of <see cref="DriverResilienceStatusBridge"/>.</summary>
|
||||
/// <param name="store">Resilience status store updated on every forwarded snapshot.</param>
|
||||
public DriverResilienceStatusBridge(IDriverResilienceStatusStore store)
|
||||
{
|
||||
_store = store;
|
||||
Receive<DriverResilienceStatusChanged>(msg => _store.Upsert(msg));
|
||||
Receive<SubscribeAck>(_ => { /* DPS confirmation */ });
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
protected override void PreStart() =>
|
||||
DistributedPubSub.Get(Context.System).Mediator.Tell(new Subscribe(TopicName, Self));
|
||||
}
|
||||
@@ -11,6 +11,7 @@ public static class HubServiceCollectionExtensions
|
||||
public const string AlertSignalRBridgeName = "alert-signalr-bridge";
|
||||
public const string ScriptLogSignalRBridgeName = "script-log-signalr-bridge";
|
||||
public const string DriverStatusSignalRBridgeName = "driver-status-signalr-bridge";
|
||||
public const string DriverResilienceStatusBridgeName = "driver-resilience-status-bridge";
|
||||
|
||||
/// <summary>
|
||||
/// Registers the in-process live-push services the AdminUI's Blazor Server panels read
|
||||
@@ -28,6 +29,7 @@ public static class HubServiceCollectionExtensions
|
||||
public static IServiceCollection AddOtOpcUaDriverStatusServices(this IServiceCollection services)
|
||||
{
|
||||
services.AddSingleton<IDriverStatusSnapshotStore, InMemoryDriverStatusSnapshotStore>();
|
||||
services.AddSingleton<IDriverResilienceStatusStore, InMemoryDriverResilienceStatusStore>();
|
||||
services.AddSingleton(typeof(IInProcessBroadcaster<>), typeof(InProcessBroadcaster<>));
|
||||
return services;
|
||||
}
|
||||
@@ -70,6 +72,12 @@ public static class HubServiceCollectionExtensions
|
||||
var driverStatusStore = resolver.GetService<IDriverStatusSnapshotStore>();
|
||||
var driverStatusBridge = system.ActorOf(DriverStatusSignalRBridge.Props(driverStatusHub, driverStatusStore), DriverStatusSignalRBridgeName);
|
||||
registry.Register<DriverStatusSignalRBridgeKey>(driverStatusBridge);
|
||||
|
||||
// Resilience-status bridge: DPS topic -> in-process store (no SignalR hub — the panel reads
|
||||
// the store directly, and resilience has no browser-JS consumer).
|
||||
var resilienceStore = resolver.GetService<IDriverResilienceStatusStore>();
|
||||
var resilienceBridge = system.ActorOf(DriverResilienceStatusBridge.Props(resilienceStore), DriverResilienceStatusBridgeName);
|
||||
registry.Register<DriverResilienceStatusBridgeKey>(resilienceBridge);
|
||||
});
|
||||
return builder;
|
||||
}
|
||||
@@ -80,3 +88,4 @@ public sealed class FleetStatusSignalRBridgeKey { }
|
||||
public sealed class AlertSignalRBridgeKey { }
|
||||
public sealed class ScriptLogSignalRBridgeKey { }
|
||||
public sealed class DriverStatusSignalRBridgeKey { }
|
||||
public sealed class DriverResilienceStatusBridgeKey { }
|
||||
|
||||
@@ -0,0 +1,36 @@
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Drivers;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.AdminUI.Hubs;
|
||||
|
||||
/// <summary>
|
||||
/// Singleton last-snapshot-per-<c>(instance, host)</c> cache of driver resilience status
|
||||
/// (breaker open / consecutive failures / in-flight depth). Populated by
|
||||
/// <c>DriverResilienceStatusBridge</c> as it forwards <c>driver-resilience-status</c> DPS
|
||||
/// messages; read in-process by the driver status panel's resilience section via
|
||||
/// <see cref="SnapshotChanged"/> + <see cref="GetForInstance"/>. Mirrors
|
||||
/// <see cref="IDriverStatusSnapshotStore"/> — the panel reads this singleton directly instead of
|
||||
/// opening a self-targeted SignalR connection (which a server-side Blazor component cannot reach
|
||||
/// behind a reverse proxy).
|
||||
/// </summary>
|
||||
public interface IDriverResilienceStatusStore
|
||||
{
|
||||
/// <summary>Stores or replaces the last-known resilience snapshot for a <c>(instance, host)</c> pair.</summary>
|
||||
/// <param name="snapshot">The resilience status snapshot to store.</param>
|
||||
void Upsert(DriverResilienceStatusChanged snapshot);
|
||||
|
||||
/// <summary>Returns every stored host snapshot for one driver instance (empty if none).</summary>
|
||||
/// <param name="driverInstanceId">The driver instance's stable logical ID.</param>
|
||||
/// <returns>The per-host resilience snapshots for that instance.</returns>
|
||||
IReadOnlyList<DriverResilienceStatusChanged> GetForInstance(string driverInstanceId);
|
||||
|
||||
/// <summary>Returns a point-in-time snapshot of every tracked <c>(instance, host)</c> pair.</summary>
|
||||
/// <returns>A read-only collection of the last-known resilience snapshot for each pair.</returns>
|
||||
IReadOnlyCollection<DriverResilienceStatusChanged> GetAll();
|
||||
|
||||
/// <summary>
|
||||
/// Raised after every <see cref="Upsert"/> with the just-stored snapshot. Lets the Blazor Server
|
||||
/// driver status panel receive live updates by reading this singleton directly. Handlers run on
|
||||
/// the caller's thread (the bridge actor), so subscribers must marshal to their own sync context.
|
||||
/// </summary>
|
||||
event Action<DriverResilienceStatusChanged>? SnapshotChanged;
|
||||
}
|
||||
@@ -0,0 +1,33 @@
|
||||
using System.Collections.Concurrent;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Drivers;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.AdminUI.Hubs;
|
||||
|
||||
/// <summary>
|
||||
/// Thread-safe in-memory implementation of <see cref="IDriverResilienceStatusStore"/>.
|
||||
/// Keyed by <c>(DriverInstanceId, HostName)</c>; last write wins.
|
||||
/// </summary>
|
||||
public sealed class InMemoryDriverResilienceStatusStore : IDriverResilienceStatusStore
|
||||
{
|
||||
private readonly ConcurrentDictionary<(string Instance, string Host), DriverResilienceStatusChanged> _byPair = new();
|
||||
|
||||
/// <inheritdoc />
|
||||
public event Action<DriverResilienceStatusChanged>? SnapshotChanged;
|
||||
|
||||
/// <inheritdoc />
|
||||
public void Upsert(DriverResilienceStatusChanged snapshot)
|
||||
{
|
||||
_byPair[(snapshot.DriverInstanceId, snapshot.HostName)] = snapshot;
|
||||
// Capture-then-invoke so a concurrent unsubscribe can't null the delegate mid-raise.
|
||||
SnapshotChanged?.Invoke(snapshot);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public IReadOnlyList<DriverResilienceStatusChanged> GetForInstance(string driverInstanceId) =>
|
||||
_byPair.Values
|
||||
.Where(s => string.Equals(s.DriverInstanceId, driverInstanceId, StringComparison.Ordinal))
|
||||
.ToList();
|
||||
|
||||
/// <inheritdoc />
|
||||
public IReadOnlyCollection<DriverResilienceStatusChanged> GetAll() => _byPair.Values.ToArray();
|
||||
}
|
||||
@@ -70,6 +70,13 @@ public static class DriverFactoryBootstrap
|
||||
logger: sp.GetService<ILoggerFactory>()?.CreateLogger("ZB.MOM.WW.OtOpcUa.Core.Resilience.DriverCapabilityInvokerFactory"));
|
||||
});
|
||||
|
||||
// The tracker's operator-facing reader: a periodic DPS publisher that mirrors each snapshot to the
|
||||
// AdminUI resilience panel (R2-10 — closes the "tracker fed by everything, read by nothing" gap).
|
||||
// Lives on driver nodes exactly where the tracker exists. The lazy Func<ActorSystem> is idempotent
|
||||
// with the one Program.cs registers for the alarm-command router.
|
||||
services.TryAddSingleton<Func<Akka.Actor.ActorSystem>>(sp => () => sp.GetRequiredService<Akka.Actor.ActorSystem>());
|
||||
services.AddHostedService<DriverResilienceStatusPublisherService>();
|
||||
|
||||
// Driver nodes also carry the probe set so a fused admin,driver node has it; the admin-only
|
||||
// case is covered by Program.cs calling AddOtOpcUaDriverProbes() in the hasAdmin block.
|
||||
services.AddOtOpcUaDriverProbes();
|
||||
|
||||
@@ -0,0 +1,93 @@
|
||||
using Akka.Actor;
|
||||
using Akka.Cluster.Tools.PublishSubscribe;
|
||||
using Microsoft.Extensions.Hosting;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Drivers;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Resilience;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Host.Drivers;
|
||||
|
||||
/// <summary>
|
||||
/// The operator-facing reader for <see cref="DriverResilienceStatusTracker"/>. Runs on driver-role
|
||||
/// nodes (registered next to the tracker it reads) and every <see cref="_interval"/> publishes the
|
||||
/// full tracker snapshot — one <see cref="DriverResilienceStatusChanged"/> per <c>(instance, host)</c>
|
||||
/// — to the <c>driver-resilience-status</c> DistributedPubSub topic. The AdminUI
|
||||
/// <c>DriverResilienceStatusBridge</c> subscribes and feeds the resilience section of the driver
|
||||
/// status panel. Mirrors the driver-health flow; publishing the full set every tick self-heals
|
||||
/// late-subscribing admin nodes and gives the panel staleness detection for free.
|
||||
/// </summary>
|
||||
public sealed class DriverResilienceStatusPublisherService : BackgroundService
|
||||
{
|
||||
private static readonly TimeSpan DefaultInterval = TimeSpan.FromSeconds(5);
|
||||
|
||||
private readonly DriverResilienceStatusTracker _tracker;
|
||||
private readonly Func<ActorSystem> _actorSystemAccessor;
|
||||
private readonly ILogger<DriverResilienceStatusPublisherService> _logger;
|
||||
private readonly TimeSpan _interval;
|
||||
|
||||
/// <summary>Initializes a new instance of <see cref="DriverResilienceStatusPublisherService"/>.</summary>
|
||||
/// <param name="tracker">The process-singleton resilience-status tracker to read each tick.</param>
|
||||
/// <param name="actorSystemAccessor">Lazy accessor for the Akka <see cref="ActorSystem"/> whose DPS mediator publishes the snapshots.</param>
|
||||
/// <param name="logger">Logger for publish diagnostics.</param>
|
||||
/// <param name="interval">Publish cadence; defaults to 5 s when null.</param>
|
||||
public DriverResilienceStatusPublisherService(
|
||||
DriverResilienceStatusTracker tracker,
|
||||
Func<ActorSystem> actorSystemAccessor,
|
||||
ILogger<DriverResilienceStatusPublisherService> logger,
|
||||
TimeSpan? interval = null)
|
||||
{
|
||||
_tracker = tracker;
|
||||
_actorSystemAccessor = actorSystemAccessor;
|
||||
_logger = logger;
|
||||
_interval = interval ?? DefaultInterval;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Map the tracker's current snapshot to one wire message per <c>(instance, host)</c> pair.
|
||||
/// Pure + unit-testable; the timer/publish shell in <see cref="ExecuteAsync"/> calls this each tick.
|
||||
/// </summary>
|
||||
/// <param name="tracker">The tracker to read.</param>
|
||||
/// <param name="publishedUtc">The publish timestamp stamped on every emitted message.</param>
|
||||
/// <returns>One <see cref="DriverResilienceStatusChanged"/> per tracked pair (empty when the tracker is empty).</returns>
|
||||
public static IReadOnlyList<DriverResilienceStatusChanged> BuildMessages(
|
||||
DriverResilienceStatusTracker tracker, DateTime publishedUtc) =>
|
||||
tracker.Snapshot()
|
||||
.Select(entry => new DriverResilienceStatusChanged(
|
||||
entry.DriverInstanceId,
|
||||
entry.HostName,
|
||||
entry.Snapshot.IsBreakerOpen,
|
||||
entry.Snapshot.ConsecutiveFailures,
|
||||
entry.Snapshot.CurrentInFlight,
|
||||
entry.Snapshot.LastBreakerOpenUtc,
|
||||
entry.Snapshot.LastSampledUtc,
|
||||
publishedUtc))
|
||||
.ToList();
|
||||
|
||||
/// <inheritdoc />
|
||||
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
|
||||
{
|
||||
using var timer = new PeriodicTimer(_interval);
|
||||
while (await timer.WaitForNextTickAsync(stoppingToken).ConfigureAwait(false))
|
||||
{
|
||||
try
|
||||
{
|
||||
var messages = BuildMessages(_tracker, DateTime.UtcNow);
|
||||
if (messages.Count == 0)
|
||||
continue;
|
||||
|
||||
var mediator = DistributedPubSub.Get(_actorSystemAccessor()).Mediator;
|
||||
foreach (var message in messages)
|
||||
mediator.Tell(new Publish(DriverResilienceStatusChanged.TopicName, message));
|
||||
}
|
||||
catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested)
|
||||
{
|
||||
break;
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
// A transient DPS / actor-system hiccup must never kill the publisher — the next tick retries.
|
||||
_logger.LogWarning(ex, "DriverResilienceStatusPublisherService: publish tick failed; retrying next interval");
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -235,6 +235,63 @@ public sealed class CapabilityInvokerTests
|
||||
builder.CachedPipelineCount.ShouldBe(1, "two writes to the same host share one cached no-retry pipeline");
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// R2-10: a successful capability call must reset <c>ConsecutiveFailures</c> so the counter
|
||||
/// is a true "consecutive" gauge — nonzero only during an active retry storm, not stuck at the
|
||||
/// last blip's count forever. Previously only the breaker's <c>OnClosed</c> reset it, so a
|
||||
/// transient 2-retry blip that never opened the breaker read "2 failures" indefinitely.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task ExecuteAsync_OnSuccess_ResetsConsecutiveFailures()
|
||||
{
|
||||
var tracker = new DriverResilienceStatusTracker();
|
||||
tracker.RecordFailure("drv-test", "host-1", DateTime.UtcNow);
|
||||
tracker.RecordFailure("drv-test", "host-1", DateTime.UtcNow);
|
||||
tracker.TryGet("drv-test", "host-1")!.ConsecutiveFailures.ShouldBe(2, "seeded failures");
|
||||
|
||||
var invoker = new CapabilityInvoker(
|
||||
new DriverResiliencePipelineBuilder(),
|
||||
"drv-test",
|
||||
() => new DriverResilienceOptions { Tier = DriverTier.A },
|
||||
statusTracker: tracker);
|
||||
|
||||
await invoker.ExecuteAsync(
|
||||
DriverCapability.Read,
|
||||
"host-1",
|
||||
_ => ValueTask.FromResult(1),
|
||||
CancellationToken.None);
|
||||
|
||||
tracker.TryGet("drv-test", "host-1")!.ConsecutiveFailures.ShouldBe(0,
|
||||
"a successful capability call must clear the consecutive-failure counter");
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// R2-10 companion: a successful non-idempotent write must also reset the consecutive-failure
|
||||
/// counter on the caller's host (the write arm has its own success path).
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task ExecuteWriteAsync_NonIdempotent_OnSuccess_ResetsConsecutiveFailures()
|
||||
{
|
||||
var tracker = new DriverResilienceStatusTracker();
|
||||
tracker.RecordFailure("drv-test", "host-1", DateTime.UtcNow);
|
||||
tracker.TryGet("drv-test", "host-1")!.ConsecutiveFailures.ShouldBe(1, "seeded failure");
|
||||
|
||||
var invoker = new CapabilityInvoker(
|
||||
new DriverResiliencePipelineBuilder(),
|
||||
"drv-test",
|
||||
() => new DriverResilienceOptions { Tier = DriverTier.A },
|
||||
statusTracker: tracker);
|
||||
|
||||
await invoker.ExecuteWriteAsync(
|
||||
"host-1",
|
||||
isIdempotent: false,
|
||||
_ => ValueTask.FromResult(0),
|
||||
CancellationToken.None);
|
||||
|
||||
tracker.TryGet("drv-test", "host-1")!.ConsecutiveFailures.ShouldBe(0,
|
||||
"a successful non-idempotent write must clear the consecutive-failure counter");
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Core-009 regression — companion consistency assertion: the non-idempotent branch must
|
||||
/// not observe two different option snapshots even if the accessor's returned value changes
|
||||
|
||||
+53
@@ -277,6 +277,44 @@ public sealed class DriverResiliencePipelineBuilderTests
|
||||
snap!.LastBreakerOpenUtc.ShouldNotBeNull();
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// R2-10 truthfulness fix: after the breaker opens and later recovers (a successful probe
|
||||
/// past <c>BreakDuration</c>), the pipeline builder's <c>OnClosed</c> hook must record the
|
||||
/// breaker close on the tracker — so <see cref="ResilienceStatusSnapshot.IsBreakerOpen"/>
|
||||
/// reads false again and <see cref="ResilienceStatusSnapshot.LastBreakerClosedUtc"/> is stamped.
|
||||
/// Uses a controllable <see cref="TimeProvider"/> so the 15 s break duration elapses deterministically.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task Tracker_RecordsBreakerClose_OnRecovery()
|
||||
{
|
||||
var clock = new ManualTimeProvider(new DateTime(2026, 7, 13, 0, 0, 0, DateTimeKind.Utc));
|
||||
var tracker = new DriverResilienceStatusTracker();
|
||||
var builder = new DriverResiliencePipelineBuilder(timeProvider: clock, statusTracker: tracker);
|
||||
var pipeline = builder.GetOrCreate("drv-trk", "host-c", DriverCapability.Write, TierAOptions);
|
||||
|
||||
// Drive the breaker open with the tier's throughput of failures.
|
||||
var threshold = TierAOptions.Resolve(DriverCapability.Write).BreakerFailureThreshold;
|
||||
for (var i = 0; i < threshold; i++)
|
||||
{
|
||||
await Should.ThrowAsync<InvalidOperationException>(async () =>
|
||||
await pipeline.ExecuteAsync(async _ =>
|
||||
{
|
||||
await Task.Yield();
|
||||
throw new InvalidOperationException("boom");
|
||||
}));
|
||||
}
|
||||
|
||||
tracker.TryGet("drv-trk", "host-c")!.IsBreakerOpen.ShouldBeTrue("breaker must be open after the failure threshold");
|
||||
|
||||
// Elapse the break duration (15 s) so the breaker half-opens, then let a probe succeed → OnClosed.
|
||||
clock.Advance(TimeSpan.FromSeconds(16));
|
||||
await pipeline.ExecuteAsync(async _ => await Task.Yield());
|
||||
|
||||
var snap = tracker.TryGet("drv-trk", "host-c")!;
|
||||
snap.LastBreakerClosedUtc.ShouldNotBeNull("OnClosed must stamp the close time on recovery");
|
||||
snap.IsBreakerOpen.ShouldBeFalse("the breaker must read closed after a successful recovery probe");
|
||||
}
|
||||
|
||||
/// <summary>Verifies that the tracker isolates counters per host.</summary>
|
||||
[Fact]
|
||||
public async Task Tracker_IsolatesCounters_PerHost()
|
||||
@@ -404,6 +442,21 @@ public sealed class DriverResiliencePipelineBuilderTests
|
||||
attempts.ShouldBe(1, "the new generation must honor optsB (retryCount 0), not the stale optsA re-cache");
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// A controllable <see cref="TimeProvider"/> whose clock advances only on <see cref="Advance"/>.
|
||||
/// Drives both wall-clock (<see cref="GetUtcNow"/>) and monotonic (<see cref="GetTimestamp"/>)
|
||||
/// reads off the same tick counter so Polly's circuit-breaker break-duration timing is deterministic.
|
||||
/// </summary>
|
||||
private sealed class ManualTimeProvider : TimeProvider
|
||||
{
|
||||
private long _ticks;
|
||||
public ManualTimeProvider(DateTime start) => _ticks = start.Ticks;
|
||||
public void Advance(TimeSpan by) => _ticks += by.Ticks;
|
||||
public override DateTimeOffset GetUtcNow() => new(_ticks, TimeSpan.Zero);
|
||||
public override long GetTimestamp() => _ticks;
|
||||
public override long TimestampFrequency => TimeSpan.TicksPerSecond;
|
||||
}
|
||||
|
||||
/// <summary>Minimal in-memory <see cref="ILogger"/> capturing (level, formatted-message) pairs so the
|
||||
/// resilience logging contract can be asserted without a real logging provider.</summary>
|
||||
private sealed class CapturingLogger : ILogger
|
||||
|
||||
+35
@@ -55,6 +55,41 @@ public sealed class DriverResilienceStatusTrackerTests
|
||||
tracker.TryGet("drv", "host")!.LastBreakerOpenUtc.ShouldBe(Now);
|
||||
}
|
||||
|
||||
/// <summary>Verifies that RecordBreakerClose stamps the close timestamp.</summary>
|
||||
[Fact]
|
||||
public void RecordBreakerClose_StampsCloseTime()
|
||||
{
|
||||
var tracker = new DriverResilienceStatusTracker();
|
||||
tracker.RecordBreakerOpen("drv", "host", Now);
|
||||
|
||||
tracker.RecordBreakerClose("drv", "host", Now.AddSeconds(20));
|
||||
|
||||
tracker.TryGet("drv", "host")!.LastBreakerClosedUtc.ShouldBe(Now.AddSeconds(20));
|
||||
}
|
||||
|
||||
/// <summary>Verifies that IsBreakerOpen tracks open → close → reopen transitions.</summary>
|
||||
[Fact]
|
||||
public void IsBreakerOpen_TracksOpenCloseReopen()
|
||||
{
|
||||
var tracker = new DriverResilienceStatusTracker();
|
||||
|
||||
// A snapshot with no breaker event is never "open".
|
||||
tracker.RecordFailure("drv", "host", Now);
|
||||
tracker.TryGet("drv", "host")!.IsBreakerOpen.ShouldBeFalse();
|
||||
|
||||
// Opened.
|
||||
tracker.RecordBreakerOpen("drv", "host", Now.AddSeconds(1));
|
||||
tracker.TryGet("drv", "host")!.IsBreakerOpen.ShouldBeTrue();
|
||||
|
||||
// Closed later → no longer open.
|
||||
tracker.RecordBreakerClose("drv", "host", Now.AddSeconds(2));
|
||||
tracker.TryGet("drv", "host")!.IsBreakerOpen.ShouldBeFalse();
|
||||
|
||||
// Re-opened after the last close → open again.
|
||||
tracker.RecordBreakerOpen("drv", "host", Now.AddSeconds(3));
|
||||
tracker.TryGet("drv", "host")!.IsBreakerOpen.ShouldBeTrue();
|
||||
}
|
||||
|
||||
/// <summary>Verifies that RecordRecycle populates the LastRecycleUtc timestamp.</summary>
|
||||
[Fact]
|
||||
public void RecordRecycle_Populates_LastRecycleUtc()
|
||||
|
||||
@@ -0,0 +1,87 @@
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.OtOpcUa.AdminUI.Hubs;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Drivers;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.AdminUI.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Covers the in-process push contract the Blazor Server driver status panel's resilience section
|
||||
/// relies on: <see cref="IDriverResilienceStatusStore.SnapshotChanged"/> fires on every
|
||||
/// <see cref="IDriverResilienceStatusStore.Upsert"/>, keyed per <c>(instance, host)</c> with
|
||||
/// last-write-wins, and <c>GetForInstance</c> returns only the queried instance's hosts. Mirrors
|
||||
/// <see cref="DriverStatusSnapshotStoreTests"/>; the panel reads this store directly (no
|
||||
/// self-targeted SignalR connection).
|
||||
/// </summary>
|
||||
public sealed class DriverResilienceStatusStoreTests
|
||||
{
|
||||
private static DriverResilienceStatusChanged Msg(
|
||||
string instance, string host, bool breakerOpen = false, int failures = 0) =>
|
||||
new(instance, host, breakerOpen, failures, 0, null,
|
||||
new DateTime(2026, 7, 13, 0, 0, 0, DateTimeKind.Utc),
|
||||
new DateTime(2026, 7, 13, 0, 0, 0, DateTimeKind.Utc));
|
||||
|
||||
[Fact]
|
||||
public void Upsert_raises_SnapshotChanged_with_the_stored_snapshot()
|
||||
{
|
||||
var store = new InMemoryDriverResilienceStatusStore();
|
||||
var received = new List<DriverResilienceStatusChanged>();
|
||||
store.SnapshotChanged += received.Add;
|
||||
|
||||
var msg = Msg("drv-1", "host-a", breakerOpen: true);
|
||||
store.Upsert(msg);
|
||||
|
||||
received.Count.ShouldBe(1);
|
||||
received[0].ShouldBeSameAs(msg);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Upsert_is_last_write_wins_per_instance_host()
|
||||
{
|
||||
var store = new InMemoryDriverResilienceStatusStore();
|
||||
store.Upsert(Msg("drv-1", "host-a", failures: 3));
|
||||
store.Upsert(Msg("drv-1", "host-a", failures: 0));
|
||||
|
||||
var hosts = store.GetForInstance("drv-1");
|
||||
hosts.Count.ShouldBe(1);
|
||||
hosts[0].ConsecutiveFailures.ShouldBe(0);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void GetForInstance_returns_only_the_queried_instances_hosts()
|
||||
{
|
||||
var store = new InMemoryDriverResilienceStatusStore();
|
||||
store.Upsert(Msg("drv-1", "host-a"));
|
||||
store.Upsert(Msg("drv-1", "host-b"));
|
||||
store.Upsert(Msg("drv-2", "host-a"));
|
||||
|
||||
var hosts = store.GetForInstance("drv-1");
|
||||
|
||||
hosts.Count.ShouldBe(2);
|
||||
hosts.ShouldAllBe(h => h.DriverInstanceId == "drv-1");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void GetForInstance_returns_empty_for_unknown_instance()
|
||||
{
|
||||
var store = new InMemoryDriverResilienceStatusStore();
|
||||
store.Upsert(Msg("drv-1", "host-a"));
|
||||
|
||||
store.GetForInstance("nope").ShouldBeEmpty();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Unsubscribed_handler_stops_receiving_after_removal()
|
||||
{
|
||||
var store = new InMemoryDriverResilienceStatusStore();
|
||||
var count = 0;
|
||||
void Handler(DriverResilienceStatusChanged _) => count++;
|
||||
|
||||
store.SnapshotChanged += Handler;
|
||||
store.Upsert(Msg("drv-1", "host-a"));
|
||||
store.SnapshotChanged -= Handler;
|
||||
store.Upsert(Msg("drv-1", "host-a"));
|
||||
|
||||
count.ShouldBe(1);
|
||||
}
|
||||
}
|
||||
+56
@@ -0,0 +1,56 @@
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Resilience;
|
||||
using ZB.MOM.WW.OtOpcUa.Host.Drivers;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Host.IntegrationTests;
|
||||
|
||||
/// <summary>
|
||||
/// Pure-DI (no fixture) coverage of the publisher's snapshot→message mapping — the only unit-testable
|
||||
/// seam of <see cref="DriverResilienceStatusPublisherService"/>. The DPS <c>Publish</c> shell is
|
||||
/// verified by the R2-10 live gate; this asserts the mapping the shell publishes.
|
||||
/// </summary>
|
||||
public sealed class DriverResilienceStatusPublisherServiceTests
|
||||
{
|
||||
private static readonly DateTime Published = new(2026, 7, 13, 9, 0, 0, DateTimeKind.Utc);
|
||||
private static readonly DateTime Sampled = new(2026, 7, 13, 8, 59, 0, DateTimeKind.Utc);
|
||||
|
||||
[Fact]
|
||||
public void BuildMessages_EmptyTracker_ReturnsEmpty()
|
||||
{
|
||||
var tracker = new DriverResilienceStatusTracker();
|
||||
|
||||
var messages = DriverResilienceStatusPublisherService.BuildMessages(tracker, Published);
|
||||
|
||||
messages.ShouldBeEmpty();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void BuildMessages_MapsTrackerSnapshot_OneMessagePerInstanceHost()
|
||||
{
|
||||
var tracker = new DriverResilienceStatusTracker();
|
||||
// (drv-1, host-a): an open breaker with a standing failure count.
|
||||
tracker.RecordFailure("drv-1", "host-a", Sampled);
|
||||
tracker.RecordFailure("drv-1", "host-a", Sampled);
|
||||
tracker.RecordBreakerOpen("drv-1", "host-a", Sampled);
|
||||
// (drv-1, host-b): a healthy host with in-flight calls.
|
||||
tracker.RecordCallStart("drv-1", "host-b");
|
||||
|
||||
var messages = DriverResilienceStatusPublisherService.BuildMessages(tracker, Published);
|
||||
|
||||
messages.Count.ShouldBe(2);
|
||||
|
||||
var open = messages.Single(m => m.HostName == "host-a");
|
||||
open.DriverInstanceId.ShouldBe("drv-1");
|
||||
open.BreakerOpen.ShouldBeTrue();
|
||||
open.ConsecutiveFailures.ShouldBe(2);
|
||||
open.LastBreakerOpenUtc.ShouldBe(Sampled);
|
||||
open.PublishedUtc.ShouldBe(Published);
|
||||
|
||||
var healthy = messages.Single(m => m.HostName == "host-b");
|
||||
healthy.BreakerOpen.ShouldBeFalse();
|
||||
healthy.ConsecutiveFailures.ShouldBe(0);
|
||||
healthy.CurrentInFlight.ShouldBe(1);
|
||||
healthy.PublishedUtc.ShouldBe(Published);
|
||||
}
|
||||
}
|
||||
+30
@@ -0,0 +1,30 @@
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.Hosting;
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.OtOpcUa.Host.Drivers;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Host.IntegrationTests;
|
||||
|
||||
/// <summary>
|
||||
/// Guards that the resilience-status tracker now HAS a production reader — the exact
|
||||
/// "built-but-never-wired" class of bug the whole arch-review targets (mirrors
|
||||
/// <see cref="ResilienceInvokerFactoryRegistrationTests"/>). <c>AddOtOpcUaDriverFactories</c> (the
|
||||
/// driver-node bootstrap) must register <see cref="DriverResilienceStatusPublisherService"/> as an
|
||||
/// <see cref="IHostedService"/>. A refactor that drops the registration fails this test, not production.
|
||||
/// </summary>
|
||||
public sealed class ResilienceStatusReaderRegistrationTests
|
||||
{
|
||||
[Fact]
|
||||
public void AddOtOpcUaDriverFactories_registers_the_resilience_status_reader_hosted_service()
|
||||
{
|
||||
var services = new ServiceCollection();
|
||||
services.AddLogging();
|
||||
services.AddOtOpcUaDriverFactories();
|
||||
|
||||
services.ShouldContain(
|
||||
d => d.ServiceType == typeof(IHostedService)
|
||||
&& d.ImplementationType == typeof(DriverResilienceStatusPublisherService),
|
||||
"the resilience-status tracker must have a production reader — the periodic DPS publisher hosted service on driver nodes");
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user