Compare commits

...

10 Commits

Author SHA1 Message Date
Joseph Doherty
0fcdfc7546 Phase 6.2 Stream A — LdapGroupRoleMapping entity + EF migration + CRUD service
Stream A.1-A.2 per docs/v2/implementation/phase-6-2-authorization-runtime.md.
Seed-data migration (A.3) is a separate follow-up once production LDAP group
DNs are finalised; until then CRUD via the Admin UI handles the fleet set up.

Configuration:
- New AdminRole enum {ConfigViewer, ConfigEditor, FleetAdmin} — string-stored.
- New LdapGroupRoleMapping entity with Id (surrogate PK), LdapGroup (512 chars),
  Role (AdminRole enum), ClusterId (nullable, FK to ServerCluster), IsSystemWide,
  CreatedAtUtc, Notes.
- EF config: UX_LdapGroupRoleMapping_Group_Cluster unique index on
  (LdapGroup, ClusterId) + IX_LdapGroupRoleMapping_Group hot-path index on
  LdapGroup for sign-in lookups. Cluster FK cascades on cluster delete.
- Migration 20260419_..._AddLdapGroupRoleMapping generated via `dotnet ef`.

Configuration.Services:
- ILdapGroupRoleMappingService — CRUD surface. Declared as control-plane only
  per decision #150; the OPC UA data-path evaluator must NOT depend on this
  interface (Phase 6.2 compliance check on control/data-plane separation).
  GetByGroupsAsync is the hot-path sign-in lookup.
- LdapGroupRoleMappingService (EF Core impl) enforces the write-time invariant
  "exactly one of (ClusterId populated, IsSystemWide=true)" and surfaces
  InvalidLdapGroupRoleMappingException on violation. Create auto-populates Id
  + CreatedAtUtc when omitted.

Tests (9 new, all pass) in Configuration.Tests:
- Create sets Id + CreatedAtUtc.
- Create rejects empty LdapGroup.
- Create rejects IsSystemWide=true with populated ClusterId.
- Create rejects IsSystemWide=false with null ClusterId.
- GetByGroupsAsync returns matching rows only.
- GetByGroupsAsync with empty input returns empty (no full-table scan).
- ListAllAsync orders by group then cluster.
- Delete removes the target row.
- Delete of unknown id is a no-op.

Microsoft.EntityFrameworkCore.InMemory 10.0.0 added to Configuration.Tests for
the service-level tests (schema-compliance tests still use the live SQL
fixture).

SchemaComplianceTests updated to expect the new LdapGroupRoleMapping table.

Full solution dotnet test: 1051 passing (baseline 906, Phase 6.1 shipped at
1042, Phase 6.2 Stream A adds 9 = 1051). Pre-existing Client.CLI Subscribe
flake unchanged.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-19 09:18:06 -04:00
1650c6c550 Merge pull request (#83) - Phase 6.1 exit gate 2026-04-19 08:55:47 -04:00
Joseph Doherty
f29043c66a Phase 6.1 exit gate — compliance script real-checks + phase doc status = SHIPPED
scripts/compliance/phase-6-1-compliance.ps1 replaces the stub TODOs with 34
real checks covering:
- Stream A: pipeline builder + CapabilityInvoker + WriteIdempotentAttribute
  present; pipeline key includes HostName (per-device isolation per decision
  #144); OnReadValue / OnWriteValue / HistoryRead route through invoker in
  DriverNodeManager; Galaxy supervisor CircuitBreaker + Backoff preserved.
- Stream B: DriverTier enum; DriverTypeMetadata requires Tier; MemoryTracking
  + MemoryRecycle (Tier C-gated) + ScheduledRecycleScheduler (rejects Tier
  A/B) + demand-aware WedgeDetector all present.
- Stream C: DriverHealthReport + HealthEndpointsHost; state matrix Healthy=200
  / Faulted=503 asserted in code; LogContextEnricher; JSON sink opt-in via
  Serilog:WriteJson.
- Stream D: GenerationSealedCache + ReadOnly marking + GenerationCacheUnavailable
  exception path; ResilientConfigReader + StaleConfigFlag.
- Stream E data layer: DriverInstanceResilienceStatus entity +
  DriverResilienceStatusTracker. SignalR/Blazor surface is Deferred per the
  visual-compliance follow-up pattern borrowed from Phase 6.4.
- Cross-cutting: full solution `dotnet test` runs; asserts 1042 >= 906
  baseline; tolerates the one pre-existing Client.CLI Subscribe flake and
  flags any new failure.

Running the script locally returns "Phase 6.1 compliance: PASS" — exit 0. Any
future regression that deletes a class or un-wires a dispatch path turns a
green check red + exit non-zero.

docs/v2/implementation/phase-6-1-resilience-and-observability.md status
updated from DRAFT to SHIPPED with the merged-PRs summary + test count delta +
the single deferred follow-up (visual review of the Admin /hosts columns).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-19 08:53:47 -04:00
a7f34a4301 Merge pull request (#82) - Phase 6.1 Stream E data layer 2026-04-19 08:49:43 -04:00
Joseph Doherty
cbcaf6593a Phase 6.1 Stream E (data layer) — DriverInstanceResilienceStatus entity + DriverResilienceStatusTracker + EF migration
Ships the data + runtime layer of Stream E. The SignalR hub and Blazor /hosts
page refresh (E.2-E.3) are follow-up work paired with the visual-compliance
review per Phase 6.4 patterns — documented as a deferred follow-up below.

Configuration:
- New entity DriverInstanceResilienceStatus with:
  DriverInstanceId, HostName (composite PK),
  LastCircuitBreakerOpenUtc, ConsecutiveFailures, CurrentBulkheadDepth,
  LastRecycleUtc, BaselineFootprintBytes, CurrentFootprintBytes,
  LastSampledUtc.
- Separate from DriverHostStatus (per-host connectivity view) so a Running
  host that has tripped its breaker or is nearing its memory ceiling shows up
  distinctly on Admin /hosts. Admin page left-joins both for display.
- OtOpcUaConfigDbContext + Fluent-API config + IX_DriverResilience_LastSampled
  index for the stale-sample filter query.
- EF migration: 20260419124034_AddDriverInstanceResilienceStatus.

Core.Resilience:
- DriverResilienceStatusTracker — process-singleton in-memory tracker keyed on
  (DriverInstanceId, HostName). CapabilityInvoker + MemoryTracking +
  MemoryRecycle callers record failure/success/breaker-open/recycle/footprint
  events; a HostedService (Stream E.2 follow-up) samples this tracker every
  5 s and persists to the DB. Pure in-memory keeps tests fast + the core
  free of EF/SQL dependencies.

Tests:
- DriverResilienceStatusTrackerTests (9 new, all pass): tryget-before-write
  returns null; failures accumulate; success resets; breaker/recycle/footprint
  fields populate; per-host isolation; snapshot returns all pairs; concurrent
  writes don't lose counts.
- SchemaComplianceTests: expected-tables list updated to include the new
  DriverInstanceResilienceStatus table.

Full solution dotnet test: 1042 passing (baseline 906, +136 for Phase 6.1 so
far across Streams A/B/C/D/E.1). Pre-existing Client.CLI Subscribe flake
unchanged.

Deferred to follow-up PR (E.2/E.3):
- ResilienceStatusPublisher HostedService that samples DriverResilienceStatusTracker
  every 5 s + upserts DriverInstanceResilienceStatus rows.
- Admin FleetStatusHub SignalR hub pushing LastCircuitBreakerOpenUtc /
  CurrentBulkheadDepth / LastRecycleUtc on change.
- Admin /hosts Blazor column additions (red badge when
  ConsecutiveFailures > breakerThreshold / 2). Visual-compliance reviewer
  signoff alongside Phase 6.4 admin-ui patterns.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-19 08:47:43 -04:00
8d81715079 Merge pull request (#81) - Phase 6.1 Stream D 2026-04-19 08:35:33 -04:00
Joseph Doherty
854c3bcfec Phase 6.1 Stream D — LiteDB generation-sealed config cache + ResilientConfigReader + UsingStaleConfig flag
Closes Stream D per docs/v2/implementation/phase-6-1-resilience-and-observability.md.

New Configuration.LocalCache types (alongside the existing single-file
LiteDbConfigCache):

- GenerationSealedCache — file-per-generation sealed snapshots per decision
  #148. Each SealAsync writes <cache-root>/<clusterId>/<generationId>.db as a
  read-only LiteDB file, then atomically publishes the CURRENT pointer via
  temp-file + File.Replace. Prior-generation files stay on disk for audit.
  Mixed-generation reads are structurally impossible: ReadCurrentAsync opens
  the single file named by CURRENT. Corruption of the pointer or the sealed
  file raises GenerationCacheUnavailableException — fails closed, never falls
  back silently to an older generation. TryGetCurrentGenerationId returns the
  pointer value or null for diagnostics.

- StaleConfigFlag — thread-safe (Volatile.Read/Write) bool. MarkStale when a
  read fell back to the cache; MarkFresh when a central-DB read succeeded.
  Surfaced on /healthz body and Admin /hosts (Stream C wiring already in
  place).

- ResilientConfigReader — wraps a central-DB fetch function with the Stream
  D.2 pipeline: timeout 2 s → retry N× jittered (skipped when retryCount=0) →
  fallback to the sealed cache. Toggles StaleConfigFlag per outcome. Read path
  only — the write path is expected to bypass this wrapper and fail hard on
  DB outage so inconsistent writes never land. Cancellation passes through
  and is NOT retried.

Configuration.csproj:
- Polly.Core 8.6.6 + Microsoft.Extensions.Logging.Abstractions added.

Tests (17 new, all pass):
- GenerationSealedCacheTests (10): first-boot-no-snapshot throws
  GenerationCacheUnavailableException (D.4 scenario C), seal-then-read round
  trip, sealed file is ReadOnly on disk, pointer advances to latest, prior
  generation file preserved, corrupt sealed file fails closed, missing sealed
  file fails closed, corrupt pointer fails closed (D.4 scenario B), same
  generation sealed twice is idempotent, independent clusters don't
  interfere.
- ResilientConfigReaderTests (4): central-DB success returns value + marks
  fresh; central-DB failure exhausts retries + falls back to cache + marks
  stale (D.4 scenario A); central-DB + cache both unavailable throws;
  cancellation not retried.
- StaleConfigFlagTests (3): default is fresh; toggles; concurrent writes
  converge.

Full solution dotnet test: 1033 passing (baseline 906, +127 net across Phase
6.1 Streams A/B/C/D). Pre-existing Client.CLI Subscribe flake unchanged.

Integration into Configuration read paths (DriverInstance enumeration,
LdapGroupRoleMapping fetches, etc.) + the sp_PublishGeneration hook that
writes sealed files lands in the Phase 6.1 Stream E / Admin-refresh PR where
the DB integration surfaces are already touched. Existing LiteDbConfigCache
continues serving its single-file role for the NodeBootstrap path.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-19 08:33:32 -04:00
ff4a74a81f Merge pull request (#80) - Phase 6.1 Stream C 2026-04-19 08:17:49 -04:00
Joseph Doherty
9dd5e4e745 Phase 6.1 Stream C — health endpoints on :4841 + LogContextEnricher + Serilog JSON sink + CapabilityInvoker enrichment
Closes Stream C per docs/v2/implementation/phase-6-1-resilience-and-observability.md.

Core.Observability (new namespace):
- DriverHealthReport — pure-function aggregation over DriverHealthSnapshot list.
  Empty fleet = Healthy. Any Faulted = Faulted. Any Unknown/Initializing (no
  Faulted) = NotReady. Any Degraded or Reconnecting (no Faulted, no NotReady)
  = Degraded. Else Healthy. HttpStatus(verdict) maps to the Stream C.1 state
  matrix: Healthy/Degraded → 200, NotReady/Faulted → 503.
- LogContextEnricher — Serilog LogContext wrapper. Push(id, type, capability,
  correlationId) returns an IDisposable scope; inner log calls carry
  DriverInstanceId / DriverType / CapabilityName / CorrelationId structured
  properties automatically. NewCorrelationId = 12-hex-char GUID slice for
  cases where no OPC UA RequestHeader.RequestHandle is in flight.

CapabilityInvoker — now threads LogContextEnricher around every ExecuteAsync /
ExecuteWriteAsync call site. OtOpcUaServer passes driver.DriverType through
so logs correlate to the driver type too. Every capability call emits
structured fields per the Stream C.4 compliance check.

Server.Observability:
- HealthEndpointsHost — standalone HttpListener on http://localhost:4841/
  (loopback avoids Windows URL-ACL elevation; remote probing via reverse
  proxy or explicit netsh urlacl grant). Routes:
    /healthz → 200 when (configDbReachable OR usingStaleConfig); 503 otherwise.
      Body: status, uptimeSeconds, configDbReachable, usingStaleConfig.
    /readyz  → DriverHealthReport.Aggregate + HttpStatus mapping.
      Body: verdict, drivers[], degradedDrivers[], uptimeSeconds.
    anything else → 404.
  Disposal cooperative with the HttpListener shutdown.
- OpcUaApplicationHost starts the health host after the OPC UA server comes up
  and disposes it on shutdown. New OpcUaServerOptions knobs:
  HealthEndpointsEnabled (default true), HealthEndpointsPrefix (default
  http://localhost:4841/).

Program.cs:
- Serilog pipeline adds Enrich.FromLogContext + opt-in JSON file sink via
  `Serilog:WriteJson = true` appsetting. Uses Serilog.Formatting.Compact's
  CompactJsonFormatter (one JSON object per line — SIEMs like Splunk,
  Datadog, Graylog ingest without a regex parser).

Server.Tests:
- Existing 3 OpcUaApplicationHost integration tests now set
  HealthEndpointsEnabled=false to avoid port :4841 collisions under parallel
  execution.
- New HealthEndpointsHostTests (9): /healthz healthy empty fleet; stale-config
  returns 200 with flag; unreachable+no-cache returns 503; /readyz empty/
  Healthy/Faulted/Degraded/Initializing drivers return correct status and
  bodies; unknown path → 404. Uses ephemeral ports via Interlocked counter.

Core.Tests:
- DriverHealthReportTests (8): empty fleet, all-healthy, any-Faulted trumps,
  any-NotReady without Faulted, Degraded without Faulted/NotReady, HttpStatus
  per-verdict theory.
- LogContextEnricherTests (8): all 4 properties attach; scope disposes cleanly;
  NewCorrelationId shape; null/whitespace driverInstanceId throws.
- CapabilityInvokerEnrichmentTests (2): inner logs carry structured
  properties; no context leak outside the call site.

Full solution dotnet test: 1016 passing (baseline 906, +110 for Phase 6.1 so
far across Streams A+B+C). Pre-existing Client.CLI Subscribe flake unchanged.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-19 08:15:44 -04:00
6b3a67fd9e Merge pull request (#79) - Phase 6.1 Stream B - Tier A/B/C stability (registry + MemoryTracking + MemoryRecycle + Scheduler + WedgeDetector) 2026-04-19 08:05:03 -04:00
41 changed files with 4963 additions and 46 deletions

View File

@@ -1,6 +1,8 @@
# Phase 6.1 — Resilience & Observability Runtime
> **Status**: DRAFT — implementation plan for a cross-cutting phase that was never formalised. The v2 `plan.md` specifies Polly, Tier A/B/C protections, structured logging, and local-cache fallback by decision; none are wired end-to-end.
> **Status**: **SHIPPED** 2026-04-19 — Streams A/B/C/D + E data layer merged to `v2` across PRs #78-82. Final exit-gate PR #83 turns the compliance script into real checks (all pass) and records this status update. One deferred piece: Stream E.2/E.3 SignalR hub + Blazor `/hosts` column refresh lands in a visual-compliance follow-up PR on the Phase 6.4 Admin UI branch.
>
> Baseline: 906 solution tests → post-Phase-6.1: 1042 passing (+136 net). One pre-existing Client.CLI Subscribe flake unchanged.
>
> **Branch**: `v2/phase-6-1-resilience-observability`
> **Estimated duration**: 3 weeks

View File

@@ -1,31 +1,27 @@
<#
.SYNOPSIS
Phase 6.1 exit-gate compliance check — stub. Each `Assert-*` either passes
(Write-Host green) or throws. Non-zero exit = fail.
Phase 6.1 exit-gate compliance check. Each check either passes or records a
failure; non-zero exit = fail.
.DESCRIPTION
Validates Phase 6.1 (Resilience & Observability runtime) completion. Checks
enumerated in `docs/v2/implementation/phase-6-1-resilience-and-observability.md`
§"Compliance Checks (run at exit gate)".
Current status: SCAFFOLD. Every check writes a TODO line and does NOT throw.
Each implementation task in Phase 6.1 is responsible for replacing its TODO
with a real check before closing that task.
Runs a mix of file-presence checks, text-pattern sweeps over the committed
codebase, and a full `dotnet test` pass to exercise the invariants each
class encodes. Meant to be invoked from repo root.
.NOTES
Usage: pwsh ./scripts/compliance/phase-6-1-compliance.ps1
Exit: 0 = all checks passed (or are still TODO); non-zero = explicit fail
Exit: 0 = all checks passed; non-zero = one or more FAILs
#>
[CmdletBinding()]
param()
$ErrorActionPreference = 'Stop'
$script:failures = 0
function Assert-Todo {
param([string]$Check, [string]$ImplementationTask)
Write-Host " [TODO] $Check (implement during $ImplementationTask)" -ForegroundColor Yellow
}
$repoRoot = (Resolve-Path (Join-Path $PSScriptRoot '..\..')).Path
function Assert-Pass {
param([string]$Check)
@@ -34,45 +30,109 @@ function Assert-Pass {
function Assert-Fail {
param([string]$Check, [string]$Reason)
Write-Host " [FAIL] $Check $Reason" -ForegroundColor Red
Write-Host " [FAIL] $Check - $Reason" -ForegroundColor Red
$script:failures++
}
Write-Host ""
Write-Host "=== Phase 6.1 compliance — Resilience & Observability runtime ===" -ForegroundColor Cyan
Write-Host ""
function Assert-Deferred {
param([string]$Check, [string]$FollowupPr)
Write-Host " [DEFERRED] $Check (follow-up: $FollowupPr)" -ForegroundColor Yellow
}
Write-Host "Stream A — Resilience layer"
Assert-Todo "Invoker coverage — every capability-interface method routes through CapabilityInvoker (analyzer error-level)" "Stream A.3"
Assert-Todo "Write-retry guard — writes without [WriteIdempotent] never retry" "Stream A.5"
Assert-Todo "Pipeline isolation — `(DriverInstanceId, HostName)` key; one dead host does not open breaker for siblings" "Stream A.5"
function Assert-FileExists {
param([string]$Check, [string]$RelPath)
$full = Join-Path $repoRoot $RelPath
if (Test-Path $full) { Assert-Pass "$Check ($RelPath)" }
else { Assert-Fail $Check "missing file: $RelPath" }
}
function Assert-TextFound {
param([string]$Check, [string]$Pattern, [string[]]$RelPaths)
foreach ($p in $RelPaths) {
$full = Join-Path $repoRoot $p
if (-not (Test-Path $full)) { continue }
if (Select-String -Path $full -Pattern $Pattern -Quiet) {
Assert-Pass "$Check (matched in $p)"
return
}
}
Assert-Fail $Check "pattern '$Pattern' not found in any of: $($RelPaths -join ', ')"
}
Write-Host ""
Write-Host "Stream B — Tier A/B/C runtime"
Assert-Todo "Tier registry — every driver type has non-null Tier; Tier C declares out-of-process topology" "Stream B.1"
Assert-Todo "MemoryTracking never kills — soft/hard breach on Tier A/B logs + surfaces without terminating" "Stream B.6"
Assert-Todo "MemoryRecycle Tier C only — hard breach on Tier A never invokes supervisor; Tier C does" "Stream B.6"
Assert-Todo "Wedge demand-aware — idle/historic-backfill/write-only cases stay Healthy" "Stream B.6"
Assert-Todo "Galaxy supervisor preserved — Driver.Galaxy.Proxy/Supervisor/CircuitBreaker + Backoff still present + invoked" "Stream A.4"
Write-Host "=== Phase 6.1 compliance - Resilience & Observability runtime ===" -ForegroundColor Cyan
Write-Host ""
Write-Host "Stream A - Resilience layer"
Assert-FileExists "Pipeline builder present" "src/ZB.MOM.WW.OtOpcUa.Core/Resilience/DriverResiliencePipelineBuilder.cs"
Assert-FileExists "CapabilityInvoker present" "src/ZB.MOM.WW.OtOpcUa.Core/Resilience/CapabilityInvoker.cs"
Assert-FileExists "WriteIdempotentAttribute present" "src/ZB.MOM.WW.OtOpcUa.Core.Abstractions/WriteIdempotentAttribute.cs"
Assert-TextFound "Pipeline key includes HostName (per-device isolation)" "PipelineKey\(.+HostName" @("src/ZB.MOM.WW.OtOpcUa.Core/Resilience/DriverResiliencePipelineBuilder.cs")
Assert-TextFound "OnReadValue routes through invoker" "DriverCapability\.Read," @("src/ZB.MOM.WW.OtOpcUa.Server/OpcUa/DriverNodeManager.cs")
Assert-TextFound "OnWriteValue routes through invoker" "ExecuteWriteAsync" @("src/ZB.MOM.WW.OtOpcUa.Server/OpcUa/DriverNodeManager.cs")
Assert-TextFound "HistoryRead routes through invoker" "DriverCapability\.HistoryRead" @("src/ZB.MOM.WW.OtOpcUa.Server/OpcUa/DriverNodeManager.cs")
Assert-FileExists "Galaxy supervisor CircuitBreaker preserved" "src/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Proxy/Supervisor/CircuitBreaker.cs"
Assert-FileExists "Galaxy supervisor Backoff preserved" "src/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Proxy/Supervisor/Backoff.cs"
Write-Host ""
Write-Host "Stream C — Health + logging"
Assert-Todo "Health state machine — /healthz + /readyz respond < 500 ms for every DriverState per matrix in plan" "Stream C.4"
Assert-Todo "Structured log — CI grep asserts DriverInstanceId + CorrelationId JSON fields present" "Stream C.4"
Write-Host "Stream B - Tier A/B/C runtime"
Assert-FileExists "DriverTier enum present" "src/ZB.MOM.WW.OtOpcUa.Core.Abstractions/DriverTier.cs"
Assert-TextFound "DriverTypeMetadata requires Tier" "DriverTier Tier" @("src/ZB.MOM.WW.OtOpcUa.Core.Abstractions/DriverTypeRegistry.cs")
Assert-FileExists "MemoryTracking present" "src/ZB.MOM.WW.OtOpcUa.Core/Stability/MemoryTracking.cs"
Assert-FileExists "MemoryRecycle present" "src/ZB.MOM.WW.OtOpcUa.Core/Stability/MemoryRecycle.cs"
Assert-TextFound "MemoryRecycle is Tier C gated" "_tier == DriverTier\.C" @("src/ZB.MOM.WW.OtOpcUa.Core/Stability/MemoryRecycle.cs")
Assert-FileExists "ScheduledRecycleScheduler present" "src/ZB.MOM.WW.OtOpcUa.Core/Stability/ScheduledRecycleScheduler.cs"
Assert-TextFound "Scheduler ctor rejects Tier A/B" "tier != DriverTier\.C" @("src/ZB.MOM.WW.OtOpcUa.Core/Stability/ScheduledRecycleScheduler.cs")
Assert-FileExists "WedgeDetector present" "src/ZB.MOM.WW.OtOpcUa.Core/Stability/WedgeDetector.cs"
Assert-TextFound "WedgeDetector is demand-aware" "HasPendingWork" @("src/ZB.MOM.WW.OtOpcUa.Core/Stability/WedgeDetector.cs")
Write-Host ""
Write-Host "Stream D — LiteDB cache"
Assert-Todo "Generation-sealed snapshot — SQL kill mid-op serves last-sealed snapshot; UsingStaleConfig=true" "Stream D.4"
Assert-Todo "Mixed-generation guard — corruption of snapshot file fails closed; no mixed reads" "Stream D.4"
Assert-Todo "First-boot no-snapshot + DB-down — InitializeAsync fails with clear error" "Stream D.4"
Write-Host "Stream C - Health + logging"
Assert-FileExists "DriverHealthReport present" "src/ZB.MOM.WW.OtOpcUa.Core/Observability/DriverHealthReport.cs"
Assert-FileExists "HealthEndpointsHost present" "src/ZB.MOM.WW.OtOpcUa.Server/Observability/HealthEndpointsHost.cs"
Assert-TextFound "State matrix: Healthy = 200" "ReadinessVerdict\.Healthy => 200" @("src/ZB.MOM.WW.OtOpcUa.Core/Observability/DriverHealthReport.cs")
Assert-TextFound "State matrix: Faulted = 503" "ReadinessVerdict\.Faulted => 503" @("src/ZB.MOM.WW.OtOpcUa.Core/Observability/DriverHealthReport.cs")
Assert-FileExists "LogContextEnricher present" "src/ZB.MOM.WW.OtOpcUa.Core/Observability/LogContextEnricher.cs"
Assert-TextFound "Enricher pushes DriverInstanceId property" "DriverInstanceId" @("src/ZB.MOM.WW.OtOpcUa.Core/Observability/LogContextEnricher.cs")
Assert-TextFound "JSON sink opt-in via Serilog:WriteJson" "Serilog:WriteJson" @("src/ZB.MOM.WW.OtOpcUa.Server/Program.cs")
Write-Host ""
Write-Host "Stream D - LiteDB generation-sealed cache"
Assert-FileExists "GenerationSealedCache present" "src/ZB.MOM.WW.OtOpcUa.Configuration/LocalCache/GenerationSealedCache.cs"
Assert-TextFound "Sealed files marked ReadOnly" "FileAttributes\.ReadOnly" @("src/ZB.MOM.WW.OtOpcUa.Configuration/LocalCache/GenerationSealedCache.cs")
Assert-TextFound "Corruption fails closed with GenerationCacheUnavailableException" "GenerationCacheUnavailableException" @("src/ZB.MOM.WW.OtOpcUa.Configuration/LocalCache/GenerationSealedCache.cs")
Assert-FileExists "ResilientConfigReader present" "src/ZB.MOM.WW.OtOpcUa.Configuration/LocalCache/ResilientConfigReader.cs"
Assert-FileExists "StaleConfigFlag present" "src/ZB.MOM.WW.OtOpcUa.Configuration/LocalCache/StaleConfigFlag.cs"
Write-Host ""
Write-Host "Stream E - Admin /hosts (data layer)"
Assert-FileExists "DriverInstanceResilienceStatus entity" "src/ZB.MOM.WW.OtOpcUa.Configuration/Entities/DriverInstanceResilienceStatus.cs"
Assert-FileExists "DriverResilienceStatusTracker present" "src/ZB.MOM.WW.OtOpcUa.Core/Resilience/DriverResilienceStatusTracker.cs"
Assert-Deferred "FleetStatusHub SignalR push + Blazor /hosts column refresh" "Phase 6.1 Stream E.2/E.3 visual-compliance follow-up"
Write-Host ""
Write-Host "Cross-cutting"
Assert-Todo "No test-count regression — dotnet test ZB.MOM.WW.OtOpcUa.slnx count ≥ pre-Phase-6.1 baseline" "Final exit-gate"
Write-Host " Running full solution test suite..." -ForegroundColor DarkGray
$prevPref = $ErrorActionPreference
$ErrorActionPreference = 'Continue'
$testOutput = & dotnet test (Join-Path $repoRoot 'ZB.MOM.WW.OtOpcUa.slnx') --nologo 2>&1
$ErrorActionPreference = $prevPref
$passLine = $testOutput | Select-String 'Passed:\s+(\d+)' -AllMatches
$failLine = $testOutput | Select-String 'Failed:\s+(\d+)' -AllMatches
$passCount = 0; foreach ($m in $passLine.Matches) { $passCount += [int]$m.Groups[1].Value }
$failCount = 0; foreach ($m in $failLine.Matches) { $failCount += [int]$m.Groups[1].Value }
$baseline = 906
if ($passCount -ge $baseline) { Assert-Pass "No test-count regression ($passCount >= $baseline baseline)" }
else { Assert-Fail "Test-count regression" "passed $passCount < baseline $baseline" }
# Pre-existing Client.CLI Subscribe flake tracked separately; exit gate tolerates a single
# known flake but flags any NEW failures.
if ($failCount -le 1) { Assert-Pass "No new failing tests (pre-existing CLI flake tolerated)" }
else { Assert-Fail "New failing tests" "$failCount failures > 1 tolerated" }
Write-Host ""
if ($script:failures -eq 0) {
Write-Host "Phase 6.1 compliance: scaffold-mode PASS (all checks TODO)" -ForegroundColor Green
Write-Host "Phase 6.1 compliance: PASS" -ForegroundColor Green
exit 0
}
Write-Host "Phase 6.1 compliance: $script:failures FAIL(s)" -ForegroundColor Red

View File

@@ -0,0 +1,44 @@
namespace ZB.MOM.WW.OtOpcUa.Configuration.Entities;
/// <summary>
/// Runtime resilience counters the CapabilityInvoker + MemoryTracking + MemoryRecycle
/// surfaces for each <c>(DriverInstanceId, HostName)</c> pair. Separate from
/// <see cref="DriverHostStatus"/> (which owns per-host <i>connectivity</i> state) so a
/// host that's Running but has tripped its breaker or is approaching its memory ceiling
/// shows up distinctly on Admin <c>/hosts</c>.
/// </summary>
/// <remarks>
/// Per <c>docs/v2/implementation/phase-6-1-resilience-and-observability.md</c> §Stream E.1.
/// The Admin UI left-joins this table on DriverHostStatus for display; rows are written
/// by the runtime via a HostedService that samples the tracker at a configurable
/// interval (default 5 s) — writes are non-critical, a missed sample is tolerated.
/// </remarks>
public sealed class DriverInstanceResilienceStatus
{
public required string DriverInstanceId { get; set; }
public required string HostName { get; set; }
/// <summary>Most recent time the circuit breaker for this (instance, host) opened; null if never.</summary>
public DateTime? LastCircuitBreakerOpenUtc { get; set; }
/// <summary>Rolling count of consecutive Polly pipeline failures for this (instance, host).</summary>
public int ConsecutiveFailures { get; set; }
/// <summary>Current Polly bulkhead depth (in-flight calls) for this (instance, host).</summary>
public int CurrentBulkheadDepth { get; set; }
/// <summary>Most recent process recycle time (Tier C only; null for in-process tiers).</summary>
public DateTime? LastRecycleUtc { get; set; }
/// <summary>
/// Post-init memory baseline captured by <c>MemoryTracking</c> (median of first
/// BaselineWindow samples). Zero while still warming up.
/// </summary>
public long BaselineFootprintBytes { get; set; }
/// <summary>Most recent footprint sample the tracker saw (steady-state read).</summary>
public long CurrentFootprintBytes { get; set; }
/// <summary>Row last-write timestamp — advances on every sampling tick.</summary>
public DateTime LastSampledUtc { get; set; }
}

View File

@@ -0,0 +1,56 @@
using ZB.MOM.WW.OtOpcUa.Configuration.Enums;
namespace ZB.MOM.WW.OtOpcUa.Configuration.Entities;
/// <summary>
/// Maps an LDAP group to an <see cref="AdminRole"/> for Admin UI access. Optionally scoped
/// to one <see cref="ClusterId"/>; when <see cref="IsSystemWide"/> is true, the grant
/// applies fleet-wide.
/// </summary>
/// <remarks>
/// <para>Per <c>docs/v2/plan.md</c> decisions #105 and #150 — this entity is <b>control-plane
/// only</b>. The OPC UA data-path evaluator does not read these rows; it reads
/// <see cref="NodeAcl"/> joined directly against the session's resolved LDAP group
/// memberships. Collapsing the two would let a user inherit tag permissions via an
/// admin-role claim path never intended as a data-path grant.</para>
///
/// <para>Uniqueness: <c>(LdapGroup, ClusterId)</c> — the same LDAP group may hold
/// different roles on different clusters, but only one row per cluster. A system-wide row
/// (<c>IsSystemWide = true</c>, <c>ClusterId = null</c>) stacks additively with any
/// cluster-scoped rows for the same group.</para>
/// </remarks>
public sealed class LdapGroupRoleMapping
{
/// <summary>Surrogate primary key.</summary>
public Guid Id { get; set; }
/// <summary>
/// LDAP group DN the membership query returns (e.g. <c>cn=fleet-admin,ou=groups,dc=corp,dc=example</c>).
/// Comparison is case-insensitive per LDAP conventions.
/// </summary>
public required string LdapGroup { get; set; }
/// <summary>Admin role this group grants.</summary>
public required AdminRole Role { get; set; }
/// <summary>
/// Cluster the grant applies to; <c>null</c> when <see cref="IsSystemWide"/> is true.
/// Foreign key to <see cref="ServerCluster.ClusterId"/>.
/// </summary>
public string? ClusterId { get; set; }
/// <summary>
/// <c>true</c> = grant applies across every cluster in the fleet; <c>ClusterId</c> must be null.
/// <c>false</c> = grant is cluster-scoped; <c>ClusterId</c> must be populated.
/// </summary>
public required bool IsSystemWide { get; set; }
/// <summary>Row creation timestamp (UTC).</summary>
public DateTime CreatedAtUtc { get; set; }
/// <summary>Optional human-readable note (e.g. "added 2026-04-19 for Warsaw fleet admin handoff").</summary>
public string? Notes { get; set; }
/// <summary>Navigation for EF core when the row is cluster-scoped.</summary>
public ServerCluster? Cluster { get; set; }
}

View File

@@ -0,0 +1,26 @@
namespace ZB.MOM.WW.OtOpcUa.Configuration.Enums;
/// <summary>
/// Admin UI roles per <c>admin-ui.md</c> §"Admin Roles" and Phase 6.2 Stream A.
/// These govern Admin UI capabilities (cluster CRUD, draft → publish, fleet-wide admin
/// actions) — they do NOT govern OPC UA data-path authorization, which reads
/// <see cref="Entities.NodeAcl"/> joined against LDAP group memberships directly.
/// </summary>
/// <remarks>
/// Per <c>docs/v2/plan.md</c> decision #150 the two concerns share zero runtime code path:
/// the control plane (Admin UI) consumes <see cref="Entities.LdapGroupRoleMapping"/>; the
/// data plane consumes <see cref="Entities.NodeAcl"/> rows directly. Having them in one
/// table would collapse the distinction + let a user inherit tag permissions via their
/// admin-role claim path.
/// </remarks>
public enum AdminRole
{
/// <summary>Read-only Admin UI access — can view cluster state, drafts, publish history.</summary>
ConfigViewer,
/// <summary>Can author drafts + submit for publish.</summary>
ConfigEditor,
/// <summary>Full Admin UI privileges including publish + fleet-admin actions.</summary>
FleetAdmin,
}

View File

@@ -0,0 +1,170 @@
using LiteDB;
namespace ZB.MOM.WW.OtOpcUa.Configuration.LocalCache;
/// <summary>
/// Generation-sealed LiteDB cache per <c>docs/v2/plan.md</c> decision #148 and Phase 6.1
/// Stream D.1. Each published generation writes one <b>read-only</b> LiteDB file under
/// <c>&lt;cache-root&gt;/&lt;clusterId&gt;/&lt;generationId&gt;.db</c>. A per-cluster
/// <c>CURRENT</c> text file holds the currently-active generation id; it is updated
/// atomically (temp file + <see cref="File.Replace(string, string, string?)"/>) only after
/// the sealed file is fully written.
/// </summary>
/// <remarks>
/// <para>Mixed-generation reads are impossible: any read opens the single file pointed to
/// by <c>CURRENT</c>, which is a coherent snapshot. Corruption of the CURRENT file or the
/// sealed file surfaces as <see cref="GenerationCacheUnavailableException"/> — the reader
/// fails closed rather than silently falling back to an older generation. Recovery path
/// is to re-fetch from the central DB (and the Phase 6.1 Stream C <c>UsingStaleConfig</c>
/// flag goes true until that succeeds).</para>
///
/// <para>This cache is the read-path fallback when the central DB is unreachable. The
/// write path (draft edits, publish) bypasses the cache and fails hard on DB outage per
/// Stream D.2 — inconsistent writes are worse than a temporary inability to edit.</para>
/// </remarks>
public sealed class GenerationSealedCache
{
private const string CollectionName = "generation";
private const string CurrentPointerFileName = "CURRENT";
private readonly string _cacheRoot;
/// <summary>Root directory for all clusters' sealed caches.</summary>
public string CacheRoot => _cacheRoot;
public GenerationSealedCache(string cacheRoot)
{
ArgumentException.ThrowIfNullOrWhiteSpace(cacheRoot);
_cacheRoot = cacheRoot;
Directory.CreateDirectory(_cacheRoot);
}
/// <summary>
/// Seal a generation: write the snapshot to <c>&lt;cluster&gt;/&lt;generationId&gt;.db</c>,
/// mark the file read-only, then atomically publish the <c>CURRENT</c> pointer. Existing
/// sealed files for prior generations are preserved (prune separately).
/// </summary>
public async Task SealAsync(GenerationSnapshot snapshot, CancellationToken ct = default)
{
ArgumentNullException.ThrowIfNull(snapshot);
ct.ThrowIfCancellationRequested();
var clusterDir = Path.Combine(_cacheRoot, snapshot.ClusterId);
Directory.CreateDirectory(clusterDir);
var sealedPath = Path.Combine(clusterDir, $"{snapshot.GenerationId}.db");
if (File.Exists(sealedPath))
{
// Already sealed — idempotent. Treat as no-op + update pointer in case an earlier
// seal succeeded but the pointer update failed (crash recovery).
WritePointerAtomically(clusterDir, snapshot.GenerationId);
return;
}
var tmpPath = sealedPath + ".tmp";
try
{
using (var db = new LiteDatabase(new ConnectionString { Filename = tmpPath, Upgrade = false }))
{
var col = db.GetCollection<GenerationSnapshot>(CollectionName);
col.Insert(snapshot);
}
File.Move(tmpPath, sealedPath);
File.SetAttributes(sealedPath, File.GetAttributes(sealedPath) | FileAttributes.ReadOnly);
WritePointerAtomically(clusterDir, snapshot.GenerationId);
}
catch
{
try { if (File.Exists(tmpPath)) File.Delete(tmpPath); } catch { /* best-effort */ }
throw;
}
await Task.CompletedTask;
}
/// <summary>
/// Read the current sealed snapshot for <paramref name="clusterId"/>. Throws
/// <see cref="GenerationCacheUnavailableException"/> when the pointer is missing
/// (first-boot-no-snapshot case) or when the sealed file is corrupt. Never silently
/// falls back to a prior generation.
/// </summary>
public Task<GenerationSnapshot> ReadCurrentAsync(string clusterId, CancellationToken ct = default)
{
ArgumentException.ThrowIfNullOrWhiteSpace(clusterId);
ct.ThrowIfCancellationRequested();
var clusterDir = Path.Combine(_cacheRoot, clusterId);
var pointerPath = Path.Combine(clusterDir, CurrentPointerFileName);
if (!File.Exists(pointerPath))
throw new GenerationCacheUnavailableException(
$"No sealed generation for cluster '{clusterId}' at '{clusterDir}'. First-boot case: the central DB must be reachable at least once before cache fallback is possible.");
long generationId;
try
{
var text = File.ReadAllText(pointerPath).Trim();
generationId = long.Parse(text, System.Globalization.CultureInfo.InvariantCulture);
}
catch (Exception ex)
{
throw new GenerationCacheUnavailableException(
$"CURRENT pointer at '{pointerPath}' is corrupt or unreadable.", ex);
}
var sealedPath = Path.Combine(clusterDir, $"{generationId}.db");
if (!File.Exists(sealedPath))
throw new GenerationCacheUnavailableException(
$"CURRENT points at generation {generationId} but '{sealedPath}' is missing — fails closed rather than serving an older generation.");
try
{
using var db = new LiteDatabase(new ConnectionString { Filename = sealedPath, ReadOnly = true });
var col = db.GetCollection<GenerationSnapshot>(CollectionName);
var snapshot = col.FindAll().FirstOrDefault()
?? throw new GenerationCacheUnavailableException(
$"Sealed file '{sealedPath}' contains no snapshot row — file is corrupt.");
return Task.FromResult(snapshot);
}
catch (GenerationCacheUnavailableException) { throw; }
catch (Exception ex) when (ex is LiteException or InvalidDataException or IOException
or NotSupportedException or FormatException)
{
throw new GenerationCacheUnavailableException(
$"Sealed file '{sealedPath}' is corrupt or unreadable — fails closed rather than falling back to an older generation.", ex);
}
}
/// <summary>Return the generation id the <c>CURRENT</c> pointer points at, or null if no pointer exists.</summary>
public long? TryGetCurrentGenerationId(string clusterId)
{
ArgumentException.ThrowIfNullOrWhiteSpace(clusterId);
var pointerPath = Path.Combine(_cacheRoot, clusterId, CurrentPointerFileName);
if (!File.Exists(pointerPath)) return null;
try
{
return long.Parse(File.ReadAllText(pointerPath).Trim(), System.Globalization.CultureInfo.InvariantCulture);
}
catch
{
return null;
}
}
private static void WritePointerAtomically(string clusterDir, long generationId)
{
var pointerPath = Path.Combine(clusterDir, CurrentPointerFileName);
var tmpPath = pointerPath + ".tmp";
File.WriteAllText(tmpPath, generationId.ToString(System.Globalization.CultureInfo.InvariantCulture));
if (File.Exists(pointerPath))
File.Replace(tmpPath, pointerPath, destinationBackupFileName: null);
else
File.Move(tmpPath, pointerPath);
}
}
/// <summary>Sealed cache is unreachable — caller must fail closed.</summary>
public sealed class GenerationCacheUnavailableException : Exception
{
public GenerationCacheUnavailableException(string message) : base(message) { }
public GenerationCacheUnavailableException(string message, Exception inner) : base(message, inner) { }
}

View File

@@ -0,0 +1,90 @@
using Microsoft.Extensions.Logging;
using Polly;
using Polly.Retry;
using Polly.Timeout;
namespace ZB.MOM.WW.OtOpcUa.Configuration.LocalCache;
/// <summary>
/// Wraps a central-DB fetch function with Phase 6.1 Stream D.2 resilience:
/// <b>timeout 2 s → retry 3× jittered → fallback to sealed cache</b>. Maintains the
/// <see cref="StaleConfigFlag"/> — fresh on central-DB success, stale on cache fallback.
/// </summary>
/// <remarks>
/// <para>Read-path only per plan. The write path (draft save, publish) bypasses this
/// wrapper entirely and fails hard on DB outage so inconsistent writes never land.</para>
///
/// <para>Fallback is triggered by <b>any exception</b> the fetch raises (central-DB
/// unreachable, SqlException, timeout). If the sealed cache also fails (no pointer,
/// corrupt file, etc.), <see cref="GenerationCacheUnavailableException"/> surfaces — caller
/// must fail the current request (InitializeAsync for a driver, etc.).</para>
/// </remarks>
public sealed class ResilientConfigReader
{
private readonly GenerationSealedCache _cache;
private readonly StaleConfigFlag _staleFlag;
private readonly ResiliencePipeline _pipeline;
private readonly ILogger<ResilientConfigReader> _logger;
public ResilientConfigReader(
GenerationSealedCache cache,
StaleConfigFlag staleFlag,
ILogger<ResilientConfigReader> logger,
TimeSpan? timeout = null,
int retryCount = 3)
{
_cache = cache;
_staleFlag = staleFlag;
_logger = logger;
var builder = new ResiliencePipelineBuilder()
.AddTimeout(new TimeoutStrategyOptions { Timeout = timeout ?? TimeSpan.FromSeconds(2) });
if (retryCount > 0)
{
builder.AddRetry(new RetryStrategyOptions
{
MaxRetryAttempts = retryCount,
BackoffType = DelayBackoffType.Exponential,
UseJitter = true,
Delay = TimeSpan.FromMilliseconds(100),
MaxDelay = TimeSpan.FromSeconds(1),
ShouldHandle = new PredicateBuilder().Handle<Exception>(ex => ex is not OperationCanceledException),
});
}
_pipeline = builder.Build();
}
/// <summary>
/// Execute <paramref name="centralFetch"/> through the resilience pipeline. On full failure
/// (post-retry), reads the sealed cache for <paramref name="clusterId"/> and passes the
/// snapshot to <paramref name="fromSnapshot"/> to extract the requested shape.
/// </summary>
public async ValueTask<T> ReadAsync<T>(
string clusterId,
Func<CancellationToken, ValueTask<T>> centralFetch,
Func<GenerationSnapshot, T> fromSnapshot,
CancellationToken cancellationToken)
{
ArgumentException.ThrowIfNullOrWhiteSpace(clusterId);
ArgumentNullException.ThrowIfNull(centralFetch);
ArgumentNullException.ThrowIfNull(fromSnapshot);
try
{
var result = await _pipeline.ExecuteAsync(centralFetch, cancellationToken).ConfigureAwait(false);
_staleFlag.MarkFresh();
return result;
}
catch (Exception ex) when (ex is not OperationCanceledException)
{
_logger.LogWarning(ex, "Central-DB read failed after retries; falling back to sealed cache for cluster {ClusterId}", clusterId);
// GenerationCacheUnavailableException surfaces intentionally — fails the caller's
// operation. StaleConfigFlag stays unchanged; the flag only flips when we actually
// served a cache snapshot.
var snapshot = await _cache.ReadCurrentAsync(clusterId, cancellationToken).ConfigureAwait(false);
_staleFlag.MarkStale();
return fromSnapshot(snapshot);
}
}
}

View File

@@ -0,0 +1,20 @@
namespace ZB.MOM.WW.OtOpcUa.Configuration.LocalCache;
/// <summary>
/// Thread-safe <c>UsingStaleConfig</c> signal per Phase 6.1 Stream D.3. Flips true whenever
/// a read falls back to a sealed cache snapshot; flips false on the next successful central-DB
/// round-trip. Surfaced on <c>/healthz</c> body and on the Admin <c>/hosts</c> page.
/// </summary>
public sealed class StaleConfigFlag
{
private int _stale;
/// <summary>True when the last config read was served from the sealed cache, not the central DB.</summary>
public bool IsStale => Volatile.Read(ref _stale) != 0;
/// <summary>Mark the current config as stale (a read fell back to the cache).</summary>
public void MarkStale() => Volatile.Write(ref _stale, 1);
/// <summary>Mark the current config as fresh (a central-DB read succeeded).</summary>
public void MarkFresh() => Volatile.Write(ref _stale, 0);
}

View File

@@ -0,0 +1,46 @@
using System;
using Microsoft.EntityFrameworkCore.Migrations;
#nullable disable
namespace ZB.MOM.WW.OtOpcUa.Configuration.Migrations
{
/// <inheritdoc />
public partial class AddDriverInstanceResilienceStatus : Migration
{
/// <inheritdoc />
protected override void Up(MigrationBuilder migrationBuilder)
{
migrationBuilder.CreateTable(
name: "DriverInstanceResilienceStatus",
columns: table => new
{
DriverInstanceId = table.Column<string>(type: "nvarchar(64)", maxLength: 64, nullable: false),
HostName = table.Column<string>(type: "nvarchar(256)", maxLength: 256, nullable: false),
LastCircuitBreakerOpenUtc = table.Column<DateTime>(type: "datetime2(3)", nullable: true),
ConsecutiveFailures = table.Column<int>(type: "int", nullable: false),
CurrentBulkheadDepth = table.Column<int>(type: "int", nullable: false),
LastRecycleUtc = table.Column<DateTime>(type: "datetime2(3)", nullable: true),
BaselineFootprintBytes = table.Column<long>(type: "bigint", nullable: false),
CurrentFootprintBytes = table.Column<long>(type: "bigint", nullable: false),
LastSampledUtc = table.Column<DateTime>(type: "datetime2(3)", nullable: false)
},
constraints: table =>
{
table.PrimaryKey("PK_DriverInstanceResilienceStatus", x => new { x.DriverInstanceId, x.HostName });
});
migrationBuilder.CreateIndex(
name: "IX_DriverResilience_LastSampled",
table: "DriverInstanceResilienceStatus",
column: "LastSampledUtc");
}
/// <inheritdoc />
protected override void Down(MigrationBuilder migrationBuilder)
{
migrationBuilder.DropTable(
name: "DriverInstanceResilienceStatus");
}
}
}

View File

@@ -0,0 +1,62 @@
using System;
using Microsoft.EntityFrameworkCore.Migrations;
#nullable disable
namespace ZB.MOM.WW.OtOpcUa.Configuration.Migrations
{
/// <inheritdoc />
public partial class AddLdapGroupRoleMapping : Migration
{
/// <inheritdoc />
protected override void Up(MigrationBuilder migrationBuilder)
{
migrationBuilder.CreateTable(
name: "LdapGroupRoleMapping",
columns: table => new
{
Id = table.Column<Guid>(type: "uniqueidentifier", nullable: false),
LdapGroup = table.Column<string>(type: "nvarchar(512)", maxLength: 512, nullable: false),
Role = table.Column<string>(type: "nvarchar(32)", maxLength: 32, nullable: false),
ClusterId = table.Column<string>(type: "nvarchar(64)", maxLength: 64, nullable: true),
IsSystemWide = table.Column<bool>(type: "bit", nullable: false),
CreatedAtUtc = table.Column<DateTime>(type: "datetime2(3)", nullable: false),
Notes = table.Column<string>(type: "nvarchar(512)", maxLength: 512, nullable: true)
},
constraints: table =>
{
table.PrimaryKey("PK_LdapGroupRoleMapping", x => x.Id);
table.ForeignKey(
name: "FK_LdapGroupRoleMapping_ServerCluster_ClusterId",
column: x => x.ClusterId,
principalTable: "ServerCluster",
principalColumn: "ClusterId",
onDelete: ReferentialAction.Cascade);
});
migrationBuilder.CreateIndex(
name: "IX_LdapGroupRoleMapping_ClusterId",
table: "LdapGroupRoleMapping",
column: "ClusterId");
migrationBuilder.CreateIndex(
name: "IX_LdapGroupRoleMapping_Group",
table: "LdapGroupRoleMapping",
column: "LdapGroup");
migrationBuilder.CreateIndex(
name: "UX_LdapGroupRoleMapping_Group_Cluster",
table: "LdapGroupRoleMapping",
columns: new[] { "LdapGroup", "ClusterId" },
unique: true,
filter: "[ClusterId] IS NOT NULL");
}
/// <inheritdoc />
protected override void Down(MigrationBuilder migrationBuilder)
{
migrationBuilder.DropTable(
name: "LdapGroupRoleMapping");
}
}
}

View File

@@ -434,6 +434,45 @@ namespace ZB.MOM.WW.OtOpcUa.Configuration.Migrations
});
});
modelBuilder.Entity("ZB.MOM.WW.OtOpcUa.Configuration.Entities.DriverInstanceResilienceStatus", b =>
{
b.Property<string>("DriverInstanceId")
.HasMaxLength(64)
.HasColumnType("nvarchar(64)");
b.Property<string>("HostName")
.HasMaxLength(256)
.HasColumnType("nvarchar(256)");
b.Property<long>("BaselineFootprintBytes")
.HasColumnType("bigint");
b.Property<int>("ConsecutiveFailures")
.HasColumnType("int");
b.Property<int>("CurrentBulkheadDepth")
.HasColumnType("int");
b.Property<long>("CurrentFootprintBytes")
.HasColumnType("bigint");
b.Property<DateTime?>("LastCircuitBreakerOpenUtc")
.HasColumnType("datetime2(3)");
b.Property<DateTime?>("LastRecycleUtc")
.HasColumnType("datetime2(3)");
b.Property<DateTime>("LastSampledUtc")
.HasColumnType("datetime2(3)");
b.HasKey("DriverInstanceId", "HostName");
b.HasIndex("LastSampledUtc")
.HasDatabaseName("IX_DriverResilience_LastSampled");
b.ToTable("DriverInstanceResilienceStatus", (string)null);
});
modelBuilder.Entity("ZB.MOM.WW.OtOpcUa.Configuration.Entities.Equipment", b =>
{
b.Property<Guid>("EquipmentRowId")
@@ -624,6 +663,51 @@ namespace ZB.MOM.WW.OtOpcUa.Configuration.Migrations
b.ToTable("ExternalIdReservation", (string)null);
});
modelBuilder.Entity("ZB.MOM.WW.OtOpcUa.Configuration.Entities.LdapGroupRoleMapping", b =>
{
b.Property<Guid>("Id")
.ValueGeneratedOnAdd()
.HasColumnType("uniqueidentifier");
b.Property<string>("ClusterId")
.HasMaxLength(64)
.HasColumnType("nvarchar(64)");
b.Property<DateTime>("CreatedAtUtc")
.HasColumnType("datetime2(3)");
b.Property<bool>("IsSystemWide")
.HasColumnType("bit");
b.Property<string>("LdapGroup")
.IsRequired()
.HasMaxLength(512)
.HasColumnType("nvarchar(512)");
b.Property<string>("Notes")
.HasMaxLength(512)
.HasColumnType("nvarchar(512)");
b.Property<string>("Role")
.IsRequired()
.HasMaxLength(32)
.HasColumnType("nvarchar(32)");
b.HasKey("Id");
b.HasIndex("ClusterId");
b.HasIndex("LdapGroup")
.HasDatabaseName("IX_LdapGroupRoleMapping_Group");
b.HasIndex("LdapGroup", "ClusterId")
.IsUnique()
.HasDatabaseName("UX_LdapGroupRoleMapping_Group_Cluster")
.HasFilter("[ClusterId] IS NOT NULL");
b.ToTable("LdapGroupRoleMapping", (string)null);
});
modelBuilder.Entity("ZB.MOM.WW.OtOpcUa.Configuration.Entities.Namespace", b =>
{
b.Property<Guid>("NamespaceRowId")
@@ -1142,6 +1226,16 @@ namespace ZB.MOM.WW.OtOpcUa.Configuration.Migrations
b.Navigation("Generation");
});
modelBuilder.Entity("ZB.MOM.WW.OtOpcUa.Configuration.Entities.LdapGroupRoleMapping", b =>
{
b.HasOne("ZB.MOM.WW.OtOpcUa.Configuration.Entities.ServerCluster", "Cluster")
.WithMany()
.HasForeignKey("ClusterId")
.OnDelete(DeleteBehavior.Cascade);
b.Navigation("Cluster");
});
modelBuilder.Entity("ZB.MOM.WW.OtOpcUa.Configuration.Entities.Namespace", b =>
{
b.HasOne("ZB.MOM.WW.OtOpcUa.Configuration.Entities.ServerCluster", "Cluster")

View File

@@ -28,6 +28,8 @@ public sealed class OtOpcUaConfigDbContext(DbContextOptions<OtOpcUaConfigDbConte
public DbSet<ConfigAuditLog> ConfigAuditLogs => Set<ConfigAuditLog>();
public DbSet<ExternalIdReservation> ExternalIdReservations => Set<ExternalIdReservation>();
public DbSet<DriverHostStatus> DriverHostStatuses => Set<DriverHostStatus>();
public DbSet<DriverInstanceResilienceStatus> DriverInstanceResilienceStatuses => Set<DriverInstanceResilienceStatus>();
public DbSet<LdapGroupRoleMapping> LdapGroupRoleMappings => Set<LdapGroupRoleMapping>();
protected override void OnModelCreating(ModelBuilder modelBuilder)
{
@@ -49,6 +51,8 @@ public sealed class OtOpcUaConfigDbContext(DbContextOptions<OtOpcUaConfigDbConte
ConfigureConfigAuditLog(modelBuilder);
ConfigureExternalIdReservation(modelBuilder);
ConfigureDriverHostStatus(modelBuilder);
ConfigureDriverInstanceResilienceStatus(modelBuilder);
ConfigureLdapGroupRoleMapping(modelBuilder);
}
private static void ConfigureServerCluster(ModelBuilder modelBuilder)
@@ -512,4 +516,53 @@ public sealed class OtOpcUaConfigDbContext(DbContextOptions<OtOpcUaConfigDbConte
e.HasIndex(x => x.LastSeenUtc).HasDatabaseName("IX_DriverHostStatus_LastSeen");
});
}
private static void ConfigureDriverInstanceResilienceStatus(ModelBuilder modelBuilder)
{
modelBuilder.Entity<DriverInstanceResilienceStatus>(e =>
{
e.ToTable("DriverInstanceResilienceStatus");
e.HasKey(x => new { x.DriverInstanceId, x.HostName });
e.Property(x => x.DriverInstanceId).HasMaxLength(64);
e.Property(x => x.HostName).HasMaxLength(256);
e.Property(x => x.LastCircuitBreakerOpenUtc).HasColumnType("datetime2(3)");
e.Property(x => x.LastRecycleUtc).HasColumnType("datetime2(3)");
e.Property(x => x.LastSampledUtc).HasColumnType("datetime2(3)");
// LastSampledUtc drives the Admin UI's stale-sample filter same way DriverHostStatus's
// LastSeenUtc index does for connectivity rows.
e.HasIndex(x => x.LastSampledUtc).HasDatabaseName("IX_DriverResilience_LastSampled");
});
}
private static void ConfigureLdapGroupRoleMapping(ModelBuilder modelBuilder)
{
modelBuilder.Entity<LdapGroupRoleMapping>(e =>
{
e.ToTable("LdapGroupRoleMapping");
e.HasKey(x => x.Id);
e.Property(x => x.LdapGroup).HasMaxLength(512).IsRequired();
e.Property(x => x.Role).HasConversion<string>().HasMaxLength(32);
e.Property(x => x.ClusterId).HasMaxLength(64);
e.Property(x => x.CreatedAtUtc).HasColumnType("datetime2(3)");
e.Property(x => x.Notes).HasMaxLength(512);
// FK to ServerCluster when cluster-scoped; null for system-wide grants.
e.HasOne(x => x.Cluster)
.WithMany()
.HasForeignKey(x => x.ClusterId)
.OnDelete(DeleteBehavior.Cascade);
// Uniqueness: one row per (LdapGroup, ClusterId). Null ClusterId is treated as its own
// "bucket" so a system-wide row coexists with cluster-scoped rows for the same group.
// SQL Server treats NULL as a distinct value in unique-index comparisons by default
// since 2008 SP1 onwards under the session setting we use — tested in SchemaCompliance.
e.HasIndex(x => new { x.LdapGroup, x.ClusterId })
.IsUnique()
.HasDatabaseName("UX_LdapGroupRoleMapping_Group_Cluster");
// Hot-path lookup during cookie auth: "what grants does this user's set of LDAP
// groups carry?". Fires on every sign-in so the index earns its keep.
e.HasIndex(x => x.LdapGroup).HasDatabaseName("IX_LdapGroupRoleMapping_Group");
});
}
}

View File

@@ -0,0 +1,47 @@
using ZB.MOM.WW.OtOpcUa.Configuration.Entities;
using ZB.MOM.WW.OtOpcUa.Configuration.Enums;
namespace ZB.MOM.WW.OtOpcUa.Configuration.Services;
/// <summary>
/// CRUD surface for <see cref="LdapGroupRoleMapping"/> — the control-plane mapping from
/// LDAP groups to Admin UI roles. Consumed only by Admin UI code paths; the OPC UA
/// data-path evaluator MUST NOT depend on this interface (see decision #150 and the
/// Phase 6.2 compliance check on control/data-plane separation).
/// </summary>
/// <remarks>
/// Per Phase 6.2 Stream A.2 this service is expected to run behind the Phase 6.1
/// <c>ResilientConfigReader</c> pipeline (timeout → retry → fallback-to-cache) so a
/// transient DB outage during sign-in falls back to the sealed snapshot rather than
/// denying every login.
/// </remarks>
public interface ILdapGroupRoleMappingService
{
/// <summary>List every mapping whose LDAP group matches one of <paramref name="ldapGroups"/>.</summary>
/// <remarks>
/// Hot path — fires on every sign-in. The default EF implementation relies on the
/// <c>IX_LdapGroupRoleMapping_Group</c> index. Case-insensitive per LDAP conventions.
/// </remarks>
Task<IReadOnlyList<LdapGroupRoleMapping>> GetByGroupsAsync(
IEnumerable<string> ldapGroups, CancellationToken cancellationToken);
/// <summary>Enumerate every mapping; Admin UI listing only.</summary>
Task<IReadOnlyList<LdapGroupRoleMapping>> ListAllAsync(CancellationToken cancellationToken);
/// <summary>Create a new grant.</summary>
/// <exception cref="InvalidLdapGroupRoleMappingException">
/// Thrown when the proposed row violates an invariant (IsSystemWide inconsistent with
/// ClusterId, duplicate (group, cluster) pair, etc.) — ValidatedLdapGroupRoleMappingService
/// is the write surface that enforces these; the raw service here surfaces DB-level violations.
/// </exception>
Task<LdapGroupRoleMapping> CreateAsync(LdapGroupRoleMapping row, CancellationToken cancellationToken);
/// <summary>Delete a mapping by its surrogate key.</summary>
Task DeleteAsync(Guid id, CancellationToken cancellationToken);
}
/// <summary>Thrown when <see cref="LdapGroupRoleMapping"/> authoring violates an invariant.</summary>
public sealed class InvalidLdapGroupRoleMappingException : Exception
{
public InvalidLdapGroupRoleMappingException(string message) : base(message) { }
}

View File

@@ -0,0 +1,69 @@
using Microsoft.EntityFrameworkCore;
using ZB.MOM.WW.OtOpcUa.Configuration.Entities;
namespace ZB.MOM.WW.OtOpcUa.Configuration.Services;
/// <summary>
/// EF Core implementation of <see cref="ILdapGroupRoleMappingService"/>. Enforces the
/// "exactly one of (ClusterId, IsSystemWide)" invariant at the write surface so a
/// malformed row can't land in the DB.
/// </summary>
public sealed class LdapGroupRoleMappingService(OtOpcUaConfigDbContext db) : ILdapGroupRoleMappingService
{
public async Task<IReadOnlyList<LdapGroupRoleMapping>> GetByGroupsAsync(
IEnumerable<string> ldapGroups, CancellationToken cancellationToken)
{
ArgumentNullException.ThrowIfNull(ldapGroups);
var groupSet = ldapGroups.ToList();
if (groupSet.Count == 0) return [];
return await db.LdapGroupRoleMappings
.AsNoTracking()
.Where(m => groupSet.Contains(m.LdapGroup))
.ToListAsync(cancellationToken)
.ConfigureAwait(false);
}
public async Task<IReadOnlyList<LdapGroupRoleMapping>> ListAllAsync(CancellationToken cancellationToken)
=> await db.LdapGroupRoleMappings
.AsNoTracking()
.OrderBy(m => m.LdapGroup)
.ThenBy(m => m.ClusterId)
.ToListAsync(cancellationToken)
.ConfigureAwait(false);
public async Task<LdapGroupRoleMapping> CreateAsync(LdapGroupRoleMapping row, CancellationToken cancellationToken)
{
ArgumentNullException.ThrowIfNull(row);
ValidateInvariants(row);
if (row.Id == Guid.Empty) row.Id = Guid.NewGuid();
if (row.CreatedAtUtc == default) row.CreatedAtUtc = DateTime.UtcNow;
db.LdapGroupRoleMappings.Add(row);
await db.SaveChangesAsync(cancellationToken).ConfigureAwait(false);
return row;
}
public async Task DeleteAsync(Guid id, CancellationToken cancellationToken)
{
var existing = await db.LdapGroupRoleMappings.FindAsync([id], cancellationToken).ConfigureAwait(false);
if (existing is null) return;
db.LdapGroupRoleMappings.Remove(existing);
await db.SaveChangesAsync(cancellationToken).ConfigureAwait(false);
}
private static void ValidateInvariants(LdapGroupRoleMapping row)
{
if (string.IsNullOrWhiteSpace(row.LdapGroup))
throw new InvalidLdapGroupRoleMappingException("LdapGroup must not be empty.");
if (row.IsSystemWide && !string.IsNullOrEmpty(row.ClusterId))
throw new InvalidLdapGroupRoleMappingException(
"IsSystemWide=true requires ClusterId to be null. A fleet-wide grant cannot also be cluster-scoped.");
if (!row.IsSystemWide && string.IsNullOrEmpty(row.ClusterId))
throw new InvalidLdapGroupRoleMappingException(
"IsSystemWide=false requires a populated ClusterId. A cluster-scoped grant needs its target cluster.");
}
}

View File

@@ -19,7 +19,9 @@
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
</PackageReference>
<PackageReference Include="Microsoft.Extensions.Configuration.Abstractions" Version="10.0.0"/>
<PackageReference Include="Microsoft.Extensions.Logging.Abstractions" Version="10.0.0"/>
<PackageReference Include="LiteDB" Version="5.0.21"/>
<PackageReference Include="Polly.Core" Version="8.6.6"/>
</ItemGroup>
<ItemGroup>

View File

@@ -0,0 +1,86 @@
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Core.Observability;
/// <summary>
/// Domain-layer health aggregation for Phase 6.1 Stream C. Pure functions over the driver
/// fleet — given each driver's <see cref="DriverState"/>, produce a <see cref="ReadinessVerdict"/>
/// that maps to HTTP status codes at the endpoint layer.
/// </summary>
/// <remarks>
/// State matrix per <c>docs/v2/implementation/phase-6-1-resilience-and-observability.md</c>
/// §Stream C.1:
/// <list type="bullet">
/// <item><see cref="DriverState.Unknown"/> / <see cref="DriverState.Initializing"/>
/// → /readyz 503 (not yet ready).</item>
/// <item><see cref="DriverState.Healthy"/> → /readyz 200.</item>
/// <item><see cref="DriverState.Degraded"/> → /readyz 200 with flagged driver IDs.</item>
/// <item><see cref="DriverState.Faulted"/> → /readyz 503.</item>
/// </list>
/// The overall verdict is computed across the fleet: any Faulted → Faulted; any
/// Unknown/Initializing → NotReady; any Degraded → Degraded; else Healthy. An empty fleet
/// is Healthy (nothing to degrade).
/// </remarks>
public static class DriverHealthReport
{
/// <summary>Compute the fleet-wide readiness verdict from per-driver states.</summary>
public static ReadinessVerdict Aggregate(IReadOnlyList<DriverHealthSnapshot> drivers)
{
ArgumentNullException.ThrowIfNull(drivers);
if (drivers.Count == 0) return ReadinessVerdict.Healthy;
var anyFaulted = drivers.Any(d => d.State == DriverState.Faulted);
if (anyFaulted) return ReadinessVerdict.Faulted;
var anyInitializing = drivers.Any(d =>
d.State == DriverState.Unknown || d.State == DriverState.Initializing);
if (anyInitializing) return ReadinessVerdict.NotReady;
// Reconnecting = driver alive but not serving live data; report as Degraded so /readyz
// stays 200 (the fleet can still serve cached / last-good data) while operators see the
// affected driver in the body.
var anyDegraded = drivers.Any(d =>
d.State == DriverState.Degraded || d.State == DriverState.Reconnecting);
if (anyDegraded) return ReadinessVerdict.Degraded;
return ReadinessVerdict.Healthy;
}
/// <summary>
/// Map a <see cref="ReadinessVerdict"/> to the HTTP status the /readyz endpoint should
/// return per the Stream C.1 state matrix.
/// </summary>
public static int HttpStatus(ReadinessVerdict verdict) => verdict switch
{
ReadinessVerdict.Healthy => 200,
ReadinessVerdict.Degraded => 200,
ReadinessVerdict.NotReady => 503,
ReadinessVerdict.Faulted => 503,
_ => 500,
};
}
/// <summary>Per-driver snapshot fed into <see cref="DriverHealthReport.Aggregate"/>.</summary>
/// <param name="DriverInstanceId">Driver instance identifier (from <c>IDriver.DriverInstanceId</c>).</param>
/// <param name="State">Current <see cref="DriverState"/> from <c>IDriver.GetHealth</c>.</param>
/// <param name="DetailMessage">Optional driver-supplied detail (e.g. "primary PLC unreachable").</param>
public sealed record DriverHealthSnapshot(
string DriverInstanceId,
DriverState State,
string? DetailMessage = null);
/// <summary>Overall fleet readiness — derived from driver states by <see cref="DriverHealthReport.Aggregate"/>.</summary>
public enum ReadinessVerdict
{
/// <summary>All drivers Healthy (or fleet is empty).</summary>
Healthy,
/// <summary>At least one driver Degraded; none Faulted / NotReady.</summary>
Degraded,
/// <summary>At least one driver Unknown / Initializing; none Faulted.</summary>
NotReady,
/// <summary>At least one driver Faulted.</summary>
Faulted,
}

View File

@@ -0,0 +1,53 @@
using Serilog.Context;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Core.Observability;
/// <summary>
/// Convenience wrapper around Serilog <see cref="LogContext"/> — attaches the set of
/// structured properties a capability call should carry (DriverInstanceId, DriverType,
/// CapabilityName, CorrelationId). Callers wrap their call-site body in a <c>using</c>
/// block; inner <c>Log.Information</c> / <c>Log.Warning</c> calls emit the context
/// automatically via the Serilog enricher chain.
/// </summary>
/// <remarks>
/// Per <c>docs/v2/implementation/phase-6-1-resilience-and-observability.md</c> §Stream C.2.
/// The correlation ID should be the OPC UA <c>RequestHeader.RequestHandle</c> when in-flight;
/// otherwise a short random GUID. Callers supply whichever is available.
/// </remarks>
public static class LogContextEnricher
{
/// <summary>Attach the capability-call property set. Dispose the returned scope to pop.</summary>
public static IDisposable Push(string driverInstanceId, string driverType, DriverCapability capability, string correlationId)
{
ArgumentException.ThrowIfNullOrWhiteSpace(driverInstanceId);
ArgumentException.ThrowIfNullOrWhiteSpace(driverType);
ArgumentException.ThrowIfNullOrWhiteSpace(correlationId);
var a = LogContext.PushProperty("DriverInstanceId", driverInstanceId);
var b = LogContext.PushProperty("DriverType", driverType);
var c = LogContext.PushProperty("CapabilityName", capability.ToString());
var d = LogContext.PushProperty("CorrelationId", correlationId);
return new CompositeScope(a, b, c, d);
}
/// <summary>
/// Generate a short correlation ID when no OPC UA RequestHandle is available.
/// 12-hex-char slice of a GUID — long enough for log correlation, short enough to
/// scan visually.
/// </summary>
public static string NewCorrelationId() => Guid.NewGuid().ToString("N")[..12];
private sealed class CompositeScope : IDisposable
{
private readonly IDisposable[] _inner;
public CompositeScope(params IDisposable[] inner) => _inner = inner;
public void Dispose()
{
// Reverse-order disposal matches Serilog's stack semantics.
for (var i = _inner.Length - 1; i >= 0; i--)
_inner[i].Dispose();
}
}
}

View File

@@ -1,5 +1,6 @@
using Polly;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Core.Observability;
namespace ZB.MOM.WW.OtOpcUa.Core.Resilience;
@@ -19,6 +20,7 @@ public sealed class CapabilityInvoker
{
private readonly DriverResiliencePipelineBuilder _builder;
private readonly string _driverInstanceId;
private readonly string _driverType;
private readonly Func<DriverResilienceOptions> _optionsAccessor;
/// <summary>
@@ -30,16 +32,19 @@ public sealed class CapabilityInvoker
/// Snapshot accessor for the current resilience options. Invoked per call so Admin-edit +
/// pipeline-invalidate can take effect without restarting the invoker.
/// </param>
/// <param name="driverType">Driver type name for structured-log enrichment (e.g. <c>"Modbus"</c>).</param>
public CapabilityInvoker(
DriverResiliencePipelineBuilder builder,
string driverInstanceId,
Func<DriverResilienceOptions> optionsAccessor)
Func<DriverResilienceOptions> optionsAccessor,
string driverType = "Unknown")
{
ArgumentNullException.ThrowIfNull(builder);
ArgumentNullException.ThrowIfNull(optionsAccessor);
_builder = builder;
_driverInstanceId = driverInstanceId;
_driverType = driverType;
_optionsAccessor = optionsAccessor;
}
@@ -54,8 +59,11 @@ public sealed class CapabilityInvoker
ArgumentNullException.ThrowIfNull(callSite);
var pipeline = ResolvePipeline(capability, hostName);
using (LogContextEnricher.Push(_driverInstanceId, _driverType, capability, LogContextEnricher.NewCorrelationId()))
{
return await pipeline.ExecuteAsync(callSite, cancellationToken).ConfigureAwait(false);
}
}
/// <summary>Execute a void-returning capability call, honoring the per-capability pipeline.</summary>
public async ValueTask ExecuteAsync(
@@ -67,8 +75,11 @@ public sealed class CapabilityInvoker
ArgumentNullException.ThrowIfNull(callSite);
var pipeline = ResolvePipeline(capability, hostName);
using (LogContextEnricher.Push(_driverInstanceId, _driverType, capability, LogContextEnricher.NewCorrelationId()))
{
await pipeline.ExecuteAsync(callSite, cancellationToken).ConfigureAwait(false);
}
}
/// <summary>
/// Execute a <see cref="DriverCapability.Write"/> call honoring <see cref="WriteIdempotentAttribute"/>
@@ -95,8 +106,11 @@ public sealed class CapabilityInvoker
},
};
var pipeline = _builder.GetOrCreate(_driverInstanceId, $"{hostName}::non-idempotent", DriverCapability.Write, noRetryOptions);
using (LogContextEnricher.Push(_driverInstanceId, _driverType, DriverCapability.Write, LogContextEnricher.NewCorrelationId()))
{
return await pipeline.ExecuteAsync(callSite, cancellationToken).ConfigureAwait(false);
}
}
return await ExecuteAsync(DriverCapability.Write, hostName, callSite, cancellationToken).ConfigureAwait(false);
}

View File

@@ -0,0 +1,104 @@
using System.Collections.Concurrent;
namespace ZB.MOM.WW.OtOpcUa.Core.Resilience;
/// <summary>
/// Process-singleton tracker of live resilience counters per
/// <c>(DriverInstanceId, HostName)</c>. Populated by the CapabilityInvoker and the
/// MemoryTracking layer; consumed by a HostedService that periodically persists a
/// snapshot to the <c>DriverInstanceResilienceStatus</c> table for Admin <c>/hosts</c>.
/// </summary>
/// <remarks>
/// Per Phase 6.1 Stream E. No DB dependency here — the tracker is pure in-memory so
/// tests can exercise it without EF Core or SQL Server. The HostedService that writes
/// snapshots lives in the Server project (Stream E.2); the actual SignalR push + Blazor
/// page refresh (E.3) lands in a follow-up visual-review PR.
/// </remarks>
public sealed class DriverResilienceStatusTracker
{
private readonly ConcurrentDictionary<StatusKey, ResilienceStatusSnapshot> _status = new();
/// <summary>Record a Polly pipeline failure for <paramref name="hostName"/>.</summary>
public void RecordFailure(string driverInstanceId, string hostName, DateTime utcNow)
{
var key = new StatusKey(driverInstanceId, hostName);
_status.AddOrUpdate(key,
_ => new ResilienceStatusSnapshot { ConsecutiveFailures = 1, LastSampledUtc = utcNow },
(_, existing) => existing with
{
ConsecutiveFailures = existing.ConsecutiveFailures + 1,
LastSampledUtc = utcNow,
});
}
/// <summary>Reset the consecutive-failure count on a successful pipeline execution.</summary>
public void RecordSuccess(string driverInstanceId, string hostName, DateTime utcNow)
{
var key = new StatusKey(driverInstanceId, hostName);
_status.AddOrUpdate(key,
_ => new ResilienceStatusSnapshot { ConsecutiveFailures = 0, LastSampledUtc = utcNow },
(_, existing) => existing with
{
ConsecutiveFailures = 0,
LastSampledUtc = utcNow,
});
}
/// <summary>Record a circuit-breaker open event.</summary>
public void RecordBreakerOpen(string driverInstanceId, string hostName, DateTime utcNow)
{
var key = new StatusKey(driverInstanceId, hostName);
_status.AddOrUpdate(key,
_ => new ResilienceStatusSnapshot { LastBreakerOpenUtc = utcNow, LastSampledUtc = utcNow },
(_, existing) => existing with { LastBreakerOpenUtc = utcNow, LastSampledUtc = utcNow });
}
/// <summary>Record a process recycle event (Tier C only).</summary>
public void RecordRecycle(string driverInstanceId, string hostName, DateTime utcNow)
{
var key = new StatusKey(driverInstanceId, hostName);
_status.AddOrUpdate(key,
_ => new ResilienceStatusSnapshot { LastRecycleUtc = utcNow, LastSampledUtc = utcNow },
(_, existing) => existing with { LastRecycleUtc = utcNow, LastSampledUtc = utcNow });
}
/// <summary>Capture / update the MemoryTracking-supplied baseline + current footprint.</summary>
public void RecordFootprint(string driverInstanceId, string hostName, long baselineBytes, long currentBytes, DateTime utcNow)
{
var key = new StatusKey(driverInstanceId, hostName);
_status.AddOrUpdate(key,
_ => new ResilienceStatusSnapshot
{
BaselineFootprintBytes = baselineBytes,
CurrentFootprintBytes = currentBytes,
LastSampledUtc = utcNow,
},
(_, existing) => existing with
{
BaselineFootprintBytes = baselineBytes,
CurrentFootprintBytes = currentBytes,
LastSampledUtc = utcNow,
});
}
/// <summary>Snapshot of a specific (instance, host) pair; null if no counters recorded yet.</summary>
public ResilienceStatusSnapshot? TryGet(string driverInstanceId, string hostName) =>
_status.TryGetValue(new StatusKey(driverInstanceId, hostName), out var snapshot) ? snapshot : null;
/// <summary>Copy of every currently-tracked (instance, host, snapshot) triple. Safe under concurrent writes.</summary>
public IReadOnlyList<(string DriverInstanceId, string HostName, ResilienceStatusSnapshot Snapshot)> Snapshot() =>
_status.Select(kvp => (kvp.Key.DriverInstanceId, kvp.Key.HostName, kvp.Value)).ToList();
private readonly record struct StatusKey(string DriverInstanceId, string HostName);
}
/// <summary>Snapshot of the resilience counters for one <c>(DriverInstanceId, HostName)</c> pair.</summary>
public sealed record ResilienceStatusSnapshot
{
public int ConsecutiveFailures { get; init; }
public DateTime? LastBreakerOpenUtc { get; init; }
public DateTime? LastRecycleUtc { get; init; }
public long BaselineFootprintBytes { get; init; }
public long CurrentFootprintBytes { get; init; }
public DateTime LastSampledUtc { get; init; }
}

View File

@@ -18,6 +18,7 @@
<ItemGroup>
<PackageReference Include="Polly.Core" Version="8.6.6"/>
<PackageReference Include="Serilog" Version="4.3.0"/>
</ItemGroup>
<ItemGroup>

View File

@@ -0,0 +1,181 @@
using System.Net;
using System.Text;
using System.Text.Json;
using Microsoft.Extensions.Logging;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Core.Hosting;
using ZB.MOM.WW.OtOpcUa.Core.Observability;
namespace ZB.MOM.WW.OtOpcUa.Server.Observability;
/// <summary>
/// Standalone <see cref="HttpListener"/> host for <c>/healthz</c> and <c>/readyz</c>
/// separate from the OPC UA binding. Per <c>docs/v2/implementation/phase-6-1-resilience-
/// and-observability.md</c> §Stream C.1.
/// </summary>
/// <remarks>
/// Binds to <c>http://localhost:4841</c> by default — loopback avoids the Windows URL-ACL
/// elevation requirement that binding to <c>http://+:4841</c> (wildcard) would impose.
/// When a deployment needs remote probing, a reverse proxy or explicit netsh urlacl grant
/// is the expected path; documented in <c>docs/v2/Server-Deployment.md</c> in a follow-up.
/// </remarks>
public sealed class HealthEndpointsHost : IAsyncDisposable
{
private readonly string _prefix;
private readonly DriverHost _driverHost;
private readonly Func<bool> _configDbHealthy;
private readonly Func<bool> _usingStaleConfig;
private readonly ILogger<HealthEndpointsHost> _logger;
private readonly HttpListener _listener = new();
private readonly DateTime _startedUtc = DateTime.UtcNow;
private CancellationTokenSource? _cts;
private Task? _acceptLoop;
private bool _disposed;
public HealthEndpointsHost(
DriverHost driverHost,
ILogger<HealthEndpointsHost> logger,
Func<bool>? configDbHealthy = null,
Func<bool>? usingStaleConfig = null,
string prefix = "http://localhost:4841/")
{
_driverHost = driverHost;
_logger = logger;
_configDbHealthy = configDbHealthy ?? (() => true);
_usingStaleConfig = usingStaleConfig ?? (() => false);
_prefix = prefix.EndsWith('/') ? prefix : prefix + "/";
_listener.Prefixes.Add(_prefix);
}
public void Start()
{
_listener.Start();
_cts = new CancellationTokenSource();
_acceptLoop = Task.Run(() => AcceptLoopAsync(_cts.Token));
_logger.LogInformation("Health endpoints listening on {Prefix}", _prefix);
}
private async Task AcceptLoopAsync(CancellationToken ct)
{
while (!ct.IsCancellationRequested)
{
HttpListenerContext ctx;
try
{
ctx = await _listener.GetContextAsync().ConfigureAwait(false);
}
catch (HttpListenerException) when (ct.IsCancellationRequested) { break; }
catch (ObjectDisposedException) { break; }
_ = Task.Run(() => HandleAsync(ctx), ct);
}
}
private async Task HandleAsync(HttpListenerContext ctx)
{
try
{
var path = ctx.Request.Url?.AbsolutePath ?? "/";
switch (path)
{
case "/healthz":
await WriteHealthzAsync(ctx).ConfigureAwait(false);
break;
case "/readyz":
await WriteReadyzAsync(ctx).ConfigureAwait(false);
break;
default:
ctx.Response.StatusCode = 404;
break;
}
}
catch (Exception ex)
{
_logger.LogWarning(ex, "Health endpoint handler failure");
try { ctx.Response.StatusCode = 500; } catch { /* ignore */ }
}
finally
{
try { ctx.Response.Close(); } catch { /* ignore */ }
}
}
private async Task WriteHealthzAsync(HttpListenerContext ctx)
{
var configHealthy = _configDbHealthy();
var staleConfig = _usingStaleConfig();
// /healthz is 200 when process alive + (config DB reachable OR cache-warm).
// Stale-config still serves 200 so the process isn't flagged dead when the DB
// blips; the body surfaces the stale flag for operators.
var healthy = configHealthy || staleConfig;
ctx.Response.StatusCode = healthy ? 200 : 503;
var body = JsonSerializer.Serialize(new
{
status = healthy ? "healthy" : "unhealthy",
uptimeSeconds = (int)(DateTime.UtcNow - _startedUtc).TotalSeconds,
configDbReachable = configHealthy,
usingStaleConfig = staleConfig,
});
await WriteBodyAsync(ctx, body).ConfigureAwait(false);
}
private async Task WriteReadyzAsync(HttpListenerContext ctx)
{
var snapshots = BuildSnapshots();
var verdict = DriverHealthReport.Aggregate(snapshots);
ctx.Response.StatusCode = DriverHealthReport.HttpStatus(verdict);
var body = JsonSerializer.Serialize(new
{
verdict = verdict.ToString(),
uptimeSeconds = (int)(DateTime.UtcNow - _startedUtc).TotalSeconds,
drivers = snapshots.Select(d => new
{
id = d.DriverInstanceId,
state = d.State.ToString(),
detail = d.DetailMessage,
}).ToArray(),
degradedDrivers = snapshots
.Where(d => d.State == DriverState.Degraded || d.State == DriverState.Reconnecting)
.Select(d => d.DriverInstanceId)
.ToArray(),
});
await WriteBodyAsync(ctx, body).ConfigureAwait(false);
}
private IReadOnlyList<DriverHealthSnapshot> BuildSnapshots()
{
var list = new List<DriverHealthSnapshot>();
foreach (var id in _driverHost.RegisteredDriverIds)
{
var driver = _driverHost.GetDriver(id);
if (driver is null) continue;
var health = driver.GetHealth();
list.Add(new DriverHealthSnapshot(driver.DriverInstanceId, health.State, health.LastError));
}
return list;
}
private static async Task WriteBodyAsync(HttpListenerContext ctx, string body)
{
var bytes = Encoding.UTF8.GetBytes(body);
ctx.Response.ContentType = "application/json; charset=utf-8";
ctx.Response.ContentLength64 = bytes.LongLength;
await ctx.Response.OutputStream.WriteAsync(bytes).ConfigureAwait(false);
}
public async ValueTask DisposeAsync()
{
if (_disposed) return;
_disposed = true;
_cts?.Cancel();
try { _listener.Stop(); } catch { /* ignore */ }
if (_acceptLoop is not null)
{
try { await _acceptLoop.ConfigureAwait(false); } catch { /* ignore */ }
}
_listener.Close();
_cts?.Dispose();
}
}

View File

@@ -4,6 +4,7 @@ using Opc.Ua.Configuration;
using ZB.MOM.WW.OtOpcUa.Core.Hosting;
using ZB.MOM.WW.OtOpcUa.Core.OpcUa;
using ZB.MOM.WW.OtOpcUa.Core.Resilience;
using ZB.MOM.WW.OtOpcUa.Server.Observability;
using ZB.MOM.WW.OtOpcUa.Server.Security;
namespace ZB.MOM.WW.OtOpcUa.Server.OpcUa;
@@ -26,6 +27,7 @@ public sealed class OpcUaApplicationHost : IAsyncDisposable
private readonly ILogger<OpcUaApplicationHost> _logger;
private ApplicationInstance? _application;
private OtOpcUaServer? _server;
private HealthEndpointsHost? _healthHost;
private bool _disposed;
public OpcUaApplicationHost(OpcUaServerOptions options, DriverHost driverHost,
@@ -68,6 +70,17 @@ public sealed class OpcUaApplicationHost : IAsyncDisposable
_logger.LogInformation("OPC UA server started — endpoint={Endpoint} driverCount={Count}",
_options.EndpointUrl, _server.DriverNodeManagers.Count);
// Phase 6.1 Stream C: health endpoints on :4841 (loopback by default — see
// HealthEndpointsHost remarks for the Windows URL-ACL tradeoff).
if (_options.HealthEndpointsEnabled)
{
_healthHost = new HealthEndpointsHost(
_driverHost,
_loggerFactory.CreateLogger<HealthEndpointsHost>(),
prefix: _options.HealthEndpointsPrefix);
_healthHost.Start();
}
// Drive each driver's discovery through its node manager. The node manager IS the
// IAddressSpaceBuilder; GenericDriverNodeManager captures alarm-condition sinks into
// its internal map and wires OnAlarmEvent → sink routing.
@@ -221,6 +234,12 @@ public sealed class OpcUaApplicationHost : IAsyncDisposable
{
_logger.LogWarning(ex, "OPC UA server stop threw during dispose");
}
if (_healthHost is not null)
{
try { await _healthHost.DisposeAsync().ConfigureAwait(false); }
catch (Exception ex) { _logger.LogWarning(ex, "Health endpoints host dispose threw"); }
}
await Task.CompletedTask;
}
}

View File

@@ -58,6 +58,20 @@ public sealed class OpcUaServerOptions
/// </summary>
public bool AutoAcceptUntrustedClientCertificates { get; init; } = true;
/// <summary>
/// Whether to start the Phase 6.1 Stream C <c>/healthz</c> + <c>/readyz</c> HTTP listener.
/// Defaults to <c>true</c>; set false in embedded deployments that don't need HTTP
/// (e.g. tests that only exercise the OPC UA surface).
/// </summary>
public bool HealthEndpointsEnabled { get; init; } = true;
/// <summary>
/// URL prefix the health endpoints bind to. Default <c>http://localhost:4841/</c> — loopback
/// avoids Windows URL-ACL elevation. Production deployments that need remote probing should
/// either reverse-proxy or use <c>http://+:4841/</c> with netsh urlacl granted.
/// </summary>
public string HealthEndpointsPrefix { get; init; } = "http://localhost:4841/";
/// <summary>
/// Security profile advertised on the endpoint. Default <see cref="OpcUaSecurityProfile.None"/>
/// preserves the PR 17 endpoint shape; set to <see cref="OpcUaSecurityProfile.Basic256Sha256SignAndEncrypt"/>

View File

@@ -57,7 +57,7 @@ public sealed class OtOpcUaServer : StandardServer
// per-type tiers into DriverTypeRegistry. Read ResilienceConfig JSON from the
// DriverInstance row in a follow-up PR; for now every driver gets Tier A defaults.
var options = new DriverResilienceOptions { Tier = DriverTier.A };
var invoker = new CapabilityInvoker(_pipelineBuilder, driver.DriverInstanceId, () => options);
var invoker = new CapabilityInvoker(_pipelineBuilder, driver.DriverInstanceId, () => options, driver.DriverType);
var manager = new DriverNodeManager(server, configuration, driver, invoker, logger);
_driverNodeManagers.Add(manager);
}

View File

@@ -4,6 +4,7 @@ using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
using Serilog;
using Serilog.Formatting.Compact;
using ZB.MOM.WW.OtOpcUa.Configuration;
using ZB.MOM.WW.OtOpcUa.Configuration.LocalCache;
using ZB.MOM.WW.OtOpcUa.Core.Hosting;
@@ -13,11 +14,25 @@ using ZB.MOM.WW.OtOpcUa.Server.Security;
var builder = Host.CreateApplicationBuilder(args);
Log.Logger = new LoggerConfiguration()
// Per Phase 6.1 Stream C.3: SIEMs (Splunk, Datadog) ingest the JSON file without a
// regex parser. Plain-text rolling file stays on by default for human readability;
// JSON file is opt-in via appsetting `Serilog:WriteJson = true`.
var writeJson = builder.Configuration.GetValue<bool>("Serilog:WriteJson");
var loggerBuilder = new LoggerConfiguration()
.ReadFrom.Configuration(builder.Configuration)
.Enrich.FromLogContext()
.WriteTo.Console()
.WriteTo.File("logs/otopcua-.log", rollingInterval: RollingInterval.Day)
.CreateLogger();
.WriteTo.File("logs/otopcua-.log", rollingInterval: RollingInterval.Day);
if (writeJson)
{
loggerBuilder = loggerBuilder.WriteTo.File(
new CompactJsonFormatter(),
"logs/otopcua-.json.log",
rollingInterval: RollingInterval.Day);
}
Log.Logger = loggerBuilder.CreateLogger();
builder.Services.AddSerilog();
builder.Services.AddWindowsService(o => o.ServiceName = "OtOpcUa");

View File

@@ -21,6 +21,7 @@
<PackageReference Include="Serilog.Settings.Configuration" Version="9.0.0"/>
<PackageReference Include="Serilog.Sinks.Console" Version="6.0.0"/>
<PackageReference Include="Serilog.Sinks.File" Version="7.0.0"/>
<PackageReference Include="Serilog.Formatting.Compact" Version="3.0.0"/>
<PackageReference Include="OPCFoundation.NetStandard.Opc.Ua.Server" Version="1.5.374.126"/>
<PackageReference Include="OPCFoundation.NetStandard.Opc.Ua.Configuration" Version="1.5.374.126"/>
<PackageReference Include="Novell.Directory.Ldap.NETStandard" Version="3.6.0"/>

View File

@@ -0,0 +1,157 @@
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Configuration.LocalCache;
namespace ZB.MOM.WW.OtOpcUa.Configuration.Tests;
[Trait("Category", "Unit")]
public sealed class GenerationSealedCacheTests : IDisposable
{
private readonly string _root = Path.Combine(Path.GetTempPath(), $"otopcua-sealed-{Guid.NewGuid():N}");
public void Dispose()
{
try
{
if (!Directory.Exists(_root)) return;
// Remove ReadOnly attribute first so Directory.Delete can clean sealed files.
foreach (var f in Directory.EnumerateFiles(_root, "*", SearchOption.AllDirectories))
File.SetAttributes(f, FileAttributes.Normal);
Directory.Delete(_root, recursive: true);
}
catch { /* best-effort cleanup */ }
}
private GenerationSnapshot MakeSnapshot(string clusterId, long generationId, string payload = "{\"sample\":true}") =>
new()
{
ClusterId = clusterId,
GenerationId = generationId,
CachedAt = DateTime.UtcNow,
PayloadJson = payload,
};
[Fact]
public async Task FirstBoot_NoSnapshot_ReadThrows()
{
var cache = new GenerationSealedCache(_root);
await Should.ThrowAsync<GenerationCacheUnavailableException>(
() => cache.ReadCurrentAsync("cluster-a"));
}
[Fact]
public async Task SealThenRead_RoundTrips()
{
var cache = new GenerationSealedCache(_root);
var snapshot = MakeSnapshot("cluster-a", 42, "{\"hello\":\"world\"}");
await cache.SealAsync(snapshot);
var read = await cache.ReadCurrentAsync("cluster-a");
read.GenerationId.ShouldBe(42);
read.ClusterId.ShouldBe("cluster-a");
read.PayloadJson.ShouldBe("{\"hello\":\"world\"}");
}
[Fact]
public async Task SealedFile_IsReadOnly_OnDisk()
{
var cache = new GenerationSealedCache(_root);
await cache.SealAsync(MakeSnapshot("cluster-a", 5));
var sealedPath = Path.Combine(_root, "cluster-a", "5.db");
File.Exists(sealedPath).ShouldBeTrue();
var attrs = File.GetAttributes(sealedPath);
attrs.HasFlag(FileAttributes.ReadOnly).ShouldBeTrue("sealed file must be read-only");
}
[Fact]
public async Task SealingTwoGenerations_PointerAdvances_ToLatest()
{
var cache = new GenerationSealedCache(_root);
await cache.SealAsync(MakeSnapshot("cluster-a", 1));
await cache.SealAsync(MakeSnapshot("cluster-a", 2));
cache.TryGetCurrentGenerationId("cluster-a").ShouldBe(2);
var read = await cache.ReadCurrentAsync("cluster-a");
read.GenerationId.ShouldBe(2);
}
[Fact]
public async Task PriorGenerationFile_Survives_AfterNewSeal()
{
var cache = new GenerationSealedCache(_root);
await cache.SealAsync(MakeSnapshot("cluster-a", 1));
await cache.SealAsync(MakeSnapshot("cluster-a", 2));
File.Exists(Path.Combine(_root, "cluster-a", "1.db")).ShouldBeTrue(
"prior generations preserved for audit; pruning is separate");
File.Exists(Path.Combine(_root, "cluster-a", "2.db")).ShouldBeTrue();
}
[Fact]
public async Task CorruptSealedFile_ReadFailsClosed()
{
var cache = new GenerationSealedCache(_root);
await cache.SealAsync(MakeSnapshot("cluster-a", 7));
// Corrupt the sealed file: clear read-only, truncate, leave pointer intact.
var sealedPath = Path.Combine(_root, "cluster-a", "7.db");
File.SetAttributes(sealedPath, FileAttributes.Normal);
File.WriteAllBytes(sealedPath, [0x00, 0x01, 0x02]);
await Should.ThrowAsync<GenerationCacheUnavailableException>(
() => cache.ReadCurrentAsync("cluster-a"));
}
[Fact]
public async Task MissingSealedFile_ReadFailsClosed()
{
var cache = new GenerationSealedCache(_root);
await cache.SealAsync(MakeSnapshot("cluster-a", 3));
// Delete the sealed file but leave the pointer — corruption scenario.
var sealedPath = Path.Combine(_root, "cluster-a", "3.db");
File.SetAttributes(sealedPath, FileAttributes.Normal);
File.Delete(sealedPath);
await Should.ThrowAsync<GenerationCacheUnavailableException>(
() => cache.ReadCurrentAsync("cluster-a"));
}
[Fact]
public async Task CorruptPointerFile_ReadFailsClosed()
{
var cache = new GenerationSealedCache(_root);
await cache.SealAsync(MakeSnapshot("cluster-a", 9));
var pointerPath = Path.Combine(_root, "cluster-a", "CURRENT");
File.WriteAllText(pointerPath, "not-a-number");
await Should.ThrowAsync<GenerationCacheUnavailableException>(
() => cache.ReadCurrentAsync("cluster-a"));
}
[Fact]
public async Task SealSameGenerationTwice_IsIdempotent()
{
var cache = new GenerationSealedCache(_root);
await cache.SealAsync(MakeSnapshot("cluster-a", 11));
await cache.SealAsync(MakeSnapshot("cluster-a", 11, "{\"v\":2}"));
var read = await cache.ReadCurrentAsync("cluster-a");
read.PayloadJson.ShouldBe("{\"sample\":true}", "sealed file is immutable; second seal no-ops");
}
[Fact]
public async Task IndependentClusters_DoNotInterfere()
{
var cache = new GenerationSealedCache(_root);
await cache.SealAsync(MakeSnapshot("cluster-a", 1));
await cache.SealAsync(MakeSnapshot("cluster-b", 10));
(await cache.ReadCurrentAsync("cluster-a")).GenerationId.ShouldBe(1);
(await cache.ReadCurrentAsync("cluster-b")).GenerationId.ShouldBe(10);
}
}

View File

@@ -0,0 +1,138 @@
using Microsoft.EntityFrameworkCore;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Configuration.Entities;
using ZB.MOM.WW.OtOpcUa.Configuration.Enums;
using ZB.MOM.WW.OtOpcUa.Configuration.Services;
namespace ZB.MOM.WW.OtOpcUa.Configuration.Tests;
[Trait("Category", "Unit")]
public sealed class LdapGroupRoleMappingServiceTests : IDisposable
{
private readonly OtOpcUaConfigDbContext _db;
public LdapGroupRoleMappingServiceTests()
{
var options = new DbContextOptionsBuilder<OtOpcUaConfigDbContext>()
.UseInMemoryDatabase($"ldap-grm-{Guid.NewGuid():N}")
.Options;
_db = new OtOpcUaConfigDbContext(options);
}
public void Dispose() => _db.Dispose();
private LdapGroupRoleMapping Make(string group, AdminRole role, string? clusterId = null, bool? isSystemWide = null) =>
new()
{
LdapGroup = group,
Role = role,
ClusterId = clusterId,
IsSystemWide = isSystemWide ?? (clusterId is null),
};
[Fact]
public async Task Create_SetsId_AndCreatedAtUtc()
{
var svc = new LdapGroupRoleMappingService(_db);
var row = Make("cn=fleet,dc=x", AdminRole.FleetAdmin);
var saved = await svc.CreateAsync(row, CancellationToken.None);
saved.Id.ShouldNotBe(Guid.Empty);
saved.CreatedAtUtc.ShouldBeGreaterThan(DateTime.UtcNow.AddMinutes(-1));
}
[Fact]
public async Task Create_Rejects_EmptyLdapGroup()
{
var svc = new LdapGroupRoleMappingService(_db);
var row = Make("", AdminRole.FleetAdmin);
await Should.ThrowAsync<InvalidLdapGroupRoleMappingException>(
() => svc.CreateAsync(row, CancellationToken.None));
}
[Fact]
public async Task Create_Rejects_SystemWide_With_ClusterId()
{
var svc = new LdapGroupRoleMappingService(_db);
var row = Make("cn=g", AdminRole.ConfigViewer, clusterId: "c1", isSystemWide: true);
await Should.ThrowAsync<InvalidLdapGroupRoleMappingException>(
() => svc.CreateAsync(row, CancellationToken.None));
}
[Fact]
public async Task Create_Rejects_NonSystemWide_WithoutClusterId()
{
var svc = new LdapGroupRoleMappingService(_db);
var row = Make("cn=g", AdminRole.ConfigViewer, clusterId: null, isSystemWide: false);
await Should.ThrowAsync<InvalidLdapGroupRoleMappingException>(
() => svc.CreateAsync(row, CancellationToken.None));
}
[Fact]
public async Task GetByGroups_Returns_MatchingGrants_Only()
{
var svc = new LdapGroupRoleMappingService(_db);
await svc.CreateAsync(Make("cn=fleet,dc=x", AdminRole.FleetAdmin), CancellationToken.None);
await svc.CreateAsync(Make("cn=editor,dc=x", AdminRole.ConfigEditor), CancellationToken.None);
await svc.CreateAsync(Make("cn=viewer,dc=x", AdminRole.ConfigViewer), CancellationToken.None);
var results = await svc.GetByGroupsAsync(
["cn=fleet,dc=x", "cn=viewer,dc=x"], CancellationToken.None);
results.Count.ShouldBe(2);
results.Select(r => r.Role).ShouldBe([AdminRole.FleetAdmin, AdminRole.ConfigViewer], ignoreOrder: true);
}
[Fact]
public async Task GetByGroups_Empty_Input_ReturnsEmpty()
{
var svc = new LdapGroupRoleMappingService(_db);
await svc.CreateAsync(Make("cn=fleet,dc=x", AdminRole.FleetAdmin), CancellationToken.None);
var results = await svc.GetByGroupsAsync([], CancellationToken.None);
results.ShouldBeEmpty();
}
[Fact]
public async Task ListAll_Orders_ByGroupThenCluster()
{
var svc = new LdapGroupRoleMappingService(_db);
await svc.CreateAsync(Make("cn=b,dc=x", AdminRole.FleetAdmin), CancellationToken.None);
await svc.CreateAsync(Make("cn=a,dc=x", AdminRole.ConfigEditor, clusterId: "c2", isSystemWide: false), CancellationToken.None);
await svc.CreateAsync(Make("cn=a,dc=x", AdminRole.ConfigEditor, clusterId: "c1", isSystemWide: false), CancellationToken.None);
var results = await svc.ListAllAsync(CancellationToken.None);
results[0].LdapGroup.ShouldBe("cn=a,dc=x");
results[0].ClusterId.ShouldBe("c1");
results[1].ClusterId.ShouldBe("c2");
results[2].LdapGroup.ShouldBe("cn=b,dc=x");
}
[Fact]
public async Task Delete_Removes_Matching_Row()
{
var svc = new LdapGroupRoleMappingService(_db);
var saved = await svc.CreateAsync(Make("cn=fleet,dc=x", AdminRole.FleetAdmin), CancellationToken.None);
await svc.DeleteAsync(saved.Id, CancellationToken.None);
var after = await svc.ListAllAsync(CancellationToken.None);
after.ShouldBeEmpty();
}
[Fact]
public async Task Delete_Unknown_Id_IsNoOp()
{
var svc = new LdapGroupRoleMappingService(_db);
await svc.DeleteAsync(Guid.NewGuid(), CancellationToken.None);
// no exception
}
}

View File

@@ -0,0 +1,154 @@
using Microsoft.Extensions.Logging.Abstractions;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Configuration.LocalCache;
namespace ZB.MOM.WW.OtOpcUa.Configuration.Tests;
[Trait("Category", "Unit")]
public sealed class ResilientConfigReaderTests : IDisposable
{
private readonly string _root = Path.Combine(Path.GetTempPath(), $"otopcua-reader-{Guid.NewGuid():N}");
public void Dispose()
{
try
{
if (!Directory.Exists(_root)) return;
foreach (var f in Directory.EnumerateFiles(_root, "*", SearchOption.AllDirectories))
File.SetAttributes(f, FileAttributes.Normal);
Directory.Delete(_root, recursive: true);
}
catch { /* best-effort */ }
}
[Fact]
public async Task CentralDbSucceeds_ReturnsValue_MarksFresh()
{
var cache = new GenerationSealedCache(_root);
var flag = new StaleConfigFlag { };
flag.MarkStale(); // pre-existing stale state
var reader = new ResilientConfigReader(cache, flag, NullLogger<ResilientConfigReader>.Instance);
var result = await reader.ReadAsync(
"cluster-a",
_ => ValueTask.FromResult("fresh-from-db"),
_ => "from-cache",
CancellationToken.None);
result.ShouldBe("fresh-from-db");
flag.IsStale.ShouldBeFalse("successful central-DB read clears stale flag");
}
[Fact]
public async Task CentralDbFails_ExhaustsRetries_FallsBackToCache_MarksStale()
{
var cache = new GenerationSealedCache(_root);
await cache.SealAsync(new GenerationSnapshot
{
ClusterId = "cluster-a", GenerationId = 99, CachedAt = DateTime.UtcNow,
PayloadJson = "{\"cached\":true}",
});
var flag = new StaleConfigFlag();
var reader = new ResilientConfigReader(cache, flag, NullLogger<ResilientConfigReader>.Instance,
timeout: TimeSpan.FromSeconds(10), retryCount: 2);
var attempts = 0;
var result = await reader.ReadAsync(
"cluster-a",
_ =>
{
attempts++;
throw new InvalidOperationException("SQL dead");
#pragma warning disable CS0162
return ValueTask.FromResult("never");
#pragma warning restore CS0162
},
snap => snap.PayloadJson,
CancellationToken.None);
attempts.ShouldBe(3, "1 initial + 2 retries = 3 attempts");
result.ShouldBe("{\"cached\":true}");
flag.IsStale.ShouldBeTrue("cache fallback flips stale flag true");
}
[Fact]
public async Task CentralDbFails_AndCacheAlsoUnavailable_Throws()
{
var cache = new GenerationSealedCache(_root);
var flag = new StaleConfigFlag();
var reader = new ResilientConfigReader(cache, flag, NullLogger<ResilientConfigReader>.Instance,
timeout: TimeSpan.FromSeconds(10), retryCount: 0);
await Should.ThrowAsync<GenerationCacheUnavailableException>(async () =>
{
await reader.ReadAsync<string>(
"cluster-a",
_ => throw new InvalidOperationException("SQL dead"),
_ => "never",
CancellationToken.None);
});
flag.IsStale.ShouldBeFalse("no snapshot ever served, so flag stays whatever it was");
}
[Fact]
public async Task Cancellation_NotRetried()
{
var cache = new GenerationSealedCache(_root);
var flag = new StaleConfigFlag();
var reader = new ResilientConfigReader(cache, flag, NullLogger<ResilientConfigReader>.Instance,
timeout: TimeSpan.FromSeconds(10), retryCount: 5);
using var cts = new CancellationTokenSource();
cts.Cancel();
var attempts = 0;
await Should.ThrowAsync<OperationCanceledException>(async () =>
{
await reader.ReadAsync<string>(
"cluster-a",
ct =>
{
attempts++;
ct.ThrowIfCancellationRequested();
return ValueTask.FromResult("ok");
},
_ => "cache",
cts.Token);
});
attempts.ShouldBeLessThanOrEqualTo(1);
}
}
[Trait("Category", "Unit")]
public sealed class StaleConfigFlagTests
{
[Fact]
public void Default_IsFresh()
{
new StaleConfigFlag().IsStale.ShouldBeFalse();
}
[Fact]
public void MarkStale_ThenFresh_Toggles()
{
var flag = new StaleConfigFlag();
flag.MarkStale();
flag.IsStale.ShouldBeTrue();
flag.MarkFresh();
flag.IsStale.ShouldBeFalse();
}
[Fact]
public void ConcurrentWrites_Converge()
{
var flag = new StaleConfigFlag();
Parallel.For(0, 1000, i =>
{
if (i % 2 == 0) flag.MarkStale(); else flag.MarkFresh();
});
flag.MarkFresh();
flag.IsStale.ShouldBeFalse();
}
}

View File

@@ -29,6 +29,8 @@ public sealed class SchemaComplianceTests
"DriverInstance", "Device", "Equipment", "Tag", "PollGroup",
"NodeAcl", "ExternalIdReservation",
"DriverHostStatus",
"DriverInstanceResilienceStatus",
"LdapGroupRoleMapping",
};
var actual = QueryStrings(@"

View File

@@ -14,6 +14,7 @@
<PackageReference Include="Shouldly" Version="4.3.0"/>
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.12.0"/>
<PackageReference Include="Microsoft.Data.SqlClient" Version="6.1.1"/>
<PackageReference Include="Microsoft.EntityFrameworkCore.InMemory" Version="10.0.0"/>
<PackageReference Include="xunit.runner.visualstudio" Version="3.0.2">
<PrivateAssets>all</PrivateAssets>
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>

View File

@@ -0,0 +1,72 @@
using Serilog;
using Serilog.Core;
using Serilog.Events;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Core.Resilience;
namespace ZB.MOM.WW.OtOpcUa.Core.Tests.Observability;
[Trait("Category", "Integration")]
public sealed class CapabilityInvokerEnrichmentTests
{
[Fact]
public async Task InvokerExecute_LogsInsideCallSite_CarryStructuredProperties()
{
var sink = new InMemorySink();
var logger = new LoggerConfiguration()
.Enrich.FromLogContext()
.WriteTo.Sink(sink)
.CreateLogger();
var invoker = new CapabilityInvoker(
new DriverResiliencePipelineBuilder(),
driverInstanceId: "drv-live",
optionsAccessor: () => new DriverResilienceOptions { Tier = DriverTier.A },
driverType: "Modbus");
await invoker.ExecuteAsync(
DriverCapability.Read,
"plc-1",
ct =>
{
logger.Information("inside call site");
return ValueTask.FromResult(42);
},
CancellationToken.None);
var evt = sink.Events.ShouldHaveSingleItem();
evt.Properties["DriverInstanceId"].ToString().ShouldBe("\"drv-live\"");
evt.Properties["DriverType"].ToString().ShouldBe("\"Modbus\"");
evt.Properties["CapabilityName"].ToString().ShouldBe("\"Read\"");
evt.Properties.ShouldContainKey("CorrelationId");
}
[Fact]
public async Task InvokerExecute_DoesNotLeak_ContextOutsideCallSite()
{
var sink = new InMemorySink();
var logger = new LoggerConfiguration()
.Enrich.FromLogContext()
.WriteTo.Sink(sink)
.CreateLogger();
var invoker = new CapabilityInvoker(
new DriverResiliencePipelineBuilder(),
driverInstanceId: "drv-a",
optionsAccessor: () => new DriverResilienceOptions { Tier = DriverTier.A });
await invoker.ExecuteAsync(DriverCapability.Read, "host", _ => ValueTask.FromResult(1), CancellationToken.None);
logger.Information("outside");
var outside = sink.Events.ShouldHaveSingleItem();
outside.Properties.ContainsKey("DriverInstanceId").ShouldBeFalse();
}
private sealed class InMemorySink : ILogEventSink
{
public List<LogEvent> Events { get; } = [];
public void Emit(LogEvent logEvent) => Events.Add(logEvent);
}
}

View File

@@ -0,0 +1,70 @@
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Core.Observability;
namespace ZB.MOM.WW.OtOpcUa.Core.Tests.Observability;
[Trait("Category", "Unit")]
public sealed class DriverHealthReportTests
{
[Fact]
public void EmptyFleet_IsHealthy()
{
DriverHealthReport.Aggregate([]).ShouldBe(ReadinessVerdict.Healthy);
}
[Fact]
public void AllHealthy_Fleet_IsHealthy()
{
var verdict = DriverHealthReport.Aggregate([
new DriverHealthSnapshot("a", DriverState.Healthy),
new DriverHealthSnapshot("b", DriverState.Healthy),
]);
verdict.ShouldBe(ReadinessVerdict.Healthy);
}
[Fact]
public void AnyFaulted_TrumpsEverything()
{
var verdict = DriverHealthReport.Aggregate([
new DriverHealthSnapshot("a", DriverState.Healthy),
new DriverHealthSnapshot("b", DriverState.Degraded),
new DriverHealthSnapshot("c", DriverState.Faulted),
new DriverHealthSnapshot("d", DriverState.Initializing),
]);
verdict.ShouldBe(ReadinessVerdict.Faulted);
}
[Theory]
[InlineData(DriverState.Unknown)]
[InlineData(DriverState.Initializing)]
public void Any_NotReady_WithoutFaulted_IsNotReady(DriverState initializingState)
{
var verdict = DriverHealthReport.Aggregate([
new DriverHealthSnapshot("a", DriverState.Healthy),
new DriverHealthSnapshot("b", initializingState),
]);
verdict.ShouldBe(ReadinessVerdict.NotReady);
}
[Fact]
public void Any_Degraded_WithoutFaultedOrNotReady_IsDegraded()
{
var verdict = DriverHealthReport.Aggregate([
new DriverHealthSnapshot("a", DriverState.Healthy),
new DriverHealthSnapshot("b", DriverState.Degraded),
]);
verdict.ShouldBe(ReadinessVerdict.Degraded);
}
[Theory]
[InlineData(ReadinessVerdict.Healthy, 200)]
[InlineData(ReadinessVerdict.Degraded, 200)]
[InlineData(ReadinessVerdict.NotReady, 503)]
[InlineData(ReadinessVerdict.Faulted, 503)]
public void HttpStatus_MatchesStateMatrix(ReadinessVerdict verdict, int expected)
{
DriverHealthReport.HttpStatus(verdict).ShouldBe(expected);
}
}

View File

@@ -0,0 +1,78 @@
using Serilog;
using Serilog.Core;
using Serilog.Events;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Core.Observability;
namespace ZB.MOM.WW.OtOpcUa.Core.Tests.Observability;
[Trait("Category", "Unit")]
public sealed class LogContextEnricherTests
{
[Fact]
public void Scope_Attaches_AllFour_Properties()
{
var captured = new InMemorySink();
var logger = new LoggerConfiguration()
.Enrich.FromLogContext()
.WriteTo.Sink(captured)
.CreateLogger();
using (LogContextEnricher.Push("drv-1", "Modbus", DriverCapability.Read, "abc123"))
{
logger.Information("test message");
}
var evt = captured.Events.ShouldHaveSingleItem();
evt.Properties["DriverInstanceId"].ToString().ShouldBe("\"drv-1\"");
evt.Properties["DriverType"].ToString().ShouldBe("\"Modbus\"");
evt.Properties["CapabilityName"].ToString().ShouldBe("\"Read\"");
evt.Properties["CorrelationId"].ToString().ShouldBe("\"abc123\"");
}
[Fact]
public void Scope_Dispose_Pops_Properties()
{
var captured = new InMemorySink();
var logger = new LoggerConfiguration()
.Enrich.FromLogContext()
.WriteTo.Sink(captured)
.CreateLogger();
using (LogContextEnricher.Push("drv-1", "Modbus", DriverCapability.Read, "abc123"))
{
logger.Information("inside");
}
logger.Information("outside");
captured.Events.Count.ShouldBe(2);
captured.Events[0].Properties.ContainsKey("DriverInstanceId").ShouldBeTrue();
captured.Events[1].Properties.ContainsKey("DriverInstanceId").ShouldBeFalse();
}
[Fact]
public void NewCorrelationId_Returns_12_Hex_Chars()
{
var id = LogContextEnricher.NewCorrelationId();
id.Length.ShouldBe(12);
id.ShouldMatch("^[0-9a-f]{12}$");
}
[Theory]
[InlineData(null)]
[InlineData("")]
[InlineData(" ")]
public void Push_Throws_OnMissingDriverInstanceId(string? id)
{
Should.Throw<ArgumentException>(() =>
LogContextEnricher.Push(id!, "Modbus", DriverCapability.Read, "c"));
}
private sealed class InMemorySink : ILogEventSink
{
public List<LogEvent> Events { get; } = [];
public void Emit(LogEvent logEvent) => Events.Add(logEvent);
}
}

View File

@@ -0,0 +1,110 @@
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Core.Resilience;
namespace ZB.MOM.WW.OtOpcUa.Core.Tests.Resilience;
[Trait("Category", "Unit")]
public sealed class DriverResilienceStatusTrackerTests
{
private static readonly DateTime Now = new(2026, 4, 19, 12, 0, 0, DateTimeKind.Utc);
[Fact]
public void TryGet_Returns_Null_Before_AnyWrite()
{
var tracker = new DriverResilienceStatusTracker();
tracker.TryGet("drv", "host").ShouldBeNull();
}
[Fact]
public void RecordFailure_Accumulates_ConsecutiveFailures()
{
var tracker = new DriverResilienceStatusTracker();
tracker.RecordFailure("drv", "host", Now);
tracker.RecordFailure("drv", "host", Now.AddSeconds(1));
tracker.RecordFailure("drv", "host", Now.AddSeconds(2));
tracker.TryGet("drv", "host")!.ConsecutiveFailures.ShouldBe(3);
}
[Fact]
public void RecordSuccess_Resets_ConsecutiveFailures()
{
var tracker = new DriverResilienceStatusTracker();
tracker.RecordFailure("drv", "host", Now);
tracker.RecordFailure("drv", "host", Now.AddSeconds(1));
tracker.RecordSuccess("drv", "host", Now.AddSeconds(2));
tracker.TryGet("drv", "host")!.ConsecutiveFailures.ShouldBe(0);
}
[Fact]
public void RecordBreakerOpen_Populates_LastBreakerOpenUtc()
{
var tracker = new DriverResilienceStatusTracker();
tracker.RecordBreakerOpen("drv", "host", Now);
tracker.TryGet("drv", "host")!.LastBreakerOpenUtc.ShouldBe(Now);
}
[Fact]
public void RecordRecycle_Populates_LastRecycleUtc()
{
var tracker = new DriverResilienceStatusTracker();
tracker.RecordRecycle("drv", "host", Now);
tracker.TryGet("drv", "host")!.LastRecycleUtc.ShouldBe(Now);
}
[Fact]
public void RecordFootprint_CapturesBaselineAndCurrent()
{
var tracker = new DriverResilienceStatusTracker();
tracker.RecordFootprint("drv", "host", baselineBytes: 100_000_000, currentBytes: 150_000_000, Now);
var snap = tracker.TryGet("drv", "host")!;
snap.BaselineFootprintBytes.ShouldBe(100_000_000);
snap.CurrentFootprintBytes.ShouldBe(150_000_000);
}
[Fact]
public void DifferentHosts_AreIndependent()
{
var tracker = new DriverResilienceStatusTracker();
tracker.RecordFailure("drv", "host-a", Now);
tracker.RecordFailure("drv", "host-b", Now);
tracker.RecordSuccess("drv", "host-a", Now.AddSeconds(1));
tracker.TryGet("drv", "host-a")!.ConsecutiveFailures.ShouldBe(0);
tracker.TryGet("drv", "host-b")!.ConsecutiveFailures.ShouldBe(1);
}
[Fact]
public void Snapshot_ReturnsAll_TrackedPairs()
{
var tracker = new DriverResilienceStatusTracker();
tracker.RecordFailure("drv-1", "host-a", Now);
tracker.RecordFailure("drv-1", "host-b", Now);
tracker.RecordFailure("drv-2", "host-a", Now);
var snapshot = tracker.Snapshot();
snapshot.Count.ShouldBe(3);
}
[Fact]
public void ConcurrentWrites_DoNotLose_Failures()
{
var tracker = new DriverResilienceStatusTracker();
Parallel.For(0, 500, _ => tracker.RecordFailure("drv", "host", Now));
tracker.TryGet("drv", "host")!.ConsecutiveFailures.ShouldBe(500);
}
}

View File

@@ -0,0 +1,177 @@
using System.Net.Http;
using System.Text.Json;
using Microsoft.Extensions.Logging.Abstractions;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Core.Hosting;
using ZB.MOM.WW.OtOpcUa.Server.Observability;
namespace ZB.MOM.WW.OtOpcUa.Server.Tests;
[Trait("Category", "Integration")]
public sealed class HealthEndpointsHostTests : IAsyncLifetime
{
private static int _portCounter = 48500 + Random.Shared.Next(0, 99);
private readonly int _port = Interlocked.Increment(ref _portCounter);
private string Prefix => $"http://localhost:{_port}/";
private readonly DriverHost _driverHost = new();
private HealthEndpointsHost _host = null!;
private HttpClient _client = null!;
public ValueTask InitializeAsync()
{
_client = new HttpClient { BaseAddress = new Uri(Prefix) };
return ValueTask.CompletedTask;
}
public async ValueTask DisposeAsync()
{
_client.Dispose();
if (_host is not null) await _host.DisposeAsync();
}
private HealthEndpointsHost Start(Func<bool>? configDbHealthy = null, Func<bool>? usingStaleConfig = null)
{
_host = new HealthEndpointsHost(
_driverHost,
NullLogger<HealthEndpointsHost>.Instance,
configDbHealthy,
usingStaleConfig,
prefix: Prefix);
_host.Start();
return _host;
}
[Fact]
public async Task Healthz_ReturnsHealthy_EmptyFleet()
{
Start();
var response = await _client.GetAsync("/healthz");
response.IsSuccessStatusCode.ShouldBeTrue();
var body = JsonDocument.Parse(await response.Content.ReadAsStringAsync()).RootElement;
body.GetProperty("status").GetString().ShouldBe("healthy");
body.GetProperty("configDbReachable").GetBoolean().ShouldBeTrue();
body.GetProperty("usingStaleConfig").GetBoolean().ShouldBeFalse();
}
[Fact]
public async Task Healthz_StaleConfig_Returns200_WithFlag()
{
Start(configDbHealthy: () => false, usingStaleConfig: () => true);
var response = await _client.GetAsync("/healthz");
response.StatusCode.ShouldBe(System.Net.HttpStatusCode.OK);
var body = JsonDocument.Parse(await response.Content.ReadAsStringAsync()).RootElement;
body.GetProperty("configDbReachable").GetBoolean().ShouldBeFalse();
body.GetProperty("usingStaleConfig").GetBoolean().ShouldBeTrue();
}
[Fact]
public async Task Healthz_UnreachableConfig_And_NoCache_Returns503()
{
Start(configDbHealthy: () => false, usingStaleConfig: () => false);
var response = await _client.GetAsync("/healthz");
response.StatusCode.ShouldBe(System.Net.HttpStatusCode.ServiceUnavailable);
}
[Fact]
public async Task Readyz_EmptyFleet_Is200_Healthy()
{
Start();
var response = await _client.GetAsync("/readyz");
response.StatusCode.ShouldBe(System.Net.HttpStatusCode.OK);
var body = JsonDocument.Parse(await response.Content.ReadAsStringAsync()).RootElement;
body.GetProperty("verdict").GetString().ShouldBe("Healthy");
}
[Fact]
public async Task Readyz_WithHealthyDriver_Is200()
{
await _driverHost.RegisterAsync(new StubDriver("drv-1", DriverState.Healthy), "{}", CancellationToken.None);
Start();
var response = await _client.GetAsync("/readyz");
response.StatusCode.ShouldBe(System.Net.HttpStatusCode.OK);
var body = JsonDocument.Parse(await response.Content.ReadAsStringAsync()).RootElement;
body.GetProperty("verdict").GetString().ShouldBe("Healthy");
body.GetProperty("drivers").GetArrayLength().ShouldBe(1);
}
[Fact]
public async Task Readyz_WithFaultedDriver_Is503()
{
await _driverHost.RegisterAsync(new StubDriver("dead", DriverState.Faulted), "{}", CancellationToken.None);
await _driverHost.RegisterAsync(new StubDriver("alive", DriverState.Healthy), "{}", CancellationToken.None);
Start();
var response = await _client.GetAsync("/readyz");
response.StatusCode.ShouldBe(System.Net.HttpStatusCode.ServiceUnavailable);
var body = JsonDocument.Parse(await response.Content.ReadAsStringAsync()).RootElement;
body.GetProperty("verdict").GetString().ShouldBe("Faulted");
}
[Fact]
public async Task Readyz_WithDegradedDriver_Is200_WithDegradedList()
{
await _driverHost.RegisterAsync(new StubDriver("drv-ok", DriverState.Healthy), "{}", CancellationToken.None);
await _driverHost.RegisterAsync(new StubDriver("drv-deg", DriverState.Degraded), "{}", CancellationToken.None);
Start();
var response = await _client.GetAsync("/readyz");
response.StatusCode.ShouldBe(System.Net.HttpStatusCode.OK);
var body = JsonDocument.Parse(await response.Content.ReadAsStringAsync()).RootElement;
body.GetProperty("verdict").GetString().ShouldBe("Degraded");
body.GetProperty("degradedDrivers").GetArrayLength().ShouldBe(1);
body.GetProperty("degradedDrivers")[0].GetString().ShouldBe("drv-deg");
}
[Fact]
public async Task Readyz_WithInitializingDriver_Is503()
{
await _driverHost.RegisterAsync(new StubDriver("init", DriverState.Initializing), "{}", CancellationToken.None);
Start();
var response = await _client.GetAsync("/readyz");
response.StatusCode.ShouldBe(System.Net.HttpStatusCode.ServiceUnavailable);
}
[Fact]
public async Task Unknown_Path_Returns404()
{
Start();
var response = await _client.GetAsync("/foo");
response.StatusCode.ShouldBe(System.Net.HttpStatusCode.NotFound);
}
private sealed class StubDriver : IDriver
{
private readonly DriverState _state;
public StubDriver(string id, DriverState state)
{
DriverInstanceId = id;
_state = state;
}
public string DriverInstanceId { get; }
public string DriverType => "Stub";
public Task InitializeAsync(string _, CancellationToken ct) => Task.CompletedTask;
public Task ReinitializeAsync(string _, CancellationToken ct) => Task.CompletedTask;
public Task ShutdownAsync(CancellationToken ct) => Task.CompletedTask;
public DriverHealth GetHealth() => new(_state, null, null);
public long GetMemoryFootprint() => 0;
public Task FlushOptionalCachesAsync(CancellationToken ct) => Task.CompletedTask;
}
}

View File

@@ -46,7 +46,7 @@ public sealed class HistoryReadIntegrationTests : IAsyncLifetime
ApplicationName = "OtOpcUaHistoryTest",
ApplicationUri = "urn:OtOpcUa:Server:HistoryTest",
PkiStoreRoot = _pkiRoot,
AutoAcceptUntrustedClientCertificates = true,
AutoAcceptUntrustedClientCertificates = true, HealthEndpointsEnabled = false,
};
_server = new OpcUaApplicationHost(options, _driverHost, new DenyAllUserAuthenticator(),

View File

@@ -49,7 +49,7 @@ public sealed class MultipleDriverInstancesIntegrationTests : IAsyncLifetime
ApplicationName = "OtOpcUaMultiDriverTest",
ApplicationUri = "urn:OtOpcUa:Server:MultiDriverTest",
PkiStoreRoot = _pkiRoot,
AutoAcceptUntrustedClientCertificates = true,
AutoAcceptUntrustedClientCertificates = true, HealthEndpointsEnabled = false,
};
_server = new OpcUaApplicationHost(options, _driverHost, new DenyAllUserAuthenticator(),

View File

@@ -36,7 +36,7 @@ public sealed class OpcUaServerIntegrationTests : IAsyncLifetime
ApplicationName = "OtOpcUaTest",
ApplicationUri = "urn:OtOpcUa:Server:Test",
PkiStoreRoot = _pkiRoot,
AutoAcceptUntrustedClientCertificates = true,
AutoAcceptUntrustedClientCertificates = true, HealthEndpointsEnabled = false,
};
_server = new OpcUaApplicationHost(options, _driverHost, new DenyAllUserAuthenticator(),