Files
Joseph Doherty
13d5a7968b
Admin RedundancyTab — per-cluster read-only topology view. Closes the UI slice of task #149 (Phase 6.3 Stream E — Admin UI RedundancyTab + OpenTelemetry metrics + SignalR); the OpenTelemetry metrics + RoleChanged SignalR push are split into new follow-up task #198 because each is a structural add that deserves its own test matrix + NuGet-dep decision rather than riding this UI PR. New /clusters/{ClusterId} Redundancy tab slotted between ACLs and Audit in the existing ClusterDetail tab bar. Shows each ClusterNode row in the cluster with columns Node / Role (Primary green, Secondary blue, Standalone primary-blue badge) / Host / OPC UA port / ServiceLevel base / ApplicationUri (text-break so the long urn: doesn't blow out the table) / Enabled badge / Last seen (relative age via the same FormatAge helper as Hosts.razor, with a yellow "Stale" chip once LastSeenAt crosses the 30s threshold shared with HostStatusService.StaleThreshold — a missed heartbeat plus clock-skew buffer). Four summary cards above the table — total Nodes, Primary count, Secondary count, Stale count. Two guard-rail alerts: (a) red "No Primary or Standalone" when the cluster has no authoritative write target (all rows are Secondaries — read-only until one is promoted by the server-side RedundancyCoordinator apply-lease flow); (b) red "Split-brain" when >1 Primary exists — apply-lease enforcement at the coordinator level should have made this impossible, so the alert implies a hand-edited DB row + an investigation. New ClusterNodeService with ListByClusterAsync (ordered by ServiceLevelBase descending so Primary rows with higher base float to the top) + a static IsStale predicate matching HostStatusService's 30s convention. DI-registered alongside the existing scoped services in Program.cs. Writes (role swap, enable/disable) are deliberately absent from the service — they go through the RedundancyCoordinator apply-lease flow on the server side + direct DB mutation from Admin would race with it. New ClusterNodeServiceTests covering IsStale across null/recent/old LastSeenAt + ListByClusterAsync ordering + cluster filter. 4/4 new tests passing; full Admin.Tests suite 76/76 (was 72 before this PR, +4). Admin project builds 0 errors. Task #198 captures the deferred work: (1) OpenTelemetry Meter for primary/secondary/stale counts + role_transition counter with from/to/node tags + OTLP exporter config; (2) RoleChanged SignalR push — extend FleetStatusPoller to detect RedundancyRole changes on ClusterNode rows + emit a RoleChanged hub message so the RedundancyTab refreshes instantly instead of on-page-load polling.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Admin RedundancyTab — per-cluster read-only topology view. Closes the UI slice of task #149 (Phase 6.3 Stream E — Admin UI RedundancyTab + OpenTelemetry metrics + SignalR); the OpenTelemetry metrics + RoleChanged SignalR push are split into new follow-up task #198 because each is a structural add that deserves its own test matrix + NuGet-dep decision rather than riding this UI PR. New /clusters/{ClusterId} Redundancy tab slotted between ACLs and Audit in the existing ClusterDetail tab bar. Shows each ClusterNode row in the cluster with columns Node / Role (Primary green, Secondary blue, Standalone primary-blue badge) / Host / OPC UA port / ServiceLevel base / ApplicationUri (text-break so the long urn: doesn't blow out the table) / Enabled badge / Last seen (relative age via the same FormatAge helper as Hosts.razor, with a yellow "Stale" chip once LastSeenAt crosses the 30s threshold shared with HostStatusService.StaleThreshold — a missed heartbeat plus clock-skew buffer). Four summary cards above the table — total Nodes, Primary count, Secondary count, Stale count. Two guard-rail alerts: (a) red "No Primary or Standalone" when the cluster has no authoritative write target (all rows are Secondaries — read-only until one is promoted by the server-side RedundancyCoordinator apply-lease flow); (b) red "Split-brain" when >1 Primary exists — apply-lease enforcement at the coordinator level should have made this impossible, so the alert implies a hand-edited DB row + an investigation. New ClusterNodeService with ListByClusterAsync (ordered by ServiceLevelBase descending so Primary rows with higher base float to the top) + a static IsStale predicate matching HostStatusService's 30s convention. DI-registered alongside the existing scoped services in Program.cs. Writes (role swap, enable/disable) are deliberately absent from the service — they go through the RedundancyCoordinator apply-lease flow on the server side + direct DB mutation from Admin would race with it. New ClusterNodeServiceTests covering IsStale across null/recent/old LastSeenAt + ListByClusterAsync ordering + cluster filter. 4/4 new tests passing; full Admin.Tests suite 76/76 (was 72 before this PR, +4). Admin project builds 0 errors. Task #198 captures the deferred work: (1) OpenTelemetry Meter for primary/secondary/stale counts + role_transition counter with from/to/node tags + OTLP exporter config; (2) RoleChanged SignalR push — extend FleetStatusPoller to detect RedundancyRole changes on ClusterNode rows + emit a RoleChanged hub message so the RedundancyTab refreshes instantly instead of on-page-load polling.
Admin /hosts red-badge + resilience columns + Polly telemetry observer. Closes task #164 (the remaining slice of Phase 6.1 Stream E.3 after the earlier publisher + hub PR). Three cooperating pieces wired together so the operator-facing /hosts table actually reflects the live Polly counters that the pipeline builder is producing. DriverResiliencePipelineBuilder gains an optional DriverResilienceStatusTracker ctor param — when non-null, every built pipeline wires Polly's OnRetry/OnOpened/OnClosed strategy-options callbacks into the tracker. OnRetry → tracker.RecordFailure (so ConsecutiveFailures climbs per retry), OnOpened → tracker.RecordBreakerOpen (stamps LastCircuitBreakerOpenUtc), OnClosed → tracker.RecordSuccess (resets the failure counter once the target recovers). Absent tracker = silent, preserving the unit-test constructor path + any deployment that doesn't care about resilience observability. Cancellation stays excluded from the failure count via the existing ShouldHandle predicate. HostStatusService.HostStatusRow extends with four new fields — ConsecutiveFailures, LastCircuitBreakerOpenUtc, CurrentBulkheadDepth, LastRecycleUtc — populated via a second LEFT JOIN onto DriverInstanceResilienceStatuses keyed on (DriverInstanceId, HostName). LEFT JOIN because brand-new hosts haven't been sampled yet; a missing row means zero failures + never-opened breaker, which is the correct default. New FailureFlagThreshold constant (=3, matches plan decision #143's conservative half-of-breaker convention) + IsFlagged predicate so the UI can pre-warn before the breaker actually trips. Hosts.razor paints three new columns between State and Last-transition — Fail# (bold red when flagged), In-flight (bulkhead-depth proxy), Breaker-opened (relative age). Per-row "Flagged" red badge alongside State when IsFlagged is true. Above the first cluster table, a red alert banner summarises the flagged-host count when ≥1 host is flagged, so operators see the problem before scanning rows. Three new tests in DriverResiliencePipelineBuilderTests — Tracker_RecordsFailure_OnEveryRetry verifies ConsecutiveFailures reaches RetryCount after a transient-forever operation, Tracker_StampsBreakerOpen_WhenBreakerTrips verifies LastBreakerOpenUtc is set after threshold failures on a Write pipeline, Tracker_IsolatesCounters_PerHost verifies one dead host does not leak failure counts into a healthy sibling. Full suite — Core.Tests 14/14 resilience-builder tests passing (11 existing + 3 new), Admin.Tests 72/72 passing, Admin project builds 0 errors. SignalR live push of status changes + browser visual review are deliberately left to a follow-up — this PR keeps the structural change minimal (polling refresh already exists in the page's 10s timer; SignalR would be a structural add that touches hub registration + client subscription).
AB CIP UDT Template Object shape reader. Closes the shape-reader half of task #179. CipTemplateObjectDecoder (pure-managed) parses the Read Template blob per Rockwell CIP Vol 1 + libplctag ab/cip.c handle_read_template_reply — 12-byte header (u16 member_count + u16 struct_handle + u32 instance_size + u32 member_def_size) followed by memberCount × 8-byte member blocks (u16 info with bit-15 struct flag + lower-12-bit type code matching the Symbol Object encoding, u16 array_size, u32 struct_offset) followed by semicolon-terminated strings (UDT name first, then one per member). ParseSemicolonTerminatedStrings handles the observed firmware variations — name;\0 vs name; delimiters, optional null/space padding after the semicolon, trailing-name-without-semicolon corner case. Struct-flag members decode as AbCipDataType.Structure; unknown atomic codes fall back to Structure so the shape remains valid even with unrecognised members. Zero member count + short buffer both return null; missing member names yield <member_N> placeholders. IAbCipTemplateReader + IAbCipTemplateReaderFactory abstraction — one call per template instance id returning the raw blob. LibplctagTemplateReader is the production implementation creating a libplctag Tag with name @udt/{templateId} + handing the buffer to the decoder. AbCipDriver ctor gains optional templateReaderFactory parameter (defaults to LibplctagTemplateReaderFactory) + new internal FetchUdtShapeAsync that — checks AbCipTemplateCache first, misses call the reader + decode + cache, template-read exceptions + decode failures return null so callers can fall back to declaration-driven fan-out without the whole discovery blowing up. OperationCanceledException rethrows for shutdown propagation. Unknown device host returns null without attempting a fetch. FlushOptionalCachesAsync empties the cache so a subsequent fetch re-reads. 16 new decoder tests — simple two-member UDT, struct-member flag → Structure, array member ArrayLength, 6-member mixed-type with correct offsets, unknown type code → Structure, zero member count → null, short buffer → null, missing member name → placeholder, ParseSemicolonTerminatedStrings theory across 5 shapes. 6 new AbCipFetchUdtShapeTests exercising the driver integration via reflection (method is internal) — happy-path decode + cache, different template ids get separate fetches, unknown device → null without reader creation, decode failure returns null + doesn't cache (next call retries), reader exception returns null, FlushOptionalCachesAsync clears the cache. Total AbCip unit tests now 211/211 passing (+19 from the @tags merge's 192); full solution builds 0 errors; other drivers untouched. Whole-UDT read optimization (single libplctag call returning the packed buffer + client-side member decode using the template offsets) is left as a follow-up — requires rethinking the per-tag read path + careful hardware validation; current per-member fan-out still works correctly, just with N round-trips instead of 1.
RMW pass 2 — AbCip BOOL-within-DINT + AbLegacy bit-within-word. Closes task #181. AbCip — AbCipDriver.WriteAsync now detects BOOL writes with a bit index + routes them through WriteBitInDIntAsync: strip the .N suffix to form the parent DINT tag path (via AbCipTagPath with BitIndex=null + ToLibplctagName), get/create a cached parent IAbCipTagRuntime via EnsureParentRuntimeAsync (distinct from the bit-selector tag runtime so read + write target the DINT directly), acquire a per-parent-name SemaphoreSlim, Read → Convert.ToInt32 the current DINT → (current | 1<<bit) or (current & ~(1<<bit)) → Write via EncodeValue(DInt, updated). Per-parent lock prevents concurrent writers to the same DINT from losing updates — parallels Modbus + FOCAS pass 1. DeviceState gains ParentRuntimes dict + GetRmwLock helper + _rmwLocks ConcurrentDictionary. DisposeHandles now walks ParentRuntimes too. LibplctagTagRuntime.EncodeValue's BOOL-with-bitIndex branch stays as a defensive throw (message updated to point at the new driver-level dispatch) so an accidental bypass fails loudly rather than silently clobbering the whole DINT. AbLegacy — identical pattern for PCCC N-file bit writes. AbLegacyDriver.WriteAsync detects Bit with bitIndex + PMC letter not in {B, I, O} (B-file + I/O use their own bit-addressable semantics so don't RMW at N-file word level), routes through WriteBitInWordAsync which uses Int16 for the parent word, creates + caches a parent runtime with the suffix-stripped N7:0 address, acquires per-parent lock, RMW. DeviceState extended the same way as AbCip (ParentRuntimes + GetRmwLock). LibplctagLegacyTagRuntime.EncodeValue Bit-with-bitIndex branch points at the driver dispatch. Tests — 5 new AbCipBoolInDIntRmwTests (bit set ORs + preserves, bit clear ANDs + preserves, 8-way concurrent writes to same parent compose to 0xFF, different-parent writes get separate runtimes, repeat bit writes reuse the parent runtime init-count 1 + write-count 2), 4 new AbLegacyBitRmwTests (bit set preserves, bit clear preserves 0xFFF7, 8-way concurrent 0xFF, repeat writes reuse parent). Two pre-existing tests flipped — AbCipDriverWriteTests.Bit_in_dint_write_returns_BadNotSupported + AbLegacyReadWriteTests.Bit_within_word_write_rejected_as_BadNotSupported both now assert Good instead of BadNotSupported, renamed to _now_succeeds_via_RMW. Total tests — AbCip 166/166, AbLegacy 96/96, full solution builds 0 errors; Modbus + FOCAS + TwinCAT + other drivers untouched. Task #181 done across all four libplctag-backed + non-libplctag drivers (Modbus BitInRegister + AbCip BOOL-in-DINT + AbLegacy N-file bit + FOCAS PMC Bit — all with per-parent-word serialisation).
RMW pass 1 — Modbus BitInRegister + FOCAS PMC Bit write paths. First half of task #181 — the two drivers where read-modify-write is a clean protocol-level insertion (Modbus FC03/FC06 round-trip + FOCAS pmc_rdpmcrng / pmc_wrpmcrng round-trip). Per-driver SemaphoreSlim registry keyed on the parent word address serialises concurrent bit writes so two writers targeting different bits in the same word don't lose one another's update. Modbus — ModbusDriver gains WriteBitInRegisterAsync + _rmwLocks ConcurrentDictionary. WriteOneAsync routes BitInRegister (HoldingRegisters region only) through RMW ahead of the normal encode path. Read uses FC03 Read Holding Registers for 1 register at tag.Address, bit-op on the returned ushort via (current | 1<<bit) for set / (current & ~(1<<bit)) for clear, write back via FC06 Write Single Register. Per-address lock prevents concurrent bit writes to the same register from racing. Rejects out-of-range bits (0-15) with InvalidOperationException. EncodeRegister's BitInRegister branch repurposed as a defensive guard — if a non-RMW caller ever reaches it, throw so an unintended bypass stays loud rather than silently clobbering. FOCAS — FwlibFocasClient gains WritePmcBitAsync + _rmwLocks keyed on {addrType}:{byteAddr}. Driver-layer WriteAsync routes Bit writes with a bitIndex through the new path; other Pmc writes still hit the direct pmc_wrpmcrng path. RMW uses cnc_rdpmcrng + Byte dataType to grab the parent byte, bit-op with (current | 1<<bit) or (current & ~(1<<bit)), cnc_wrpmcrng to write back. Rejects out-of-range bits (0-7, FOCAS PMC bytes are 8-bit) with InvalidOperationException. EncodePmcValue's Bit branch now treats a no-bitIndex case as whole-byte boolean (non-zero / zero); bitIndex-present writes never hit this path because they dispatch to WritePmcBitAsync upstream. Tests — 5 new ModbusBitRmwTests + 4 new FocasPmcBitRmwTests + 1 renamed pre-existing test each covering — bit set preserves other bits, bit clear preserves other bits, concurrent bit writes to same word/byte compose correctly (8-parallel stress), bit writes on different parent words proceed without contention (4-parallel), sequential bit sets compose into 0xFF after all 8. Fake PmcRmwFake in FOCAS tests simulates the PMC byte storage + surfaces it through the IFocasClient contract so the test asserts driver-level behavior without needing Fwlib32.dll. FwlibNativeHelperTests.EncodePmcValue_Bit_throws_NotSupported_for_RMW_gap replaced with EncodePmcValue_Bit_without_bit_index_writes_byte_boolean reflecting the new behavior. ModbusDataTypeTests.BitInRegister_write_is_not_supported_in_PR24 renamed to BitInRegister_EncodeRegister_still_rejects_direct_calls; the message assertion updated to match the new defensive message. Modbus tests now 182/182, FOCAS tests now 119/119; full solution builds 0 errors; AbCip/AbLegacy/TwinCAT untouched (those get their RMW pass in a follow-up since libplctag bit access may need a parallel parent-word handle). Task #181 stays pending until that second pass lands.
RMW pass 1 — Modbus BitInRegister + FOCAS PMC Bit write paths. First half of task #181 — the two drivers where read-modify-write is a clean protocol-level insertion (Modbus FC03/FC06 round-trip + FOCAS pmc_rdpmcrng / pmc_wrpmcrng round-trip). Per-driver SemaphoreSlim registry keyed on the parent word address serialises concurrent bit writes so two writers targeting different bits in the same word don't lose one another's update. Modbus — ModbusDriver gains WriteBitInRegisterAsync + _rmwLocks ConcurrentDictionary. WriteOneAsync routes BitInRegister (HoldingRegisters region only) through RMW ahead of the normal encode path. Read uses FC03 Read Holding Registers for 1 register at tag.Address, bit-op on the returned ushort via (current | 1<<bit) for set / (current & ~(1<<bit)) for clear, write back via FC06 Write Single Register. Per-address lock prevents concurrent bit writes to the same register from racing. Rejects out-of-range bits (0-15) with InvalidOperationException. EncodeRegister's BitInRegister branch repurposed as a defensive guard — if a non-RMW caller ever reaches it, throw so an unintended bypass stays loud rather than silently clobbering. FOCAS — FwlibFocasClient gains WritePmcBitAsync + _rmwLocks keyed on {addrType}:{byteAddr}. Driver-layer WriteAsync routes Bit writes with a bitIndex through the new path; other Pmc writes still hit the direct pmc_wrpmcrng path. RMW uses cnc_rdpmcrng + Byte dataType to grab the parent byte, bit-op with (current | 1<<bit) or (current & ~(1<<bit)), cnc_wrpmcrng to write back. Rejects out-of-range bits (0-7, FOCAS PMC bytes are 8-bit) with InvalidOperationException. EncodePmcValue's Bit branch now treats a no-bitIndex case as whole-byte boolean (non-zero / zero); bitIndex-present writes never hit this path because they dispatch to WritePmcBitAsync upstream. Tests — 5 new ModbusBitRmwTests + 4 new FocasPmcBitRmwTests + 1 renamed pre-existing test each covering — bit set preserves other bits, bit clear preserves other bits, concurrent bit writes to same word/byte compose correctly (8-parallel stress), bit writes on different parent words proceed without contention (4-parallel), sequential bit sets compose into 0xFF after all 8. Fake PmcRmwFake in FOCAS tests simulates the PMC byte storage + surfaces it through the IFocasClient contract so the test asserts driver-level behavior without needing Fwlib32.dll. FwlibNativeHelperTests.EncodePmcValue_Bit_throws_NotSupported_for_RMW_gap replaced with EncodePmcValue_Bit_without_bit_index_writes_byte_boolean reflecting the new behavior. ModbusDataTypeTests.BitInRegister_write_is_not_supported_in_PR24 renamed to BitInRegister_EncodeRegister_still_rejects_direct_calls; the message assertion updated to match the new defensive message. Modbus tests now 182/182, FOCAS tests now 119/119; full solution builds 0 errors; AbCip/AbLegacy/TwinCAT untouched (those get their RMW pass in a follow-up since libplctag bit access may need a parallel parent-word handle). Task #181 stays pending until that second pass lands.
TwinCAT follow-up — Symbol browser via AdsClient + SymbolLoaderFactory. Closes task #188. Adds ITwinCATClient.BrowseSymbolsAsync — IAsyncEnumerable yielding TwinCATDiscoveredSymbol (InstancePath + mapped TwinCATDataType + ReadOnly flag) from the target's flat symbol table. AdsTwinCATClient implementation uses SymbolLoaderFactory.Create(_client, new SymbolLoaderSettings(SymbolsLoadMode.Flat)) + iterates loader.Symbols, maps IEC 61131-3 type names (BOOL/SINT/INT/DINT/LINT/REAL/LREAL/STRING/WSTRING/TIME/DATE/DT/TOD + BYTE/WORD/DWORD/LWORD unsigned-word aliases) through MapSymbolTypeName, checks SymbolAccessRights.Write bit for writable vs read-only. Unsupported types (UDTs / function blocks / arrays / pointers) surface with DataType=null so callers can skip or recurse. TwinCATDriverOptions.EnableControllerBrowse — new bool, default false to preserve the strict-config path. When true, DiscoverAsync iterates each device's BrowseSymbolsAsync, filters via TwinCATSystemSymbolFilter (rejects TwinCAT_*, Constants.*, Mc_*, __*, Global_Version* prefixes + anything empty), skips null-DataType symbols, emits surviving symbols under a per-device Discovered/ sub-folder with InstancePath as both FullName + BrowseName + ReadOnly→ViewOnly/writable→Operate. Pre-declared tags from TwinCATDriverOptions.Tags always emit regardless. Browse failure is non-fatal — exception caught + swallowed, pre-declared tags stay in the address space, operators see the failure in driver health on next read. TwinCATSystemSymbolFilter static class mirrors AbCipSystemTagFilter's shape with TwinCAT-specific prefixes. Fake client updated — BrowseResults list for test setup + FireNotification-style single-invocation on each subscribe, ThrowOnBrowse flag for failure testing. 8 new unit tests — strict path emits only pre-declared when EnableControllerBrowse=false, browse enabled adds Discovered/ folder, filter rejects system prefixes, null-DataType symbols skipped, ReadOnly symbols surface ViewOnly, browse failure leaves pre-declared intact, SystemSymbolFilter theory (10 cases). Total TwinCAT unit tests now 110/110 passing (+17 from the native-notification merge's 93); full solution builds 0 errors; other drivers untouched.