Closes the R2-08 S2 live leg. The offline LdapAuthResilienceTests prove the
directory-outage circuit with fakes + an unroutable-host blackhole; this drives
the SAME circuit through the production auth path (OtOpcUaLdapAuthService +
LdapOpcUaUserAuthenticator) against a REAL GLAuth that is paused mid-run and
unpaused — and doubles as the first end-to-end verification that the PR #451
GLAuth swap (retired bitnami/openldap) binds correctly.
Triple-gated (GLAUTH_LIVE_HOST + GLAUTH_OUTAGE_START_CMD/STOP_CMD), skips clean
offline (verified: 1 skipped, 8 ms). try/finally always unpauses GLAuth.
Verified GREEN against 10.100.0.35:3894 (12 s): healthy binds (alice+bob, mapped
roles) -> docker pause -> 3 bounded failures -> circuit opens -> sub-second
"unavailable" fast-deny -> docker unpause -> half-open probe closes -> binds
succeed again. A boundary AuthTimeout feeds the circuit (authenticator :123-130),
so the docker-pause timeout shape is a faithful outage.
Integration-sweep follow-up #6 (SQL leg). The sweep flagged 7 heavy 2-node
deploy/failover/reconnect E2E tests as 'time out under amd64-emulated SQL' and
suggested raising deadlines. That diagnosis was WRONG: run against real
(native-amd64) SQL on the Docker host, they still hang — even at 4x deadline.
Root cause is that these tests had never actually run against real SQL, only the
EF in-memory provider, which does NOT enforce foreign keys.
Two distinct defects, both hidden by in-memory:
1. Missing FK-parent seed (5 tests). The deploy records a NodeDeploymentState row
per node, FK'd to ClusterNode (FK_NodeDeploymentState_ClusterNode_NodeId). The
tests never seeded ServerCluster + ClusterNode, so on SQL each node-state INSERT
throws and the deploy never seals (waits the full deadline regardless of size).
Added TwoNodeClusterHarness.SeedDefaultClusterAsync (seeds a ServerCluster +
a ClusterNode per node; Warm/NodeCount=2 to satisfy the SQL CHECK constraint
CK_ServerCluster_RedundancyMode_NodeCount AND the ClusterEnabledNodeCountMismatch
validator). Called it in DeployHappyPath x2 / Failover / FleetDiagnostics /
EquipmentNamespace; fixed DriverReconnect to seed node B + NodeCount=2.
2. Stale EquipmentId (EquipmentNamespace test; pre-existing on BOTH providers).
Seeded EquipmentId='eq-1', but the later-added EquipmentIdNotDerived validator
requires EquipmentId == DeriveEquipmentId(EquipmentUuid) ('EQ-'+first 12 hex).
Fixed the seed to a canonical id + matching UUID. Only surfaced once the FK fix
let the deploy reach equipment validation.
Also added OTOPCUA_HARNESS_SQL_HOST / OTOPCUA_HARNESS_LDAP_HOST overrides so the
harness fixtures can run on the native-amd64 Docker host (avoids arm64 mssql
emulation entirely).
Verified: the 5 SQL-backed classes go 11/11 green vs native SQL; in-memory default
unregressed (12/12). RoslynVirtualTagEvaluatorTests racing test is a pre-existing
flaky race (not SQL-backed; passes in isolation) — left as-is.
The Host.IntegrationTests opt-in real-LDAP mode (OTOPCUA_HARNESS_USE_LDAP=1)
used bitnami/openldap:2.6, gone since Bitnami deprecated their free image
catalog in 2025. Replaced it with GLAuth — the same LDAP server the rest of
the project already uses (docker-dev + the live OPC UA data-plane auth, both
against the shared GLAuth on :3893) — removing the lone OpenLDAP outlier and
the gone-image breakage.
- Added tests/.../Host.IntegrationTests/glauth/config.toml: baseDN dc=zb,dc=local,
a search-capable serviceaccount, dev users alice (full access) / bob (read-only),
and role groups with the same gidnumbers as scadaproj/infra/glauth so GroupToRole
maps identically.
- Compose ldap service -> glauth/glauth:latest, mounting the config, host :3894 -> :3893.
- Repointed the harness real-LDAP override from the OpenLDAP cn=admin/ldapadmin to
GLAuth's cn=serviceaccount/serviceaccount123.
Live-verified against a deployed container on :3894: the serviceaccount bind
searches and returns alice + her 5 memberOf groups; alice/bob bind with the
correct password; a wrong password yields Invalid credentials (49) — exactly the
OtOpcUaLdapAuthService flow (proven identical to the data-plane's shared-GLAuth path).
NB: real-LDAP mode is opt-in; the default Host run uses StubLdapAuthService, so
this never blocked the default suite. Integration-sweep follow-up #6 (LDAP leg);
the amd64-emulated-SQL deadline leg remains open.
Findings 01/S-1 (AddressSpaceApplyOutcome failure field + apply.failed logging,
no optimistic success), 06/S-1 (Galaxy write fails closed -> BadCommunicationError
+ #5 revert, no knowingly-lost raw Write), 03/S4 (PrimaryGatePolicy default-deny
unknown-role-multi-driver on all gates + scripted-alarm emit gate). T13/T15 (2-node
live gates) deferred -- T15 is the behavior-affecting S4 live gate, must run in heavy
pass. Clean merge, build clean.
AddOtOpcUaDriverFactories (the driver-node bootstrap) must bind the real
DriverCapabilityInvokerFactory — not the NullDriverCapabilityInvokerFactory
pass-through — and it must mint a real CapabilityInvoker per instance. This is the
deterministic DI half of the #10 live gate: a refactor dropping back to the
pass-through would make the whole retry/breaker/bulkhead pipeline inert in production
while every pass-through-defaulting unit test stays green. 3 tests (real factory bound,
real invoker minted, shared singletons), non-fixtured. Host.IntegrationTests.
The load-bearing integration half of Critical 1. Adds TwoNodeClusterHarness.HardKillNodeAAsync
(canonical in-process crash sim: shut down node A's remoting transport via
IRemoteActorRefProvider.Transport.Shutdown() — associations drop, heartbeats/gossip stop, NO
graceful Cluster.Leave, so node B's failure detector marks A Unreachable) + a
WaitForNodeBSoleDriverLeaderAsync helper (waits for a STABLE takeover: B sole Up member + driver
role-leader). HardKillFailoverTests crashes the oldest node and asserts the survivor takes over —
the exact scenario the graceful StopNodeBAsync path (Cluster.Leave, no downing decision) cannot
exercise.
Live-verified in-process (~35s: acceptable-heartbeat-pause 10s + stable-after 15s + convergence).
Negative control confirmed the test BITES: forcing explicit NoDowning
(akka.cluster.downing-provider-class = "") makes it time out with the node stuck Up-but-Unreachable.
FINDING surfaced by the negative control: removing Critical 1's typed
ClusterOptions.SplitBrainResolver did NOT break failover, because Akka.Cluster.Hosting's
WithClustering applies SplitBrainResolverOption.Default when the option is null — enabling the SBR
downing provider that reads the akka.conf keep-oldest block (present on master BEFORE Critical 1).
So the cluster was NOT running NoDowning before Critical 1; the typed option reinforces the akka.conf
strategy rather than being the sole activator, and Critical 1's 'HOCON inert / NoDowning' premise is
inaccurate. The test guards the failover OUTCOME regardless of activation path. Comments corrected to
reflect this; flagged for review.
U2 (Critical): RoslynVirtualTagEvaluator ran scripts with a CancellationToken
that a synchronous Roslyn script can never observe, so one CPU-bound/infinite-
loop virtual-tag script hung the owning VirtualTagActor forever (no timeout, no
recovery). Route evaluation through the existing TimedScriptEvaluator (Task.Run
+ WaitAsync) so the wall-clock budget is real; a runaway script now returns Bad
within the timeout instead of wedging the actor.
U3 (High): the live path cached evaluators in a raw ConcurrentDictionary never
cleared on republish, leaking a collectible AssemblyLoadContext per edited
script forever. Swap in the purpose-built CompiledScriptCache (Lazy single-
compile, dispose-on-Clear) and add an apply-boundary clear: a new narrow
IScriptCacheOwner seam the VirtualTagHostActor calls on each ApplyVirtualTags
generation (mirrors ScriptedAlarmEngine's per-generation clear).
Tests: infinite-loop-fails-within-timeout (bounded by WaitAsync so a regression
fails instead of hanging), happy-path-after-wrapping, ClearCompiledScripts-
empties-cache, and a host-actor wiring guard asserting each apply generation
calls ClearCompiledScripts (theme #1 production-caller assertion). Host.Integration
VT suite 18/18, Runtime VT suite 12/12.
The HistorianGateway driver is now the sole historian read/write+alarm backend, so the
Wonderware sidecar projects are dead code. Removes the 5 Wonderware projects (driver,
.Client, .Client.Contracts, + their 2 test projects) from the solution and tree, and fully
retires the vestigial 'Historian.Wonderware' driver type (UI/probe-only; it had no driver
factory): the Host probe registration, the AdminUI driver-config surface (driver page,
tag-config editor/model/validator entry, address picker/builder, driver-type catalog +
dropdown + edit-router entries), and their tests. Prunes the now-unused Wonderware
connection fields (Host/Port/UseTls/ServerCertThumbprint/SharedSecret) from
AlarmHistorianOptions (keeping Enabled + the SQLite store-and-forward knobs) and refreshes
the stale XML docs that named Wonderware as the production backend.
Claude-Session: https://claude.ai/code/session_012SDSQ3AcaXqPcBtDESBRii
AdminUI driver-instance pages serialized enum config fields (S7 CpuType,
Modbus DataType/Region, AbCip PlcFamily, ...) as JSON *numbers* because each
page's _jsonOpts lacked a JsonStringEnumConverter. The driver factories,
however, deserialize into string-typed DTOs (+ lenient ParseEnum) and throw
when binding a JSON number to a string? — so an AdminUI-authored config
containing any enum field produced a blob the driver could not parse,
faulting the driver on deploy. Proven end-to-end for S7 and Modbus; latent
for AbCip/AbLegacy/TwinCAT/FOCAS/Galaxy/Historian. Only OpcUaClient was safe
(its factory + probe already carried the converter).
Add JsonStringEnumConverter to all 9 driver-instance pages' _jsonOpts and the
8 missing driver probes' _opts (factories unchanged — already string-via-
ParseEnum; strictly more permissive, also lets pages load hand-seeded
string-enum configs back into the form).
Also fix DriverProbeHandshakeE2eTests.AbCip_Green_AgainstSim to probe a real
sim tag (TestDINT) — the no-tags @raw_cpu_type fallback is rejected by the
ab_server sim with ErrorBadParam (a real ControlLogix returns ErrorNotFound,
which the probe treats as reachable; hardware-gated follow-up).
Tests: reflection guard over all driver pages' _jsonOpts (AdminUI.Tests);
factory round-trip + numeric-form-throws guards for S7 and Modbus.
Found by running the never-before-run FB-9/FB-10 live verifies.
The OPC UA address-space build pipeline was named after a v2-roadmap
milestone number rather than its domain. Rename the family to describe
what it does (build/diff/apply the OPC UA address space):
Phase7Composer -> AddressSpaceComposer
Phase7CompositionResult -> AddressSpaceComposition
Phase7Planner -> AddressSpacePlanner
Phase7Plan -> AddressSpacePlan
Phase7Applier -> AddressSpaceApplier
Phase7ApplyOutcome -> AddressSpaceApplyOutcome
The 9 Phase7*Tests suites follow suit; Phase7ScriptingEntitiesTests ->
ScriptingEntitiesTests (it tests the scripting migration, not the
pipeline). Log-message prefixes move to the new class names.
Pure mechanical rename, no behavioral change. EF migration classes/IDs
(AddPhase7ScriptingTables, ExtendComputeGenerationDiffWithPhase7) are
immutable and left untouched, as are historical design docs.
Build clean; OpcUaServer 261/261, Runtime 272/272, ScriptingEntities
12/12 green.
Two bugs caught by live verification against the mxaccessgw at 10.100.0.48:5120:
- MaxAttempts=1 produced an invalid Polly RetryStrategyOptions -> the probe failed
on every real gateway. Removed the Retry override (matches GalaxyDriver); fail-fast
is already guaranteed by the TCP preflight + the per-call deadline.
- A rejected key surfaces as a typed MxGatewayAuthenticationException, not a raw
RpcException, so 'auth-rejection = reachable' was bypassed. Catch the typed auth/
authorization exceptions -> Ok=true.
Adds DriverProbeHandshakeE2eTests: direct-probe, skip-gated cross-protocol green/red
discrimination (Modbus, OpcUaClient, Galaxy + a local real OPC UA server).
All five suppressed advisories are now resolved at baseline/resolved versions,
so every NuGetAuditSuppress is removed repo-wide:
- System.Security.Cryptography.Xml (GHSA-37gx-xxp4-5rgx / GHSA-w3x6-4m5h-cxqf)
-> fixed by the .NET 10 baseline (10.0.6)
- OPCFoundation Opc.Ua.Core (GHSA-h958-fxgg-g7w3) -> fixed at resolved 1.5.378.106
Two were still live and are now patched via direct security pins:
- OpenTelemetry.Api 1.9.0 -> 1.15.3 (GHSA-g94r-2vxg-569j) pinned in Cluster;
Runtime/ControlPlane/AdminUI + tests inherit via project reference
- Tmds.DBus.Protocol 0.20.0 -> 0.21.3 (GHSA-xrw6-gwf8-vvr9) pinned in Client.UI
Also correct the Historian sidecar runtime comments (x86 -> x64, matching the
csproj PlatformTarget). Solution audit: 0 vulnerable packages; full build clean.
Composes the one Layer-0 hop existing tests left uncovered together:
ScriptLogSignalRBridge subscribing to the script-logs DPS topic and
fanning a ScriptLogEntry out to the IInProcessBroadcaster<ScriptLogEntry>
singleton resolved from the SAME DI container the /script-log page injects.
Mirrors DriverStatusHubE2eTests. Confirms the server-side topic→page chain
delivers end-to-end (only the live Blazor circuit remains manual).
A thin gateway over the admin-operations cluster singleton so CI/scripts can trigger a
deployment without the Blazor button. Forwards to the same IAdminOperationsClient.
StartDeploymentAsync; mounted on admin-role nodes. Auth is a fixed-time X-Api-Key check
against Security:DeployApiKey (orthogonal to the cookie-only web auth); AllowAnonymous so the
auth fallback doesn't 401 it, self-disabling (503) until the key is set. Outcome->status:
202/200/409/422. Unit tests for the key check + outcome mapping; HTTP E2E (real auth + real
deploy via the 2-node harness). Documented in docs/security.md.
Seed a 1-area/1-line/1-equipment/1-tag Equipment namespace, StartDeployment via the
in-process 2-node harness, and assert the persisted artifact decodes (ParseComposition)
to the equipment signal (FullName from TagConfig) + friendly UNS folder names. Covers the
ConfigComposer -> ArtifactBlob -> ParseComposition.EquipmentTags seam the unit tests only
approximated with hand-built JSON. (OPC UA browse is covered against a real SDK node manager
in Phase7ApplierHierarchyTests; the cluster harness binds the no-op sink.)
Standardize the control-plane admin role VALUES on the canonical six
(ZB.MOM.WW.Auth CanonicalRole). OtOpcUa uses four:
ConfigViewer -> Viewer
ConfigEditor -> Designer
FleetAdmin -> Administrator
DriverOperator -> Operator (appsettings-only string role)
This is a rename, not a permission change: enforcement semantics are
preserved (whoever could deploy/administer/operate before still can).
- AdminRole enum members renamed (persisted as string names via
HasConversion<string>); RoleGrants.razor dropdown default updated.
- EF DATA migration CanonicalizeAdminRoles rewrites existing
LdapGroupRoleMapping.Role rows old->new (Up) and back (Down); schema /
model snapshot byte-identical (no pending model changes).
- Enforcement role STRINGS canonicalized:
* Security policies keep their NAMES ("DriverOperator"/"FleetAdmin")
but require canonical roles: RequireRole("Operator","Administrator")
and RequireRole("Administrator").
* Deployments.razor [Authorize(Roles="Administrator,Designer")].
* DevStub now grants "Administrator"; LdapOptions/doc-comment examples
canonicalized.
- Data-plane authorization (NodePermissions/NodeAcl/IPermissionEvaluator/
TriePermissionEvaluator/UserAuthorizationState) UNTOUCHED.
- New CanonicalAdminRolesTests pins canonical claim values end-to-end and
the real registered policies; existing role-string tests updated.
Both bugs surfaced only on split-role deployments (the MAIN cluster's
admin-only nodes), where the AdminUI runs without the driver role.
- Test Connect returned "No probe registered" for every driver: the
IDriverProbe set was registered only under the driver role, but the
admin-operations singleton that consumes it is pinned to admin. Extract
AddOtOpcUaDriverProbes() (idempotent via TryAddEnumerable) and call it
in the hasAdmin path too.
- Live driver-status/alerts/script-log panels showed "SignalR error:
Connection refused": these Blazor Server components opened a HubConnection
to their own hub via the browser's public URL, which server-side code
can't reach behind Traefik (host :9200 -> container :9000). Read the
in-process source directly instead -- DriverStatus via
IDriverStatusSnapshotStore.SnapshotChanged, Alerts/ScriptLog via a new
IInProcessBroadcaster<T>. Fleet status was unaffected (reads DB/ActorSystem).
Adds unit tests for probe registration, the snapshot-store event, and the
broadcaster.
- DriverTestConnectE2eTests: 3 scenarios (sim/wrong-port/black-hole)
against the Modbus Docker fixture. Sim + wrong-port skip if fixture
unreachable; black-hole uses ModbusDriverProbe directly (no fixture).
- DriverReconnectE2eTests: message round-trip through AdminOperationsActor
cluster singleton — Ok=true + audit write, without live driver side effect.
- DriverStatusHubE2eTests: bridge-mocked fallback — spawns
DriverStatusSignalRBridge in the harness ActorSystem with a mock
IHubContext, publishes DriverHealthChanged to the driver-health DPS
topic, asserts store upsert + hub SendAsync call.
- DockerFixtureAvailability helper: TCP-connect probe for skip guards.
- Moq 4.20.72 added to central package management for hub mocking.
- Design doc §8.3 replaced with concrete pre-ship operator runbook.
Adds <summary>, <param>, <typeparam>, and <inheritdoc/> tags to public
members surfaced by commentchecker — resolves 5,847 of 5,869 issues
(99.6%) across three /fixdocs passes.
Joseph Doherty