1. Wiped-node back-fill (live gate check 4). Fixed upstream in
ZB.MOM.WW.LocalDb 0.1.2 and pinned here. The gate's diagnosis was slightly
off: the oplog cap was not the gate. Snapshot detection measured the peer's
gap against the oldest surviving oplog row and read an empty oplog as "no
gap possible" — and an empty oplog is the steady state of a converged pair,
since ack-pruning deletes everything the peer confirmed. The healthy state
was the one state that could not heal a wiped peer. Now measured against
last_acked_seq when the oplog is empty.
Pinned at this level too, not just the library's: the pair harness grows a
WipePassiveAsync (a NEW database — a rebuilt node comes back with a new node
id and a zero watermark), and the new scenario asserts the emptied oplog as
its precondition before wiping, then requires the deployment artifact to
come back byte-identical with no new deploy. Verified RED against the pinned
0.1.1 (times out waiting for back-fill) and green on 0.1.2.
2. Oplog growth on default-OFF nodes (live gate check 8), fixed at the source:
StoreAsync now skips entirely when the pointer already names this
deployment/revision, the SHA matches the bytes, and the expected chunk count
is present. Re-caching an artifact the node already holds — every restart's
boot-from-cache, every RestoreApplied — writes nothing and mints no oplog
rows, where before it cost a delete plus an insert per chunk plus a pointer
update, all identical to what was already there.
Identity is over the bytes and the chunks are counted, both deliberately: a
re-composed artifact can carry the same ids with different bytes, and a
pointer can name a deployment whose chunks are missing, where skipping would
make an unreadable cache permanent. Both guards verified RED against a naive
pointer-only skip.
The gate's stated bound was also wrong and is corrected in the doc: growth
was never headed for the 1M row cap. AddZbLocalDbReplication is registered
unconditionally, so MaintenanceBackgroundService runs on default-OFF nodes
and the 7-day MaxOplogAge cap prunes.
Runtime.Tests 412/0/31, Host.IntegrationTests LocalDb 45/45. Pre-existing and
unrelated: AbCip_Green_AgainstSim (needs the AB CIP docker fixture, red on a
clean master too) and a flaky Roslyn race test (green 2/2 in isolation).
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
The Kestrel re-bind read only urls/ASPNETCORE_URLS. The aspnet:8.0+ base images
set ASPNETCORE_HTTP_PORTS=8080 (all interfaces) as the CONTAINER default in place
of ASPNETCORE_URLS, so a driver node that never sets URLS fell straight through to
Kestrel's localhost:5000 default — silently moving its health/metrics surface to
loopback when the sync listener took over Kestrel. Now falls through URLS →
HTTP_PORTS/HTTPS_PORTS (wildcard, all-interfaces) → localhost:5000, mirroring
Kestrel's own source precedence. Found on the docker-dev live gate (central nodes
were fine — they set ASPNETCORE_URLS=9000 explicitly).
Two driver nodes replicate the deployment-artifact cache over a real loopback
h2c transport through the real fail-closed LocalDbSyncAuthInterceptor; both DBs
init via the production LocalDbSetup.OnReady. Scenarios: byte-identical
convergence with matching pointer HLC+origin, retention prune reaching B as
tombstones, writes-while-transport-down surviving rejoin, and wrong-key
non-convergence with a matching-key positive control.
DoD positive control: commenting out both RegisterReplicated calls turns all
four scenarios red (verified locally, restored before commit).
Builds a container from the same AddOtOpcUa* methods Program.cs's driver branch
calls (AddOtOpcUaLocalDb / AddGrpc / AddOtOpcUaHealth / AddOtOpcUaObservability) and
resolves through it - DI extensions have shipped inert three times in this family.
Does not boot the real host: WebApplicationFactory<Program> would start hosted
services (Akka, OPC UA server, ValidateOnStart) needing SQL + a mesh, per D-5. The
seam under test is DI resolution, so the container is built and resolved without
starting.
Five pins: ILocalDb singleton + exact replicated-table set; IDeploymentArtifactCache
-> concrete; default-OFF ISyncStatus; localdb-replication health check registered and
constructible; admin-only graph has NO ILocalDb and does not demand LocalDb:Path.
LocalDbReplicationHealthCheck reports Healthy when replication is default-OFF or
LocalDb is absent (admin-only graphs) - so a plain node is never degraded by it -
Degraded when configured-but-disconnected, connected-with-unknown-backlog (a failed
oplog poll must not read as zero), or connected-with-backlog past the threshold.
Never Unhealthy: a replication fault does not stop the node serving its address
space. Pure decision core is table-tested; registered unconditionally via a factory
so AddOtOpcUaHealth keeps its no-arg signature and resolves ISyncStatus optionally.
Adds LocalDbMetrics.MeterName to the observability allowlist. o.Meters is a strict
allowlist with no wildcard, so the localdb.sync.* / localdb.oplog.depth series were
otherwise silently absent from /metrics - the omission a ScadaBridge live gate
caught.
Bumps the four ZB.MOM.WW.Secrets pins 0.2.1 -> 0.2.2 (closes the OtOpcUa side
of scadaproj#1, tracked here as #482). 0.2.1's Akka replicator deadlocked any
hosted process at startup when Secrets:Replication:Enabled was true: the
package's DI graph closed a circular singleton dependency through factory
lambdas (store decorator -> replicator -> actor provider -> cache invalidator
-> resolver -> store), which MS.DI's StackGuard turns into a silent
cross-thread call-site-lock deadlock. 0.2.2 defers the invalidator edge to
first eviction. The flag stays default-false; enabling remains a
per-environment decision.
The_startup_hook_actually_creates_the_replication_actor is now a real test:
SecretReplicationStarter's docs had promised it since the adoption, and the
upstream fix finally makes a provider-based resolve runnable - container built
exactly as the host does, hook started under a watchdog, replication actor
proven to exist by ActorSelection on a self-joined single-node cluster (no
TestKit needed, which matters because Akka.TestKit.Xunit2 is xunit-v2-only and
this project is on xunit.v3). Also corrects the stale rationale that blamed
the old hang on DistributedPubSub needing a joined cluster - the actor
constructor was never reached; it was the DI cycle.
Verified: SecretsReplicationRegistrationTests 8/8 on the 0.2.2 feed packages;
full slnx build 0 errors; the 2-node Akka live convergence gate re-run against
the published 0.2.2 packages passes 6/6 (write->peer, tombstone propagation
without resurrection, delete visibility through the resolver cache, reverse
direction, wrong-KEK fail-closed).
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Akka.TestKit is the family convention for non-end-to-end Akka testing and would
be the right tool here, but Akka.TestKit.Xunit2 is xunit-v2-only and this project
is on xunit.v3 - adding it produces CS0433 type conflicts. Directory.Packages.props
already documents the same constraint for the three projects deliberately held on
xunit v2 for TestKit. Revisit when Akka ships an xunit.v3 TestKit.
0.2.0's AddZbSecretsAkkaReplication never bound its own ISecretReplicator: it
called AddZbSecrets first, which TryAdds NoOpSecretReplicator, so the package's
own TryAdd was silently discarded. Replication published into a no-op sink and no
actor was ever spawned - no exception, no log line. Found by the Task 6 wiring
work, which is the first code that ever built a container around that extension.
Fixed upstream in 0.2.1.
Test consequence worth recording: the replication-enabled registration tests
previously passed against a plain ActorSystem only BECAUSE of that bug - the
no-op never touched Akka. With 0.2.1 they resolve a real replicator, which spawns
the actor, whose PreStart needs DistributedPubSub and therefore a joined cluster,
so provider-based assertions hang. Standing a real cluster up inside this test
assembly was attempted and did not work, so these assertions are now made against
the ServiceCollection, which is precisely where the defect lives. Actor creation
and convergence remain covered upstream by the library's TwoNodeClusterReplication
tests against a genuine 2-node cluster.
7 registration tests pass in 23ms. Full solution builds 0 errors; no warnings
originate from these files.
Routes the host's secrets registration through a new AddOtOpcUaSecrets extension
that gates the ISecretStore implementation on Secrets:Replication:Enabled.
Opt-in gate (default FALSE)
This call decides which ISecretStore every node resolves — including driver-role
nodes with no auth/AdminUI, where a wrong store surfaces as drivers failing to
open sessions rather than as a failing test. With the flag false the wiring is
the pre-existing AddZbSecrets(config, "Secrets") call, unchanged, so current
behavior is byte-identical. With it true, AddZbSecretsAkkaReplication replaces
that call (it invokes AddZbSecrets internally; calling both would double-register).
Extracted to a named extension specifically so the registration is testable:
Program.cs is top-level statements and cannot be exercised by a container test,
which is how a "registered but never resolvable" defect ships unnoticed.
Serializer HOCON
AkkaSecretsReplication.SerializationConfig is merged into the ActorSystem config
inside the AddAkka configurator, conditionally on the same gate — a non-replicating
node carries no bindings for messages it will never see. Merged via
AddHocon(..., HoconAddMode.Append), Akka.Hosting's fallback merge and the same mode
the existing base-config merge uses; a raw Config.WithFallback would fight the
builder's own assembly.
Lazy-actor mitigation
The replication actor is created lazily on first ISecretStore resolution, so a node
that never touches a secret would never announce a manifest and would silently never
converge. SecretReplicationStarter (IHostedService) resolves the store once at
startup to make participation unconditional.
KNOWN BLOCKER — replication is currently NON-FUNCTIONAL; do not enable
ZB.MOM.WW.Secrets.Replicator.AkkaDotNet 0.2.0 never binds its own ISecretReplicator.
AddZbSecretsAkkaReplication calls AddZbSecrets FIRST, which does
TryAddSingleton<ISecretReplicator, NoOpSecretReplicator>(); the package's own
TryAddSingleton<ISecretReplicator>(AkkaSecretReplicator) that follows is therefore
a no-op. Verified empirically in a built container: with Enabled=true,
ISecretReplicator resolves to NoOpSecretReplicator, so ReplicatingSecretStore
publishes into a sink and no actor is ever spawned.
Consequence: the startup hook cannot create the actor, and the test asserting it
does is committed Skipped with the evidence. Not worked around here — the fix
belongs upstream (AddSingleton, or register before calling AddZbSecrets).
Because the flag defaults false, this commit is inert in production.
Tests: SecretsReplicationRegistrationTests (new) — disabled path resolves plain
SqliteSecretStore and needs no ActorSystem; enabled path resolves
ReplicatingSecretStore AND the undecorated concrete SqliteSecretStore the decorator
is built from (the exact registration gap that shipped once); startup hook registered
only when enabled. Red before wiring (4 assertion failures), green after: 6 pass,
1 skipped (blocker above).
Build: 861 warnings / 0 errors, unchanged from baseline (full --no-incremental A/B).
Host.IntegrationTests: 123 pass, 6 skip, 1 fail — AbCip_Green_AgainstSim, verified
pre-existing on the stashed tree (fixture-gated).
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
The full-suite gate caught a real regression: AddOtOpcUaDriverFactories' registration
now resolves ISecretResolver (for Galaxy/OpcUaClient secret: refs, Tasks 7-8), so the
two ResilienceInvokerFactoryRegistrationTests that build a minimal container and resolve
the invoker factory threw 'No service for ISecretResolver'. The real host always
registers it via AddZbSecrets (unconditional, Task 3), and GetRequiredService correctly
fails-fast for a genuinely misconfigured host — so the fix is to complete the test
container with a stub resolver, matching production composition. Test-only.
Pre-existing stale test surfaced by Batch 4's full gate (Batch 2/3 gates were
live-/run and never ran Host.IntegrationTests): Batch 2 registered CalculationProbe
in DriverFactoryBootstrap (9 probes) but AddOtOpcUaDriverProbes_is_idempotent's
AdminUiDriverTypeKeys still listed 8, so distinctTypes(9) != Length(8). Calculation
is a real DriverTypeNames.Calculation pseudo-driver with a probe; add it and refresh
the now-stale *DriverPage.razor comment (that routed flow retired in Batch 2). Not a
Batch-4 code change — a test-correctness fix.
Claude-Session: https://claude.ai/code/session_01LVneM3eh1UtJxEisFXgmox
H1 (HIGH): write-routing key now (AddressSpaceRealm, bareId), not bare-only.
A raw s=<RawPath> and a UNS s=<Area/Line/Equip/Eff> can collide as bare
strings; the bare-only key let a colliding raw+UNS pair route to the WRONG
driver ref (last-writer-wins). The realm the node manager resolves (RealmOf)
is now threaded through IOpcUaNodeWriteGateway.WriteAsync -> RouteNodeWrite ->
_driverRefByNodeId keyed by (realm, bareId). New regression test:
Colliding_raw_and_uns_bare_ids_route_to_their_own_driver_by_realm.
M1 (MEDIUM): discovered-node injection made coherently DORMANT. HandleDiscoveredNodes
hard-short-circuits (single enforcement point; _discoveredByDriver never
populates so the re-inject tail is inert too), with a clear log pointing at the
/raw browse-commit flow. New pin: Discovered_nodes_are_ignored_dormant_in_v3;
the 16+2 v2 injection scenarios re-pointed to an accurate skip reason
(DiscoveryInjectionDormantV3).
M2 (MEDIUM): realm-qualified dual-node self-correction tests —
Failed_uns_write_reverts_uns_node_and_leaves_raw_node_untouched +
Raw_realm_revert_reverts_raw_node_only (the second fails if the realm is dropped).
L1: removed the = AddressSpaceRealm.Uns defaults from the consequential
node-manager mutation methods (WriteValue/WriteAlarmCondition/MaterialiseAlarmCondition/
EnsureFolder/EnsureVariable/UpdateFolderDisplayName/UpdateTagAttributes/
RaiseNodesAddedModelChange/Remove*/RevertOptimisticWriteIfNeeded) + the
AttributeValueUpdate/AlarmStateUpdate records, so the compiler forces explicit
realm; read-only accessors + internal builders retain their defaults.
L3: fixed the stale VirtualTagHostActor class comment (V3NodeIds.Uns, not the
retired EquipmentNodeIds.Variable).
Also: DeploymentArtifactRawUnsParityTests — Raw/UNS node-set byte-parity
round-trip between AddressSpaceComposer.Compose and DeploymentArtifact.ParseComposition.
Claude-Session: https://claude.ai/code/session_01LVneM3eh1UtJxEisFXgmox
Un-darken the address-space composition for the v3 dual namespace.
Composer (AddressSpaceComposer):
- Raw subtree: RawContainerNode (Folder/Driver/Device/TagGroup as
Object/Folder nodes, keyed s=<RawPath>, parent-before-child) + RawTagPlan
(raw tag Variables keyed (realm=Raw, s=<RawPath>) carrying DataType /
writable / historize+historian tagname / array shape). Native-alarm intent
attaches at the RAW tag (ConditionId = RawPath) with the list of referencing
equipment UNS paths (one alarm at the raw tag, not one per equipment).
- UNS subtree: UnsReferenceVariable per UnsTagReference, keyed
(realm=Uns, s=<Area>/<Line>/<Equipment>/<EffectiveName>), carrying its
backing RawPath (Organizes UNS->Raw target + fan-out) and inheriting
DataType/AccessLevel from the backing raw tag. Effective name =
DisplayNameOverride else backing raw Tag.Name.
- All RawPaths flow through one shared RawTopology (RawPathResolver +
memoised container paths), byte-parity with EquipmentReferenceMap.
- Every new node carries an AddressSpaceRealm (so WP3's applier picks the
namespace at the call site). Additive/defaulted model changes only — the
un-migrated AddressSpaceApplier still compiles.
Planner (AddressSpacePlanner.Compute) + AddressSpacePlan:
- Per-realm diff sets: RawContainers + RawTags keyed by RawPath (NodeId) so a
rename = remove+add; UnsReferenceVariables keyed by the stable
UnsTagReferenceId so a backing-tag rename re-points (BackingRawPath moves,
NodeId stable) and a display-name-override edit is UNS-only.
- PINNED: raw-tag rename = remove+add in Raw AND re-point in UNS.
Classifier folds the new sets in (adds->PureAdd, removes->PureRemove,
changed->Rebuild safe default).
Tests: AddressSpaceComposerDualNamespaceTests (7),
AddressSpacePlannerDualNamespaceTests (8, incl. the rename pin),
AddressSpaceChangeClassifierDualNamespaceTests (8); reactivated
EquipmentNamespaceMaterializationTests' Batch-4-pending case, re-authored to
drive the composer over the seeded config (the artifact-decode seam is WP3).
Claude-Session: https://claude.ai/code/session_01LVneM3eh1UtJxEisFXgmox
Corrects the #459 finding. 2-node keep-oldest recovery works fine (the ScadaBridge
sister project proves it); OtOpcUa was missing the supervision pieces that make it
automatic, and docs/Redundancy.md wrongly claimed in-place oldest-crash failover.
Mechanism (confirmed on a 2-container rig + by decompiling Akka KeepOldest.OldestDecision):
on an OLDEST-node crash keep-oldest downs the LONE survivor (DownReachable including
myself) — down-if-alone can't rescue a lone survivor (its branch needs >=2 survivors).
Recovery is exit-and-rejoin: run-coordinated-shutdown-when-down terminates the node and
the service supervisor restarts it. My earlier 'total outage' was a docker-dev artifact
(no restart policy); production Install-Services.ps1 already has sc.exe failure restart.
Changes (ScadaBridge parity):
- ActorSystemTerminationWatchdog (Host, registered after AddAkka): watches
ActorSystem.WhenTerminated and on an unexpected self-down calls StopApplication so the
process exits (supervisor restarts it) instead of idling with a dead actor system.
Distinguishes graceful shutdown via _stopRequested + ApplicationStopping. 3 unit tests.
- docker-dev: restart: unless-stopped on the host anchor (models production supervision) +
both redundancy peers in SeedNodes so a restarted node re-forms via either peer.
- docs/Redundancy.md: rewrote the split-brain recovery section — younger-loss = in-place
fast failover; oldest-loss = exit-and-rejoin under supervision (not in-place); the three
requirements (supervisor + watchdog + both-node seeds); flagged HardKillFailoverTests as
non-representative (Transport.Shutdown, not a real crash). Instant in-place takeover on
ANY single loss needs 3+ members.
Cluster.Tests 29/29 (SBR guards), watchdog tests 3/3, full solution builds.
Live re-verify of the watchdog image pending (host docker disk full).
Closes the R2-08 S2 live leg. The offline LdapAuthResilienceTests prove the
directory-outage circuit with fakes + an unroutable-host blackhole; this drives
the SAME circuit through the production auth path (OtOpcUaLdapAuthService +
LdapOpcUaUserAuthenticator) against a REAL GLAuth that is paused mid-run and
unpaused — and doubles as the first end-to-end verification that the PR #451
GLAuth swap (retired bitnami/openldap) binds correctly.
Triple-gated (GLAUTH_LIVE_HOST + GLAUTH_OUTAGE_START_CMD/STOP_CMD), skips clean
offline (verified: 1 skipped, 8 ms). try/finally always unpauses GLAuth.
Verified GREEN against 10.100.0.35:3894 (12 s): healthy binds (alice+bob, mapped
roles) -> docker pause -> 3 bounded failures -> circuit opens -> sub-second
"unavailable" fast-deny -> docker unpause -> half-open probe closes -> binds
succeed again. A boundary AuthTimeout feeds the circuit (authenticator :123-130),
so the docker-pause timeout shape is a faithful outage.
Integration-sweep follow-up #6 (SQL leg). The sweep flagged 7 heavy 2-node
deploy/failover/reconnect E2E tests as 'time out under amd64-emulated SQL' and
suggested raising deadlines. That diagnosis was WRONG: run against real
(native-amd64) SQL on the Docker host, they still hang — even at 4x deadline.
Root cause is that these tests had never actually run against real SQL, only the
EF in-memory provider, which does NOT enforce foreign keys.
Two distinct defects, both hidden by in-memory:
1. Missing FK-parent seed (5 tests). The deploy records a NodeDeploymentState row
per node, FK'd to ClusterNode (FK_NodeDeploymentState_ClusterNode_NodeId). The
tests never seeded ServerCluster + ClusterNode, so on SQL each node-state INSERT
throws and the deploy never seals (waits the full deadline regardless of size).
Added TwoNodeClusterHarness.SeedDefaultClusterAsync (seeds a ServerCluster +
a ClusterNode per node; Warm/NodeCount=2 to satisfy the SQL CHECK constraint
CK_ServerCluster_RedundancyMode_NodeCount AND the ClusterEnabledNodeCountMismatch
validator). Called it in DeployHappyPath x2 / Failover / FleetDiagnostics /
EquipmentNamespace; fixed DriverReconnect to seed node B + NodeCount=2.
2. Stale EquipmentId (EquipmentNamespace test; pre-existing on BOTH providers).
Seeded EquipmentId='eq-1', but the later-added EquipmentIdNotDerived validator
requires EquipmentId == DeriveEquipmentId(EquipmentUuid) ('EQ-'+first 12 hex).
Fixed the seed to a canonical id + matching UUID. Only surfaced once the FK fix
let the deploy reach equipment validation.
Also added OTOPCUA_HARNESS_SQL_HOST / OTOPCUA_HARNESS_LDAP_HOST overrides so the
harness fixtures can run on the native-amd64 Docker host (avoids arm64 mssql
emulation entirely).
Verified: the 5 SQL-backed classes go 11/11 green vs native SQL; in-memory default
unregressed (12/12). RoslynVirtualTagEvaluatorTests racing test is a pre-existing
flaky race (not SQL-backed; passes in isolation) — left as-is.
The Host.IntegrationTests opt-in real-LDAP mode (OTOPCUA_HARNESS_USE_LDAP=1)
used bitnami/openldap:2.6, gone since Bitnami deprecated their free image
catalog in 2025. Replaced it with GLAuth — the same LDAP server the rest of
the project already uses (docker-dev + the live OPC UA data-plane auth, both
against the shared GLAuth on :3893) — removing the lone OpenLDAP outlier and
the gone-image breakage.
- Added tests/.../Host.IntegrationTests/glauth/config.toml: baseDN dc=zb,dc=local,
a search-capable serviceaccount, dev users alice (full access) / bob (read-only),
and role groups with the same gidnumbers as scadaproj/infra/glauth so GroupToRole
maps identically.
- Compose ldap service -> glauth/glauth:latest, mounting the config, host :3894 -> :3893.
- Repointed the harness real-LDAP override from the OpenLDAP cn=admin/ldapadmin to
GLAuth's cn=serviceaccount/serviceaccount123.
Live-verified against a deployed container on :3894: the serviceaccount bind
searches and returns alice + her 5 memberOf groups; alice/bob bind with the
correct password; a wrong password yields Invalid credentials (49) — exactly the
OtOpcUaLdapAuthService flow (proven identical to the data-plane's shared-GLAuth path).
NB: real-LDAP mode is opt-in; the default Host run uses StubLdapAuthService, so
this never blocked the default suite. Integration-sweep follow-up #6 (LDAP leg);
the amd64-emulated-SQL deadline leg remains open.
Findings 01/S-1 (AddressSpaceApplyOutcome failure field + apply.failed logging,
no optimistic success), 06/S-1 (Galaxy write fails closed -> BadCommunicationError
+ #5 revert, no knowingly-lost raw Write), 03/S4 (PrimaryGatePolicy default-deny
unknown-role-multi-driver on all gates + scripted-alarm emit gate). T13/T15 (2-node
live gates) deferred -- T15 is the behavior-affecting S4 live gate, must run in heavy
pass. Clean merge, build clean.
AddOtOpcUaDriverFactories (the driver-node bootstrap) must bind the real
DriverCapabilityInvokerFactory — not the NullDriverCapabilityInvokerFactory
pass-through — and it must mint a real CapabilityInvoker per instance. This is the
deterministic DI half of the #10 live gate: a refactor dropping back to the
pass-through would make the whole retry/breaker/bulkhead pipeline inert in production
while every pass-through-defaulting unit test stays green. 3 tests (real factory bound,
real invoker minted, shared singletons), non-fixtured. Host.IntegrationTests.
The load-bearing integration half of Critical 1. Adds TwoNodeClusterHarness.HardKillNodeAAsync
(canonical in-process crash sim: shut down node A's remoting transport via
IRemoteActorRefProvider.Transport.Shutdown() — associations drop, heartbeats/gossip stop, NO
graceful Cluster.Leave, so node B's failure detector marks A Unreachable) + a
WaitForNodeBSoleDriverLeaderAsync helper (waits for a STABLE takeover: B sole Up member + driver
role-leader). HardKillFailoverTests crashes the oldest node and asserts the survivor takes over —
the exact scenario the graceful StopNodeBAsync path (Cluster.Leave, no downing decision) cannot
exercise.
Live-verified in-process (~35s: acceptable-heartbeat-pause 10s + stable-after 15s + convergence).
Negative control confirmed the test BITES: forcing explicit NoDowning
(akka.cluster.downing-provider-class = "") makes it time out with the node stuck Up-but-Unreachable.
FINDING surfaced by the negative control: removing Critical 1's typed
ClusterOptions.SplitBrainResolver did NOT break failover, because Akka.Cluster.Hosting's
WithClustering applies SplitBrainResolverOption.Default when the option is null — enabling the SBR
downing provider that reads the akka.conf keep-oldest block (present on master BEFORE Critical 1).
So the cluster was NOT running NoDowning before Critical 1; the typed option reinforces the akka.conf
strategy rather than being the sole activator, and Critical 1's 'HOCON inert / NoDowning' premise is
inaccurate. The test guards the failover OUTCOME regardless of activation path. Comments corrected to
reflect this; flagged for review.
U2 (Critical): RoslynVirtualTagEvaluator ran scripts with a CancellationToken
that a synchronous Roslyn script can never observe, so one CPU-bound/infinite-
loop virtual-tag script hung the owning VirtualTagActor forever (no timeout, no
recovery). Route evaluation through the existing TimedScriptEvaluator (Task.Run
+ WaitAsync) so the wall-clock budget is real; a runaway script now returns Bad
within the timeout instead of wedging the actor.
U3 (High): the live path cached evaluators in a raw ConcurrentDictionary never
cleared on republish, leaking a collectible AssemblyLoadContext per edited
script forever. Swap in the purpose-built CompiledScriptCache (Lazy single-
compile, dispose-on-Clear) and add an apply-boundary clear: a new narrow
IScriptCacheOwner seam the VirtualTagHostActor calls on each ApplyVirtualTags
generation (mirrors ScriptedAlarmEngine's per-generation clear).
Tests: infinite-loop-fails-within-timeout (bounded by WaitAsync so a regression
fails instead of hanging), happy-path-after-wrapping, ClearCompiledScripts-
empties-cache, and a host-actor wiring guard asserting each apply generation
calls ClearCompiledScripts (theme #1 production-caller assertion). Host.Integration
VT suite 18/18, Runtime VT suite 12/12.
The HistorianGateway driver is now the sole historian read/write+alarm backend, so the
Wonderware sidecar projects are dead code. Removes the 5 Wonderware projects (driver,
.Client, .Client.Contracts, + their 2 test projects) from the solution and tree, and fully
retires the vestigial 'Historian.Wonderware' driver type (UI/probe-only; it had no driver
factory): the Host probe registration, the AdminUI driver-config surface (driver page,
tag-config editor/model/validator entry, address picker/builder, driver-type catalog +
dropdown + edit-router entries), and their tests. Prunes the now-unused Wonderware
connection fields (Host/Port/UseTls/ServerCertThumbprint/SharedSecret) from
AlarmHistorianOptions (keeping Enabled + the SQLite store-and-forward knobs) and refreshes
the stale XML docs that named Wonderware as the production backend.
Claude-Session: https://claude.ai/code/session_012SDSQ3AcaXqPcBtDESBRii
AdminUI driver-instance pages serialized enum config fields (S7 CpuType,
Modbus DataType/Region, AbCip PlcFamily, ...) as JSON *numbers* because each
page's _jsonOpts lacked a JsonStringEnumConverter. The driver factories,
however, deserialize into string-typed DTOs (+ lenient ParseEnum) and throw
when binding a JSON number to a string? — so an AdminUI-authored config
containing any enum field produced a blob the driver could not parse,
faulting the driver on deploy. Proven end-to-end for S7 and Modbus; latent
for AbCip/AbLegacy/TwinCAT/FOCAS/Galaxy/Historian. Only OpcUaClient was safe
(its factory + probe already carried the converter).
Add JsonStringEnumConverter to all 9 driver-instance pages' _jsonOpts and the
8 missing driver probes' _opts (factories unchanged — already string-via-
ParseEnum; strictly more permissive, also lets pages load hand-seeded
string-enum configs back into the form).
Also fix DriverProbeHandshakeE2eTests.AbCip_Green_AgainstSim to probe a real
sim tag (TestDINT) — the no-tags @raw_cpu_type fallback is rejected by the
ab_server sim with ErrorBadParam (a real ControlLogix returns ErrorNotFound,
which the probe treats as reachable; hardware-gated follow-up).
Tests: reflection guard over all driver pages' _jsonOpts (AdminUI.Tests);
factory round-trip + numeric-form-throws guards for S7 and Modbus.
Found by running the never-before-run FB-9/FB-10 live verifies.
The OPC UA address-space build pipeline was named after a v2-roadmap
milestone number rather than its domain. Rename the family to describe
what it does (build/diff/apply the OPC UA address space):
Phase7Composer -> AddressSpaceComposer
Phase7CompositionResult -> AddressSpaceComposition
Phase7Planner -> AddressSpacePlanner
Phase7Plan -> AddressSpacePlan
Phase7Applier -> AddressSpaceApplier
Phase7ApplyOutcome -> AddressSpaceApplyOutcome
The 9 Phase7*Tests suites follow suit; Phase7ScriptingEntitiesTests ->
ScriptingEntitiesTests (it tests the scripting migration, not the
pipeline). Log-message prefixes move to the new class names.
Pure mechanical rename, no behavioral change. EF migration classes/IDs
(AddPhase7ScriptingTables, ExtendComputeGenerationDiffWithPhase7) are
immutable and left untouched, as are historical design docs.
Build clean; OpcUaServer 261/261, Runtime 272/272, ScriptingEntities
12/12 green.
Two bugs caught by live verification against the mxaccessgw at 10.100.0.48:5120:
- MaxAttempts=1 produced an invalid Polly RetryStrategyOptions -> the probe failed
on every real gateway. Removed the Retry override (matches GalaxyDriver); fail-fast
is already guaranteed by the TCP preflight + the per-call deadline.
- A rejected key surfaces as a typed MxGatewayAuthenticationException, not a raw
RpcException, so 'auth-rejection = reachable' was bypassed. Catch the typed auth/
authorization exceptions -> Ok=true.
Adds DriverProbeHandshakeE2eTests: direct-probe, skip-gated cross-protocol green/red
discrimination (Modbus, OpcUaClient, Galaxy + a local real OPC UA server).
All five suppressed advisories are now resolved at baseline/resolved versions,
so every NuGetAuditSuppress is removed repo-wide:
- System.Security.Cryptography.Xml (GHSA-37gx-xxp4-5rgx / GHSA-w3x6-4m5h-cxqf)
-> fixed by the .NET 10 baseline (10.0.6)
- OPCFoundation Opc.Ua.Core (GHSA-h958-fxgg-g7w3) -> fixed at resolved 1.5.378.106
Two were still live and are now patched via direct security pins:
- OpenTelemetry.Api 1.9.0 -> 1.15.3 (GHSA-g94r-2vxg-569j) pinned in Cluster;
Runtime/ControlPlane/AdminUI + tests inherit via project reference
- Tmds.DBus.Protocol 0.20.0 -> 0.21.3 (GHSA-xrw6-gwf8-vvr9) pinned in Client.UI
Also correct the Historian sidecar runtime comments (x86 -> x64, matching the
csproj PlatformTarget). Solution audit: 0 vulnerable packages; full build clean.
Joseph Doherty