cd4b29a36879110c44311a4fe38540d3fd7aadb6
107 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Joseph Doherty
|
f10f4c3754 |
fix(auth): #543 whitespace-only configured key fails closed in the three pre-existing gRPC auth interceptors
TelemetryStreamAuthInterceptor, ConfigServeAuthInterceptor and LocalDbSyncAuthInterceptor all decided "is a key configured?" with string.IsNullOrEmpty. A whitespace-only configured value therefore passed that guard and fell through to the constant-time comparison — where it is satisfied by a whitespace-only bearer token. The gate would be open to anyone who guessed a blank credential. This backports the check the two NEW mesh interceptors already use: MeshCommandAuthInterceptor (and its central-side clone) validate with string.IsNullOrWhiteSpace, so the four/five planes now answer the same way to the same misconfiguration. Nothing else changes — the denial status codes (PermissionDenied / Unauthenticated / PermissionDenied respectively) and every denial message are untouched, and no key material is logged. No deployment configures a whitespace key: the docker-dev rig and production both set real keys, so this is a fail-closed hardening of an unreachable-today misconfiguration rather than a live exposure. Tests: each of the three test files gains a ..._WithBlankKeyConfigured_IsDenied fact mirroring MeshCommandAuthInterceptorTests.ExecuteMethod_WithBlankPskConfigured_IsDenied (configured key " ", presented "Bearer "), verified red before the fix. ConfigServeAuthInterceptor and LocalDbSyncAuthInterceptor also gain the HasExactlyOnePublicConstructor pin their two siblings already had — Grpc.AspNetCore silently stops invoking an interceptor with more than one public constructor, which would disable auth without failing anything. Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1 |
||
|
Joseph Doherty
|
cd7135570c |
refactor(mesh): collapse MeshPaths to local actor names — no longer a wire contract (review follow-up)
MeshPaths.CentralCommunication / .NodeCommunication (the "/user/..." full-path constants) had
ZERO production consumers after the ClusterClient deletion: both gRPC services resolve their
comm actor through ActorRegistry by typed key, never by ActorSelection over a path. Only the
path-probe test still read them, while the class doc went on calling them "the wire contract
of the central↔node boundary" — a claim that stopped being true the moment nothing addressed
an actor across a process boundary by name.
- The two full-path constants are deleted; the two Name constants remain.
- MeshPaths' doc now says what it is (the local actor-naming convention the registry-keyed
services are spawned under) and records what it stopped being, so the next reader does not
re-derive a contract from the file's history.
- MeshCommActorPathTests composes "/user/" + Name itself — the absolute path is that test's
own addressing detail now, not something the product declares. Renamed
Both_mesh_comm_actors_exist_at_their_contract_paths ->
..._are_spawned_under_their_declared_names, because what it still pins is that the two
registration sites actually ran on a real booted node; a rename is now a compile error, so
"contract" was the wrong word for what would break.
- NodeCommunicationActor's summary no longer cross-references the deleted constant or calls
the name an address. CentralCommunicationActor was checked and makes no such claim.
|
||
|
Joseph Doherty
|
4d2dfb5b50 |
test(mesh): step 7 — replace the boundary test with MeshGrpcBoundaryTests and sweep the comments
MeshGrpcBoundaryTests replaces the MeshClusterClientBoundaryTests deleted in step 1. Three
self-seeded single-member clusters sharing the ActorSystem name, joined ONLY by gRPC, driving
the REAL CentralCommunicationActor -> MeshGrpcCommandDialer -> Kestrel h2c ->
MeshCommandAuthInterceptor -> NodeCommandGrpcService -> NodeCommunicationActor -> EventStream:
1. No_cross_mesh_Akka_association_forms_between_central_and_either_site — the program's own
point, kept as a test rather than a live-gate note: each cluster stays single-member and
none ever holds another's address as a member.
2. One_dispatch_through_central_reaches_a_node_in_every_cluster_mesh — the fan, with each
site holding its OWN per-cluster PSK, so a dial that selected the wrong cluster's key
would be refused at the far side.
3. Falsifiability_control_a_dial_carrying_the_wrong_cluster_key_delivers_nothing — without
it, (2) cannot distinguish "the gRPC boundary carried it" from "something else in the
process did". The same node then accepts the right key, so the silence is the auth gate
and not a dead endpoint.
All three run fully offline — no Docker fixture — and pass.
Comment sweep across every remaining clusterclient|receptionist hit in src/ and tests/:
the program label becomes "mesh gRPC command plane", and every comment that described the
deleted transport as live now describes what actually runs (Dps = pair-local default,
Grpc = cross-mesh). mesh_transport.proto's header/service comments are reworded; no message,
field or enum is touched — the contract is locked. MeshCommActorPathTests is reshaped rather
than retired: the two paths still exist and still matter, but what a mismatch costs is now a
misleading Unavailable from the gRPC ingress rather than a silently dropped receptionist send.
The only surviving mentions of the old names are the ones that must survive: the fail-closed
rules that name "ClusterClient" as a retired value refused at boot, their tests, and this
test class's own account of what it replaces.
|
||
|
Joseph Doherty
|
0db40a5e7f |
refactor(mesh): step 1 — delete the central ClusterClient factory and per-cluster client cache
IMeshClusterClientFactory/DefaultMeshClusterClientFactory are gone, along with
CentralCommunicationActor's _clients map, RebuildClient, the SendToAll fan and the
receptionist-address half of the contact load. LoadContactsFromDb SURVIVES — it feeds the
gRPC dialer's endpoint resolution and the known-cluster gate — but now projects only the
columns those two need (the AkkaPort/ActorPath.TryParse parse and its Malformed list went
with the receptionist addresses).
Ripples fixed in this step because the build must be green at every commit:
- ControlPlane/ServiceCollectionExtensions drops the factory argument.
- CentralCommunicationActorTests: the six ClusterClient-only facts are deleted (each has a
Grpc counterpart that already asserted the same property independently), the three
transport-agnostic facts move to Grpc mode, and the Dps fact keeps its own coverage.
- MeshClusterClientFactoryTests deleted (its subject is gone).
- MeshClusterClientBoundaryTests deleted here rather than in step 7: it drives
DefaultMeshClusterClientFactory directly, so it cannot compile past this commit. Its
replacement, MeshGrpcBoundaryTests, lands in step 7 and keeps the load-bearing assertion
(two separate meshes, no cross-mesh Akka association).
|
||
|
Joseph Doherty
|
6b3da93151 |
fix(browse): commit the browsed data type, and fix the probe classification the fixture was hiding (#536)
A browse-committed AB CIP / TwinCAT / FOCAS tag bound with its model's DEFAULT
data type, not the one the device reported. A browsed REAL committed as
`{"tagPath":"Pump_1_Speed","dataType":"DInt"}` — and those three drivers decode
the wire bytes by that field, not by the raw row's OPC UA DataType — so the tag
read four bytes of a float through an integer decoder and published a plausible
wrong number with no error raised anywhere. FOCAS was missed when #536 was filed.
The type is CARRIED, not derived. Reverse-mapping the leaf's driver-agnostic
DriverDataType looks like the smaller fix and is wrong: SInt/Int/DInt/USInt/UInt
all collapse into Int32 on the way out of the driver, so that version would bind
a 16-bit INT as a 32-bit DINT while appearing to fix the REAL. Drivers now state
their own type name on DriverAttributeInfo.NativeDataType, which reaches the
picker as AttributeInfo.AddressFields["dataType"] and the commit mapper from
there. A leaf that states nothing keeps the model default — the pre-#536
behaviour, kept deliberately as the fallback rather than a guess.
Numeric strings are refused before parsing: Enum.TryParse accepts "9" for any
enum and Enum.IsDefined then confirms it, so an unguarded parse would bind a tag
to a type nobody named. A test caught this in the first version of the guard.
--- and the other half, which the fixture was hiding ---
Test Connect on a device with NO authored tags — every AB CIP device at the
moment an operator first presses the button — reported "CIP handshake failed"
against a healthy controller. That was written down as a fixture limitation
("ab_server doesn't implement @raw_cpu_type"). Both halves of that were wrong:
* @raw_cpu_type is not a libplctag special tag at all (only @raw, @tags and
@udt/ are), so it goes out as an ordinary symbolic name that no controller
defines, and the controller's "tag does not exist" IS the successful answer.
* ab_server answered an unresolvable tag path with CIP 0x03 where a Logix
controller answers 0x04 — ErrorBadParam vs ErrorNotFound, which the probe
classifies oppositely. Fixed in vendored patch 0002.
* With the fixture then telling the truth, the real defect surfaced:
AbCipDriverProbe.IsReachableException matched the native C status spellings
("STATUS_NOT_FOUND"), but libplctag.NET throws with the MANAGED enum name
("ErrorNotFound"). Those substrings can never match, so the reachable
classification could not fire on the exception path — which is the path every
no-authored-tags probe takes. This was failing on real hardware too, not just
against the sim. It now recovers the Status and defers to IsReachableStatus,
so there is one classification rather than two that can drift.
Rejected: switching the fallback to @tags. libplctag builds a tag-listing tag
with NO network I/O, so it reports OK against any TCP listener — measured green
against a Mosquitto broker.
Verification: full solution suite green. Falsification runs confirm exactly the
intended tests go red — 9 for the type carry, 3 (the reachable statuses only) for
the probe classification. Live on docker-dev: browse-committed TestString/
Motor_Run/TestSINT now store String/Bool/SInt, alongside last session's
Pump_1_Speed still holding the wrong DInt. TestSINT is the falsifier — its row
DataType is Int32, identical to TestDINT's, so no reverse map could have produced
"SInt".
Known consequence: pre-existing browse-committed tags keep their wrong dataType;
the fix applies to new commits only.
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
|
||
|
Joseph Doherty
|
df340b8bba |
fix(tests): three real failures the integration suite had stopped catching
None of these were fixture absence. Each is a guard that went quietly wrong and then reported the wrong thing. PrimaryGateFailoverTests — per-cluster mesh Phase 6 ( |
||
|
Joseph Doherty
|
65fda7237f |
refactor(acl): delete the LDAP data-plane authenticator and its tests (#520)
Stage 2b. LdapOpcUaUserAuthenticator was left registered nowhere by stage 2a; this removes it, its three test files, and the config surface that existed only to serve it. Deleted: LdapOpcUaUserAuthenticator, LdapOpcUaUserAuthenticatorTests, LdapAuthResilienceTests, LdapGlauthOutageLiveTests (19 tests). Also removed four Security:Ldap keys — AuthTimeoutMs, MaxConcurrentAuthentications, OutageFailureThreshold, OutageCooldownSeconds — plus their validator rules and binding/validator test coverage. Verified first that the ONLY consumers were the deleted class and the validator rules asserting them; none appear in any checked-in appsettings. Leaving them would have created exactly the dead-config surface #521/#524/#525 were filed to clear up. Worth stating why that resilience machinery does not need relocating rather than just deleting. It existed because the OPC UA SDK raises impersonation SYNCHRONOUSLY under a SessionManager-wide event lock, so a hung LDAP bind there serialized every session activation server-wide — Anonymous and X509 included. Key verification is one indexed lookup plus one HMAC with no network I/O, so the hazard the wall-clock bound, concurrency cap and outage circuit guarded against no longer exists on that path. LDAP now runs only on AdminUI login, an ordinary async request path where a slow directory affects the one request making it. ConnectionTimeoutMs is RETAINED: it is projected into the shared library and is not authenticator-specific. Docs corrected rather than left to rot: - CLAUDE.md's "LDAP Authentication" section described the deleted file as the data-plane authenticator. Rewritten, including a warning not to "restore" it on seeing the LDAP wiring that AdminUI legitimately still needs. - infra/README §5b (the GLAuth outage live gate) marked RETIRED with the reason, rather than silently dropped — the harness GLAuth is still used by OTOPCUA_HARNESS_USE_LDAP=1 and is unaffected. - docs/security.md's five-key resilience table cut to the one surviving key, and a new API-key identity section added. Historical archreview/ records are left as-is; they are dated accounts. Host.IntegrationTests 211 tests (was 230, i.e. exactly the 19 removed) with the same 3 pre-existing failures verified against clean master. Host 37, Security 89. Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo |
||
|
Joseph Doherty
|
b815446940 |
refactor(drivers): delete GetMemoryFootprint / FlushOptionalCachesAsync (#525)
Both `IDriver` members had no production call site. Their only consumer was `MemoryTracking`, deleted with the driver-tier recycle machinery in #522, and `GetMemoryFootprint`'s doc-comment claimed "polled every 30s by Core" — which had not been true since v3, even before #522 removed the poller. Owner decision on #525 was (a) delete rather than (b) build the process-wide watchdog that would consume them. The driver-side implementations are in git history and were mostly reusable; what was always missing is the trigger, and a member nobody calls is how the misleading doc-comment survived this long. Removed from `IDriver`, all 12 driver implementations, and every test stub — plus the support code that existed only to feed them, which is the part a member-only deletion would have left behind as dead weight: - MTConnect's three per-entry byte-weight constants, its cached `ProbeModelDataItemCount`, and `CountDataItems` — a full device-tree walk run on every cache fill to populate a number with no remaining reader. - OpcUaClient's `_discoveredNodeCount` field and the discovery-pass write to it. - MQTT's `ApproxBytesPerAuthoredTag`. Two behavioural notes, neither a change in what the software does: - `AbCipTemplateCache` now has no production reset path. Its only caller was AB CIP's flush, which itself had no caller — so nothing changes except how visible the fact is. Recorded on the type, along with the correction that its comment's claim about a device `ReinitializeAsync` also flushing it was never true. A UDT redefined on a live controller keeps its stale shape until the driver instance is rebuilt. - MTConnect's `/probe` cache is still dropped, by `InvalidateProbeModel()` on an Agent restart. That is now `internal` rather than `private` so the tests which used the flush purely to empty the cache drive the real path instead — they now cover a case that actually occurs in production. Several unrelated doc comments cited `FlushOptionalCachesAsync` as the canonical CAS example and are repointed to it. Tests: the ones that existed only to assert on the removed members are deleted (3 OpcUaClient, 3 Galaxy, 2 MQTT, 1 MTConnect, 1 AbCip). Worth noting one was vacuous — `FlushOptionalCachesAsync_resets_footprint_counter` asserted a footprint of 0 before the flush and 0 after with no discovery in between, so it would have passed against a flush that did nothing at all. Full suite green except 7 failures verified identical on master before and after (4 AbLegacy integration with the compose profile down, 3 Host.IntegrationTests), checked by name rather than assumed. Docs corrected where they described this as live: `DriverLifecycle.md` (which carried the #525 placeholder), `drivers/AbCip.md`, `drivers/AbLegacy.md`, and `v2/driver-stability.md` — whose "polls this every 30 s" + per-driver `DriverConfig` memory budget were never built as written and are now banner-ed as design intent rather than description. Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo |
||
|
Joseph Doherty
|
a107a1a8b9 |
feat(mtconnect): host registration + DriverTypeNames const + guard parity (Task 16)
Register the MTConnect factory + probe with the Host, add the DriverTypeNames.MTConnect constant, and keep DriverTypeNamesGuardTests green. - DriverTypeNames: new MTConnect const + appended to All. - DriverFactoryBootstrap: MTConnectDriverFactoryExtensions.Register in Register(...) at the default Tier A (managed HttpClient/XML, in-process; Tier C is the only tier arming process-recycle, wrong here), and TryAddEnumerable(IDriverProbe -> MTConnectDriverProbe) in AddOtOpcUaDriverProbes so the probe reaches admin-only nodes (the admin-pinned Test-Connect singleton) without double-registering on a fused admin,driver node. - Host.csproj: ProjectReference to Driver.MTConnect (required to compile the Register call). - Core.Abstractions.Tests.csproj: ProjectReference to Driver.MTConnect so the guard's bin scan discovers the factory. This and the constant MUST land together -- verified RED (2/4 facts) with the constant alone. - DriverProbeRegistrationTests: MTConnect added to AdminUiDriverTypeKeys; its idempotency fact asserts distinct-probe-count and goes RED without this (and RED if TryAddEnumerable is downgraded to AddSingleton). Reconciles all four "MTConnect" literals onto the one constant: the factory's DriverTypeName, MTConnectDriver.DriverType, and MTConnectDriverProbe.DriverType now all alias DriverTypeNames.MTConnect (no circular reference -- Core.Abstractions is a leaf both driver projects already reference). This is the repo's TwinCat/Focas anti-drift convention. |
||
|
Joseph Doherty
|
6531ec1984 |
feat(mesh): one ClusterClient per Cluster in CentralCommunicationActor (Phase 6)
After the fleet splits into one Akka mesh per application Cluster, a ClusterClientReceptionist serves only its own mesh — so the single fleet-wide client's SendToAll reached only the mesh whose receptionist answered, leaving every other cluster silently on its old configuration. Central now holds one ClusterClient per ClusterId and fans SendToAll across all of them. LoadContactsFromDb selects ClusterId and groups the receptionist contacts per cluster (keeping the per-row TryParse guard and the enabled/non-maintenance filter); ContactsLoaded carries ContactsByCluster; HandleContactsLoaded diffs per cluster (rebuild changed, stop+drop vanished, warn-don't-create on empty); RebuildClient builds a per-cluster client under a clusterId-derived actor name. IMeshClusterClientFactory.Create gained a clusterId parameter so the per-cluster clients get distinct, diagnosable names. ApplyAck handling and the DPS branch are unchanged; the deploy path stays payload-free. Tests: focused unit coverage of the grouping + fan-out (one-client-per-cluster, fan-out SendToAll to every cluster client) via the recording factory double, plus the real two-mesh boundary test extended to central + two separate site meshes proving one dispatch reaches both and a client scoped to one cluster never crosses into another. Red-before-green verified: crippling the fan-out to a single client fails the reaches-both-meshes assertion. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
af9c9d78da |
test(mesh-phase5): 3-port listener reachability + comment fix for the telemetry endpoint
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
8e5090edf3 |
feat(mesh-phase5): tap the 4 telemetry publish seams into the local hub (DPS intact)
Each of the four telemetry publish seams now also emits its DPS payload into the node-local ITelemetryLocalHub (Phase 5), leaving the existing DistributedPubSub publish untouched: - driver-health AkkaDriverHealthPublisher -> TelemetryItem.Health - resilience DriverResilienceStatusPublisherService -> TelemetryItem.Resilience - alerts ScriptedAlarmHostActor + DriverHostActor (native) -> TelemetryItem.Alarm - script-logs VirtualTagActor + DpsScriptLogPublisher -> TelemetryItem.Script DI services take the hub as a required ctor param; actors take an optional nullable hub threaded through their Props and the real spawn sites (DriverHostActor <- ServiceCollectionExtensions; VirtualTagActor/ScriptedAlarmHostActor <- DriverHostActor), so existing test Props constructions keep compiling and simply do not emit. The hub is a no-op until a gRPC client subscribes, so the emit is safe on every node. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
6f21213f70 |
test(mesh-phase4): pin driver-only ConfigDb-free DI graph + fused-node mapper ordering
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
d907160747 |
feat(mesh-phase4): DriverHostActor runs ConfigDb-free (LocalDb alarm store, guarded acks)
Nullable ConfigDb factory; UpsertNodeDeploymentState no-ops when absent (central persists acks from the ApplyAck); scripted-alarm condition state served from the LocalDb store instead of the ConfigDb-backed Ef store. Combines Phase-4 Tasks 4 + 6. Removes the interim dbFactory! cast. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
4f4cdd05ec |
feat(mesh-phase4): LocalDb alarm-condition-state store (replicated, pair-local)
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
a41582ea6a |
feat(mesh-phase4): register ConfigDb only on admin-role nodes
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
ffbcaa93f2 |
test(mesh): real two-Host gRPC artifact-fetch boundary test with wrong-key control
Phase 3 Task 8. Stands a real h2c-only Kestrel listener serving the real DeploymentArtifactGrpcService behind the real ConfigServeAuthInterceptor (in-memory config DB seeded with one sealed deployment), and drives the real GrpcDeploymentArtifactFetcher across it — proving a stream crosses the real boundary, not just that the fetcher's logic is right (Task 4's fakes). Asserts: (1) right key + >256 KB blob reassembles byte-equal and verifies against RevisionHash; (2) FALSIFIABILITY CONTROL — wrong key returns null while the right key succeeds on the SAME server (so the null is the auth gate, not a dead server); (3) unknown id -> null (NotFound-hidden); (4) [dead-port, real-port] failover still returns the bytes. h2c GrpcChannel over http:// works out of the box in .NET 10. Tagged [Trait Category=ArtifactBoundary]. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
4d88f67c0d |
feat(mesh): config-serve auth interceptor + dedicated h2c listener (fail-closed bearer)
Phase 3 Task 3. Central serves the deployment artifact over a dedicated h2c Kestrel listener (ConfigServe:GrpcListenPort), gated by the ConfigServeAuthInterceptor (shared bearer key, FixedTimeEquals, fail-closed, path-scoped to /deployment_artifact.v1.DeploymentArtifactService/). The listener block is merged with the LocalDb-sync listener so a fused admin+driver node re-applies its existing HTTP surface exactly ONCE and adds both dedicated h2c ports on top — double-binding would throw 'address already in use'; zero re-binding would silently unbind the AdminUI behind Traefik. AddGrpc now registers under hasDriver || hasAdmin with both path-scoped interceptors sharing one pipeline. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
5468b79d19 |
test(mesh): real two-ActorSystem ClusterClient boundary test with falsifiability control
Two genuinely separate single-member clusters sharing the ActorSystem name and joined only by ClusterClient, so the mesh boundary is exercised over the wire rather than through an in-process relay: a relay cannot fail for a missing receptionist extension, an unregistered service, a malformed contact path or a serialization break, which are the Phase 2 failure modes. Covers both legs plus the control: * central SendToAll -> node comm actor -> node EventStream * node ApplyAck -> central comm actor, through the node's own client * a node without the receptionist extension receives nothing Sabotage-verified. Suppressing the node-side EventStream publish and the outbound ack Tell reddens one leg each. Restoring the receptionist and the service registration on the control node reddens the control, which is what proves the control's send is live rather than silently misaddressed. The control removes the extension AND the RegisterService call, since resolving the receptionist to register would materialise the extension whose absence is the point; it falsifies "delivery happens without a receptionist boundary", not the extension line alone. Recorded in the test. Also corrects MeshCommActorPathTests: with one node per side, per-node and singleton registration are indistinguishable here, so the property the dropped Task 6 assertion covered belongs to live-gate step 8, not to this class. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
16d598560a |
feat(mesh): register comm actors with the receptionist; pin the wire-contract paths
Phase 2 Task 6. Both comm actors are now spawned and receptionist-registered: CentralCommunicationActor from WithOtOpcUaControlPlaneSingletons, and NodeCommunicationActor from WithOtOpcUaRuntimeActors (before DriverHostActor, because it is the host's ackRouter under ClusterClient mode; it has no dependency of its own on the host, since inbound commands travel via the node's EventStream). Both are plain WithActors registrations, NOT singletons -- a driver node's ClusterClient rotates across contact points and must find a live comm actor at whichever node answers. Both register with the receptionist in BOTH transport modes: an idle registration costs nothing, whereas registering only under ClusterClient would mean flipping the flag on a running fleet needs a restart before anything can be reached, turning a config change back into a deployment. The coordinator ref is resolved lazily per message (Func<IActorRef?> + registry.TryGet) so the comm actor never depends on the order in which Akka.Hosting materialises the singleton proxy relative to this block. Adds MeshCommActorPathTests, which boots the real two-node host and Identify- probes both paths. These strings are the wire contract and nothing else asserts they agree: a rename would compile, pass every unit test, deploy, and deliver nothing, because a ClusterClient send to an unregistered path is dropped silently. Sabotage-verified with an actual rename. DELIBERATELY NOT TESTED, with the reason recorded in the file: that the actors are per-node rather than singletons. Two discriminators were tried and both were invalid -- "the path resolves on both nodes" passes under either shape because a ClusterSingletonManager is also created on every node in the role, and "no /singleton child" is null even for the KNOWN singleton /user/config-publish, so Akka.Hosting does not lay them out that way. A sabotage re-registering the actor via WithSingleton left both green, which is what exposed them. Rather than ship an assertion that cannot fail, the property is left to the Task 8 boundary test, where a send only arrives if the target really is registered. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
7654f24dab |
test(harness): serialize Host.IntegrationTests; drop the reconciler's unused singleton proxy
Closes the last branch-only failures. Measured across five full-suite runs on this branch versus two on master in a clean worktree: master failed only AbCip in this assembly (2/2), while the branch additionally failed one or two E2E deploy tests, rotating between DeployHappyPath, DriverReconnect and EquipmentNamespaceMaterialization (4/4). Each passed in isolation, and the assembly on its own was 195/201 throughout — the failures only appeared under full-suite CPU contention. Cause is contention, not correctness. Every test here builds a real two-node Akka cluster — two Kestrel hosts, two ActorSystems with remoting, six admin singletons, an EF context, a deploy pipeline — and xUnit ran them concurrently with each other and with every other assembly. That was survivable while the coordinator sealed instantly on an empty expected-ack set; now that it waits for a real ack from every configured node, these tests measure an end-to-end round-trip and starve. Serialising this one assembly makes them deterministic: 195/201 with only AbCip_Green_AgainstSim, and the full-solution failure set is now IDENTICAL to master's — 7 shared, zero branch-only. Cost is wall-clock for this assembly, 40s -> 3m35s; it is a handful of heavyweight E2E tests, not a broad unit suite, and a widened timeout alone did not fix it (tried first, at 45s). Also drops createProxyToo for the address reconciler. Nothing resolves ClusterNodeAddressReconcilerKey from the registry — the actor only reads cluster state and logs — so the proxy was a second actor per node hunting for a singleton nobody addresses. Kept because it removes dead machinery, not because it fixed the flakiness; measured on its own, it did not. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
0b7c53f64f |
test(harness): widen the deploy-seal timeout to match a real two-node deploy
Comparing full-suite runs on this branch against master in a clean worktree: both fail 11 tests, 9 identical. Master's two extras are load-flaky unit tests (AbCip Probe_loops, Galaxy EventPumpBoundedChannel); this branch's two extras were both Host.IntegrationTests deploy-path tests — the area this change re-times — so they are attributable here rather than to background flakiness. Cause is margin, not correctness. These waits used to observe a coordinator that sealed instantly on an empty expected-ack set; they now observe a real ApplyAck round-trip from every configured node. 15s was enormous margin against "instant" and thin against the real thing, so they failed only under full-suite CPU contention. Hoisted to TwoNodeClusterHarness.DeploySealTimeout (45s) so the reason is recorded once rather than as four unexplained numbers. DriverReconnectE2eTests is deliberately left alone: it seeds both ClusterNode rows itself and unconditionally, so its expected-ack set is identical before and after this change. Widening its timeout would be papering over a flake this change did not cause. Host.IntegrationTests 195/201; sole failure AbCip_Green_AgainstSim, verified failing on master. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
14746f2995 |
test(harness): seed ClusterNode rows in in-memory mode; retarget the failover deploy test
The full-suite run surfaced two load-dependent failures that were green in isolation. Root cause: SeedDefaultClusterAsync no-opped unless OTOPCUA_HARNESS_USE_SQL=1, on the reasoning that "the in-memory provider ignores FK constraints, so deploy E2E tests pass without it". Phase 1 invalidated that. With no ClusterNode rows the coordinator's expected-ack set is empty, so it seals IMMEDIATELY. "Wait for Sealed" therefore stopped implying "every node has applied" — and because DriverHostActor.UpsertNodeDeploymentState writes each node's row on its own schedule, every test counting those rows after a seal became a race. Reproducibly green alone, red under suite load, and it surfaced in a different test each run, which is what made it look like flakiness rather than a contract change. Seeding in both modes restores the invariant those tests were written against and is closer to production either way: a real fleet always has these rows, because the FK requires them. Deployment_started_with_node_b_down_seals_with_one_node_state asserted the behaviour Phase 1 deliberately removed — its own comment documented membership snapshotting, i.e. sealing green while a configured node never received the deployment. Replaced by two tests covering the new contract: a stopped node is still expected (both state rows exist, B still Applying, deployment does not seal), and MaintenanceMode is what makes it seal with one. The new "does not seal" test asserts both rows exist rather than only the absence of a seal, so it cannot pass against a coordinator that simply died. Host.IntegrationTests 195/201, sole remaining failure AbCip_Green_AgainstSim — verified failing on master in a clean worktree, so pre-existing fixture baseline, not a regression. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
8838e92d0a |
fix(tests): StopNodeBAsync also has to stop the node, not just dispose it
Completes
|
||
|
Joseph Doherty
|
835fc08c3f |
fix(tests): stop the integration host from ballooning to 30 GB
User-reported: the integration tests leave instances behind and take 18+ GB.
Measured before changing anything — a SINGLE Host.IntegrationTests process peaked
at 30,153 MB RSS across its ~200 tests.
The cause was not a leak and not parallelism. Referencing the Host drags in the
ASP.NET Core framework reference, which turns Server GC on by default: one heap
per core, tuned for throughput and deliberately reluctant to hand memory back.
Correct for a server, wrong for a test host. Workstation GC takes the same suite
from 30,153 MB to 6,717 MB — 78% less — with no change in runtime (45s).
Two hypotheses were measured and rejected on the way, recorded here so nobody
re-runs the experiment: capping xunit maxParallelThreads to 4 cost 29% wall-clock
and saved nothing (30,153 -> 31,115 MB, i.e. noise), and it turns out a single
process was doing all of it. Reverted.
The harness fix is real but was worth only 4% on its own: TwoNodeClusterHarness
disposed each node without stopping it first, and IHost.DisposeAsync only disposes
the service provider — it never invokes IHostedService.StopAsync, which is where
Akka.Hosting terminates the ActorSystem. So every test left two live ActorSystems,
remoting transports and dispatcher pools included, rooted for the process
lifetime. The class doc-comment asserted the opposite ("DisposeAsync, which runs
CoordinatedShutdown"); it was wrong, and StopNodeBAsync inherited the same bug on
a path failover tests depend on.
The payoff is larger than the memory number. Two failures I had classified as
known-flaky baseline noise now pass consistently:
RoslynVirtualTagEvaluatorTests.Evaluate_racing_ClearCompiledScripts_never_fails_with_disposed
and ContinuousHistorizationRecorderTests.Retry_after_writer_failure_eventually_acks.
They were never flaky — they were starved. Full solution: peak summed test-process
RSS 15.2 GB, zero new failures, two fewer.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
|
||
|
Joseph Doherty
|
f9f1b8fcee |
fix(localdb): phase-2 live gate — 4 production defects found and fixed
Gate record: docs/plans/2026-07-20-localdb-phase2-live-gate.md. Checks 1, 2, 5, 6 pass. Checks 3 and 4 are NOT satisfied — the defect they were meant to confirm turned out to be the opposite of what the plan assumed. Three defects crash-looped every driver node before check 1 could even run: 1. An empty ServerHistorian:ApiKey kills the host. ServerHistorianOptions- Validator exists to turn exactly that class of failure into a named OptionsValidationException, but its documented fail tier explicitly excluded ApiKey on the reasoning that a keyless client "degrades — the gateway rejects calls". It does not: the client validates its own options at construction, so the process dies during Akka startup and never makes a call. 2. UseTls disagreeing with the endpoint scheme kills the host too, in both directions (both messages confirmed in the shipped client assembly). Moving an endpoint from https to http without clearing UseTls is an ordinary migration slip. 3. Plaintext h2c was UNREACHABLE. HistorianGatewayClientAdapter forwarded the TLS-only options unconditionally, and AllowUntrustedServerCertificate defaults to false, so it always sent RequireCertificateValidation=true — which the client rejects outright when UseTls=false. Every http:// deployment crashed, though the scheme is documented as the supported way to select h2c, and the only workaround was to assert a certificate posture for a connection that has no certificate. The fourth was the blocker, and it is Phase 2's own: 4. The drain gate deferred to a Primary that cannot deliver. Redundancy roles are elected CLUSTER-WIDE; the alarm queue is PAIR-LOCAL. On the rig the elected driver Primary is central-1 — it carries the driver Akka role, replicates nobody's LocalDb and does not even run the alarm historian — so every driver node logged "Historian drain suspended", including the two site-b nodes that have no peer at all. Nothing drained anywhere, where before Phase 2 it drained fine. The cost is not a duplicate; it is the buffer growing to the capacity wall and evicting the audit trail it exists to protect. Fixed in three layers: a separate ShouldDrainAlarmHistory policy (unknown role drains; the two gates now deliberately disagree, and a test pins that); peer- host matching in DriverHostActor so a node stands down only for a Primary holding its rows; and AddAlarmHistorian short-circuiting the gate when replication is unconfigured — testing BOTH Replication:PeerAddress and SyncListenPort, since only the dialing half sets the former while both halves share the queue. Every one of these follows from the asymmetry: a false allow costs a duplicate row, which at-least-once delivery already accepts and payload-hash ids collapse; a false deny loses data silently. A third vacuous test, caught by the same delete-the-guard discipline: the role-view tests stayed green with the guard removed, because AwaitAssert polls until an assertion passes and the assertion was "reads open" — which is the SEEDED value, satisfied at the first poll before the actor processed anything. They now assert the sequence of published values through a recording view; the control then goes red for exactly the cases that matter. Migration evidence: 11 legacy rows across two deliberately overlapping files converged to exactly 9 identical rows on both nodes, proving D-6's payload-hash identity on real nodes rather than in a fixture. Open design fork, recorded in the gate doc rather than decided here: a pair cannot currently identify its own Primary, so both halves drain. Safe in every topology — nothing loses data — but the gate's de-duplication benefit is unrealised until roles are scoped per pair. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
271fcc4e15 |
test(localdb): alarm S&F convergence scenarios
Two driver nodes replicating the alarm buffer over the real loopback h2c
transport, through the real fail-closed interceptor. This is the test the
phase exists to pass: schema, registration, sink rewire and drain gate can
all be individually green while a pair still fails to share the undelivered
alarm history that is the whole point of moving the queue.
Rows are written through the production sink rather than hand-rolled
INSERTs, so the id derivation, column set and delete-on-ack semantics under
test are the ones production uses. The scenarios cover a buffered burst
converging with the oplog draining to zero, delivered rows being removed from
BOTH nodes as tombstones (the no-redeliver-after-failover property), writes
during a transport outage surviving the rejoin, and the same event accepted
on both nodes collapsing to one row.
The positive control found a weak assertion. With
RegisterReplicated("alarm_sf_events") commented out, three of the four went
red immediately -- but the same-event-on-both-nodes scenario still PASSED,
because with replication off each node trivially holds its own single copy
and "one row on each" is satisfied by two databases that never spoke. It now
also enqueues a distinct event on B and requires both nodes to hold two rows,
which cannot be satisfied without convergence; it goes red under the control
like the others. Control restored, all four green.
Also updates the second exact-set replicated-tables pin, in LocalDbWiringTests.
The plan predicted these pins would go red here, and that is them working:
one of them is asserted through the full driver DI graph rather than the
OnReady callback, so it catches a registration that exists in the callback
but never reaches a running host.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
|
||
|
Joseph Doherty
|
af545efdf5 |
feat(localdb): one-time alarm-historian.db migrator
Copies the pre-consolidation store-and-forward queue into the consolidated
database on first boot, then renames the legacy file aside. Rows are in that
file precisely because the historian could not be reached, so dropping them
on upgrade would discard exactly the alarm audit trail the queue exists to
protect.
Runs last in OnReady, after every RegisterReplicated call. It is the only
thing in OnReady that writes rows, and capture is trigger-based: a migration
that ran before registration would recover the backlog locally and never
replicate a line of it, silently and permanently.
Ids are derived from the payload rather than the plan's mig-{node}-{legacyId}
scheme. Node-prefixing solves the collision the legacy AUTOINCREMENT key
would cause -- node A's row 7 and node B's row 7 are different alarms -- but
it preserves a duplication that should be collapsed instead. A warm pair's
two legacy files OVERLAP: HistorianAdapterActor default-writes while its
redundancy role is unknown, so both nodes accepted the same transitions
during every boot window. Prefixed ids would carry those duplicates into the
merged buffer forever; equal-payload ids converge them. The same property
makes a crash between commit and rename harmless under INSERT OR IGNORE.
OnReady now takes IConfiguration rather than offering an overload that skips
the migration. A wiring mistake that silently discarded a node's undelivered
alarm history is not a mistake worth making possible.
The copy is restricted to the columns the legacy table actually has. Naming
a column an older build never wrote throws "no such column", which would
discard every row in the table rather than the one field.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
|
||
|
Joseph Doherty
|
8a9cb40a72 |
feat(localdb): alarm_sf_events replicated table
Adds the alarm store-and-forward buffer to the consolidated LocalDb file and registers it for replication, so a node that dies with undelivered alarm history no longer takes it to the grave. The table keeps the legacy queue's shape rather than the status column the plan sketched. An acknowledged row is deleted, not marked delivered: that needs no second sweeper to keep the table bounded, leaves the capacity semantics untouched, and the replication engine carries the delete as a tombstone so the peer drops its copy anyway -- which is what the status column was for. last_error is retained because it is the only operator-facing record of why a row was dead-lettered. The primary key is app-minted TEXT. The legacy AUTOINCREMENT RowId cannot replicate under last-writer-wins: two nodes would independently allocate rowid 7 to different alarms and silently overwrite each other. The drain index therefore orders by enqueued_at_utc rather than insertion order, with id as a tiebreak so the ordering is total. Tables are created unconditionally, independent of AlarmHistorian:Enabled. An empty registered table costs three triggers; creating it lazily would mean a node that enables the historian later writes rows before its capture triggers exist, which is exactly the silent-loss shape OnReady's ordering comment warns about. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
232bff5df7 |
fix(localdb): close both Phase 1 live-gate follow-ups
1. Wiped-node back-fill (live gate check 4). Fixed upstream in ZB.MOM.WW.LocalDb 0.1.2 and pinned here. The gate's diagnosis was slightly off: the oplog cap was not the gate. Snapshot detection measured the peer's gap against the oldest surviving oplog row and read an empty oplog as "no gap possible" — and an empty oplog is the steady state of a converged pair, since ack-pruning deletes everything the peer confirmed. The healthy state was the one state that could not heal a wiped peer. Now measured against last_acked_seq when the oplog is empty. Pinned at this level too, not just the library's: the pair harness grows a WipePassiveAsync (a NEW database — a rebuilt node comes back with a new node id and a zero watermark), and the new scenario asserts the emptied oplog as its precondition before wiping, then requires the deployment artifact to come back byte-identical with no new deploy. Verified RED against the pinned 0.1.1 (times out waiting for back-fill) and green on 0.1.2. 2. Oplog growth on default-OFF nodes (live gate check 8), fixed at the source: StoreAsync now skips entirely when the pointer already names this deployment/revision, the SHA matches the bytes, and the expected chunk count is present. Re-caching an artifact the node already holds — every restart's boot-from-cache, every RestoreApplied — writes nothing and mints no oplog rows, where before it cost a delete plus an insert per chunk plus a pointer update, all identical to what was already there. Identity is over the bytes and the chunks are counted, both deliberately: a re-composed artifact can carry the same ids with different bytes, and a pointer can name a deployment whose chunks are missing, where skipping would make an unreadable cache permanent. Both guards verified RED against a naive pointer-only skip. The gate's stated bound was also wrong and is corrected in the doc: growth was never headed for the 1M row cap. AddZbLocalDbReplication is registered unconditionally, so MaintenanceBackgroundService runs on default-OFF nodes and the 7-day MaxOplogAge cap prunes. Runtime.Tests 412/0/31, Host.IntegrationTests LocalDb 45/45. Pre-existing and unrelated: AbCip_Green_AgainstSim (needs the AB CIP docker fixture, red on a clean master too) and a flaky Roslyn race test (green 2/2 in isolation). Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
ce9fa07ff8 |
fix(localdb): re-bind ASPNETCORE_HTTP_PORTS surface, not just ASPNETCORE_URLS
The Kestrel re-bind read only urls/ASPNETCORE_URLS. The aspnet:8.0+ base images set ASPNETCORE_HTTP_PORTS=8080 (all interfaces) as the CONTAINER default in place of ASPNETCORE_URLS, so a driver node that never sets URLS fell straight through to Kestrel's localhost:5000 default — silently moving its health/metrics surface to loopback when the sync listener took over Kestrel. Now falls through URLS → HTTP_PORTS/HTTPS_PORTS (wildcard, all-interfaces) → localhost:5000, mirroring Kestrel's own source precedence. Found on the docker-dev live gate (central nodes were fine — they set ASPNETCORE_URLS=9000 explicitly). |
||
|
Joseph Doherty
|
afa5be713a |
test(localdb): 2-node convergence harness + scenarios
Two driver nodes replicate the deployment-artifact cache over a real loopback h2c transport through the real fail-closed LocalDbSyncAuthInterceptor; both DBs init via the production LocalDbSetup.OnReady. Scenarios: byte-identical convergence with matching pointer HLC+origin, retention prune reaching B as tombstones, writes-while-transport-down surviving rejoin, and wrong-key non-convergence with a matching-key positive control. DoD positive control: commenting out both RegisterReplicated calls turns all four scenarios red (verified locally, restored before commit). |
||
|
Joseph Doherty
|
5a21be0a62 |
test(localdb): DI pins over the real host composition methods
Builds a container from the same AddOtOpcUa* methods Program.cs's driver branch calls (AddOtOpcUaLocalDb / AddGrpc / AddOtOpcUaHealth / AddOtOpcUaObservability) and resolves through it - DI extensions have shipped inert three times in this family. Does not boot the real host: WebApplicationFactory<Program> would start hosted services (Akka, OPC UA server, ValidateOnStart) needing SQL + a mesh, per D-5. The seam under test is DI resolution, so the container is built and resolved without starting. Five pins: ILocalDb singleton + exact replicated-table set; IDeploymentArtifactCache -> concrete; default-OFF ISyncStatus; localdb-replication health check registered and constructible; admin-only graph has NO ILocalDb and does not demand LocalDb:Path. |
||
|
Joseph Doherty
|
b350fba233 |
feat(localdb): replication health check + meter export
LocalDbReplicationHealthCheck reports Healthy when replication is default-OFF or LocalDb is absent (admin-only graphs) - so a plain node is never degraded by it - Degraded when configured-but-disconnected, connected-with-unknown-backlog (a failed oplog poll must not read as zero), or connected-with-backlog past the threshold. Never Unhealthy: a replication fault does not stop the node serving its address space. Pure decision core is table-tested; registered unconditionally via a factory so AddOtOpcUaHealth keeps its no-arg signature and resolves ISyncStatus optionally. Adds LocalDbMetrics.MeterName to the observability allowlist. o.Meters is a strict allowlist with no wildcard, so the localdb.sync.* / localdb.oplog.depth series were otherwise silently absent from /metrics - the omission a ScadaBridge live gate caught. |
||
|
Joseph Doherty
|
b9ddf20edd | feat(localdb): dedicated h2c listener + MapZbLocalDbSync gated on LocalDb:SyncListenPort | ||
|
Joseph Doherty
|
3d29be834e | feat(localdb): fail-closed bearer interceptor for the sync endpoint | ||
|
Joseph Doherty
|
3b65c24bfb | feat(localdb): deployment-cache schema + OnReady registration | ||
|
Joseph Doherty
|
2254ae3dea |
fix(secrets): consume Secrets 0.2.2 - clustered-secrets DI deadlock fixed upstream
Bumps the four ZB.MOM.WW.Secrets pins 0.2.1 -> 0.2.2 (closes the OtOpcUa side of scadaproj#1, tracked here as #482). 0.2.1's Akka replicator deadlocked any hosted process at startup when Secrets:Replication:Enabled was true: the package's DI graph closed a circular singleton dependency through factory lambdas (store decorator -> replicator -> actor provider -> cache invalidator -> resolver -> store), which MS.DI's StackGuard turns into a silent cross-thread call-site-lock deadlock. 0.2.2 defers the invalidator edge to first eviction. The flag stays default-false; enabling remains a per-environment decision. The_startup_hook_actually_creates_the_replication_actor is now a real test: SecretReplicationStarter's docs had promised it since the adoption, and the upstream fix finally makes a provider-based resolve runnable - container built exactly as the host does, hook started under a watchdog, replication actor proven to exist by ActorSelection on a self-joined single-node cluster (no TestKit needed, which matters because Akka.TestKit.Xunit2 is xunit-v2-only and this project is on xunit.v3). Also corrects the stale rationale that blamed the old hang on DistributedPubSub needing a joined cluster - the actor constructor was never reached; it was the DI cycle. Verified: SecretsReplicationRegistrationTests 8/8 on the 0.2.2 feed packages; full slnx build 0 errors; the 2-node Akka live convergence gate re-run against the published 0.2.2 packages passes 6/6 (write->peer, tombstone propagation without resurrection, delete visibility through the resolver cache, reverse direction, wrong-KEK fail-closed). Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts |
||
|
Joseph Doherty
|
1ccc237cb6 |
docs(test): record why these assert descriptors, not a built provider
Akka.TestKit is the family convention for non-end-to-end Akka testing and would be the right tool here, but Akka.TestKit.Xunit2 is xunit-v2-only and this project is on xunit.v3 - adding it produces CS0433 type conflicts. Directory.Packages.props already documents the same constraint for the three projects deliberately held on xunit v2 for TestKit. Revisit when Akka ships an xunit.v3 TestKit. |
||
|
Joseph Doherty
|
b3d1a26f38 |
fix(secrets): consume Secrets 0.2.1 - Akka replication was inert in 0.2.0
0.2.0's AddZbSecretsAkkaReplication never bound its own ISecretReplicator: it called AddZbSecrets first, which TryAdds NoOpSecretReplicator, so the package's own TryAdd was silently discarded. Replication published into a no-op sink and no actor was ever spawned - no exception, no log line. Found by the Task 6 wiring work, which is the first code that ever built a container around that extension. Fixed upstream in 0.2.1. Test consequence worth recording: the replication-enabled registration tests previously passed against a plain ActorSystem only BECAUSE of that bug - the no-op never touched Akka. With 0.2.1 they resolve a real replicator, which spawns the actor, whose PreStart needs DistributedPubSub and therefore a joined cluster, so provider-based assertions hang. Standing a real cluster up inside this test assembly was attempted and did not work, so these assertions are now made against the ServiceCollection, which is precisely where the defect lives. Actor creation and convergence remain covered upstream by the library's TwoNodeClusterReplication tests against a genuine 2-node cluster. 7 registration tests pass in 23ms. Full solution builds 0 errors; no warnings originate from these files. |
||
|
Joseph Doherty
|
3336ec08c7 |
feat(secrets): opt-in Akka cluster secret replication (default OFF; upstream blocker documented)
Routes the host's secrets registration through a new AddOtOpcUaSecrets extension that gates the ISecretStore implementation on Secrets:Replication:Enabled. Opt-in gate (default FALSE) This call decides which ISecretStore every node resolves — including driver-role nodes with no auth/AdminUI, where a wrong store surfaces as drivers failing to open sessions rather than as a failing test. With the flag false the wiring is the pre-existing AddZbSecrets(config, "Secrets") call, unchanged, so current behavior is byte-identical. With it true, AddZbSecretsAkkaReplication replaces that call (it invokes AddZbSecrets internally; calling both would double-register). Extracted to a named extension specifically so the registration is testable: Program.cs is top-level statements and cannot be exercised by a container test, which is how a "registered but never resolvable" defect ships unnoticed. Serializer HOCON AkkaSecretsReplication.SerializationConfig is merged into the ActorSystem config inside the AddAkka configurator, conditionally on the same gate — a non-replicating node carries no bindings for messages it will never see. Merged via AddHocon(..., HoconAddMode.Append), Akka.Hosting's fallback merge and the same mode the existing base-config merge uses; a raw Config.WithFallback would fight the builder's own assembly. Lazy-actor mitigation The replication actor is created lazily on first ISecretStore resolution, so a node that never touches a secret would never announce a manifest and would silently never converge. SecretReplicationStarter (IHostedService) resolves the store once at startup to make participation unconditional. KNOWN BLOCKER — replication is currently NON-FUNCTIONAL; do not enable ZB.MOM.WW.Secrets.Replicator.AkkaDotNet 0.2.0 never binds its own ISecretReplicator. AddZbSecretsAkkaReplication calls AddZbSecrets FIRST, which does TryAddSingleton<ISecretReplicator, NoOpSecretReplicator>(); the package's own TryAddSingleton<ISecretReplicator>(AkkaSecretReplicator) that follows is therefore a no-op. Verified empirically in a built container: with Enabled=true, ISecretReplicator resolves to NoOpSecretReplicator, so ReplicatingSecretStore publishes into a sink and no actor is ever spawned. Consequence: the startup hook cannot create the actor, and the test asserting it does is committed Skipped with the evidence. Not worked around here — the fix belongs upstream (AddSingleton, or register before calling AddZbSecrets). Because the flag defaults false, this commit is inert in production. Tests: SecretsReplicationRegistrationTests (new) — disabled path resolves plain SqliteSecretStore and needs no ActorSystem; enabled path resolves ReplicatingSecretStore AND the undecorated concrete SqliteSecretStore the decorator is built from (the exact registration gap that shipped once); startup hook registered only when enabled. Red before wiring (4 assertion failures), green after: 6 pass, 1 skipped (blocker above). Build: 861 warnings / 0 errors, unchanged from baseline (full --no-incremental A/B). Host.IntegrationTests: 123 pass, 6 skip, 1 fail — AbCip_Green_AgainstSim, verified pre-existing on the stashed tree (fixture-gated). Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts |
||
|
Joseph Doherty
|
872cf7e37a |
test(secrets): register ISecretResolver in driver-factory resilience tests (Task 10 fix)
The full-suite gate caught a real regression: AddOtOpcUaDriverFactories' registration now resolves ISecretResolver (for Galaxy/OpcUaClient secret: refs, Tasks 7-8), so the two ResilienceInvokerFactoryRegistrationTests that build a minimal container and resolve the invoker factory threw 'No service for ISecretResolver'. The real host always registers it via AddZbSecrets (unconditional, Task 3), and GetRequiredService correctly fails-fast for a genuinely misconfigured host — so the fix is to complete the test container with a stub resolver, matching production composition. Test-only. |
||
|
Joseph Doherty
|
1badc10445 |
test(v3-batch4): add Calculation to the driver-probe idempotency key set
Pre-existing stale test surfaced by Batch 4's full gate (Batch 2/3 gates were live-/run and never ran Host.IntegrationTests): Batch 2 registered CalculationProbe in DriverFactoryBootstrap (9 probes) but AddOtOpcUaDriverProbes_is_idempotent's AdminUiDriverTypeKeys still listed 8, so distinctTypes(9) != Length(8). Calculation is a real DriverTypeNames.Calculation pseudo-driver with a probe; add it and refresh the now-stale *DriverPage.razor comment (that routed flow retired in Batch 2). Not a Batch-4 code change — a test-correctness fix. Claude-Session: https://claude.ai/code/session_01LVneM3eh1UtJxEisFXgmox |
||
|
Joseph Doherty
|
b8208b3312 |
test+docs(v3-batch4-wp5): 2-node dual-namespace harness tests + address-space docs
Tests: - OpcUaServer.IntegrationTests/DualNamespaceAddressSpaceTests.cs (NEW, over-the-wire, offline-safe): both namespace URIs registered + distinct; Raw + UNS subtrees browse and read; UNS variable Organizes-references its raw node; single-source fan-out parity (identical value/quality/timestamp on both NodeIds); HistoryRead via either NodeId -> GoodNoData under the shared tagname; WriteOperate gate symmetric across both NodeIds. - Host.IntegrationTests/EquipmentNamespaceMaterializationTests.cs (extended): full deploy -> persisted-artifact -> ParseComposition round-trip carrying both realms, sealing across the redundant 2-node cluster (redundancy non-interference). In-memory harness, offline. Docs (dual-namespace reality): - CLAUDE.md: new "v3 OPC UA Address Space (Batch 4)" section + Batch-4 testing paragraph. - docs/Uns.md: address-space projection (two namespaces, Organizes edge, effective-name leaf). - docs/Historian.md: dual-registration (both NodeIds -> one tagname); updated CLI examples. - docs/ScriptedAlarms.md + docs/AlarmTracking.md: multi-notifier fan-out, ConditionId=RawPath. - docs/ScriptEditor.md: dual-namespace clarification (script tag-path semantics unchanged). Claude-Session: https://claude.ai/code/session_01LVneM3eh1UtJxEisFXgmox |
||
|
Joseph Doherty
|
2e0743ad25 |
fix(v3-batch4-wp3): realm-qualified write routing + dormant discovery guard + self-correction/byte-parity tests (Wave B review H1/M1/M2/L1/L3)
H1 (HIGH): write-routing key now (AddressSpaceRealm, bareId), not bare-only. A raw s=<RawPath> and a UNS s=<Area/Line/Equip/Eff> can collide as bare strings; the bare-only key let a colliding raw+UNS pair route to the WRONG driver ref (last-writer-wins). The realm the node manager resolves (RealmOf) is now threaded through IOpcUaNodeWriteGateway.WriteAsync -> RouteNodeWrite -> _driverRefByNodeId keyed by (realm, bareId). New regression test: Colliding_raw_and_uns_bare_ids_route_to_their_own_driver_by_realm. M1 (MEDIUM): discovered-node injection made coherently DORMANT. HandleDiscoveredNodes hard-short-circuits (single enforcement point; _discoveredByDriver never populates so the re-inject tail is inert too), with a clear log pointing at the /raw browse-commit flow. New pin: Discovered_nodes_are_ignored_dormant_in_v3; the 16+2 v2 injection scenarios re-pointed to an accurate skip reason (DiscoveryInjectionDormantV3). M2 (MEDIUM): realm-qualified dual-node self-correction tests — Failed_uns_write_reverts_uns_node_and_leaves_raw_node_untouched + Raw_realm_revert_reverts_raw_node_only (the second fails if the realm is dropped). L1: removed the = AddressSpaceRealm.Uns defaults from the consequential node-manager mutation methods (WriteValue/WriteAlarmCondition/MaterialiseAlarmCondition/ EnsureFolder/EnsureVariable/UpdateFolderDisplayName/UpdateTagAttributes/ RaiseNodesAddedModelChange/Remove*/RevertOptimisticWriteIfNeeded) + the AttributeValueUpdate/AlarmStateUpdate records, so the compiler forces explicit realm; read-only accessors + internal builders retain their defaults. L3: fixed the stale VirtualTagHostActor class comment (V3NodeIds.Uns, not the retired EquipmentNodeIds.Variable). Also: DeploymentArtifactRawUnsParityTests — Raw/UNS node-set byte-parity round-trip between AddressSpaceComposer.Compose and DeploymentArtifact.ParseComposition. Claude-Session: https://claude.ai/code/session_01LVneM3eh1UtJxEisFXgmox |
||
|
Joseph Doherty
|
f4f3e17e3e |
feat(v3-batch4-wp1): composer/planner emit both Raw + UNS subtrees
Un-darken the address-space composition for the v3 dual namespace. Composer (AddressSpaceComposer): - Raw subtree: RawContainerNode (Folder/Driver/Device/TagGroup as Object/Folder nodes, keyed s=<RawPath>, parent-before-child) + RawTagPlan (raw tag Variables keyed (realm=Raw, s=<RawPath>) carrying DataType / writable / historize+historian tagname / array shape). Native-alarm intent attaches at the RAW tag (ConditionId = RawPath) with the list of referencing equipment UNS paths (one alarm at the raw tag, not one per equipment). - UNS subtree: UnsReferenceVariable per UnsTagReference, keyed (realm=Uns, s=<Area>/<Line>/<Equipment>/<EffectiveName>), carrying its backing RawPath (Organizes UNS->Raw target + fan-out) and inheriting DataType/AccessLevel from the backing raw tag. Effective name = DisplayNameOverride else backing raw Tag.Name. - All RawPaths flow through one shared RawTopology (RawPathResolver + memoised container paths), byte-parity with EquipmentReferenceMap. - Every new node carries an AddressSpaceRealm (so WP3's applier picks the namespace at the call site). Additive/defaulted model changes only — the un-migrated AddressSpaceApplier still compiles. Planner (AddressSpacePlanner.Compute) + AddressSpacePlan: - Per-realm diff sets: RawContainers + RawTags keyed by RawPath (NodeId) so a rename = remove+add; UnsReferenceVariables keyed by the stable UnsTagReferenceId so a backing-tag rename re-points (BackingRawPath moves, NodeId stable) and a display-name-override edit is UNS-only. - PINNED: raw-tag rename = remove+add in Raw AND re-point in UNS. Classifier folds the new sets in (adds->PureAdd, removes->PureRemove, changed->Rebuild safe default). Tests: AddressSpaceComposerDualNamespaceTests (7), AddressSpacePlannerDualNamespaceTests (8, incl. the rename pin), AddressSpaceChangeClassifierDualNamespaceTests (8); reactivated EquipmentNamespaceMaterializationTests' Batch-4-pending case, re-authored to drive the composer over the seeded config (the artifact-decode seam is WP3). Claude-Session: https://claude.ai/code/session_01LVneM3eh1UtJxEisFXgmox |
||
|
Joseph Doherty
|
2c7fc65bae |
test(v3): migrate OpcUaServer.Tests + Host.IntegrationTests to v3 dark address space
Migrate the server composer/applier + 2-node harness tests off the retired
Namespace/NamespaceKind + equipment→driver/device binding schema onto the v3
greenfield shape (raw-only Tag; DriverInstance-RawFolderId; equipment via UnsLine).
OpcUaServer.Tests (134 build errors -> green, 335 pass / 4 skip):
- LIVE (migrated, passing): VirtualTag-historize + ScriptedAlarm composition,
composer purity/hierarchy, applier MaterialiseHierarchy + MaterialiseEquipmentTags
idempotency (hand-fed applier, not dark).
- SKIP dark-until-Batch-4: {{equip}} token expansion, applier equipment-namespace
E2E (equipment-tag variable materialization).
- RETIRE-with-comment / skip: alias-tag (Tag.EquipmentId binding gone) + device-host
equipment binding (architecturally removed).
- Add local DarkAddressSpaceReasons Skip-reason constants (mirrors Runtime.Tests).
Host.IntegrationTests (green, in-memory): drop deleted-Namespace seeds from
MultiClusterScoping + DriverReconnect; reshape EquipmentNamespaceMaterialization to
assert the DARK contract (folder nodes decode, EquipmentTags empty despite a seeded
v3 raw tag) with the full raw-tag->EquipmentTag materialization kept as a
Batch-4-pending Skip.
Tests-only; no src/ changes.
|
||
|
Joseph Doherty
|
b0eb653bad |
fix(redundancy): 2-node SBR exit-and-rejoin recovery — watchdog + restart supervision + both-node seeds (#459)
Corrects the #459 finding. 2-node keep-oldest recovery works fine (the ScadaBridge sister project proves it); OtOpcUa was missing the supervision pieces that make it automatic, and docs/Redundancy.md wrongly claimed in-place oldest-crash failover. Mechanism (confirmed on a 2-container rig + by decompiling Akka KeepOldest.OldestDecision): on an OLDEST-node crash keep-oldest downs the LONE survivor (DownReachable including myself) — down-if-alone can't rescue a lone survivor (its branch needs >=2 survivors). Recovery is exit-and-rejoin: run-coordinated-shutdown-when-down terminates the node and the service supervisor restarts it. My earlier 'total outage' was a docker-dev artifact (no restart policy); production Install-Services.ps1 already has sc.exe failure restart. Changes (ScadaBridge parity): - ActorSystemTerminationWatchdog (Host, registered after AddAkka): watches ActorSystem.WhenTerminated and on an unexpected self-down calls StopApplication so the process exits (supervisor restarts it) instead of idling with a dead actor system. Distinguishes graceful shutdown via _stopRequested + ApplicationStopping. 3 unit tests. - docker-dev: restart: unless-stopped on the host anchor (models production supervision) + both redundancy peers in SeedNodes so a restarted node re-forms via either peer. - docs/Redundancy.md: rewrote the split-brain recovery section — younger-loss = in-place fast failover; oldest-loss = exit-and-rejoin under supervision (not in-place); the three requirements (supervisor + watchdog + both-node seeds); flagged HardKillFailoverTests as non-representative (Transport.Shutdown, not a real crash). Instant in-place takeover on ANY single loss needs 3+ members. Cluster.Tests 29/29 (SBR guards), watchdog tests 3/3, full solution builds. Live re-verify of the watchdog image pending (host docker disk full). |
||
|
Joseph Doherty
|
4d2dcff5df |
test(ldap): live GLAuth outage/recovery gate (R2-08 03/S2)
Closes the R2-08 S2 live leg. The offline LdapAuthResilienceTests prove the directory-outage circuit with fakes + an unroutable-host blackhole; this drives the SAME circuit through the production auth path (OtOpcUaLdapAuthService + LdapOpcUaUserAuthenticator) against a REAL GLAuth that is paused mid-run and unpaused — and doubles as the first end-to-end verification that the PR #451 GLAuth swap (retired bitnami/openldap) binds correctly. Triple-gated (GLAUTH_LIVE_HOST + GLAUTH_OUTAGE_START_CMD/STOP_CMD), skips clean offline (verified: 1 skipped, 8 ms). try/finally always unpauses GLAuth. Verified GREEN against 10.100.0.35:3894 (12 s): healthy binds (alice+bob, mapped roles) -> docker pause -> 3 bounded failures -> circuit opens -> sub-second "unavailable" fast-deny -> docker unpause -> half-open probe closes -> binds succeed again. A boundary AuthTimeout feeds the circuit (authenticator :123-130), so the docker-pause timeout shape is a faithful outage. |
||
|
Joseph Doherty
|
859d63178a |
fix(harness): make deploy E2E tests pass against real SQL (seed cluster FKs)
Integration-sweep follow-up #6 (SQL leg). The sweep flagged 7 heavy 2-node deploy/failover/reconnect E2E tests as 'time out under amd64-emulated SQL' and suggested raising deadlines. That diagnosis was WRONG: run against real (native-amd64) SQL on the Docker host, they still hang — even at 4x deadline. Root cause is that these tests had never actually run against real SQL, only the EF in-memory provider, which does NOT enforce foreign keys. Two distinct defects, both hidden by in-memory: 1. Missing FK-parent seed (5 tests). The deploy records a NodeDeploymentState row per node, FK'd to ClusterNode (FK_NodeDeploymentState_ClusterNode_NodeId). The tests never seeded ServerCluster + ClusterNode, so on SQL each node-state INSERT throws and the deploy never seals (waits the full deadline regardless of size). Added TwoNodeClusterHarness.SeedDefaultClusterAsync (seeds a ServerCluster + a ClusterNode per node; Warm/NodeCount=2 to satisfy the SQL CHECK constraint CK_ServerCluster_RedundancyMode_NodeCount AND the ClusterEnabledNodeCountMismatch validator). Called it in DeployHappyPath x2 / Failover / FleetDiagnostics / EquipmentNamespace; fixed DriverReconnect to seed node B + NodeCount=2. 2. Stale EquipmentId (EquipmentNamespace test; pre-existing on BOTH providers). Seeded EquipmentId='eq-1', but the later-added EquipmentIdNotDerived validator requires EquipmentId == DeriveEquipmentId(EquipmentUuid) ('EQ-'+first 12 hex). Fixed the seed to a canonical id + matching UUID. Only surfaced once the FK fix let the deploy reach equipment validation. Also added OTOPCUA_HARNESS_SQL_HOST / OTOPCUA_HARNESS_LDAP_HOST overrides so the harness fixtures can run on the native-amd64 Docker host (avoids arm64 mssql emulation entirely). Verified: the 5 SQL-backed classes go 11/11 green vs native SQL; in-memory default unregressed (12/12). RoslynVirtualTagEvaluatorTests racing test is a pre-existing flaky race (not SQL-backed; passes in isolation) — left as-is. |