b4a03b4baab964a810041bb2ca4a5446aa511e56
84 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Joseph Doherty
|
05b424788f |
fix(tagconfig): close the four #551 residuals — accept-gate-first Inspect, three authoring rules, CSV/service gate, doc sweep
Gitea #551. 1. CSV import ran no driver-typed config gate. Its typed columns were always strictly parsed, but the free-form TagConfigJson column reached the database unread, so a CSV could author a tag the driver rejects — one that deploys green and reads BadNodeIdUnknown forever. RawTagCsvMapper.Parse now names the offending row in the staged review grid, and RawTreeService enforces the same gate on CreateTag/UpdateTag/ImportTags: a form cannot be the guarantee (the API and any future caller reach none of them, and this repo has no bUnit). A rejected import row is an ERROR — the batch inserts nothing and the row is named — never a skip. Note the brief's premise that manual entry rides ImportTagsAsync is wrong: RawManualTagEntryModal commits row-by-row via CreateTagAsync, which is why the gate had to land on all three write paths. 2/3. The seven pre-#549 Inspect methods ran a hand-list of DescribeInvalidEnum checks, so every rejection cause not on the list was invisible. They are now ACCEPT-GATE-FIRST (the MTConnect/Sql/OpcUaClient/Calculation shape): ask the driver's own reader for the verdict, then explain the refusal, with a catch-all so "the parse rejects but Inspect said nothing" is unreachable. That closes S7's absent dataType, AbCip members[], Modbus stringByteOrder and the type-dependent guards, plus causes the issue did not name (absent region/address/tagPath/symbolPath/topic, TwinCAT Structure, MQTT's Sparkplug tuple). FOCAS and MQTT keep advisories about ACCEPTED blobs, so they diagnose rather than return empty. Authoring half: ModbusTagConfigModel gains stringByteOrder; AbCipTagConfigModel asks the new AbCipTagDefinitionFactory.DescribeInvalidMember; S7TagConfigModel captures whether the SOURCE blob declared dataType and asks S7TagDefinitionFactory.DataTypeKey/DeclaresDataType. One authority per rule. 4. Doc sweep: the six factory Inspect summaries, EquipmentTagConfigInspector's class remarks and AdminOperationsActor's comment all claimed invalid enums are "silently defaulted by the lenient runtime" — false since R2-11 Phase C made the runtime reads strict. They describe rejections now. |
||
|
Joseph Doherty
|
b4f8cb8de7 |
refactor(opcuaclient): move NamespaceMap to a .Session leaf so Contracts is dependency-light
G-7 wired OpcUaClientTagDefinitionFactory.Inspect into the deploy-time TagConfig gate, which gave ControlPlane a ProjectReference to Driver.OpcUaClient.Contracts — and with it, transitively, the whole OPC UA client stack (5 OPCFoundation packages) into every host that loads the control plane. NamespaceMap was the only reason Contracts carried OPCFoundation.NetStandard.Opc.Ua.Client: its entire public surface is ISession/NamespaceTable/NodeId-typed. It moves to a new leaf project, ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.Session, referenced by the runtime .Driver and by .Browser. Contracts is left with Core.Abstractions and NOTHING else — the same shape as Calculation.Contracts. The namespace stays ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient, so no consumer's `using` changed. The .Session -> .Contracts direction is deliberate: the blob read NamespaceMap needs is OpcUaClientTagDefinitionFactory.TryReadNodeId, the one authority it shares with the deploy gate, and that must stay in the dependency-light half. Why a new project rather than folding NamespaceMap into the runtime .Driver and having .Browser reference it (the Driver.Mqtt.Browser #512-style deliberate exception): the AdminUI project-references .Browser and NOT .Driver, so that shape would pull the OpcUaClient driver runtime — plus Core, Opc.Ua.Configuration and Secrets.Abstractions — into the AdminUI's build/deploy graph. That is exactly the violation #512 spent two extractions removing from Mqtt.Browser; re-introducing it to save one csproj is a bad trade. Guard: ContractsIsDependencyLightTests (Driver.OpcUaClient.Tests) asserts the Contracts manifest names no Opc.Ua* assembly, plus a counterpart asserting .Session does. An ABSENCE assertion is sound on the manifest where a presence one is not — the compiler elides an unused reference, but it MUST emit one the moment Opc.Ua-typed code is added. Falsified by temporarily re-adding the package plus an Opc.Ua.NodeId-typed member: red on Opc.Ua.Types, green again on removal. Evidence — `dotnet list ControlPlane package --include-transitive | grep -i opc`: before: OPCFoundation.NetStandard.Opc.Ua.{Client,Configuration,Core, Security.Certificates,Types} 1.5.378.106 after: none (and 0 Opc.Ua*.dll in ControlPlane's output directory) |
||
|
Joseph Doherty
|
1469fbb4d1 |
feat(deploy): wire the four G-7 inspectors into the deploy-time TagConfig gate
Adds the Sql / MTConnect / Calculation / OpcUaClient Contracts ProjectReferences to ControlPlane and their entries to EquipmentTagConfigInspector, closing the G-7 asymmetry: the deploy gate now checks every driver the AdminUI checks. Galaxy is the sole remaining unmapped type and is so by design — its dotted tag_name.AttributeName reference has no typed shape, and the AdminUI has no typed editor for it either, so the two gates agree. Adding the ProjectReference is what makes EquipmentTagConfigInspectorCoverageTests demand a map entry (its sweep scans the output directory, so an unreferenced Contracts assembly is invisible to it). Falsified: deleting the Sql entry fails Every_driver_shipping_an_Inspect_is_wired_into_the_dispatch_map with ["SqlTagDefinitionFactory"]; restored. Tests: three cases per driver in EquipmentTagConfigInspectorTests (canonical valid config warns nothing / present-but-invalid field draws a warning naming it / broken JSON warns without throwing), plus a Calculation control pinning that the two shapes the runtime accepts and the editor refuses draw nothing. OpcUaClient moved out of the unmapped-driver theory. New TagConfigAuthoringDeployParityTests (in AdminUI.Tests, the only suite that sees both gates) pins the direction that matters: a config the AdminUI model's own ToJson produces and Validate accepts draws no deploy warning. Its falsifiability control — the same driver types warning about a blob the editor refuses — caught a pre-existing, unrelated authoring-side gap: MTConnectTagConfigModel reads dataType leniently, so a misspelled name loads as String and saves clean while the strict runtime rejects it. Left alone (rewriting the editors is out of scope); the control uses the digits-only shape both gates refuse, and the case is documented inline. |
||
|
Joseph Doherty
|
b5607d355a |
refactor(fleet): #542 review residue — drop dead proxyOptions local, retire stale compliance defer line
The proxyOptions local in WithOtOpcUaControlPlaneSingletons was already unused on master (pre-existing dead code, not orphaned by the deletion commit — every registration passes singletonOptions); removed while the file is open. The phase-6-1 compliance script still Assert-Deferred'd the FleetStatusHub /hosts follow-up — an item that is now deleted, not deferred — replaced with a retirement comment. Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1 |
||
|
Joseph Doherty
|
f1c188114b |
refactor(fleet): #542 delete the dead fleet-status chain (broadcaster, SignalR bridge, hub, messages)
#542 filed FleetStatusBroadcaster as blind to site nodes since the Phase 6 per-cluster mesh split: it derives "the fleet" from central's own Akka gossip ring, which after the split is just central's own pair. Scoping the cross-mesh rewrite found the deeper truth — the whole chain is dead end to end, so this deletes it rather than reviving it. - Zero consumers. FleetStatusBroadcaster published FleetStatusChanged on the DPS "fleet-status" topic; FleetStatusSignalRBridge forwarded it to FleetStatusHub ("/hubs/fleet-status"); nothing anywhere connected. The AdminUI bans self-connecting HubConnections, no browser JS subscribes, and /fleet (Fleet.razor) reads NodeDeploymentState from the ConfigDb plus IClusterRoleInfo on a 5 s poll instead. - Its second input never existed. DriverHostStatusHeartbeat had no production sender — only the broadcaster's own test told it — so CurrentRevision was always null and the 30 s staleness -> Degraded flip could never fire. - Post-Phase-6 DPS is pair-local, so even the central-only view it did produce was an accident of topology rather than a design. The fleet visibility that actually works is unchanged and already cross-mesh-correct: /fleet over NodeDeploymentState, and the four Phase 5 per-node telemetry dials. Precedent for deleting inert machinery rather than activating it: #522 (driver stability tiers), #507 (discovered-node injection), #525 (GetMemoryFootprint). Deleted: FleetStatusBroadcaster (+ its nested DriverHostStatusHeartbeat), its admin singleton registration and FleetStatusBroadcasterKey, FleetStatusSignalRBridge and its key, FleetStatusHub and its route mapping, and the FleetStatusChanged / FleetNodeStatus / FleetNodeHealth messages. Both Fleet/ directories survive — ControlPlane keeps ClusterNodeAddressReconciler, Commons keeps GetDiagnostics. Docs swept where they claimed a fleet-status channel exists (Telemetry.md's Phase 5 non-migrated channels, Configuration.md, CLAUDE.md); completed-program plan docs under docs/plans and docs/v2 are left as the historical record they are. Build 0 errors, 868 warnings — byte-identical to master's rebuild baseline. ControlPlane.Tests 231/231, AdminUI.Tests 1109/1109. Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1 |
||
|
Joseph Doherty
|
4d2dfb5b50 |
test(mesh): step 7 — replace the boundary test with MeshGrpcBoundaryTests and sweep the comments
MeshGrpcBoundaryTests replaces the MeshClusterClientBoundaryTests deleted in step 1. Three
self-seeded single-member clusters sharing the ActorSystem name, joined ONLY by gRPC, driving
the REAL CentralCommunicationActor -> MeshGrpcCommandDialer -> Kestrel h2c ->
MeshCommandAuthInterceptor -> NodeCommandGrpcService -> NodeCommunicationActor -> EventStream:
1. No_cross_mesh_Akka_association_forms_between_central_and_either_site — the program's own
point, kept as a test rather than a live-gate note: each cluster stays single-member and
none ever holds another's address as a member.
2. One_dispatch_through_central_reaches_a_node_in_every_cluster_mesh — the fan, with each
site holding its OWN per-cluster PSK, so a dial that selected the wrong cluster's key
would be refused at the far side.
3. Falsifiability_control_a_dial_carrying_the_wrong_cluster_key_delivers_nothing — without
it, (2) cannot distinguish "the gRPC boundary carried it" from "something else in the
process did". The same node then accepts the right key, so the silence is the auth gate
and not a dead endpoint.
All three run fully offline — no Docker fixture — and pass.
Comment sweep across every remaining clusterclient|receptionist hit in src/ and tests/:
the program label becomes "mesh gRPC command plane", and every comment that described the
deleted transport as live now describes what actually runs (Dps = pair-local default,
Grpc = cross-mesh). mesh_transport.proto's header/service comments are reworded; no message,
field or enum is touched — the contract is locked. MeshCommActorPathTests is reshaped rather
than retired: the two paths still exist and still matter, but what a mismatch costs is now a
misleading Unavailable from the gRPC ingress rather than a silently dropped receptionist send.
The only surviving mentions of the old names are the ones that must survive: the fail-closed
rules that name "ClusterClient" as a retired value refused at boot, their tests, and this
test class's own account of what it replaces.
|
||
|
Joseph Doherty
|
5d7ecdf737 |
refactor(mesh): step 3 — delete both ClusterClientReceptionist registrations
Central's CentralCommunicationActor and a node's NodeCommunicationActor no longer register with a receptionist; the ActorRegistry entry each already carried is now the whole reachability story, and it is what CentralCommandGrpcService / NodeCommandGrpcService resolve to hand an inbound message over. Both `using Akka.Cluster.Tools.Client` directives are gone with them. |
||
|
Joseph Doherty
|
0db40a5e7f |
refactor(mesh): step 1 — delete the central ClusterClient factory and per-cluster client cache
IMeshClusterClientFactory/DefaultMeshClusterClientFactory are gone, along with
CentralCommunicationActor's _clients map, RebuildClient, the SendToAll fan and the
receptionist-address half of the contact load. LoadContactsFromDb SURVIVES — it feeds the
gRPC dialer's endpoint resolution and the known-cluster gate — but now projects only the
columns those two need (the AkkaPort/ActorPath.TryParse parse and its Malformed list went
with the receptionist addresses).
Ripples fixed in this step because the build must be green at every commit:
- ControlPlane/ServiceCollectionExtensions drops the factory argument.
- CentralCommunicationActorTests: the six ClusterClient-only facts are deleted (each has a
Grpc counterpart that already asserted the same property independently), the three
transport-agnostic facts move to Grpc mode, and the Dps fact keeps its own coverage.
- MeshClusterClientFactoryTests deleted (its subject is gone).
- MeshClusterClientBoundaryTests deleted here rather than in step 7: it drives
DefaultMeshClusterClientFactory directly, so it cannot compile past this commit. Its
replacement, MeshGrpcBoundaryTests, lands in step 7 and keeps the load-bearing assertion
(two separate meshes, no cross-mesh Akka association).
|
||
|
Joseph Doherty
|
d7c9ead64e |
test(mesh): close review findings on the PreStart-race logging fix — non-vacuous once-per-incarnation pin, broken-dialer symmetry, comment idiom
Follow-up to
|
||
|
Joseph Doherty
|
a94eb8a335 |
fix(mesh): PreStart resolve race logs honestly — quiet on heal, loud on permanent absence
PreStart's eager resolve of IMeshCommandDialer/IMeshPskProvider loses a
millisecond-wide race against the Akka builder's remaining starters on
essentially every admin boot (live gates: 1A 2/3 boots, 1B 4/4 boots) and
self-heals at the memoized Dialer/PskProvider properties' first use. Before
this change that produced two FALSE Errors per healthy boot, which made a
soak error-grep on a healthy rig impossible.
Resolve() now takes a `firstUse` flag and logs at Debug on BOTH call sites
(PreStart and the memoized properties' retry) — from inside Resolve a lost
race and a genuinely broken factory are indistinguishable, so severity is
decided by the CONSUMER of a still-null result after the retry, not by
Resolve itself:
- Dialer: DispatchOverGrpc's existing per-dispatch Error ("no command
dialer is wired") is unchanged and is now the ONLY loud signal for that
dependency.
- PskProvider: PublishKnownClusters gained a new Error carrying the old
PreStart guard's diagnostic content (gate never published, cached keys
never dropped, rotated key needs restart), guarded to fire ONCE per
actor incarnation via `_pskProviderAbsenceReported` — that handler runs
on the contact-refresh timer for the actor's whole life, so an
unguarded Error there would repeat every ContactRefreshSeconds forever.
Design choice for the "factory never supplied" case (Dps/ClusterClient test
constructions that pass no factories at all): kept silent at BOTH call
sites for both dependencies, same as before. For the dialer this was
already true (DispatchOverGrpc's check doesn't care why the dialer is
null). For the PSK provider the new once-guarded Error is gated on
`_pskProviderFactory is not null` specifically so the many existing
ClusterClient/Dps facts in CentralCommunicationActorTests that never wire a
PSK provider (using the bare Spawn helper) keep passing unmodified — a
never-supplied factory is a deliberate wiring/test choice, not a resolve
failure, and Resolve() already stayed silent about it for the same reason.
Two new tests pin the net behavior: a healthy boot (factories that throw
once then succeed) produces zero Warning/Error end-to-end through PreStart,
the first contact load and the first dispatch; a permanently-broken PSK
provider logs the absence exactly once across two contact loads, not once
per refresh.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
|
||
|
Joseph Doherty
|
e54883a8e0 | fix(mesh): memoize dialer/provider on first use — a lost PreStart race self-heals at the first refresh (review-prescribed) | ||
|
Joseph Doherty
|
c0475f2df4 |
fix(mesh): resolve dialer + PSK provider lazily — central boots again
Phase 1A live gate defect 4. Neither central node booted on this branch,
in ANY transport mode:
Akka.Hosting.MissingActorRegistryEntryException: No actor registered
for key ...ControlPlane.AuditWriterActorKey
at AdminUI.Audit.ActorAuditWriter..ctor(ActorRegistry registry)
at ControlPlane.ServiceCollectionExtensions.<WithOtOpcUaControlPlaneSingletons>
WithOtOpcUaControlPlaneSingletons' first WithActors starter resolved
IMeshCommandDialer and IMeshPskProvider eagerly for the Props. Their
graph reaches back into the registry — dialer -> IMeshPskProvider ->
ISecretResolver -> AddZbSecrets -> IAuditWriter -> ActorAuditWriter,
whose ctor calls registry.Get<AuditWriterActorKey>() — and
WithSingleton<AuditWriterActorKey> is registered ~60 lines LATER in the
same builder. Boot died before the entry existed. Mode-independent: the
resolve happened before anything consulted MeshTransport:Mode, so Dps
(the default, and the rollback target) died exactly like Grpc.
Both dependencies now arrive as Func<> and are invoked in PreStart, by
which point the host is fully built — the same shape, for the same
class of reason, as the coordinator proxy Func one argument earlier.
PreStart re-resolves on restart rather than caching across one; both
are container singletons, so a restart gets the same instances. A
resolution throw is caught and logged rather than propagated: a throw
out of PreStart is an actor failure, the supervisor answers it with a
restart, and a broken DI graph is not transient — that is an unbounded
restart loop that would take the transport-independent ack path down
with it.
Regression pin: ControlPlaneStarterOrderTests boots a REAL admin host
through WithOtOpcUaClusterBootstrap + WithOtOpcUaControlPlaneSingletons
in production starter order, with the production graph from
IMeshCommandDialer down to a registry-bound IAuditWriter, and asserts
StartAsync itself — the failure occurs while the builder's starters
execute, so building a ServiceProvider and resolving by hand cannot
reproduce it. Verified RED against the eager resolve: 3/3 modes fail
with MissingActorRegistryEntryException. A second fact pins that the
deferral did not become 'never resolved' — the actor must resolve the
provider from the container and drive it through the contact refresh,
signalled by a TaskCompletionSource (PreStart runs on the actor thread
and the contact load is a piped DB read, so no synchronous assertion
after StartAsync is correct).
Why nothing caught this: the dialer's container-activation test resolves
from a collection with no secrets registration, so its graph stops one
hop in; the actor facts inject fakes and never touch DI; and the
existing whole-host boots register no mesh provider, so the eager
lookups returned null and walked nothing.
Two aggravators recorded by the gate, no code here: the failed process
did not exit (docker Up while /health/ready 503s — Phase 2 readiness
concern, tracked), and this is the third instance of the same
eager-resolve-during-AddAkka class after scadaproj#1 and mesh Phase 6's
IClusterRoleInfo boot crash.
ControlPlane.Tests 236 -> 240, Host.Tests 92, Cluster.Tests 184,
0 warnings in every touched project.
|
||
|
Joseph Doherty
|
48dc2120c6 |
fix(mesh): gate follows the endpoint not the mode + review fixes
The known-cluster gate published only under Mode=Grpc, but the interceptor it feeds is registered on every admin node and 1B maps CentralCommandService by LISTENER PORT — neither consults the mode. A dark-switch rollback to ClusterClient or Dps would therefore have left that endpoint serving with the gate permanently inactive, silently restoring both the timing oracle and the pre-auth secret-store/audit flood, as a side effect of a rollback nobody would connect to authentication. The gate is central's cluster inventory, not transport state, so it is now decoupled from the mode entirely: - SetKnownClusters + InvalidateAll run on EVERY successful contact load in EVERY mode, and the contact-refresh timer starts in all three. Under Dps this is a ConfigDb query every ContactRefreshSeconds that did not run before; central always has a ConfigDb, and one indexed read a minute is the cheaper half of the trade. - The GATE set is built from ALL ClusterNode rows. The eligibility filter moved out of the query's WHERE and into the loop, so the FAN set is byte-for-byte what it was (enabled && !maintenance) while the gate widens. This also dissolves the maintenance coupling flagged in 1A.2: a cluster whose nodes are all disabled or in maintenance can still ack, and refusing it would make maintenance mode an authentication switch. Central still refuses clusters it has never heard of. - HandleContactsLoaded keeps a documented order — targets adopted, gate published, cache dropped — now pinned by test. Guards: PreStart mirrors the null-dialer error for a null PSK provider (silent absence is the worst outcome there — nothing malfunctions, so nothing is investigated, while the gate never publishes and keys never rotate); a failed contact load now says out loud that the gate is stale and, before any load has succeeded, inactive; MeshGrpcCommandDialer clamps a non-positive DispatchTimeoutSeconds to 10 with a construction Warning, because the validator for that key is driver-role-scoped and a pure-admin central reads it unvalidated; the dialer's local PSK refusal was shortened so its remediation survives the actor's 200-char truncation of remote-authored status detail; the portless-node warn-once set is rebuilt each refresh, so a node that regains then loses a port warns again — the regression is the occurrence worth seeing. Docs/pins: dialer disposal comment no longer claims one-way; the two interceptor-test comments claiming Matches has a default implementation corrected (it deliberately has none); new pins for per-refresh invalidation, the SetKnownClusters-before-InvalidateAll order, and one channel per endpoint via a new CachedChannelCount. The_known_cluster_set_is_not_published_by_the_other_two_modes is replaced — its premise is obsolete — by three facts: every mode publishes, gate set is a strict superset of the fan set, and the handler ordering. ControlPlane.Tests 230 -> 236, Host.Tests 92, Cluster.Tests 184, 0 warnings. |
||
|
Joseph Doherty
|
6a86201105 |
feat(mesh): gRPC command fan-out behind Mode=Grpc
MeshGrpcCommandDialer places one NodeCommandService.Execute call on one node: a GrpcChannel cached per endpoint (the GrpcDeploymentArtifactFetcher shape), the TARGET node's cluster's bearer plus x-otopcua-cluster naming that cluster, and a per-call deadline linked to the caller's token. A cluster with no resolvable key is refused locally with Unauthenticated — there is no arm on which a command leaves central unauthenticated, even one the far side would have rejected. CentralCommunicationActor gains the third mode branch. The contact load now also projects GrpcPort and, from the SAME rows in the SAME pass, builds the dispatch set — which is what makes 'the gRPC fan reaches exactly the SendToAll recipient set' structural rather than two queries that have to keep agreeing. A row with a null GrpcPort is skipped warn-once (the TelemetryNodeSource precedent). Per-node results are piped back for LOGGING ONLY: named node, gRPC status, dropped. No retry, no queue. Every dial is scheduled on the thread pool rather than called from the receive. Found by review and pinned red-first: an async method runs synchronously to its first await, and the dialer's pre-await work is TryGetPsk, which blocks on the secret store — bounded but seconds, and repeated after every refresh since the refresh drops the cache. Dialing inline parked the actor's mailbox, so deployment acks queued behind a slow node; the regression test tells an ApplyAck while a dial is held and fails without the hop. The contact refresh also wires the Phase 1A register: it publishes the loaded cluster ids as the PSK provider's known-cluster set and drops the key cache, so a rotated key takes effect within one ContactRefreshSeconds. Neither happens under Dps or ClusterClient. Both existing mode branches are behaviourally untouched and every existing fact passes unmodified. The one edit to shared text: the Status.Failure warning drops the word 'ClusterClient', since that handler now serves all three transports. ControlPlane.Tests 205 -> 230, Host.Tests 92, Cluster.Tests 184, 0 warnings. |
||
|
Joseph Doherty
|
14d21496fe |
feat(mesh): known-cluster gate + Matches on the PSK provider
Phase 1A security register items 1 and 2, landed ahead of the command fan-out because they are independently testable and the fan-out is the first caller of TryGetPsk. Item 1 — known-cluster gate. MeshPskProvider gains SetKnownClusters, an atomic whole-set swap consulted by TryGetPsk BEFORE the config map and before the secret store. A cluster id outside the set is refused in O(1) with no store read and no audit write, which closes two things at once: the pre-auth store/audit flood an unauthenticated caller could drive with a forged x-otopcua-cluster header, and the timing oracle that defeated CentralCommandAuthInterceptor's uniform denial text (unknown clusters are never cached, so they paid a store read on every probe while known ones answered from memory). Empty or never-published leaves the gate inactive — central publishes the set from its DB contact refresh, and 'the DB told me nothing' must not become a fleet-wide authentication outage during admin boot. Item 2 — bool Matches(clusterId, presented) on IMeshPskProvider, with FixedTimeEquals inside the provider. CentralCommandAuthInterceptor now calls it and never holds key material; its two provider-side denial logs collapse into one (the provider answers both as false) and the operator hint is restated there. No production caller publishes a cluster set yet, so behaviour is unchanged until the fan-out commit wires the contact refresh. ControlPlane.Tests 190 -> 205, Host.Tests 91 -> 92. |
||
|
Joseph Doherty
|
24e55d3395 | fix(mesh): Batch B review fixes — bounded store read, cache poison guard, test hardening | ||
|
Joseph Doherty
|
68a416e4fc | feat(mesh): per-cluster PSK provider + central interceptor | ||
|
Joseph Doherty
|
aa21f61fa0 | feat(mesh): Grpc mode value + validation | ||
|
Joseph Doherty
|
39db75083d |
feat(drivers): every driver reports the authored tags it skipped (Gitea #541)
A driver builds its RawPath -> definition table at init and tolerates a
per-tag miss: an unmappable TagConfig, an unparseable address, a
capability-matrix rejection. That tolerance is right -- one bad tag must not
take down the other few thousand. What was missing is any statement of the
fact afterwards. A skip is not an error and happens before any read, so the
driver stays Healthy, errors/5 min stays 0, Last error stays empty, and Last
read advances live off the tags that did map. On the wonder Makino in #540
that reading held while focas-z-34184 ignored 20 of its 22 tags.
New optional capability ISkippedTagReporter, backed by a shared
SkippedTagLedger so the reset/commit/cap rules are one implementation rather
than one per driver. The report -- authored count, exact skipped count, and a
capped sample of RawPath + reason -- rides DriverHealthChanged to a new Tags
column on /hosts, over the Phase 5 telemetry stream. No table: a driver-only
node has no ConfigDb to write to, the lesson from #521.
The issue named eight drivers. It is twelve -- every driver in
DriverTypeNames.All has an authored-tag skip path. Calculation (no scriptId),
MTConnect (no readable DataItem id), Galaxy (no attributeRef) and OpcUaClient
(no nodeId) were unlisted, and OpcUaClient's was the quietest of the twelve:
NamespaceMap drops such an entry with a bare `continue` and not even a log
line. SkippedTagReporterFleetTests pins the invariant reflectively over every
IDriver in src/Drivers/, so a thirteenth cannot re-introduce the silence.
Sql drops a tag at TWO points and its report is the union of both: a blob
that does not map, AND an identifier the 8.1 catalog gate rejects against the
live catalog. The second is what schema drift produces -- a renamed column, a
dropped view -- months after a deployment sealed green, and reporting only
the first would have that driver claim "0 skipped" while polling half its
tags, which is worse than reporting nothing because the column would then be
actively reassuring.
Deliberately additive to driver state, never a change of it. Latching
non-Healthy for a config problem would satisfy both halves of the #527
comms-loss conjunction -- 90 s without a successful read AND non-Healthy --
and a driver whose tags were ALL skipped has no successful reads by
construction, so it would reconnect-loop forever against a healthy device.
Three details that are load-bearing rather than incidental:
- The skip digest is part of the publish fingerprint. A skipped tag leaves
every other fingerprint component identical to a clean driver's, so without
it PublishHealthSnapshot's dedup swallows exactly the publish carrying the
signal -- the trap that already bit the rediscovery and host-status
channels. The digest covers the sample, not just the counts: a redeploy
that fixes one tag and breaks another leaves both counts unchanged.
- The count is exact; only the sample is capped, and the tooltip says so when
it is showing fewer than it counted. A truncated list that looked complete
would have an operator fix twenty tags and believe the job done.
- The ledger stages and commits, so an init that throws part-way leaves the
previous report standing. A partial snapshot would read as "we now skip
fewer tags", the opposite of the truth.
The proto field needs no companion presence flag, unlike host_statuses: a
singular message field carries real presence in proto3, and a has_ flag would
collide with protoc's generated HasSkippedTags.
Verified red-first three ways: dropping SkippedTagDigest from the fingerprint
fails the two dedup tests, removing the seam from one driver fails the fleet
guard, and dropping the catalog-gate leg fails the Sql union test. Full
solution builds with zero new warnings against
|
||
|
Joseph Doherty
|
06c56d7719 |
fix(cluster,acl): OTOPCUA_ROLES fallback + one string-identity rule (#532, #533)
Two independent defects surfaced in review of #520 stage 3.
#532 — Cluster:Roles had no OTOPCUA_ROLES fallback, despite two doc sites
promising one. A node configured only through the env var — which drives ALL of
Program.cs's conditional wiring, so it reads as the authoritative knob — bound an
empty Roles array and joined the cluster carrying no Akka roles at all. It ran,
served, and logged nothing wrong, while central reported "no driver-role nodes
are Up" and every deployment to it timed out. Not hypothetical: the field fix on
wonder-app-vd03 was to hand-add Cluster__Roles__0/1 to a Windows service's
registry environment. Anyone reading the code to diagnose that box would have
concluded from the XML doc that the fallback was already applied.
AddOtOpcUaCluster now post-configures AkkaClusterOptions through the shared
RoleParser.ResolveEffectiveRoles, so BuildClusterOptions (Akka's own member
roles), ClusterRoleInfo, the ControlPlane singleton scope and
ConfigDbApiKeyStore's ClusterId all see one answer. Two sibling validators
(ConfigSource, Telemetry) read Cluster:Roles directly and were exempting the
exact node they exist for — an env-only driver-only node has no ConfigDb and
could sit on ConfigSource:Mode=Direct unchallenged. They resolve the same way now.
Also collapsed the four open-coded `Roles.FirstOrDefault(IsClusterRole)` +
ClusterIdFromRole copies into RoleParser.ClusterIdFromRoles. They had drifted:
three classified the cluster role ordinally, so a verbatim-bound `Cluster-SITE-A`
yielded a null ClusterId — and a null ClusterId applies NO cluster filter, which
made #528's cross-cluster over-grant guard fail OPEN. IsClusterRole is now
case-insensitive; the ClusterId suffix keeps its authored spelling, because
normalizing it would silently rename the Akka role of a running deployment.
#533 — three string-identity rules in one authorization path. AclScopeRef used
default (ordinal) equality while the trie builder filtered ClusterId and hashed
principals OrdinalIgnoreCase; the deploy gate's cross-cluster check was ordinal
too, which was the worse half — it could reject a whole deploy over a ClusterId
case difference the runtime would have accepted. It only worked because resolver
and builder were handed the same NodeAcl instances.
The rule is now stated once, on AclScopeRef, and applied throughout:
identifiers (ScopeId, ClusterId, Principal) compare OrdinalIgnoreCase; path
segments compare Ordinal. AclScopePathResolver re-keys its own maps rather than
documenting a comparer contract for callers, so the two independent producers —
EF at the gate, artifact JSON on the node — cannot diverge from the outside.
PermissionTrieNode.Children stays Ordinal: those are browse names, and Speed /
speed are two nodes that can coexist under one device.
Split PermissionTrieBuildDiagnosticReason.RetiredScopeKind out of
UnresolvedScope. A legacy FolderSegment row and a typo'd id had the same message
and different remedies.
Falsified: five mutations, each reddening only its own guard — remove the
PostConfigure (3 tests), revert IsClusterRole to ordinal (3), revert AclScopeRef
equality (1), revert the resolver's map comparer (6), revert the gate's
comparisons (2), remove the FolderSegment short-circuit (1).
Test isolation: OTOPCUA_ROLES is process-global and xUnit runs classes in
parallel, so the three validator suites that now resolve effective roles share a
non-parallel collection. Without it, cases passing no roles would intermittently
observe another class's env var.
Full solution build clean; full suite green apart from two pre-existing failures
confirmed identical on
|
||
|
Joseph Doherty
|
a438ac1640 |
feat(acl): keys + ACLs reach driver-only nodes via the artifact (#520 stage 4)
A driver-only node has had no ConfigDb since per-cluster mesh Phase 4, so it registered NullOpcUaUserAuthenticator and denied every UserName token. Its keys now ride the deployment artifact instead. - ConfigComposer snapshots ApiKeys alongside the NodeAcls it already carried. Only the peppered HMAC ships, never a secret, and the pepper is node config that is deliberately not in the artifact. - DeploymentArtifact.ParseDataPlaneAuthority decodes both node-side. Grants are scoped to the node's own cluster; keys are not, because ApiKey carries no ClusterId — a key is a fleet-wide credential and the ACL rows are what bind it. - OpcUaPublishActor installs them on DataPlaneAuthorityState on every apply, BEFORE its plan.IsEmpty early-return: an artifact that only revokes a key produces an empty address-space plan, so feeding after that return would leave the revoked key working until an unrelated structural change redeployed. - ArtifactApiKeyStore serves them to session activation; a node with a ConfigDb keeps ConfigDbApiKeyStore, whose per-request read is what makes revocation immediate there. - OtOpcUaNodeManager gains DataPlaneAuthority — the cluster identity blocker #4 named. NodeScope.ClusterId is required and the evaluator fails closed on a mismatch, so until now no scope it built could have evaluated. Nothing reads it until stages 5/6. The generation is a node-local COUNTER driven by the RevisionHash, not derived from its bits. PermissionTrieCache treats the highest GenerationId as current, so a generation taken from hash bytes would install a lower number about half the time and the cache would go on serving the previous deployment's ACLs, silently. Sessions are not stamped with it yet — that belongs with the first seam that reads a UserAuthorizationState, in stage 5, rather than being built now and consulted by nothing. Live-gated on docker-dev against site-a-1 (driver-only, no ConfigDb): key authenticates, wrong secret denied, both halves of the revocation asymmetry observed, and a SITE-A grant confirmed not to reach site-b-1. Key minted with an independent Python HMAC so the verifier was checked against something other than itself. Rig left clean. Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo |
||
|
Joseph Doherty
|
30d0697c28 |
feat(hosts): surface per-host connectivity on /hosts; drop the unwritable table (#521)
`IHostConnectivityProbe` was a dead surface: eleven drivers implement it, `GetHostStatuses()` had ZERO production call sites, and `OnHostStatusChanged` had no subscriber outside the Galaxy driver's own aggregator. Per-host connectivity was computed by every driver and read by nobody. The issue offered "build the publisher or delete it". The publisher as its entity doc described it — driver nodes upserting `DriverHostStatus` rows — is not buildable: per-cluster mesh Phase 4 gates `AddOtOpcUaConfigDb` on the `admin` role, so a driver-only node has no ConfigDb connection to write rows with. So the capability is kept and the transport changed. `DriverHealthChanged.HostStatuses` now carries the probe result to `/hosts` as a Hosts column. That channel already reached the page, already survives the mesh split via the Phase 5 gRPC telemetry stream, and already replays a last-value snapshot on re-subscribe — so per-host state re-primes after a reconnect without a durable store. Both halves of the interface finally do what they are for: the event triggers a prompt publish, the pull is the source of truth. The point of the column is the case the driver-level state chip structurally cannot express: a multi-device driver stays aggregate-Healthy while ONE of its devices is unreachable. Two traps, both pinned by tests that were falsified against the prod code: - The host digest MUST be in the publish fingerprint. On a single-host-down transition every other fingerprint component is unchanged, so the dedup would swallow exactly the publish carrying the news — the trap that already bit the rediscovery signal. Removing it turns the guard test red, verified. - null (no probe) must stay distinct from empty (probe with no hosts). proto3 cannot tell an absent repeated field from an empty one, hence the explicit `has_host_statuses` flag; collapsing them would render every probe-less driver as one whose devices are all fine. Dropped: the DriverHostStatus entity, enum, DbSet, model config and table (migration DropDriverHostStatusTable — empty on every deployment, so the scaffolder's data-loss warning is moot, and Down() recreates it exactly). Found en route, NOT fixed here: `DriverInstanceResilienceStatus` is the identical defect — no writer, no reader, only a DbSet declaration, while the live data rides the `driver-resilience-status` telemetry channel. Its doc-comment now states that rather than describing the sampler and AdminUI join that were never built. Filed as #524 rather than widening this schema change beyond what was asked. Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo |
||
|
Joseph Doherty
|
4624141a5d |
test+docs: retire the stale EquipmentTagsDarkBatch4 skips, guard the deploy-time
tag gate, and fix three silently-broken OpcUaClient integration tests
Continues working through deferment.md's remaining items.
SKIPPED TESTS (13 -> 0 for this reason). The skip said "dark until Batch 4";
Batch 4 shipped as v3.0 and RETIRED the equipment-tag path rather than lighting
it, so every one of them was waiting on something that had already happened and
gone the other way. Resolved individually rather than in bulk:
- REVIVED onto the raw/UNS shape (6): four DriverHostActor primary-gate cases,
the cluster-scoping rebuild, and the real-SDK dual-namespace materialisation
E2E. The behaviour never went dark, only the v2 seeding did.
- FOLDED into a live parity test (3): DeploymentArtifactRawUnsParityTests
already compared RawTagPlan record-equal and RawTagPlan carries array /
historize / alarm intent, so widening its corpus by two tags covers what
three empty placeholder suites had been promising. Those suites are deleted.
- DELETED (4): subjects genuinely retired -- equipment-namespace EquipmentTags,
and both dot-joint {{equip}}.X token suites (that resolver was deleted in
Batch 3; the slash-joint form is covered by DeploymentArtifactEquipRefParityTests).
Falsified, not assumed: removing the seeded UnsTagReference turns two revived
primary-gate tests red; re-homing the SITE-A node to MAIN turns the scoping test
red. Widening a parity corpus also needed direct value assertions -- parity alone
cannot distinguish "both seams right" from "both seams blind".
Runtime.Tests skips 13->3, OpcUaServer.Tests 4->1; all remaining are deliberate
EquipmentDeviceBindingRetired tombstones.
G-7 (deploy-time TagConfig gate). MQTT shipped an Inspect() nobody ever called;
now wired. Replaced the hand-maintained list with a reflection guard, because the
existing test enumerates driver types by hand and so guards renames but can never
notice a gap. NOTE: the first version of that guard was hollow -- it discovered
candidates via GetReferencedAssemblies(), which is circular, since the compiler
elides a reference nothing in the IL uses. Removing MQTT from the map also removed
its Contracts assembly from the manifest, and the guard passed green with the bug
reintroduced. Rewritten to scan the output directory; re-falsified and it now
fails naming MqttTagDefinitionFactory. Residual recorded at the code: Sql,
MTConnect, Calculation and OpcUaClient have no Contracts-side Inspect at all, so
the deploy gate stays weaker than the AdminUI's authoring gate for them.
OPCUACLIENT INTEGRATION TESTS (3 of 4 failing, on master too). Not fixture rot:
the simulator was healthy and Client.CLI read the very same node Good. v3 made the
driver resolve every reference through its authored RawTags table, so a bare
"ns=3;s=StepUp" fails LOCALLY at the resolver with BadNodeIdInvalid and never
reaches the wire -- the tests were exercising the unresolved-reference guard.
Fixed by seeding OpcPlcProfile.RawTags in the deploy artifact's TagConfig shape
and addressing by RawPath. 4/4 pass.
DOCS. Applied the Phase7* -> AddressSpace* rename across the four live docs
(historical bannered files deliberately untouched), resolving the three that are
not renames to their real members. Found en route that the earlier banner pass
missed three files: VirtualTags.md, OpcUaServer.md and ScriptedAlarms.md all
still presented the test-scaffolding GenericDriverNodeManager as the production
dispatch path. All three now carry the correction.
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
|
||
|
Joseph Doherty
|
09a401b881 |
feat(drivers): consume IRediscoverable as an operator re-browse prompt (§8.2, #518)
Nine drivers raise IRediscoverable.OnRediscoveryNeeded when they observe that a remote's tag set may have changed — a Galaxy redeploy, a TwinCAT symbol-version bump, an MTConnect agent restart, a Sparkplug rebirth. Nothing in src/ subscribed, so every raise went into the void. Drivers even wrap the invoke in try/catch for "subscriber threw"; there has never been a subscriber. DriverInstanceActor now attaches the event in PreStart and detaches in PostStop, mirroring AttachAlarmSource/DetachAlarmSource. Attach is per-actor, not per-connect: the raise has no connection affinity and can land while the driver is between connects. The detach is load-bearing rather than tidy — the IDriver instance outlives the actor when the host respawns a child around the same driver object, so a missing -= accumulates one handler per respawn, each holding a dead Self. The signal rides the existing driver-health snapshot to /hosts as a "re-browse" chip. It is ADVISORY: the served address space is deliberately not rebuilt, because v3 authors raw tags explicitly through the /raw browse-commit flow and a runtime graft would materialise nodes no operator approved and no deployment artifact records. Two things worth knowing: - PublishHealthSnapshot dedups on a health fingerprint, and a rediscovery raise on an otherwise-healthy driver changes none of the four fields it hashed. Without adding the timestamp to that tuple the dedup swallows the exact publish carrying the signal. Reverting that one line turns 3 of the 5 new tests red, which is how it was confirmed rather than assumed. - The new test stub implements IDriver from scratch instead of deriving from the shared StubDriver, whose GetHealth() returns DateTime.UtcNow — its fingerprint differs on every call, so the dedup never engages and the dedup test would have passed vacuously either way. The planned discovery-result drift detector was NOT built. Modbus, S7, MQTT, AbLegacy and Sql all echo authored config from DiscoverAsync rather than browsing the device, so a discovered-vs-authored diff is permanently empty for them — it would have been another plausible-looking inert seam, which is the class this audit exists to remove. IRediscoverable is the trustworthy signal because the driver asserts it deliberately. DriverHealthChanged, IDriverHealthPublisher.Publish and HostsDriverRow gain defaulted optional parameters, so no existing call site changed. telemetry.proto gains fields 8/9 and both Phase-5 mappers carry them. Runtime.Tests 512 passed / 31 skipped. |
||
|
Joseph Doherty
|
88adec047b |
refactor(health): adopt the shared active-node check (Health 0.3.0)
ClusterPrimaryHealthCheck was written three days' worth of debugging ago because the shared ActiveNodeHealthCheck selected by RoleLeader and reported Healthy for any node lacking the role. Health 0.3.0 fixes the shared check — oldest Up member, role-preference scoping, Unhealthy when the node owns no active work — so the private copy is now duplication rather than a workaround, and it is deleted. RolePreference [admin, driver] reproduces exactly what it did: a fused central node answers for admin (where the singletons and the AdminUI are pinned), a driver-only site node answers for driver. SelectOldestUpMemberOfRole now delegates to the shared ClusterActiveNode instead of re-implementing the age ordering. This is the point of the change rather than a tidy-up: the redundancy snapshot drives the OPC UA ServiceLevel 250/240 split while the health tier drives Traefik's admin routing, so two copies of "oldest Up member of a role" would be two chances to advertise one node as authoritative while gating the data plane on another. ControlPlane takes a ZB.MOM.WW.Health.Akka reference for it; the layering trade-off is recorded at the PackageReference. Behaviour note: a node carrying neither admin nor driver now answers 503 on the active tier instead of 200. That case is reachable — RoleParser admits dev-only and cluster-role-only nodes — and 503 is correct, since such a node owns no active work and must not be in an active-tier pool. No docker-dev node is affected: every rig node is admin+driver or driver-only. Tests replaced in kind. The rule itself is now pinned in the library against a real two-node cluster; what stays OtOpcUa's own decision is the wiring, so ActiveTierRegistrationTests pins which check is on the active tag, that it is registered on driver-only nodes too, and that it is not on the ready tier. Verified: Host.Tests 25/25, ControlPlane redundancy 15/15. |
||
|
Joseph Doherty
|
4550486144 |
fix(health): active tier reports the cluster Primary, not the admin role leader
Closes the remaining half of #494, and corrects the half fixed in |
||
|
Joseph Doherty
|
3a4ed8ddb5 |
fix(mesh): break Phase 6 boot-crash — cluster-redundancy singleton scope must not resolve IClusterRoleInfo eagerly
Phase 6 wired the re-homed redundancy singleton in Program.cs as `WithOtOpcUaClusterRedundancySingleton(sp.GetRequiredService<IClusterRoleInfo>())` inside the AddAkka configurator lambda. That lambda runs WHILE the ActorSystem is being built, but ClusterRoleInfo depends on the ActorSystem (it is a live Cluster.State view) — so resolving it there recurses into ActorSystem construction and every node dies at boot with StackOverflowException. It compiles cleanly (a runtime DI cycle) and RedundancyStateSingletonRehomeTests passed a hand-built FakeClusterRoleInfo straight into the extension, mocking around the composition-root line that overflows. The docker-dev live gate caught it on first boot. Derive the singleton's cluster-role scope from AkkaClusterOptions (pure config, no ActorSystem) instead: BuildClusterRedundancySingletonOptions / the extension now take AkkaClusterOptions and compute Role = Roles.FirstOrDefault(IsClusterRole) ?? driver — identical to ClusterRoleInfo.ClusterRole, which derives from the same config. Program.cs passes IOptions<AkkaClusterOptions>.Value, which has no ActorSystem dependency. Live-verified: rebuilt image boots with zero stack-overflow lines, cluster singletons form per mesh. 5/5 rehome tests pass; Host builds clean. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
8a173bbaf6 |
docs(mesh): Phase 6 — mark done, remove single-mesh caveats, document pair-local meshes
Redundancy.md: KNOWN-LIMITATION (per-Akka-cluster election) marked RESOLVED;
new 'Per-cluster meshes (Phase 6)' section + pair-local secrets note; manual-
failover mesh-scope caveat removed; auto-down 1-vs-1 note updated (every mesh is
now two nodes, drill deferred to Phase 7). IManualFailoverService + ClusterRedundancy
razor caveats updated to pair-local reality. Configuration.md documents the
cluster-{ClusterId} role, per-pair self-first seeds, SplitTopologyTransportValidator,
and the intentional Dps default. Program + design status tables + CLAUDE.md marked
Phase 6 DONE.
(Task 8 subagent completed the edits but died on an API error before committing;
reviewed and committed by the controller.)
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
|
||
|
Joseph Doherty
|
d98b32640b |
docs(mesh): correct the all-malformed-cluster comments in CentralCommunicationActor
The per-cluster contact map only gains a key when a row parses, so an all-malformed cluster has no key at all and is caught by the absent-key arm; the empty-set guard is defensive/unreachable for the DB producer. Comment-only (Task 4 code-review minor follow-up). Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
1ce2042e14 |
fix(mesh): also scope the reconciler's observed membership to the own cluster
Task 3 review: scoping only the rows/enabled DB queries by ClusterId left the observed set (from Cluster.State.Members) filtered by the driver role alone. In the mixed-mesh window — DB scoping live via a cluster role, but the physical Akka mesh not yet split so central still sees foreign members via gossip — a foreign-cluster driver member stays in observed while its row is filtered out, hitting the RunningNodeHasNoRow branch, which logs at ERROR. That traded the Warning-level false positive for a worse Error-level one. Filter observed by the node's own cluster ROLE too. Extracted the projection into a static FilterOwnClusterDriverMembers(members, ownClusterRole) so the membership filter is unit-testable without seeding a live cluster (the actor harness joins as admin only, so observed is always empty). Sourced at registration from IClusterRoleInfo.ClusterRole alongside ClusterId. Null role (legacy, no cluster role) keeps every driver member — reconcile-all, unchanged. Also clarified the ownClusterId doc wording (non-null-and-non-empty). Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
6531ec1984 |
feat(mesh): one ClusterClient per Cluster in CentralCommunicationActor (Phase 6)
After the fleet splits into one Akka mesh per application Cluster, a ClusterClientReceptionist serves only its own mesh — so the single fleet-wide client's SendToAll reached only the mesh whose receptionist answered, leaving every other cluster silently on its old configuration. Central now holds one ClusterClient per ClusterId and fans SendToAll across all of them. LoadContactsFromDb selects ClusterId and groups the receptionist contacts per cluster (keeping the per-row TryParse guard and the enabled/non-maintenance filter); ContactsLoaded carries ContactsByCluster; HandleContactsLoaded diffs per cluster (rebuild changed, stop+drop vanished, warn-don't-create on empty); RebuildClient builds a per-cluster client under a clusterId-derived actor name. IMeshClusterClientFactory.Create gained a clusterId parameter so the per-cluster clients get distinct, diagnosable names. ApplyAck handling and the DPS branch are unchanged; the deploy path stays payload-free. Tests: focused unit coverage of the grouping + fan-out (one-client-per-cluster, fan-out SendToAll to every cluster client) via the recording factory double, plus the real two-mesh boundary test extended to central + two separate site meshes proving one dispatch reaches both and a client scoped to one cluster never crosses into another. Red-before-green verified: crippling the fan-out to a single client fails the reaches-both-meshes assertion. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
2535df019b |
fix(mesh): scope ClusterNode address reconcile to the admin node's own cluster
Phase 6 splits the fleet into one mesh per Cluster, so an admin (central) node's Cluster.State.Members shows only its own pair — it no longer sees site members via gossip. The ClusterNodeAddressReconciler singleton compared live membership against EVERY ClusterNode row fleet-wide, so post-split every foreign-cluster row would log EnabledRowNotInCluster forever. Scope the actor's two DB queries to rows whose ClusterId matches the admin node's own (IClusterRoleInfo.ClusterId, sourced at registration). A legacy admin node with no cluster role (null ClusterId) still reconciles the whole fleet — it genuinely sees every member via gossip. The pure Reconcile function and AddressMismatchKind semantics are unchanged; scoping lives entirely in the actor's queries. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
a2c5d57fa0 |
feat(mesh): re-home the redundancy singleton to a per-cluster driver singleton
Phase 6 splits the single fleet-wide Akka mesh into one 2-node mesh per
application Cluster. RedundancyStateActor was an admin-scoped cluster singleton
that elected ONE fleet-wide driver Primary — but after the split a driver-only
site pair has no admin node in its mesh to host it, so its Primary would never
be elected.
Re-home it: remove the redundancy WithSingleton block from the admin
WithOtOpcUaControlPlaneSingletons set, and add WithOtOpcUaClusterRedundancySingleton,
a singleton spawned from Program.cs's hasDriver branch on EVERY driver node (the
fused central included). It scopes to the node's own cluster-{ClusterId} role when
present, and falls back to the fixed `driver` role otherwise. Each mesh then has
exactly one, electing its own pair-local Primary and publishing redundancy-state
on its own mesh's DistributedPubSub. The election logic (oldest Up driver member)
and the DPS publish are unchanged — they become pair-local after the split.
The driver-role fallback (Decision 2) deliberately does NOT throw when a node has
no cluster role: that is the pre-Phase-6 fleet-wide behavior on a legacy single
mesh, so legacy/harness (TwoNodeClusterHarness: admin,driver) and not-yet-migrated
deployments keep booting unchanged. On a genuinely split 2-node mesh the `driver`
role is already pair-local (the mesh IS the pair), so the fallback is correct in
both worlds. Cluster-scope, when present, additionally survives an accidental
two-mesh merge by keeping one singleton per cluster.
The role scope + driver-role fallback are pinned by unit tests against the
extracted BuildClusterRedundancySingletonOptions helper (the BuildDowningHocon
pattern); admin-removal + driver-registration are pinned behaviorally against a
booted node's ActorRegistry.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
|
||
|
Joseph Doherty
|
87ff00fd30 |
feat(mesh): RoleParser accepts cluster-{ClusterId} roles; consolidate the driver-role literal
Phase 6 (per-cluster mesh split) needs nodes to carry a cluster-scoped role like cluster-SITE-A; RoleParser previously rejected anything outside the fixed admin/driver/dev set. Adds IsClusterRole/ ClusterIdFromRole plus well-known-role constants, and points the three duplicate "driver" string literals (RedundancyStateActor, ClusterNodeAddressReconcilerActor, ServiceCollectionExtensions) at the new RoleParser.Driver so the value has one source, without renaming any existing DriverRole symbol. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
a279a43ed4 |
fix(mesh-phase5): harden telemetry client onError contract (validate/isolate/defend)
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
c78034f0b9 |
feat(mesh-phase5): central telemetry dialer client + proto->domain converters
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
2a2e54f083 |
feat(mesh): route deploy, driver-control and alarm commands through the mesh router
Phase 2 Task 7. The five central->node publish sites -- ConfigPublishCoordinator's deploy dispatch, and AdminOperationsActor's restart, reconnect, acknowledge and shelve -- now hand a MeshCommand to the transport router instead of telling the DistributedPubSub mediator directly. Neither actor knows which transport is in force any more. _meshRouter is NULLABLE with a mediator fallback, and that is deliberate: about a dozen existing coordinator and admin-operations tests construct these actors without a router, and they must keep exercising the real DPS path rather than a silently-disabled one. The fallback is not a permanent seam -- Phase 6 deletes it with the DPS branch and the single mesh. Wiring note: the comm actor's WithActors block moved ABOVE the singleton registrations so both singletons can resolve it with registry.Get, which is the ordering-by-registration idiom AdminOperationsActor already uses for the coordinator. An earlier version used ActorSelection.ResolveOne().GetAwaiter() .GetResult() inside the props factory -- that blocks inside actor construction and races the very spawn it waits for. Sabotage-verified by inverting the branch so the mediator always wins: all six new tests go red, including the fallback test (which proves it is asserting the mediator path rather than passing by accident). Suites after: ControlPlane 118/118, Runtime 450/450, Host.IntegrationTests 196/202 -- sole failure AbCip_Green_AgainstSim, the fixture baseline that fails on master too. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
16d598560a |
feat(mesh): register comm actors with the receptionist; pin the wire-contract paths
Phase 2 Task 6. Both comm actors are now spawned and receptionist-registered: CentralCommunicationActor from WithOtOpcUaControlPlaneSingletons, and NodeCommunicationActor from WithOtOpcUaRuntimeActors (before DriverHostActor, because it is the host's ackRouter under ClusterClient mode; it has no dependency of its own on the host, since inbound commands travel via the node's EventStream). Both are plain WithActors registrations, NOT singletons -- a driver node's ClusterClient rotates across contact points and must find a live comm actor at whichever node answers. Both register with the receptionist in BOTH transport modes: an idle registration costs nothing, whereas registering only under ClusterClient would mean flipping the flag on a running fleet needs a restart before anything can be reached, turning a config change back into a deployment. The coordinator ref is resolved lazily per message (Func<IActorRef?> + registry.TryGet) so the comm actor never depends on the order in which Akka.Hosting materialises the singleton proxy relative to this block. Adds MeshCommActorPathTests, which boots the real two-node host and Identify- probes both paths. These strings are the wire contract and nothing else asserts they agree: a rename would compile, pass every unit test, deploy, and deliver nothing, because a ClusterClient send to an unregistered path is dropped silently. Sabotage-verified with an actual rename. DELIBERATELY NOT TESTED, with the reason recorded in the file: that the actors are per-node rather than singletons. Two discriminators were tried and both were invalid -- "the path resolves on both nodes" passes under either shape because a ClusterSingletonManager is also created on every node in the role, and "no /singleton child" is null even for the KNOWN singleton /user/config-publish, so Akka.Hosting does not lay them out that way. A sabotage re-registering the actor via WithSingleton left both green, which is what exposed them. Rather than ship an assertion that cannot fail, the property is left to the Task 8 boundary test, where a send only arrives if the target really is registered. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
d5b5cb6ede |
feat(mesh): CentralCommunicationActor — dark-switched fan-out, DB-sourced contacts
Phase 2 Task 3. Central-side end of the boundary: routes every central->node command out over DPS or ClusterClient per MeshTransport:Mode, and forwards inbound ApplyAcks to the deploy coordinator singleton (Forward, not Tell, so the coordinator sees the node rather than this relay). Three things here are load-bearing and easy to get wrong later: SendToAll, never Send. Today's DPS publish reaches EVERY DriverHostActor and the node side has no ClusterId or node filter to compensate -- scoping happens later, inside the artifact. ClusterClient.Send delivers to exactly ONE registered actor, so it would deploy to a single node while every other node silently kept its old config, and the deployment could still seal green. Sabotage-verified: swapping to Send reddens exactly that one test. Exactly ONE ClusterClient, fleet-wide. A receptionist serves its whole cluster, so SendToAll reaches every registered node-comm actor in the mesh regardless of which contact point was dialled. One client per application Cluster -- the shape Phase 6 wants -- would fan each command out once per cluster while the fleet is still one mesh: N x duplicate DispatchDeployment and ApplyAck. Marked TODO(Phase 6). The contact set does NOT scope delivery. Excluding a maintenance-mode node from the contacts does not stop it receiving commands; it still receives them, as it does under today's broadcast. The filter is about which receptionists are worth dialling, not about who gets the message. Stated in the code because the opposite is the natural assumption. Also carries the sister project's shipped fixes: per-row address parsing so one malformed ClusterNode cannot abort the refresh and leave the contact set half-built (regression test orders the bad row FIRST), the cache entry cleared before recreate so a failed create cannot leave commands routing into a stopping actor, and a Status.Failure handler so a DB outage is a Warning rather than a silent debug-level unhandled message. Two test-harness bugs found and fixed while writing this, both mine: SubscribeAck goes to the SENDER (so the mediator subscribe needed an explicit sender), and EventFilter matches case-INSENSITIVELY, so a "was NOT" filter also caught this actor's own "was not created" warning. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
0d6669d5ff |
feat(mesh): MeshCommand envelope + generation-suffixed ClusterClient factory
Phase 2 Task 2. Two small pieces the comm actors are built on. MeshCommand carries a command plus the DPS topic it would have been published on, so the admin singletons stop knowing which transport is in force. Without it the dark switch would have to be threaded through five publish sites across two actors, each free to drift. Central-side only -- it never crosses the wire, so it carries no serialization contract. DefaultMeshClusterClientFactory appends a generation counter to the actor name. Context.Stop is asynchronous and the name stays reserved until termination completes, so recreating the same name inside one message handler throws InvalidActorNameException -- which is exactly what a contact-set change does (stop the old client, create the replacement, one handler). Ported from the sister project, where it was a shipped bug fix rather than foresight. Sabotage-verified: freezing the name to a constant reddens both name tests with InvalidActorNameException. Dropped a third test I had written -- it asserted ClusterClientSettings directly and never touched the factory, so it was coverage theatre. Replaced with a comment saying where contact propagation IS covered (the Task 8 boundary test, the only place it can actually fail). Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
7654f24dab |
test(harness): serialize Host.IntegrationTests; drop the reconciler's unused singleton proxy
Closes the last branch-only failures. Measured across five full-suite runs on this branch versus two on master in a clean worktree: master failed only AbCip in this assembly (2/2), while the branch additionally failed one or two E2E deploy tests, rotating between DeployHappyPath, DriverReconnect and EquipmentNamespaceMaterialization (4/4). Each passed in isolation, and the assembly on its own was 195/201 throughout — the failures only appeared under full-suite CPU contention. Cause is contention, not correctness. Every test here builds a real two-node Akka cluster — two Kestrel hosts, two ActorSystems with remoting, six admin singletons, an EF context, a deploy pipeline — and xUnit ran them concurrently with each other and with every other assembly. That was survivable while the coordinator sealed instantly on an empty expected-ack set; now that it waits for a real ack from every configured node, these tests measure an end-to-end round-trip and starve. Serialising this one assembly makes them deterministic: 195/201 with only AbCip_Green_AgainstSim, and the full-solution failure set is now IDENTICAL to master's — 7 shared, zero branch-only. Cost is wall-clock for this assembly, 40s -> 3m35s; it is a handful of heavyweight E2E tests, not a broad unit suite, and a widened timeout alone did not fix it (tried first, at 45s). Also drops createProxyToo for the address reconciler. Nothing resolves ClusterNodeAddressReconcilerKey from the registry — the actor only reads cluster state and logs — so the proxy was a second actor per node hunting for a singleton nobody addresses. Kept because it removes dead machinery, not because it fixed the flakiness; measured on its own, it did not. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
ee69caa270 |
feat(config): add ClusterNode.MaintenanceMode — the hatch the live gate proved missing
Phase 1 gate step 4 failed: setting Enabled = 0 to take a node out of service returns 422 ClusterEnabledNodeCountMismatch, because DraftValidator.ValidateClusterTopology requires the enabled-node count to equal ServerCluster.NodeCount. On a Warm/Hot pair — every cluster on the rig, and every cluster in the target topology — Enabled can therefore never be the maintenance hatch. The plan called step 4 "the check that makes step 3 acceptable to ship", so Task 3's behaviour change was not shippable as it stood: a node down for maintenance would block every deployment to its cluster. The two rules were each reasonable and contradictory together — the validator reads Enabled as "part of the declared topology", Phase 1 additionally read it as "expect an ack". Split the meanings rather than weaken either rule: Enabled part of the declared topology (validator, untouched) MaintenanceMode expected to participate now (coordinator + reconciler) Rejected alternatives: counting configured rather than enabled nodes (drops the guard against booting a pair into InvalidTopology); downgrading the rule to a warning (weakens a deploy gate for everyone); shipping with no hatch. Sabotage: dropping !n.MaintenanceMode from the coordinator query turns the new test red. It also asserts both nodes remain Enabled, so the fix cannot quietly regress to disabling the row after all. Configuration.Tests 95/95, ControlPlane.Tests 101/101, solution builds clean. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
90c1302f71 |
feat(fleet): reconcile ClusterNode dial targets against live membership
ClusterNode.AkkaPort and the node's own Cluster:Port are the same fact in two places and nothing made them agree. Phase 2 dials the row instead of gossiping, so a node binding 4054 while its row says 4053 becomes unreachable from central — and the symptom is a silent absence of acks rather than an error. That is the shape of the Modbus/ModbusTcp and TwinCat/Focas drifts already in this repo. Since Phase 1 also made the rows the deploy path's expected-ack set, drift already costs a failed deployment today. Implemented as the plan's preferred option: an admin-role singleton comparing rows against the membership an admin node can already see, rather than each driver node asserting its own row — Phase 4 removes the driver nodes' ConfigDb connection, so a self-assertion written there would have to be deleted again. Three shapes, split by severity: a row whose dial target disagrees with its own NodeId and a running node with no row are Errors; an enabled row with no matching member is a Warning, because a node down for maintenance is a legitimate state. Findings are logged only when the set changes — a check that reprints the same warning every sweep trains operators to filter it out. Documented limitation: Phase 2 must revisit this. Once the fleet splits into one mesh per cluster an admin node cannot see site members, and every site row would report EnabledRowNotInCluster forever. Sabotage: removing the change-detection guard turns the repeat test red. ControlPlane.Tests 100/100. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
d88e245503 |
feat(deploy): coordinator sources its expected-ack set from ClusterNode rows
Cuts ConfigPublishCoordinator's last genuinely mesh-bound dependency: central must be able to name the nodes a deployment is for without sharing a gossip ring with them, because Phase 2 splits the fleet into one mesh per Cluster. Behaviour change, confirmed before implementing: a configured node that is switched off is now expected, so deploying while it is down fails at the apply deadline instead of sealing green without it. Under the membership rule the operator was told the fleet was deployed when it was not. ClusterNode.Enabled = 0 is the maintenance hatch, and the deadline log now names the silent nodes and points at that hatch — "4/5 acks landed" would leave an operator reading logs on five machines. Two assumptions are now documented on the class rather than left implicit: every ClusterNode row is a driver node (the DB has no role column and deliberately does not gain one — it would drift from Cluster:Roles), and ServerCluster.Enabled is not consulted because nothing else consults it. Dropped from the plan: cluster-scope filtering of the expected set. There is no cluster-scoped deployment to filter on — Deployment has no ClusterId, ConfigComposer always snapshots the whole DB, and ResolveClusterScope is node-side self-scoping of a fleet-wide artifact. Positive control: reverting DiscoverDriverNodes to the membership scan turns three of the four new tests red. The fourth pins the seal-empty branch and is documented as derivation-insensitive rather than left to look like coverage. ControlPlane.Tests 90/90. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
69b697bc3e |
feat(redundancy): manual failover service — graceful Leave of the driver Primary
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
50b55ee4b9 |
fix(redundancy): elect the Primary by cluster age, not by address
RedundancyStateActor derived the driver Primary from ClusterState.RoleLeader("driver").
Akka offers two different notions of a "first" member and they are not the same
one: role leader is the lowest-ADDRESSED Up member (host, then port), while
ClusterSingletonManager places singletons on the OLDEST — lowest up-number.
They agree on a freshly-formed cluster, which is why every existing test passed.
They diverge after any restart: the restarted node re-joins as the youngest while
keeping its address, so if it holds the lower address it becomes role leader while
the singletons stay put. The snapshot would then name a Primary that is not
hosting the work, and every Primary-gated surface follows it — inbound device
writes, native-alarm acks, the fleet-wide alerts emit, and the alarm-history
drain would all enable on the wrong node while the node actually running the
singletons stayed gated off.
BuildSnapshot now selects the oldest Up member carrying the driver role, matching
singleton placement. Leaving members are excluded: a node handing its singletons
over must not be named Primary.
NodeRedundancyState.IsRoleLeaderForDriver is renamed IsDriverPrimary, and
NodeHealthInputs.IsDriverRoleLeader likewise. Keeping the old names would have
left the wire contract asserting a derivation the code no longer uses — the same
drift that made this defect invisible.
Proven by a real two-node cluster rather than a mock. RedundancyPrimaryElectionTests
binds the first-joining node to the HIGHER port, so oldest and lowest-address name
different nodes, and includes a fixture assertion that the divergence actually
occurred — without it the real assertion could pass for the wrong reason. Positive
control: restoring the RoleLeader derivation turns exactly the two election tests
red while the fixture check stays green.
Runtime.Tests 440 passed, ControlPlane.Tests 82, Cluster.Tests 36.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
|
||
|
Joseph Doherty
|
96471d2345 |
refactor(adminui): B2-WP8 DriverTypeNames rewire + retire routed driver-authoring flow
Part 1 — rewire the three DriverType dispatch maps from hand-authored string
literals to the canonical DriverTypeNames constants (single source of truth):
- TagConfigEditorMap, TagConfigValidator (AdminUI)
- EquipmentTagConfigInspector (ControlPlane)
This FIXES the live drift ("TwinCat"->TwinCAT, "Focas"->FOCAS). The maps are
OrdinalIgnoreCase so behavior is identical; the value is no future drift.
Calculation is not added to the inspector (no CalculationTagDefinitionFactory.Inspect).
Added TagConfigDriverTypeNameGuardTests pinning the maps to the constants.
Part 2 — retire the routed /clusters/{id}/drivers driver-authoring flow (/raw
Waves A-C now cover driver/device/tag authoring end-to-end):
- Deleted DriverTypePicker, DriverEditRouter, the 8 *DriverPage shells, and the
ClusterDrivers list page.
- Removed the "Drivers" tab from ClusterNav. Routes /clusters/{id}/drivers* now
simply do not exist -> clean 404.
- The Forms/DeviceForms bodies, shared driver sections, and address pickers used
by the /raw modals are untouched.
Part 3 — updated PageAuthorizationGuardTests census: removed the 10 deleted
ConfigEditor pages + ClusterDrivers, updated group-count comments (ConfigEditor
21->11, AuthenticatedRead 17->16). Removed the now-dead Clusters.Drivers usings
from the 4 *DriverPageFormSerializationTests + the guard test.
Claude-Session: https://claude.ai/code/session_01LVneM3eh1UtJxEisFXgmox
|
||
|
Joseph Doherty
|
e81ac352ed |
v3 B1-WP4/WP5: pipeline rewire to greenfield schema + dark address space
- ConfigComposer: snapshot v3 tables (RawFolders/TagGroups/UnsTagReferences), drop the retired Namespaces snapshot; RevisionHash follows the new blob. - DeploymentArtifact: compute each raw tag's RawPath (RawFolder->Driver->Device-> TagGroup ancestry via shared RawPathResolver); per-driver RawTags + merged DriverConfig+DeviceConfig config injected into DriverInstanceSpec via DriverDeviceConfigMerger; ParseComposition emits an empty EquipmentTags set (DARK until Batch 4); EquipmentNode driver/device hooks always null; cluster scoping attributes equipment by UNS line only. - AddressSpaceComposer: rewritten to the v3 shape (no Namespace/NamespaceKind, no dropped Tag/Equipment fields); emits empty EquipmentTags (parity mirror). - DriverHostActor: fan-out/write map tuple member FullName -> RawPath. - Commons: new RawPathResolver (shared identity authority) + DriverDeviceConfigMerger (single/multi-device endpoint+RawTags merge). - DraftValidator: v3 rules — raw-name charset (RawPaths.ValidateSegment), historized effective-tagname <=255, UNS effective-leaf uniqueness across UnsTagReference/VirtualTag/ScriptedAlarm; DraftSnapshot(+Factory) carry the new tables. - AdminOperationsActor: tag-config inspection resolves driver via Device (no EquipmentId/DriverInstanceId on Tag). - Tests: RawPathResolver/merger unit tests (Commons.Tests, green) + DeploymentArtifact RawPath/dark test (Runtime.Tests, awaits WP6 corpus fixes). |
||
|
Joseph Doherty
|
8e8dd2e824 | v3 batch1 Wave-B reconcile: rename protocol parsers -> *TagDefinitionFactory in EquipmentTagConfigInspector | ||
|
Joseph Doherty
|
aafb9d4929 |
v3(b1-modbus): RawPath identity seam — RawTagEntry-driven Modbus driver
Wave-B EXEMPLAR. Retire the pre-declared/blob-parse tag model; the driver now builds its RawPath -> definition table from the artifact's RawTagEntry set via a pure mapper. - Contracts: rename ModbusEquipmentTagParser -> ModbusTagDefinitionFactory; TryParse(reference) -> FromTagConfig(tagConfig, rawPath) (def.Name = rawPath, no leading-brace heuristic); add inverse ToTagConfig(def) serializer; keep all strict-enum reads + guards; also read stringByteOrder/deadband/coalesceProhibited so a RawTagEntry round-trips the full authored def. Inspect() unchanged. - Options: Tags(IReadOnlyList<ModbusTagDefinition>) -> RawTags(IReadOnlyList<RawTagEntry>). - Driver: _tagsByName -> _tagsByRawPath (Ordinal); resolver byRawPath-only; build the table from RawTags at Initialize threading entry.WriteIdempotent onto each def; Discover/Teardown updated. - Factory: remove ModbusTagDto/BuildTag/ValidateStringLength + ConfigDto.Tags; bind RawTags from driver-config JSON. - CLI ModbusCommandBase.BuildOptions: serialise typed defs -> RawTagEntry via ToTagConfig. - Tests: migrate Tags= -> RawTags via ModbusRawTags.Entries helper; parser tests -> FromTagConfig(rawPath); factory String-length/enum tests re-seated on the mapper seam. - Propagated rename to ControlPlane EquipmentTagConfigInspector (outside Modbus projects). Modbus.Tests 315/315, Addressing.Tests 161/161, Cli.Tests 72/72 green. Docker-gated Driver.Modbus.IntegrationTests left untouched (won't compile against the new API; expected). |