8b10d327281e4e3dfdffb687439b02f71cb7015c
154 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Joseph Doherty
|
4d2dfb5b50 |
test(mesh): step 7 — replace the boundary test with MeshGrpcBoundaryTests and sweep the comments
MeshGrpcBoundaryTests replaces the MeshClusterClientBoundaryTests deleted in step 1. Three
self-seeded single-member clusters sharing the ActorSystem name, joined ONLY by gRPC, driving
the REAL CentralCommunicationActor -> MeshGrpcCommandDialer -> Kestrel h2c ->
MeshCommandAuthInterceptor -> NodeCommandGrpcService -> NodeCommunicationActor -> EventStream:
1. No_cross_mesh_Akka_association_forms_between_central_and_either_site — the program's own
point, kept as a test rather than a live-gate note: each cluster stays single-member and
none ever holds another's address as a member.
2. One_dispatch_through_central_reaches_a_node_in_every_cluster_mesh — the fan, with each
site holding its OWN per-cluster PSK, so a dial that selected the wrong cluster's key
would be refused at the far side.
3. Falsifiability_control_a_dial_carrying_the_wrong_cluster_key_delivers_nothing — without
it, (2) cannot distinguish "the gRPC boundary carried it" from "something else in the
process did". The same node then accepts the right key, so the silence is the auth gate
and not a dead endpoint.
All three run fully offline — no Docker fixture — and pass.
Comment sweep across every remaining clusterclient|receptionist hit in src/ and tests/:
the program label becomes "mesh gRPC command plane", and every comment that described the
deleted transport as live now describes what actually runs (Dps = pair-local default,
Grpc = cross-mesh). mesh_transport.proto's header/service comments are reworded; no message,
field or enum is touched — the contract is locked. MeshCommActorPathTests is reshaped rather
than retired: the two paths still exist and still matter, but what a mismatch costs is now a
misleading Unavailable from the gRPC ingress rather than a silently dropped receptionist send.
The only surviving mentions of the old names are the ones that must survive: the fail-closed
rules that name "ClusterClient" as a retired value refused at boot, their tests, and this
test class's own account of what it replaces.
|
||
|
Joseph Doherty
|
667639e6ab |
fix(mesh): per-endpoint deadline + review fixes — failover survives a hanging central
CRITICAL. The ack deadline was one absolute DateTime computed once for the whole endpoint walk. Against the characteristic WAN failure — a central that HANGS (SYN black-hole, frozen host) rather than refusing — endpoint 1 burned the whole AckTimeoutSeconds and endpoint 2 was then dialled with a deadline already in the past: instant DeadlineExceeded, no packet leaving the node, the surviving central never really tried. Failover looked like it ran and did not. It also contradicted MeshTransportOptions' own documented "applied per endpoint, not per ack". The deadline is now computed fresh inside the loop; headers stay hoisted (read-only, sequential walk). Two test gaps that let that survive are closed: nothing pinned the ackRouter gate (narrowing it back to ClusterClient-only compiled and passed every test while dead-ending every Grpc ack to DPS), and the deadline fact's lower bound accepted any positive timeout at all. Both new pins are mutation-verified. Docs corrected where they overstated the case: duplicate acks CANNOT mis-seal a deployment — ConfigPublishCoordinator.HandleAck keys _acks by NodeId behind an expected-set guard, so a duplicate overwrites its own entry. The single-endpoint rule is defence in depth and a bound on wasted work, not a correctness dependency; and the walk is at-least-once on the deadline path, not at-most-once. |
||
|
Joseph Doherty
|
39db75083d |
feat(drivers): every driver reports the authored tags it skipped (Gitea #541)
A driver builds its RawPath -> definition table at init and tolerates a
per-tag miss: an unmappable TagConfig, an unparseable address, a
capability-matrix rejection. That tolerance is right -- one bad tag must not
take down the other few thousand. What was missing is any statement of the
fact afterwards. A skip is not an error and happens before any read, so the
driver stays Healthy, errors/5 min stays 0, Last error stays empty, and Last
read advances live off the tags that did map. On the wonder Makino in #540
that reading held while focas-z-34184 ignored 20 of its 22 tags.
New optional capability ISkippedTagReporter, backed by a shared
SkippedTagLedger so the reset/commit/cap rules are one implementation rather
than one per driver. The report -- authored count, exact skipped count, and a
capped sample of RawPath + reason -- rides DriverHealthChanged to a new Tags
column on /hosts, over the Phase 5 telemetry stream. No table: a driver-only
node has no ConfigDb to write to, the lesson from #521.
The issue named eight drivers. It is twelve -- every driver in
DriverTypeNames.All has an authored-tag skip path. Calculation (no scriptId),
MTConnect (no readable DataItem id), Galaxy (no attributeRef) and OpcUaClient
(no nodeId) were unlisted, and OpcUaClient's was the quietest of the twelve:
NamespaceMap drops such an entry with a bare `continue` and not even a log
line. SkippedTagReporterFleetTests pins the invariant reflectively over every
IDriver in src/Drivers/, so a thirteenth cannot re-introduce the silence.
Sql drops a tag at TWO points and its report is the union of both: a blob
that does not map, AND an identifier the 8.1 catalog gate rejects against the
live catalog. The second is what schema drift produces -- a renamed column, a
dropped view -- months after a deployment sealed green, and reporting only
the first would have that driver claim "0 skipped" while polling half its
tags, which is worse than reporting nothing because the column would then be
actively reassuring.
Deliberately additive to driver state, never a change of it. Latching
non-Healthy for a config problem would satisfy both halves of the #527
comms-loss conjunction -- 90 s without a successful read AND non-Healthy --
and a driver whose tags were ALL skipped has no successful reads by
construction, so it would reconnect-loop forever against a healthy device.
Three details that are load-bearing rather than incidental:
- The skip digest is part of the publish fingerprint. A skipped tag leaves
every other fingerprint component identical to a clean driver's, so without
it PublishHealthSnapshot's dedup swallows exactly the publish carrying the
signal -- the trap that already bit the rediscovery and host-status
channels. The digest covers the sample, not just the counts: a redeploy
that fixes one tag and breaks another leaves both counts unchanged.
- The count is exact; only the sample is capped, and the tooltip says so when
it is showing fewer than it counted. A truncated list that looked complete
would have an operator fix twenty tags and believe the job done.
- The ledger stages and commits, so an init that throws part-way leaves the
previous report standing. A partial snapshot would read as "we now skip
fewer tags", the opposite of the truth.
The proto field needs no companion presence flag, unlike host_statuses: a
singular message field carries real presence in proto3, and a has_ flag would
collide with protoc's generated HasSkippedTags.
Verified red-first three ways: dropping SkippedTagDigest from the fingerprint
fails the two dedup tests, removing the seam from one driver fails the fleet
guard, and dropping the catalog-gate leg fails the Sql union test. Full
solution builds with zero new warnings against
|
||
|
Joseph Doherty
|
e7fbe97759 |
fix(historian): gate historization capture on the Primary, quarantine a poison tag (#537, #538)
Both defects were found by #491's live gate against a real HistorianGateway and filed rather than fixed at the time, because each one's fix is a design decision rather than a patch. #537 — both nodes of a redundant pair captured every value, so the historian received duplicate live samples for every historized tag (measured: the gateway's write counter advanced twice per value and halved when one central was stopped). The recorder now reads a capture gate per value, fed from the new PrimaryGatePolicy.ShouldRecordHistorizedValues through IHistorizationRoleView — a third verdict on the same RedundancyRoleView object, registered unconditionally on driver nodes rather than alongside the alarm section's IRedundancyRoleView (which only exists when that section is enabled; piggybacking is the #511 registration accident). Two things are deliberate and easy to get backwards: * It gates the CAPTURE, not the drain — the opposite of the alarm sink. That queue is in LocalDb and replicates, so the Secondary already holds the Primary's rows. The historization outbox is a node-local FasterLog replicating to nobody, so a drain-only gate would leave the standby appending forever to a queue no one empties: unbounded disk, then a flood of stale duplicates the moment it is promoted. Entries a node already captured are still drained by that node. * The peer it stands down for must RECORD THE SAME REFS, not merely hold the same rows — a Primary in this node's own per-cluster mesh. A node carrying no cluster-scoped role cannot establish that and never stands down: under a fleet-wide election the elected Primary is usually running a different cluster's configuration, and deferring to it would end this cluster's historization silently, with the drain reporting success on an empty outbox forever. That is the scope mismatch that once suspended the alarm drain on every node of the rig. An unknown role captures, matching the alarm drain — a duplicate sample is recoverable, a hole in the trend is not. #538 — the drain writes every per-tag group and then acks all-or-nothing (deliberate: RemoveAsync truncates THROUGH an id, so partial acking could drop older unwritten entries). One permanently failing tag therefore pinned the FIFO head while every healthy tag in the batch was re-submitted to the historian on every pass, indefinitely — duplicate samples and an outbox that never truncated, observed live at 8.3 MB from a single unprovisionable tag. The drain now tracks consecutive failures per historian tag and abandons that tag's entries once it exceeds MaxWriteAttempts (new, default 5), so the head advances. The counter only advances on a pass where some OTHER tag wrote successfully: that is what separates "this tag is poison" from "the backend is down". A count-only quarantine looks identical on the poison-tag case and silently discards the whole backlog during an ordinary gateway restart, turning a duplicate-samples bug into a lost-history one. Abandonment is also deferred until every failing tag is over budget, so nothing is discarded on a pass that could not have advanced the head anyway. Each abandonment logs a Warning naming the tag; the total rides RecorderStatus.AbandonedEntries. Capacity also gains a non-zero default (500,000 entries) as the disk backstop for the one case the quarantine deliberately cannot resolve — a historian refusing every write, where retaining is correct. Set 0 to restore unbounded retention. Every new test was falsified by reverting the production change it guards, and each one reddens alone: * drop the backend-reachable evidence -> only A_backend_that_fails_every_tag_abandons_nothing * never abandon -> only A_permanently_failing_tag_is_abandoned_so_healthy_tags_stop_being_rewritten * latch the capture gate at construction -> only The_capture_gate_is_re_read_per_value... * treat any Primary as a scope peer -> only the two scope-mismatch cases Full suite green. (Four OpcUaClient.Browser failures during the run were the opc-plc fixture being down on the shared Docker host; brought it back up and they pass.) Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo |
||
|
Joseph Doherty
|
3a6b2a1daf |
feat(acl): variable-depth scope model + the scopePaths producer (#520 stage 3)
Makes the /raw device tree addressable by an ACL rule, and builds the
scope-path producer PermissionTrieBuilder has accepted since it was written
but which had no caller anywhere.
The plan for this stage said "add NodeHierarchyKind.Raw, whose path is the
RawPath segments Folder/Driver/Device/TagGroup/Tag". That shape is wrong.
RawFolder.ParentRawFolderId and TagGroup.ParentTagGroupId are both
self-referencing, so folders and tag groups each nest without bound and a real
RawPath is folder…/Driver/Device/group…/Tag. NodeScope was fixed-arity and
PermissionTrie.CollectMatches was five hardcoded steps, so the scope model
could not address a real raw tree at all. NodeScope is now {ClusterId, Kind,
Segments} and the walk is a segment loop. The trie STRUCTURE was already
variable-depth (NodeAclPath.Segments is a list, Children is a recursive
dictionary), so only the walker and the scope record had to change.
Four decisions worth keeping:
- The namespace level became the realm. Level 1 gated on NodeScope.NamespaceId,
but v3 has no Namespace entity — the snapshot was retired when the address
space went dual-namespace, and the identifier appears nowhere in production
outside NodeScope. So the level could only ever fail to match, which silently
made every sub-cluster grant unreachable while cluster grants kept working.
It is now a realm token derived from the scope's Kind, not carried on it, so
the walk and the producer cannot disagree about its spelling.
- Provenance comes from the granting row's own ScopeKind, recorded on TrieGrant
at build time. Depth-derived provenance could not name a raw level at all,
since the depth of "Device" varies with folder nesting.
- The trie comparer moved from OrdinalIgnoreCase to Ordinal. Raw sibling-name
uniqueness is ordinal, so Speed and speed can both exist under one device and
a grant on either would have matched both.
- The null-scopePaths "deterministic test mode" is deleted. It planted
unresolvable rows at a direct child of the trie root, a position the walker
cannot reach, so the grant looked placed and could never match — and every
test using that mode proved nothing about production placement. scopePaths is
now required, and a row that does not resolve is dropped with a diagnostic.
A scope resolves to a LIST of paths: a raw tag projected into N equipments is
reachable at N+1 NodeIds and a write through any of them lands on the same
driver reference, so a Tag-scoped grant covering only the raw node would be
bypassable through a UNS one. The lookup is keyed on (ScopeKind, ScopeId), not
ScopeId alone — ScopeId has no FK and no type discriminator, so an id collision
across entity types would hand a grant the wrong node's paths.
Also closed here, all found by review rather than asked for:
- NodeAcl rows had no deploy-time validation of any kind and were not reachable
from DraftValidator at all (DraftSnapshot carried no NodeAcls). They are now
gated for dangling ids, cross-cluster scopes, blank/contradictory ScopeId, the
retired FolderSegment kind, and undefined ordinals.
- ConfigDbApiKeyStore conferred roles from Cluster-scoped grants authored in ANY
cluster, because it filtered on scope kind with no ClusterId predicate while
ArtifactApiKeyStore read an already-cluster-filtered snapshot. On a fused
admin+driver node that was a cross-cluster over-grant. Its doc comment claimed
the two stores matched.
- The two AdminUI ACL banners still said enforcement was LDAP-group -> role
mapping, which stages 1/2 made false in both directions.
- The /apikeys page was never classified in PageAuthorizationGuardTests, so that
test has been red since the key-minting commit merged — I did not run
AdminUI.Tests before merging stage 4.
Falsified rather than assumed: 8 mutations against the trie tests, 5 against the
deploy gate (including one proving a cross-cluster row is not silently
mis-diagnosed as unresolved), a staged red->green for the over-grant regression,
and the realm gate, the comparer and the malformed-path guard each broken
individually to confirm only their own tests go red.
Node ACLs are still not ENFORCED — nothing constructs a NodeScope or calls the
evaluator yet. That is stages 5 and 6; this stage makes the model able to
express what those will enforce.
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
|
||
|
Joseph Doherty
|
a438ac1640 |
feat(acl): keys + ACLs reach driver-only nodes via the artifact (#520 stage 4)
A driver-only node has had no ConfigDb since per-cluster mesh Phase 4, so it registered NullOpcUaUserAuthenticator and denied every UserName token. Its keys now ride the deployment artifact instead. - ConfigComposer snapshots ApiKeys alongside the NodeAcls it already carried. Only the peppered HMAC ships, never a secret, and the pepper is node config that is deliberately not in the artifact. - DeploymentArtifact.ParseDataPlaneAuthority decodes both node-side. Grants are scoped to the node's own cluster; keys are not, because ApiKey carries no ClusterId — a key is a fleet-wide credential and the ACL rows are what bind it. - OpcUaPublishActor installs them on DataPlaneAuthorityState on every apply, BEFORE its plan.IsEmpty early-return: an artifact that only revokes a key produces an empty address-space plan, so feeding after that return would leave the revoked key working until an unrelated structural change redeployed. - ArtifactApiKeyStore serves them to session activation; a node with a ConfigDb keeps ConfigDbApiKeyStore, whose per-request read is what makes revocation immediate there. - OtOpcUaNodeManager gains DataPlaneAuthority — the cluster identity blocker #4 named. NodeScope.ClusterId is required and the evaluator fails closed on a mismatch, so until now no scope it built could have evaluated. Nothing reads it until stages 5/6. The generation is a node-local COUNTER driven by the RevisionHash, not derived from its bits. PermissionTrieCache treats the highest GenerationId as current, so a generation taken from hash bytes would install a lower number about half the time and the cache would go on serving the previous deployment's ACLs, silently. Sessions are not stamped with it yet — that belongs with the first seam that reads a UserAuthorizationState, in stage 5, rather than being built now and consulted by nothing. Live-gated on docker-dev against site-a-1 (driver-only, no ConfigDb): key authenticates, wrong secret denied, both halves of the revocation asymmetry observed, and a SITE-A grant confirmed not to reach site-b-1. Key minted with an independent Python HMAC so the verifier was checked against something other than itself. Rig left clean. Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo |
||
|
Joseph Doherty
|
5a7a08deca |
fix(drivers): detect mid-run comms loss and recover from it (#527)
DriverInstanceActor.DisconnectObserved had no production sender. The only paths into Reconnecting were a failed *initial* connect and an operator's Force Reconnect, so a driver that connected and then lost its device stayed Connected indefinitely: publishing Bad for every tag, never re-initialising, and NOT recovering when the device returned. PublishHealthSnapshot read GetHealth() for reporting only and never acted on a driver reporting itself unhealthy. Both comms-loss quality paths -- #477's native-condition annotation and #481's scripted-alarm input fan-out -- hang off ConnectivityChanged, so both were correctly wired to a signal that never fired. The Connected health tick now evaluates a watchdog and self-sends DisconnectObserved when the driver reports a non-Healthy state AND its LastSuccessfulRead has been frozen for DefaultCommsLossTimeout (90s). The conjunction is the design, and the issue's own proposal -- "non-Healthy for N consecutive polls" -- is wrong. Degraded means *connected but some data is bad*; several drivers write health per tag inside the read loop (ModbusDriver.ReadAsync), so one unreadable address latches Degraded forever against a perfectly reachable device, and a state-only watchdog would reconnect that driver every 90s indefinitely -- worse than the bug. LastSuccessfulRead is the honest evidence: preserved across Degraded/Faulted writes, advanced by any success. The state check remains the second conjunct so an idle Healthy driver with nothing subscribed is not declared dead for not reading. Verified across all 12 drivers that a sustained failure does reach a non-Healthy state -- nine poll drivers via Degraded, MQTT via live connection state, Galaxy via its reconnect supervisor. CalculationDriver reports Healthy while running and so can never trip, which is correct: it has no transport. Also: the Reconnecting retry now calls ShutdownAsync before InitializeAsync. ModbusDriver (_transport) and MqttDriver (_connection) overwrite an IAsyncDisposable field inside InitializeAsync without disposing the previous one, so re-initialising without a teardown leaks a socket per attempt -- latent while only a manual Force Reconnect reached this path, but the watchdog makes it automatic and unbounded (~360 leaks for an hour-long outage at the 10s retry). Tests: 8 new in DriverInstanceActorCommsLossWatchdogTests, each absence assertion carrying a positive control (a proven count of processed health-poll ticks). Falsified five ways -- dropping either conjunct reds exactly the test written for it, and removing the teardown, the null baseline, or the watchdog call each reds its own guard. Live gate PASSED on docker-dev (Modbus + otopcua-pymodbus-standard), run twice: - fixture stopped -> watchdog fires at ~108s, "declaring comms lost" - fan-out observed at Debug: "pushed BadNoCommunication for 2 mux ref(s)" (#481 Layer 4) and the #477 native-condition annotation on a driver owning conditions - fixture restarted -> reconnected in ~8s with NO node restart; both the raw (ns=2) and UNS (ns=3) NodeIds read Good with fresh timestamps Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo |
||
|
Joseph Doherty
|
c81cef39f5 |
feat(alarms): bridge driver comms-loss into scripted-alarm input quality (#481)
Layer 4 of the condition-Quality chain (#477 -> #478). #478 gave a scripted condition the worst quality of its input tags, but only for inputs whose driver PUBLISHES a Bad data change. A poll driver whose device goes unreachable publishes nothing — it raises ConnectivityChanged(false) and goes silent — so the engine held its last Good value, WorstInputStatus stayed Good, and worse, AreInputsReady stayed true, so the predicate went on evaluating a stale value after the source had died. `DriverHostActor.FanCommsLossToDependencyMux` pushes a synthetic BadNoCommunication with a null value into the dependency mux for every value ref the disconnected driver owns. The issue asked for a driver -> all-tag-refs map to be built for this. It already exists: `_nodeIdByDriverRef`, keyed (DriverInstanceId, RawPath) and rebuilt from the composition on every apply. Alarm-bearing tags live in the separate `_alarmNodeIdByDriverRef` and are deliberately excluded — a native condition is not a mux value and takes the Part 9 quality path instead. Nothing is pushed on reconnect, deliberately: publishing Good would assert a value quality this actor has not observed. Recovery rides the driver's own republish, and the mechanism was verified rather than assumed — the Connected -> Reconnecting edge calls DetachSubscription, which nulls the subscription handle, so the reconnect's ResubscribeDesired builds a FRESH subscription whose PollGroupEngine initial push runs with forceRaise: true. Without that detach the poll engine would compare against its retained LastValues, see an unchanged value, raise nothing, and strand these refs Bad forever — the value-dedup trap that bit the v3 Batch 4 virtual-tag reassert. Closes #478 Finding 3, which the above depends on. WorstInputStatus skipped every input whose VALUE was null as "no data yet" — and Bad-with-no-value is exactly the shape a comms-lost driver produces, so the skip was hiding the case that matters most; without this half the fan-out above would be silently inert. The upstream source now stamps a never-pushed path BadWaitingForInitialData — the honest Part 4 code for that state — and the engine skips that specific code rather than every null value. Deleting the null-skip outright is the obvious fix and is wrong in the other direction: it makes every not-yet-published input drag its condition Bad at deploy, flooding the quality path with load-time annotations, which is what the skip existed to prevent. Both mistakes are pinned by a test pair that goes red under the opposite error, and both were falsified by reverting the production code: old null-skip restored -> A_pushed_Bad_carrying_no_value RED null-skip deleted outright -> An_input_that_was_never_pushed RED fan-out call removed -> the three DriverHostActor tests RED (1 of 7) The never-pushed test was vacuous on first writing and is worth recording: the quality bucket only emits on CHANGE, so the load-time evaluation is the one that flashes Bad, and subscribing to OnEvent after LoadAsync made the assertion unfalsifiable — it passed against the broken behaviour. It now subscribes first. `FakeUpstream.ReadTag` returned 0x80340000 (BadNotConnected) for an unset path, a different Bad code from production's placeholder. Harmless while any Bad behaved alike; load-bearing now, so it is aligned and pinned by a reflection guard asserting the engine's private constant agrees with what the production source stamps. Behaviour change to weigh, flagged in the issue and confirmed: virtual tags computed from a comms-lost input now go Bad too. That follows from using the shared mux and reads as correct, but it is broader than scripted alarms. Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo |
||
|
Joseph Doherty
|
b815446940 |
refactor(drivers): delete GetMemoryFootprint / FlushOptionalCachesAsync (#525)
Both `IDriver` members had no production call site. Their only consumer was `MemoryTracking`, deleted with the driver-tier recycle machinery in #522, and `GetMemoryFootprint`'s doc-comment claimed "polled every 30s by Core" — which had not been true since v3, even before #522 removed the poller. Owner decision on #525 was (a) delete rather than (b) build the process-wide watchdog that would consume them. The driver-side implementations are in git history and were mostly reusable; what was always missing is the trigger, and a member nobody calls is how the misleading doc-comment survived this long. Removed from `IDriver`, all 12 driver implementations, and every test stub — plus the support code that existed only to feed them, which is the part a member-only deletion would have left behind as dead weight: - MTConnect's three per-entry byte-weight constants, its cached `ProbeModelDataItemCount`, and `CountDataItems` — a full device-tree walk run on every cache fill to populate a number with no remaining reader. - OpcUaClient's `_discoveredNodeCount` field and the discovery-pass write to it. - MQTT's `ApproxBytesPerAuthoredTag`. Two behavioural notes, neither a change in what the software does: - `AbCipTemplateCache` now has no production reset path. Its only caller was AB CIP's flush, which itself had no caller — so nothing changes except how visible the fact is. Recorded on the type, along with the correction that its comment's claim about a device `ReinitializeAsync` also flushing it was never true. A UDT redefined on a live controller keeps its stale shape until the driver instance is rebuilt. - MTConnect's `/probe` cache is still dropped, by `InvalidateProbeModel()` on an Agent restart. That is now `internal` rather than `private` so the tests which used the flush purely to empty the cache drive the real path instead — they now cover a case that actually occurs in production. Several unrelated doc comments cited `FlushOptionalCachesAsync` as the canonical CAS example and are repointed to it. Tests: the ones that existed only to assert on the removed members are deleted (3 OpcUaClient, 3 Galaxy, 2 MQTT, 1 MTConnect, 1 AbCip). Worth noting one was vacuous — `FlushOptionalCachesAsync_resets_footprint_counter` asserted a footprint of 0 before the flush and 0 after with no discovery in between, so it would have passed against a flush that did nothing at all. Full suite green except 7 failures verified identical on master before and after (4 AbLegacy integration with the compose profile down, 3 Host.IntegrationTests), checked by name rather than assumed. Docs corrected where they described this as live: `DriverLifecycle.md` (which carried the #525 placeholder), `drivers/AbCip.md`, `drivers/AbLegacy.md`, and `v2/driver-stability.md` — whose "polls this every 30 s" + per-driver `DriverConfig` memory budget were never built as written and are now banner-ed as design intent rather than description. Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo |
||
|
Joseph Doherty
|
7f2108dcf2 |
feat(mqtt): Sparkplug primary-host STATE publishing, Primary-gated (#511)
`ActAsPrimaryHost` was accepted by config and by the AdminUI form and did nothing but
log a warning. It now delivers the whole three-part contract or none of it: a retained
`ONLINE` on `spBv1.0/STATE/{hostId}`, a matching retained `OFFLINE` armed as the MQTT
Last Will, and an explicit `OFFLINE` on a clean stand-down.
**The claim is a dedicated connection, not a message.** A Will can only be registered in
the CONNECT packet — it cannot be added to a live session, changed, or withdrawn. So a
claim that is acquired and released at runtime cannot ride the long-lived ingest session.
`SparkplugPrimaryHost` opens its own connection (client id `{clientId}-spbhost`, forced
clean-session) when the node gains Primary and closes it when the node loses Primary; the
connection's lifetime *is* the claim, so every leg follows from its lifecycle. Ingest is
untouched by a role change — no dropped subscriptions, no lost birth cache, no data gap at
the Secondary->Primary transition.
**Primary-gated** (owner decision, recorded on the issue). Both pair nodes subscribe
independently, so an ungated claim means both claim the same host id — and the sharp
failure is not the two fighting over the retained STATE, it is that the SECONDARY's death
fires its Will and publishes OFFLINE while the Primary is healthy, stopping every edge node
in the group. Registering the Will only on whichever node is Primary at connect time fails
the other way: a node that later gains Primary would publish ONLINE with no Will behind it.
New seam: `IDriverPrimaryRoleSource` (`Core.Abstractions`) carrying the *data-plane* verdict
(`PrimaryGatePolicy.ShouldServiceAsPrimary`), not the alarm-drain verdict — an unknown role
with a driver peer present must deny. Registered unconditionally by `AddOtOpcUaRuntime`,
deliberately NOT alongside `IRedundancyRoleView`, which `AddAlarmHistorian` registers only
when that section is enabled: piggybacking would leave a pair with no alarm historian
falling back to "claim unconditionally". Polled on a 1s timer rather than evented — a
process singleton raising events at per-instance drivers outlives every driver it holds.
Accepted trade: a brief edge-visible OFFLINE window during failover. That is a genuine
state transition, not a glitch to hide.
Live-gated 5/5 against the Mosquitto fixture, including the Will leg — `AbandonForTests`
drops the socket with no DISCONNECT, because a clean disconnect makes the broker discard
the Will by protocol rule, so a test driven through `StopAsync` would observe the
stand-down OFFLINE and pass against an implementation that never armed a Will. Falsified:
removing the Will turns that leg and only that leg red.
Also corrects two stale claims found en route: the AdminUI form still said MQTT tags are
read-only (#508 shipped), and the driver doc still said nothing consumes `IRediscoverable`.
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
|
||
|
Joseph Doherty
|
2d20aeb779 |
fix(opcua,drivers): carry driver status sub-codes to clients (#519) + health-write ordering (#517)
#519 — a driver's OPC UA status code now reaches a client with its sub-code intact. DriverInstanceActor projected the driver's uint onto a 3-state Good/Uncertain/Bad enum and OtOpcUaNodeManager re-expanded Bad to a bare 0x80000000, so BadNoCommunication, BadTypeMismatch and BadNotSupported were indistinguishable on the wire and every driver's status constants (incl. the 16 #497 had just corrected) were diagnostic in logs only. Resolved by option (b): the raw uint rides AttributeValuePublished -> AttributeValueUpdate -> IOpcUaAddressSpaceSink.WriteValue, OpcUaQuality is deleted, and severity classification is a helper (OpcUaStatus.IsGood/IsUncertain/IsBad). The surface was wider than the issue costed (41 files, not "the sink and 3 implementations"): OpcUaQuality had non-driver producers with no uint to carry. Each got its natural code — BadInternalError for a failed virtual-tag script, BadNoCommunication for driver comms loss — and ScriptedAlarmHostActor's pair of mutually-inverse uint<->enum mappers is deleted, having round-tripped a code the engine already held. AddressSpaceApplier's removal writes deliberately keep the generic Bad: not losing a code the driver had is this issue; inventing one for the deploy path is a separate call. Behaviour change: WriteAlarmQuality fires its Part 9 event on a status-code change rather than a severity-bucket change. While the annotation was 3-state those were the same predicate; now a move between two Bad reasons changes what a subscriber reads. New SDK-free named constants live on Commons.OpcUa.OpcUaStatus, and StatusCodeParityTests is widened to cover them — that guard is reflection-driven, so it can only see a named constant, never an inline literal. #517 — the symptom is real but the issue named the wrong path and its fix was unsafe. ShutdownAsync cannot race: DriverInstanceActor calls it from PostStop, after the mailbox drains. (Core/Hosting/DriverHost, the one class with a concurrent GetHealth, has no production call site.) InitializeAsync can: it runs on a fire-and-forget Task.Run, and MTConnectDriver writes Healthy then starts loops that can throw into a catch which disposes the client before writing Faulted. Moving the write earlier, as suggested, inverts the terminal state — teardown drains in-flight loops that write health as they unwind, so a late loop could leave Healthy at rest with every connection closed. The fix is a bracket (conservative before, terminal after), applied to MTConnect's init-failure path only; shutdown keeps its single trailing write, and the rule plus the reachability table are documented on IDriver.ShutdownAsync. CalculationDriver.GetHealth returned a hard-coded Healthy that consulted nothing — Healthy before initialize, after shutdown and after dispose. Now derived from _running. Its test, named Health_is_always_Connected, asserted the defect and is replaced. Every new guard was falsified by breaking the production code and watching it go red. The two sub-code tests each assert two different Bad codes, since a single-code assertion could pass against a hard-coded constant under the old projection. Pre-existing failures, verified unchanged on a stashed tree and not chased: 4 in Driver.AbLegacy.IntegrationTests (slc500/micrologix/plc5 fixtures not up) and 3 in Host.IntegrationTests. Closes #517 Closes #519 Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo |
||
|
Joseph Doherty
|
30d0697c28 |
feat(hosts): surface per-host connectivity on /hosts; drop the unwritable table (#521)
`IHostConnectivityProbe` was a dead surface: eleven drivers implement it, `GetHostStatuses()` had ZERO production call sites, and `OnHostStatusChanged` had no subscriber outside the Galaxy driver's own aggregator. Per-host connectivity was computed by every driver and read by nobody. The issue offered "build the publisher or delete it". The publisher as its entity doc described it — driver nodes upserting `DriverHostStatus` rows — is not buildable: per-cluster mesh Phase 4 gates `AddOtOpcUaConfigDb` on the `admin` role, so a driver-only node has no ConfigDb connection to write rows with. So the capability is kept and the transport changed. `DriverHealthChanged.HostStatuses` now carries the probe result to `/hosts` as a Hosts column. That channel already reached the page, already survives the mesh split via the Phase 5 gRPC telemetry stream, and already replays a last-value snapshot on re-subscribe — so per-host state re-primes after a reconnect without a durable store. Both halves of the interface finally do what they are for: the event triggers a prompt publish, the pull is the source of truth. The point of the column is the case the driver-level state chip structurally cannot express: a multi-device driver stays aggregate-Healthy while ONE of its devices is unreachable. Two traps, both pinned by tests that were falsified against the prod code: - The host digest MUST be in the publish fingerprint. On a single-host-down transition every other fingerprint component is unchanged, so the dedup would swallow exactly the publish carrying the news — the trap that already bit the rediscovery signal. Removing it turns the guard test red, verified. - null (no probe) must stay distinct from empty (probe with no hosts). proto3 cannot tell an absent repeated field from an empty one, hence the explicit `has_host_statuses` flag; collapsing them would render every probe-less driver as one whose devices are all fine. Dropped: the DriverHostStatus entity, enum, DbSet, model config and table (migration DropDriverHostStatusTable — empty on every deployment, so the scaffolder's data-loss warning is moot, and Down() recreates it exactly). Found en route, NOT fixed here: `DriverInstanceResilienceStatus` is the identical defect — no writer, no reader, only a DbSet declaration, while the live data rides the `driver-resilience-status` telemetry channel. Its doc-comment now states that rather than describing the sampler and AdminUI join that were never built. Filed as #524 rather than widening this schema change beyond what was asked. Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo |
||
|
Joseph Doherty
|
5fff597324 |
revert(drivers): remove the periodic 5-minute device re-browse (#523)
Removes the tag-set drift detector shipped in |
||
|
Joseph Doherty
|
e77c8a3569 |
feat(drivers): tag-set drift detection, gated on SupportsOnlineDiscovery
The second thing #516/§8.2 deliberately did not build. The original objection stands — Modbus, S7, MQTT, AbLegacy and Sql echo authored config from DiscoverAsync, so diffing discovered against authored is a tautology for them — but it is now ENCODED rather than used as a reason not to build. The discriminator already existed: ITagDiscovery.SupportsOnlineDiscovery is documented as "enumerates the tag set from the live backend rather than replaying pre-declared/authored tags", and it separates the fleet cleanly — FOCAS, TwinCAT, MTConnect and AbCip true; the five config-echo drivers explicitly false. The check runs only for a driver that is SupportsOnlineDiscovery AND reports the new AuthoredDiscoveryRefs. That is nullable rather than empty-by-default on purpose: an empty collection legitimately means "nothing authored, everything discovered is new", so conflating the two would report total drift for a driver that never opted in. Where the value is: AbCip and FOCAS browse a live backend but implement no IRediscoverable, so before this they had no change signal at all. A periodic compare is the only way to notice a PLC re-download. A finding that changed the design: FOCAS, TwinCAT and AbCip all re-emit their authored tags into the same DiscoverAsync stream as device-derived ones, so the browse picker can show an operator their existing tags. Discovered is therefore always a superset of authored, and a VANISHED tag can never appear missing — one whole direction is structurally undetectable for three of the four. Shipping a Vanished list that is permanently empty for them would have been the half-inert seam this whole exercise removes. So the driver declares DiscoveryStreamIncludesAuthoredTags, the detector does not compute the undetectable half, and the operator message says "missing-tag detection unavailable for this driver" rather than letting the absence of a missing-tag clause read as reassurance. MTConnect is the only driver where both directions work — its stream is purely probe-model-derived. Behaviour: a failed browse is not drift (an unreachable device would otherwise report every authored tag as vanished); a truncated capture is skipped for the same reason; an unchanged drift is reported once rather than re-stamping its timestamp; drift that resolves clears the prompt. Interval defaults to 5 minutes — it browses a real device, and a tag set changing is an engineering event, not a runtime one. It reuses the Stage-2 surfacing, raising the same /hosts re-browse prompt, and never rebuilds the served address space. Comparison logic is a pure, separately-tested type rather than actor-inline. 14 new tests. The gate was verified load-bearing by deleting the SupportsOnlineDiscovery half: the tautology-driver test goes red, and it asserts discovery was never invoked rather than merely "no prompt raised", which would have passed for the wrong reason if the timer never fired. Full suite green except the same three pre-existing fixture-gated integration suites, identical counts. |
||
|
Joseph Doherty
|
d32d89c340 |
fix(drivers): stop silently discarding driver config edits (§8.3, #516)
Five drivers ignored the config handed to them on reinitialize, serving the
options their constructor captured — and the deployment sealed green anyway.
Fixed on both halves, as chosen.
Seam (load-bearing): DriverSpawnPlanner routes a changed DriverConfig to
ToStop + ToSpawn instead of an in-place delta, making the factory the single
parse authority. This REVERSES the deliberate decision documented at
DriverSpawnPlan.cs:49-50 ("a pure DriverConfig change stays an in-place delta
— no reconnect"). That reasoning was correct about the resilience pipeline and
wrong about the driver. The accepted price is a reconnect per config edit.
Per-driver, and this split was not anticipated:
- Re-parse in place — Modbus, AbLegacy, OpcUaClient. ParseOptions extracted
from each factory, called from InitializeAsync behind a HasConfigBody guard
so "{}" still keeps the constructor options.
- Respawn-only, deliberately NOT re-parsed — Sql and FOCAS. Each builds more
than options from config (Sql's dialect + resolved connection string,
FOCAS's client-factory backend, both injected at construction), so adopting
new options alone would run a NEW tag set against an OLD connection. Half a
re-parse is worse than none. SqlDriver's doc-comment asserted the opposite
premise — "config parsing belongs to the factory, which builds a fresh
instance" — which was false when written and is true only now.
Two green seals removed from ApplyChildDelta: it overwrote the cached Spec
synchronously BEFORE the child dequeued the message, so the host believed the
new config was live and the next reconcile computed no delta, sealing the
drift permanently; and it Tell'd with no Receive<ApplyResult> registered, so a
failed reinit — including Galaxy's deliberate NotSupportedException —
dead-lettered.
Exposed (not created) by the change: a factory throw is a CONFIG error, and
SpawnChild catches it and silently substitutes a stub. Only a brand-new driver
could reach that before; an ordinary config edit can now. Raised Warning ->
Error with an actionable message. It still does not fail the deployment —
doing so would let one malformed driver block a fleet deploy, so that is a
follow-up rather than a drive-by.
Tests: every pre-existing reinit test in these suites passes "{}", the exact
input a guarded re-parser treats as "keep constructor options" — blind to this
defect by construction. New tests pass CHANGED json; the Modbus one was
verified falsifiable by deleting the re-parse (goes red). Two tests asserted
the old behaviour and were rewritten, including the positive control in
DriverHostActorUnreadableArtifactTests, whose observable moved from
UnsubscribeAsync to ShutdownAsync now that teardown happens by stopping the
child rather than emptying its desired set.
Remaining full-suite failures are pre-existing and environmental, verified
identical on the pre-change tree: Host.IntegrationTests (3) and
Driver.AbLegacy.IntegrationTests (4, docker fixture-gated).
|
||
|
Joseph Doherty
|
adce27e7fa |
refactor(drivers): delete the dead discovered-node injection path (§8.2, #507)
#507 was filed as "injection inert in v3, re-migrate onto the raw subtree". Reading it, the retained code is not revivable: it resolves equipment from EquipmentNode.DriverInstanceId UNION EquipmentTags, and BOTH are structurally empty in v3 (AddressSpaceComposer always constructs EquipmentNode with a null DriverInstanceId; DeploymentArtifact hard-codes an empty EquipmentTags set). Removing the guard would have changed a log line and injected nothing. So it is deleted rather than fixed, and #507 closes as superseded by /raw browse-commit. Removed: HandleDiscoveredNodes, PartitionDiscoveredByDeviceHost, ShouldWarnPartition, PlansRoutingEqual, ApplyDiscoveredPlansForDriver, _discoveredByDriver, the redeploy re-inject tail, DiscoveredNodeMapper, DiscoveredInjection, AddressSpaceApplier.MaterialiseDiscoveredNodes, OpcUaPublishActor.MaterialiseDiscoveredNodes, and the Runtime-local copies of CapturingAddressSpaceBuilder/DiscoveredNode. The connect-time discovery loop goes too (StartDiscovery, RediscoverTick, HandleRediscoverAsync, DiscoveredNodesReady, TriggerRediscovery). With injection gone it had no consumer, and leaving it would either dead-letter or keep browsing real devices up to ~15x per connect to drop the result. ITagDiscovery itself stays — the /raw browse picker drives it through Commons/Browsing/DiscoveryDriverBrowser, which never went through the actor, so the picker is unaffected. deferment.md §4.4 called the 18 DiscoveryInjectionDormantV3 tests "Real — blocked on #507". They are not: they assert an equipment-rooted graft (EquipmentRootNodeId == "EQ-1") that v3 cannot produce, so they could never have unskipped as written. Deleted along with DiscoveredNodeMapperTests, the CapturingAddressSpaceBuilder tests, and the MaterialiseDiscoveredNodes tests in AddressSpaceApplierTests/OpcUaPublishActorTests. Runtime.Tests skipped: 31 -> 13. IHostConnectivityProbe is deliberately NOT resolved here. GetHostStatuses() has no production call site and nothing writes a DriverHostStatus row, but the table was re-created in the v3 initial migration, so deleting on inference would be wrong. Split to Gitea #521 with both options costed. CLAUDE.md, docs/drivers/{Galaxy,TwinCAT,MTConnect}.md updated — they described the seam as dead. Build clean; Runtime.Tests 476 passed / 13 skipped; OpcUaServer.Tests 362 passed / 4 skipped. |
||
|
Joseph Doherty
|
09a401b881 |
feat(drivers): consume IRediscoverable as an operator re-browse prompt (§8.2, #518)
Nine drivers raise IRediscoverable.OnRediscoveryNeeded when they observe that a remote's tag set may have changed — a Galaxy redeploy, a TwinCAT symbol-version bump, an MTConnect agent restart, a Sparkplug rebirth. Nothing in src/ subscribed, so every raise went into the void. Drivers even wrap the invoke in try/catch for "subscriber threw"; there has never been a subscriber. DriverInstanceActor now attaches the event in PreStart and detaches in PostStop, mirroring AttachAlarmSource/DetachAlarmSource. Attach is per-actor, not per-connect: the raise has no connection affinity and can land while the driver is between connects. The detach is load-bearing rather than tidy — the IDriver instance outlives the actor when the host respawns a child around the same driver object, so a missing -= accumulates one handler per respawn, each holding a dead Self. The signal rides the existing driver-health snapshot to /hosts as a "re-browse" chip. It is ADVISORY: the served address space is deliberately not rebuilt, because v3 authors raw tags explicitly through the /raw browse-commit flow and a runtime graft would materialise nodes no operator approved and no deployment artifact records. Two things worth knowing: - PublishHealthSnapshot dedups on a health fingerprint, and a rediscovery raise on an otherwise-healthy driver changes none of the four fields it hashed. Without adding the timestamp to that tuple the dedup swallows the exact publish carrying the signal. Reverting that one line turns 3 of the 5 new tests red, which is how it was confirmed rather than assumed. - The new test stub implements IDriver from scratch instead of deriving from the shared StubDriver, whose GetHealth() returns DateTime.UtcNow — its fingerprint differs on every call, so the dedup never engages and the dedup test would have passed vacuously either way. The planned discovery-result drift detector was NOT built. Modbus, S7, MQTT, AbLegacy and Sql all echo authored config from DiscoverAsync rather than browsing the device, so a discovered-vs-authored diff is permanently empty for them — it would have been another plausible-looking inert seam, which is the class this audit exists to remove. IRediscoverable is the trustworthy signal because the driver asserts it deliberately. DriverHealthChanged, IDriverHealthPublisher.Publish and HostsDriverRow gain defaulted optional parameters, so no existing call site changed. telemetry.proto gains fields 8/9 and both Phase-5 mappers carry them. Runtime.Tests 512 passed / 31 skipped. |
||
|
Joseph Doherty
|
4b9bcddbcc |
feat(drivers): periodic reconcile of desired vs actual subscriptions (#488)
Option A from the issue — the cheap consistency check, no interface change. The desired ref set was re-applied on a Connected TRANSITION (`ResubscribeDesired`) and on a deploy's `SetDesiredSubscriptions`, and nowhere else. A subscription lost while the driver STAYED Connected — a subscribe that failed and was never retried, a handle dropped down an error path — left the actor subscribed to nothing while still reporting Healthy, indefinitely. `ReconcileSubscription` now rides the existing 30 s `health-poll` tick: while Connected, a driver holding a desired set but no live handle is re-subscribed. Gated on `ISubscribable`. A driver that cannot subscribe at all can still be handed a desired set, and without the guard it would be told `Subscribe` every tick and fail every tick, forever. This is a state-machine consistency check, NOT a probe — it can only see that this actor holds no handle, never that a handle is present but dead server-side, because `ISubscriptionHandle` is opaque. That is option B, and it stays deferred until a real occurrence shows the handle outliving the subscription; it would need a liveness member implemented across all eight drivers with different subscription mechanics. A redundant `Subscribe` is possible (a tick queued ahead of an already-queued one observes the same no-handle state) and is harmless: `HandleSubscribeAsync` is a `ReceiveAsync`, so no tick is processed mid-subscribe, and it drops any prior subscription before establishing the new one. Suppressing it would need a pending-subscribe flag threaded through every self-tell site — more state than the race is worth. `healthPollInterval` becomes an optional Props parameter, mirroring the existing `rediscoverInterval` seam, so a test can drive the reconcile without waiting 30 s. Three tests. The reconcile itself is proven by a stub whose FIRST subscribe throws: nothing else would ever retry it, so a second attempt can only come from the reconcile — disabling `ReconcileSubscription` turns it red. The two guard tests are absence assertions and are bounded by a positive control rather than a sleep: a counting health publisher makes ticks PROCESSED observable, so "still one subscribe" means the reconcile ran and declined, not that the timer had not fired. Explicitly not deploy-time drift (#486) — re-asserting an already-correct desired set would not have helped there. |
||
|
Joseph Doherty
|
8e5090edf3 |
feat(mesh-phase5): tap the 4 telemetry publish seams into the local hub (DPS intact)
Each of the four telemetry publish seams now also emits its DPS payload into the node-local ITelemetryLocalHub (Phase 5), leaving the existing DistributedPubSub publish untouched: - driver-health AkkaDriverHealthPublisher -> TelemetryItem.Health - resilience DriverResilienceStatusPublisherService -> TelemetryItem.Resilience - alerts ScriptedAlarmHostActor + DriverHostActor (native) -> TelemetryItem.Alarm - script-logs VirtualTagActor + DpsScriptLogPublisher -> TelemetryItem.Script DI services take the hub as a required ctor param; actors take an optional nullable hub threaded through their Props and the real spawn sites (DriverHostActor <- ServiceCollectionExtensions; VirtualTagActor/ScriptedAlarmHostActor <- DriverHostActor), so existing test Props constructions keep compiling and simply do not emit. The hub is a no-op until a gRPC client subscribes, so the emit is safe on every node. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
3a590a0cb7 |
refactor(mesh-phase4): retire EfAlarmConditionStateStore + drop dead ScriptedAlarmState table (Task 9)
Scripted-alarm condition state lives in LocalDb (Phase 4); the ConfigDb-backed Ef store + ScriptedAlarmState table are now dead. Removes the test-harness Ef fallback (a DB-backed node with no store now skips the alarm host), deletes the store + entity + model config, and drops the table via migration. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
d907160747 |
feat(mesh-phase4): DriverHostActor runs ConfigDb-free (LocalDb alarm store, guarded acks)
Nullable ConfigDb factory; UpsertNodeDeploymentState no-ops when absent (central persists acks from the ApplyAck); scripted-alarm condition state served from the LocalDb store instead of the ConfigDb-backed Ef store. Combines Phase-4 Tasks 4 + 6. Removes the interim dbFactory! cast. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
6d00473866 |
feat(mesh): FetchAndCache bootstrap from the LocalDb pointer; no driver-side config SQL read
Phase 3 Task 6. In FetchAndCache mode Bootstrap() branches to BootstrapFromCache(), which restores served state from the LocalDb pointer (GetCurrentUnkeyedAsync → set _currentRevision → ApplyCachedArtifact from the cached bytes) with NO central-SQL read. Distinct from the Direct-mode TryBootFromCache fallback: reading the cache is NORMAL operation here so _isRunningFromCache stays false (the node can still fetch new deploys). An empty OR unreadable cache lands Steady-with-no-revision (the first dispatch fetches), never Stale — Stale means 'central SQL down, retry it', and FetchAndCache has no config SQL read to recover. TryRecoverFromStale gains a defensive FetchAndCache guard (it is unreachable in that mode, since the retry-db timer only starts in Stale). Proven with a ThrowingDbFactory in every test: reaching Steady + applying a dispatch proves the boot never touched central SQL. Sabotaging the mode branch (fall through to the SQL read) reddens all three — test A flips RunningFromCache; B/C enter Stale, which ignores the dispatch. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
648f173b18 |
feat(mesh): FetchAndCache apply path — fetch->cache->apply-from-bytes, null=apply-failure
Phase 3 Task 5. Under ConfigSource:Mode=FetchAndCache, a DispatchDeployment whose revision differs from _currentRevision kicks off a gRPC artifact fetch that PipeTo's its result back to Self as FetchedForApply (never a blocking await in a receive — that would freeze the mailbox for the fetch deadline). HandleFetchedForApply then applies synchronously on the actor thread, mirroring ApplyAndAck: null bytes = apply FAILURE (keep last-known-good, do not advance the revision, ack Failed — #485); non-null bytes reconcile drivers, rebuild the address space and push subscriptions FROM the bytes in hand (OpcUaPublishActor never reads central SQL), then cache them. A faulted fetch task maps to null bytes so it can't escape as an unhandled message. Extracted ReconcileDriversFromBlob from ReconcileDrivers so Direct-read, FetchAndCache and boot-from-cache share one reconcile body (the #485 empty guard moves into it); ApplyCachedArtifact now reuses it instead of duplicating the spawn-plan logic. DI: DriverHostActor.Props gains fetchAndCacheMode + artifactFetcher (LAST, per the positional-forwarding warning); Runtime resolves them from IOptions<ConfigSourceOptions> (fetcher only in FetchAndCache mode); Host registers GrpcDeploymentArtifactFetcher under hasDriver. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
5cb0e72166 |
feat(mesh): node subscribers accept EventStream commands; _coordinatorOverride -> _ackRouter
Phase 2 Task 5. DriverHostActor and ScriptedAlarmHostActor now subscribe to the node-local EventStream alongside their existing DPS topic subscriptions, so the transport flag lives ENTIRELY on the publishing side -- exactly one of the two ever publishes, and the switch can be flipped or reverted without touching a single subscriber. Note the alarm case is not symmetric: the DPS subscribe there still carries the node-LOCAL publisher (OtOpcUaServerHostedService's OPC UA Part 9 method calls), which never crosses the boundary and stays on DPS in both modes. Only the central AdminUI leg moves. Renamed DriverHostActor._coordinatorOverride to _ackRouter. Under ClusterClient mode that ref is the NodeCommunicationActor, emphatically not a coordinator, and the old name would send the next reader looking for one. Adds two wiring tests, because the unit tests either side of this seam BOTH pass with the subscription deleted -- the comm actor's test asserts a probe receives the message, and the host tests drive the actor by direct Tell. Neither notices a missing PreStart subscribe. Sabotage-verified: deleting either subscription reddens exactly its own wiring test and nothing else. Three things went wrong while writing those tests, all worth keeping: - The first version raced. ActorOf returns before PreStart runs, and an EventStream.Publish with no subscriber is dropped silently -- no buffering, no dead letter. Fixed with a warm-up whose ack proves PreStart completed; the comment says why it is not ceremony. - The alarm version was VACUOUS: it told the host directly INSIDE the assertion window alongside the relay, so the direct Tell alone produced the log the assertion waited for. Split into warm-up (outside) and relay (inside). - The alarm test needs its own ActorSystem: the shared harness pins loglevel = WARNING and the only signal an unowned alarm gives is a Debug line. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
f9f1b8fcee |
fix(localdb): phase-2 live gate — 4 production defects found and fixed
Gate record: docs/plans/2026-07-20-localdb-phase2-live-gate.md. Checks 1, 2, 5, 6 pass. Checks 3 and 4 are NOT satisfied — the defect they were meant to confirm turned out to be the opposite of what the plan assumed. Three defects crash-looped every driver node before check 1 could even run: 1. An empty ServerHistorian:ApiKey kills the host. ServerHistorianOptions- Validator exists to turn exactly that class of failure into a named OptionsValidationException, but its documented fail tier explicitly excluded ApiKey on the reasoning that a keyless client "degrades — the gateway rejects calls". It does not: the client validates its own options at construction, so the process dies during Akka startup and never makes a call. 2. UseTls disagreeing with the endpoint scheme kills the host too, in both directions (both messages confirmed in the shipped client assembly). Moving an endpoint from https to http without clearing UseTls is an ordinary migration slip. 3. Plaintext h2c was UNREACHABLE. HistorianGatewayClientAdapter forwarded the TLS-only options unconditionally, and AllowUntrustedServerCertificate defaults to false, so it always sent RequireCertificateValidation=true — which the client rejects outright when UseTls=false. Every http:// deployment crashed, though the scheme is documented as the supported way to select h2c, and the only workaround was to assert a certificate posture for a connection that has no certificate. The fourth was the blocker, and it is Phase 2's own: 4. The drain gate deferred to a Primary that cannot deliver. Redundancy roles are elected CLUSTER-WIDE; the alarm queue is PAIR-LOCAL. On the rig the elected driver Primary is central-1 — it carries the driver Akka role, replicates nobody's LocalDb and does not even run the alarm historian — so every driver node logged "Historian drain suspended", including the two site-b nodes that have no peer at all. Nothing drained anywhere, where before Phase 2 it drained fine. The cost is not a duplicate; it is the buffer growing to the capacity wall and evicting the audit trail it exists to protect. Fixed in three layers: a separate ShouldDrainAlarmHistory policy (unknown role drains; the two gates now deliberately disagree, and a test pins that); peer- host matching in DriverHostActor so a node stands down only for a Primary holding its rows; and AddAlarmHistorian short-circuiting the gate when replication is unconfigured — testing BOTH Replication:PeerAddress and SyncListenPort, since only the dialing half sets the former while both halves share the queue. Every one of these follows from the asymmetry: a false allow costs a duplicate row, which at-least-once delivery already accepts and payload-hash ids collapse; a false deny loses data silently. A third vacuous test, caught by the same delete-the-guard discipline: the role-view tests stayed green with the guard removed, because AwaitAssert polls until an assertion passes and the assertion was "reads open" — which is the SEEDED value, satisfied at the first poll before the actor processed anything. They now assert the sequence of published values through a recording view; the control then goes red for exactly the cases that matter. Migration evidence: 11 legacy rows across two deliberately overlapping files converged to exactly 9 identical rows on both nodes, proving D-6's payload-hash identity on real nodes rather than in a fixture. Open design fork, recorded in the gate doc rather than decided here: a pair cannot currently identify its own Primary, so both halves drain. Safe in every topology — nothing loses data — but the gate's de-duplication benefit is unrealised until roles are scoped per pair. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
71379816e7 |
feat(localdb): alarm S&F on LocalDb with primary-gated drain (cutover)
Moves the alarm store-and-forward buffer out of its own alarm-historian.db and into the node's consolidated LocalDb, where it replicates to the redundant pair peer. A node that dies holding undelivered alarm history no longer takes it to the grave. Tasks 2 and 3 land together, as the plan anticipated. They are not separable: the gate is a constructor argument of the rewritten sink, and a commit that replicated the queue without gating the drain would be a commit in which both nodes of a pair deliver every alarm event, continuously. That drain gate is the load-bearing part of the change, and the recon explains why it is new work rather than a refinement. Exactly-once delivery across a pair is enforced today on the ENQUEUE side, by HistorianAdapterActor: only the Primary enqueues, so the Secondary's queue is empty and its ungated drain has nothing to send. Replicating the table destroys that invariant. The gate is a Func<bool> the caller supplies, because the drain runs on a timer the sink owns rather than on a mailbox, and Core.AlarmHistorian cannot reference PrimaryGatePolicy in Runtime. Runtime supplies it via a new IRedundancyRoleView singleton that DriverHostActor publishes its Primary-gate verdict to -- the same verdict the inbound-write and native-ack gates use, so there is no second notion of am-I-the-Primary to drift. Two failure modes are deliberately closed: - The view is seeded OPEN, matching the policy's own answer for an unknown role with no driver peer. A deployment that runs no redundancy never publishes to it, and defaulting closed would silently stop its alarm history forever. - A gate that throws is read as not-now, never as permission, and a closed gate reports the new HistorianDrainState.NotPrimary rather than Idle. A Secondary's rising queue is supposed to look different from a stalled drain, and if BOTH nodes report NotPrimary the pair is misconfigured and says so instead of quietly filling toward the capacity ceiling. Row ids are a hash of the payload rather than fresh GUIDs. Both adapters accept the same fanned transition in the window before the first redundancy snapshot arrives, and under last-writer-wins an equal key converges those two accepts into one row instead of duplicating them. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
ef41a43102 |
fix(drivers): fail the apply when the artifact could not be read (#486)
ApplyAndAck advanced _currentRevision and recorded NodeDeploymentState=Applied immediately after ReconcileDrivers, which returns null when the artifact could not be read. So a node that applied NOTHING reported success, claimed a revision whose configuration it never applied, and — because HandleDispatchFromSteady short-circuits on a revision match — could never be healed by re-dispatching that revision. Only a later, different revision recovered it. Now a null blob fails the apply: the revision is left where it was, NodeDeploymentState records Failed with the reason, and the coordinator gets a Failed ACK. Leaving the revision alone is what makes the retry actually land instead of being waved through. This is about telling the truth, not about tearing anything down — the node keeps serving its last-known-good address space, drivers and subscriptions throughout (#485). Tests, RED-first (both verified failing with "should be Failed but was Applied"): the ACK/revision assertion, plus the one that matters — after a failed apply, re-dispatching the SAME revision now genuinely applies. Its recovered artifact adds a second driver precisely so a short-circuited ACK (which has no side effects) cannot satisfy it. Four existing tests changed, and both changes are deliberate: - DriverHostActorTests.SeedDeployment never set ArtifactBlob at all. Its three consumers are about the apply/ACK/state machine, not the artifact, and now need a genuine apply — so the helper seeds a well-formed artifact declaring no drivers. That is what the composer emits for an empty configuration, and is precisely what "bytes we could not read" is NOT. - EmptyArtifact_IsNotCached asserted "the apply still succeeds — an empty artifact is a legitimate no-op deployment". That premise is the bug. Flipped to Failed; the test's actual subject (nothing is cached) is untouched and now holds for two independent reasons. Closes #486 Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
b0c031f09f |
fix(drivers): don't tear down field I/O when a deploy's artifact reads back empty (#485)
The driver-side half of the same mistake the address-space fix corrected. ReconcileDrivers and PushDesiredSubscriptionsFromArtifact both read the artifact as `...FirstOrDefault() ?? Array.Empty<byte>()`. Their THROW paths already returned early, but empty bytes flowed onwards as a real answer: zero driver specs, so DriverSpawnPlanner planned every running child for StopChild, and then an empty desired set dropped each surviving driver's live subscription handle. A node lost its entire field I/O and still ACKed Applied. Same rule as the address space: no bytes is no answer, not "a configuration with no drivers". Nothing legitimate produces a zero-length blob — deleting the last driver still deploys a JSON document with empty arrays. Both sites now skip and keep what is running. The two guards are independent because PushDesiredSubscriptions does its OWN ConfigDb read, so the row can go missing between them. Also corrects the ReconcileDrivers doc comment, which claimed an empty blob made it "effectively a no-op" — with children running it was the opposite. Tests: DriverHostActorUnreadableArtifactTests, RED-first (verified failing — after the empty dispatch the driver list was empty). A zero-length ArtifactBlob reproduces byte-for-byte what the missing-row case delivers, so the race does not have to be constructed. Two controls, both load-bearing: - a READABLE driverless artifact still stops the driver, so the guard keys on "the read gave us nothing", not on "fewer drivers than before"; - dropping a driver's LAST TAG still clears its subscription. This one was added after the absence assertion was caught passing on a race: the unsubscribe is an async self-tell, so with the second guard deleted the test still went green. The control observes that same unsubscribe arriving well inside the settle window, which is what makes the absence meaningful — with the guard deleted the suite now correctly goes RED. SubscribableStubDriver gains an UnsubscribeCount for that observation. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
c6a9f93a0c |
fix(localdb): re-cache the artifact on RestoreApplied so a wiped node self-heals
CacheAppliedArtifact ran only on a fresh apply (ApplyAndAck), never on the bootstrap RestoreApplied path. So a node whose cache was lost — fresh/wiped volume, disk failure — recovered its served state from central yet stayed cache-less, unable to boot-from-cache on the NEXT central outage until some future new deploy happened to land. Replication does not heal this: a peer's already-acked cache rows are pruned from its oplog, so a fully-wiped, converged node is never back-filled. RestoreApplied now writes the served artifact back to the cache (invariant: the cache holds what the node serves). Found on the docker-dev live gate wiping a node's LocalDb volume. |
||
|
Joseph Doherty
|
9137cb41eb |
fix(localdb): materialise the address space from the cached blob on boot-from-cache
Boot-from-cache told OpcUaPublishActor to RebuildAddressSpace(deploymentId), which re-loads the artifact from the ConfigDb by id — the very database that is unreachable during the outage this path exists to survive. The rebuild no-op'd, so a cache-booted node ran its drivers but served OPC UA clients an EMPTY address space. RebuildAddressSpace now carries an optional in-hand Artifact blob; ApplyCachedArtifact passes the cached bytes so materialisation happens from them directly. Found on the docker-dev live gate: healthy peer served 16 nodes, the cache-booted node served 1. |
||
|
Joseph Doherty
|
a27eff3298 |
feat(localdb): boot from the pair-local artifact cache when central SQL is unreachable
Hooks the Bootstrap catch that previously went straight to Stale. On a cache hit the node serves its last-known-good configuration through the outage instead of coming up with an empty address space; on a miss, behaviour is byte-for-byte what it was. The read is unkeyed because ClusterId is only derivable from an artifact you already hold or from the unreachable central DB (recon D-1). Newest pointer wins, which is correct in every real topology - a node belongs to one cluster and its peer replicates that same row. Splits PushDesiredSubscriptionsFromArtifact out of PushDesiredSubscriptions: the cache path runs precisely when the ConfigDb read cannot succeed, so re-reading the artifact there would fail by definition. RunningFromCache is surfaced on NodeDiagnosticsSnapshot (optional param, existing call sites unaffected). A node running from cache looks entirely healthy from outside - full address space, live values - but its config is frozen and no deploy can reach it. It clears only on a real apply from central. |
||
|
Joseph Doherty
|
1becf59168 |
feat(localdb): DriverHostActor stores applied artifacts in the pair-local cache
New Props/ctor parameter appended LAST at all three positions - the forwarding list inside Props.Create is positional and compiles into an expression tree, so a mis-ordered argument is usually type-compatible and binds the wrong dependency at runtime. All 26 existing test construction sites use named arguments and needed no change. ReconcileDrivers now returns the blob it loaded. It catches its own DB failures and returns without rethrowing, so ApplyAndAck can reach its success path having spawned zero drivers; caching on 'the apply succeeded' would persist an empty artifact as last-known-good and boot the node into an empty address space during the next outage. Verified: weakening the guard to null-only turns EmptyArtifact_IsNotCached red. The store runs in its own try/catch because an Applied ACK has already been sent by then - an escaping exception would land in ApplyAndAck's catch and emit a second, contradictory Failed ACK for a deployment the fleet believes is live. |
||
|
Joseph Doherty
|
db751d12a5 |
feat(alarms): condition Quality tracks source connectivity (#477)
Part 9 ConditionType.Quality was never assigned; default(StatusCode)==Good so every native + scripted condition reported Good unconditionally — a comms-lost device still showed a healthy, inactive, Good condition (a wrong-VALUE bug, distinct from the null-value #473/#475). Clients (and HMIs bucketing on IsGood) could not tell "genuinely inactive" from "lost contact". Layer 1 — make Quality a real, plumbed field: - AlarmConditionSnapshot gains OpcUaQuality Quality (default Good). - MaterialiseAlarmCondition sets it (native BadWaitingForInitialData, scripted Good). - WriteAlarmCondition projects snapshot.Quality; the delta-gate gains a Quality member so a quality-bucket change fires a Part 9 event. Layer 2 — drive native quality from driver connectivity (a comms-lost driver emits no alarm transitions, and an alarm-bearing raw tag has no value variable, so quality can't come from either existing channel): - DriverInstanceActor Tells parent ConnectivityChanged on Connected/Reconnecting. - DriverHostActor fans it to every native condition the driver owns as OpcUaPublishActor.AlarmQualityUpdate (Good on connect, Bad on disconnect). - New dedicated IOpcUaAddressSpaceSink.WriteAlarmQuality sets ONLY Quality and fires only on a bucket change — never touches Active/Acked/Retain (an active alarm that loses comms stays active). Not a full-snapshot re-projection, so it can't clobber severity/message and works for a never-fired condition. Forwarded through DeferredAddressSpaceSink (F10b trap; auto-verified by the reflection forwarding guard). Ungated by redundancy role; no /alerts row. Scripted conditions stay Good; worst-of-input quality deferred to #478 (Layer 3). Tests: node-level (materialise/project/no-clobber/unknown-node no-op), NativeAlarmProjector, DriverInstanceActor connectivity emission, DriverHostActor fan-out, OpcUaPublishActor routing, and the wire-level guard (Condition_event_Quality_tracks_source_connectivity_on_the_wire) — RED-verified against a simulated pre-fix always-Good server. Existing DriverInstanceActor parent probes ignore the new ConnectivityChanged. Docs: docs/AlarmTracking.md §"Condition source-data Quality (#477)"; design doc docs/plans/2026-07-17-alarm-condition-quality-477-design.md. |
||
|
Joseph Doherty
|
90e52a4415 |
fix(v3-batch4): thread ReferencingEquipmentPaths into native AlarmTransitionEvent
Closes the WP3/WP4 boundary seam WP4 flagged: ForwardNativeAlarm (DriverHostActor, a WP3-owned file WP4 could not edit) built the AlarmTransitionEvent without the referencing-equipment paths, so the /alerts equipment-list chips WP4 added never populated in production (would fail live-gate leg 5). The RawTagPlan already carries ReferencingEquipmentPaths (the Area/Line/Equipment UNS folder paths); this rides them onto the per-condition alarm meta and into every transition. Coordinator-owned 1-file change (WP3 merged, no parallel agent owns this file this wave). Claude-Session: https://claude.ai/code/session_01LVneM3eh1UtJxEisFXgmox |
||
|
Joseph Doherty
|
2e0743ad25 |
fix(v3-batch4-wp3): realm-qualified write routing + dormant discovery guard + self-correction/byte-parity tests (Wave B review H1/M1/M2/L1/L3)
H1 (HIGH): write-routing key now (AddressSpaceRealm, bareId), not bare-only. A raw s=<RawPath> and a UNS s=<Area/Line/Equip/Eff> can collide as bare strings; the bare-only key let a colliding raw+UNS pair route to the WRONG driver ref (last-writer-wins). The realm the node manager resolves (RealmOf) is now threaded through IOpcUaNodeWriteGateway.WriteAsync -> RouteNodeWrite -> _driverRefByNodeId keyed by (realm, bareId). New regression test: Colliding_raw_and_uns_bare_ids_route_to_their_own_driver_by_realm. M1 (MEDIUM): discovered-node injection made coherently DORMANT. HandleDiscoveredNodes hard-short-circuits (single enforcement point; _discoveredByDriver never populates so the re-inject tail is inert too), with a clear log pointing at the /raw browse-commit flow. New pin: Discovered_nodes_are_ignored_dormant_in_v3; the 16+2 v2 injection scenarios re-pointed to an accurate skip reason (DiscoveryInjectionDormantV3). M2 (MEDIUM): realm-qualified dual-node self-correction tests — Failed_uns_write_reverts_uns_node_and_leaves_raw_node_untouched + Raw_realm_revert_reverts_raw_node_only (the second fails if the realm is dropped). L1: removed the = AddressSpaceRealm.Uns defaults from the consequential node-manager mutation methods (WriteValue/WriteAlarmCondition/MaterialiseAlarmCondition/ EnsureFolder/EnsureVariable/UpdateFolderDisplayName/UpdateTagAttributes/ RaiseNodesAddedModelChange/Remove*/RevertOptimisticWriteIfNeeded) + the AttributeValueUpdate/AlarmStateUpdate records, so the compiler forces explicit realm; read-only accessors + internal builders retain their defaults. L3: fixed the stale VirtualTagHostActor class comment (V3NodeIds.Uns, not the retired EquipmentNodeIds.Variable). Also: DeploymentArtifactRawUnsParityTests — Raw/UNS node-set byte-parity round-trip between AddressSpaceComposer.Compose and DeploymentArtifact.ParseComposition. Claude-Session: https://claude.ai/code/session_01LVneM3eh1UtJxEisFXgmox |
||
|
Joseph Doherty
|
3efcf8014b |
feat(v3-batch4-wp3): raw-only binding + UNS fan-out + write routing
Wave B of Batch 4 — the runtime binding seam for the dual namespace. Applier (both realms, explicit realm at every sink call site): - MaterialiseRawSubtree: Raw containers as folders + Raw tags as variables keyed by RawPath (native-alarm tag → single Part 9 condition at the RawPath), all in AddressSpaceRealm.Raw; historian tagname = override else RawPath. - MaterialiseUnsReferences: each UNS reference Variable under its equipment folder (Uns realm) + an Organizes UNS->Raw edge; inherits writable/array/ historian tagname from the backing raw tag (both NodeIds -> one tagname). - FeedHistorizedRefs / ProvisionHistorizedTags now source RAW tags (mux ref stays single, keyed by RawPath); ApplyPureRemove tears down raw tags + UNS refs in place (raw-container removal falls back to rebuild). DriverHostActor (dual-NodeId, single-source fan-out): - _nodeIdByDriverRef value gains a realm (NodeRealmRef); rebuilt from RawTags UNION UnsReferenceVariables so one (DriverInstanceId, RawPath) fans to the raw NodeId AND every referencing UNS NodeId with identical value/quality/timestamp. Write inverse map keyed by the bare id; the ns-qualified NodeId the write hook passes is normalised (BareNodeId) so a write to either NodeId resolves the same driver ref (-> RawPath write). - Native raw alarm condition routing is realm-tagged (Raw); AttributeValueUpdate + AlarmStateUpdate carry the realm through to the sink. Retire EquipmentNodeIds -> V3NodeIds.Uns (applier/VirtualTagHostActor) and RawPaths.Combine (DiscoveredNodeMapper, discovered nodes are Raw now). DeploymentArtifact.ParseComposition emits the Raw + UNS subtrees byte-parity with the composer (reconstruct entities -> AddressSpaceComposer.Compose). Sink surface: removed the transitional `= AddressSpaceRealm.Uns` defaults from IOpcUaAddressSpaceSink / ISurgicalAddressSpaceSink / SdkAddressSpaceSink / DeferredAddressSpaceSink / NullOpcUaAddressSpaceSink — every call site is now explicit (realm reordered before the trailing optionals on EnsureVariable + MaterialiseAlarmCondition). Node-manager convenience methods keep their defaults (they are not the interface impl; Sdk delegates explicitly). Tests: rewrote DriverHostActorLiveValueTests (fan-out drift, 1:N) + DriverHostActorWriteRoutingTests (dual-NodeId raw/uns write routing) to the v3 raw+uns model; new AddressSpaceApplierRawUnsTests; migrated the EquipmentTags provisioning/feed tests to RawTags; swept EquipmentNodeIds test callers. Claude-Session: https://claude.ai/code/session_01LVneM3eh1UtJxEisFXgmox |
||
|
Joseph Doherty
|
dc0d7653b9 |
feat(v3-batch3): B3-WP4 — {{equip}}/<RefName> reference-relative resolution
Replace the deleted equipment-base-prefix {{equip}} mechanism (impossible now
that equipment is reference-only) with per-reference resolution:
{{equip}}/<RefName> resolves through the owning equipment's UnsTagReference rows
(by effective name) to the backing raw tag's RawPath. Every unresolved <RefName>
is a clear deploy error + authoring rejection + Monaco diagnostic — preserving
"editor accepts <=> publish accepts".
Commons:
- EquipmentScriptPaths: delete DeriveEquipmentBase; SubstituteEquipmentToken now
takes the equipment's reference map (effectiveName -> RawPath) and substitutes
{{equip}}/<RefName> inside path literals (unresolved left intact, never throws);
add ExtractEquipReferenceNames (path-literal scoped) +
ExtractEquipReferenceNamesFromText (message templates). Slash syntax replaces
the v2 dot joint.
- New EquipmentReferenceMap: shared, input-shape-agnostic builder (entity + JSON
sides) over RawPathResolver — the single authority both compose seams + the
validator use, so resolved RawPaths agree byte-for-byte.
Compose seams (byte-parity kept):
- AddressSpaceComposer.Compose + DeploymentArtifact.ParseComposition both build
the per-equipment reference map from UnsTagReferences + Tags + raw topology and
substitute VirtualTag AND ScriptedAlarm-predicate sources before dependency
extraction (runtime dep refs = resolved RawPaths).
Deploy gate + authoring + editor:
- DraftValidator.ValidateEquipReferenceResolution: EquipReferenceUnresolved error
for unresolved {{equip}}/<RefName> in VirtualTag/ScriptedAlarm scripts + alarm
message-template tokens (naming script, equipment, missing ref).
- UnsTreeService.ValidateEquipTokenAsync (Wave-A M1): reference-relative authoring
rejection, replacing the stale DeriveEquipmentBase(empty)->reject-all logic.
- ScriptAnalysisService: {{equip}}/ completion offers the equipment's reference
effective names; DiagnoseAsync flags unresolved refs (OTSCRIPT_EQUIPREF) same as
the deploy gate. Requests carry optional EquipmentId (threaded MonacoEditor ->
monaco-init.js -> fetch bodies). IScriptTagCatalog.GetEquipmentRelativeLeavesAsync
repurposed to GetEquipmentReferenceNamesAsync(equipmentId, filter).
Tests: EquipmentScriptPaths substitution/extraction (slash pinned); composer<->
artifact reference-resolution byte-parity; DraftValidator unresolved-ref;
ScriptAnalysis completion+diagnostic; M1 authoring gate. Build clean (0/0); all
five affected suites green.
Claude-Session: https://claude.ai/code/session_01LVneM3eh1UtJxEisFXgmox
|
||
|
Joseph Doherty
|
af5e062ba4 |
fix(driver-calc): rebuild calc tag state on ReinitializeAsync + re-register deps post-apply (review HIGH)
A calc-tag add/remove/edit or a script edit changes the merged DriverConfig, which DriverSpawnPlanner routes to an in-place ApplyDelta (not a respawn), i.e. CalculationDriver.ReinitializeAsync. The old ReinitializeAsync ignored the new config: the tag table + dependency-ref set were built once at construction and never rebuilt, so after a redeploy added tags never computed, removed tags kept publishing, and edited scripts served stale results (deploy reported success). Part 1 (HIGH): extract RebuildTagTable(config) — shared by the ctor and reinit so they cannot drift — and have ReinitializeAsync re-bind the new config (ParseOptions) and rebuild _tagsByRawPath/_changeDependents/_dependencyRefs/ _stateByRawPath under _lock, preserving last-known state for surviving tags, dropping removed tags, and starting new tags fresh. _dependencyRefs now reflects the new authored tags. Corrected the now-false ReinitializeAsync comment. Part 2 (MEDIUM ordering hazard): the host's ReRegister was Tell-ed synchronously alongside the ApplyDelta Tell, so the adapter re-read DependencyRefs before the async ReinitializeAsync rebuilt them (stale set, no self-correction). Now DriverInstanceActor sends DeltaApplied(driverInstanceId) to DriverHostActor AFTER ReinitializeAsync completes, and HandleDeltaApplied re-registers only that driver's adapter — sequenced after the rebuild. Removed the inline loop. Test: added a redeploy test to CalculationDependencyFlowTests exercising the real DependencyMuxActor + adapter + driver — a new tag reading a NEW dep flows + computes, a removed tag stops publishing, an edited script serves the new result, and DependencyRefs re-registers the new set. Fails on pre-fix code (DependencyRefs stays stale ["src/a"]), passes after the fix. Claude-Session: https://claude.ai/code/session_01LVneM3eh1UtJxEisFXgmox |
||
|
Joseph Doherty
|
53d222e0f7 |
feat(driver-calc): B2-WP7 Calculation driver — IDependencyConsumer + triggers + deploy cycle gate
Adds the Calculation pseudo-driver (tags computed by C# scripts over other tags' live values) plus the host seam that feeds it: - IDependencyConsumer capability interface (Core.Abstractions) + DriverTypeNames.Calculation. - CalculationDriver : IDriver, ISubscribable, IReadable, IDependencyConsumer — change-gate + dedupe (VirtualTagActor parity), timer-group scheduler, Bad-transition + script-log emission, Good recovery; reuses the T0-4 CalculationEvaluator. - DriverHostActor spawns a DependencyConsumerMuxAdapter per dependency-consuming driver, registering its refs on the per-node DependencyMuxActor and forwarding DependencyValueChanged → OnDependencyValue; re-registers on every apply, torn down with the driver. - Factory + probe registered in DriverFactoryBootstrap (keeps the T0-3 guard green); guard test csproj references the driver so the bin-scan discovers the factory. - DeploymentArtifact injects the resolved scriptSource (scriptId → SourceCode) into a calc tag's delivered TagConfig so the host-blind driver can compile it. - DraftValidator deploy gates: CalculationScriptMissing / CalculationScriptNotFound (scriptId existence) + CalculationDependencyCycle (DependencyGraph.DetectCycles over calc→calc edges). - Tests: driver units (dep-ref extraction, change-gate, dedupe, Bad+script-log+recovery, timer, read), tag-config parsing, DraftValidator gates (self/2-cycle/cross-driver-terminal), and a Runtime integration test proving values FLOW mux → adapter → driver → calc-of-calc end-to-end. Claude-Session: https://claude.ai/code/session_01LVneM3eh1UtJxEisFXgmox |
||
|
Joseph Doherty
|
e81ac352ed |
v3 B1-WP4/WP5: pipeline rewire to greenfield schema + dark address space
- ConfigComposer: snapshot v3 tables (RawFolders/TagGroups/UnsTagReferences), drop the retired Namespaces snapshot; RevisionHash follows the new blob. - DeploymentArtifact: compute each raw tag's RawPath (RawFolder->Driver->Device-> TagGroup ancestry via shared RawPathResolver); per-driver RawTags + merged DriverConfig+DeviceConfig config injected into DriverInstanceSpec via DriverDeviceConfigMerger; ParseComposition emits an empty EquipmentTags set (DARK until Batch 4); EquipmentNode driver/device hooks always null; cluster scoping attributes equipment by UNS line only. - AddressSpaceComposer: rewritten to the v3 shape (no Namespace/NamespaceKind, no dropped Tag/Equipment fields); emits empty EquipmentTags (parity mirror). - DriverHostActor: fan-out/write map tuple member FullName -> RawPath. - Commons: new RawPathResolver (shared identity authority) + DriverDeviceConfigMerger (single/multi-device endpoint+RawTags merge). - DraftValidator: v3 rules — raw-name charset (RawPaths.ValidateSegment), historized effective-tagname <=255, UNS effective-leaf uniqueness across UnsTagReference/VirtualTag/ScriptedAlarm; DraftSnapshot(+Factory) carry the new tables. - AdminOperationsActor: tag-config inspection resolves driver via Device (no EquipmentId/DriverInstanceId on Tag). - Tests: RawPathResolver/merger unit tests (Commons.Tests, green) + DeploymentArtifact RawPath/dark test (Runtime.Tests, awaits WP6 corpus fixes). |
||
|
Joseph Doherty
|
1a698cbb97 |
Merge R2-11 TagConfig consolidation (arch-review round 2) [PR #434]
Findings 01/C-1, 01/P-1 (single TagConfigIntent.Parse in Commons, 4 copies delegate, parse-once) + 05 CONV-2/UNDER-1/UNDER-6 (shared strict TagConfigJson readers, per-driver Inspect() + writable key, FOCAS capability pre-flight, Modbus probe timeoutMs, deploy-gate Warn|Error). Phase C runtime-strict flip deferred. T22/T24 deferred. Auto-merged AddressSpaceApplier.cs + DriverHostActor.cs with R2-04/R2-10; verified OpcUaServer.Tests 286/286 + Runtime.Tests 396/396. Build clean. |
||
|
Joseph Doherty
|
72b1600245 | fix(r2-04): primary gates default-DENY on multi-driver clusters while role unknown (03/S4) | ||
|
Joseph Doherty
|
f2e899aaad | feat(r2-04): PrimaryGatePolicy - membership-aware default-deny + denial meter (03/S4) | ||
|
Joseph Doherty
|
c1e19ea954 | refactor(runtime): DeploymentArtifact decodes TagConfig via shared TagConfigIntent (R2-11, 01/C-1) | ||
|
Joseph Doherty
|
7bbec77716 | refactor(runtime,opcuaserver): device-host normalization re-homed to Commons DeviceConfigIntent (R2-11) | ||
|
Joseph Doherty
|
07eec11d31 | docs+test(r2-02): bulkhead doc sweep + parsed-knob-must-be-wired guard (01/U-6, OVERALL theme 1) | ||
|
Joseph Doherty
|
496f97da20 | fix(r2-02): write dispatch defaults to non-idempotent — no-retry arm authoritative (03/S12 layer 2) | ||
|
Joseph Doherty
|
75403caa1a |
feat(archreview #13): apply per-instance ResilienceConfig from the deploy artifact
The DriverInstance.ResilienceConfig column was authored in AdminUI, persisted to the entity, and serialized into the deployment artifact by ConfigComposer — but the runtime read path dropped it: DriverInstanceSpec didn't carry it and the invoker factory always passed null, so every driver got tier defaults regardless of its configured overrides (a silent dead-config gap — #10's residual sub-finding). Read-path plumbing (write side was already complete): - DriverInstanceSpec gains ResilienceConfig; DeploymentArtifact.TryReadSpec reads the column. - IDriverCapabilityInvokerFactory.Create takes resilienceConfigJson; DriverHostActor.SpawnChild threads spec.ResilienceConfig; the concrete factory parses it (ParseOrDefaults, layering on the tier), logs any parse diagnostic (never throws), and builds the invoker with the merged options. - Invalidate-on-change: the pipeline cache keys on (instance, host, capability) and ignores options on a hit, so Create() now Invalidate()s the instance's cached pipelines first (no-op on first spawn) — a respawn with changed options rebuilds them. - DriverSpawnPlanner treats a ResilienceConfig change as a stop+respawn (the invoker/options are bound to the child at spawn); a pure DriverConfig change stays an in-place delta (no reconnect). - Host DI passes a logger to the factory for the parse diagnostic. Verification (deterministic): factory Create applies a retryCount:0 override to actual execution (control test proves tier default retries), invalidates the instance's cache on re-create (scoped — sibling survives), malformed config logs+falls-back; planner respawns on ResilienceConfig change (incl null→json) and stays delta on a pure config change; artifact parse carries/omits the column. Core.Tests 243 (+5), Runtime.Tests 363 (+5), Host builds clean. |
||
|
Joseph Doherty
|
bacea1a44f |
fix(archreview #10): wire CapabilityInvoker into prod dispatch via a Polly-free seam
Remediates RESILIENCE-DISPATCH-GAP (surfaced by the 07/C-1 analyzer): the Phase 6.1 CapabilityInvoker resilience pipeline (retry / breaker / bulkhead / telemetry) was constructed only in tests — the production dispatch layer called driver-capability methods directly, bypassing it entirely. The filed plan (thread CapabilityInvoker directly into DriverInstanceActor) is INFEASIBLE: Runtime is deliberately Polly-free (references Core.Abstractions, not Core — the same boundary IDriverFactory documents). So this introduces a seam, mirroring IDriverFactory exactly: - Core.Abstractions: IDriverCapabilityInvoker (+ NullDriverCapabilityInvoker pass-through) and IDriverCapabilityInvokerFactory (+ null factory). - Core: CapabilityInvoker now implements the interface; new DriverCapabilityInvokerFactory builds a per-instance invoker over the process-singleton pipeline builder + status tracker + tier resolver. - Runtime: DriverInstanceActor takes an IDriverCapabilityInvoker (default = pass-through) and routes all 6 dispatch sites (write, alarm-ack, subscribe, unsubscribe, alarm-subscribe, discover) through it; per-host key resolved via IPerCallHostResolver for single-ref calls, driver-instance key for bulk calls. DriverHostActor builds + injects the real invoker per spawned driver. - Host: DriverFactoryBootstrap registers the tracker, pipeline builder, and concrete factory (needs DriverFactoryRegistry.GetTier). Runtime SCE resolves the factory from DI like IDriverFactory; pass-through on nodes without it bound. - Analyzer: IDriverCapabilityInvoker.ExecuteAsync/ExecuteWriteAsync are now recognized wrapper homes (interface-typed invoker calls must suppress OTOPCUA0001 too). All 6 RESILIENCE-DISPATCH-GAP pragmas removed — the analyzer (an error in Runtime) is now the standing regression guard. Also adds retry / breaker-open / breaker-close LOGGING to the pipeline builder — the operator-facing observability surface (the Admin /hosts reader, Phase 6.1 Stream E.2/E.3, was never built, so the pipeline otherwise runs silently). Scope notes: - Tier-DEFAULT policy only: DriverInstanceSpec (the deploy artifact) does not carry the per-instance ResilienceConfig JSON, so overrides aren't applied yet (tracked follow-up: plumb ResilienceConfig through the composer/artifact). - GenericDriverNodeManager's call is NOT wired: that class is test-only scaffolding (zero production references — the Server has its own address-space path); its pragma is re-annotated accordingly, not left as a "tracked gap". Verification (unit + analyzer; live behavioral gate still pending — see FOLLOWUP-10): - Negative control: unwrapping any site fails the Runtime build with OTOPCUA0001. - New Runtime guard (recording invoker): write routes via ExecuteWriteAsync with the IPerCallHostResolver host; subscribe via ExecuteAsync — proves runtime routing. - New analyzer test: the interface is a valid wrapper home. - New pipeline-builder test: retry events are logged. - Full solution builds clean (0 errors); Runtime.Tests 357, Core.Tests 238, Analyzers.Tests 32 all green; pass-through default keeps existing dispatch tests byte-for-byte unchanged. |
||
|
Joseph Doherty
|
f0082af5b9 |
build(analyzers): wire OTOPCUA0001 tree-wide + triage the surfaced hits (arch-review 07/C-1)
Inject the custom UnwrappedCapabilityCallAnalyzer as an OutputItemType=Analyzer ProjectReference from Directory.Build.props (excluding the analyzer + its test project) so OTOPCUA0001 runs on every src/ and tests/ compilation — it previously enforced its CapabilityInvoker-wrapping rule against nothing but its own 31 unit tests (the 'built-but-never-wired' failure mode). Triage of the ~280 surfaced hits, three categories: 1. RESILIENCE-DISPATCH-GAP (7 sites, DriverInstanceActor x6 + GenericDriverNodeManager x1): a REAL, previously-untracked gap the analyzer caught on first wiring — the Phase 6.1 CapabilityInvoker resilience pipeline (retry/breaker/bulkhead/telemetry) is constructed ONLY in tests and was never wired into the production dispatch layer. Scoped per-site #pragma with a greppable RESILIENCE-DISPATCH-GAP marker explicitly noting these are tracked-but-not-intentional, pending the dispatch-wiring remediation (filed as a follow-up). Keeps the analyzer live everywhere else in those projects so a NEW unwrapped call still fails the build. 2. Driver-INTERNAL self-calls (3 sites, AbCipAlarmProjection x2 + S7Driver x1): a driver's own poll/ack path calling its own capability method. The invoker wraps the driver from the dispatch layer OUTWARD; a driver re-wrapping its own internal calls would double-wrap. Genuinely intentional — scoped #pragma with that rationale. 3. Wire-level test suites + manual-testing CLIs (12 projects): invoke drivers directly by design — the analyzer's own documented intentional case. Project-level NoWarn with a comment. Verified: full solution build green, 0 OTOPCUA0001 hits; analyzer's 31 tests pass; negative control — dropping one dispatch-gap pragma re-fires OTOPCUA0001 and fails the Runtime build, proving the analyzer is genuinely live tree-wide, not disabled. |