d8a17f2e345b96f76dccf7d971c192cedc48a6e7
297 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Joseph Doherty
|
140bf0c131 | perf(adminui): pooled DbContext factory; role-grants reads per-call, not per-circuit | ||
|
Joseph Doherty
|
d787502d54 |
fix(opcuaserver): degraded applies re-diff instead of losing nodes; in-place UNS mirror updates; sever inverse edges
C1 — a non-rebuild apply reporting failedNodes > 0 no longer advances _lastApplied. Pre-delta-materialisation the full passes re-ensured every node on every apply, which silently repaired any prior swallowed sink fault. The delta path removed that accidental self-healing: a failed node sat inside _lastApplied, never appeared in a diff set again, and EnsureVariable early-returns on the ones that landed — so it was lost until an unrelated change forced a rebuild. The advance also moved to AFTER the materialise passes, so a throwing pass leaves _lastApplied at the previous composition too. A rebuild still advances even with failures (the ratchet is its resync path). I2 — a UnsReferenceDelta whose identity is unchanged (same NodeId, EquipmentId and BackingRawPath) is now applied in place via UpdateTagAttributes(Uns) instead of remove+recreate, so the mirror's MonitoredItems survive an operator historizing or reshaping the raw tag behind them. Remove+recreate stays for identity-changing deltas (re-point, effective-name edit). I3 — RemoveVariableNode severs the inverse half of every outgoing Organizes edge, the variable-node counterpart of UnwireAlarmNotifiers. Rebuilds used to sweep these up; surgical applies made rebuilds rare, so they accumulated on the raw node for the life of the process. I5 — added UNS reference variables are announced (their equipment folder is the parent, already the Uns realm). Raw adds still are not; the gap comment now says so honestly rather than calling it neutral and pre-existing. I4 + M1 — corrected two comments that no longer described the code: the "re-point → rebuild" claim in ApplyPureRemove, and the "N lock acquisitions" rationale for the delta passes (Ensure* early-return on a dictionary probe before the lock). |
||
|
Joseph Doherty
|
0dbbb5ac6b | fix(runtime): stash dedup + applying-wedge watchdog; cache-failure diagnostics | ||
|
Joseph Doherty
|
bfbdce32e4 | perf(opcuaserver): non-rebuild applies materialise the delta, not the composition | ||
|
Joseph Doherty
|
14c31941c7 | fix(runtime): parallel-safe decompression limit; tighten ParsedDeploymentArtifact surface | ||
|
Joseph Doherty
|
5f2e60278c | perf(runtime): apply-path cache write off-mailbox; stash deferred dispatches | ||
|
Joseph Doherty
|
9dbd5e96f7 |
perf(controlplane): gzip artifact blob inside the hashed bytes (~10-20x smaller)
ConfigComposer gzips the serialised snapshot (CompressionLevel.Fastest) immediately after SerializeToUtf8Bytes and BEFORE taking the SHA-256, so the compressed bytes ARE the artifact everywhere: Deployment.ArtifactBlob, the LocalDb cache chunks, the gRPC fetch stream, and the RevisionHash a FetchAndCache node verifies against. The invariant the deploy path rests on — RevisionHash == SHA-256(stored/wire bytes) — is preserved, which is what leaves the fetcher, the cache and the serve path untouched: nothing between the composer and DeploymentArtifact.Open ever looks inside the blob. Read side: Open() sniffs the gzip magic 1F 8B (valid JSON can never start with 0x1F, so no framing of our own is needed) and inflates through a size guard — MaxDecompressedBytes, 512 MiB, internal + mutable so the guard can be exercised without inflating half a gigabyte per test run. Pre-upgrade uncompressed blobs take the identical path they always did. Every decompression failure — corrupt stream, truncated stream, over-limit inflation — is reported as JsonException, the same exception family the malformed-JSON path already owns. So the lenient static wrappers still degrade to empty results and the apply paths still fail the apply and keep last-known-good (#485); no caller's decision changes because the reason the bytes are unusable changed. gzip output is not guaranteed byte-stable across .NET releases. Harmless here: the hash is computed FROM the emitted bytes at compose time and never re-derived from the JSON, so the worst an implementation change costs is one spurious "changed" revision after a runtime upgrade. Snapshot determinism is unaffected — compression wraps the final bytes and reorders nothing. Tests: gzip/uncompressed parity through both the instance parses and the statics, corrupt-gzip leniency through the wrappers, the size guard (with a falsifiability control proving the same bytes inflate fine under the production limit), and composer-side blob-starts-with-1F8B + RevisionHash == SHA256(blob) with a control proving the hash is NOT over the JSON. |
||
|
Joseph Doherty
|
2fd60ecce1 |
perf(runtime): apply path parses the artifact once per actor
Both apply handlers (DriverHostActor.ApplyAndAck + HandleFetchedForApply)
and OpcUaPublishActor.HandleRebuild used to rebuild a JsonDocument over the
same artifact bytes for every question they asked of it — driver instances,
composition, cluster scope, data-plane authority. Each now opens ONE
ParsedDeploymentArtifact per apply and threads it through:
- ReconcileDriversForApply / ReconcileDriversFromParsed return the blob
plus the parse the rest of the apply reads; the handler owns it and
disposes it in its finally.
- PushDesiredSubscriptionsFromParsed and CacheAppliedArtifact take that
parse instead of re-parsing the bytes.
- HandleRebuild opens once after the #485 guard and feeds both the
data-plane authority and the composition from it.
The blob-based variants stay as the lenient wrappers the boot paths need
(BootstrapFromCache, RestoreApplied), so the three remaining static Parse*
call sites are boot-only — none on an apply path.
The RebuildAddressSpace message still carries raw BYTES, deliberately: the
publish actor is a different actor with its own mailbox and
ParsedDeploymentArtifact is a disposable, non-thread-safe object owned by
the sending handler. It opens its own single parse on the far side.
FeedDataPlaneAuthority still SHA-256s the RAW bytes for the revision id — a
re-serialisation of the parsed document would be a different byte sequence
and would not match the deployment's RevisionHash.
Behaviour note: Open() throws JsonException on bytes the static wrappers
swallowed to an empty result. That is routed into each path's EXISTING
failure handling — ApplyAndAck's catch (Failed ack, children left running)
and HandleRebuild's catch (Error log, last-known-good still served) — not
as a new outcome. An empty composition is not a benign no-op there: the
planner diffs it as a PureRemove and tears the address space down, so the
throw is the honest reading of unusable bytes.
|
||
|
Joseph Doherty
|
59202017cd |
refactor(runtime): ParsedDeploymentArtifact — one JsonDocument per apply
Every public entry point on DeploymentArtifact took ReadOnlySpan<byte> and did JsonDocument.Parse(blob.ToArray()) — a full copy plus a full document build per call — so one apply paid ~8-11 full parses of the same bytes (the scoped ParseComposition alone parsed three times: scope, composition, BuildClusterSets). DeploymentArtifact.Open(byte[]) now parses once into ParsedDeploymentArtifact, which owns the JsonDocument and exposes an instance method mirroring each static entry point. The moved bodies operate on Root; BuildClusterSets takes JsonElement instead of re-parsing the blob; ResolveClusterScope is memoized per node id, since every consumer of one apply asks about the same node. Every static is now a thin wrapper (Open + delegate + dispose) keeping its ReadOnlySpan<byte> signature and its lenient empty/malformed contract — the wrapper pays the one copy and owns the JsonException catch, so output shapes are unchanged. Hot callers migrate to Open() separately. ParsedDeploymentArtifactTests pins same-bytes-same-output through both doors for all six entry points over a rich multi-cluster artifact (raw chain + UNS refs + scripts + keys + ACLs), with falsifiability controls proving the fixture actually scopes, plus memoization and after-dispose-use coverage. |
||
|
Joseph Doherty
|
cf69d03f9b | perf(runtime): Direct-mode apply threads the in-hand artifact (1 SQL read per apply, not 3) | ||
|
Joseph Doherty
|
44b99a9939 |
fix(runtime,opcuaserver): review fast-follows — event-filter throw test, loud MapKey fallback, log shape
Accumulated fast-follows from the Task 1-11 reviews. No behaviour change on the value path. - Strengthen OpcUaPublishActorBatchTests.Sink_throw_is_swallowed_and_logged. The survival assertion alone did NOT prove the catch exists: Akka's default supervision RESTARTS a throwing actor and the restart preserves the mailbox, so the follow-up message landed either way and the test passed against a build with no catch at all. It now asserts via EventFilter that the exception NEVER reaches supervision (Expect(0) on the InvalidOperationException) and that the handler's own Warning IS logged, keeping the survival assertion. Everything that must have happened goes inside the Expect(0) action — it counts matches at the end of the action, so an assertion left outside would race the actor and pass vacuously. Verified falsifiable: deleting the catch turns the test red. - MapKey(NodeId): the non-string identifier branch collapsed to a string.Empty key silently. Such a key cannot match anything the manager stored, so the caller sees an unexplained miss rather than an error. It now logs a Utils.LogWarning naming the seam. Every current call site is provably string-only, so this should never fire. - The two WireAlarmNotifiers Utils.LogInfo sites interpolated the RealmKey struct, rendering a raw struct dump. Interpolate .Id and .Realm as separate placeholders so log-grep patterns stay consistent with the file's other sites. - Stale comments: the _eventNotifierSources key is a RealmKey now, not a namespace-qualified NodeId string; and three comments still named DriverHostActor.ForwardToMux, which Task 8 replaced with ForwardBatchToMux. |
||
|
Joseph Doherty
|
ec6f8bb0ea | refactor(runtime): delete dead per-value host ingress; batch fan-out review fixes | ||
|
Joseph Doherty
|
d7edc709cf |
feat(runtime): dependency mux accepts batched publishes
DependencyMuxActor now receives DriverInstanceActor.AttributeValueBatchPublished — the primary value ingress since DriverHostActor's batched fan-out — and filters it per item, emitting one DependencyValueChanged per genuinely interested (item, subscriber) pair with that item's own value/timestamp/status sub-code (#519). The per-item AttributeValuePublished receive stays: DriverHostActor's comms-loss fan fabricates it, and per-tag drivers still take that path. Indexed for-loop over Items to avoid an interface-enumerator allocation per burst. |
||
|
Joseph Doherty
|
c51568f2a5 |
feat(runtime): DriverHostActor batched value fan-out
Replaces the temporary per-item unwrap bridge with a real batched handler: one AttributeValueBatchPublished burst now costs ONE hop to the dependency mux and ONE OpcUaPublishActor.AttributeValueBatchUpdate, instead of one message per value. Routing per item is unchanged — the v3 Batch 4 single-source fan-out (raw NodeId + every referencing UNS NodeId, identical value/status/timestamp), with an unmapped ref Debug-logged and dropped without disturbing the rest of the burst. A burst in which nothing resolves sends no publish message at all rather than an empty batch. Indexed for-loop over Items is deliberate: no interface-enumerator allocation per burst on the hottest routing seam. ForwardToMux and its registrations stay for the comms-loss fan and per-tag drivers. |
||
|
Joseph Doherty
|
6bf1fb3754 | fix(opcuaserver): chunked, per-item-isolated WriteValues with typed partial-failure outcome | ||
|
Joseph Doherty
|
8ef02506f7 | fix(runtime): harden batch forwarding — detach pins, malformed-raise guard, struct items | ||
|
Joseph Doherty
|
e04ff2488c | feat(runtime): DriverInstanceActor forwards batched data changes | ||
|
Joseph Doherty
|
04e6a4690c | feat(runtime): OpcUaPublishActor batched value updates | ||
|
Joseph Doherty
|
1320f3fa08 | perf(runtime): raise opcua pinned dispatcher throughput 1 -> 100 | ||
|
Joseph Doherty
|
dac7e1885d |
feat(resilience): route IDriver.InitializeAsync through the capability invoker (R2-10 T10 closed)
The connect path was the one capability call the resilience pipeline never saw, so a driver that could not connect contributed zero breaker input and the /hosts resilience panel read 'All targets healthy' against a dead endpoint. InitializeAsync now routes through the invoker as DriverCapability.Initialize, keyed on the driver-instance id. - No pipeline retry: the actor's RetryConnect timer is the retry loop (parser forces retryCount overrides to 0, same funnel as write-shaped). - Breaker threshold 2, not the Read-sized 5: the breaker samples 30 s and the reconnect timer fires every 10 s, so at most three attempts fit one window — threshold 5 could never open. Pinned by its own test. - An open breaker fast-fails into the same InitializeFailed path as a real connect failure; the 15 s break window costs at most one extra tick. The best-effort teardown ShutdownAsync stays unwrapped. - Companion fix: DriverCapabilityInvokerFactory.Create now invalidates the instance's tracker snapshots with its pipelines — a respawn during an outage discarded the only breaker whose OnClosed could clear the OPEN flag, stranding a false 'Breaker OPEN' chip forever. Live gate PASSED on docker-dev (image symbol-verified per container): rtu-gate against dead 10.100.0.35:5021 opened its Initialize breaker on both centrals with the red 'BREAKER OPEN' chip on /hosts; a socat 5021→5020 forwarder recovered it (both breakers closed within 1 s, panel back to 'All targets healthy'); healthy MAIN-modbus recorded zero breaker events. Full suite 50/50 green. Evidence: docs/plans/2026-08-07-adminui-followups-and-live-gates.md §7. |
||
|
Joseph Doherty
|
cd7135570c |
refactor(mesh): collapse MeshPaths to local actor names — no longer a wire contract (review follow-up)
MeshPaths.CentralCommunication / .NodeCommunication (the "/user/..." full-path constants) had
ZERO production consumers after the ClusterClient deletion: both gRPC services resolve their
comm actor through ActorRegistry by typed key, never by ActorSelection over a path. Only the
path-probe test still read them, while the class doc went on calling them "the wire contract
of the central↔node boundary" — a claim that stopped being true the moment nothing addressed
an actor across a process boundary by name.
- The two full-path constants are deleted; the two Name constants remain.
- MeshPaths' doc now says what it is (the local actor-naming convention the registry-keyed
services are spawned under) and records what it stopped being, so the next reader does not
re-derive a contract from the file's history.
- MeshCommActorPathTests composes "/user/" + Name itself — the absolute path is that test's
own addressing detail now, not something the product declares. Renamed
Both_mesh_comm_actors_exist_at_their_contract_paths ->
..._are_spawned_under_their_declared_names, because what it still pins is that the two
registration sites actually ran on a real booted node; a rename is now a compile error, so
"contract" was the wrong word for what would break.
- NodeCommunicationActor's summary no longer cross-references the deleted constant or calls
the name an address. CentralCommunicationActor was checked and makes no such claim.
|
||
|
Joseph Doherty
|
4d2dfb5b50 |
test(mesh): step 7 — replace the boundary test with MeshGrpcBoundaryTests and sweep the comments
MeshGrpcBoundaryTests replaces the MeshClusterClientBoundaryTests deleted in step 1. Three
self-seeded single-member clusters sharing the ActorSystem name, joined ONLY by gRPC, driving
the REAL CentralCommunicationActor -> MeshGrpcCommandDialer -> Kestrel h2c ->
MeshCommandAuthInterceptor -> NodeCommandGrpcService -> NodeCommunicationActor -> EventStream:
1. No_cross_mesh_Akka_association_forms_between_central_and_either_site — the program's own
point, kept as a test rather than a live-gate note: each cluster stays single-member and
none ever holds another's address as a member.
2. One_dispatch_through_central_reaches_a_node_in_every_cluster_mesh — the fan, with each
site holding its OWN per-cluster PSK, so a dial that selected the wrong cluster's key
would be refused at the far side.
3. Falsifiability_control_a_dial_carrying_the_wrong_cluster_key_delivers_nothing — without
it, (2) cannot distinguish "the gRPC boundary carried it" from "something else in the
process did". The same node then accepts the right key, so the silence is the auth gate
and not a dead endpoint.
All three run fully offline — no Docker fixture — and pass.
Comment sweep across every remaining clusterclient|receptionist hit in src/ and tests/:
the program label becomes "mesh gRPC command plane", and every comment that described the
deleted transport as live now describes what actually runs (Dps = pair-local default,
Grpc = cross-mesh). mesh_transport.proto's header/service comments are reworded; no message,
field or enum is touched — the contract is locked. MeshCommActorPathTests is reshaped rather
than retired: the two paths still exist and still matter, but what a mismatch costs is now a
misleading Unavailable from the gRPC ingress rather than a silently dropped receptionist send.
The only surviving mentions of the old names are the ones that must survive: the fail-closed
rules that name "ClusterClient" as a retired value refused at boot, their tests, and this
test class's own account of what it replaces.
|
||
|
Joseph Doherty
|
5d7ecdf737 |
refactor(mesh): step 3 — delete both ClusterClientReceptionist registrations
Central's CentralCommunicationActor and a node's NodeCommunicationActor no longer register with a receptionist; the ActorRegistry entry each already carried is now the whole reachability story, and it is what CentralCommandGrpcService / NodeCommandGrpcService resolve to hand an inbound message over. Both `using Akka.Cluster.Tools.Client` directives are gone with them. |
||
|
Joseph Doherty
|
ab5b958d47 |
refactor(mesh): step 2 — delete the node-side ClusterClient and its centralClient parameter
WithOtOpcUaRuntimeActors no longer builds a ClusterClient from MeshTransport:CentralContactPoints, and NodeCommunicationActor.Props takes only the gRPC IApplyAckSender. The ackRouter gate narrows from `useClusterClient || useGrpc` to `useGrpc`, and the CentralClusterClientName constant is gone. NodeCommunicationActor's no-transport warning now names MeshTransport:CentralGrpcEndpoints and MeshTransport:Mode only — the old text pointed an operator at a key that no longer exists. Ripples in this step: the two ClusterClient-shaped facts in NodeCommunicationActorTests (Send-not-SendToAll, and sender-wins-over-a-client) are deleted — the first has no subject left, the second guarded a both-transports-present state that is now unconstructible. The Runtime ServiceCollectionExtensions fact keeps its ackRouter pin and drops its "no ClusterClient actor exists" half for the same reason. |
||
|
Joseph Doherty
|
07bc1878fc | docs(mesh): sweep the remaining doubled-quorum comment sites — the coordinator absorbs duplicates idempotently | ||
|
Joseph Doherty
|
667639e6ab |
fix(mesh): per-endpoint deadline + review fixes — failover survives a hanging central
CRITICAL. The ack deadline was one absolute DateTime computed once for the whole endpoint walk. Against the characteristic WAN failure — a central that HANGS (SYN black-hole, frozen host) rather than refusing — endpoint 1 burned the whole AckTimeoutSeconds and endpoint 2 was then dialled with a deadline already in the past: instant DeadlineExceeded, no packet leaving the node, the surviving central never really tried. Failover looked like it ran and did not. It also contradicted MeshTransportOptions' own documented "applied per endpoint, not per ack". The deadline is now computed fresh inside the loop; headers stay hoisted (read-only, sequential walk). Two test gaps that let that survive are closed: nothing pinned the ackRouter gate (narrowing it back to ClusterClient-only compiled and passed every test while dead-ending every Grpc ack to DPS), and the deadline fact's lower bound accepted any positive timeout at all. Both new pins are mutation-verified. Docs corrected where they overstated the case: duplicate acks CANNOT mis-seal a deployment — ConfigPublishCoordinator.HandleAck keys _acks by NodeId behind an expected-set guard, so a duplicate overwrites its own entry. The single-endpoint rule is defence in depth and a bound on wasted work, not a correctness dependency; and the walk is at-least-once on the deadline path, not at-most-once. |
||
|
Joseph Doherty
|
308c6aaa0f |
feat(mesh): gRPC ApplyAck path behind Mode=Grpc
Task 1B.2 — the node→central half of the ClusterClient→gRPC migration. GrpcApplyAckSender walks MeshTransport:CentralGrpcEndpoints in order and STOPS AT THE FIRST SUCCESS. That single `return` is the whole single-count guarantee: the transport it replaces was ClusterClient.Send, deliberately not SendToAll, because both central nodes forward to the SAME ConfigPublishCoordinator singleton proxy — an ack posted to both would have the coordinator count one node twice and could seal a deployment on a doubled count. The list is an availability list, not a fan-out list. No retry beyond that one ordered pass, no queue: the receptionist buffered nothing and retried nothing, and this is like-for-like, not an upgrade to delivery guarantees. All endpoints failing is still a DROP — but a loud one, naming the node, deployment, correlation id and the status each endpoint answered with, which is what separates "central is down" from "central rejected this node's key". The bearer never reaches a log; only endpoints and status codes do. Per-endpoint GrpcChannel cache and the endpoint loop clone GrpcDeploymentArtifactFetcher; the cluster id in x-otopcua-cluster comes from this node's own cluster-<id> role via RoleParser, off the same bound Cluster:Roles ClusterRoleInfo reads. SendAsync never throws: the caller discards the task, so an escaping exception would become an unobserved fault — a drop with no log line, the exact failure mode this migration removes. NodeCommunicationActor gains an OPTIONAL ackSender; every existing test compiles and passes unedited. It is checked before the ClusterClient so the two can never both fire. The registration arm creates no ClusterClient under Grpc. NOTE — one wiring change beyond the actor: DriverHostActor's ackRouter is now `useClusterClient || useGrpc ? nodeComm : null`. Left at ClusterClient-only, a Grpc node's ack would have gone out on the DPS deployment-acks topic and never reached the sender at all. |
||
|
Joseph Doherty
|
39db75083d |
feat(drivers): every driver reports the authored tags it skipped (Gitea #541)
A driver builds its RawPath -> definition table at init and tolerates a
per-tag miss: an unmappable TagConfig, an unparseable address, a
capability-matrix rejection. That tolerance is right -- one bad tag must not
take down the other few thousand. What was missing is any statement of the
fact afterwards. A skip is not an error and happens before any read, so the
driver stays Healthy, errors/5 min stays 0, Last error stays empty, and Last
read advances live off the tags that did map. On the wonder Makino in #540
that reading held while focas-z-34184 ignored 20 of its 22 tags.
New optional capability ISkippedTagReporter, backed by a shared
SkippedTagLedger so the reset/commit/cap rules are one implementation rather
than one per driver. The report -- authored count, exact skipped count, and a
capped sample of RawPath + reason -- rides DriverHealthChanged to a new Tags
column on /hosts, over the Phase 5 telemetry stream. No table: a driver-only
node has no ConfigDb to write to, the lesson from #521.
The issue named eight drivers. It is twelve -- every driver in
DriverTypeNames.All has an authored-tag skip path. Calculation (no scriptId),
MTConnect (no readable DataItem id), Galaxy (no attributeRef) and OpcUaClient
(no nodeId) were unlisted, and OpcUaClient's was the quietest of the twelve:
NamespaceMap drops such an entry with a bare `continue` and not even a log
line. SkippedTagReporterFleetTests pins the invariant reflectively over every
IDriver in src/Drivers/, so a thirteenth cannot re-introduce the silence.
Sql drops a tag at TWO points and its report is the union of both: a blob
that does not map, AND an identifier the 8.1 catalog gate rejects against the
live catalog. The second is what schema drift produces -- a renamed column, a
dropped view -- months after a deployment sealed green, and reporting only
the first would have that driver claim "0 skipped" while polling half its
tags, which is worse than reporting nothing because the column would then be
actively reassuring.
Deliberately additive to driver state, never a change of it. Latching
non-Healthy for a config problem would satisfy both halves of the #527
comms-loss conjunction -- 90 s without a successful read AND non-Healthy --
and a driver whose tags were ALL skipped has no successful reads by
construction, so it would reconnect-loop forever against a healthy device.
Three details that are load-bearing rather than incidental:
- The skip digest is part of the publish fingerprint. A skipped tag leaves
every other fingerprint component identical to a clean driver's, so without
it PublishHealthSnapshot's dedup swallows exactly the publish carrying the
signal -- the trap that already bit the rediscovery and host-status
channels. The digest covers the sample, not just the counts: a redeploy
that fixes one tag and breaks another leaves both counts unchanged.
- The count is exact; only the sample is capped, and the tooltip says so when
it is showing fewer than it counted. A truncated list that looked complete
would have an operator fix twenty tags and believe the job done.
- The ledger stages and commits, so an init that throws part-way leaves the
previous report standing. A partial snapshot would read as "we now skip
fewer tags", the opposite of the truth.
The proto field needs no companion presence flag, unlike host_statuses: a
singular message field carries real presence in proto3, and a has_ flag would
collide with protoc's generated HasSkippedTags.
Verified red-first three ways: dropping SkippedTagDigest from the fingerprint
fails the two dedup tests, removing the seam from one driver fails the fleet
guard, and dropping the catalog-gate leg fails the Sql union test. Full
solution builds with zero new warnings against
|
||
|
Joseph Doherty
|
e7fbe97759 |
fix(historian): gate historization capture on the Primary, quarantine a poison tag (#537, #538)
Both defects were found by #491's live gate against a real HistorianGateway and filed rather than fixed at the time, because each one's fix is a design decision rather than a patch. #537 — both nodes of a redundant pair captured every value, so the historian received duplicate live samples for every historized tag (measured: the gateway's write counter advanced twice per value and halved when one central was stopped). The recorder now reads a capture gate per value, fed from the new PrimaryGatePolicy.ShouldRecordHistorizedValues through IHistorizationRoleView — a third verdict on the same RedundancyRoleView object, registered unconditionally on driver nodes rather than alongside the alarm section's IRedundancyRoleView (which only exists when that section is enabled; piggybacking is the #511 registration accident). Two things are deliberate and easy to get backwards: * It gates the CAPTURE, not the drain — the opposite of the alarm sink. That queue is in LocalDb and replicates, so the Secondary already holds the Primary's rows. The historization outbox is a node-local FasterLog replicating to nobody, so a drain-only gate would leave the standby appending forever to a queue no one empties: unbounded disk, then a flood of stale duplicates the moment it is promoted. Entries a node already captured are still drained by that node. * The peer it stands down for must RECORD THE SAME REFS, not merely hold the same rows — a Primary in this node's own per-cluster mesh. A node carrying no cluster-scoped role cannot establish that and never stands down: under a fleet-wide election the elected Primary is usually running a different cluster's configuration, and deferring to it would end this cluster's historization silently, with the drain reporting success on an empty outbox forever. That is the scope mismatch that once suspended the alarm drain on every node of the rig. An unknown role captures, matching the alarm drain — a duplicate sample is recoverable, a hole in the trend is not. #538 — the drain writes every per-tag group and then acks all-or-nothing (deliberate: RemoveAsync truncates THROUGH an id, so partial acking could drop older unwritten entries). One permanently failing tag therefore pinned the FIFO head while every healthy tag in the batch was re-submitted to the historian on every pass, indefinitely — duplicate samples and an outbox that never truncated, observed live at 8.3 MB from a single unprovisionable tag. The drain now tracks consecutive failures per historian tag and abandons that tag's entries once it exceeds MaxWriteAttempts (new, default 5), so the head advances. The counter only advances on a pass where some OTHER tag wrote successfully: that is what separates "this tag is poison" from "the backend is down". A count-only quarantine looks identical on the poison-tag case and silently discards the whole backlog during an ordinary gateway restart, turning a duplicate-samples bug into a lost-history one. Abandonment is also deferred until every failing tag is over budget, so nothing is discarded on a pass that could not have advanced the head anyway. Each abandonment logs a Warning naming the tag; the total rides RecorderStatus.AbandonedEntries. Capacity also gains a non-zero default (500,000 entries) as the disk backstop for the one case the quarantine deliberately cannot resolve — a historian refusing every write, where retaining is correct. Set 0 to restore unbounded retention. Every new test was falsified by reverting the production change it guards, and each one reddens alone: * drop the backend-reachable evidence -> only A_backend_that_fails_every_tag_abandons_nothing * never abandon -> only A_permanently_failing_tag_is_abandoned_so_healthy_tags_stop_being_rewritten * latch the capture gate at construction -> only The_capture_gate_is_re_read_per_value... * treat any Primary as a scope peer -> only the two scope-mismatch cases Full suite green. (Four OpcUaClient.Browser failures during the run were the opc-plc fixture being down on the shared Docker host; brought it back up and they pass.) Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo |
||
|
Joseph Doherty
|
ad74473452 |
fix(historian): the recorder died on its first value — live gate for continuous historization (#491)
Ran #491's live gate against a real ZB.MOM.WW.HistorianGateway talking to a real AVEVA Historian, and it found a production-fatal defect on the first value it ever saw. ContinuousHistorizationRecorder.OnValueChangedAsync touched `Self` after awaiting IHistorizationOutbox.AppendAsync(...).ConfigureAwait(false). ConfigureAwait(false) discards Akka's ActorTaskScheduler synchronization context, so `Self` — which reads ActorBase.Context — threw NotSupportedException. The throw escaped into supervision, which RESTARTED the actor; because production seeds an EMPTY ref set (the deployed set arrives later via UpdateHistorizedRefs), the restart silently discarded every historized ref. Net effect: one value captured, then nothing, with no further error to show for it. Fixed by capturing `IActorRef self = Self` before the await — the idiom the GetStatus handler already applied to Sender. Thirteen unit tests missed it because InMemoryOutbox.AppendAsync returned ValueTask.CompletedTask: a completed ValueTask never yields, so the continuation ran inline with the context still installed. The production FasterLog outbox fsyncs per entry and does yield. The first regression test was itself vacuous — `await Task.Yield()` posts back to the CAPTURED context, so the append still completed on the actor thread and reverting the fix left the test green. The fake must complete on a POOL thread (Task.Delay(1).ConfigureAwait(false)), as a real fsync does; with that, reverting the fix reddens exactly one test. The rig could not run this gate at all: docker-dev carried ServerHistorian__* but no ContinuousHistorization__* keys, so the recorder could never spawn. Both centrals now carry them (default OFF — the recorder needs ContinuousHistorization:Enabled AND ServerHistorian:Enabled), with the outbox on the per-node named volume so it survives a container recreate. ServerHistorian__UseTls was hard-coded true and is now parameterized; a plaintext h2c gateway was unreachable without it. Gate results: outbox append, drain, WriteLiveValues accepted by a real historian, and restart convergence (capture resumes after a restart with no new deployment) all pass. The read-back hop is NOT proven — the LAN historian accepts writes but serves no reads under a "License violation: Database tag count exceeds license tag count", and the licensed corporate one needs the VPN. EnsureTags provisioning is likewise unexercised there (RemoteGrpc-only, and that CAP is not listening). #491 stays open for the read-back hop alone. Also recorded, not fixed: the recorder has NO Primary gate, unlike the alarm store-and-forward sink, so both nodes of a redundant pair send duplicate live values. Gating it would introduce a failover window with no capture — that is a design decision, not a cleanup. Evidence: docs/plans/2026-08-01-continuous-historization-live-gate.md Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo |
||
|
Joseph Doherty
|
f692f6afe4 |
feat(security): enforce node ACLs on write, alarms and HistoryRead (#520 stage 5)
`Security:Acl:Enabled` (default false) makes scoped `NodeAcl` grants decide data-plane authorization at the three seams that have one, and gives HistoryRead a check it never had. Read, Browse and Subscribe stay ungated — stage 6. The ACL decision REPLACES the server-wide role check rather than chaining with it. Chained, a grant would have to confer the role as well as pass the scope check, and only Cluster-scoped grants confer a role — so every narrow grant would have stayed inert and the flag would have looked inert too. The load-bearing test is an ALLOW: a session carrying no roles at all writing successfully on the strength of its grant. A deny-only test passes against a build where the ACL is never consulted, because the role gate denies too. Found en route: the principal never reached the session. The authenticator passed it as OpcUaUserAuthResult.DisplayName, but RoleCarryingUserIdentity derives DisplayName from the presented token — for a key-bearing client that is the static username `apikey`, identical fleet-wide. So the trie's matching key was dropped at the identity boundary, and the audit ClientUserId on a rejected write recorded `apikey` for every consuming system. Principal and ApiKeyId are now carried explicitly. Beyond the stage as sketched: - Revocation reaches an OPEN session. Every decision re-checks the authenticating key against the current snapshot; without it a key revoked by a deployment kept its authority until the client chose to reconnect, which for a machine-to-machine consumer can be indefinitely. - UserAuthorizationState is built per CALL, not per session. The per-session shape exists so an LDAP session can re-resolve membership that changed out of band; nothing can change out of band under API keys, so the generation stamp is the whole invalidation signal and a per-call record picks it up for free. IsStale/NeedsRefresh therefore cannot fire here and are documented as vestigial. - The trie is built inside DataPlaneAuthorityState.Apply, under the same generation as the snapshot, rather than by a second wiring at the publish actor. An uninstalled trie denies everything — indistinguishable from working enforcement. Deleting the install turns 13 of the 22 new tests red. - The deploy log reports what the BUILD dropped instead of re-deriving it. The old count agreed with the builder on one of its four drop reasons; a rule with a blank ScopeId, a retired scope kind, or a path not starting at a realm vanished with the log reading zero. Both IApiKeyStore implementations widened to the full grant set. The narrowing was redundant — ApiKeyInterimRoleMapper applies the same filter itself — and became a trap once scope was enforced. Pinned by asserting the roles are unchanged, with the two grants carrying different flags so it cannot pass by accident. Falsified by mutation: enforcement never engaging, the write seam ignoring the ACL, the realm dropped from the scope, the revocation check removed, and the trie never installed each turn the intended tests red and no others. Full solution suite green. Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo |
||
|
Joseph Doherty
|
3a6b2a1daf |
feat(acl): variable-depth scope model + the scopePaths producer (#520 stage 3)
Makes the /raw device tree addressable by an ACL rule, and builds the
scope-path producer PermissionTrieBuilder has accepted since it was written
but which had no caller anywhere.
The plan for this stage said "add NodeHierarchyKind.Raw, whose path is the
RawPath segments Folder/Driver/Device/TagGroup/Tag". That shape is wrong.
RawFolder.ParentRawFolderId and TagGroup.ParentTagGroupId are both
self-referencing, so folders and tag groups each nest without bound and a real
RawPath is folder…/Driver/Device/group…/Tag. NodeScope was fixed-arity and
PermissionTrie.CollectMatches was five hardcoded steps, so the scope model
could not address a real raw tree at all. NodeScope is now {ClusterId, Kind,
Segments} and the walk is a segment loop. The trie STRUCTURE was already
variable-depth (NodeAclPath.Segments is a list, Children is a recursive
dictionary), so only the walker and the scope record had to change.
Four decisions worth keeping:
- The namespace level became the realm. Level 1 gated on NodeScope.NamespaceId,
but v3 has no Namespace entity — the snapshot was retired when the address
space went dual-namespace, and the identifier appears nowhere in production
outside NodeScope. So the level could only ever fail to match, which silently
made every sub-cluster grant unreachable while cluster grants kept working.
It is now a realm token derived from the scope's Kind, not carried on it, so
the walk and the producer cannot disagree about its spelling.
- Provenance comes from the granting row's own ScopeKind, recorded on TrieGrant
at build time. Depth-derived provenance could not name a raw level at all,
since the depth of "Device" varies with folder nesting.
- The trie comparer moved from OrdinalIgnoreCase to Ordinal. Raw sibling-name
uniqueness is ordinal, so Speed and speed can both exist under one device and
a grant on either would have matched both.
- The null-scopePaths "deterministic test mode" is deleted. It planted
unresolvable rows at a direct child of the trie root, a position the walker
cannot reach, so the grant looked placed and could never match — and every
test using that mode proved nothing about production placement. scopePaths is
now required, and a row that does not resolve is dropped with a diagnostic.
A scope resolves to a LIST of paths: a raw tag projected into N equipments is
reachable at N+1 NodeIds and a write through any of them lands on the same
driver reference, so a Tag-scoped grant covering only the raw node would be
bypassable through a UNS one. The lookup is keyed on (ScopeKind, ScopeId), not
ScopeId alone — ScopeId has no FK and no type discriminator, so an id collision
across entity types would hand a grant the wrong node's paths.
Also closed here, all found by review rather than asked for:
- NodeAcl rows had no deploy-time validation of any kind and were not reachable
from DraftValidator at all (DraftSnapshot carried no NodeAcls). They are now
gated for dangling ids, cross-cluster scopes, blank/contradictory ScopeId, the
retired FolderSegment kind, and undefined ordinals.
- ConfigDbApiKeyStore conferred roles from Cluster-scoped grants authored in ANY
cluster, because it filtered on scope kind with no ClusterId predicate while
ArtifactApiKeyStore read an already-cluster-filtered snapshot. On a fused
admin+driver node that was a cross-cluster over-grant. Its doc comment claimed
the two stores matched.
- The two AdminUI ACL banners still said enforcement was LDAP-group -> role
mapping, which stages 1/2 made false in both directions.
- The /apikeys page was never classified in PageAuthorizationGuardTests, so that
test has been red since the key-minting commit merged — I did not run
AdminUI.Tests before merging stage 4.
Falsified rather than assumed: 8 mutations against the trie tests, 5 against the
deploy gate (including one proving a cross-cluster row is not silently
mis-diagnosed as unresolved), a staged red->green for the over-grant regression,
and the realm gate, the comparer and the malformed-path guard each broken
individually to confirm only their own tests go red.
Node ACLs are still not ENFORCED — nothing constructs a NodeScope or calls the
evaluator yet. That is stages 5 and 6; this stage makes the model able to
express what those will enforce.
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
|
||
|
Joseph Doherty
|
a438ac1640 |
feat(acl): keys + ACLs reach driver-only nodes via the artifact (#520 stage 4)
A driver-only node has had no ConfigDb since per-cluster mesh Phase 4, so it registered NullOpcUaUserAuthenticator and denied every UserName token. Its keys now ride the deployment artifact instead. - ConfigComposer snapshots ApiKeys alongside the NodeAcls it already carried. Only the peppered HMAC ships, never a secret, and the pepper is node config that is deliberately not in the artifact. - DeploymentArtifact.ParseDataPlaneAuthority decodes both node-side. Grants are scoped to the node's own cluster; keys are not, because ApiKey carries no ClusterId — a key is a fleet-wide credential and the ACL rows are what bind it. - OpcUaPublishActor installs them on DataPlaneAuthorityState on every apply, BEFORE its plan.IsEmpty early-return: an artifact that only revokes a key produces an empty address-space plan, so feeding after that return would leave the revoked key working until an unrelated structural change redeployed. - ArtifactApiKeyStore serves them to session activation; a node with a ConfigDb keeps ConfigDbApiKeyStore, whose per-request read is what makes revocation immediate there. - OtOpcUaNodeManager gains DataPlaneAuthority — the cluster identity blocker #4 named. NodeScope.ClusterId is required and the evaluator fails closed on a mismatch, so until now no scope it built could have evaluated. Nothing reads it until stages 5/6. The generation is a node-local COUNTER driven by the RevisionHash, not derived from its bits. PermissionTrieCache treats the highest GenerationId as current, so a generation taken from hash bytes would install a lower number about half the time and the cache would go on serving the previous deployment's ACLs, silently. Sessions are not stamped with it yet — that belongs with the first seam that reads a UserAuthorizationState, in stage 5, rather than being built now and consulted by nothing. Live-gated on docker-dev against site-a-1 (driver-only, no ConfigDb): key authenticates, wrong secret denied, both halves of the revocation asymmetry observed, and a SITE-A grant confirmed not to reach site-b-1. Key minted with an independent Python HMAC so the verifier was checked against something other than itself. Rig left clean. Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo |
||
|
Joseph Doherty
|
5a7a08deca |
fix(drivers): detect mid-run comms loss and recover from it (#527)
DriverInstanceActor.DisconnectObserved had no production sender. The only paths into Reconnecting were a failed *initial* connect and an operator's Force Reconnect, so a driver that connected and then lost its device stayed Connected indefinitely: publishing Bad for every tag, never re-initialising, and NOT recovering when the device returned. PublishHealthSnapshot read GetHealth() for reporting only and never acted on a driver reporting itself unhealthy. Both comms-loss quality paths -- #477's native-condition annotation and #481's scripted-alarm input fan-out -- hang off ConnectivityChanged, so both were correctly wired to a signal that never fired. The Connected health tick now evaluates a watchdog and self-sends DisconnectObserved when the driver reports a non-Healthy state AND its LastSuccessfulRead has been frozen for DefaultCommsLossTimeout (90s). The conjunction is the design, and the issue's own proposal -- "non-Healthy for N consecutive polls" -- is wrong. Degraded means *connected but some data is bad*; several drivers write health per tag inside the read loop (ModbusDriver.ReadAsync), so one unreadable address latches Degraded forever against a perfectly reachable device, and a state-only watchdog would reconnect that driver every 90s indefinitely -- worse than the bug. LastSuccessfulRead is the honest evidence: preserved across Degraded/Faulted writes, advanced by any success. The state check remains the second conjunct so an idle Healthy driver with nothing subscribed is not declared dead for not reading. Verified across all 12 drivers that a sustained failure does reach a non-Healthy state -- nine poll drivers via Degraded, MQTT via live connection state, Galaxy via its reconnect supervisor. CalculationDriver reports Healthy while running and so can never trip, which is correct: it has no transport. Also: the Reconnecting retry now calls ShutdownAsync before InitializeAsync. ModbusDriver (_transport) and MqttDriver (_connection) overwrite an IAsyncDisposable field inside InitializeAsync without disposing the previous one, so re-initialising without a teardown leaks a socket per attempt -- latent while only a manual Force Reconnect reached this path, but the watchdog makes it automatic and unbounded (~360 leaks for an hour-long outage at the 10s retry). Tests: 8 new in DriverInstanceActorCommsLossWatchdogTests, each absence assertion carrying a positive control (a proven count of processed health-poll ticks). Falsified five ways -- dropping either conjunct reds exactly the test written for it, and removing the teardown, the null baseline, or the watchdog call each reds its own guard. Live gate PASSED on docker-dev (Modbus + otopcua-pymodbus-standard), run twice: - fixture stopped -> watchdog fires at ~108s, "declaring comms lost" - fan-out observed at Debug: "pushed BadNoCommunication for 2 mux ref(s)" (#481 Layer 4) and the #477 native-condition annotation on a driver owning conditions - fixture restarted -> reconnected in ~8s with NO node restart; both the raw (ns=2) and UNS (ns=3) NodeIds read Good with fresh timestamps Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo |
||
|
Joseph Doherty
|
c81cef39f5 |
feat(alarms): bridge driver comms-loss into scripted-alarm input quality (#481)
Layer 4 of the condition-Quality chain (#477 -> #478). #478 gave a scripted condition the worst quality of its input tags, but only for inputs whose driver PUBLISHES a Bad data change. A poll driver whose device goes unreachable publishes nothing — it raises ConnectivityChanged(false) and goes silent — so the engine held its last Good value, WorstInputStatus stayed Good, and worse, AreInputsReady stayed true, so the predicate went on evaluating a stale value after the source had died. `DriverHostActor.FanCommsLossToDependencyMux` pushes a synthetic BadNoCommunication with a null value into the dependency mux for every value ref the disconnected driver owns. The issue asked for a driver -> all-tag-refs map to be built for this. It already exists: `_nodeIdByDriverRef`, keyed (DriverInstanceId, RawPath) and rebuilt from the composition on every apply. Alarm-bearing tags live in the separate `_alarmNodeIdByDriverRef` and are deliberately excluded — a native condition is not a mux value and takes the Part 9 quality path instead. Nothing is pushed on reconnect, deliberately: publishing Good would assert a value quality this actor has not observed. Recovery rides the driver's own republish, and the mechanism was verified rather than assumed — the Connected -> Reconnecting edge calls DetachSubscription, which nulls the subscription handle, so the reconnect's ResubscribeDesired builds a FRESH subscription whose PollGroupEngine initial push runs with forceRaise: true. Without that detach the poll engine would compare against its retained LastValues, see an unchanged value, raise nothing, and strand these refs Bad forever — the value-dedup trap that bit the v3 Batch 4 virtual-tag reassert. Closes #478 Finding 3, which the above depends on. WorstInputStatus skipped every input whose VALUE was null as "no data yet" — and Bad-with-no-value is exactly the shape a comms-lost driver produces, so the skip was hiding the case that matters most; without this half the fan-out above would be silently inert. The upstream source now stamps a never-pushed path BadWaitingForInitialData — the honest Part 4 code for that state — and the engine skips that specific code rather than every null value. Deleting the null-skip outright is the obvious fix and is wrong in the other direction: it makes every not-yet-published input drag its condition Bad at deploy, flooding the quality path with load-time annotations, which is what the skip existed to prevent. Both mistakes are pinned by a test pair that goes red under the opposite error, and both were falsified by reverting the production code: old null-skip restored -> A_pushed_Bad_carrying_no_value RED null-skip deleted outright -> An_input_that_was_never_pushed RED fan-out call removed -> the three DriverHostActor tests RED (1 of 7) The never-pushed test was vacuous on first writing and is worth recording: the quality bucket only emits on CHANGE, so the load-time evaluation is the one that flashes Bad, and subscribing to OnEvent after LoadAsync made the assertion unfalsifiable — it passed against the broken behaviour. It now subscribes first. `FakeUpstream.ReadTag` returned 0x80340000 (BadNotConnected) for an unset path, a different Bad code from production's placeholder. Harmless while any Bad behaved alike; load-bearing now, so it is aligned and pinned by a reflection guard asserting the engine's private constant agrees with what the production source stamps. Behaviour change to weigh, flagged in the issue and confirmed: virtual tags computed from a comms-lost input now go Bad too. That follows from using the shared mux and reads as correct, but it is broader than scripted alarms. Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo |
||
|
Joseph Doherty
|
b815446940 |
refactor(drivers): delete GetMemoryFootprint / FlushOptionalCachesAsync (#525)
Both `IDriver` members had no production call site. Their only consumer was `MemoryTracking`, deleted with the driver-tier recycle machinery in #522, and `GetMemoryFootprint`'s doc-comment claimed "polled every 30s by Core" — which had not been true since v3, even before #522 removed the poller. Owner decision on #525 was (a) delete rather than (b) build the process-wide watchdog that would consume them. The driver-side implementations are in git history and were mostly reusable; what was always missing is the trigger, and a member nobody calls is how the misleading doc-comment survived this long. Removed from `IDriver`, all 12 driver implementations, and every test stub — plus the support code that existed only to feed them, which is the part a member-only deletion would have left behind as dead weight: - MTConnect's three per-entry byte-weight constants, its cached `ProbeModelDataItemCount`, and `CountDataItems` — a full device-tree walk run on every cache fill to populate a number with no remaining reader. - OpcUaClient's `_discoveredNodeCount` field and the discovery-pass write to it. - MQTT's `ApproxBytesPerAuthoredTag`. Two behavioural notes, neither a change in what the software does: - `AbCipTemplateCache` now has no production reset path. Its only caller was AB CIP's flush, which itself had no caller — so nothing changes except how visible the fact is. Recorded on the type, along with the correction that its comment's claim about a device `ReinitializeAsync` also flushing it was never true. A UDT redefined on a live controller keeps its stale shape until the driver instance is rebuilt. - MTConnect's `/probe` cache is still dropped, by `InvalidateProbeModel()` on an Agent restart. That is now `internal` rather than `private` so the tests which used the flush purely to empty the cache drive the real path instead — they now cover a case that actually occurs in production. Several unrelated doc comments cited `FlushOptionalCachesAsync` as the canonical CAS example and are repointed to it. Tests: the ones that existed only to assert on the removed members are deleted (3 OpcUaClient, 3 Galaxy, 2 MQTT, 1 MTConnect, 1 AbCip). Worth noting one was vacuous — `FlushOptionalCachesAsync_resets_footprint_counter` asserted a footprint of 0 before the flush and 0 after with no discovery in between, so it would have passed against a flush that did nothing at all. Full suite green except 7 failures verified identical on master before and after (4 AbLegacy integration with the compose profile down, 3 Host.IntegrationTests), checked by name rather than assumed. Docs corrected where they described this as live: `DriverLifecycle.md` (which carried the #525 placeholder), `drivers/AbCip.md`, `drivers/AbLegacy.md`, and `v2/driver-stability.md` — whose "polls this every 30 s" + per-driver `DriverConfig` memory budget were never built as written and are now banner-ed as design intent rather than description. Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo |
||
|
Joseph Doherty
|
7f2108dcf2 |
feat(mqtt): Sparkplug primary-host STATE publishing, Primary-gated (#511)
`ActAsPrimaryHost` was accepted by config and by the AdminUI form and did nothing but
log a warning. It now delivers the whole three-part contract or none of it: a retained
`ONLINE` on `spBv1.0/STATE/{hostId}`, a matching retained `OFFLINE` armed as the MQTT
Last Will, and an explicit `OFFLINE` on a clean stand-down.
**The claim is a dedicated connection, not a message.** A Will can only be registered in
the CONNECT packet — it cannot be added to a live session, changed, or withdrawn. So a
claim that is acquired and released at runtime cannot ride the long-lived ingest session.
`SparkplugPrimaryHost` opens its own connection (client id `{clientId}-spbhost`, forced
clean-session) when the node gains Primary and closes it when the node loses Primary; the
connection's lifetime *is* the claim, so every leg follows from its lifecycle. Ingest is
untouched by a role change — no dropped subscriptions, no lost birth cache, no data gap at
the Secondary->Primary transition.
**Primary-gated** (owner decision, recorded on the issue). Both pair nodes subscribe
independently, so an ungated claim means both claim the same host id — and the sharp
failure is not the two fighting over the retained STATE, it is that the SECONDARY's death
fires its Will and publishes OFFLINE while the Primary is healthy, stopping every edge node
in the group. Registering the Will only on whichever node is Primary at connect time fails
the other way: a node that later gains Primary would publish ONLINE with no Will behind it.
New seam: `IDriverPrimaryRoleSource` (`Core.Abstractions`) carrying the *data-plane* verdict
(`PrimaryGatePolicy.ShouldServiceAsPrimary`), not the alarm-drain verdict — an unknown role
with a driver peer present must deny. Registered unconditionally by `AddOtOpcUaRuntime`,
deliberately NOT alongside `IRedundancyRoleView`, which `AddAlarmHistorian` registers only
when that section is enabled: piggybacking would leave a pair with no alarm historian
falling back to "claim unconditionally". Polled on a 1s timer rather than evented — a
process singleton raising events at per-instance drivers outlives every driver it holds.
Accepted trade: a brief edge-visible OFFLINE window during failover. That is a genuine
state transition, not a glitch to hide.
Live-gated 5/5 against the Mosquitto fixture, including the Will leg — `AbandonForTests`
drops the socket with no DISCONNECT, because a clean disconnect makes the broker discard
the Will by protocol rule, so a test driven through `StopAsync` would observe the
stand-down OFFLINE and pass against an implementation that never armed a Will. Falsified:
removing the Will turns that leg and only that leg red.
Also corrects two stale claims found en route: the AdminUI form still said MQTT tags are
read-only (#508 shipped), and the driver doc still said nothing consumes `IRediscoverable`.
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
|
||
|
Joseph Doherty
|
2d20aeb779 |
fix(opcua,drivers): carry driver status sub-codes to clients (#519) + health-write ordering (#517)
#519 — a driver's OPC UA status code now reaches a client with its sub-code intact. DriverInstanceActor projected the driver's uint onto a 3-state Good/Uncertain/Bad enum and OtOpcUaNodeManager re-expanded Bad to a bare 0x80000000, so BadNoCommunication, BadTypeMismatch and BadNotSupported were indistinguishable on the wire and every driver's status constants (incl. the 16 #497 had just corrected) were diagnostic in logs only. Resolved by option (b): the raw uint rides AttributeValuePublished -> AttributeValueUpdate -> IOpcUaAddressSpaceSink.WriteValue, OpcUaQuality is deleted, and severity classification is a helper (OpcUaStatus.IsGood/IsUncertain/IsBad). The surface was wider than the issue costed (41 files, not "the sink and 3 implementations"): OpcUaQuality had non-driver producers with no uint to carry. Each got its natural code — BadInternalError for a failed virtual-tag script, BadNoCommunication for driver comms loss — and ScriptedAlarmHostActor's pair of mutually-inverse uint<->enum mappers is deleted, having round-tripped a code the engine already held. AddressSpaceApplier's removal writes deliberately keep the generic Bad: not losing a code the driver had is this issue; inventing one for the deploy path is a separate call. Behaviour change: WriteAlarmQuality fires its Part 9 event on a status-code change rather than a severity-bucket change. While the annotation was 3-state those were the same predicate; now a move between two Bad reasons changes what a subscriber reads. New SDK-free named constants live on Commons.OpcUa.OpcUaStatus, and StatusCodeParityTests is widened to cover them — that guard is reflection-driven, so it can only see a named constant, never an inline literal. #517 — the symptom is real but the issue named the wrong path and its fix was unsafe. ShutdownAsync cannot race: DriverInstanceActor calls it from PostStop, after the mailbox drains. (Core/Hosting/DriverHost, the one class with a concurrent GetHealth, has no production call site.) InitializeAsync can: it runs on a fire-and-forget Task.Run, and MTConnectDriver writes Healthy then starts loops that can throw into a catch which disposes the client before writing Faulted. Moving the write earlier, as suggested, inverts the terminal state — teardown drains in-flight loops that write health as they unwind, so a late loop could leave Healthy at rest with every connection closed. The fix is a bracket (conservative before, terminal after), applied to MTConnect's init-failure path only; shutdown keeps its single trailing write, and the rule plus the reachability table are documented on IDriver.ShutdownAsync. CalculationDriver.GetHealth returned a hard-coded Healthy that consulted nothing — Healthy before initialize, after shutdown and after dispose. Now derived from _running. Its test, named Health_is_always_Connected, asserted the defect and is replaced. Every new guard was falsified by breaking the production code and watching it go red. The two sub-code tests each assert two different Bad codes, since a single-code assertion could pass against a hard-coded constant under the old projection. Pre-existing failures, verified unchanged on a stashed tree and not chased: 4 in Driver.AbLegacy.IntegrationTests (slc500/micrologix/plc5 fixtures not up) and 3 in Host.IntegrationTests. Closes #517 Closes #519 Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo |
||
|
Joseph Doherty
|
30d0697c28 |
feat(hosts): surface per-host connectivity on /hosts; drop the unwritable table (#521)
`IHostConnectivityProbe` was a dead surface: eleven drivers implement it, `GetHostStatuses()` had ZERO production call sites, and `OnHostStatusChanged` had no subscriber outside the Galaxy driver's own aggregator. Per-host connectivity was computed by every driver and read by nobody. The issue offered "build the publisher or delete it". The publisher as its entity doc described it — driver nodes upserting `DriverHostStatus` rows — is not buildable: per-cluster mesh Phase 4 gates `AddOtOpcUaConfigDb` on the `admin` role, so a driver-only node has no ConfigDb connection to write rows with. So the capability is kept and the transport changed. `DriverHealthChanged.HostStatuses` now carries the probe result to `/hosts` as a Hosts column. That channel already reached the page, already survives the mesh split via the Phase 5 gRPC telemetry stream, and already replays a last-value snapshot on re-subscribe — so per-host state re-primes after a reconnect without a durable store. Both halves of the interface finally do what they are for: the event triggers a prompt publish, the pull is the source of truth. The point of the column is the case the driver-level state chip structurally cannot express: a multi-device driver stays aggregate-Healthy while ONE of its devices is unreachable. Two traps, both pinned by tests that were falsified against the prod code: - The host digest MUST be in the publish fingerprint. On a single-host-down transition every other fingerprint component is unchanged, so the dedup would swallow exactly the publish carrying the news — the trap that already bit the rediscovery signal. Removing it turns the guard test red, verified. - null (no probe) must stay distinct from empty (probe with no hosts). proto3 cannot tell an absent repeated field from an empty one, hence the explicit `has_host_statuses` flag; collapsing them would render every probe-less driver as one whose devices are all fine. Dropped: the DriverHostStatus entity, enum, DbSet, model config and table (migration DropDriverHostStatusTable — empty on every deployment, so the scaffolder's data-loss warning is moot, and Down() recreates it exactly). Found en route, NOT fixed here: `DriverInstanceResilienceStatus` is the identical defect — no writer, no reader, only a DbSet declaration, while the live data rides the `driver-resilience-status` telemetry channel. Its doc-comment now states that rather than describing the sampler and AdminUI join that were never built. Filed as #524 rather than widening this schema change beyond what was asked. Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo |
||
|
Joseph Doherty
|
5fff597324 |
revert(drivers): remove the periodic 5-minute device re-browse (#523)
Removes the tag-set drift detector shipped in |
||
|
Joseph Doherty
|
e77c8a3569 |
feat(drivers): tag-set drift detection, gated on SupportsOnlineDiscovery
The second thing #516/§8.2 deliberately did not build. The original objection stands — Modbus, S7, MQTT, AbLegacy and Sql echo authored config from DiscoverAsync, so diffing discovered against authored is a tautology for them — but it is now ENCODED rather than used as a reason not to build. The discriminator already existed: ITagDiscovery.SupportsOnlineDiscovery is documented as "enumerates the tag set from the live backend rather than replaying pre-declared/authored tags", and it separates the fleet cleanly — FOCAS, TwinCAT, MTConnect and AbCip true; the five config-echo drivers explicitly false. The check runs only for a driver that is SupportsOnlineDiscovery AND reports the new AuthoredDiscoveryRefs. That is nullable rather than empty-by-default on purpose: an empty collection legitimately means "nothing authored, everything discovered is new", so conflating the two would report total drift for a driver that never opted in. Where the value is: AbCip and FOCAS browse a live backend but implement no IRediscoverable, so before this they had no change signal at all. A periodic compare is the only way to notice a PLC re-download. A finding that changed the design: FOCAS, TwinCAT and AbCip all re-emit their authored tags into the same DiscoverAsync stream as device-derived ones, so the browse picker can show an operator their existing tags. Discovered is therefore always a superset of authored, and a VANISHED tag can never appear missing — one whole direction is structurally undetectable for three of the four. Shipping a Vanished list that is permanently empty for them would have been the half-inert seam this whole exercise removes. So the driver declares DiscoveryStreamIncludesAuthoredTags, the detector does not compute the undetectable half, and the operator message says "missing-tag detection unavailable for this driver" rather than letting the absence of a missing-tag clause read as reassurance. MTConnect is the only driver where both directions work — its stream is purely probe-model-derived. Behaviour: a failed browse is not drift (an unreachable device would otherwise report every authored tag as vanished); a truncated capture is skipped for the same reason; an unchanged drift is reported once rather than re-stamping its timestamp; drift that resolves clears the prompt. Interval defaults to 5 minutes — it browses a real device, and a tag set changing is an engineering event, not a runtime one. It reuses the Stage-2 surfacing, raising the same /hosts re-browse prompt, and never rebuilds the served address space. Comparison logic is a pure, separately-tested type rather than actor-inline. 14 new tests. The gate was verified load-bearing by deleting the SupportsOnlineDiscovery half: the tautology-driver test goes red, and it asserts discovery was never invoked rather than merely "no prompt raised", which would have passed for the wrong reason if the timer never fired. Full suite green except the same three pre-existing fixture-gated integration suites, identical counts. |
||
|
Joseph Doherty
|
d32d89c340 |
fix(drivers): stop silently discarding driver config edits (§8.3, #516)
Five drivers ignored the config handed to them on reinitialize, serving the
options their constructor captured — and the deployment sealed green anyway.
Fixed on both halves, as chosen.
Seam (load-bearing): DriverSpawnPlanner routes a changed DriverConfig to
ToStop + ToSpawn instead of an in-place delta, making the factory the single
parse authority. This REVERSES the deliberate decision documented at
DriverSpawnPlan.cs:49-50 ("a pure DriverConfig change stays an in-place delta
— no reconnect"). That reasoning was correct about the resilience pipeline and
wrong about the driver. The accepted price is a reconnect per config edit.
Per-driver, and this split was not anticipated:
- Re-parse in place — Modbus, AbLegacy, OpcUaClient. ParseOptions extracted
from each factory, called from InitializeAsync behind a HasConfigBody guard
so "{}" still keeps the constructor options.
- Respawn-only, deliberately NOT re-parsed — Sql and FOCAS. Each builds more
than options from config (Sql's dialect + resolved connection string,
FOCAS's client-factory backend, both injected at construction), so adopting
new options alone would run a NEW tag set against an OLD connection. Half a
re-parse is worse than none. SqlDriver's doc-comment asserted the opposite
premise — "config parsing belongs to the factory, which builds a fresh
instance" — which was false when written and is true only now.
Two green seals removed from ApplyChildDelta: it overwrote the cached Spec
synchronously BEFORE the child dequeued the message, so the host believed the
new config was live and the next reconcile computed no delta, sealing the
drift permanently; and it Tell'd with no Receive<ApplyResult> registered, so a
failed reinit — including Galaxy's deliberate NotSupportedException —
dead-lettered.
Exposed (not created) by the change: a factory throw is a CONFIG error, and
SpawnChild catches it and silently substitutes a stub. Only a brand-new driver
could reach that before; an ordinary config edit can now. Raised Warning ->
Error with an actionable message. It still does not fail the deployment —
doing so would let one malformed driver block a fleet deploy, so that is a
follow-up rather than a drive-by.
Tests: every pre-existing reinit test in these suites passes "{}", the exact
input a guarded re-parser treats as "keep constructor options" — blind to this
defect by construction. New tests pass CHANGED json; the Modbus one was
verified falsifiable by deleting the re-parse (goes red). Two tests asserted
the old behaviour and were rewritten, including the positive control in
DriverHostActorUnreadableArtifactTests, whose observable moved from
UnsubscribeAsync to ShutdownAsync now that teardown happens by stopping the
child rather than emptying its desired set.
Remaining full-suite failures are pre-existing and environmental, verified
identical on the pre-change tree: Host.IntegrationTests (3) and
Driver.AbLegacy.IntegrationTests (4, docker fixture-gated).
|
||
|
Joseph Doherty
|
adce27e7fa |
refactor(drivers): delete the dead discovered-node injection path (§8.2, #507)
#507 was filed as "injection inert in v3, re-migrate onto the raw subtree". Reading it, the retained code is not revivable: it resolves equipment from EquipmentNode.DriverInstanceId UNION EquipmentTags, and BOTH are structurally empty in v3 (AddressSpaceComposer always constructs EquipmentNode with a null DriverInstanceId; DeploymentArtifact hard-codes an empty EquipmentTags set). Removing the guard would have changed a log line and injected nothing. So it is deleted rather than fixed, and #507 closes as superseded by /raw browse-commit. Removed: HandleDiscoveredNodes, PartitionDiscoveredByDeviceHost, ShouldWarnPartition, PlansRoutingEqual, ApplyDiscoveredPlansForDriver, _discoveredByDriver, the redeploy re-inject tail, DiscoveredNodeMapper, DiscoveredInjection, AddressSpaceApplier.MaterialiseDiscoveredNodes, OpcUaPublishActor.MaterialiseDiscoveredNodes, and the Runtime-local copies of CapturingAddressSpaceBuilder/DiscoveredNode. The connect-time discovery loop goes too (StartDiscovery, RediscoverTick, HandleRediscoverAsync, DiscoveredNodesReady, TriggerRediscovery). With injection gone it had no consumer, and leaving it would either dead-letter or keep browsing real devices up to ~15x per connect to drop the result. ITagDiscovery itself stays — the /raw browse picker drives it through Commons/Browsing/DiscoveryDriverBrowser, which never went through the actor, so the picker is unaffected. deferment.md §4.4 called the 18 DiscoveryInjectionDormantV3 tests "Real — blocked on #507". They are not: they assert an equipment-rooted graft (EquipmentRootNodeId == "EQ-1") that v3 cannot produce, so they could never have unskipped as written. Deleted along with DiscoveredNodeMapperTests, the CapturingAddressSpaceBuilder tests, and the MaterialiseDiscoveredNodes tests in AddressSpaceApplierTests/OpcUaPublishActorTests. Runtime.Tests skipped: 31 -> 13. IHostConnectivityProbe is deliberately NOT resolved here. GetHostStatuses() has no production call site and nothing writes a DriverHostStatus row, but the table was re-created in the v3 initial migration, so deleting on inference would be wrong. Split to Gitea #521 with both options costed. CLAUDE.md, docs/drivers/{Galaxy,TwinCAT,MTConnect}.md updated — they described the seam as dead. Build clean; Runtime.Tests 476 passed / 13 skipped; OpcUaServer.Tests 362 passed / 4 skipped. |
||
|
Joseph Doherty
|
09a401b881 |
feat(drivers): consume IRediscoverable as an operator re-browse prompt (§8.2, #518)
Nine drivers raise IRediscoverable.OnRediscoveryNeeded when they observe that a remote's tag set may have changed — a Galaxy redeploy, a TwinCAT symbol-version bump, an MTConnect agent restart, a Sparkplug rebirth. Nothing in src/ subscribed, so every raise went into the void. Drivers even wrap the invoke in try/catch for "subscriber threw"; there has never been a subscriber. DriverInstanceActor now attaches the event in PreStart and detaches in PostStop, mirroring AttachAlarmSource/DetachAlarmSource. Attach is per-actor, not per-connect: the raise has no connection affinity and can land while the driver is between connects. The detach is load-bearing rather than tidy — the IDriver instance outlives the actor when the host respawns a child around the same driver object, so a missing -= accumulates one handler per respawn, each holding a dead Self. The signal rides the existing driver-health snapshot to /hosts as a "re-browse" chip. It is ADVISORY: the served address space is deliberately not rebuilt, because v3 authors raw tags explicitly through the /raw browse-commit flow and a runtime graft would materialise nodes no operator approved and no deployment artifact records. Two things worth knowing: - PublishHealthSnapshot dedups on a health fingerprint, and a rediscovery raise on an otherwise-healthy driver changes none of the four fields it hashed. Without adding the timestamp to that tuple the dedup swallows the exact publish carrying the signal. Reverting that one line turns 3 of the 5 new tests red, which is how it was confirmed rather than assumed. - The new test stub implements IDriver from scratch instead of deriving from the shared StubDriver, whose GetHealth() returns DateTime.UtcNow — its fingerprint differs on every call, so the dedup never engages and the dedup test would have passed vacuously either way. The planned discovery-result drift detector was NOT built. Modbus, S7, MQTT, AbLegacy and Sql all echo authored config from DiscoverAsync rather than browsing the device, so a discovered-vs-authored diff is permanently empty for them — it would have been another plausible-looking inert seam, which is the class this audit exists to remove. IRediscoverable is the trustworthy signal because the driver asserts it deliberately. DriverHealthChanged, IDriverHealthPublisher.Publish and HostsDriverRow gain defaulted optional parameters, so no existing call site changed. telemetry.proto gains fields 8/9 and both Phase-5 mappers carry them. Runtime.Tests 512 passed / 31 skipped. |
||
|
Joseph Doherty
|
4b9bcddbcc |
feat(drivers): periodic reconcile of desired vs actual subscriptions (#488)
Option A from the issue — the cheap consistency check, no interface change. The desired ref set was re-applied on a Connected TRANSITION (`ResubscribeDesired`) and on a deploy's `SetDesiredSubscriptions`, and nowhere else. A subscription lost while the driver STAYED Connected — a subscribe that failed and was never retried, a handle dropped down an error path — left the actor subscribed to nothing while still reporting Healthy, indefinitely. `ReconcileSubscription` now rides the existing 30 s `health-poll` tick: while Connected, a driver holding a desired set but no live handle is re-subscribed. Gated on `ISubscribable`. A driver that cannot subscribe at all can still be handed a desired set, and without the guard it would be told `Subscribe` every tick and fail every tick, forever. This is a state-machine consistency check, NOT a probe — it can only see that this actor holds no handle, never that a handle is present but dead server-side, because `ISubscriptionHandle` is opaque. That is option B, and it stays deferred until a real occurrence shows the handle outliving the subscription; it would need a liveness member implemented across all eight drivers with different subscription mechanics. A redundant `Subscribe` is possible (a tick queued ahead of an already-queued one observes the same no-handle state) and is harmless: `HandleSubscribeAsync` is a `ReceiveAsync`, so no tick is processed mid-subscribe, and it drops any prior subscription before establishing the new one. Suppressing it would need a pending-subscribe flag threaded through every self-tell site — more state than the race is worth. `healthPollInterval` becomes an optional Props parameter, mirroring the existing `rediscoverInterval` seam, so a test can drive the reconcile without waiting 30 s. Three tests. The reconcile itself is proven by a stub whose FIRST subscribe throws: nothing else would ever retry it, so a second attempt can only come from the reconcile — disabling `ReconcileSubscription` turns it red. The two guard tests are absence assertions and are bounded by a positive control rather than a sleep: a counting health publisher makes ticks PROCESSED observable, so "still one subscribe" means the reconcile ran and declined, not that the timer had not fired. Explicitly not deploy-time drift (#486) — re-asserting an already-correct desired set would not have helped there. |
||
|
Joseph Doherty
|
549c656489 |
fix(alarms): re-assert non-normal scripted-alarm conditions after a (re)load (#487)
A deploy that triggers a FULL address-space rebuild clears `_alarmConditions` and re-materialises every condition node fresh — inactive, acked, confirmed, unshelved. The engine reloads its persisted state and re-derives Active from the predicate, so it ends up correct; but there is no transition to report, so `LoadAsync` yields `EmissionKind.None` and `OnEngineEmission` drops it. Nothing writes the node. Live-reproduced on the docker-dev rig against the pre-fix image: an alarm that had fired and cleared but was still UNACKNOWLEDGED came back from a rebuild reporting Acknowledged with Retain=false — so it disappeared from ConditionRefresh entirely, while the engine still held Unacknowledged. An operator's outstanding alarm silently vanishes from the alarm list on deploy. `ScriptedAlarmHostActor.ReassertConditionNodes` closes it. After each load it reads the new `ScriptedAlarmEngine.GetProjections()` and sends one `AlarmStateUpdate` per alarm NOT in the Part 9 no-event position. Load-bearing properties: - Node-only. It sends `AlarmStateUpdate` and nothing else — never the `alerts` topic, never the telemetry hub. `alerts` is the historization path, so a row per alarm per deploy would append a duplicate historian/AVEVA record every time anyone deploys. This is why it reads a projection rather than re-emitting: `ScriptedAlarmProjection` carries no `EmissionKind`, so there is nothing for the emission machinery — or a future refactor — to mistake for a transition. The issue's sketch said `GetState(alarmId)` + `LoadedAlarmIds`; that alone would have clobbered the node's severity and message, so the projection also carries the definition's severity, the resolved message template, and the last-observed worst-input quality. - Only non-normal alarms. One in the no-event position already matches what the materialise built, so a never-fired alarm stays byte-identical to a deploy without this pass (including its Message, which the materialise seeds with the display name), and an all-normal deploy writes nothing at all. - Timestamp is the condition's own `LastTransitionUtc`, not the deploy instant. - No duplicate Part 9 events: `WriteAlarmCondition`'s delta-gate compares against the node's CURRENT state, so a re-assert onto a freshly materialised node is a genuine delta (fires once — correct, the rebuild reset what clients see) while one onto a surgically-preserved node is suppressed. Tests: 6 new host tests + 6 new engine tests. Falsifiability checked by disabling the call — 4 of the 6 host tests go red; the other 2 are absence guards against an over-broad fix and pass either way by design. Live gate (docker-dev, central-2, pre-fix image vs. rebuilt image): - pre-fix: condition absent from ConditionRefresh after a rebuild; - post-fix: present, Unacknowledged, Retain=True, stamped with its own LastTransitionUtc rather than the restart instant; - /alerts stayed empty across two re-asserts, with the panel proven live by a real ACTIVATED transition immediately afterwards — no duplicate history. |
||
|
Joseph Doherty
|
87ff00fd30 |
feat(mesh): RoleParser accepts cluster-{ClusterId} roles; consolidate the driver-role literal
Phase 6 (per-cluster mesh split) needs nodes to carry a cluster-scoped role like cluster-SITE-A; RoleParser previously rejected anything outside the fixed admin/driver/dev set. Adds IsClusterRole/ ClusterIdFromRole plus well-known-role constants, and points the three duplicate "driver" string literals (RedundancyStateActor, ClusterNodeAddressReconcilerActor, ServiceCollectionExtensions) at the new RoleParser.Driver so the value has one source, without renaming any existing DriverRole symbol. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
909d75357b |
test(mesh-phase5): cover the ScriptedAlarmHostActor + VirtualTagActor telemetry taps
Closes two test-completeness gaps flagged in review of the seam-tap commit — the ScriptedAlarmHostActor and VirtualTagActor hub emits had no test proving the hub actually receives them. - ScriptedAlarm_activation_emits_one_Alarm_item_with_matching_payload: spawns the host with a capturing fake hub, drives a real engine Inactive->Active transition through the existing ScriptedAlarms harness, and asserts exactly one TelemetryItem.Alarm carrying the Activated transition. - VirtualTag_script_log_emits_one_Script_item_with_same_payload: passes a fake hub via Props + a publisherFactory, drives an evaluator failure, and asserts a TelemetryItem.Script carrying the SAME ScriptLogEntry instance handed to the publisher. To make the VirtualTagActor tap observable, its hub emit moved to the TOP of PublishLog, before the test-only publisherFactory early-return branch, so the emit fires on BOTH the production DPS path and the factory path. Production behavior is unchanged: production passes publisherFactory: null, so it still reaches the DPS Tell; the emit just moved a couple lines earlier (both are fire-and-forget). Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
7bdb6d00ec |
fix(mesh-phase5): telemetry hub — plain Dictionary under gate, concurrency stress test, priming/eviction notes
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
8e5090edf3 |
feat(mesh-phase5): tap the 4 telemetry publish seams into the local hub (DPS intact)
Each of the four telemetry publish seams now also emits its DPS payload into the node-local ITelemetryLocalHub (Phase 5), leaving the existing DistributedPubSub publish untouched: - driver-health AkkaDriverHealthPublisher -> TelemetryItem.Health - resilience DriverResilienceStatusPublisherService -> TelemetryItem.Resilience - alerts ScriptedAlarmHostActor + DriverHostActor (native) -> TelemetryItem.Alarm - script-logs VirtualTagActor + DpsScriptLogPublisher -> TelemetryItem.Script DI services take the hub as a required ctor param; actors take an optional nullable hub threaded through their Props and the real spawn sites (DriverHostActor <- ServiceCollectionExtensions; VirtualTagActor/ScriptedAlarmHostActor <- DriverHostActor), so existing test Props constructions keep compiling and simply do not emit. The hub is a no-op until a gRPC client subscribes, so the emit is safe on every node. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |