f13a89dbc349eabb218948a7eb29d2ad20bbe8f6
688 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Joseph Doherty
|
f13a89dbc3 | docs(mtconnect): final-review nits — two stale read-only comments; task statuses | ||
|
Joseph Doherty
|
203000a80a | docs: MTConnect write-back recorded; stale MQTT write-through/browse-gap records reconciled | ||
|
Joseph Doherty
|
e784545844 | docs(plan): MTConnect write-back design + implementation plan; MQTT P3 recorded as already shipped | ||
|
Joseph Doherty
|
de036c62aa | chore(plan): mark the raw-modal picker live-config gate task complete | ||
|
Joseph Doherty
|
8af5742406 |
docs(raw): live /run gate PASSED for the raw-modal picker live config; records Gitea #539
All three legs pass against the patched AbCip fixture: the edit modal and the manual-entry grid both browse the live nine-tag symbol table, and an unreachable device degrades to the graceful error chip while the modal still saves. The gate also found a defect unrelated to this change and filed it as #539: the AbCip discovery path (LibplctagTagEnumerator, LibplctagTemplateReader) passes libplctag the bare host, so an authored non-default EtherNet/IP port is dropped and the browse reaches whatever answers on 44818 — #534's defect surviving in the discovery half. Discovered because the first negative-leg endpoint differed from the fixture only by port, which never reached the wire. |
||
|
Joseph Doherty
|
4b88dc6f9a | docs(raw): final-review nits — commit count, closed cross-reference, forced accessibility asymmetry | ||
|
Joseph Doherty
|
c1e8762cd4 |
docs(raw): close the #468 follow-up — raw modals now feed pickers live merged config
Records the outcome in the design doc §6 (including the stale-open race the review found and fixed in both modals), closes the follow-up in the tracking doc and the universal-browser plan, and registers RawBrowseModal's shared race shape in deferment.md §0.1 as the deliberate leftover. |
||
|
Joseph Doherty
|
ebcf73f72c | docs(plan): implementation plan for the raw-modal picker live-config fix | ||
|
Joseph Doherty
|
3b9009da47 |
docs(drivers): tracker catches up with the #468 gate closure; design for the last dangling follow-up
The tracking doc and program-design §9 still showed Wave 0's live gate open —
it closed 2026-08-01 (
|
||
|
Joseph Doherty
|
4bdce91002 |
docs(historian): live gate for #537 and #538 — both PASSED
Ran on the docker-dev central pair against the local HistorianGateway container talking to the real
AVEVA Historian on wonder-sql-vd03 (VPN up), with the host image rebuilt at
|
||
|
Joseph Doherty
|
e7fbe97759 |
fix(historian): gate historization capture on the Primary, quarantine a poison tag (#537, #538)
Both defects were found by #491's live gate against a real HistorianGateway and filed rather than fixed at the time, because each one's fix is a design decision rather than a patch. #537 — both nodes of a redundant pair captured every value, so the historian received duplicate live samples for every historized tag (measured: the gateway's write counter advanced twice per value and halved when one central was stopped). The recorder now reads a capture gate per value, fed from the new PrimaryGatePolicy.ShouldRecordHistorizedValues through IHistorizationRoleView — a third verdict on the same RedundancyRoleView object, registered unconditionally on driver nodes rather than alongside the alarm section's IRedundancyRoleView (which only exists when that section is enabled; piggybacking is the #511 registration accident). Two things are deliberate and easy to get backwards: * It gates the CAPTURE, not the drain — the opposite of the alarm sink. That queue is in LocalDb and replicates, so the Secondary already holds the Primary's rows. The historization outbox is a node-local FasterLog replicating to nobody, so a drain-only gate would leave the standby appending forever to a queue no one empties: unbounded disk, then a flood of stale duplicates the moment it is promoted. Entries a node already captured are still drained by that node. * The peer it stands down for must RECORD THE SAME REFS, not merely hold the same rows — a Primary in this node's own per-cluster mesh. A node carrying no cluster-scoped role cannot establish that and never stands down: under a fleet-wide election the elected Primary is usually running a different cluster's configuration, and deferring to it would end this cluster's historization silently, with the drain reporting success on an empty outbox forever. That is the scope mismatch that once suspended the alarm drain on every node of the rig. An unknown role captures, matching the alarm drain — a duplicate sample is recoverable, a hole in the trend is not. #538 — the drain writes every per-tag group and then acks all-or-nothing (deliberate: RemoveAsync truncates THROUGH an id, so partial acking could drop older unwritten entries). One permanently failing tag therefore pinned the FIFO head while every healthy tag in the batch was re-submitted to the historian on every pass, indefinitely — duplicate samples and an outbox that never truncated, observed live at 8.3 MB from a single unprovisionable tag. The drain now tracks consecutive failures per historian tag and abandons that tag's entries once it exceeds MaxWriteAttempts (new, default 5), so the head advances. The counter only advances on a pass where some OTHER tag wrote successfully: that is what separates "this tag is poison" from "the backend is down". A count-only quarantine looks identical on the poison-tag case and silently discards the whole backlog during an ordinary gateway restart, turning a duplicate-samples bug into a lost-history one. Abandonment is also deferred until every failing tag is over budget, so nothing is discarded on a pass that could not have advanced the head anyway. Each abandonment logs a Warning naming the tag; the total rides RecorderStatus.AbandonedEntries. Capacity also gains a non-zero default (500,000 entries) as the disk backstop for the one case the quarantine deliberately cannot resolve — a historian refusing every write, where retaining is correct. Set 0 to restore unbounded retention. Every new test was falsified by reverting the production change it guards, and each one reddens alone: * drop the backend-reachable evidence -> only A_backend_that_fails_every_tag_abandons_nothing * never abandon -> only A_permanently_failing_tag_is_abandoned_so_healthy_tags_stop_being_rewritten * latch the capture gate at construction -> only The_capture_gate_is_re_read_per_value... * treat any Primary as a scope peer -> only the two scope-mismatch cases Full suite green. (Four OpcUaClient.Browser failures during the run were the opc-plc fixture being down on the shared Docker host; brought it back up and they pass.) Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo |
||
|
Joseph Doherty
|
e77ea4aa73 |
docs(historian): close #491 — read-back and EnsureTags proven against wonder-sql-vd03
The 2026-08-01 run closed every leg except read-back, which no reachable historian could
serve that day (the LAN instance accepts writes but serves no reads under a licence tag-count
violation; the licensed one needed the VPN). With the VPN up the run was repeated against
wonder-sql-vd03 and the remaining legs pass:
- EnsureTags provisioning: ensured=2, failed=0 — the path the LAN box could not exercise at
all (it is RemoteGrpc-only and that CAP was not listening).
- Read-back: two Modbus registers carrying DISTINCT NON-ZERO constants were historized, and
each read back as its own value (HR20 -> 8224, HR3 -> 120, quality 192). A first attempt
read back 0, which proves nothing — the same trap #536 recorded, so the proof was rebuilt
on values a crossed or fabricated answer could not fake.
Found during the run and FILED rather than fixed:
- #538 — one permanently-failing tag re-writes every healthy tag on every drain pass, forever.
DrainOnceAsync writes EVERY per-tag group and then acks all-or-nothing, so a tag whose
historian tag does not exist stops the FIFO head advancing while the healthy tags' values
are re-submitted every drain interval; the outbox never truncates (Capacity defaults to 0 =
unbounded, observed at 8.3 MB). The all-or-nothing ack is deliberate — RemoveAsync truncates
THROUGH an id, so partial acking could drop older unwritten entries — so the fix changes the
outbox contract and does not belong in this commit.
Also re-confirmed live: EnsureTags provisioning fires only for newly-ADDED historized tags, so
flipping an existing tag's historianTagname provisions nothing (Issue 5 of
2026-06-27-historian-gateway-integration-issues.md).
No production code changes — the recorder fix shipped in
|
||
|
Joseph Doherty
|
ad74473452 |
fix(historian): the recorder died on its first value — live gate for continuous historization (#491)
Ran #491's live gate against a real ZB.MOM.WW.HistorianGateway talking to a real AVEVA Historian, and it found a production-fatal defect on the first value it ever saw. ContinuousHistorizationRecorder.OnValueChangedAsync touched `Self` after awaiting IHistorizationOutbox.AppendAsync(...).ConfigureAwait(false). ConfigureAwait(false) discards Akka's ActorTaskScheduler synchronization context, so `Self` — which reads ActorBase.Context — threw NotSupportedException. The throw escaped into supervision, which RESTARTED the actor; because production seeds an EMPTY ref set (the deployed set arrives later via UpdateHistorizedRefs), the restart silently discarded every historized ref. Net effect: one value captured, then nothing, with no further error to show for it. Fixed by capturing `IActorRef self = Self` before the await — the idiom the GetStatus handler already applied to Sender. Thirteen unit tests missed it because InMemoryOutbox.AppendAsync returned ValueTask.CompletedTask: a completed ValueTask never yields, so the continuation ran inline with the context still installed. The production FasterLog outbox fsyncs per entry and does yield. The first regression test was itself vacuous — `await Task.Yield()` posts back to the CAPTURED context, so the append still completed on the actor thread and reverting the fix left the test green. The fake must complete on a POOL thread (Task.Delay(1).ConfigureAwait(false)), as a real fsync does; with that, reverting the fix reddens exactly one test. The rig could not run this gate at all: docker-dev carried ServerHistorian__* but no ContinuousHistorization__* keys, so the recorder could never spawn. Both centrals now carry them (default OFF — the recorder needs ContinuousHistorization:Enabled AND ServerHistorian:Enabled), with the outbox on the per-node named volume so it survives a container recreate. ServerHistorian__UseTls was hard-coded true and is now parameterized; a plaintext h2c gateway was unreachable without it. Gate results: outbox append, drain, WriteLiveValues accepted by a real historian, and restart convergence (capture resumes after a restart with no new deployment) all pass. The read-back hop is NOT proven — the LAN historian accepts writes but serves no reads under a "License violation: Database tag count exceeds license tag count", and the licensed corporate one needs the VPN. EnsureTags provisioning is likewise unexercised there (RemoteGrpc-only, and that CAP is not listening). #491 stays open for the read-back hop alone. Also recorded, not fixed: the recorder has NO Primary gate, unlike the alarm store-and-forward sink, so both nodes of a redundant pair send duplicate live values. Gating it would introduce a failover window with no capture — that is a design decision, not a cleanup. Evidence: docs/plans/2026-08-01-continuous-historization-live-gate.md Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo |
||
|
Joseph Doherty
|
6b3da93151 |
fix(browse): commit the browsed data type, and fix the probe classification the fixture was hiding (#536)
A browse-committed AB CIP / TwinCAT / FOCAS tag bound with its model's DEFAULT
data type, not the one the device reported. A browsed REAL committed as
`{"tagPath":"Pump_1_Speed","dataType":"DInt"}` — and those three drivers decode
the wire bytes by that field, not by the raw row's OPC UA DataType — so the tag
read four bytes of a float through an integer decoder and published a plausible
wrong number with no error raised anywhere. FOCAS was missed when #536 was filed.
The type is CARRIED, not derived. Reverse-mapping the leaf's driver-agnostic
DriverDataType looks like the smaller fix and is wrong: SInt/Int/DInt/USInt/UInt
all collapse into Int32 on the way out of the driver, so that version would bind
a 16-bit INT as a 32-bit DINT while appearing to fix the REAL. Drivers now state
their own type name on DriverAttributeInfo.NativeDataType, which reaches the
picker as AttributeInfo.AddressFields["dataType"] and the commit mapper from
there. A leaf that states nothing keeps the model default — the pre-#536
behaviour, kept deliberately as the fallback rather than a guess.
Numeric strings are refused before parsing: Enum.TryParse accepts "9" for any
enum and Enum.IsDefined then confirms it, so an unguarded parse would bind a tag
to a type nobody named. A test caught this in the first version of the guard.
--- and the other half, which the fixture was hiding ---
Test Connect on a device with NO authored tags — every AB CIP device at the
moment an operator first presses the button — reported "CIP handshake failed"
against a healthy controller. That was written down as a fixture limitation
("ab_server doesn't implement @raw_cpu_type"). Both halves of that were wrong:
* @raw_cpu_type is not a libplctag special tag at all (only @raw, @tags and
@udt/ are), so it goes out as an ordinary symbolic name that no controller
defines, and the controller's "tag does not exist" IS the successful answer.
* ab_server answered an unresolvable tag path with CIP 0x03 where a Logix
controller answers 0x04 — ErrorBadParam vs ErrorNotFound, which the probe
classifies oppositely. Fixed in vendored patch 0002.
* With the fixture then telling the truth, the real defect surfaced:
AbCipDriverProbe.IsReachableException matched the native C status spellings
("STATUS_NOT_FOUND"), but libplctag.NET throws with the MANAGED enum name
("ErrorNotFound"). Those substrings can never match, so the reachable
classification could not fire on the exception path — which is the path every
no-authored-tags probe takes. This was failing on real hardware too, not just
against the sim. It now recovers the Status and defers to IsReachableStatus,
so there is one classification rather than two that can drift.
Rejected: switching the fallback to @tags. libplctag builds a tag-listing tag
with NO network I/O, so it reports OK against any TCP listener — measured green
against a Mosquitto broker.
Verification: full solution suite green. Falsification runs confirm exactly the
intended tests go red — 9 for the type carry, 3 (the reachable statuses only) for
the probe classification. Live on docker-dev: browse-committed TestString/
Motor_Run/TestSINT now store String/Bool/SInt, alongside last session's
Pump_1_Speed still holding the wrong DInt. TestSINT is the falsifier — its row
DataType is Int32, identical to TestDINT's, so no reverse map could have produced
"SInt".
Known consequence: pre-existing browse-committed tags keep their wrong dataType;
the fix applies to new commits only.
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
|
||
|
Joseph Doherty
|
c2c3c52274 |
fix(abcip): unblock the controller-browse gate, and fix the decoder it exposed (#468)
The AbCip controller browse could never be verified end to end: libplctag's ab_server answers CIP service 0x55 (Get Instance Attribute List) on the Symbol Object class with "unsupported", so the `@tags` walk had no backend. #468's own definition of done offered a sim that serves `@tags` as an alternative to real hardware; this is that. The fixture now applies a vendored patch teaching ab_server to serve the Symbol Object listing, with partial-transfer paging. Correctness is checked against libplctag's own `list_tags_logix` rather than against our decoder — our decoder is the thing under test, so its agreement would prove nothing. The clone is also pinned to a concrete tag: the Dockerfile called `--branch release` "pinned" while tracking a branch that moves. That immediately exposed a real defect. CipSymbolObjectDecoder advanced to an even byte boundary after each symbol name, but entries are packed with no padding — libplctag walks them by sizeof(entry) + string_len, and a captured reply confirms it. Any odd-length name therefore desynchronised every entry behind it, and odd-length names are ordinary. In the live browse below, `Motor_Run` is 9 bytes and first on the wire, so the old decoder would have shown that one tag and silently dropped the other eight. Every existing multi-entry test passed throughout, because the test builder padded exactly as the decoder did: the two agreed with each other and neither agreed with a controller. The builder no longer pads, and the layout is pinned by a captured reply (CipSymbolObjectWireCaptureTests) instead of by a second statement of the assumption. Restoring the alignment step reddens exactly the three new tests and none of the 29 others. Live gate on docker-dev, all three of #468's criteria: - populated tree — all nine seeded tags render as leaves, in wire order - refresh re-captures - leaf commit — Pump_1_Speed + TestDINT saved with their CIP tag paths The gate also surfaced two things it was not asked about: browse-commit drops the discovered data type, so a browsed REAL commits as a DINT and the driver decodes it as one (filed as #536), and Test Connect fails against this fixture regardless of the device because it probes `@raw_cpu_type` (documented). TwinCAT and FOCAS still have no browsable backend; #468's AbCip half is closed, theirs is not. Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo |
||
|
Joseph Doherty
|
1a1dbc3f00 |
fix(uns): enforce the UNS segment rule on every authoring path (#535)
An area or line could be named "Packaging" or "Line 1", save cleanly, and then fail the NEXT deployment with UnsSegmentInvalid. Because a deployment snapshots the whole database, one badly-named area blocked EVERY cluster's deploy - and whoever hit that was usually not whoever typed the name. Only the equipment form carried the rule; the deploy gate held its own private copy. The rule now lives once, on Configuration.Validation.UnsSegment, and both ends call it. DraftValidator no longer restates it. Enforced in UnsTreeService, not just on the forms: - Create/Update Area, Line and Equipment reject an invalid name before touching the row. - The CSV import reports a bad name as an ERROR, not a skip - a skip means "you already have this row", which would tell the operator the equipment exists when it does not. Good rows in the same batch still insert. A form attribute could never have covered the import, and with no bUnit in this repo a rule only a form enforces is also a rule no test can reach. The forms keep an attribute for the immediate message, but it now references the shared pattern rather than restating it - a second copy is how these drifted apart. One behaviour change beyond the fix: UnsSegment.AttributePattern admits "_default", which the equipment form's hand-written literal rejected. The form was stricter than the gate it was mirroring, so a _default row was uneditable through the AdminUI. Tests: 41, falsified with three production mutations - dropping the CreateArea guard reds exactly the CreateArea rejections, turning the import error back into a skip reds only the import test, and narrowing AttributePattern reds only the attribute/IsValid agreement. Every rejection is paired with a name that must still be accepted, or a guard that refused everything would pass. Live-verified on docker-dev: the area modal now refuses "Packaging" in place with the rule spelled out, "packaging-b" saves, and a CSV import of two rows reports Inserted 1 / Skipped 0 / Errors 1 naming the offending row - the path no form attribute reaches. Rig cleaned up and a further deployment sealed. Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo |
||
|
Joseph Doherty
|
9303289d05 |
docs(plan): correct the self-form watchdog record — rejected, TCP guard was interim
Family-wide docs correction (paired with scadaproj 6d222ea4). This plan's superseded
banner was right but its task body still read as if the watchdog were merely patched.
- Dated 2026-08-01 correction section (history preserved): the timer calling
Cluster.Join(SelfAddress) was implemented and REJECTED in both repos after live
testing — outside Akka's join handshake, it fires mid-join on a routine standby
restart (peer alive, join stalled behind stale-incarnation removal) and forms a
permanent second cluster at the same address. ScadaBridge measured a split unhealed
after 90s; Task 8 hit the same shape live here.
- The 'a booting node only self-forms when NO seed answers InitJoin' claim in the Task 2
doc comment is marked FALSE for the watchdog: that guarantee belongs to Akka's native
FirstSeedNodeProcess, which runs inside the handshake; the timer does not, and fires
while a seed is answering.
- Task 8's TCP reachability guard is recorded as an INTERIM patch, not the fix — the
race survived it, which is why the watchdog was deleted rather than patched.
- Shipped design restated: self-first seed ordering (AkkaClusterOptionsValidator +
SelfFirstSeedBootstrapTests) plus the opt-in, default-OFF bootstrap guard (
|
||
|
Joseph Doherty
|
a88cefd5ee |
docs(security): close the stage-7 gate's UNS gap — legs 12-15
The first pass could not exercise UNS-realm scope selection because SITE-A had no UNS. Authored one on docker-dev (area -> line -> equipment) and re-ran that part: the picker's UNS column offers UnsArea/UnsLine/Equipment, selecting the equipment resolves, and the grant deploys Sealed. Leg 15 is the one worth keeping. Re-pointing the SITE-A grant at MAIN's equipment is the only rejection whose id is REAL, so a check that asked only "does a row with this id exist?" would accept it and confer authority across a cluster boundary. The list page and the deploy gate both catch it and both name the owning cluster. Also found and filed (Gitea #535, not fixed here): the /uns area and line modals do not validate the UNS segment format, so "Packaging" and "Line 1" saved happily and then failed the deploy with UnsSegmentInvalid. Only the equipment form carries the rule. A deployment snapshots the whole database, so one bad area name blocks every cluster's deploy - the same authoring-vs-gate gap stage 7 just closed for NodeAcl rows. Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo |
||
|
Joseph Doherty
|
d542ab5b7c |
feat(security): author ACL grants from the real tree, not free text (#520 stage 7)
The last stage of #520. Enforcement landed in stages 5-6; this closes the authoring end, where a grant could be saved, deployed and sealed green while resolving to nothing. Scope kind and scope target are now chosen TOGETHER from a picker over the cluster's raw and UNS trees (AclScopePickerModal -> AclScopeTree, backed by IAclScopeService). They were previously an unrelated dropdown and a free-text box, so "RawDevice + a TagId" was a well-formed row. Because the trees are rooted at the grant's own cluster and each node supplies its own kind, NodeAclScopeCrossCluster, NodeAclScopeIdUnexpected, NodeAclScopeIdMissing and NodeAclScopeKindRetired all become unauthorable by construction. CheckScopeAsync mirrors DraftValidator.ValidateNodeAclScopes rule for rule and blocks the save when a scope resolves to nothing - re-checked on submit, not trusted from form load, because a row whose entity was deleted since is exactly where the operator never touches the picker. The list page marks such rows unresolved with the reason. The principal field suggests known principals and reports how many live keys reach the one typed; reaching nobody warns rather than blocks, since authoring ahead of minting is legitimate. Deliberate departures from the plan line, all recorded in the design doc: - The banners are reshaped, not removed. "Not enforced" stopped being true at stage 6, but enforcement is still a per-node flag the AdminUI cannot see, so a narrow grant really does confer nothing on a default-configured node. - A Cluster row produces a NULL ScopeId. The deploy gate rejects one that carries an id, so filling in the cluster id would author undeployable rows. - AclScopeTree is a new component rather than a third mode on RawTree, which already carries the /raw editing tree and the equipment reference picker. - UNS tags are not offered: a UNS signal is a UnsTagReference, and a Tag grant already covers the raw node and every UNS projection of it. Also fixed: the ACL editor had a DataAnnotationsValidator and no ValidationSummary, so an invalid NodeAclId made Create a silent no-op. Tests: 30 in AdminUI.Tests/Acls, falsified with six production mutations - each turned exactly the intended tests red and no others. Live gate PASSED on docker-dev, 11 legs; both central containers asserted on the same image by container id. The pair that matters is legs 8 and 11: the page and the deploy gate reject the same row with the same reason. Legs 5 and 9 are the positive controls - without them the gate passes against a page that refuses everything. Not covered: UNS-realm scope selection (SITE-A has no UNS areas). Rig restored to zero ACL rows. Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo |
||
|
Joseph Doherty
|
c4f537a60c |
docs(security): record the stage 6 live gate — PASSED on docker-dev
site-a-1 (driver-only, no ConfigDb) with Security:Acl:Enabled=true; central-1
on the same artifact as the A/B control. 11 legs.
The two that only a live rig could settle, as a transcript:
13:49:39 SiteAOnly = 53 (0x00000000) streaming ~1/s
13:49:40 SiteAOnly = (0x801F0000) revoking deploy lands; client TOLD
… 25 seconds of silence, ~25 updates that never arrived …
13:50:05 SiteAOnly = 157 (0x00000000) restoring deploy; feed resumes
53 -> 157 across the gap: the device counter ran throughout, so delivery
stopped, not the source. The 0x801F0000 frame is exactly what the
MonitoringMode.Disabled implementation would have swallowed.
Also recorded: the leg that distinguishes working enforcement from a trie that
denies everything is resolvedScopes=1 in the install log, not any denial.
Stated plainly as NOT covered: a UNS-realm grant (the rig's SITE-A has no UNS
equipment) and alarm-command scoping (no alarms there). Both covered offline,
neither seen live.
Two rig facts worth keeping: :9200 round-robins central-1/central-2, and a
stale central-2 rejected the deploy outright because its image predated the
stage-3 scope kinds — image parity must be asserted per container, not assumed
from one build. And SITEA-modbus had an empty DriverConfig, so it ran as a stub
serving nothing; until that was fixed every read returned
BadWaitingForInitialData and no allow leg could be told from a denial.
Rig left as found.
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
|
||
|
Joseph Doherty
|
00aa486677 |
feat(security): enforce node ACLs on read, browse and subscribe (#520 stage 6)
Completes data-plane enforcement behind Security:Acl:Enabled. Read (every attribute, not just Value), Browse, TranslateBrowsePath, CreateMonitoredItems, SubscribeToEvents and SetMonitoringMode now consult scoped NodeAcl grants. With the flag off nothing changes. Four things went beyond the stage as written, each because leaving it out would have made the feature look like it worked: - Browse needed ancestor visibility. The trie inherits downward, and a client reaches a deep node by browsing the nodes above it, so without this every grant narrower than the cluster was authored, shipped and unreachable. It is a separate non-inheriting marker, NOT a synthesized Browse grant on each ancestor: the additive downward walk would carry such a grant from Plant onto Plant/Siemens/dev9 and every other sibling subtree. Pinned by a test that goes through the evaluator — probing the trie helper directly passes against the broken design, because a sibling is absent from the trie either way. - Revocation had to reach LIVE subscriptions. Every other operation re-asks per request; a subscription is authorized once and then streams, and streaming is how machine consumers actually read this server — so stopping only the re-asking operations would have made revocation close to cosmetic. IDataPlaneAuthoritySource gained a Changed event. A denied item is queued BadUserAccessDenied and then DETACHED from its MonitoredNode2 fan-out, not set MonitoringMode.Disabled. Disabling discards the item's queued notifications, so the client is silenced without ever being told why. The over-the-wire test caught this; the unit-level path reports the item as stood down either way. Detaching is also reversible, so a restored grant resumes the feed with no client action. - The decision cache holds the grant WALK, not the decision. Grants and the trie are fixed for a generation; a key's ExpiresUtc passes with the clock, so caching the whole decision would let an expired key keep working until the next deploy — the stage-5 open-session hole in a subtler form. - TranslateBrowsePath is gated, without which browse filtering is decorative: a client refused a browse asks TranslateBrowsePathsToNodeIds for the same path. Found en route: UserAuthorizationState's staleness ceiling couples two clocks. MembershipResolvedUtc is stamped from DataPlaneAuthorizer's TimeProvider while IsStale is evaluated against TriePermissionEvaluator's; they are the same instance in production and diverge the moment anything injects a test clock, at which point the evaluator denies EVERYTHING with a verdict indistinguishable from "no grant matched". Stage 5's claim that these predicates cannot fire was accidentally true; they are now disabled explicitly. Also fixed: ContinuationPoint.NodeToBrowse is the manager HANDLE, not a NodeState. The declared type is object, so the wrong cast compiled cleanly and silently yielded null — skipping the browse source gate entirely. Falsified: disabling the transit marking reddens exactly the three tests that assert transit works; implementing transit as an ancestor grant reddens the sibling-leak test; skipping the fan-out detach reddens the revocation test. Full solution suite green. The stage 8 live gate has NOT run yet. Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo |
||
|
Joseph Doherty
|
f692f6afe4 |
feat(security): enforce node ACLs on write, alarms and HistoryRead (#520 stage 5)
`Security:Acl:Enabled` (default false) makes scoped `NodeAcl` grants decide data-plane authorization at the three seams that have one, and gives HistoryRead a check it never had. Read, Browse and Subscribe stay ungated — stage 6. The ACL decision REPLACES the server-wide role check rather than chaining with it. Chained, a grant would have to confer the role as well as pass the scope check, and only Cluster-scoped grants confer a role — so every narrow grant would have stayed inert and the flag would have looked inert too. The load-bearing test is an ALLOW: a session carrying no roles at all writing successfully on the strength of its grant. A deny-only test passes against a build where the ACL is never consulted, because the role gate denies too. Found en route: the principal never reached the session. The authenticator passed it as OpcUaUserAuthResult.DisplayName, but RoleCarryingUserIdentity derives DisplayName from the presented token — for a key-bearing client that is the static username `apikey`, identical fleet-wide. So the trie's matching key was dropped at the identity boundary, and the audit ClientUserId on a rejected write recorded `apikey` for every consuming system. Principal and ApiKeyId are now carried explicitly. Beyond the stage as sketched: - Revocation reaches an OPEN session. Every decision re-checks the authenticating key against the current snapshot; without it a key revoked by a deployment kept its authority until the client chose to reconnect, which for a machine-to-machine consumer can be indefinitely. - UserAuthorizationState is built per CALL, not per session. The per-session shape exists so an LDAP session can re-resolve membership that changed out of band; nothing can change out of band under API keys, so the generation stamp is the whole invalidation signal and a per-call record picks it up for free. IsStale/NeedsRefresh therefore cannot fire here and are documented as vestigial. - The trie is built inside DataPlaneAuthorityState.Apply, under the same generation as the snapshot, rather than by a second wiring at the publish actor. An uninstalled trie denies everything — indistinguishable from working enforcement. Deleting the install turns 13 of the 22 new tests red. - The deploy log reports what the BUILD dropped instead of re-deriving it. The old count agreed with the builder on one of its four drop reasons; a rule with a blank ScopeId, a retired scope kind, or a path not starting at a realm vanished with the log reading zero. Both IApiKeyStore implementations widened to the full grant set. The narrowing was redundant — ApiKeyInterimRoleMapper applies the same filter itself — and became a trap once scope was enforced. Pinned by asserting the roles are unchanged, with the two grants carrying different flags so it cannot pass by accident. Falsified by mutation: enforcement never engaging, the write seam ignoring the ACL, the realm dropped from the scope, the revocation check removed, and the trie never installed each turn the intended tests red and no others. Full solution suite green. Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo |
||
|
Joseph Doherty
|
06c56d7719 |
fix(cluster,acl): OTOPCUA_ROLES fallback + one string-identity rule (#532, #533)
Two independent defects surfaced in review of #520 stage 3.
#532 — Cluster:Roles had no OTOPCUA_ROLES fallback, despite two doc sites
promising one. A node configured only through the env var — which drives ALL of
Program.cs's conditional wiring, so it reads as the authoritative knob — bound an
empty Roles array and joined the cluster carrying no Akka roles at all. It ran,
served, and logged nothing wrong, while central reported "no driver-role nodes
are Up" and every deployment to it timed out. Not hypothetical: the field fix on
wonder-app-vd03 was to hand-add Cluster__Roles__0/1 to a Windows service's
registry environment. Anyone reading the code to diagnose that box would have
concluded from the XML doc that the fallback was already applied.
AddOtOpcUaCluster now post-configures AkkaClusterOptions through the shared
RoleParser.ResolveEffectiveRoles, so BuildClusterOptions (Akka's own member
roles), ClusterRoleInfo, the ControlPlane singleton scope and
ConfigDbApiKeyStore's ClusterId all see one answer. Two sibling validators
(ConfigSource, Telemetry) read Cluster:Roles directly and were exempting the
exact node they exist for — an env-only driver-only node has no ConfigDb and
could sit on ConfigSource:Mode=Direct unchallenged. They resolve the same way now.
Also collapsed the four open-coded `Roles.FirstOrDefault(IsClusterRole)` +
ClusterIdFromRole copies into RoleParser.ClusterIdFromRoles. They had drifted:
three classified the cluster role ordinally, so a verbatim-bound `Cluster-SITE-A`
yielded a null ClusterId — and a null ClusterId applies NO cluster filter, which
made #528's cross-cluster over-grant guard fail OPEN. IsClusterRole is now
case-insensitive; the ClusterId suffix keeps its authored spelling, because
normalizing it would silently rename the Akka role of a running deployment.
#533 — three string-identity rules in one authorization path. AclScopeRef used
default (ordinal) equality while the trie builder filtered ClusterId and hashed
principals OrdinalIgnoreCase; the deploy gate's cross-cluster check was ordinal
too, which was the worse half — it could reject a whole deploy over a ClusterId
case difference the runtime would have accepted. It only worked because resolver
and builder were handed the same NodeAcl instances.
The rule is now stated once, on AclScopeRef, and applied throughout:
identifiers (ScopeId, ClusterId, Principal) compare OrdinalIgnoreCase; path
segments compare Ordinal. AclScopePathResolver re-keys its own maps rather than
documenting a comparer contract for callers, so the two independent producers —
EF at the gate, artifact JSON on the node — cannot diverge from the outside.
PermissionTrieNode.Children stays Ordinal: those are browse names, and Speed /
speed are two nodes that can coexist under one device.
Split PermissionTrieBuildDiagnosticReason.RetiredScopeKind out of
UnresolvedScope. A legacy FolderSegment row and a typo'd id had the same message
and different remedies.
Falsified: five mutations, each reddening only its own guard — remove the
PostConfigure (3 tests), revert IsClusterRole to ordinal (3), revert AclScopeRef
equality (1), revert the resolver's map comparer (6), revert the gate's
comparisons (2), remove the FolderSegment short-circuit (1).
Test isolation: OTOPCUA_ROLES is process-global and xUnit runs classes in
parallel, so the three validator suites that now resolve effective roles share a
non-parallel collection. Without it, cases passing no roles would intermittently
observe another class's env var.
Full solution build clean; full suite green apart from two pre-existing failures
confirmed identical on
|
||
|
Joseph Doherty
|
3a6b2a1daf |
feat(acl): variable-depth scope model + the scopePaths producer (#520 stage 3)
Makes the /raw device tree addressable by an ACL rule, and builds the
scope-path producer PermissionTrieBuilder has accepted since it was written
but which had no caller anywhere.
The plan for this stage said "add NodeHierarchyKind.Raw, whose path is the
RawPath segments Folder/Driver/Device/TagGroup/Tag". That shape is wrong.
RawFolder.ParentRawFolderId and TagGroup.ParentTagGroupId are both
self-referencing, so folders and tag groups each nest without bound and a real
RawPath is folder…/Driver/Device/group…/Tag. NodeScope was fixed-arity and
PermissionTrie.CollectMatches was five hardcoded steps, so the scope model
could not address a real raw tree at all. NodeScope is now {ClusterId, Kind,
Segments} and the walk is a segment loop. The trie STRUCTURE was already
variable-depth (NodeAclPath.Segments is a list, Children is a recursive
dictionary), so only the walker and the scope record had to change.
Four decisions worth keeping:
- The namespace level became the realm. Level 1 gated on NodeScope.NamespaceId,
but v3 has no Namespace entity — the snapshot was retired when the address
space went dual-namespace, and the identifier appears nowhere in production
outside NodeScope. So the level could only ever fail to match, which silently
made every sub-cluster grant unreachable while cluster grants kept working.
It is now a realm token derived from the scope's Kind, not carried on it, so
the walk and the producer cannot disagree about its spelling.
- Provenance comes from the granting row's own ScopeKind, recorded on TrieGrant
at build time. Depth-derived provenance could not name a raw level at all,
since the depth of "Device" varies with folder nesting.
- The trie comparer moved from OrdinalIgnoreCase to Ordinal. Raw sibling-name
uniqueness is ordinal, so Speed and speed can both exist under one device and
a grant on either would have matched both.
- The null-scopePaths "deterministic test mode" is deleted. It planted
unresolvable rows at a direct child of the trie root, a position the walker
cannot reach, so the grant looked placed and could never match — and every
test using that mode proved nothing about production placement. scopePaths is
now required, and a row that does not resolve is dropped with a diagnostic.
A scope resolves to a LIST of paths: a raw tag projected into N equipments is
reachable at N+1 NodeIds and a write through any of them lands on the same
driver reference, so a Tag-scoped grant covering only the raw node would be
bypassable through a UNS one. The lookup is keyed on (ScopeKind, ScopeId), not
ScopeId alone — ScopeId has no FK and no type discriminator, so an id collision
across entity types would hand a grant the wrong node's paths.
Also closed here, all found by review rather than asked for:
- NodeAcl rows had no deploy-time validation of any kind and were not reachable
from DraftValidator at all (DraftSnapshot carried no NodeAcls). They are now
gated for dangling ids, cross-cluster scopes, blank/contradictory ScopeId, the
retired FolderSegment kind, and undefined ordinals.
- ConfigDbApiKeyStore conferred roles from Cluster-scoped grants authored in ANY
cluster, because it filtered on scope kind with no ClusterId predicate while
ArtifactApiKeyStore read an already-cluster-filtered snapshot. On a fused
admin+driver node that was a cross-cluster over-grant. Its doc comment claimed
the two stores matched.
- The two AdminUI ACL banners still said enforcement was LDAP-group -> role
mapping, which stages 1/2 made false in both directions.
- The /apikeys page was never classified in PageAuthorizationGuardTests, so that
test has been red since the key-minting commit merged — I did not run
AdminUI.Tests before merging stage 4.
Falsified rather than assumed: 8 mutations against the trie tests, 5 against the
deploy gate (including one proving a cross-cluster row is not silently
mis-diagnosed as unresolved), a staged red->green for the over-grant regression,
and the realm gate, the comparer and the malformed-path guard each broken
individually to confirm only their own tests go red.
Node ACLs are still not ENFORCED — nothing constructs a NodeScope or calls the
evaluator yet. That is stages 5 and 6; this stage makes the model able to
express what those will enforce.
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
|
||
|
Joseph Doherty
|
a438ac1640 |
feat(acl): keys + ACLs reach driver-only nodes via the artifact (#520 stage 4)
A driver-only node has had no ConfigDb since per-cluster mesh Phase 4, so it registered NullOpcUaUserAuthenticator and denied every UserName token. Its keys now ride the deployment artifact instead. - ConfigComposer snapshots ApiKeys alongside the NodeAcls it already carried. Only the peppered HMAC ships, never a secret, and the pepper is node config that is deliberately not in the artifact. - DeploymentArtifact.ParseDataPlaneAuthority decodes both node-side. Grants are scoped to the node's own cluster; keys are not, because ApiKey carries no ClusterId — a key is a fleet-wide credential and the ACL rows are what bind it. - OpcUaPublishActor installs them on DataPlaneAuthorityState on every apply, BEFORE its plan.IsEmpty early-return: an artifact that only revokes a key produces an empty address-space plan, so feeding after that return would leave the revoked key working until an unrelated structural change redeployed. - ArtifactApiKeyStore serves them to session activation; a node with a ConfigDb keeps ConfigDbApiKeyStore, whose per-request read is what makes revocation immediate there. - OtOpcUaNodeManager gains DataPlaneAuthority — the cluster identity blocker #4 named. NodeScope.ClusterId is required and the evaluator fails closed on a mismatch, so until now no scope it built could have evaluated. Nothing reads it until stages 5/6. The generation is a node-local COUNTER driven by the RevisionHash, not derived from its bits. PermissionTrieCache treats the highest GenerationId as current, so a generation taken from hash bytes would install a lower number about half the time and the cache would go on serving the previous deployment's ACLs, silently. Sessions are not stamped with it yet — that belongs with the first seam that reads a UserAuthorizationState, in stage 5, rather than being built now and consulted by nothing. Live-gated on docker-dev against site-a-1 (driver-only, no ConfigDb): key authenticates, wrong secret denied, both halves of the revocation asymmetry observed, and a SITE-A grant confirmed not to reach site-b-1. Key minted with an independent Python HMAC so the verifier was checked against something other than itself. Rig left clean. Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo |
||
|
Joseph Doherty
|
dd939d3c97 |
feat(acl): anonymous becomes a normal grantable principal (#520 D3)
Anonymous sessions used to short-circuit in HandleImpersonation — "let the SDK's
default validation stand" — so they always connected and always carried no roles.
Now admission and authority are separated:
admission -> OpcUa:AllowAnonymous, the HOST's decision (it owns token policies)
authority -> the authenticator, resolving the reserved principal "anonymous"
through the SAME grant path a key's principal goes through
So anonymous can be granted WRITE, not merely tolerated as read-only, and it
arrives at the downstream gates as the same RoleCarryingUserIdentity a key
produces — which is the real payoff: no gate carries an anonymous-shaped branch,
so there is no bypass path to get wrong.
AllowAnonymous defaults to TRUE on purpose. Defaulting it false would silently
deny every client without a key the moment this deploys, including ScadaBridge's
Data Connection Layer in another repo. Making keys mandatory is a rollout
decision, not an upgrade side effect. When false, the Anonymous token policy is
withheld so a client is refused at DISCOVERY rather than only at activation, and
activation is rejected independently in case a client attempts a token type the
endpoint never advertised.
Two fail-closed choices, both falsified:
- A failure to read anonymous's grants DENIES rather than admitting with no
roles. Admitting is the pre-#520 behaviour, and defaulting to it exactly when
the server cannot determine what anonymous may do stops being harmless once
reads are gated. Making it admit reds Anonymous_denies_when_grants_cannot_be_read.
- A key may not authenticate as the reserved "anonymous" principal, or a
credentialled client silently inherits whatever anonymous was granted.
AuthenticateAnonymousAsync is a DEFAULT interface implementation returning
admit-with-no-roles, i.e. exactly the pre-#520 behaviour, so
NullOpcUaUserAuthenticator and every test double keep working rather than being
forced to reimplement it — important because driver-only nodes run the Null one
and would otherwise black out. Note a default interface method is only reachable
through the interface-typed reference, which is how OpcUaApplicationHost holds it.
An existing test asserted the old fall-through and correctly went red; it is
rewritten to pin the new contract rather than deleted, with the reason recorded.
OpcUaServer 366, Host 42 green.
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
|
||
|
Joseph Doherty
|
65fda7237f |
refactor(acl): delete the LDAP data-plane authenticator and its tests (#520)
Stage 2b. LdapOpcUaUserAuthenticator was left registered nowhere by stage 2a; this removes it, its three test files, and the config surface that existed only to serve it. Deleted: LdapOpcUaUserAuthenticator, LdapOpcUaUserAuthenticatorTests, LdapAuthResilienceTests, LdapGlauthOutageLiveTests (19 tests). Also removed four Security:Ldap keys — AuthTimeoutMs, MaxConcurrentAuthentications, OutageFailureThreshold, OutageCooldownSeconds — plus their validator rules and binding/validator test coverage. Verified first that the ONLY consumers were the deleted class and the validator rules asserting them; none appear in any checked-in appsettings. Leaving them would have created exactly the dead-config surface #521/#524/#525 were filed to clear up. Worth stating why that resilience machinery does not need relocating rather than just deleting. It existed because the OPC UA SDK raises impersonation SYNCHRONOUSLY under a SessionManager-wide event lock, so a hung LDAP bind there serialized every session activation server-wide — Anonymous and X509 included. Key verification is one indexed lookup plus one HMAC with no network I/O, so the hazard the wall-clock bound, concurrency cap and outage circuit guarded against no longer exists on that path. LDAP now runs only on AdminUI login, an ordinary async request path where a slow directory affects the one request making it. ConnectionTimeoutMs is RETAINED: it is projected into the shared library and is not authenticator-specific. Docs corrected rather than left to rot: - CLAUDE.md's "LDAP Authentication" section described the deleted file as the data-plane authenticator. Rewritten, including a warning not to "restore" it on seeing the LDAP wiring that AdminUI legitimately still needs. - infra/README §5b (the GLAuth outage live gate) marked RETIRED with the reason, rather than silently dropped — the harness GLAuth is still used by OTOPCUA_HARNESS_USE_LDAP=1 and is unaffected. - docs/security.md's five-key resilience table cut to the one surviving key, and a new API-key identity section added. Historical archreview/ records are left as-is; they are dated accounts. Host.IntegrationTests 211 tests (was 230, i.e. exactly the 19 removed) with the same 3 pre-existing failures verified against clean master. Host 37, Security 89. Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo |
||
|
Joseph Doherty
|
f88eb49bd1 |
feat(acl): ApiKey entity + key format, and put Core on the server's graph (#520)
Stage 1b. Still no enforcement wired — this adds the credential model the identity cutover (stage 2) will read. ApiKey holds one principal per key, deliberately. The reason to separate keys from principals is so many keys share a grant set and rotating a key never touches an ACL row, and that holds with one principal per key since nothing stops two keys naming the same principal. A key belonging to several principals at once buys little and costs a join table. UserAuthorizationState.Principals stays a list, so adding it later needs no ACL-side change. ApiKeySecret is the security-critical half and is pure, so it is tested on its own: otua_<keyId>_<secret>, peppered HMAC-SHA256, FixedTimeEquals. Notes why a plain HMAC rather than an iterated KDF (the secret is 32 bytes of CSPRNG, and the call runs inline on session activation, which the SDK serializes server-wide behind its SessionManager event lock), and fails closed on an unset pepper rather than hashing under an empty key. Found and fixed a bug in my own first draft: the key id was url-safe base64, whose alphabet maps '/' to '_' — the very character TryParse splits on. Roughly one key in five would have had its id truncated at a random position, an intermittent auth failure reproducing only for SOME keys. The id is hex now, and the guard is falsified: reverting the alphabet reds both The_generated_key_id_never_contains_the_separator and, as a bonus signal, Two_generated_keys_never_collide, because truncated ids collide. ApiKeyId is NOT NULL, unlike the nullable NodeAclId precedent whose filtered unique index would admit two NULL ids. Fine for a config row, not for a credentials lookup key. OpcUaServer.csproj now references Core. Until now it did not, so IPermissionEvaluator was literally uncallable from the node manager — the place enforcement has to happen. No new transitive package surface. Schema doc updated for both, and its NodeAcl block flagged for pre-existing drift (it still documents the GenerationId column v3 dropped) without fixing it here. Core 247, OpcUaServer 364, Configuration 128 — all green. Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo |
||
|
Joseph Doherty
|
804c32e5c0 |
docs(acl): design for API-key identity + node ACL enforcement (#520)
Owner decisions 2026-07-31: the OPC UA data plane is machine-to-machine, so identity is an API key with a static username rather than an LDAP user; LDAP is replaced entirely on the data plane (AdminUI login keeps its own); and Anonymous becomes a normal grantable principal behind an enable setting rather than a special case in the evaluator. Re-verified all six of #520's claims on master first — they still hold. The identity decision dissolves blocker #2 (session carries roles, not groups) rather than solving it. Blocker #1 (raw-realm nodes have no representable NodeScope) turns out to be tractable: NodeAclScopeKind already carries Namespace and FolderSegment, so adding NodeHierarchyKind.Raw over the RawPath segments makes FolderSegment matchable for the first time. Seam survey corrects #520 in both directions: HistoryRead already has four overrides to hang a check on, but Read/Browse/Subscribe genuinely have none and sit on the hot publish path, so decisions must be cached per session+node. Records two things the issue did not: enforcement ships behind a default-off Security:Acl:Enabled dark switch (otherwise the first deploy silently denies ScadaBridge, which consumes this address space from another repo), and because Phase 4 leaves driver-only nodes with no ConfigDb, keys ride the deployment artifact — so key revocation requires a deploy. Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo |
||
|
Joseph Doherty
|
be3152e184 |
docs(raw): record that a raw rename rebinds live — #489 closed as fixed-by-#516
Re-ran #489's own repro recipe on docker-dev. It does not reproduce: renaming pymodbus/plc/HR200X -> HR200Z and deploying (no node restart, container at 0 restarts) yields the new raw NodeId serving live ~1 Hz data, the old NodeId gone with BadNodeIdUnknown, and the UNS projection still Good. The mechanism is NOT the ApplyDelta/ReinitializeAsync chain the issue's 2026-07-26 code-trace predicted. The node log shows a stop-and-respawn: DriverHost: stopped driver child MAIN-modbus DriverHost: spawned Modbus driver MAIN-modbus (stub=False) DriverInstance MAIN-modbus: subscribed to 2 refs (poll-sub-1) #516 moved a DriverConfig change out of toDelta and into toStop+toSpawn so the factory is the single parse authority. A rename changes the merged DriverConfig, so the child is rebuilt against the new RawPaths and re-subscribes on the same deploy. #489 was fixed as a side effect and needs no code change. Also gated the sibling case the issue did not cover: renaming the *device* (the middle RawPath segment, plc -> plcz) rebinds identically. Records the operator-facing fact in docs/Raw.md, including the flip side #516 accepts — a rename reconnects the driver, briefly dropping every tag on it. Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo |
||
|
Joseph Doherty
|
5a7a08deca |
fix(drivers): detect mid-run comms loss and recover from it (#527)
DriverInstanceActor.DisconnectObserved had no production sender. The only paths into Reconnecting were a failed *initial* connect and an operator's Force Reconnect, so a driver that connected and then lost its device stayed Connected indefinitely: publishing Bad for every tag, never re-initialising, and NOT recovering when the device returned. PublishHealthSnapshot read GetHealth() for reporting only and never acted on a driver reporting itself unhealthy. Both comms-loss quality paths -- #477's native-condition annotation and #481's scripted-alarm input fan-out -- hang off ConnectivityChanged, so both were correctly wired to a signal that never fired. The Connected health tick now evaluates a watchdog and self-sends DisconnectObserved when the driver reports a non-Healthy state AND its LastSuccessfulRead has been frozen for DefaultCommsLossTimeout (90s). The conjunction is the design, and the issue's own proposal -- "non-Healthy for N consecutive polls" -- is wrong. Degraded means *connected but some data is bad*; several drivers write health per tag inside the read loop (ModbusDriver.ReadAsync), so one unreadable address latches Degraded forever against a perfectly reachable device, and a state-only watchdog would reconnect that driver every 90s indefinitely -- worse than the bug. LastSuccessfulRead is the honest evidence: preserved across Degraded/Faulted writes, advanced by any success. The state check remains the second conjunct so an idle Healthy driver with nothing subscribed is not declared dead for not reading. Verified across all 12 drivers that a sustained failure does reach a non-Healthy state -- nine poll drivers via Degraded, MQTT via live connection state, Galaxy via its reconnect supervisor. CalculationDriver reports Healthy while running and so can never trip, which is correct: it has no transport. Also: the Reconnecting retry now calls ShutdownAsync before InitializeAsync. ModbusDriver (_transport) and MqttDriver (_connection) overwrite an IAsyncDisposable field inside InitializeAsync without disposing the previous one, so re-initialising without a teardown leaks a socket per attempt -- latent while only a manual Force Reconnect reached this path, but the watchdog makes it automatic and unbounded (~360 leaks for an hour-long outage at the 10s retry). Tests: 8 new in DriverInstanceActorCommsLossWatchdogTests, each absence assertion carrying a positive control (a proven count of processed health-poll ticks). Falsified five ways -- dropping either conjunct reds exactly the test written for it, and removing the teardown, the null baseline, or the watchdog call each reds its own guard. Live gate PASSED on docker-dev (Modbus + otopcua-pymodbus-standard), run twice: - fixture stopped -> watchdog fires at ~108s, "declaring comms lost" - fan-out observed at Debug: "pushed BadNoCommunication for 2 mux ref(s)" (#481 Layer 4) and the #477 native-condition annotation on a driver owning conditions - fixture restarted -> reconnected in ~8s with NO node restart; both the raw (ns=2) and UNS (ns=3) NodeIds read Good with fresh timestamps Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo |
||
|
Joseph Doherty
|
6cd91cc30a |
docs(alarms): record the #481 live-gate finding — the comms-loss signal never fires (#527)
The docker-dev gate for #481 could not close on its headline scenario, and the reason is worth recording next to the feature rather than only on the tracker: DriverInstanceActor.DisconnectObserved has no production sender, so a driver whose device dies mid-run is never detected. Filed as #527 with the live evidence (Modbus stayed Connected >10 min publishing Bad, and did not recover when its fixture returned until the node was restarted). What the gate DID establish: the change deploys and runs (scripted alarm materialised on central-1 against the seeded Modbus tag), and the #478 Layer 3 path does carry Bad into scripted input quality for Modbus, because that driver publishes Bad per tag rather than going silent. Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo |
||
|
Joseph Doherty
|
c81cef39f5 |
feat(alarms): bridge driver comms-loss into scripted-alarm input quality (#481)
Layer 4 of the condition-Quality chain (#477 -> #478). #478 gave a scripted condition the worst quality of its input tags, but only for inputs whose driver PUBLISHES a Bad data change. A poll driver whose device goes unreachable publishes nothing — it raises ConnectivityChanged(false) and goes silent — so the engine held its last Good value, WorstInputStatus stayed Good, and worse, AreInputsReady stayed true, so the predicate went on evaluating a stale value after the source had died. `DriverHostActor.FanCommsLossToDependencyMux` pushes a synthetic BadNoCommunication with a null value into the dependency mux for every value ref the disconnected driver owns. The issue asked for a driver -> all-tag-refs map to be built for this. It already exists: `_nodeIdByDriverRef`, keyed (DriverInstanceId, RawPath) and rebuilt from the composition on every apply. Alarm-bearing tags live in the separate `_alarmNodeIdByDriverRef` and are deliberately excluded — a native condition is not a mux value and takes the Part 9 quality path instead. Nothing is pushed on reconnect, deliberately: publishing Good would assert a value quality this actor has not observed. Recovery rides the driver's own republish, and the mechanism was verified rather than assumed — the Connected -> Reconnecting edge calls DetachSubscription, which nulls the subscription handle, so the reconnect's ResubscribeDesired builds a FRESH subscription whose PollGroupEngine initial push runs with forceRaise: true. Without that detach the poll engine would compare against its retained LastValues, see an unchanged value, raise nothing, and strand these refs Bad forever — the value-dedup trap that bit the v3 Batch 4 virtual-tag reassert. Closes #478 Finding 3, which the above depends on. WorstInputStatus skipped every input whose VALUE was null as "no data yet" — and Bad-with-no-value is exactly the shape a comms-lost driver produces, so the skip was hiding the case that matters most; without this half the fan-out above would be silently inert. The upstream source now stamps a never-pushed path BadWaitingForInitialData — the honest Part 4 code for that state — and the engine skips that specific code rather than every null value. Deleting the null-skip outright is the obvious fix and is wrong in the other direction: it makes every not-yet-published input drag its condition Bad at deploy, flooding the quality path with load-time annotations, which is what the skip existed to prevent. Both mistakes are pinned by a test pair that goes red under the opposite error, and both were falsified by reverting the production code: old null-skip restored -> A_pushed_Bad_carrying_no_value RED null-skip deleted outright -> An_input_that_was_never_pushed RED fan-out call removed -> the three DriverHostActor tests RED (1 of 7) The never-pushed test was vacuous on first writing and is worth recording: the quality bucket only emits on CHANGE, so the load-time evaluation is the one that flashes Bad, and subscribing to OnEvent after LoadAsync made the assertion unfalsifiable — it passed against the broken behaviour. It now subscribes first. `FakeUpstream.ReadTag` returned 0x80340000 (BadNotConnected) for an unset path, a different Bad code from production's placeholder. Harmless while any Bad behaved alike; load-bearing now, so it is aligned and pinned by a reflection guard asserting the engine's private constant agrees with what the production source stamps. Behaviour change to weigh, flagged in the issue and confirmed: virtual tags computed from a comms-lost input now go Bad too. That follows from using the shared mux and reads as correct, but it is broader than scripted alarms. Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo |
||
|
Joseph Doherty
|
b815446940 |
refactor(drivers): delete GetMemoryFootprint / FlushOptionalCachesAsync (#525)
Both `IDriver` members had no production call site. Their only consumer was `MemoryTracking`, deleted with the driver-tier recycle machinery in #522, and `GetMemoryFootprint`'s doc-comment claimed "polled every 30s by Core" — which had not been true since v3, even before #522 removed the poller. Owner decision on #525 was (a) delete rather than (b) build the process-wide watchdog that would consume them. The driver-side implementations are in git history and were mostly reusable; what was always missing is the trigger, and a member nobody calls is how the misleading doc-comment survived this long. Removed from `IDriver`, all 12 driver implementations, and every test stub — plus the support code that existed only to feed them, which is the part a member-only deletion would have left behind as dead weight: - MTConnect's three per-entry byte-weight constants, its cached `ProbeModelDataItemCount`, and `CountDataItems` — a full device-tree walk run on every cache fill to populate a number with no remaining reader. - OpcUaClient's `_discoveredNodeCount` field and the discovery-pass write to it. - MQTT's `ApproxBytesPerAuthoredTag`. Two behavioural notes, neither a change in what the software does: - `AbCipTemplateCache` now has no production reset path. Its only caller was AB CIP's flush, which itself had no caller — so nothing changes except how visible the fact is. Recorded on the type, along with the correction that its comment's claim about a device `ReinitializeAsync` also flushing it was never true. A UDT redefined on a live controller keeps its stale shape until the driver instance is rebuilt. - MTConnect's `/probe` cache is still dropped, by `InvalidateProbeModel()` on an Agent restart. That is now `internal` rather than `private` so the tests which used the flush purely to empty the cache drive the real path instead — they now cover a case that actually occurs in production. Several unrelated doc comments cited `FlushOptionalCachesAsync` as the canonical CAS example and are repointed to it. Tests: the ones that existed only to assert on the removed members are deleted (3 OpcUaClient, 3 Galaxy, 2 MQTT, 1 MTConnect, 1 AbCip). Worth noting one was vacuous — `FlushOptionalCachesAsync_resets_footprint_counter` asserted a footprint of 0 before the flush and 0 after with no discovery in between, so it would have passed against a flush that did nothing at all. Full suite green except 7 failures verified identical on master before and after (4 AbLegacy integration with the compose profile down, 3 Host.IntegrationTests), checked by name rather than assumed. Docs corrected where they described this as live: `DriverLifecycle.md` (which carried the #525 placeholder), `drivers/AbCip.md`, `drivers/AbLegacy.md`, and `v2/driver-stability.md` — whose "polls this every 30 s" + per-driver `DriverConfig` memory budget were never built as written and are now banner-ed as design intent rather than description. Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo |
||
|
Joseph Doherty
|
803f4e6efe |
refactor(sql): split ISqlDialect/SqlServerDialect into a .Dialect leaf (#526)
`Driver.Sql.Browser` referenced the runtime `Driver.Sql` project. The AdminUI references that browser, so the whole SQL driver runtime — and `ZB.MOM.WW.OtOpcUa.Core` through it — sat in the AdminUI's build/deploy graph. This is the same layering violation #512 removed from the MQTT browser, in a second driver, and it is why #512's expected removal of `Core.dll` from the AdminUI's output did not happen. MEASURED, both directions, on a clean AdminUI build: before: ZB.MOM.WW.OtOpcUa.Core.dll + ZB.MOM.WW.OtOpcUa.Driver.Sql.dll present after: both absent The old csproj comment's premise was sound and survives — the dialect's catalog SQL *is* the browse engine, so it must be shared rather than duplicated. What was wrong was the conclusion that sharing it required referencing the runtime driver. `ISqlDialect`/`SqlServerDialect` depend on nothing in `Driver.Sql`, so they move to a `.Dialect` leaf that both consume. This is the `.Transport` shape, not the cheap one: `ISqlDialect` exposes `DbProviderFactory` and `SqlServerDialect` returns `SqlClientFactory.Instance`, so `.Contracts` (deliberately provider-free) cannot hold them. Assembly name is `.Dialect`; the NAMESPACE stays `ZB.MOM.WW.OtOpcUa.Driver.Sql`, matching the `Driver.Mqtt.Transport` precedent — the extraction cost zero `using` changes. Two dependencies the cut edge surfaced as the implicit ones they were: - The AdminUI itself read `SqlDriver.DriverTypeName` in three places — the tag editor map, the tag config validator, and the Sql address picker. All three now read `DriverTypeNames.Sql`, which is exactly what that constant resolves to and exactly the repoint `SqlDriverBrowser`'s own comment had deferred until a factory was registered against the shared constant. One exists now. - `SqlDriverFormContractTests` asserts the AdminUI form's emitted JSON against the runtime factory that reads it, so it genuinely needs the runtime driver. `AdminUI.Tests` now references it explicitly. A test project referencing a runtime driver is fine — tests are not in the AdminUI's deploy graph. New guard: `DriverBrowserLayeringTests` (Commons.Tests) scans the csproj graph of every `*.Browser` project in the tree, transitively. It reads csproj files rather than loaded assemblies because the cost is a build-graph cost — the compiler emits no assembly reference for an unused `ProjectReference`, so `MqttBrowserLayeringTests`' assembly-reference approach cannot see that case. The two are complementary and each now points at the other. Falsified three ways, each red then green again: the direct reference re-added to the browser; the same reference one hop out via `.Dialect` (proving the walk is really transitive); and both controls — browsers-discovered and references-parsed — which exist so an empty discovery cannot pass the absence assertion vacuously. #526's claim that the reference "carries no comment at all" was wrong; it carried a detailed one from the scaffold commit. Corrected on the issue and in the design doc's project table, which had recorded the deviation as deliberate. Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo |
||
|
Joseph Doherty
|
c74ccc55b5 |
feat(mqtt): flatten Sparkplug Template/DataSet metrics to sibling tags (#515)
`SparkplugCodec` handled scalars; `DataSet`, `Template` and `PropertySet` were skipped
with a warning. They now flatten to addressable tags (owner decision, recorded on the
issue):
Template → sibling metrics, `/`-joined, recursively: Motor → Motor/Speed, Motor/Stator/Temp
DataSet → one metric per cell, {column}/{rowIndex}: Recipe → Recipe/Step/0, Recipe/Target/1
PropertySet → node metadata, never tags — it describes a metric rather than being one
Flattening was chosen over OPC UA Structures because flattened scalars are consumable by
any client with no type dictionary, and the result mirrors how the `/raw` tree works.
**The row index is always present, including on row 0.** Emitting a bare column name for
a single-row table and an indexed one otherwise would make a tag's name depend on how many
rows the plant published — a table growing from one row to two would rename every tag
already authored against it.
**A DATA metric carries the parent's alias and no name**, so member names cannot be
composed until the alias table has resolved the parent. Expansion therefore happens in
`SparkplugIngestor.PublishStructuredMembers` — after resolution, at the single funnel both
the DATA and birth-value paths already pass through — not at decode time. The alias table
installs a binding per member so browse enumerates them and a DDATA member that omits its
datatype falls back to the birth's (guessing from the CLR type would read a signed Int32 of
-42 out of its unsigned field as 4294967254).
**Skip-and-warn survives, and refusal is whole** — the explicit requirement on the issue.
A template definition, an unnamed member, an extension value, a column/type mismatch, a
short row, or a payload past MaxDepth(5)/MaxMembers(512) refuses the ENTIRE metric. A
silently half-flattened UDT looks exactly like a complete one in the picker.
The browse picker offers the members and deliberately NOT the structured parent, which has
no value and no servable datatype. The rediscovery signature likewise tracks members rather
than the parent, or a UDT gaining a member would leave the signature unmoved.
Member paths contain `/`, which #509 already solved: the slashed name travels intact as
`metricName` while the derived tag Name is coerced to a legal RawPath segment.
Live-gated 3/3 against the Mosquitto fixture with the project-owned simulator extended to
publish a nested Template and a 2x2 DataSet. Falsified at three points: dropping the member
bindings, not expanding the parent, and refusing to flatten templates each turn exactly the
expected tests red.
Also fixes a regression from the #511 commit in the same branch: `AddOtOpcUaDriverFactories`
resolved `IDriverPrimaryRoleSource` with `GetRequiredService` but only `AddOtOpcUaRuntime`
registered it, so a graph wiring driver factories without the full runtime failed at
resolution. Two Host integration suites do exactly that and caught it; both extensions now
TryAdd the same concrete `RedundancyRoleView`, so the registration is order-independent and
the fail-loud resolve is kept.
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
|
||
|
Joseph Doherty
|
7f2108dcf2 |
feat(mqtt): Sparkplug primary-host STATE publishing, Primary-gated (#511)
`ActAsPrimaryHost` was accepted by config and by the AdminUI form and did nothing but
log a warning. It now delivers the whole three-part contract or none of it: a retained
`ONLINE` on `spBv1.0/STATE/{hostId}`, a matching retained `OFFLINE` armed as the MQTT
Last Will, and an explicit `OFFLINE` on a clean stand-down.
**The claim is a dedicated connection, not a message.** A Will can only be registered in
the CONNECT packet — it cannot be added to a live session, changed, or withdrawn. So a
claim that is acquired and released at runtime cannot ride the long-lived ingest session.
`SparkplugPrimaryHost` opens its own connection (client id `{clientId}-spbhost`, forced
clean-session) when the node gains Primary and closes it when the node loses Primary; the
connection's lifetime *is* the claim, so every leg follows from its lifecycle. Ingest is
untouched by a role change — no dropped subscriptions, no lost birth cache, no data gap at
the Secondary->Primary transition.
**Primary-gated** (owner decision, recorded on the issue). Both pair nodes subscribe
independently, so an ungated claim means both claim the same host id — and the sharp
failure is not the two fighting over the retained STATE, it is that the SECONDARY's death
fires its Will and publishes OFFLINE while the Primary is healthy, stopping every edge node
in the group. Registering the Will only on whichever node is Primary at connect time fails
the other way: a node that later gains Primary would publish ONLINE with no Will behind it.
New seam: `IDriverPrimaryRoleSource` (`Core.Abstractions`) carrying the *data-plane* verdict
(`PrimaryGatePolicy.ShouldServiceAsPrimary`), not the alarm-drain verdict — an unknown role
with a driver peer present must deny. Registered unconditionally by `AddOtOpcUaRuntime`,
deliberately NOT alongside `IRedundancyRoleView`, which `AddAlarmHistorian` registers only
when that section is enabled: piggybacking would leave a pair with no alarm historian
falling back to "claim unconditionally". Polled on a 1s timer rather than evented — a
process singleton raising events at per-instance drivers outlives every driver it holds.
Accepted trade: a brief edge-visible OFFLINE window during failover. That is a genuine
state transition, not a glitch to hide.
Live-gated 5/5 against the Mosquitto fixture, including the Will leg — `AbandonForTests`
drops the socket with no DISCONNECT, because a clean disconnect makes the broker discard
the Will by protocol rule, so a test driven through `StopAsync` would observe the
stand-down OFFLINE and pass against an implementation that never armed a Will. Falsified:
removing the Will turns that leg and only that leg red.
Also corrects two stale claims found en route: the AdminUI form still said MQTT tags are
read-only (#508 shipped), and the driver doc still said nothing consumes `IRediscoverable`.
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
|
||
|
Joseph Doherty
|
72be19225d |
feat(mqtt): write-through (IWritable) — Plain publish + Sparkplug NCMD/DCMD (#508)
MqttDriver implemented no IWritable, so every MQTT node materialized read-only
regardless of what the operator authored. It now publishes.
Plain -> the tag's writeTopic; UTF-8 text (Raw/Scalar) or a bare JSON
literal (Json)
Sparkplug -> DCMD for a device metric, NCMD for an edge-node metric, carrying
the metric BY NAME with an explicit Datatype and no seq
Both at QoS 0, retain:false. Retain is deliberately not configurable — a retained
command is replayed by the broker on every future subscribe, including the
device's own reconnect, so one stale write would be re-applied forever.
writeTopic is a new, separate TagConfig key, and its ABSENCE is meaningful: the
write is refused BadNotWritable and the node is discovered ViewOnly. The issue's
"do not skip" warning drove the design — the one outcome worse than read-only
nodes is a node that looks writable, returns Good and publishes nothing. Reusing
the read topic would be worse still: it would echo back and read as a confirmed
device value. Browse-time writability is computed by the SAME encoder the write
path uses, so a node cannot advertise writability it would then refuse.
Good means published, not applied — MQTT has no application-level write ack, so
this is Galaxy's optimistic-Good posture and inherits its property: a failed
DEVICE write can never be surfaced. A failed PUBLISH is reported Bad; the
optimism covers what the device did with the command, never whether it was sent.
WriteIdempotent stays default-off.
The encoder is pure and lives in .Contracts, so both wire shapes are testable
without a broker. Unrepresentable values and types with no Sparkplug encoding are
refused rather than coerced — a Reference must not silently become a String on
the wire, and "not-a-number" must not become 0 on a live plant.
LIVE GATE PASSED against the real Mosquitto fixture on 10.100.0.35: 21/21
integration tests, including 6 new ones that observe the published message with
an INDEPENDENT broker subscriber — a fake asserting "publish was called" could
not tell the feature working from the feature publishing to the wrong topic.
The gate earned its keep: the NCMD test initially failed because NCMD is a SHARED
channel — the driver publishes its own rebirth request (Node Control/Rebirth) on
the very topic an edge-node metric write uses, and it arrived first. Correct
driver behaviour, wrong test; the watcher now matches on metric name. Documented,
because anything consuming that topic needs to know. DCMD has no contention.
Offline: 24 encoder + 9 driver tests. The four new status constants are NAMED so
StatusCodeParityTests can see them (#497/#519) — it went 258 -> 262 and validated
each against the pinned SDK.
Also corrects the Mqtt.md "Known gaps" table: prunes the six rows closed in this
round, and fixes the #507 row, which still described rediscovery as inert after
that issue was closed as superseded and its injection path deleted.
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
|
||
|
Joseph Doherty
|
2d20aeb779 |
fix(opcua,drivers): carry driver status sub-codes to clients (#519) + health-write ordering (#517)
#519 — a driver's OPC UA status code now reaches a client with its sub-code intact. DriverInstanceActor projected the driver's uint onto a 3-state Good/Uncertain/Bad enum and OtOpcUaNodeManager re-expanded Bad to a bare 0x80000000, so BadNoCommunication, BadTypeMismatch and BadNotSupported were indistinguishable on the wire and every driver's status constants (incl. the 16 #497 had just corrected) were diagnostic in logs only. Resolved by option (b): the raw uint rides AttributeValuePublished -> AttributeValueUpdate -> IOpcUaAddressSpaceSink.WriteValue, OpcUaQuality is deleted, and severity classification is a helper (OpcUaStatus.IsGood/IsUncertain/IsBad). The surface was wider than the issue costed (41 files, not "the sink and 3 implementations"): OpcUaQuality had non-driver producers with no uint to carry. Each got its natural code — BadInternalError for a failed virtual-tag script, BadNoCommunication for driver comms loss — and ScriptedAlarmHostActor's pair of mutually-inverse uint<->enum mappers is deleted, having round-tripped a code the engine already held. AddressSpaceApplier's removal writes deliberately keep the generic Bad: not losing a code the driver had is this issue; inventing one for the deploy path is a separate call. Behaviour change: WriteAlarmQuality fires its Part 9 event on a status-code change rather than a severity-bucket change. While the annotation was 3-state those were the same predicate; now a move between two Bad reasons changes what a subscriber reads. New SDK-free named constants live on Commons.OpcUa.OpcUaStatus, and StatusCodeParityTests is widened to cover them — that guard is reflection-driven, so it can only see a named constant, never an inline literal. #517 — the symptom is real but the issue named the wrong path and its fix was unsafe. ShutdownAsync cannot race: DriverInstanceActor calls it from PostStop, after the mailbox drains. (Core/Hosting/DriverHost, the one class with a concurrent GetHealth, has no production call site.) InitializeAsync can: it runs on a fire-and-forget Task.Run, and MTConnectDriver writes Healthy then starts loops that can throw into a catch which disposes the client before writing Faulted. Moving the write earlier, as suggested, inverts the terminal state — teardown drains in-flight loops that write health as they unwind, so a late loop could leave Healthy at rest with every connection closed. The fix is a bracket (conservative before, terminal after), applied to MTConnect's init-failure path only; shutdown keeps its single trailing write, and the rule plus the reachability table are documented on IDriver.ShutdownAsync. CalculationDriver.GetHealth returned a hard-coded Healthy that consulted nothing — Healthy before initialize, after shutdown and after dispose. Now derived from _running. Its test, named Health_is_always_Connected, asserted the defect and is replaced. Every new guard was falsified by breaking the production code and watching it go red. The two sub-code tests each assert two different Bad codes, since a single-code assertion could pass against a hard-coded constant under the old projection. Pre-existing failures, verified unchanged on a stashed tree and not chased: 4 in Driver.AbLegacy.IntegrationTests (slc500/micrologix/plc5 fixtures not up) and 3 in Host.IntegrationTests. Closes #517 Closes #519 Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo |
||
|
Joseph Doherty
|
cbb882293b |
refactor(drivers): delete the inert driver-tier recycle machinery (#522)
The tier system was documented, operator-authorable, and inert. Deleted rather than activated,
because its premise is gone rather than merely unused.
"Tier C" meant a driver running out-of-process behind an IDriverSupervisor that could restart its
Host without tearing down the OPC UA session. No such process exists anywhere: Galaxy reaches
MXAccess over gRPC to the external mxaccessgw sidecar (PR 7.2 retired the in-process
Galaxy.Host/Proxy/Shared projects) and FOCAS has run in-process since its managed wire client
landed 2026-04-24. Consistently, IDriverSupervisor had ZERO implementations and there was nothing
for one to implement against.
The issue understated the inertness. It says the Tier-C-only protections never engaged, implying
the Tier A/B parts did. They did not: nothing constructs MemoryTracking, MemoryRecycle or
ScheduledRecycleScheduler outside their own unit tests, so the whole Core/Stability recycle layer
was dead — meaning option (a), "pass real tiers", was never a flag flip. It would have meant
writing the wiring that never existed AND arming it.
Deleted: MemoryTracking, MemoryRecycle, ScheduledRecycleScheduler, IDriverSupervisor, the
vestigial DriverTypeRegistry (referenced only by its own tests), and the RecycleIntervalSeconds
knob — which the AdminUI let an operator author and the parser validated while it configured
nothing.
DriverTier itself SURVIVES and is load-bearing: DriverResilienceOptions.GetTierDefaults supplies
the real per-capability timeout/retry/breaker policies via DriverFactoryRegistry.GetTier and
DriverCapabilityInvokerFactory. Only the isolation-and-recycle layer above it is gone.
Deliberately NOT deleted:
- WedgeDetector came along in the same directory and is equally dead in production, but it is
tier-agnostic and is not recycle machinery — it only shares the folder. Restored rather than
swept up in a decision that was not about it.
- IDriver.GetMemoryFootprint() and FlushOptionalCachesAsync() lose their only consumer here.
Removing them touches all 12 drivers and every test stub, so they are documented as
consumerless and filed as #525 instead of buried in this diff.
Compatibility: a deployed ResilienceConfig blob still carrying "recycleIntervalSeconds" parses
cleanly (unknown keys are ignored — guarded by a new test, because a blob that suddenly failed to
parse would fall back to tier defaults and silently discard the operator's real overrides), and
the AdminUI's preserve-unknown-keys bag keeps the key rather than rewriting stored config on an
unrelated edit.
The 01/U-6 knob-inertness guard carried an explicit carve-out admitting RecycleIntervalSeconds was
dormant and out of scope; that carve-out is now gone, so the expected set is literally what the
test's name claims.
Note: Host.IntegrationTests has 2 failures (DriverProbeRegistrationTests.is_idempotent,
PrimaryGateFailoverTests) — verified pre-existing by reproducing both on clean master
|
||
|
Joseph Doherty
|
30d0697c28 |
feat(hosts): surface per-host connectivity on /hosts; drop the unwritable table (#521)
`IHostConnectivityProbe` was a dead surface: eleven drivers implement it, `GetHostStatuses()` had ZERO production call sites, and `OnHostStatusChanged` had no subscriber outside the Galaxy driver's own aggregator. Per-host connectivity was computed by every driver and read by nobody. The issue offered "build the publisher or delete it". The publisher as its entity doc described it — driver nodes upserting `DriverHostStatus` rows — is not buildable: per-cluster mesh Phase 4 gates `AddOtOpcUaConfigDb` on the `admin` role, so a driver-only node has no ConfigDb connection to write rows with. So the capability is kept and the transport changed. `DriverHealthChanged.HostStatuses` now carries the probe result to `/hosts` as a Hosts column. That channel already reached the page, already survives the mesh split via the Phase 5 gRPC telemetry stream, and already replays a last-value snapshot on re-subscribe — so per-host state re-primes after a reconnect without a durable store. Both halves of the interface finally do what they are for: the event triggers a prompt publish, the pull is the source of truth. The point of the column is the case the driver-level state chip structurally cannot express: a multi-device driver stays aggregate-Healthy while ONE of its devices is unreachable. Two traps, both pinned by tests that were falsified against the prod code: - The host digest MUST be in the publish fingerprint. On a single-host-down transition every other fingerprint component is unchanged, so the dedup would swallow exactly the publish carrying the news — the trap that already bit the rediscovery signal. Removing it turns the guard test red, verified. - null (no probe) must stay distinct from empty (probe with no hosts). proto3 cannot tell an absent repeated field from an empty one, hence the explicit `has_host_statuses` flag; collapsing them would render every probe-less driver as one whose devices are all fine. Dropped: the DriverHostStatus entity, enum, DbSet, model config and table (migration DropDriverHostStatusTable — empty on every deployment, so the scaffolder's data-loss warning is moot, and Down() recreates it exactly). Found en route, NOT fixed here: `DriverInstanceResilienceStatus` is the identical defect — no writer, no reader, only a DbSet declaration, while the live data rides the `driver-resilience-status` telemetry channel. Its doc-comment now states that rather than describing the sampler and AdminUI join that were never built. Filed as #524 rather than widening this schema change beyond what was asked. Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo |
||
|
Joseph Doherty
|
f3cd685da5 |
docs(drivers): write the two missing driver guides (Sql, Calculation)
Closes the §7.2 priority-3 item. Sql was a shipped, merged driver with NO
user-facing documentation at all -- only design and plan files, which describe
what was intended rather than what an operator needs to author one.
Sql.md covers: SQL Server as the only constructed provider (the other four
SqlProvider names are reserved, and authoring one fails rather than falling
back), the KeyValue / WideRow tag models with worked TagConfig, Query being
rejected end-to-end, the config table with real defaults read from
SqlDriverOptions, connectionStringRef never carrying credentials into an
artifact, per-source query grouping and its case-sensitive group keys, and the
value-bound-vs-identifier-validated split that makes it injection-safe.
Calculation.md covers: what a pseudo-driver is and why it occupies a driver slot,
a Calculation-vs-VirtualTags decision table (the two are easy to confuse), the
literal-only ctx.GetTag dependency extraction and the runtime-built-path trap,
the two deploy gates, the all-deps-arrived publish gate, and the Good->Bad-once
error semantics.
Two stale claims found by checking against source rather than inheriting them:
- The driver README's Calculation row still carried G-1 ("no typed config form,
RunTimeout unauthorable"), fixed on 2026-07-27 by CalculationDriverForm.
- docs/Raw.md's "one auto-created default Engine device" NEVER SHIPPED. I had
copied it into the new doc before checking; "Engine" has zero occurrences in
src/ and nothing special-cases Calculation at device creation. Corrected in
both files. The config key was also wrong in my first draft: runTimeoutMs, an
int in milliseconds, not a TimeSpan string -- and it is ignored on reinit
because the evaluator's timeout is constructor-fixed.
Full solution: 49 suites pass; the 2 failures are the known pre-existing
environment ones (AbLegacy.IntegrationTests docker fixture, Host.IntegrationTests).
OpcUaClient.IntegrationTests, which was silently failing, is now green.
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
|
||
|
Joseph Doherty
|
4624141a5d |
test+docs: retire the stale EquipmentTagsDarkBatch4 skips, guard the deploy-time
tag gate, and fix three silently-broken OpcUaClient integration tests
Continues working through deferment.md's remaining items.
SKIPPED TESTS (13 -> 0 for this reason). The skip said "dark until Batch 4";
Batch 4 shipped as v3.0 and RETIRED the equipment-tag path rather than lighting
it, so every one of them was waiting on something that had already happened and
gone the other way. Resolved individually rather than in bulk:
- REVIVED onto the raw/UNS shape (6): four DriverHostActor primary-gate cases,
the cluster-scoping rebuild, and the real-SDK dual-namespace materialisation
E2E. The behaviour never went dark, only the v2 seeding did.
- FOLDED into a live parity test (3): DeploymentArtifactRawUnsParityTests
already compared RawTagPlan record-equal and RawTagPlan carries array /
historize / alarm intent, so widening its corpus by two tags covers what
three empty placeholder suites had been promising. Those suites are deleted.
- DELETED (4): subjects genuinely retired -- equipment-namespace EquipmentTags,
and both dot-joint {{equip}}.X token suites (that resolver was deleted in
Batch 3; the slash-joint form is covered by DeploymentArtifactEquipRefParityTests).
Falsified, not assumed: removing the seeded UnsTagReference turns two revived
primary-gate tests red; re-homing the SITE-A node to MAIN turns the scoping test
red. Widening a parity corpus also needed direct value assertions -- parity alone
cannot distinguish "both seams right" from "both seams blind".
Runtime.Tests skips 13->3, OpcUaServer.Tests 4->1; all remaining are deliberate
EquipmentDeviceBindingRetired tombstones.
G-7 (deploy-time TagConfig gate). MQTT shipped an Inspect() nobody ever called;
now wired. Replaced the hand-maintained list with a reflection guard, because the
existing test enumerates driver types by hand and so guards renames but can never
notice a gap. NOTE: the first version of that guard was hollow -- it discovered
candidates via GetReferencedAssemblies(), which is circular, since the compiler
elides a reference nothing in the IL uses. Removing MQTT from the map also removed
its Contracts assembly from the manifest, and the guard passed green with the bug
reintroduced. Rewritten to scan the output directory; re-falsified and it now
fails naming MqttTagDefinitionFactory. Residual recorded at the code: Sql,
MTConnect, Calculation and OpcUaClient have no Contracts-side Inspect at all, so
the deploy gate stays weaker than the AdminUI's authoring gate for them.
OPCUACLIENT INTEGRATION TESTS (3 of 4 failing, on master too). Not fixture rot:
the simulator was healthy and Client.CLI read the very same node Good. v3 made the
driver resolve every reference through its authored RawTags table, so a bare
"ns=3;s=StepUp" fails LOCALLY at the resolver with BadNodeIdInvalid and never
reaches the wire -- the tests were exercising the unresolved-reference guard.
Fixed by seeding OpcPlcProfile.RawTags in the deploy artifact's TagConfig shape
and addressing by RawPath. 4/4 pass.
DOCS. Applied the Phase7* -> AddressSpace* rename across the four live docs
(historical bannered files deliberately untouched), resolving the three that are
not renames to their real members. Found en route that the earlier banner pass
missed three files: VirtualTags.md, OpcUaServer.md and ScriptedAlarms.md all
still presented the test-scaffolding GenericDriverNodeManager as the production
dispatch path. All three now carry the correction.
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
|
||
|
Joseph Doherty
|
88ce8df099 |
docs: close the plan-bookkeeping drift and state the tier truth (§8.5, §8.6, #522)
Bookkeeping: 13 .tasks.json files closed — the 8 the audit named, plus
mesh-phase4 (Task 9 landed
|
||
|
Joseph Doherty
|
adce27e7fa |
refactor(drivers): delete the dead discovered-node injection path (§8.2, #507)
#507 was filed as "injection inert in v3, re-migrate onto the raw subtree". Reading it, the retained code is not revivable: it resolves equipment from EquipmentNode.DriverInstanceId UNION EquipmentTags, and BOTH are structurally empty in v3 (AddressSpaceComposer always constructs EquipmentNode with a null DriverInstanceId; DeploymentArtifact hard-codes an empty EquipmentTags set). Removing the guard would have changed a log line and injected nothing. So it is deleted rather than fixed, and #507 closes as superseded by /raw browse-commit. Removed: HandleDiscoveredNodes, PartitionDiscoveredByDeviceHost, ShouldWarnPartition, PlansRoutingEqual, ApplyDiscoveredPlansForDriver, _discoveredByDriver, the redeploy re-inject tail, DiscoveredNodeMapper, DiscoveredInjection, AddressSpaceApplier.MaterialiseDiscoveredNodes, OpcUaPublishActor.MaterialiseDiscoveredNodes, and the Runtime-local copies of CapturingAddressSpaceBuilder/DiscoveredNode. The connect-time discovery loop goes too (StartDiscovery, RediscoverTick, HandleRediscoverAsync, DiscoveredNodesReady, TriggerRediscovery). With injection gone it had no consumer, and leaving it would either dead-letter or keep browsing real devices up to ~15x per connect to drop the result. ITagDiscovery itself stays — the /raw browse picker drives it through Commons/Browsing/DiscoveryDriverBrowser, which never went through the actor, so the picker is unaffected. deferment.md §4.4 called the 18 DiscoveryInjectionDormantV3 tests "Real — blocked on #507". They are not: they assert an equipment-rooted graft (EquipmentRootNodeId == "EQ-1") that v3 cannot produce, so they could never have unskipped as written. Deleted along with DiscoveredNodeMapperTests, the CapturingAddressSpaceBuilder tests, and the MaterialiseDiscoveredNodes tests in AddressSpaceApplierTests/OpcUaPublishActorTests. Runtime.Tests skipped: 31 -> 13. IHostConnectivityProbe is deliberately NOT resolved here. GetHostStatuses() has no production call site and nothing writes a DriverHostStatus row, but the table was re-created in the v3 initial migration, so deleting on inference would be wrong. Split to Gitea #521 with both options costed. CLAUDE.md, docs/drivers/{Galaxy,TwinCAT,MTConnect}.md updated — they described the seam as dead. Build clean; Runtime.Tests 476 passed / 13 skipped; OpcUaServer.Tests 362 passed / 4 skipped. |
||
|
Joseph Doherty
|
53ede679c3 |
docs+ui(security): state plainly that node ACLs are not enforced (§8.1, #520)
deferment.md §8.1 asked for a decision: wire IPermissionEvaluator into the node manager, or say plainly that ACLs are not enforced. Decision: the latter. The evaluator stays in the tree; #520 tracks the wire-up. Verifying the claim turned up four things worse than the audit recorded: - docs/security.md, not ReadWriteOperations.md, was the highest-risk doc. It claimed an anonymous session is "default-denie[d] any node a session has no ACL grant for". The opposite holds: anonymous can Browse/Read/Subscribe/ HistoryRead the entire address space, and is refused writes and alarm acks only because it carries no roles. - Three of five documented data-plane role strings do nothing. OpcUaDataPlaneRoles declares exactly WriteOperate and AlarmAck; ReadOnly, WriteTune and WriteConfigure are compared against nowhere in src/. The doc called all five "exact, case-insensitive, and code-true". - ReadWriteOperations.md was fiction beyond the ACL claims. OnReadValue has zero occurrences in src/ — the documented read path did not exist. Reads never reach a driver: the node manager is push-model and the SDK serves a Read from the cached pushed value. WriteAuthzPolicy, _sourceByFullRef, _writeIdempotentByFullRef and IRoleBearer are likewise zero-hit, and CapabilityInvoker is not referenced by the OpcUaServer project at all. This file is rewritten rather than bannered. - v2-release-readiness.md claimed a whole enforcement layer shipped — FilterBrowseReferences, GateCallMethodRequests, MapCallOperation, AuthorizationBootstrap, Node:Authorization:* keys. None exist. Whatever landed on the v2 branch did not survive into the shipped tree. The UI change is the operative one: ClusterAcls.razor and AclEdit.razor now warn that a grant is saved and shipped in every deployment artifact but never evaluated, so an operator cannot author a deny rule believing it works. deferment.md gains a §9 execution log tracking §8 remediation. |
||
|
Joseph Doherty
|
e08855fb9d |
docs: source-verified deferment register + correct 17 drifted docs
Adds deferment.md — a source-verified inventory of new-driver status, all 17
open issues, in-code deferrals, open live gates and plan bookkeeping. Every
claim was checked against src/ and git, not against documentation.
Headline finding: three subsystems are authored, persisted and shipped but
never executed —
* Node ACLs: IPermissionEvaluator/TriePermissionEvaluator have zero
production consumers and OtOpcUaNodeManager never references them, yet
ClusterAcls.razor authors NodeAcl rows and ConfigComposer.cs:51 ships them
in every artifact. An authored deny rule has no effect.
* IRediscoverable / IHostConnectivityProbe raise into the void (#518/#507);
nothing ever writes a DriverHostStatus row.
* DriverTypeRegistry is vestigial, so no factory passes a tier and every
driver runs Tier A with the Tier-C protections dormant.
Also records the Calculation driver as picker-visible but unauthorable
(DriverConfigModal has no case) — the "registered but unauthorable" class
recurring after the Sql picker defect — and notes that no parity test guards
DriverConfigModal/DeviceModal, which is why it survived review.
Documentation corrections (source-verified):
* ReadWriteOperations.md claimed "a denied read never hits the driver" via
four types that do not exist in src/. Bannered + struck through; a security
review reading that page would have concluded a per-node ACL gate exists.
* CLAUDE.md: the Change Detection sentence was false (DriverHost consumes
nothing); the mesh Phase 4 and Phase 5 live gates and the auto-down 1-vs-1
gate had all PASSED; the ScriptedAlarmState table was already dropped.
* driver-expansion tracking: Modbus RTU and SQL poll are merged, not
pending — its command table would have made someone rebuild two shipped
drivers in a fresh worktree.
* drivers/README.md: dead DriverTypeRegistry paragraph, retired
SystemPlatform namespace kind, missing Sql + Calculation rows, missing
Modbus RTU-over-TCP transport.
* TwinCAT.md/Galaxy.md promised an address-space rebuild that never happens.
* Historian.md gained the #491 unproven-value-capture pointer.
* IncrementalSync.md and AddressSpace.md bannered as wholesale v2-era (the
rewrite is recorded as still owed, not done here); four v2 status docs
bannered as historical; three wrong-name-for-a-live-type fixes.
Left deliberately untouched: the secrets NoOpSecretReplicator line, which
makes a security claim and needs the real behaviour identified rather than a
rename.
|
||
|
Joseph Doherty
|
813e445936 |
test(mtconnect): close live-gate leg 5 — deploy -> OPC UA read/subscribe PASSED
Re-ran the gate on the rebased branch against a freshly rebuilt
otopcua-host:mtconnect image, so it exercises the merged code.
The 2026-07-24 blocker (antiforgery cookie collision between the running
otopcua-dev rig on :9200 and the isolated stack on :9220) is sidestepped
rather than worked around: the headless deploy API
(POST /api/deployments + X-Api-Key) is AllowAnonymous().DisableAntiforgery()
by design, so cookie scoping is structurally irrelevant to it. No browser
needed — the recommended recipe for any future isolated-stack gate.
Both deploys sealed (DeploymentStatus.Sealed, FailureReason NULL) with both
MAIN nodes acking. Read + subscribe verified over opc.tcp://localhost:4860
across all three type paths: Float64 SAMPLE (sinusoid), Int64 EVENT
(monotonic), String EVENT (READY -> INTERRUPTED). This closes the last
unproven link — driver seam was already covered by the live integration
suite; leg 5 adds DeploymentArtifact -> address space -> OPC UA publish.
Two findings recorded in the plan:
- A bad authored dataType degrades to exactly ONE skipped tag, loudly
(per-tag warning naming the expected keys, BadNodeIdUnknown, "every other
tag is unaffected") while the rest keep streaming — the intended
fail-isolated behaviour, demonstrated rather than argued. Correcting it
took effect with no restart, showing MTConnect is not subject to the
separately-tracked config-edits-discarded defect.
- fixture_asset_changed reading 0x80000000 is NOT an MTConnect defect. The
driver maps the agent's UNAVAILABLE to BadNoCommunication correctly; the
shared publish path collapses every status to a 3-state OpcUaQuality enum
(DriverInstanceActor.QualityFromStatus, statusCode >> 30) and re-expands
Bad to StatusCodes.Bad (OtOpcUaNodeManager.StatusFromQuality). Deliberate
per the enum's own doc comment, but the client-visible consequence appears
undocumented: no driver's Bad/Uncertain sub-code ever reaches a client.
Tracked separately; out of scope here.
Also records the rebase onto master
|
||
|
Joseph Doherty
|
9d430bee62 | docs(mtconnect): correct the MaintenanceMode note; record status-code parity pre-verification | ||
|
Joseph Doherty
|
1d0989d3b4 | test(mtconnect): live gate on an isolated rig — 4/5 legs pass, deploy leg blocked on a cookie collision (Task 21) |