5ba9f1be6f474761550f8017d88f776e29f24296
1473 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Joseph Doherty
|
5ba9f1be6f |
fix(mtconnect): make a /sample stream end loud, and correct the gap contract (Task 7 review)
Code review of c46540ae approved the framing algorithm but moved the risk onto the contract around it. C1 (critical). SampleAsync returned normally on three different events: EOF, the closing --boundary--, and a response that was not multipart at all (where it yielded one parsed snapshot and finished). The seam documents the stream as yielding until ct is cancelled, so a Task 11 pump written to that contract would silently drop its subscription or hot-loop reconnecting; the non-multipart leg reported a configuration error as a healthy finished stream, which is the #485 quiet-successful-termination shape aimed at the very next task. Every non-cancellation end now throws: MTConnectStreamEndedException (ConnectionClosed / ClosingBoundary - transient) or MTConnectStreamNotSupportedException (configuration - reconnecting reproduces it forever), sharing one catchable base. The non-multipart body is still parsed first so an Agent's own MTConnectError text wins, but the document is NOT yielded. I1. IsSequenceGap moved to IMTConnectAgentClient (static) and its first parameter is now expectedFrom. The old name and doc were wrong for every chunk after the first - the correct comparand is the PREVIOUS chunk's NextSequence, and comparing against the opening "from" argument reports a gap on every chunk once the ring buffer rolls, i.e. an endless /current re-baseline storm. Both doc blocks also now record that cppagent answers from < firstSequence with an OUT_OF_RANGE MTConnectError under HTTP 200, which surfaces as InvalidDataException and NOT as a gap - so Task 11 must treat that parse failure as a re-baseline trigger too. I2. Every multipart test served the whole body from one buffer, so every framing test completed in a single ReadAsync - the split-boundary path, the split-header path and the multi-fill loop were correct by inspection only. A ChunkedStream double (N bytes per read, N = 1/3/7) now drives the fixtures through a split transport, plus a >16 KB document that forces a buffer resize. Falsifiability: removing either scan overlap, or collapsing the fill loop, fails ONLY the new chunked tests. I3. Disposal mid-enumeration surfaces as OperationCanceledException via an internal dispose-linked token, not a raw ObjectDisposedException that Task 9's re-init would inflict on an enumerating pump. I5. HeartbeatMs, SampleIntervalMs and SampleCount join RequestTimeoutMs in construction-time positive-value validation. All three reach the query string and an Agent answers heartbeat=0 with an HTTP-200 MTConnectError that names no config key. I4/M2. The no-Content-length framing fallback logs a one-shot warning (the client takes an optional ILogger); a multipart response with no boundary parameter fails fast instead of degrading into a watchdog timeout. M5/M6. Shared element/attribute reading rules extracted to MTConnectXml; the probe parser documents why it alone does not trim BOM/whitespace. The literal U+FEFF in source became a '' escape. Also, from Task 8: MTConnectObservation gained IsStructured (default false), set from element.HasElements. A DATA_SET/TABLE observation's <Entry> children concatenate through element.Value to nonsense ("12" for two entries) that the index would publish as Good, and only the parser can tell that apart from a legitimate space-bearing Message. False for CONDITION observations - their value comes from the element name, so child content cannot corrupt it. 275/275 green in this suite's own files. |
||
|
Joseph Doherty
|
ac0a284055 | feat(mtconnect): observation index + UNAVAILABLE->BadNoCommunication (Task 8) | ||
|
Joseph Doherty
|
cf43f8d0ab |
feat(mtconnect): current/sample parse + sequence-gap detection (Task 7)
Adds the /current and /sample legs to MTConnectAgentClient: - MTConnectStreamsParser: hand-rolled System.Xml.Linq parse of an MTConnectStreams document into MTConnectStreamsResult. Observations are every element child of <Samples>/<Events>/<Condition>, keyed by dataItemId (the element NAME is the data item's type in PascalCase, so a fixed name set would drop observations). A CONDITION's value is its element name, not its text - <Normal/> is empty - and <Unavailable/> normalizes onto the same UNAVAILABLE sentinel a Sample/Event carries as text, so Task 8's no-comms mapping sees one token. Timestamps are normalized to DateTimeKind.Utc explicitly. - IsSequenceGap(requestedFrom, chunk) => chunk.FirstSequence > requestedFrom, so Task 11's pump can re-baseline after a ring-buffer overflow. Equality is NOT a gap. - SampleAsync: ResponseHeadersRead + a hand-rolled multipart/x-mixed-replace frame reader (Content-length fast path, boundary-scan fallback), bounded by a heartbeat watchdog derived from HeartbeatMs so a silent Agent faults instead of wedging the pump - the S7 frozen-peer shape (arch-review R2-01). The long poll gets its own HttpClient with an infinite Timeout so the unary /probe + /current deadline stays bounded. Task 0 package decision, option (c): both TrakHound PackageReferences and their PackageVersions are dropped. MTConnectHttpClientStream takes a URL and dials its own socket (no handler/stream seam, so it cannot serve a socket-free unit suite), and it emits already-parsed documents through the formatter lookup Task 6 proved is missing - framing-only reuse was never actually on offer. The driver now depends on nothing but the BCL. Recorded in the plan's Task 0 CORRECTION block. Also folds in two Task 6 review carry-overs: the stale IMTConnectAgentClient doc comment claiming a TrakHound-backed implementation, and a probe test constructing a mixed-namespace vendor extension (<x:Pump> with default-namespace <DataItems> children) that backs the parser's ignore-the-namespace claim with a test. 187/187 green. |
||
|
Joseph Doherty
|
1990d21733 | feat(mtconnect): agent client /probe parse to device model (Task 6) | ||
|
Joseph Doherty
|
b13cf5c6e8 | fix(mtconnect): options carry the authored tag definitions (Task 2 follow-up) | ||
|
Joseph Doherty
|
79e30d1ead | fix(mtconnect): infer numeric EVENT types from the probe's UPPER_SNAKE spelling (Task 3 follow-up) | ||
|
Joseph Doherty
|
3ca8ddf6f7 | test(mtconnect): canned probe/current/sample XML fixtures incl. forced-gap (Task 4) | ||
|
Joseph Doherty
|
c0729ae5c0 | feat(mtconnect): pure data-type inference table + golden test (Task 3) | ||
|
Joseph Doherty
|
18742ea92b | feat(mtconnect): driver options + tag definition records (Task 2) | ||
|
Joseph Doherty
|
e36b920113 | build(mtconnect): scaffold Contracts+Driver+Tests projects, add to slnx (Task 1) | ||
|
Joseph Doherty
|
c3a2b0f773 |
Merge branch 'feat/mqtt-sparkplug-driver'
# Conflicts: # src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/DriverTypeNames.cs # src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Components/Shared/Drivers/DriverConfigModal.razor # src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/EndpointRouteBuilderExtensions.cs # src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Uns/TagEditors/TagConfigEditorMap.cs # src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Uns/TagEditors/TagConfigValidator.cs # src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/ZB.MOM.WW.OtOpcUa.AdminUI.csproj # src/Server/ZB.MOM.WW.OtOpcUa.Host/Drivers/DriverFactoryBootstrap.cs |
||
|
Joseph Doherty
|
123ddc3f40 |
fix(scripteditor): complete + hover absolute RawPaths (#490)
The issue reports a missing feature. The cause is a stale projection, and the
consequence was worse than the gap described.
`ScriptTagCatalog` still emitted `Tag.TagConfig.FullName` — the pre-v3 driver wire
address. Since v3 the mux key is the **RawPath**: `AddressSpaceComposer` harvests the
`ctx.GetTag("…")` literals into `DependencyRefs`, those become `VirtualTagActor`
dependency refs registered with `DependencyMuxActor`, whose `_byRef` is keyed Ordinal
by `AttributeValuePublished.FullReference` — and a driver's wire ref IS the RawPath
(`DriverHostActor._nodeIdByDriverRef` is keyed `(DriverInstanceId, RawPath)`). The
`{{equip}}/<RefName>` form already substitutes to RawPaths at both compose seams.
So the catalog was wrong in BOTH directions, not merely incomplete: completion
offered paths that could never resolve at runtime, while a CORRECT absolute RawPath
got no completion and hovered as "not a known configured tag path". The file's own
comment admitted the raw/UNS catalog was left for "Batch 2/3" and it never came.
Now projected through the shared `RawPathResolver` — the same byte-parity authority
`DraftValidator` and `AddressSpaceComposer` use — so a suggested path is identical to
the one the deploy gate and the runtime compute. A tag with a broken ancestry chain
resolves to null and is omitted: the runtime could not route it either, so offering
it would be a lie. Hover additionally gains the owning `DriverInstanceId`, which the
topology now makes available (it was always null before).
Editor-accepts <=> publish-accepts is preserved: no diagnostic keys off this catalog
(`OTSCRIPT_EQUIPREF` only marks `{{equip}}` paths), so this changes completion and
hover only — it cannot newly reject a script.
The existing tests pinned the v2 contract and were re-authored: a RawPath only exists
if the RawFolder -> DriverInstance -> Device -> TagGroup -> Tag chain does, so the
seed now builds one. Added coverage for group-nested paths, a driver at the cluster
root (no folder segment), a broken chain being omitted rather than guessed, and the
pre-v3 FullName no longer resolving.
Live-gated on docker-dev (rig rebuilt, both central nodes):
- completion, empty literal -> every RawPath incl. group-nested
`opcua1/plc/OpcPlc/Telemetry/Basic/AlternatingBoolean`
- completion, prefix `pymod` -> `pymodbus/plc/HR200X`, `pymodbus/plc/ImportedTag`
- hover `pymodbus/plc/HR200X` -> "Tag · Type UInt16 · Driver MAIN-modbus"
- hover an unknown path -> "Not a known configured tag path"
AdminUI 742/742; script-analysis 55/55; solution builds 0 errors.
|
||
|
Joseph Doherty
|
0f38f48679 |
merge: Modbus RTU-over-TCP transport (#495)
Wave 1 of the driver-expansion program. The branch was complete and live-gated but had never been merged; it sat 15 commits ahead and 50 behind master. Purely additive behind the existing IModbusTransport seam — the application protocol (function codes, codecs, planner, coalescing, write path, probe, materialisation, HistoryRead) is identical across TCP and RTU; only the wire framing differs ([addr][PDU][CRC-16], no MBAP, no TxId). Zero changes to codecs/planner/health. Selected by a new Transport config field (ModbusTransportMode: Tcp | RtuOverTcp), routed through ModbusTransportFactory (throws on unknown mode), with the string-enum guard on options/DTO/factory and a Transport selector on the AdminUI Modbus form. ModbusSocketLifecycle was extracted from ModbusTcpTransport (behaviour-preserving) and is composed by both transports. Descoped by design: direct-serial (System.IO.Ports) and Modbus ASCII — RTU buses are reached exclusively via a serial->Ethernet gateway. No per-tag config changes; the existing per-tag UnitId already makes a multi-drop bus work. Re-validated against current master at merge time (the branch's own gate predates 50 commits of master): no overlapping files, clean merge, solution builds 0 errors, and Modbus 344/344, Modbus.Addressing 161/161, AdminUI 740/740, Runtime 507/507, Configuration 131/131 all pass. Confirmed the Transport selector is still reachable through master's current authoring path (DriverConfigModal -> ModbusDriverForm), so the branch's live /run gate result still applies. |
||
|
Joseph Doherty
|
4b9bcddbcc |
feat(drivers): periodic reconcile of desired vs actual subscriptions (#488)
Option A from the issue — the cheap consistency check, no interface change. The desired ref set was re-applied on a Connected TRANSITION (`ResubscribeDesired`) and on a deploy's `SetDesiredSubscriptions`, and nowhere else. A subscription lost while the driver STAYED Connected — a subscribe that failed and was never retried, a handle dropped down an error path — left the actor subscribed to nothing while still reporting Healthy, indefinitely. `ReconcileSubscription` now rides the existing 30 s `health-poll` tick: while Connected, a driver holding a desired set but no live handle is re-subscribed. Gated on `ISubscribable`. A driver that cannot subscribe at all can still be handed a desired set, and without the guard it would be told `Subscribe` every tick and fail every tick, forever. This is a state-machine consistency check, NOT a probe — it can only see that this actor holds no handle, never that a handle is present but dead server-side, because `ISubscriptionHandle` is opaque. That is option B, and it stays deferred until a real occurrence shows the handle outliving the subscription; it would need a liveness member implemented across all eight drivers with different subscription mechanics. A redundant `Subscribe` is possible (a tick queued ahead of an already-queued one observes the same no-handle state) and is harmless: `HandleSubscribeAsync` is a `ReceiveAsync`, so no tick is processed mid-subscribe, and it drops any prior subscription before establishing the new one. Suppressing it would need a pending-subscribe flag threaded through every self-tell site — more state than the race is worth. `healthPollInterval` becomes an optional Props parameter, mirroring the existing `rediscoverInterval` seam, so a test can drive the reconcile without waiting 30 s. Three tests. The reconcile itself is proven by a stub whose FIRST subscribe throws: nothing else would ever retry it, so a second attempt can only come from the reconcile — disabling `ReconcileSubscription` turns it red. The two guard tests are absence assertions and are bounded by a positive control rather than a sleep: a counting health publisher makes ticks PROCESSED observable, so "still one subscribe" means the reconcile ran and declined, not that the timer had not fired. Explicitly not deploy-time drift (#486) — re-asserting an already-correct desired set would not have helped there. |
||
|
Joseph Doherty
|
2193d9f2e4 |
fix(config): refuse to store a Sql credential in a node config override (#499)
`ClusterNode.DriverConfigOverridesJson` is the third surface a Sql driver's config is persisted on, and the only one nothing gated. It is a map keyed by `DriverInstanceId` merged onto the cluster-level `DriverConfig`, so a literal `connectionString` pasted there leaks exactly as one pasted into the driver — the leak #498 closed for `DriverInstance.DriverConfig` and `Device.DeviceConfig`. Gated at the SAVE, not at the deploy — a deliberate departure from both options the issue offered: - `DraftSnapshot` carries no `ClusterNode` rows, and adding them would widen the snapshot and every builder of one for a single rule about a value that never enters the artifact. - More to the point, a deploy gate is the wrong instrument here. Node overrides are not in the artifact, so blocking a deploy would not stop the credential being stored — it is already in the database and replicated by then. Refusing the save is the only point where "refuse to store it" is literally true, which is #498's own framing: discarding a secret on read is not the same as refusing to store it. The check moves into a shared `SqlCredentialGuard` so the two enforcement points cannot drift; `DraftValidator` now calls it instead of its own private helper. Kept deliberately narrow — the `Sql` driver's `connectionString` only, and only for instance ids that ARE Sql drivers. The issue asked whether to generalise to credential-shaped keys across every driver type; not doing that, for the same reason #498 did not: a broader sweep would start refusing configs that are legitimate today for drivers which never made an indirect-credential guarantee, which is a regression rather than defence in depth. Widen per driver, as each gains its own contract. The error names the offending driver instance(s) and NEVER the value — it reaches the AdminUI and the audit trail. 14 new cases cover both halves, including the case variants (System.Text.Json binds `ConnectionString` to `connectionString`, so a case variant is the same key, not a bypass), non-Sql ids being ignored, and every blank/malformed/non-object shape. |
||
|
Joseph Doherty
|
240d7aa8aa |
fix(test): give the three absence assertions a positive control (#501)
`AwaitAssert` returns on its FIRST success, so an assertion that is already true at t=0 passes instantly and spends none of its duration. All three sites asserted an absence against a collection that starts empty, so none of them ever gave the handler the time their comments claimed. Replaced with mailbox-ordering positive controls rather than sleeps: - `RouteNativeAlarmAck_unknown_node_is_dropped` — follow the unmapped ack with a MAPPED one. The host forwards via `Tell` and the child handles `RouteAlarmAck` with `ReceiveAsync` (which suspends its own mailbox), so once the control reaches the driver the unmapped one has provably been handled. - `RouteNativeAlarmAck_on_non_primary_is_dropped` — the role itself is the control: return the node to Primary and re-send. Comments (`gated` / `control`) are the discriminator, so a leaked ack is visible rather than merged into a count. - `PeerProbeSupervisorTests.Single_node_snapshot_spawns_no_children` — a FOURTH site the issue did not list. It says the other `ChildCount.ShouldBe(0)` waits are each preceded by a `ShouldBe(1)`; that is true of three of the four, but not this one, which asserts the initial state. Now followed by a two-node snapshot: the count reaching 1 proves the single-node snapshot was processed, and a local-node child would have made it 2. The issue warned these may turn red, which would be a real defect surfacing. They did not — the routing is correct. Proven by breaking it deliberately instead: disabling the Primary gate reddens the Secondary test, and routing unmapped ids to an arbitrary mapping reddens the unknown-node test. Under the old form both breaks passed. |
||
|
Joseph Doherty
|
95295210b0 |
fix(test): scope the EventPump counter capture to its own pump (#503)
Root cause was cross-test measurement leakage, not a timing budget. EventPump's `Meter` is STATIC — one instance per process — and a `MeterListener` can only select by *meter* in `InstrumentPublished`. `StartMeterCapture` therefore received measurements from every `EventPump` alive anywhere in the assembly. xUnit runs test classes in parallel and several build pumps (`EventPumpStreamFaultTests`, the `Driver-X` test in this same file), so a sibling's events landed in these counters. That is why the flake needed a busy box: it needed two tests to overlap. Measured directly during the gate — `Received = 11` in a run that emitted 10. The old `Received == Dispatched + Dropped + InFlight` assertion is what turned the leak into a failure. `InFlight` is DERIVED as `Received - Dispatched - Dropped`, so the identity is a tautology that cannot fail on a real defect — only on a foreign increment landing between its four separate `Interlocked.Read` calls. Hence the reported subject, `counters.Received`. It is deleted, along with the `InFlight` member, and replaced by direct assertions on the three MEASURED counters. Three further defects in the same test, all found on the way: - **The "held" dispatch loop was never held.** `OnDataChange` is `EventHandler<T>` — void-returning — so `async (_, _) => await gate` is async VOID: it returned to the dispatch loop at the first await and blocked nothing. Measured `Dispatched = 2..3`, never 0. Drops happened only when the producer happened to outrun the consumer, so the arrangement the doc comment describes was fiction and the assertions rode on scheduling luck. Now a synchronous `ManualResetEventSlim.Wait()` on the loop's own thread, pinned by `Dispatched.ShouldBe(0)`. - **Fixed `Task.Delay(150)`** replaced by a presence poll on the drop count (#500's rule). Dropped starts at 0, so it cannot be satisfied by the initial state. - **Emission is staged** — one event, wait for the dispatcher to enter the handler, then the other nine — so the arithmetic is exact (`10 - 1 held - 2 buffered = 7`) rather than scheduler-dependent. The gate release moved into `finally`, ahead of disposal: `DisposeAsync` awaits the dispatch loop, which is now genuinely parked on that gate, so a failed assertion would otherwise hang instead of failing. Verified: 15/15 clean under the exact contention the issue measured (Runtime + AdminUI + Core concurrently) — where the half-fixed version still failed on run 4. Falsifiability: restoring the async-void handler turns the new assertions red. |
||
|
Joseph Doherty
|
47d148daf9 |
fix(adminui): never slice a DB-sourced string with the range operator (#504)
`Scripts.razor` rendered a hash prefix as `@s.SourceHash[..12]…`. `SourceHash`
is a plain nvarchar(64) with no length floor, so any value shorter than 12
characters threw ArgumentOutOfRangeException out of BuildRenderTree — unhandled
in Blazor, which takes the WHOLE page to an HTTP 500, not just the offending
row. Live-reproduced on docker-dev, where two Script rows carry `h1` and
`h-abs-hr200`: /scripts was a hard 500.
New `Components/Shared/DisplayText.Abbreviate(string?, int)`:
- null/blank renders the Admin UI's em-dash placeholder rather than throwing;
- a value already within budget renders verbatim with NO ellipsis, so the
display never implies text that isn't there (the old markup appended "…"
unconditionally, outside the slice — `h1` would have shown as `h1…`);
- a maxLength < 1 is clamped, so a display bug cannot become a page crash.
Applied to every unguarded range-operator slice over a DB-sourced string found
by sweeping the AdminUI for `[..N]`:
Scripts.razor SourceHash (the reported crash)
Certificates.razor Thumbprint (read off the on-disk store)
Deployments.razor x2 RevisionHash
Clusters/ClusterOverview.razor RevisionHash
The other `[..N]` hits are safe and untouched: Guid.ToString("N") slices
(always 32 chars) and the `Length > 60 ? [..60]` ternaries that self-guard.
NOTE — the issue text blamed the docker-dev seed; that was wrong and is
corrected on #504. `seed-clusters.sql` inserts only ServerCluster, ClusterNode
and LdapGroupRoleMapping. The short-hash rows came from earlier live-gate
authoring. A freshly seeded rig is fine; the trigger is any Script row that did
not go through ScriptEdit / UnsTreeService.HashSource — REST-API authored,
hand-inserted, or migrated in.
Tests: 14 new in DisplayTextTests, covering the short/null/blank/clamped cases
and a never-throws sweep over every value x budget combination. AdminUI suite
739/739.
Live-verified on docker-dev (AdminUI has no bUnit): /scripts 500 -> 200 across
both central nodes, rendering `hash=h1` and `hash=h-abs-hr200` in full;
/deployments still truncates 64-char hashes at 12 with the ellipsis;
/certificates and /clusters/MAIN unchanged.
|
||
|
Joseph Doherty
|
549c656489 |
fix(alarms): re-assert non-normal scripted-alarm conditions after a (re)load (#487)
A deploy that triggers a FULL address-space rebuild clears `_alarmConditions` and re-materialises every condition node fresh — inactive, acked, confirmed, unshelved. The engine reloads its persisted state and re-derives Active from the predicate, so it ends up correct; but there is no transition to report, so `LoadAsync` yields `EmissionKind.None` and `OnEngineEmission` drops it. Nothing writes the node. Live-reproduced on the docker-dev rig against the pre-fix image: an alarm that had fired and cleared but was still UNACKNOWLEDGED came back from a rebuild reporting Acknowledged with Retain=false — so it disappeared from ConditionRefresh entirely, while the engine still held Unacknowledged. An operator's outstanding alarm silently vanishes from the alarm list on deploy. `ScriptedAlarmHostActor.ReassertConditionNodes` closes it. After each load it reads the new `ScriptedAlarmEngine.GetProjections()` and sends one `AlarmStateUpdate` per alarm NOT in the Part 9 no-event position. Load-bearing properties: - Node-only. It sends `AlarmStateUpdate` and nothing else — never the `alerts` topic, never the telemetry hub. `alerts` is the historization path, so a row per alarm per deploy would append a duplicate historian/AVEVA record every time anyone deploys. This is why it reads a projection rather than re-emitting: `ScriptedAlarmProjection` carries no `EmissionKind`, so there is nothing for the emission machinery — or a future refactor — to mistake for a transition. The issue's sketch said `GetState(alarmId)` + `LoadedAlarmIds`; that alone would have clobbered the node's severity and message, so the projection also carries the definition's severity, the resolved message template, and the last-observed worst-input quality. - Only non-normal alarms. One in the no-event position already matches what the materialise built, so a never-fired alarm stays byte-identical to a deploy without this pass (including its Message, which the materialise seeds with the display name), and an all-normal deploy writes nothing at all. - Timestamp is the condition's own `LastTransitionUtc`, not the deploy instant. - No duplicate Part 9 events: `WriteAlarmCondition`'s delta-gate compares against the node's CURRENT state, so a re-assert onto a freshly materialised node is a genuine delta (fires once — correct, the rebuild reset what clients see) while one onto a surgically-preserved node is suppressed. Tests: 6 new host tests + 6 new engine tests. Falsifiability checked by disabling the call — 4 of the 6 host tests go red; the other 2 are absence guards against an over-broad fix and pass either way by design. Live gate (docker-dev, central-2, pre-fix image vs. rebuilt image): - pre-fix: condition absent from ConditionRefresh after a rebuild; - post-fix: present, Unacknowledged, Retain=True, stamped with its own LastTransitionUtc rather than the restart instant; - /alerts stayed empty across two re-asserts, with the panel proven live by a real ACTIVATED transition immediately afterwards — no duplicate history. |
||
|
Joseph Doherty
|
154171f48c |
test(runtime): fix the intermittent Runtime.Tests failure — 4 ordering/logic defects + the presence-budget class (#500)
The reported symptom was "Runtime.Tests occasionally reports Failed: 1" with no test
name. Instrumenting 30 full-assembly runs reproduced it at 13% (4 runs) and showed it
was never ONE flaky test: five failures across three distinct tests in that batch, and
five more distinct tests over the verification rounds that followed.
Two hypotheses were tested and DISPROVED by measurement before anything was changed:
- Cluster-formation timeout. Every test in this assembly forms a real single-node Akka
cluster over TCP in RuntimeActorTestBase's constructor and waits 5 s for MemberStatus.Up
— 219 formations per run. Instrumented: idle max 960 ms, under-load max 1470 ms, zero
timeouts. A 3.4x margin; not the cause.
- Ephemeral-port exhaustion from that bind churn. TIME_WAIT measured at ~0 throughout.
Not the cause.
FOUR GENUINE ORDERING/LOGIC DEFECTS (none is a timeout):
1. VirtualTagHostActorTests.ApplyVirtualTags_respawns_child_when_plan_changes_in_place
The test loops to accept RegisterInterest/UnregisterInterest in EITHER order — and
then asserts on mux.LastSender, which DEPENDS on that order. Under the [Register,
Unregister] interleaving LastSender is the DYING child and the assertion fails. Now
captures the sender of the Register message as it arrives. Fully deterministic; no
timing involved. (Swept the other four LastSender sites: all drain the Unregister
first, so their ordering is fixed. Only this one was unsafe.)
2. DriverHostActorWriteRoutingTests.Primary_routes_write_via_uns_nodeid_to_driver_by_rawpath
ApplyAck marks the end of the APPLY, not the point a write can succeed: the child is
still in Connecting, which deliberately fast-fails writes ("driver not connected"),
and the NodeId->driver reverse map has not been pushed. Now retries until accepted.
This does not weaken the assertion — every rejection branch replies WITHOUT reaching
the driver, so Writes.Count.ShouldBe(1) still means exactly what it did.
3. HistorianAdapterActorTests.Redundancy_snapshot_without_local_node_...
One 500 ms constant served both presence budgets (AwaitAssert) and absence windows
(ExpectNoMsg). Different quantities that happened to share a number, so the presence
budget could not be raised without slowing every absence check. Split into
AssertTimeout (5 s) and Settle (500 ms).
4. ContinuousHistorizationRecorderTests.Retry_after_writer_failure_eventually_acks
Waited on `outbox == 0`, which is ALSO true before the first append — so the poll
could win the race against the recorder and return having observed the INITIAL state,
after which the CallCount>=2 check outside the block failed against a recorder that
had not run. Both conditions are now polled together with the retry count as the
discriminator. (The preceding test in the same file documents this exact trap.)
THE PRESENCE-BUDGET CLASS:
After those four, three consecutive 30-run rounds each still failed once — on a
DIFFERENT test, in a DIFFERENT file, every time (OpcUaPublishActorApplyFailureTests 2 s,
OpcUaPublishActorRebuildTests 2 s, OpcUaPublishActorTests 500 ms). That is a
distribution, not a defect, and fixing it one test at a time was the wrong method.
RuntimeActorTestBase now carries a documented PresenceBudget (15 s) and 57 sub-3 s
AwaitAssert/AwaitCondition durations across four files route through it.
Why this is safe rather than sloppy: a presence budget is an upper bound before giving
up, not a wait — these helpers poll and return the instant the condition holds. Raising
one costs nothing on the happy path and CANNOT make a genuinely failing assertion pass;
it only changes how quickly a real breakage reports. Absence windows are the opposite
(the elapsed time IS the assertion) and were deliberately left untouched with their own
short, individually-calibrated literals.
ScriptedAlarmHostActorTests is a special case, diagnosed rather than merely raised: its
8 s budget was sized for message passing but actually waits on a REAL Roslyn compile
(ApplyScriptedAlarms -> ScriptedAlarmEngine.LoadAsync compiles every predicate). Cold
script compiles are the slowest and most variable thing in the assembly, so that class
gets 30 s with the reason recorded.
Verification: 30 consecutive full-assembly runs, 30 passed / 0 failed, against a 13%
baseline. Full solution builds; all 41 unit-test projects green.
NOT fixed here, filed as #501: DriverHostActorNativeAlarmAckRoutingTests:90 and :116 use
AwaitAssert for an ABSENCE assertion, so they return instantly and prove nothing. They
are excluded from the budget change on purpose — a bigger number does not fix them, and
rewriting them to settle-then-assert may legitimately turn them red.
|
||
|
Joseph Doherty
|
4dc2ad8084 |
feat(sql): implement the design §8.1 catalog identifier-validation gate (#496)
Until now ISqlDialect.QuoteIdentifier was the SOLE defence for authored
table/column names: an identifier went from the TagConfig blob straight into a
command text, bracket-quoted. The residual risk was bounded — a hostile name is
quoted into one nonexistent object, the query fails after the connection opens,
and the tag Bad-codes — but "bounded" is not "filtered", and the design promised
a filter. Both ISqlDialect and SqlServerDialect carried a doc paragraph saying so.
SqlCatalogGate + SqlCatalogLoader now resolve every authored identifier against
the live catalog at Initialize, and REPLACE it with the catalog's own spelling.
The identifier text in an emitted poll query is therefore a string this driver
read back out of ListSchemas/ListTables/ListColumns — not operator input. Quoting
becomes the backstop it was documented to be.
Decisions worth knowing before touching this:
- Substitution, not just validation. Matching is exact-ordinal first, then a
UNIQUE case-insensitive hit: SQL Server's default collation is CI so case
variants have always worked, and rejecting them would break valid deployments.
An ambiguous CI match under a case-sensitive collation is refused rather than
guessed — picking one would publish another column's data under the operator's
node, which is worse than rejecting the tag.
- Charset check BEFORE catalog lookup. Each identifier goes through
QuoteIdentifier for its rejection rules first, so a name carrying a control or
Unicode format character is rejected WITHOUT its value being echoed into a log
line (Trojan-Source). A name that passes is safe to render, which is why
catalog-miss messages do name it — an operator hunting a typo has to see what
they wrote. Both halves are pinned by tests.
- A rejected tag keeps its node. It is dropped from the POLLED table but stays in
the AUTHORED table, so it still materializes and reads BadNodeIdUnknown —
§8.1's specified outcome. My first wiring dropped it from both, which deleted
the node instead; an existing test (ReinitializeAsync_recoversFromFaulted)
caught it. A status code can only be published by a node that exists, and a
missing address-space entry is far harder to diagnose than a Bad quality.
- An unreadable catalog FAULTS Initialize; it does not reject every tag. That is
the absence of evidence about the tags, not evidence against them — rejecting
all of them would serve a confidently-empty address space and send the operator
hunting typos that do not exist. Zero visible schemas is treated the same way,
because that is exactly what a missing GRANT looks like. Faulting lands
DriverInstanceActor in Reconnecting with its retry timer running.
- Bounded load: one schema list, one default-schema scalar, then one ListTables
per distinct authored schema and one ListColumns per distinct authored relation
— never a full catalog enumeration, and nothing after Initialize. Every
authored name reaches the catalog queries as a bound @schema/@table parameter,
so building the allow-list cannot itself be an injection vector.
- ISqlDialect gains DefaultSchemaSql ("SELECT SCHEMA_NAME()"). An unqualified
`TagValues` must resolve in whatever schema the SERVER reports; hardcoding dbo
would be a silent lie on any estate that maps service accounts to their own
default schema. It is a query, not a constant, because the answer is
per-connection.
- Accepted v1 limitation: a 3-part db.schema.table (or linked-server name)
addresses a catalog this connection cannot enumerate, so it cannot be
allow-listed and is rejected with a message pointing at the fix — expose the
data through a view in the connected database.
VerifyLivenessAsync's wall-clock pattern is extracted to RunBoundedAsync and
reused, so the new I/O gets the same R2-01 / STAB-14 protection rather than a
second hand-rolled copy. (It also fixes a latent bug in the extracted code: the
OperationCanceledException arm detached the wrong task.)
Tests: 24 pure gate tests + 9 end-to-end driver tests against the real SQLite
catalog + 3 new live tests against the real SQL Server on 10.100.0.35 (21 in that
suite now pass, exercising the real SELECT SCHEMA_NAME() + INFORMATION_SCHEMA
path). Verified falsifiable: bypassing the gate turns exactly the three rejection
assertions red and leaves the rest green.
SqlInjectionRegressionTests is deliberately NOT rewritten to expect
BadNodeIdUnknown. It drives SqlPollReader directly, below the gate, and pins the
quoting backstop on its own — defence in depth is only worth the name if each
layer holds independently. Rewriting those assertions would delete the backstop's
only coverage and leave the gate a single point of failure. Its scope note, which
said the gate does not exist, is updated to say why it stays where it is.
|
||
|
Joseph Doherty
|
1919a8e538 |
feat(config): reject a persisted Sql connectionString at the deploy gate (#498)
The Sql driver's "a pasted literal connection string is never persisted" guarantee was
enforced only on the READ path: SqlDriverConfigDto has no connectionString property and
UnmappedMemberHandling.Skip drops the key on deserialization. Nothing stopped it being
WRITTEN. Config blobs are schemaless JSON columns and the fallback authoring pattern for
a driver without a typed form is a raw-JSON textarea, so an operator pasting
{"connectionString":"Server=...;Password=..."} would put a live credential in the
ConfigDb — persisted, replicated to every node's artifact cache, readable by anyone with
config access — while the runtime silently ignored it and the driver failed to connect.
Discarding a secret on read is not the same as refusing to store it.
DraftValidator.ValidateSqlConnectionStringNotPersisted fails the deploy
(SqlConnectionStringPersisted) when a Sql-typed driver instance's DriverConfig, or the
DeviceConfig of any device beneath it, carries a top-level connectionString key.
- Both surfaces are checked because DeviceConfig is merged onto DriverConfig before the
DTO sees it — a credential pasted into the device blob is the identical leak.
- The key is matched case-insensitively: System.Text.Json binds ConnectionString to a
connectionString property by default, so an ordinal match would leave the obvious
bypass open.
- Only the top level is scanned; the DTO is flat, so a nested occurrence cannot bind and
is not the mistake this rule exists to catch.
- The message never echoes the value — validation errors reach the AdminUI, the deploy
log and the audit trail, and repeating the string there would leak the credential the
rule is refusing to store. Pinned by a test.
- Malformed/blank/non-object JSON never throws: a parse failure would take down every
other rule in the same pass.
16 tests. Verified falsifiable — with the rule disabled the 6 positive assertions fail
and the 10 negatives stay green, so none of them passes vacuously.
Design §8.2's "if an inline connectionString is ever permitted (dev convenience only),
the AdminUI redacts it" carve-out is withdrawn in the same change: redaction only hides a
credential that has already been written.
Not covered, and filed as a follow-up: ClusterNode.DriverConfigOverridesJson is a third
persistence surface for a driver config blob, but it is not part of DraftSnapshot, so
this validator cannot see it.
|
||
|
Joseph Doherty
|
b95b99ba8e |
fix(drivers): correct 16 wrong OPC UA status-code constants + guard them by reflection (#497)
Every value verified against Opc.Ua.StatusCodes in the pinned SDK (1.5.378.106) by reflection, not by copying siblings. All are client-visible: OPC UA clients branch on status. The three named in #497: - Historian.Gateway SampleMapper "BadNoData" 0x800E0000 -> 0x809B0000 (0x800E0000 is BadServerHalted) - Galaxy "BadTimeout" 0x800B0000 -> 0x800A0000 (0x800B0000 is BadServiceUnsupported) - TwinCAT BadTypeMismatch 0x80730000 -> 0x80740000 (0x80730000 is BadWriteNotSupported) BadTypeMismatch was wrong in three MORE drivers the issue did not name — FOCAS, AbLegacy and AbCip carried the same 0x80730000. Every call site is a FormatException/InvalidCastException conversion failure, so the name was right and the value was wrong in all four. Adding the reflection guard then surfaced nine further defects offline tests had never checked: - Galaxy GoodLocalOverride 0x00D80000 -> 0x00960000 (not a UA code at all) - Galaxy UncertainLastUsableValue 0x40A40000 -> 0x40900000 - Galaxy UncertainSensorNotAccurate 0x408D0000 -> 0x40930000 - Galaxy UncertainEngineeringUnitsExceeded 0x408E0000 -> 0x40940000 - Galaxy UncertainSubNormal 0x408F0000 -> 0x40950000 - TwinCAT BadOutOfService 0x80BE0000 -> 0x808D0000 (was BadProtocolVersionUnsupported) - TwinCAT BadInvalidState 0x80350000 -> 0x80AF0000 (was BadAttributeIdInvalid) - AbCip/AbLegacy GoodMoreData 0x00A70000 -> 0x00A60000 (was GoodCommunicationEvent) - Historian.Gateway GatewayQualityMapper Good_LocalOverride 0x00D80000 -> 0x00960000 Galaxy's whole Uncertain block read as though the OPC DA quality byte could be shifted into the UA substatus position; it cannot — the substatuses are an unrelated enumeration. GatewayQualityMapper already held the correct table, which is what the Galaxy values are now reconciled against. Guard: StatusCodeParityTests (Core.Abstractions.Tests, which already project- references every driver) reflects over every `const uint` in the deployed ZB.MOM.WW.OtOpcUa.Driver.*.dll whose name reads as a status code, and asserts it equals Opc.Ua.StatusCodes.<name>. Discovery is by convention, so a new driver assembly referenced by that project is covered with no edit here. It went red on all nine defects above before the fixes and now checks 109 constants green. Because reflection can only see a NAMED constant, two inline literals were hoisted so the guard can reach them: Galaxy's BadTimeout/Bad into StatusCodeMap, and GatewayQualityMapper's 15-entry table into a new HistorianStatusCodes. An inline `0x800B0000u, // BadTimeout` at a call site is exactly how the Galaxy defect survived. The drivers stay SDK-free by design; only the test project references Opc.Ua.Core, as the oracle. Also corrected: tests that pinned the wrong values (Galaxy StatusCodeMapTests, SampleMapperTests, GatewayQualityMapperTests — all written from the same bad source), a mislabelled comment in DriverInstanceActorTests, and stale constants in two design docs, one of them the unexecuted MTConnect driver design that would have propagated BadNoData's wrong value into a new driver. The CLI's SnapshotFormatter value->name table was checked against the SDK and is correct; no change needed. |
||
|
Joseph Doherty
|
ee12568cab |
refactor(modbus-rtu): distinct CrcMismatch/UnitMismatch desync reasons
The RTU framer overloaded DesyncReason.TruncatedFrame for three distinct failures — a genuine short read, a CRC-16 validation failure, and a CRC-valid reply from the wrong RS-485 slave. Split the latter two into dedicated CrcMismatch / UnitMismatch enum members so a future .Reason consumer (metrics, operator diagnostics) can tell a wiring/EMI CRC fault apart from a mis-addressed multi-drop reply. Behaviour is unchanged — all three still throw ModbusTransportDesyncException and tear the socket down. Framing tests now assert the specific Reason on each path. Follow-up #12 from the Modbus RTU-over-TCP plan. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
d5bd4226ee |
docs(mqtt): Sparkplug P2 live-verified; MQTT driver complete
Task 26 — the P2 milestone gate. Ran the live /run verification on an isolated
docker-dev rig (project otopcua-mqtt, ports 9210/4850-4851/14350, image built
from this branch) against the real Mosquitto TLS+auth broker and the C#
Sparkplug edge-node simulator on 10.100.0.35, and recorded the result.
The gate found three defects. Two are the same defect class the P1 gate found
twice — a hand-maintained AdminUI surface left behind by a driver-side feature —
and the third is a pre-existing cross-driver bug that only a Sparkplug flow
could surface.
1. CRITICAL, FIXED — MqttDriverForm still shipped its P1 Sparkplug PLACEHOLDER.
Switching Mode to SparkplugB rendered a "not available yet" notice and NO
Group ID field, so with Sparkplug ingest fully shipped there was still no
way to author a Sparkplug driver from the AdminUI at all. Sparkplug.GroupId
is the driver's entire subscription filter (spBv1.0/{GroupId}/#): blank ⇒
connected, Healthy, ingesting nothing. Now authors all five Sparkplug keys,
MERGES over the existing sub-object rather than replacing it (so a key a
newer driver adds inside it survives an older AdminUI), leaves it untouched
in Plain mode, and validates the group id as the topic segment it is.
Pinned by 8 MqttDriverFormModelTests cases, verified falsifiable — 8 RED
with the fix stubbed out.
2. PRE-EXISTING + CROSS-DRIVER, FIXED — every node label in the shared
DriverBrowseTree was an <a href="#">. In a Blazor Web App, blazor.web.js's
enhanced-navigation click interceptor resolves a bare "#" against
<base href="/">, so clicking ANY browse-tree node label navigated the whole
AdminUI to "/", tore down the circuit, and destroyed the hosting modal —
losing the browse session AND the tag selection. @onclick:preventDefault
does not help: it suppresses the browser's default action, not Blazor's own
interceptor. Labels are now <button type="button">. Dates to the component's
introduction (2026-05-28) and affects EVERY driver's picker; it surfaced
only now because choosing a rebirth scope is the first flow that requires
clicking a label rather than the ▶ toggle (always a button, always fine).
3. FIXED (mitigation) — the browse tree rendered exactly once, at open, and
never again, while OpenAsync returns as soon as the SUBSCRIBE is granted.
On Sparkplug that means it is almost always empty forever, because births
are never retained. Added a Refresh button that re-reads the root against
the SAME session (nothing reconnects, nothing observed is lost, the tag
selection is kept, only the armed rebirth scope is cleared).
Live gate result (all against the real broker + simulator, group OtOpcUaSim):
- driver authored end-to-end through the fixed MqttDriverForm; blob is
Mode:SparkplugB + Sparkplug.GroupId:OtOpcUaSim with enums as names
- browse: BROWSER OPEN chip, Sparkplug-only rebirth panel, tree renders
Group → EdgeNode → [Device] → Metric with the device folder Filler1 and
node-level metric leaves SIDE BY SIDE, and "Node Control/Rebirth" as ONE
leaf (the metric-name-is-not-a-topic-segment rule holding)
- rebirth: group scope names the group + the 32-node cap, Cancel published
NOTHING (simulator log unchanged); edge-node scope → "1 command published"
and the simulator logged "rebirth NCMD #1 received; republishing metadata";
group scope → "2 commands published", both edge nodes re-birthed; a device
and a metric both resolve UP to "edge node EdgeA"; the panel does NOT render
for a Plain MQTT device
- browse-commit wrote bindable tuples with no topic/address key —
{"groupId","edgeNodeId","metricName","deviceId"} for the device metric and
deviceId ABSENT (not blank) for the node-level one, datatypes inherited from
the birth (Int64 / Float)
- deployed (Sealed), and both nodes served live changing values through OPC UA
at the 2 s cadence — Good 0x00000000, no BadNodeIdUnknown
- death→STALE: stopping the simulator drove both nodes to 0x80000000 with an
empty value ~2 s later, and restarting it recovered them to Good on the new
birth (FillCount back to its birth-declared 1000, then counting)
- rediscovery fired exactly TWICE (once per authored scope: OtOpcUaSim/EdgeA
7 metrics, OtOpcUaSim/EdgeA/Filler1 3 metrics) and was then gated across
many subsequent births/rebirths — the anti-storm change gate working, and
EdgeB correctly filtered out as an unauthored scope
- tag editor: opens in SparkplugB (mode inference on reopen) with all four
descriptor fields populated; blank group → "A Sparkplug group ID is
required."; "Plant/1" rejected; "Node Control/Rebirth" ACCEPTED and saved
with the slash whole; dataType override persists and its key is ABSENT again
after selecting "(from birth certificate)"
Two things are NOT verified, and are recorded rather than claimed:
- Rediscovery is INERT in v3 and this is a platform gap, not an MQTT one:
nothing subscribes to IRediscoverable.OnRediscoveryNeeded and
DriverHostActor.HandleDiscoveredNodes hard-returns. A DBIRTH introducing a
metric does NOT change the OPC UA tree; redeploy. Verified from the driver's
log line only, exactly as far as it can be verified.
- A metric name containing "/" cannot be BROWSE-COMMITTED: the derived raw tag
Name is a RawPath segment and may not contain "/", so the spec-mandatory
"Node Control/Rebirth" is refused ("Row 3: Name must not contain '/'"). The
refusal is loud and all-or-nothing, never a silently mis-bound tag, and
Manual entry is the working path. Fixing it needs a name-sanitisation policy
with a collision answer, so it was recorded rather than guessed at.
Also left open + documented: an empty browse tree cannot arm a rebirth (the
scope comes from a tree click, and the session side already accepts a bare
{group}/{edgeNode} — only a UI path is missing); _canRebirth is captured at
browse-open; the tag editor injects the Plain-only payloadFormat key into a
Sparkplug blob (cosmetic — the factory routes on the tuple).
Suites: offline 1528 passed / 0 failed (581 Driver.Mqtt + 809 AdminUI + 138
Core.Abstractions); live 15/15 against the broker + simulator.
Docs: docs/drivers/Mqtt.md brought fully up to date for P2 (Sparkplug config
sub-object, both tag shapes, the ingest state machine, rediscovery, browse +
rebirth + Refresh, a Known-gaps table); Mqtt-Test-Fixture.md gains the simulator
and drops its "Sparkplug has no fixture" claims; infra/README.md §3 gains the
simulator row; CLAUDE.md gains the simulator endpoint facts; the tracking doc
marks MQTT/Sparkplug COMPLETE. docker-dev/docker-compose.mqtt.yml is the
isolated-rig overlay the gate ran on.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
|
||
|
Joseph Doherty
|
8d9155682d |
fix(mqtt): browse-commit writes a bindable address + a Request-rebirth affordance
Two gaps the Task 23/24 review found, both blocking Task 26's live gate.
Gap 1 — browse-commit produced a silently dead MQTT tag. RawBrowseCommitMapper
had no `Mqtt` case, so a committed leaf fell through to the generic
`{"address": …}` key. Neither MqttTagDefinitionFactory entry point reads
`address`: the tag deployed clean and reported BadNodeIdUnknown forever, with
no signal at commit time. Predates Task 23 (Plain was affected too), but Task 23
built the Sparkplug metric tree precisely so an operator could browse and commit
a binding.
The address is a DESCRIPTOR, not a reference string, and it cannot be recovered
from the browse node id: `{group}/{node}[/{device}]::{metric}` where a metric
name legitimately contains `/` (`Node Control/Rebirth`) — the ambiguity
MetricSeparator's remarks already name. So the session STATES it, via a new
`AttributeInfo.AddressFields` seam, and the mapper reads it. Which keys are
emitted is also how the mapper learns Plain vs Sparkplug — the driver type
reaching it is just `Mqtt`, and the mode lives on the driver config. Key names
are single-sourced in the new `MqttTagConfigKeys` (producer, mapper, factory),
with the literals pinned by a test so a symmetric rename cannot silently unbind
already-persisted blobs. A leaf with no stated address is refused at commit in
words rather than committed dead.
Gap 2 — RequestRebirthAsync had no UI. Task 23 shipped it backend-only; Task 26
step 1 assumes the button, and Task 25's runbook notes the picker tree stays
empty until a birth lands, so it is the only way to fill it on demand. Added to
the /raw browse modal: scope = the clicked tree node (device/metric resolve up to
their edge node, group fans out and is refused whole past 32), an explicit
two-click confirm naming the resolved scope and its consequence, the outcome or
the refusal shown rather than swallowed. Offered only for a Sparkplug session
(new `IRebirthCapableBrowseSession.RebirthAvailable` — one session class serves
both modes, so a type test alone would draw a button that can only throw) and
only to a DriverOperator; the server-side gate remains the boundary.
Tests: MQTT 545 → 581, AdminUI 781 → 800. The round-trip tests feed the emitted
blob to the REAL factory and were falsified by mutating the emitted key name.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
|
||
|
Joseph Doherty
|
767e7031b3 |
test(mqtt): C# Sparkplug edge-node simulator + §3.6 live matrix
Adds a project-owned, controllable Sparkplug B edge node (SparkplugEdgeNode) that
encodes with the SAME generated Tahu schema the driver decodes with — so the live
gate checks encode/decode symmetry rather than the driver against itself — and
drives the §3.6 matrix end-to-end over the real TLS+auth Mosquitto fixture.
The simulator frames its topics independently of SparkplugTopic.Format: a
simulator borrowing the parser's formatter could not detect a bug in it, because
both sides of the comparison would be wrong together. It registers a real NDEATH
Last Will at CONNECT, restarts seq at 0 on NBIRTH, publishes DATA metrics as
alias-only, carries bdSeq per session, and encodes signed ints as two's
complement in the unsigned proto field.
SparkplugLiveTests (Category=LiveIntegration, env-gated, skip-clean) covers:
birth -> alias-only data -> OnDataChange under the RawPath; late-join rebirth
NCMD honoured by an independent decoder; alias reuse across a rebirth routing by
metric NAME; a stale-bdSeq NDEATH ignored while the current one stales; the real
broker-published Will staling and the next birth restoring; seq wrap 255->0
requesting no rebirth while a deliberate gap does; and the browser's passive
window asserted ON THE WIRE (a third client watching spBv1.0/{group}/NCMD/#),
plus RequestRebirthAsync node-vs-group enumeration.
Two things make the suite falsifiable rather than decorative: the seq-wrap test
builds its ingestor with rebirthDebounce: TimeSpan.Zero (at the shipped 10s
default a wrap-triggered NCMD would be swallowed and the test would pass for the
wrong reason) and pairs the silence with a deliberate gap that must produce one;
and the passivity assertion observes the broker rather than the session's own
publish counter, which can only see the seam it guards.
Docker/docker-compose.yml gains a profile-gated `sparkplug-sim` service running
the same engine standalone against the same broker (TLS, CA-pinned) — a manual
driving aid for the AdminUI picker, deliberately NOT a test dependency, since the
matrix needs an edge node it can command mid-test. Its app directory is publish
output (publish-simulator.sh), gitignored like secrets/.
Offline: 15 skipped, 0 failed with no env set. Live: 15/15 passed against
10.100.0.35:8883. Falsifiability spot-check: mutating AliasTable.RebuildFromBirth
to merge instead of replace, and neutering SparkplugCodec.ReinterpretSigned for
Int32, turned exactly the two corresponding tests red and nothing else; both
mutations reverted.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
|
||
|
Joseph Doherty
|
64ec7aa43a |
feat(mqtt): Sparkplug birth-driven browser tree + scoped Request-rebirth action
Unseals the Sparkplug branch of the MQTT address-picker browser that Task 10 deliberately sealed shut, and adds the first (and only) sanctioned publish on a browse session. - MqttBrowseSession serves Group -> EdgeNode -> [Device] -> Metric in Sparkplug mode, decoded from observed NBIRTH/DBIRTH certificates (the only Sparkplug messages that name their metrics). Node-level metrics hang directly under their edge node -- no synthesised device folder, because the authored address tuple genuinely has deviceId = null for them. Metric node ids use a '::' separator: a metric name may contain '/', so slash-joining would make a node-level metric indistinguishable from a device folder. - AttributesAsync is purely birth-derived in Sparkplug mode. The plain path's UTF-8 inference / payload-snippet machinery does not run and no payload bytes are retained: a Sparkplug body is protobuf and would be reported as "(N bytes, binary)" for every metric in the plant. A datatype ToDriverDataType cannot map is reported as the Sparkplug type name, never coerced. - RequestRebirthAsync(scope) is the one publishing member, reached only by an explicit operator action. It routes through the counted PublishAsync chokepoint via a private RebirthTransport adapter, so PublishCountForTest still proves that OpenAsync/RootAsync/ExpandAsync/AttributesAsync/Observe/ DisposeAsync publish nothing -- now in both modes. A device or metric scope resolves up to its owning edge node (NCMD is node-addressed); group scope enumerates observed edge nodes and is refused whole past MaxGroupRebirthNodes = 32. - BrowserSessionService.RequestRebirthAsync gates on the same DriverOperator policy that gates the picker, evaluated server-side where the action happens rather than only at render time, fails closed, and Info-logs the scope and the published count. Exposed through the new IRebirthCapableBrowseSession contract so "does this session write?" stays a type question. - ToBrowseOptions' Last-Will landmine re-verified: MqttDriverOptions still carries nothing will-shaped (an NDEATH is a broker-published will and would be invisible to PublishCountForTest), now pinned by a reflection guard. Falsifiability: injecting a publish into RootAsync reddens the Sparkplug and plain passivity tests; double-publishing per target reddens all four one-NCMD-per-node assertions. 572/572 MQTT tests, 24/24 AdminUI browsing tests. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
6d7a458c4d |
fix(mqtt): reset Sparkplug ingest state at every session/authoring seam
Task 21 review — two Criticals sharing one root cause, plus four follow-ups.
C1/C2 root cause: `OnReconnectedAsync` opened with `Births.Clear(); _trackers.Clear();`
— "not housekeeping, the correctness step" — and that reset was missing from the
other two seams where the view underneath a surviving cache changes.
- C1: the ingestor OUTLIVES a session rebuild. `SessionIdentity` is a strict
superset of `IngestIdentity`, so changing only Host/Port/ClientId/credentials/
TLS/a timeout gives `!SameSession && SameIngest` — `ReinitializeAsync` tears the
session down and re-establishes on the same instance, and `TeardownAsync` touches
no ingest state (the resilience layer re-running `InitializeAsync` is the second
path). A node that rebound alias 5 during the changeover would publish Pressure's
value under the Temperature RawPath at Good. Hoisted into `ResetSessionState`,
called from `AttachTo` (earliest seam — a persistent session can push between
CONNACK and our SUBSCRIBE), `EstablishAsync` and `OnReconnectedAsync`.
- C2: `Register` pruned trackers but not births. While a node is unauthored its
traffic is dropped at `Dispatch`, so its cached birth FREEZES rather than
refreshing; drop-then-re-add across a week mis-routes at Good, and `Births` grew
monotonically. Now evicted on edge-node departure.
Deliberately NARROWER than "absent from ByScope": a device with no authored tags
under a still-authored node keeps receiving traffic, so its birth is live, not
frozen — evicting it would discard correct state. Pinned both ways.
- I1: a birth whose seq is unreadable produced a permanent debounce-paced rebirth
loop. "A birth arrived" is now tracked separately from "the birth established a
baseline". Writing the test showed the cap belonged wider than the review scoped
it: data-before-birth and unknown-alias loop identically (20 messages → 23 NCMDs
against a cap of 3), so all three missing-metadata reasons share one per-node
consecutive cap, re-armed by any applied birth. A seq gap stays uncapped — it
self-limits.
- I2: the oversize `WarnOnce` key was the raw topic, off a group-wide `#`
subscription. `HandleMessage` now parses and filters to authored nodes BEFORE the
size check (also skipping the protobuf parse for discarded traffic), and `_warned`
carries a hard ceiling so a future bad derivation degrades to silence.
- I3: array metrics slipped the unsupported-type gate — `ToDriverDataType` returns
the ELEMENT type, so a String-typed tag published a packed `byte[]` as base64 at
Good. Gated on `IsSparkplugArray`.
- I4: driver-level Sparkplug coverage for the composition-root lines T22 left
untested — the `OnDataChange` re-raise, `ReadAsync` off the shared cache,
subscribe/unsubscribe dispatch, and both directions of the mode gate.
Minors: M1 the oversize test fed unparseable bytes and passed with the size check
deleted (now a real NBIRTH + an at-the-ceiling control); M2 answered in-place (an
absent seq is refused WITHOUT becoming the baseline, so the NCMD assertion is the
complete check); M3 `HostStateObserved` raise wrapped; M5 the legacy `STATE/{host}`
form is now subscribed, so the parser's tolerance is reachable in production; M8
`ResolveBinding` prefers the NAME over a disagreeing alias; M9 a narrowing float
conversion refuses instead of publishing ±Infinity at Good (a publisher's own
infinity still passes through). M4 was already resolved by T22.
Also fixed a race in the new tests: `publish.Topics.Clear()` after a call that
dispatches rebirths off-thread must drain first — it made the C1 sequence-baseline
pin pass vacuously in the first falsifiability run.
Falsifiability: dropping the reset from AttachTo+EstablishAsync reddens 4 (C1);
dropping the Register eviction reddens 2 (C2); widening the eviction predicate to
the reviewer's literal phrasing reddens the keep-live-births pin (C2b). 545/545 MQTT
unit tests green (was 524); whole-solution `--no-incremental` build clean under TWAE.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
|
||
|
Joseph Doherty
|
cd0157a3b8 |
feat(mqtt): typed tag editor Sparkplug mode + validation
Task 24 (P2): replaces the P1 Sparkplug Validate() stub (always null) and the "not available yet" editor placeholder with real groupId/edgeNodeId/deviceId/ metricName authoring, mirroring MqttTagDefinitionFactory.FromSparkplugTagConfig in both directions -- required ids + optional deviceId, dataType/qos read strictly but only when present (dataType is optional: the birth certificate declares it). One rule is deliberately stricter than the factory: group/edge- node/device ids reject '/', '+', '#' (topic-segment characters a decoded incoming id can never carry, so an authored one that does can never bind); metricName is exempt since Sparkplug's own canonical names use '/' (e.g. "Node Control/Rebirth"). No FullName key is written -- TagConfig carries no identity under v3, and the plan's snippet asking for one was corrected per the brief. A SparkplugB tag now survives save/reopen because its descriptor keys re-infer Mode on load; there was never a 'mode' key to persist. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
6917d8805d |
feat(mqtt): Sparkplug UntilStable discovery + rediscover-on-DBIRTH
RediscoverPolicy is now mode-dependent: UntilStable in Sparkplug B (a tag's dataType is optional -- the birth certificate declares it, so the discovered tree fills in asynchronously), Once in Plain (nothing about the authored set arrives on the wire). DiscoverAsync resolves each Sparkplug tag's datatype per pass from the live BirthCache, with the ingest path's own precedence: authored dataType wins, else the birth's, else the record default. An unsupported Sparkplug type (DataSet/Template/PropertySet/Unknown) falls back rather than blanking the tag. The authored tag SET is never conditional on a birth -- an authored tag is part of the declared configuration, not something the plant grants by publishing. SparkplugIngestor.BirthObserved is wired to RaiseRediscoveryNeeded behind a two-part change gate, which is the anti-storm mechanism rather than an optimisation: fire only for a scope an authored tag binds, and only when the birth's metric-name SET (ordered-distinct, ordinal) differs from the last one seen for that scope. Edge nodes re-birth freely -- on their own timer, on every reconnect via the late-join rebirth fan-out, and once per rebirth NCMD the gap policy sends -- and firing on each of those would make a healthy plant a permanent address-space-rebuild loop. No extra debounce: what survives the gate is a real edit to the served tree, and delaying it would need its own trailing-edge flush to avoid dropping the last change. The signature map is NOT cleared on death/reconnect/ingest rebuild (the ingestor drops its whole birth cache on reconnect, but "the driver forgot" is not "the address space changed"); it is pruned when a redeploy stops authoring a scope. ScopeHint is the "Mqtt" discovery folder, deliberately not the Sparkplug scope path the plan named: the discovered tree is flat, so an edge-node/device path would name a subtree that does not exist and a consumer scoping a rebuild on it would rebuild nothing. The scope is carried in Reason instead. Falsifiability: always-fire reddens the identical-rebirth + reordered-rebirth pins; always-authored-datatype reddens the birth-fill pin; dropping the authored-scope filter reddens the unauthored-device pin. 524/524 MQTT unit tests green (was 513). Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
8538fb798b |
feat(mqtt): Sparkplug ingest state machine (birth/alias/seq/rebirth/death→stale)
Task 21 — the design doc's §3.6 correctness core. `SparkplugIngestor` owns the
authored `(group, node, device?, metric) → RawPath` index, the live `BirthCache`,
one `SequenceTracker` per authored edge node, and the rebirth policy; it routes
NBIRTH/DBIRTH/NDATA/DDATA/NDEATH/DDEATH/STATE into `OnDataChange` +
`LastValueCache`, or into a bounded, per-node-debounced rebirth NCMD.
Invariants held (and each pinned by a falsifiability control):
- Published reference is the RawPath, never a composed Sparkplug reference —
`DriverHostActor`'s dual-namespace fan-out is RawPath-keyed.
- Tags bind by stable metric NAME; the alias is only the per-birth lookup, so an
alias reused across a rebirth cannot mis-route.
- Every value goes through `SparkplugMetricBinding.Reinterpret` before publish.
- NDEATH never reaches `SequenceTracker.Accept` (it carries no seq); it is tied to
its birth by bdSeq instead.
- An invalid payload mutates nothing — no tracker, no alias table, no eviction.
- `HandleMessage` never throws on MQTTnet's dispatcher thread.
Supporting changes:
- `MqttTagDefinitionFactory.FromSparkplugTagConfig` — the driver had no way to
parse the Sparkplug descriptor keys at all; the P1 stub fields on
`MqttTagDefinition` were never populated. Mode selects the parser, never a
blob heuristic. `dataType` becomes optional in the Sparkplug shape (the birth
declares it), recorded via the new `DataTypeAuthored` flag.
- `MqttConnection` implements `IMqttPublishTransport` (bounded, refusal throws).
- `MqttDriver` dispatches every capability by mode, subscribes `spBv1.0/{group}/#`
(+ the configured STATE topic) once at connect, and re-subscribes + requests a
late-join rebirth on reconnect. `IngestIdentity` now names `MqttSparkplugOptions`,
closing the silent same-ingest gap its own ⚠️ comment warned about.
Primary-host STATE *publishing* is explicitly out of scope and warned about at
construction: it needs the retained-OFFLINE Last Will set at CONNECT time, and
shipping the ONLINE half alone is worse than shipping neither.
58 new tests (MQTT suite 455 → 513, all green).
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
|
||
|
Joseph Doherty
|
31a98a1bf5 |
feat(mqtt): RebirthRequester NCMD encode
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
3fdf24659f |
fix(mqtt): AcceptBirth is NBIRTH-only -- a DBIRTH routed there wipes bdSeq
Self-review of the commit before this one caught a defect in the contract it published rather than in the code it ran. AcceptBirth's doc said it records "a birth (NBIRTH/DBIRTH)". That is wrong on both halves of this type, and the wrong version plants exactly the defect the bdSeq tie exists to prevent. Only the NBIRTH restarts the Sparkplug sequence; a DBIRTH carries the next number in the edge node's ongoing stream, so it belongs to Accept. And only the NBIRTH and NDEATH carry bdSeq at all -- a DBIRTH has none to pass. So a Task 21 that followed the old doc and routed a DBIRTH to AcceptBirth would do two wrong things at once: rebase the sequence mid-stream, and wipe the node's session token to null. A stale Last Will arriving afterwards would then find nothing to compare against, fall through the deliberate fail-toward-stale rule in IsDeathForCurrentSession, and kill the live node. Renamed AcceptBirth -> AcceptNodeBirth so the DBIRTH call site reads wrong at a glance rather than only in prose, documented the hazard on both methods, and added DeviceBirth_ContinuesTheNodeSequence_AndLeavesTheSessionTokenAlone to pin the shape at this type's own boundary -- the ingest state machine now has something to fail against if it wires the call sites the other way round. Falsifiability: mutating Accept to clear _birthBdSeq reddens exactly the new test (1 RED), reverted. 29 tests, 438/438 for the MQTT suite. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
e4e313cf1c |
feat(mqtt): SequenceTracker seq-gap + bdSeq death-tie
SequenceTracker (Driver.Mqtt/Sparkplug/) holds one edge node's Sparkplug stream-continuity state -- design doc SS3.6 invariant #3. Two independent halves: the wrapping 0-255 seq counter that reports whether a message was missed, and the bdSeq session token that ties an NDEATH to the NBIRTH it belongs to. Next-expected is (last + 1) & 0xFF, so 255 -> 0 is the sequence continuing. The boundary is the whole task because it fails differently in each direction: read the wrap as a gap and every edge node is asked to rebirth once per 256 messages, so a busy node spends its life republishing births instead of data; miss a real gap and the driver serves values it knows are behind the device, at Good quality, indefinitely. Both directions are asserted, plus two full wrap cycles so an off-by-one that only misfires at one value has nowhere to hide. A gap is reported once and the tracker then resynchronizes onto the observed value -- holding the baseline at the pre-gap number would make every following message report a gap too, so one lost message would become a permanent rebirth storm, the same pathology as a wrong wrap reached by a different route. An out-of-range seq is refused, never masked. The codec hands seq over as a ulong precisely so a spec-violating publisher's value is not truncated into a plausible-looking one, and masking would finish the job it declined to do: with the baseline at 255, a seq of 256 masks to 0 -- exactly what the wrap expects next -- and the bogus message would be accepted as contiguous. Refused values do not become the baseline either, so the next genuine message is still measured against the last credible one. A birth restarts the sequence via AcceptBirth rather than Accept, so it is never itself a gap; a gap after a birth is still detected. An in-range but non-zero birth seq is adopted rather than refused -- refusing it would demand a fresh rebirth for every message from an otherwise-followable publisher. The first message on a virgin tracker cannot be a gap (nothing to measure it against) but leaves IsBirthSynchronized false, which is how Task 21 tells a mid-stream late join from a synchronized node. bdSeq exists to stop a late Last Will from killing a live node: a node's connection drops, it reconnects and publishes a fresh NBIRTH, and only then does the broker notice the old connection died and deliver its will carrying the PREVIOUS session's bdSeq. Without the compare that stale NDEATH drives every tag of a freshly-born healthy node to Bad. Unknowns fail toward stale deliberately -- only a positive mismatch of two known tokens discards a death, because acting on a death wrongly self-corrects at the next birth (invariant #4) while ignoring a real one leaves a dead node's values flowing Good forever. bdSeq is not a top-level field, so TryReadBdSeq extracts it from the metric named bdSeq and runs it through ReinterpretSigned first: read raw, a negative Int64 bdSeq becomes an enormous ulong -- a plausible-looking session token minted out of nonsense. Falsifiability verified, each mutation reverted after observing RED: (a) no-wrap expected -> 2 RED; (b) Accept always true -> 5 RED; (c) bdSeq compare always matches -> 2 RED; (d) mask out-of-range seq into range -> 2 RED. 28 tests, 437/437 for the MQTT suite. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
2afef00eaa |
feat(mqtt): AliasTable/BirthCache — bind-by-name, rebuild-per-birth
Design §3.6 invariants #1/#2. A Sparkplug alias is scoped to ONE birth: after a rebirth an edge node may point alias 5 at a different metric. So authored tags bind by the stable metric NAME and the alias is a per-birth cache only — RebuildFromBirth builds both indexes fresh and publishes them in one assignment, never merging or upserting, including for an empty birth (which legitimately clears the scope). BirthCache keys scopes by the full (group, edgeNode, device?) triple and applies the spec's node→device rules: an NBIRTH invalidates every DBIRTH under that node (returning their metric sets for the STALE fan-out), a death evicts rather than blanks so data-before-rebirth stays detectable. Reads are lock-free on MQTTnet's dispatcher thread — an immutable snapshot swapped atomically per scope, the same discipline MqttSubscriptionManager.AuthoredTable uses, with rebuilds landing in place so a held table reference always observes the newest birth. Running values are deliberately NOT stored here — LastValueCache owns those, keyed by RawPath. SparkplugMetricBinding.Reinterpret exposes the birth datatype to the codec's two's-complement fix, since a DATA metric carries no datatype. Falsifiability: mutating RebuildFromBirth to merge reddened 5/19 tests (both plan invariants); making the alias carry metric identity across a birth reddened 3/19, including the headline alias-reuse case. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
b245f380b1 |
feat(mqtt): Sparkplug topic parse/format + datatype map
SparkplugTopic (Driver.Mqtt/Sparkplug/) parses/formats spBv1.0 topics for
every message type (NBIRTH/DBIRTH/NDATA/DDATA/NDEATH/DDEATH/NCMD/DCMD/STATE).
TryParse never throws -- it is fed every topic a live spBv1.0/{group}/#
subscription delivers -- and rejects device/node-scope mismatches and MQTT
wildcard chars in a segment. STATE is handled honestly rather than force-fit
into the {group}/{type}/{node} mould: it parses both the v3.0
spBv1.0/STATE/{hostId} form and the legacy no-prefix STATE/{hostId} form
(tolerated on receive only -- Format/FormatState always emit v3.0). Format
gives Task 20's NCMD/DCMD write path a builder instead of hand-concatenation.
SparkplugDataType is a `global using` alias for the vendored proto's
generated Org.Eclipse.Tahu.Protobuf.DataType, not a second hand-duplicated
enum -- Metric.Datatype is a raw wire uint32 (no enum-typed field forces a
second CLR type to exist), and SparkplugCodec (Task 16, landed concurrently)
already casts straight to the generated type. A duplicate enum would be the
same enum-drift hazard this repo already names systemic (CLAUDE.md's driver
enum-serialization bug) and would force every downstream task to cast
between two value-compatible-but-nominally-different enums. ToDriverDataType()
maps per design doc SS3.5: Int8/UInt8 widen to Int16/UInt16 (no 8-bit
DriverDataType member), Float/Double to Float32/Float64 (there is no
DriverDataType.Double), Text/UUID/Bytes/File to String, *Array variants to
their element type (IsSparkplugArray carries the array bit separately), and
DataSet/Template/PropertySet/PropertySetList/Unknown return null -- an
explicit "unsupported, caller must skip+warn" rather than a guessed String.
The completeness test enumerates the live generated DataType member set
(via the alias) and asserts every member is mapped or on the explicit
unsupported list, so a future Tahu proto change is caught automatically
instead of silently falling through a stale duplicate enum's default.
Falsifiability verified by hand for three defect shapes (each reverted after
observing RED): wrong Int8 widening, a dropped mapping falling through
undetected, and inverted node/device topic scoping.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
|
||
|
Joseph Doherty
|
2589774480 |
feat(mqtt): SparkplugCodec decode + golden payload vectors
Decodes Sparkplug-B wire bytes into a driver-side projection (SparkplugPayload /
SparkplugMetric / SparkplugValueKind) for the Task 21 ingest state machine.
- Never throws, for ANY input. It sits on MQTTnet's shared dispatcher thread, so
an escaping exception would stall delivery for every subscription on the
connection, not one tag. Garbage, truncation, zero-length, a valid protobuf of
another schema and an over-nested Template all resolve to a verdict.
- Zero-length input is INVALID, not "a payload with no metrics" — protobuf would
parse it as all-defaults, and that is how a truncated-to-nothing body gets
mistaken for a well-formed one.
- Explicit presence throughout (Has{Seq,Name,Alias,Datatype,Timestamp}), never a
zero-check: an NBIRTH legitimately carries seq = 0, and every DATA metric after
a birth carries an alias with no name and no datatype.
- Values are projected RAW, boxed, with the value oneof reported as an explicit
ValueKind — Absent / Null / Scalar / Unsupported all mean different things and
three of them carry a null value. DataSet/Template/extension decode as
Unsupported (v1 scope) rather than throwing or silently vanishing.
- ReinterpretSigned() undoes Sparkplug's two's-complement-in-an-unsigned-field
encoding of Int8/16/32/64. Kept out of decode because a DATA metric carries no
datatype — only the consumer, holding the birth's alias table, knows which
applies. Skipping it publishes 4294967254 for a tag whose value is -42.
- Datatype is carried as the generated Org.Eclipse.Tahu.Protobuf.DataType; the
SparkplugDataType map is Task 17's and is applied downstream (Task 21).
Golden vectors: nbirth.bin (seq=0, named/aliased catalog, a negative Int32, an
is_null metric, a DataSet metric) + ndata.bin (alias-only metrics, no name, no
datatype) are hand-built by SparkplugGoldenPayloads, committed, and pinned by a
drift guard that rebuilds and byte-compares them on every run — plus a test that
they actually reach the output directory, since a missing copy item is invisible
in source.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
|
||
|
Joseph Doherty
|
1ae26675ac |
feat(mqtt): vendor Tahu sparkplug_b.proto + Grpc.Tools codegen
Vendors the NORMATIVE (proto2) Eclipse Tahu sparkplug_b.proto and compiles it
in .Contracts with build-time Grpc.Tools, message-only (GrpcServices="None") --
Sparkplug rides MQTT, there is no gRPC service, so no Grpc.Core.Api is pulled in.
Provenance rides in the file header: repo, path, pinned commit
5736e404889d4b95910613040a99ba79589ffb13, permalink, git blob SHA-1
bf72ab5f..., content SHA-256 4432c5c4..., EPL-2.0, and a re-verify recipe. The
body below line 38 is byte-for-byte upstream; the blob SHA-1 matches the tree
entry at that commit, so the copy is provably genuine and not reconstructed.
Chose the proto2 file over upstream's sibling sparkplug_b_c_sharp.proto: the
sibling is a lossy proto3 restatement that drops explicit presence, and protoc
generates valid C# from proto2 into the identical namespace
(Org.Eclipse.Tahu.Protobuf, PascalCased from the package -- no
csharp_namespace option). Presence matters downstream: an NBIRTH legitimately
carries seq = 0 and a DATA metric legitimately omits `name`, so Has{Seq,Name,
Alias,IsNull,Datatype} is the difference between "absent" and "present, zero".
.Contracts stays transport-free: Google.Protobuf is a serialization dependency
with a framework-only graph, Grpc.Tools is PrivateAssets=all, and the resolved
graph is exactly {Google.Protobuf, Grpc.Tools, Core.Abstractions} -- no MQTTnet.
Codegen tests pin the round-trip, the namespace, the Sparkplug field numbers,
the DataType spec indices, and protoc's enum-name mangling (UInt64 -> Uint64,
UUID -> Uuid) that Task 17's datatype map has to spell correctly.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
|
||
|
Joseph Doherty
|
92ba120964 |
feat(mqtt): MqttDriverForm + MqttDeviceForm — the driver/device config forms P1 omitted
The P1 live gate had to insert MQTT driver config directly via SQL: DriverConfigModal and DeviceModal both fell through to "No typed config form for driver type Mqtt" with no raw-JSON fallback, so broker host/port/TLS/credentials were unauthorable from the AdminUI. Task 12 built the *tag* editor; these are the driver + device surfaces. MqttDriverForm authors the whole MqttDriverOptions connection surface (host/port/clientId, TLS + CA pin, credentials, protocol version/clean-session/keep-alive, connect timeout, reconnect backoffs, mode, maxPayloadBytes, and the Plain sub-object). The Sparkplug sub-object stays P2 — a marked placeholder mirroring MqttTagConfigEditor's stub, with any existing sparkplug keys preserved untouched. The connection is authored on the DRIVER, not the device — unlike Modbus/S7/OpcUaClient and contrary to what docs/drivers/Mqtt.md said. DriverDeviceConfigMerger merges a device's keys up only when the driver has exactly ONE device, so a device-authored broker connection vanishes silently the moment a second device is added. MqttDeviceForm is therefore informational (the GalaxyDeviceForm shape) and round-trips DeviceConfig verbatim; it flags legacy connection keys left on a device by a pre-form deployment, because those still win via merge-up. Serialization goes through the ONE shared MqttJson.Options instance (Task 9's decision), never a fifth per-form copy — pinned by an assertion that an ordinal-only reader CANNOT bind the emitted blob, and by extending the fleet-wide DriverPageJsonConverterTests guard to resolve a form's serializer from an explicit external-instance registry when it has no _jsonOpts field. Dropping the converter from MqttJson.Options reddens 3 tests and leaves the symmetric round-trip green — the Task 9 lesson, reproduced. RawTags is stripped from the emitted blob (the deploy artifact owns it) and unknown top-level keys survive a load->save. Validation mirrors the driver's own [Range] bounds inline, and ToOptions() additionally clamps, so an ignored error still cannot persist a connectTimeoutSeconds:0 driver-brick. A non-blank clientId raises the P1 live-gate warning inline (fixed ids make redundant pair nodes evict each other while both report Healthy). AdminUI 761/761 green (was 730), TWAE-clean; Driver.Mqtt 266/266 green. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
b295b24419 |
feat(adminui): author + configure the Sql driver in /raw
The read-only Sql driver's runtime factory/probe and tag-side editors were wired, but the AdminUI could not author the DRIVER: the /raw "New driver" type list, the DriverConfigModal switch, and the legacy identity dropdown all omitted Sql, and no SqlDriverForm existed. Live-driving the AdminUI surfaced this. - Add SqlDriverForm.razor over the SqlDriverConfigDto shape: Provider (SqlServer-only in v1), required connectionStringRef (an env-var NAME, not a connection string — label is explicit), optional poll/operation/command timeouts, maxConcurrentGroups, nullIsBad. Enums serialize by NAME via JsonStringEnumConverter; a connection-string literal can never be authored (no such field is collected); rawTags (composer-owned) + allowWrites (inert, read-only v1) are never emitted. - Wire Sql into DriverConfigModal switch, RawDriverTypeDialog type list, and DriverIdentitySection legacy dropdown. - Add SqlDriverFormContractTests: the exact JSON the form emits constructs a driver through the real factory, missing connectionStringRef is rejected, and provider round-trips as the "SqlServer" name (never an ordinal). Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
07a4a5ff5e |
docs(mqtt): P1 plain-MQTT milestone live-verified; endpoint recorded
Task 14 — the P1 milestone gate. Ran the live /run verification on a docker-dev
rig against the real Mosquitto TLS+auth fixture, and recorded the result.
The gate did its job: it found MQTT was unauthorable in production.
RawDriverTypeDialog's driver-type list is a hand-maintained array that nobody
added MQTT to, so the /raw "New driver" picker never offered it — with every
other layer (contracts, driver, browser, factory, host registration, typed tag
editor) complete and 266 green unit tests. Nothing tied that array to
DriverTypeNames and AdminUI has no bUnit, so no offline test could see it.
Fixed, plus RawDriverTypeDialogParityTests: a reflection parity guard that
fails for ANY DriverTypeNames entry missing from the picker. Verified
falsifiable — RED with "…cannot be authored from the /raw New-driver picker,
so they are unreachable in production: Mqtt" before the one-line fix.
A second gap is left OPEN and documented, not fixed (no task ever covered it):
MqttDriverForm / MqttDeviceForm do not exist, and DriverConfigModal/DeviceModal
have no raw-JSON fallback — so an operator can create an MQTT driver but cannot
author its broker endpoint or credentials. P1 is not operator-complete until
those two razor forms land. The gate authored config via SQL to get past it.
Live gate result (isolated 2-node MAIN stack, image built from this branch,
fixture at 10.100.0.35:8883, AllowUntrustedServerCertificate rather than
pinning the CA into the rig):
- typed MQTT tag editor renders and round-trips; Json shows the JSON-path
field, Raw/Scalar hide it; wildcard topic a/+/c blocked at save; data-type
list offers Float64 and no Double; qos/retainSeed absent on "(driver
default)" and qos:2 written as a number; isHistorized/historianTagname
survive a topic-only edit; SparkplugB renders the P2 placeholder and
switching back keeps Plain values; jsonPath is guidance not a gate in all
three shapes ($ seeded only for a brand-new blank tag, blank stays absent,
clearing saves fine)
- deployed, and both editor-written blobs resolved and served changing live
values through OPC UA (22.8 / 1629, StatusCode Good) — no BadNodeIdUnknown
- the bespoke #-observation browser drove a real broker session and rendered
the fixture's actual topic tree (line1/{counter,state,temperature},
noise/chatter, retained/seed); a Modbus device's picker still correctly
reports "Browsing unavailable"
Also found live and documented (not a code change): both nodes of a redundant
pair run the driver, so a FIXED clientId makes them evict each other forever —
the broker logs "already connected, closing old connection" and both nodes
reconnect every ~2s while still reporting Healthy. Unset (the default) is
correct; confirmed 0 reconnects/60s on both nodes after removing it.
Suites: offline 1134 passed / 0 failed (266 Driver.Mqtt + 730 AdminUI incl. the
3 new guards + 138 Core.Abstractions); live 7/7 against the fixture.
Docs: new docs/drivers/Mqtt.md + Mqtt-Test-Fixture.md (the per-driver + fixture
convention every other driver follows), MQTT rows in docs/drivers/README.md,
TestConnectProbes.md, root README.md, and infra/README.md §3.
CLAUDE.md drift corrected while adding the MQTT endpoints:
- the "every fixture carries project: lmxopcua" claim was false — only the
MQTT fixture does; the filter returns one stack, not the fleet
- lmxopcua-fix.ps1 is Windows-VM-only; on macOS drive the host over ssh/rsync
(and the docker-dev rig itself runs locally under OrbStack)
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
|
||
|
Joseph Doherty
|
2409d05a28 |
fix(mqtt): a broker-refused CONNECT no longer reports a Healthy driver
MQTTnet 5 does not throw on an unsuccessful CONNACK — it returns the reason in MqttClientConnectResult.ResultCode and v5 removed v4's ThrowOnNonSuccessfulConnectResponse, so ConnectCoreAsync's unconditional SetState(Connected) sealed a wrong-password deployment green: DriverState.Healthy / HostState.Running on a session that never authenticated. Caught by the Task-13 Mosquitto fixture; 240 offline tests missed it. ConnectCoreAsync now inspects the CONNACK and raises MqttConnectRejectedException. Credentials/identity/protocol/Last-Will refusals are unrecoverable and set Faulted, which stops the reconnect supervisor; transient refusals (ServerUnavailable, ServerBusy, QuotaExceeded, UseAnotherServer, ConnectionRateExceeded, and anything unrecognised) stay Reconnecting under backoff. MqttDriverProbe now shares the rejection wording, so the AdminUI Test-connect button and the running driver can no longer disagree. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
033b3700c4 |
test(mqtt): Mosquitto TLS+auth fixture + env-gated plain live suite
Adds the MQTT driver integration-test project: an eclipse-mosquitto fixture running auth AND TLS (allow_anonymous false on both listeners), a mosquitto_pub-based JSON publisher emitting retained messages, a cert/password generator script whose output is gitignored, and a live suite gated on MQTT_FIXTURE_ENDPOINT that skips clean offline. The fixture is never anonymous by design: an anonymous broker would let the driver's TLS/CA-pin and auth paths go untested, and those fail silently. The CA-pin leg carries its own falsifiability control (a foreign CA generated in-process must be rejected). Live gate on 10.100.0.35: 7/7 passed. It found a driver defect, reported not fixed here (src/ is out of this task's scope): MqttConnection.ConnectAsync RETURNS NORMALLY when Mosquitto rejects a CONNECT as not-authorized -- MQTTnet 5 does not throw on an unsuccessful CONNACK, so a wrong broker password yields DriverState.Healthy from InitializeAsync. The auth-negative test therefore asserts the invariant that holds either way (no session is ever established) and documents the finding. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
ff62b62a55 |
refactor(mqtt): accept a blank jsonPath; guide with a seeded "$" instead
Review follow-up. Dropped the hard Validate rejection of a blank jsonPath under a Json payload: the runtime defaults it to the document root, which is the real "publisher puts a bare JSON scalar on the topic" case, so rejecting it made the editor refuse what the driver accepts — inverting "authoring surface accepts <=> publish accepts" — and blocked whole CSV-import batches, since this validator also gates RawManualTagEntryModal's review grid. Replaced with guidance: an UNAUTHORED tag (no topic AND no jsonPath) seeds the field with "$", so a fresh tag emits an explicit "jsonPath":"$" while an existing path-less tag keeps the key ABSENT and the driver defaults it. The wildcard-topic rule stays strict — a wildcard has no legitimate single-Tag interpretation. Also: omit a blank "topic" rather than writing "topic":"", and record why the _lastConfigJson guard diverges from the Modbus template — the landmine is a DERIVED, NON-PERSISTED UI FIELD INFERRED FROM BLOB CONTENT (Mode), which Task 24's Sparkplug field group will be tempted to add more of. Named the host-side dependency (RawTagModal only mutates through this callback) that keeps the staleness trade benign today. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
a13ae926a8 |
fix(mqtt): gate DisposeAsync against in-flight lifecycle ops (C1) + rebuild-branch tests (I1)
C1 (Critical) — DisposeAsync went straight to TeardownAsync with no lifecycle-gate
wait, reopening the orphaned-connection class. Interleaving: ReinitializeAsync's
rebuild branch holds the gate mid-InitializeCoreAsync, past its own teardown but
before `_connection = connection`; an ungated dispose sees a null connection, does
nothing, and sets `_disposed`; the rebuild then publishes a live connection plus a
host-probe loop that every later dispose short-circuits past. Not proven reachable
today (DriverInstanceActor.PostStop calls the gated ShutdownAsync), but MqttDriver
publicly implements IAsyncDisposable, which invites `await using`.
- DisposeAsync now takes the gate with the same bounded wait + fallback teardown
ShutdownAsync uses, and sets `_disposed` BEFORE the wait.
- InitializeCoreAsync re-checks `_disposed` after the connect and disposes the
connection it just built rather than publishing it — this closes the residual
window on the bounded-timeout fallback path.
- The doc comment no longer claims parity with ShutdownAsync it did not have, and
stops conflating "don't Dispose() the semaphore object" with "don't WaitAsync".
I1 (Important) — the SameSession == false rebuild branch had no test driving it.
Adds three: an endpoint-changing delta rebuilds and Faults on a refused connect;
an ingest-only delta (MaxPayloadBytes) ALSO rebuilds, pinning that IngestIdentity
is nested inside SessionIdentity; and DisposeAsync serializes behind a lifecycle
operation parked at a new internal BeforeConnectHookForTests seam (mirroring
MqttConnection's AfterConnectHookForTests, which exists for the same race class).
Minors: comments recording that IngestIdentity names no Sparkplug field and must
grow one in P2 (tasks 21/22), and why _options/_subscriptions/_authoredRawPaths
get looser memory discipline than _health/_hostState.
Falsifiability: reverting DisposeAsync to the ungated form reddens exactly the new
serialization test ("Shouldly.ShouldAssertException : raced"); forcing SameSession
to always-true reddens exactly the two new rebuild tests. 240/240 MQTT tests pass;
forced rebuild of the driver project is 0 warnings under TreatWarningsAsErrors.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
|
||
|
Joseph Doherty
|
f79d13e2d8 |
feat(mqtt): factory + DriverTypeNames.Mqtt + host factory/probe registration
Task 9. Adds MqttDriverFactoryExtensions (DriverTypeName = DriverTypeNames.Mqtt, direct deserialization of MqttDriverOptions — no intermediate DTO, mirroring OpcUaClientDriverFactoryExtensions), the DriverTypeNames.Mqtt constant, and both host wiring sites: the factory in DriverFactoryBootstrap.Register and the probe in AddOtOpcUaDriverProbes (the admin-node path Program.cs calls in its hasAdmin block — a probe wired only on driver nodes makes Test-connect silently dead). Carried-forward items: - Converges every MQTT config-parsing seam onto ONE JsonSerializerOptions — MqttJson.Options, in .Contracts alongside MqttDriverOptions. It replaces the probe's internal JsonOpts and the browser's separate private copy; the factory, the probe, MqttDriver.ParseOptions and MqttDriverBrowser now all parse through it. .Contracts is the only assembly all four consumers reference, and the browser's reference to the runtime .Driver project is a documented layering exception scheduled for removal — anchoring the options there would resurrect the duplicate the day it goes away. - Replaces the "Mqtt" literals in MqttDriverProbe, MqttDriverBrowser and MqttDriver with the constant (string value unchanged). - Tightens MqttDriverProbeTests.ProbeAsync_EnumAsName: it asserted only Ok == false + non-empty message, which is also exactly what a JSON-parse failure produces — so it stayed green under the very regression it names. It now asserts the probe got past the parse and reached the network. Falsifiability: deleting JsonStringEnumConverter from MqttJson.Options reddens 9 tests across 4 suites, including the tightened probe test (message becomes "Config JSON is invalid: The JSON value could not be converted to MqttProtocolVersion") — which the pre-fix assertions would have passed. Also references the MQTT driver from Core.Abstractions.Tests so DriverTypeNamesGuardTests' reflective bin scan discovers the new factory and the constant/factory parity check stays honest. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
36abd871a1 |
feat(mqtt): typed tag editor + validator (plain mode)
Thin typed model over a preserved JsonObject key bag, mirroring the sibling <Driver>TagConfigModel template. Key names and strictness rules mirror MqttTagDefinitionFactory rather than inventing an editor-side schema; enums round-trip as NAMES (the factory reads them strictly by name). No FullName key: under v3 a tag is identified by its RawPath, which the factory keys the definition's Name off — a composed identity key here would be dead weight nothing reads. Mode is a UI-only sub-shape selector inferred from the blob (any Sparkplug descriptor key present) and never serialised — the driver takes Plain vs SparkplugB from its DRIVER config. Only Plain Validate() is implemented; the Sparkplug branch is a stub Task 24 fills, and its keys are preserved untouched. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
420692b6e5 |
feat(mqtt): MqttDriver shell — lifecycle + authored-only discovery (Once)
Composes MqttConnection + MqttSubscriptionManager + LastValueCache into the IDriver / ITagDiscovery / ISubscribable / IReadable / IHostConnectivityProbe / IRediscoverable capability set. - Discovery replays ONLY the authored raw tags (RediscoverPolicy = Once, SupportsOnlineDiscovery = false) — a chatty broker can never auto-provision. - Composition order is register -> AttachTo -> ConnectAsync; the manager's Reconnected handler is passed through unwrapped so its throw-on-total-failure still tears a deaf session down. - ReinitializeAsync applies a tag-only delta in place and never faults on a bad one; a session-changing delta rebuilds. - ReadAsync serves the last-value cache and degrades per reference (BadWaitingForInitialData), never throwing for the batch. - FlushOptionalCachesAsync is a no-op: the last-value cache IS IReadable. - Adds MqttDriverOptions.RawTags (the authored-tag delivery mechanism every other driver's options DTO already has) and promotes MaxPayloadBytes from a manager ctor knob to an operator-facing key. - Converges on MqttDriverProbe.JsonOpts rather than a second, divergent JsonSerializerOptions. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
a856d61510 |
feat(mqtt): MqttDriverProbe CONNECT handshake
Test-connect probe for the Mqtt driver type: parses MqttDriverOptions
(shared enum-as-name JsonSerializerOptions), opens a bounded MQTT CONNECT
via MqttConnection.BuildClientOptions with a distinct -probe-{guid8}
client-id suffix, and classifies the outcome. Live-probed against
MQTTnet 5.2.0.1603 to confirm the real contract: a broker-rejected
CONNACK is a MqttClientConnectResult.ResultCode, never a thrown
exception, and timeout classification checks the deadline token itself
rather than switching on exception type (which varies unpredictably
between MqttConnectingFailedException and bare OperationCanceledException
for the same frozen-peer shape).
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
|