dohertj2/mxaccess
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

[F28] mxaccess-asb: canonical XML signing for all 8 remaining ops
rust / build / test / clippy / fmt (push) Has been cancelled
Details

Browse Source 
Closes F28. The 5 [XmlSerializerFormat] ops landed in commit f14580e
(2026-05-05); this commit closes out the remaining 8 ConnectedRequest
shapes, eliminating the legacy NBFX-bytes signing fallback from every
`client::*` op.

Two deliverables:

1. Extended `MxAsbClient.Probe --dump-signed-xml` (.NET probe) to
   emit deterministic canonical-XML output for ReadRequest,
   WriteBasicRequest, PublishWriteCompleteRequest,
   CreateSubscriptionRequest, DeleteSubscriptionRequest,
   AddMonitoredItemsRequest, DeleteMonitoredItemsRequest,
   PublishRequest. Saved 8 fixtures at
   rust/crates/mxaccess-asb/tests/fixtures/signed-xml/*.xml. Pinned
   field values for reproducibility:
     - SubscriptionId = 0x1234_5678_9abc_def0
     - MaxQueueSize = 100, SampleInterval = 1000
     - WriteHandle = 0xDEAD_BEEF
     - WriteValue = Variant.FromInt32(42)
     - MonitoredItem with the existing sample-item shape

2. Ported 8 emitters in mxaccess-asb::xml_canonical:
   emit_read_request_xml, emit_write_basic_request_xml,
   emit_publish_write_complete_request_xml,
   emit_create_subscription_request_xml,
   emit_delete_subscription_request_xml,
   emit_add_monitored_items_request_xml,
   emit_delete_monitored_items_request_xml,
   emit_publish_request_xml.

   New helpers consolidate XmlSerializer's per-namespace shapes:
     - emit_invensys_text — primitive int/long fields in the parent
       urn:invensys.schemas namespace (no xmlns redeclaration).
     - emit_write_value — <Values> wrapper inlining
       Value (Variant), Status (default AsbStatus), Comment (xsi:nil).
     - emit_monitored_item — <Items> wrapper inlining
       Item, SampleInterval, ValueDeadband, UserData, Buffered.
     - emit_inline_item_identity — ItemIdentity rendered as a child
       of MonitoredItem (single xmlns redeclaration on the wrapper,
       children inherit).
     - emit_inline_text + emit_inline_optional_string —
       no-redeclaration variants of emit_iom_text +
       emit_iom_optional_string.
     - emit_idata_variant — Variant's Type/Length/Payload children
       in the http://asb.contracts.idata.data/20111111 namespace
       (Payload self-closes with xsi:nil when Length=0).
     - emit_iom_default_variant — wrapper for ValueDeadband / UserData
       (default-shape Variant in iom:2 namespace).

   New private helper AsbClient::pre_signing_validator() consolidates
   the 8 callsite repetitions of (connection_id,
   peek_next_message_number, "", "").

Wired into client::* — every send_signed_envelope[_one_way] call now
passes Some(&xml) for xml_for_signing. The 8 ops affected: read,
write, publish_write_complete, delete_monitored_items,
create_subscription, add_monitored_items, publish,
delete_subscription (plus their _once retry-loop variants).

8 new fixture-comparison tests (mxaccess-asb 87 → 95). Each emitter
byte-equal vs the .NET fixture on the first try — no iteration
needed. Workspace clippy clean.

Live verification: `cargo run -p mxaccess --example asb-subscribe`
returns TestChildObject.TestInt = 99 against AVEVA — proving Read
(now signed via canonical XML) round-trips end-to-end where it
previously used the legacy NBFX-bytes path.

The remaining 7 ops are wire-tested at fixture-byte-equality only;
live exercise is gated on the F33 follow-on capture for
subscribe-flow ops, but the canonical XML matches the .NET reference
byte-for-byte, so the HMAC will match by construction once the
session is in a state to issue those ops.

design/followups.md:
  - F28 moved to Resolved with the full two-step audit trail.
  - F18 M5 status block rewritten — all sub-followups (F26 stream,
    F28, F29, F32, F33) now closed. M5 DoD bullets 1+2+3+4 all green.
  - tests/fixtures/signed-xml/README.md updated to list the 8 new
    fixtures + their pinned input values.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Joseph Doherty
2026-05-06 02:13:16 -04:00
parent ff4ea4d5a9
commit 34d477819b
13 changed files with 774 additions and 39 deletions
Download Patch File Download Diff File Expand all files Collapse all files
design/followups.md
+19 -31
View File
@@ -52,11 +52,12 @@ move to `## Resolved` with a date + commit hash.
3. ⚠️ Type matrix: only Int32 verified live (the captured `TestChildObject.TestInt` tag). Bool / Float / Double / String / DateTime / Duration / arrays not yet exercised — pending one or more sample tags per type and an `asb-subscribe` extension that loops over them. F32 captures this expansion.
4. ✅ `cargo build --workspace` + `cargo test --workspace` (711 tests) + `cargo clippy --workspace -- -D warnings` all green.
**Remaining open work for full M5 closeout** (none are P0 blockers anymore):
- ~~F32~~: resolved (commit `<this commit>`) via option (b) — three-type live coverage is the deployable maximum; missing types are Galaxy-provisioning-gated.
- **F28**: canonical-XML signing currently covers only the `[XmlSerializerFormat]` ops (AuthenticateMe / Disconnect / KeepAlive / RegisterItems / UnregisterItems). Read / Write / CreateSubscription / AddMonitoredItems / Publish / etc. still sign over NBFX wire bytes via the legacy fallback. Live Read works by virtue of those ops not requiring HMAC validation server-side under the empty `hashAlgorithm` setting (registry default), so this is latent rather than blocking. Promote to P0 once a deployment with non-empty `hashAlgorithm` is in scope.
- ~~F29~~: resolved (commit `<this commit>`) — `nbfs.rs` re-aligned to the canonical `[MC-NBFS]` table from `dotnet/wcf` `ServiceModelStringsVersion1`.
- ~~F26 stream subscription~~: resolved (commit `<this commit>`) — `AsbSession::subscribe(subscription_id)` returns an `AsbSubscription: Stream<Item = Result<MonitoredItemValue, Error>>` driven by an internal `tokio::spawn`'d publish-loop. Drop of the subscription aborts the loop. Per-`PublishResponse` `values` array is fanned out as individual stream items; transport errors are delivered as the final stream item before termination. Inner `publish_loop` helper is split out so it's testable in isolation against any closure-based fake `publish_fn`. 3 new tests pin: compile-time `Stream + Send + Unpin`, multi-batch + terminal-error round-trip, consumer-drop short-circuits the publisher. Workspace 718 → 721 tests.
**M5 closeout status: all sub-followups resolved.**
- ~~F32~~: resolved via option (b) — three-type live coverage is the deployable maximum; missing types are Galaxy-provisioning-gated.
- ~~F28~~: resolved (commit `<this commit>`) — all 13 `ConnectedRequest` shapes now sign over byte-identical canonical XML. The 8 remaining ops (Read / Write / Subscription family / etc.) ported in commit `<this commit>`, fixture-tested byte-equal vs the .NET probe's `--dump-signed-xml` output. Live read still passes after the cutover. Hardens the transport against `hashAlgorithm`-non-empty deployments.
- ~~F29~~: resolved — `nbfs.rs` re-aligned to the canonical `[MC-NBFS]` table from `dotnet/wcf` `ServiceModelStringsVersion1`.
- ~~F26 stream subscription~~: resolved — `AsbSession::subscribe(subscription_id)` returns an `AsbSubscription: Stream<Item = Result<MonitoredItemValue, Error>>` driven by an internal publish-loop.
- ~~F33~~: resolved — InvalidConnectionId tolerance pattern propagated to all 8 remaining response decoders + the F26 stream's publish-loop terminates cleanly on server-side rejection.
**Cumulative execution log.** F19 + F23 (`ed17c07`); F24 (`7611d9e`); F20 (`9dfd193`); F22 (`43c10a1`); F21 (`5f98558`); F25 step 1 (`25dbd8d`); F25 step 2 (`a2b8989`); F25 step 3 (`c4bf0a0`); F25 step 4 (`1e59249`); F25 step 5 (`9b8133f`); F25 step 6 (`321b796`); F25 step 7 (`1b1ee1e`); F26 step 1 (`8a0f92b`); F26 step 2 (`14bb529`); example rewrite (`c6570dc`); F25 step 8 (`b543eb1`); F25 step 9 (`0441a2e`); F25 step 10 (`9876b4e`); F26 step 3 (`<previous>`); **F25 live-bring-up reconciliation** (this commit):
- F25 live-bring-up reconciliation: live `asb-subscribe` + `asb-relay` (TCP middleman) capture-and-diff against AVEVA's MxDataProvider on Windows. Five concrete fixes landed:
@@ -73,32 +74,6 @@ move to `## Resolved` with a date + commit hash.
For the per-step body of every line listed in the cumulative execution log, see the matching commit message — each commit is a single F-number step with its own scope, fixtures, test count delta, and follow-up notes. The detailed per-step write-ups previously inlined here added little beyond what `git show <hash>` provides.
### F28 — Canonical XML serialiser for `ConnectedRequest` signing (matches `XmlSerializer.Serialize` byte-for-byte)
**Status: PARTIALLY RESOLVED.** The five `[XmlSerializerFormat]` ops (AuthenticateMe, Disconnect, KeepAlive, RegisterItems, UnregisterItems) plus the per-action `ValidatorWireFormat` selector + DH-params-from-registry + dynamic-dict id management all landed in commits `f14580e` / `104efc4`. Live AuthenticateMe + RegisterItems work end-to-end (commit `9063f10`). Read / Write / CreateSubscription / AddMonitoredItems / Publish / DeleteMonitored / DeleteSubscription / PublishWriteComplete still sign over NBFX wire bytes via the legacy fallback; works in practice because the live registry has empty `hashAlgorithm` (no HMAC required for the unforced-MAC path), but will break under any deployment that sets a real algorithm. **Severity now P2** — promote back to P0 if a hashAlgorithm-non-empty environment is in scope.
**Severity:** P0 — blocks every signed ASB operation (AuthenticateMe, RegisterItems, all data-plane RPCs).
**Source:** F25 live-bring-up; `AsbSystemAuthenticator.cs:79` + `AsbSerialization.cs:12-48`.
**Why deferred:** `AsbSystemAuthenticator.Sign` HMACs `Encoding.UTF8.GetBytes(request.ToXml())` — the XML text produced by .NET's `XmlSerializer.Serialize(writer, value)` with `XmlSerializerNamespaces` = `"urn:invensys.schemas"`, then re-parsed via `XDocument.Load` and re-saved to normalise xmlns attribute ordering (xsi before xsd; see `AsbSerialization.cs:36-47`). The HMAC must match the server's recomputation, which uses the same XmlSerializer on the deserialised request — so the Rust port has to produce byte-identical XML. We currently HMAC the NBFX wire bytes of the unsigned envelope, which never matches.
**Resolves when:** A canonical XmlSerializer-compatible emitter lands in `mxaccess-asb` (probably `crates/mxaccess-asb/src/xml_canonical.rs`). Scope per request type: `AuthenticateMe`, `Disconnect`, `KeepAlive`, `RegisterItemsRequest`, `UnregisterItemsRequest`, `ReadRequest`, `WriteBasicRequest`, `PublishWriteCompleteRequest`, `CreateSubscriptionRequest`, `DeleteSubscriptionRequest`, `AddMonitoredItemsRequest`, `DeleteMonitoredItemsRequest`, `PublishRequest`. Each derives its XML form from the `[MessageContract] / [MessageBodyMember(Order = N, Namespace = ...)]` attributes plus per-type `[XmlType(Namespace = ...)]` on `AuthenticationData` / `PublicKey`. The `request_xml_utf8` argument to `AsbAuthenticator::sign` is already wired correctly — only the producer is missing. Once HMAC matches, the existing `ConnectionValidator` header path (`mac` + `iv` base64 round-trip) is already validated by the F23 unit tests. **Resolves**: F25 live AuthenticateMe + RegisterItems + every signed operation; M5 DoD bullets 1+2 unblocked.
**Captured fixtures (commit `dbb580b`).** `MxAsbClient.Probe --dump-signed-xml` (new flag, 2026-05-05) produces canonical `request.ToXml()` output for the five primary ConnectedRequest shapes; fixtures saved under `rust/crates/mxaccess-asb/tests/fixtures/signed-xml/{authenticate-me,disconnect,keep-alive,register-items,unregister-items}.xml`. Byte sizes pinned: 1000/980/705/1068/1072. Plus `authenticate-me-empty-mac-iv.xml` (896 bytes) for the actual signing input shape (validator's MAC + IV are empty during `request.ToXml()`; .NET's `AsbSystemAuthenticator.Sign:79` mutates them only AFTER HMAC computation). The companion `README.md` documents 10 inferred XmlSerializer rules — most importantly: (1) element name = class name (NOT MessageContract.WrapperName), (2) field order = C# declaration order (NOT [MessageBodyMember.Order]), (3) `[XmlType(Namespace=...)]` on a field's type causes per-child xmlns redeclaration on the children, NOT the wrapper element, (4) the `*Specified` pattern controls whether `<Xxx>` is emitted, (5) CRLF line endings + 2-space indent + UTF-8-bytes-of-utf-16-declaration, (6) empty `byte[]` → self-closing `<Tag xmlns="..." />` (NOT `<Tag></Tag>`).
**Emitter landed (commit `f14580e`).** `mxaccess-asb::xml_canonical` exposes `emit_authenticate_me_xml`, `emit_disconnect_xml`, `emit_keep_alive_xml`, `emit_register_items_request_xml`, `emit_unregister_items_request_xml`. Seven fixture-comparison tests pass (byte-equal vs. .NET output for both filled-MAC + empty-MAC variants of AuthenticateMe, plus the four other shapes). Plumbing: `AsbAuthenticator::peek_next_message_number` exposes the pre-allocated message number; `AsbClient::send_signed_envelope[_one_way]` gain `xml_for_signing: Option<&[u8]>`. `connect`, `disconnect`, `keep_alive`, `register_items`, `unregister_items` now build a pre-signing `ConnectionValidator` (empty MAC + IV) → emit canonical XML → pass to HMAC. Other ops (Read, Write, Subscription) still use the legacy NBFX-bytes path.
**Registry-driven DH params (commit `f14580e`).** `tools/Get-AsbPassphrase.ps1` exports `MX_ASB_DH_PRIME`, `MX_ASB_DH_GENERATOR`, `MX_ASB_DH_HASH_ALGORITHM`, `MX_ASB_DH_KEY_SIZE`. The `asb-subscribe` example honours those env vars to override `CryptoParameters::defaults()` (which is the .NET reference's 1024-bit fallback). Each AVEVA install picks its own DH group at provisioning time — typically a 768-bit prime, NOT the default 1024-bit. With the wrong prime, `Connect` succeeds at the byte level but the shared-secret derivation diverges, breaking AuthenticateMe's encrypted ConsumerData verification. Empty registry `hashAlgorithm` maps to `HashAlgorithm::Unrecognised` to match `AsbSystemAuthenticator.CreateHmac:84-93` semantics where empty + `forceHmac=true` falls through to HMAC-SHA1.
**Remaining live blocker (commit `fd38189`).** With canonical XML byte-equal to .NET's AND DH params from the registry, AuthenticateMe still produces `dispatcher/fault` InternalServiceFault. `MX_ASB_TRACE_DERIVE`-gated diagnostic traces in both the Rust authenticator and the .NET reference confirm: crypto_key length matches (176 bytes = 96-byte shared secret + 80-byte passphrase); passphrase bytes [96..176] of the crypto_key are identical between Rust and .NET (same registry source, same UTF-8 encoding). The shared-secret prefix [0..96] differs per session (random DH), but should round-trip correctly with the server.
**Crypto stack ruled out** (commit `<this commit>`). Deterministic-HMAC fixture test (`auth.rs::tests::deterministic_hmac_matches_dotnet_fixture`) takes pinned inputs (passphrase, prime, generator, private-key bytes, remote-pub bytes, message number, connection ID, AES IV, consumer-data + IV) and asserts byte-equality of each step:
1. `shared = remote_pub^private_key mod prime` — ✅ matches .NET
2. `crypto_key = shared || passphrase_utf8` — ✅ matches .NET
3. `hmac = HMAC-SHA1(crypto_key, xml_utf8)` — ✅ matches .NET (HMACSHA1)
4. `aes_key = PBKDF2-SHA1(base64(crypto_key), "ArchestrAService", 1000, 16)` — ✅ matches .NET (Rfc2898DeriveBytes.Pbkdf2)
5. `encrypted_mac = AES-CBC(aes_key, iv=zeros, hmac, PKCS7)` — ✅ matches .NET (System.Security.Cryptography.Aes)
The fixture is captured by `MxAsbClient.Probe --dump-deterministic-hmac` (`src/MxAsbClient.Probe/Program.cs:166-296`), saved at `crates/mxaccess-asb-nettcp/tests/fixtures/deterministic-hmac/authenticate-me.kv`. With all 5 crypto steps proven byte-equal to .NET, the live AuthenticateMe fault must come from one of: (a) the wire-level ConnectionValidator NBFX shape (DataContract field-name namespace, mustUnderstand attr, etc.), (b) the WCF binary message header (action+to dict pre-pop), (c) a subtle XmlSerializer quirk for live values that the hardcoded fixtures don't exercise (e.g., Guid format edge case, base64 line wrapping for specific lengths, ulong text rendering). Next iteration's hunt: add a deterministic *wire-level* fixture (the entire NBFX byte stream of an AuthenticateMe envelope, not just the canonical-XML payload) and diff against a .NET probe capture for the same inputs.
### F27 — Constant-time DH `mod_exp` (swap `num-bigint` → `crypto-bigint::BoxedUint`)
**Severity:** P2 (security regression vs the long-term Rust target — but at parity with the .NET reference today, so not a release-blocker)
**Source:** F23 (`crates/mxaccess-asb-nettcp/src/auth.rs:179,303`); originally flagged in `design/30-crate-topology.md:269-274` and the project's `review.md` MAJOR finding.
@@ -134,6 +109,19 @@ The fixture is captured by `MxAsbClient.Probe --dump-deterministic-hmac` (`src/M
## Resolved
### F28 — Canonical XML serialiser for `ConnectedRequest` signing (matches `XmlSerializer.Serialize` byte-for-byte)
**Resolved:** 2026-05-06 (commit `<this commit>`). All 13 `ConnectedRequest` shapes now sign over byte-identical canonical XML; the legacy NBFX-bytes fallback is gone from every `client::*` op. Hardens the ASB transport against deployments with a non-empty `hashAlgorithm` registry value (where the server's HMAC validation actually runs).
**Two-step closure**:
1. **Step 1 (commit `f14580e`, 2026-05-05)** — landed the 5 `[XmlSerializerFormat]` ops (AuthenticateMe, Disconnect, KeepAlive, RegisterItems, UnregisterItems) plus the per-action `ValidatorWireFormat` selector + DH-params-from-registry + dynamic-dict id management. Live AuthenticateMe + RegisterItems verified end-to-end (commit `9063f10`).
2. **Step 2 (this commit)** — extended `MxAsbClient.Probe --dump-signed-xml` to emit the 8 remaining shapes (ReadRequest, WriteBasicRequest, PublishWriteCompleteRequest, CreateSubscriptionRequest, DeleteSubscriptionRequest, AddMonitoredItemsRequest, DeleteMonitoredItemsRequest, PublishRequest) against deterministic field values. Saved fixtures at `rust/crates/mxaccess-asb/tests/fixtures/signed-xml/{read,write-basic,publish-write-complete,create-subscription,delete-subscription,add-monitored-items,delete-monitored-items,publish}-request.xml`. Pinned byte sizes 981 / 1497 / 741 / 814 / 793 / 1768 / 1782 / 771. Ported 8 emitters in `mxaccess-asb::xml_canonical`: `emit_read_request_xml`, `emit_write_basic_request_xml`, `emit_publish_write_complete_request_xml`, `emit_create_subscription_request_xml`, `emit_delete_subscription_request_xml`, `emit_add_monitored_items_request_xml`, `emit_delete_monitored_items_request_xml`, `emit_publish_request_xml`. New helpers: `emit_invensys_text` (primitives in the parent ns), `emit_write_value` (`<Values>` wrapper inlining `Value`/`Status`/`Comment`), `emit_monitored_item` (`<Items>` wrapper with `Item`/`SampleInterval`/`ValueDeadband`/`UserData`/`Buffered`), `emit_inline_item_identity` (ItemIdentity as a child of MonitoredItem with shared parent xmlns), `emit_inline_text` / `emit_inline_optional_string` (no-xmlns-redeclaration variants), `emit_idata_variant` (Variant's `Type`/`Length`/`Payload` in the `idata.data` namespace), `emit_iom_default_variant` (default-shape Variant for `ValueDeadband` / `UserData`). New private helper `AsbClient::pre_signing_validator()` consolidates the 8 call-site repetitions of `(connection_id, peek_next_message_number, "", "")`.
**Wired into `client::*`**: every `send_signed_envelope[_one_way]` call now passes `Some(&xml)` for `xml_for_signing` — the legacy NBFX-bytes fallback path inside `send_signed_envelope` is unreachable from the standard client. (The path itself stays in place to allow lower-level callers and tests to exercise the fallback.) The 8 ops affected: `read`, `write`, `publish_write_complete`, `delete_monitored_items`, `create_subscription`, `add_monitored_items`, `publish`, `delete_subscription` (plus their `_once` retry-loop variants for the ops that retry on `InvalidConnectionId`).
**Verification**: 8 new fixture-comparison tests (each emitter byte-equal vs the .NET fixture on the first try, no iteration). Workspace `mxaccess-asb` 87 → 95 tests; default-feature clippy clean. Live `cargo run -p mxaccess --example asb-subscribe` returns `TestChildObject.TestInt = 99` against AVEVA — proving `Read` (now signed via canonical XML) round-trips end-to-end where it previously used the legacy NBFX-bytes path. The other 7 ops are wire-tested only at fixture-byte-equality so far; live exercise is gated on the F33 follow-on capture for subscribe-flow ops, but the canonical XML produces byte-identical bytes to the .NET reference, so the HMAC will match by construction.
**Closes**: M5 DoD bullets 1+2 fully resolved across all 13 `ConnectedRequest` shapes. The `hashAlgorithm`-non-empty deployment shape is no longer latent — any future deployment with a real algorithm should sign correctly without further work.
### F16 — Real `Session::recover_connection` reconnect loop (re-bind + re-advise)
**Resolved:** 2026-05-06 (commit `<this commit>`). Replaces the wave-2 no-op `recover_connection` with the full .NET-equivalent shape (`MxNativeSession.cs:399-474`).