dohertj2/lmxopcua
1
0
Fork 0
Code Issues 5 Pull Requests Actions Packages Projects Releases Wiki Activity

SDK 2 clients cannot log in to SDK 1.5 OtOpcUa servers with API keys (65-char key sent as RsaEncryptedSecret -> BadIdentityTokenInvalid): rollback/QA-compat risk #880

New Issue
Closed
opened 2026-11-12 10:11:44 -05:00 by dohertj2 · 1 comment
dohertj2 commented 2026-11-12 10:11:44 -05:00
Owner
Copy Link
Copy Source

Found by the #800 Windows service gate on windev, 2026-11-12. Both client and server ran on the same box, so clock skew is ruled out (#873 does not apply).

Rollback/QA-compat risk.

Symptom: an SDK 2.0.1 client cannot log in to an SDK 1.5.378.182 OtOpcUa server (known-good master efb9197a) with a production-format API key. Each attempt gets BadIdentityTokenInvalid at ActivateSession, in both Sign and SignAndEncrypt (Basic256Sha256), on a channel the server trusts. This was seen with both our #800 probe (SDK 2.0.1) and the actual SDK 2 Client CLI at master 438484fd.

Controls (same server, same boot, same trusted client certificate):

Client Password Result
SDK 2 probe 65-char key (otua_<16 hex>_<43>, the ApiKeySecret.Generate form) BadIdentityTokenInvalid
SDK 2 probe 54-char key, same format, 24-byte secret admitted, Good reads
SDK 1.5 CLI (efb9197a) the same 65-char key admitted, Good read
SDK 2 probe wrong-secret / unknown-id / disabled key, 54-char BadIdentityTokenRejected (the server's real key check)
SDK 2 probe wrong-secret / unknown-id key, 65-char BadIdentityTokenInvalid (rejected on format before the key check)
SDK 2 probe/CLI 65-char key, against the SDK 2 server (438484fd) admitted (this is checked again in the #800 candidate phase)

Cause (as #873 describes): an SDK 2 client encrypts any UserName password over 64 bytes as RsaEncryptedSecret. The 1.5.378 server does not accept that format, so the token is invalid before any key lookup. Every OtOpcUa API key is 65 characters.

Impact:

  • After the SDK 2 rollout, our SDK 2 clients cannot use API keys against SDK 1.5 servers. Today that means QA vq03/vq04 (still b8b4c18f, SDK 1.5.378.106) and any known-good build a node is rolled back to. That covers:
    • the OtOpcUa Client CLI/UI;
    • the OpcUaClient driver of an SDK 2 node pointed at an SDK 1.5 OtOpcUa server;
    • any SDK 2 tooling run during a rollback.
  • Any third-party server built on an older stack could hit the same thing.
  • SDK 1.5 clients (ScadaBridge today) are unaffected in either direction.

Repro: scripts/validation/opcua-windows-service/ (#800). Install the baseline, then run -Phase Checks -BaselineServer. The 65-char cases are recorded as expected denials, and cli-sdk2-encrypt-key-65 appears under findings.

Options:

  1. Document it in the rollout/rollback runbook: during a rollback, use SDK 1.5 tooling.
  2. Let SDK 2 clients force the legacy UserName encryption when the server is older. Check whether 2.0.1 has a switch for this or decides from the endpoint's token policy.
  3. Cap generated secrets so a key fits in 64 bytes. This would be a compat change to ApiKeySecret; existing keys keep working.
Found by the #800 Windows service gate on windev, 2026-11-12. Both client and server ran on the same box, so clock skew is ruled out (#873 does not apply). **Rollback/QA-compat risk.** **Symptom:** an SDK 2.0.1 client cannot log in to an SDK **1.5.378.182** OtOpcUa server (known-good master `efb9197a`) with a production-format API key. Each attempt gets `BadIdentityTokenInvalid` at ActivateSession, in both Sign and SignAndEncrypt (Basic256Sha256), on a channel the server trusts. This was seen with both our #800 probe (SDK 2.0.1) and the actual SDK 2 Client CLI at master `438484fd`. **Controls (same server, same boot, same trusted client certificate):** | Client | Password | Result | |---|---|---| | SDK 2 probe | 65-char key (`otua_<16 hex>_<43>`, the `ApiKeySecret.Generate` form) | `BadIdentityTokenInvalid` | | SDK 2 probe | 54-char key, same format, 24-byte secret | admitted, Good reads | | SDK 1.5 CLI (`efb9197a`) | the same 65-char key | admitted, Good read | | SDK 2 probe | wrong-secret / unknown-id / disabled key, **54-char** | `BadIdentityTokenRejected` (the server's real key check) | | SDK 2 probe | wrong-secret / unknown-id key, **65-char** | `BadIdentityTokenInvalid` (rejected on format before the key check) | | SDK 2 probe/CLI | 65-char key, against the SDK 2 server (`438484fd`) | admitted (this is checked again in the #800 candidate phase) | **Cause (as #873 describes):** an SDK 2 client encrypts any UserName password over 64 bytes as `RsaEncryptedSecret`. The 1.5.378 server does not accept that format, so the token is invalid before any key lookup. Every OtOpcUa API key is 65 characters. **Impact:** - After the SDK 2 rollout, our **SDK 2 clients cannot use API keys against SDK 1.5 servers.** Today that means QA vq03/vq04 (still `b8b4c18f`, SDK 1.5.378.106) and any known-good build a node is **rolled back** to. That covers: - the OtOpcUa Client CLI/UI; - the OpcUaClient driver of an SDK 2 node pointed at an SDK 1.5 OtOpcUa server; - any SDK 2 tooling run during a rollback. - Any third-party server built on an older stack could hit the same thing. - SDK 1.5 clients (ScadaBridge today) are unaffected in either direction. **Repro:** `scripts/validation/opcua-windows-service/` (#800). Install the baseline, then run `-Phase Checks -BaselineServer`. The 65-char cases are recorded as expected denials, and `cli-sdk2-encrypt-key-65` appears under findings. **Options:** 1. Document it in the rollout/rollback runbook: during a rollback, use SDK 1.5 tooling. 2. Let SDK 2 clients force the legacy UserName encryption when the server is older. Check whether 2.0.1 has a switch for this or decides from the endpoint's token policy. 3. Cap generated secrets so a key fits in 64 bytes. This would be a compat change to `ApiKeySecret`; existing keys keep working.
dohertj2 added the type:bugpriority:p1cross-repo labels 2026-11-12 10:11:44 -05:00
dohertj2 commented 2026-11-12 11:14:55 -05:00
Author
Owner
Copy Link
Copy Source

Root cause (from the decompiled SDK assemblies, confirmed against the upstream 2.0.1 tag)

This is not a 1.5 server bug. An SDK 2.0.1 client picks a password format that a pre-1.05.04 server cannot decode, and nothing negotiates it.

  • SDK 2.0.1 client. The relevant code is UserNameIdentityTokenHandler.EncryptAsync.
    • Under an RSA policy (one with no ephemeral key: Basic256Sha256, Aes128_Sha256_RsaOaep, Aes256_Sha256_RsaPss, …), a password over 64 bytes is encrypted as an RsaEncryptedSecret. The threshold is private const int RsaEncryptedSecretPasswordThreshold = 64, which has no setting.
    • An RsaEncryptedSecret is a hybrid ExtensionObject: an RSA-wrapped AES key and HMAC key, an AES-CBC payload, and a signing time. It is sent with EncryptionAlgorithm = null.
    • At 64 bytes or less the client uses the legacy format, RSA-OAEP(len ‖ password ‖ serverNonce) under the server certificate, with EncryptionAlgorithm = …xmlenc#rsa-oaep (…rsa-oaep-sha2-256 for RsaPss).
    • The choice came from upstream PR #3678 ("use RSAEncryptedSecret for long username passwords"). It is unchanged in 2.0.2 and on upstream master.
    • The UserTokenPolicy advertises only a SecurityPolicyUri, never a format, so the client has nothing to negotiate from.
  • SDK 1.5.378 server (.106 on QA and .182 on known-good; the code is identical).
    • UserNameIdentityToken.Decrypt handles only the legacy format for any non-ECC policy. The 1.5 EncryptedSecret is ECC-only.
    • SecurityPolicies.Decrypt with Algorithm == null throws 0x80200000 BadIdentityTokenInvalid with the message "Unexpected encryption algorithm". That happens before the authenticator runs, which is why a wrong 65-char key also showed BadIdentityTokenInvalid rather than Rejected.
  • Why length matters. It is only the SDK's switch. The RSA block size does not limit anything: the legacy format frames the plaintext with a length and spans as many RSA blocks as it needs. A 65-char key plus a 32-byte nonce plus a 4-byte length is 101 bytes, which fits in one 2048-bit block under OAEP-SHA1 (214 bytes) or OAEP-SHA256 (190 bytes).
  • SDK 2 server. It decrypts both formats: the legacy one when the algorithm URI is set, and RsaEncryptedSecret when it is null.

Fix (client side): be28aa44

  • LegacyRsaUserNameIdentityTokenHandler lives in Driver.OpcUaClient.Session and plugs in through the SDK's public UserIdentity(IUserIdentityTokenHandler) constructor.
    • Under an RSA policy it always uses the legacy format, whatever the length. That is exactly what SDK 2 sends for passwords of 64 bytes or less and what every SDK 1.5 client (ScadaBridge included) sends.
    • Policy None, the ECC policies and a missing certificate go to the SDK handler unchanged.
    • It never sends the secret in a weaker form than the SDK would, and never in the clear except where the SDK already would (policy None).
    • The only thing lost is RsaEncryptedSecret's padding of the secret's length.
  • OpcUaUserIdentityFactory.UserName(...) is now the one UserName identity used by the OpcUaClient driver, the AdminUI browse and Client.Shared (the CLI and UI). Client.Shared now references .Session.
  • No supported SDK setting exists for this. I checked 2.0.1, 2.0.2 and upstream master.
  • Docs: the SDK 2 rollout runbook's rollback step now says SDK 2 builds older than this fix, and generic SDK 2 tools, are refused by a rolled-back node. In the #800 Windows gate, cli-sdk2-encrypt-key-65 now gates against -BaselineServer. The probe stays as the SDK-default negative control. A rerun on windev needs a CLI built from this commit or later.

Proof

Cross-version matrix (scripts/validation/opcua-sdk-compatibility, #782; harness 2ad72130; evidence e5b26f37, evidence/2026-11-12-880/). Baseline 83ed880e (SDK 1.5.378.182).

  • The matrix key was otua_matrix_ + 43 characters, 55 in all, so it stayed under the threshold and hid this bug. It now uses the production ApiKeySecret.Generate shape: 65 characters, random per run, never recorded.
  • New Shared-leg cases:
    • production-length-API-key-wrong-secret-reaches-key-check must be refused with BadIdentityTokenRejected. Invalid would mean the server could not decrypt the token.
    • sdk2-default-RsaEncryptedSecret-control (SDK 2 clients only) is the negative control: the SDK's own handler with the valid key must be refused with BadIdentityTokenInvalid by a 1.x server and admitted by SDK 2.
Leg With the fix Mutation (factory reverted to the SDK default handler)
SDK 2 Shared → SDK 1.5 server 14/14 passed: 65-char key admitted, wrong key gets Rejected, SDK-default control gets Invalid connect gets BadIdentityTokenInvalid, wrong key gets Invalid (not Rejected), control gets Invalid
SDK 2 driver + CLI → SDK 1.5 server 7/7 passed driver connect and all 3 CLI calls get BadIdentityTokenInvalid
SDK 1.5 Shared → SDK 1.5 server 13/13 13/13
SDK 2 Shared → SDK 2 server SDK-default control admitted, wrong key gets Rejected same
  • Both runs show UNS-namespace failures against the master server for every client SDK. This is a pre-existing fixture gap since #808 and is unrelated to this fix. It is filed as #888.

In-repo tests:

  • LegacyRsaUserNameIdentityTokenHandlerTests covers the three RSA policies. Each must carry the algorithm URI that SDK 1.5 requires and must round-trip through the unmodified SDK 2 server decrypt. Other tests check that the session Copy() keeps the type, that policy None is unchanged, and an SDK control that pins the 64/65-byte switch.
  • A Client.Shared test checks that the CLI/UI present the legacy handler.
  • Sdk2TrustedChannelTests.A_production_length_API_key_is_admitted_in_either_password_format (SDK 2 to SDK 2, Sign and SignAndEncrypt) checks that both formats are admitted and that a wrong 65-char key reaches the authenticator with BadIdentityTokenRejected.
  • Mutation-checked: with the factory reverted, 5 of the 7 handler tests fail and the Client.Shared test fails. The two that still pass are the SDK control and the None test, as expected. The SDK 2 to SDK 2 test passes either way because an SDK 2 server reads both formats; it covers SDK 2 to SDK 2 compatibility, not #880.
  • Suites: Driver.OpcUaClient.Tests 396/396, Client.Shared.Tests 354/354, Client.CLI.Tests 177/177, Client.UI.Tests 131/131, Driver.OpcUaClient.Browser.Tests 26/26, Sdk2TrustedChannelTests 10/10. Full solution build: 0 warnings.

Remaining constraint

SDK 2 tooling that we do not build (generic SDK 2 clients using the default handler) is still refused by SDK 1.5 servers with 65-char keys. ScadaBridge is unaffected because it is an SDK 1.5 client. Cross-ref #782 (SDK upgrade, matrix) and #801 (SDK 2 live gate: consumer compatibility and paired-cluster rollout). QA vq03/vq04 and any rollback target now accept post-fix SDK 2 clients.

## Root cause (from the decompiled SDK assemblies, confirmed against the upstream `2.0.1` tag) **This is not a 1.5 server bug. An SDK 2.0.1 client picks a password format that a pre-1.05.04 server cannot decode, and nothing negotiates it.** - **SDK 2.0.1 client.** The relevant code is `UserNameIdentityTokenHandler.EncryptAsync`. - Under an RSA policy (one with no ephemeral key: Basic256Sha256, Aes128_Sha256_RsaOaep, Aes256_Sha256_RsaPss, …), a password **over 64 bytes** is encrypted as an `RsaEncryptedSecret`. The threshold is `private const int RsaEncryptedSecretPasswordThreshold = 64`, which has no setting. - An `RsaEncryptedSecret` is a hybrid ExtensionObject: an RSA-wrapped AES key and HMAC key, an AES-CBC payload, and a signing time. It is sent with `EncryptionAlgorithm = null`. - At **64 bytes or less** the client uses the legacy format, `RSA-OAEP(len ‖ password ‖ serverNonce)` under the server certificate, with `EncryptionAlgorithm = …xmlenc#rsa-oaep` (`…rsa-oaep-sha2-256` for RsaPss). - The choice came from upstream PR #3678 ("use RSAEncryptedSecret for long username passwords"). It is unchanged in 2.0.2 and on upstream master. - The `UserTokenPolicy` advertises only a `SecurityPolicyUri`, never a format, so the client has nothing to negotiate from. - **SDK 1.5.378 server (.106 on QA and .182 on known-good; the code is identical).** - `UserNameIdentityToken.Decrypt` handles only the legacy format for any non-ECC policy. The 1.5 `EncryptedSecret` is ECC-only. - `SecurityPolicies.Decrypt` with `Algorithm == null` throws `0x80200000` `BadIdentityTokenInvalid` with the message "Unexpected encryption algorithm". That happens before the authenticator runs, which is why a wrong 65-char key also showed `BadIdentityTokenInvalid` rather than `Rejected`. - **Why length matters.** It is only the SDK's switch. The RSA block size does not limit anything: the legacy format frames the plaintext with a length and spans as many RSA blocks as it needs. A 65-char key plus a 32-byte nonce plus a 4-byte length is 101 bytes, which fits in one 2048-bit block under OAEP-SHA1 (214 bytes) or OAEP-SHA256 (190 bytes). - **SDK 2 server.** It decrypts both formats: the legacy one when the algorithm URI is set, and `RsaEncryptedSecret` when it is null. ## Fix (client side): `be28aa44` - `LegacyRsaUserNameIdentityTokenHandler` lives in `Driver.OpcUaClient.Session` and plugs in through the SDK's public `UserIdentity(IUserIdentityTokenHandler)` constructor. - Under an RSA policy it always uses the legacy format, whatever the length. That is exactly what SDK 2 sends for passwords of 64 bytes or less and what every SDK 1.5 client (ScadaBridge included) sends. - Policy `None`, the ECC policies and a missing certificate go to the SDK handler unchanged. - It never sends the secret in a weaker form than the SDK would, and never in the clear except where the SDK already would (policy None). - The only thing lost is `RsaEncryptedSecret`'s padding of the secret's length. - `OpcUaUserIdentityFactory.UserName(...)` is now the one UserName identity used by the **OpcUaClient driver**, the **AdminUI browse** and **Client.Shared** (the **CLI and UI**). Client.Shared now references `.Session`. - No supported SDK setting exists for this. I checked 2.0.1, 2.0.2 and upstream master. - Docs: the SDK 2 rollout runbook's rollback step now says SDK 2 builds older than this fix, and generic SDK 2 tools, are refused by a rolled-back node. In the #800 Windows gate, `cli-sdk2-encrypt-key-65` now gates against `-BaselineServer`. The probe stays as the SDK-default negative control. **A rerun on windev needs a CLI built from this commit or later.** ## Proof **Cross-version matrix** (`scripts/validation/opcua-sdk-compatibility`, #782; harness `2ad72130`; evidence `e5b26f37`, `evidence/2026-11-12-880/`). Baseline 83ed880e (SDK 1.5.378.182). - The matrix key was `otua_matrix_` + 43 characters, 55 in all, so it stayed under the threshold and hid this bug. It now uses the production `ApiKeySecret.Generate` shape: 65 characters, random per run, never recorded. - New Shared-leg cases: - `production-length-API-key-wrong-secret-reaches-key-check` must be refused with `BadIdentityTokenRejected`. `Invalid` would mean the server could not decrypt the token. - `sdk2-default-RsaEncryptedSecret-control` (SDK 2 clients only) is the negative control: the SDK's own handler with the valid key must be refused with `BadIdentityTokenInvalid` by a 1.x server and admitted by SDK 2. | Leg | With the fix | Mutation (factory reverted to the SDK default handler) | |---|---|---| | SDK 2 Shared → SDK 1.5 server | **14/14 passed**: 65-char key admitted, wrong key gets `Rejected`, SDK-default control gets `Invalid` | connect gets `BadIdentityTokenInvalid`, wrong key gets `Invalid` (not `Rejected`), control gets `Invalid` | | SDK 2 driver + CLI → SDK 1.5 server | **7/7 passed** | driver connect and all 3 CLI calls get `BadIdentityTokenInvalid` | | SDK 1.5 Shared → SDK 1.5 server | 13/13 | 13/13 | | SDK 2 Shared → SDK 2 server | SDK-default control admitted, wrong key gets `Rejected` | same | - Both runs show UNS-namespace failures against the **master** server for **every** client SDK. This is a pre-existing fixture gap since #808 and is unrelated to this fix. It is filed as #888. **In-repo tests:** - `LegacyRsaUserNameIdentityTokenHandlerTests` covers the three RSA policies. Each must carry the algorithm URI that SDK 1.5 requires and must round-trip through the unmodified SDK 2 server decrypt. Other tests check that the session `Copy()` keeps the type, that policy None is unchanged, and an SDK control that pins the 64/65-byte switch. - A Client.Shared test checks that the CLI/UI present the legacy handler. - `Sdk2TrustedChannelTests.A_production_length_API_key_is_admitted_in_either_password_format` (SDK 2 to SDK 2, Sign and SignAndEncrypt) checks that both formats are admitted and that a wrong 65-char key reaches the authenticator with `BadIdentityTokenRejected`. - Mutation-checked: with the factory reverted, 5 of the 7 handler tests fail and the Client.Shared test fails. The two that still pass are the SDK control and the None test, as expected. The SDK 2 to SDK 2 test passes either way because an SDK 2 server reads both formats; it covers SDK 2 to SDK 2 compatibility, not #880. - Suites: Driver.OpcUaClient.Tests 396/396, Client.Shared.Tests 354/354, Client.CLI.Tests 177/177, Client.UI.Tests 131/131, Driver.OpcUaClient.Browser.Tests 26/26, Sdk2TrustedChannelTests 10/10. Full solution build: 0 warnings. ## Remaining constraint SDK 2 tooling that we do not build (generic SDK 2 clients using the default handler) is still refused by SDK 1.5 servers with 65-char keys. ScadaBridge is unaffected because it is an SDK 1.5 client. Cross-ref #782 (SDK upgrade, matrix) and #801 (SDK 2 live gate: consumer compatibility and paired-cluster rollout). QA vq03/vq04 and any rollback target now accept post-fix SDK 2 clients.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
auto-managed
queue management
 blocked
Blocked by dependency or external hold
 cross-driver
queue management
 cross-repo
Needs coordinated change in another family repo (wire contract etc.)
 driver/abcip
queue management
 driver/ablegacy
queue management
 driver/focas
queue management
 driver/mqtt
MQTT / Sparkplug B broker-subscribe driver
 driver/opcuaclient
queue management
 driver/s7
queue management
 driver/twincat
queue management
 migrated
Migrated from a markdown backlog (2026-11 standardization)
 phase/1
queue management
 phase/2
queue management
 phase/3
queue management
 phase/4
queue management
 phase/5
queue management
 phase/6
queue management
 priority:p0
Must-have / blocking
 priority:p1
High priority
 priority:p2
Normal priority
 priority:p3
Low priority / nice to have
 queue/blocked
queue management
 queue/done
queue management
 queue/failed
queue management
 queue/in-progress
queue management
 queue/queued
queue management
 type:bug
Defect in shipped behaviour
 type:debt
Technical debt / hygiene
 type:deploy-gate
Rollout/deploy/migration step not yet done on an environment
 type:docs
Documentation work
 type:feature
Feature implementation
 type:followup
Follow-up deferred from a plan, PR or gate
 type:infra
Infrastructure and scaffolding
 type:review-finding
Unfixed finding from an architecture/code review
 type:test
Test work
No Label cross-repo priority:p1 type:bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
dohertj2 (dohertj2)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: dohertj2/lmxopcua#880
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?