lmxopcua/scripts/validation/opcua-client-lifecycle/README.md

# Production OPC UA client lifecycle diagnostic

This standalone harness runs the unchanged production `OpcUaClientService`,
`OpcUaClientDriver`, and `OpcUaClientDriverBrowser` in one persistent Linux client
process against a separate process using the production SDK server and node
manager. Both processes load OPC Foundation SDK 2.0.1. The server fixture is a
local copy of the compatibility harness at commit `7b353d7b`; only this directory's
copy has configurable lifetime and atomic public resource snapshots.

Run a short diagnostic from a checkout containing the SDK 2.0.1 changes:

```sh
python3 scripts/validation/opcua-client-lifecycle/run.py \
  --duration 90 --output /tmp/otopcua-client-lifecycle-diagnostic-unique
```

The coordinator must launch and own the full process separately:

```sh
python3 scripts/validation/opcua-client-lifecycle/run.py \
  --full --duration 7200 --output /tmp/otopcua-client-lifecycle-full-unique
```

The output path must not exist. `--full` rejects a duration below 7200 seconds;
`fullTwoHourLifecycle` is true only after the client monotonic clock has actually
elapsed at least 7200 seconds and every final check completes. A diagnostic pass
cannot satisfy that gate. The outer process also has a duration plus 300-second
hard deadline and forcibly removes its own container on timeout.

The launcher publishes this standalone project, then uses the locally cached
official `mcr.microsoft.com/dotnet/sdk:10.0.103-noble` Linux arm64 image with
`--network none`, no host ports and fresh task-owned PKI. `XDG_DATA_HOME` is the
only storage environment override; it never changes `HOME`. A generated
credential stays in the child process environment. The output root is mode 0700
and contains generated private keys: **never commit the entire output directory**.
Commit only the selected redacted JSON/log receipts under `evidence/`.

Preparation invokes bounded failing connections through each product's public
API to create its own application certificate. It does not reflectively invoke
configuration builders or replace the store. Before the live session, public DER
certificates are exchanged explicitly among the three clients and server.
Autoaccept is disabled on both ends. Negotiation must be Basic256Sha256
SignAndEncrypt and API-key-shaped credentials must pass the fixture authenticator
and production authority/ACL path. Existing own DER hashes must remain unchanged
across all loop iterations.

Every cycle performs two Shared connect/read/write/subscribe/unsubscribe/
disconnect rounds on the same object. It performs two driver initialize/read/
write/subscribe/unsubscribe rounds, each followed by a live reinitialize with an
active subscription, then shutdown. It opens, browses and disposes two production
AdminUI browser sessions. Scalar Int32 and Int32[3] writes must roundtrip exactly,
and both Shared and driver scalar data notifications must arrive.

The external harness reads ownership fields for diagnostics only. Fields are
looked up by exact name, with missing/null layouts failing the run. It does not
write reflected fields or change SDK state. For SDK 2.0.1.0 only, each captured
`CertificateManager.m_disposed` boolean must be false while connected and true
after release. An independently created, deliberately disposed SDK manager proves
the before/after probe. All strong manager references are dropped each pass.
Eight released managers are checked per cycle: two Shared, four driver (including
live reinitialization), and two browser. Production session and data subscription
collections must clear, and public Shared connection information and driver health
must return to disconnected/Unknown. Collected weak references to disposed client,
driver, browser and browse-session objects must have no survivors at samples.

After every cycle the server's public session/subscription/monitored-item counts
must return to zero within a bounded settle period. Resource samples capture GC,
managed allocation/retained bytes, RSS/private bytes, global FDs and task/PKI FDs.
Owned-path FD matching canonicalizes absolute paths and uses a path-separator
boundary. Sampling starts cold, then every ten cycles and at completion. RSS and
FD observations require interpretation after warmup; they are not performance
budgets or an automatic G4 acceptance decision.

Artifacts are `result.json`, `source.json`, `command.json`, `build.log`,
`container.log`, and `task/{summary.json,resources.jsonl,sdk-assemblies.json,
client.log,server.log,prepare.log,server/live-resources.json}`. A failure additionally
writes a redacted `task/failure.txt`. The server uses a fixed application URI and
the three client certificate stores are reused for the entire process lifetime.

This is a small local lifecycle gate, not the entire Task 12 gate. It does not
cover certificate revocation, network stalls, node surgical edits/rebuild,
continuation-point abandonment/churn, representative driver fanout/load,
Windows install/publish, ScadaBridge or AVEVA interoperability. It makes no
baseline/candidate performance or full G4 claim. The coordinator's separate server
lifecycle harness supplies complementary SDK session/CP/server mutation coverage.

The checked-in 2026-11-10 diagnostic completed **91.104 seconds**, 68 cycles,
136 Shared rounds, 136 driver rounds (plus 136 live driver reinitializations),
136 browser sessions, and 544 verified released certificate managers. The final
sample had zero surviving weak product objects and zero owned PKI FDs; server
sessions/subscriptions/items returned to zero after every cycle. Warm samples
showed 146–148 total FDs; the final managed heap was 4,737,536 bytes and RSS was
181,018,624 bytes. The deliberate manager before/after control passed and
`--full --duration 90` exited 2 before creating artifacts. This is diagnostic
coverage only; the two-hour gate was pending at that point. The completed full
run is recorded below. Receipts are under `evidence/2026-11-10-diagnostic/`.


The coordinator-owned full run completed on 2026-11-10 with **exit 0 and
7,200.984 seconds of actual client observation**: 5,420 cycles; 10,840 Shared
rounds, 10,840 driver rounds plus 10,840 live reinitializations, 10,840 browser
sessions, and **43,360 verified manager disposals**. All samples report zero weak
product-object survivors and zero open PKI descriptors; each cycle returned server
public session/subscription/item counts to zero. Final retained managed memory was
3,988,520 bytes and RSS 86,102,016 bytes, with 148 global FDs. Own certificate hashes
were unchanged. The manager before/after positive control passed. Selected counter
histories, loaded SDK hashes, artifact manifest and verification hashes are in
`evidence/2026-11-10-full/`; private PKI and credentials are excluded.

This is a **completed two-hour client lifecycle gate**, not a performance-budget
or complete G4 pass. Its source is `c25561c6`, exact SDK 2.0.1, framework-dependent
Linux ARM64 on runtime 10.0.3 with the pinned image ID recorded in the receipt.
The later HG 1.0 consumer mapping is independently tested and does not retroactively
change these immutable artifact/source records. The failed canonical server RSS
workload remains separate in #799.