Joseph Doherty
142 lines
8.4 KiB
Markdown
142 lines
8.4 KiB
Markdown
# Linux SDK 2 server lifecycle soak
|
||
|
||
This launcher publishes and runs the existing
|
||
`Sdk2LifecycleSoakTests.Repeated_lifecycles_return_resources_to_a_steady_baseline`
|
||
without copying, rewriting, suppressing or changing its test logic. It imports the
|
||
canonical descriptor fixes from `642aefe3`: physical symlink resolution, path
|
||
separator boundaries, open/closed PKI positive controls, and bounded probe process
|
||
and stream cleanup. The same test still owns the production server, node manager,
|
||
sessions, subscriptions, continuation points, revocation, stalled TCP proxy,
|
||
surgical changes/rebuild and same-PKI restarts.
|
||
|
||
A diagnostic run:
|
||
|
||
```sh
|
||
python3 scripts/validation/opcua-server-lifecycle-linux/run.py \
|
||
--duration 90 --output /tmp/otopcua-server-linux-diagnostic-unique
|
||
```
|
||
|
||
The coordinator must launch and own a full run in its persistent execution
|
||
session. A diagnostic cannot satisfy the full gate:
|
||
|
||
```sh
|
||
python3 scripts/validation/opcua-server-lifecycle-linux/run.py \
|
||
--full --duration 7200 --output /tmp/otopcua-server-linux-full-unique
|
||
```
|
||
|
||
An already published artifact can be reused with
|
||
`--published-app /absolute/path/to/app`. Its parent must contain the launcher's
|
||
`publish-receipt.json`; the test source, apphost and test assembly hashes must
|
||
match. Reuse only a publish produced from the current canonical test source:
|
||
|
||
```sh
|
||
python3 scripts/validation/opcua-server-lifecycle-linux/run.py \
|
||
--full --duration 7200 \
|
||
--published-app /absolute/path/to/current-diagnostic/app \
|
||
--output /tmp/otopcua-server-linux-full-another-unique
|
||
```
|
||
|
||
The historical 2026-11-10 diagnostic publish predates the later pre-assert/GC/JIT
|
||
instrumentation. Its recorded full-run command is retained in the failed-run
|
||
receipt; a current launcher correctly rejects that publish's test-source hash.
|
||
Republish to measure the current instrumented test.
|
||
|
||
The output directory must be fresh. It is mode 0700 and can contain task-owned
|
||
private PKI under `task/tmp` during a run or after a failure. Never commit the
|
||
entire output. The existing successful test removes its own stores. Only selected
|
||
redacted receipt/log/JSONL/TRX files belong under this directory's `evidence/`.
|
||
|
||
The exact publish command uses the host .NET SDK (recorded as 10.0.403 for the
|
||
verified diagnostic), RID `linux-arm64`, self-contained runtime 10.0.14, explicit
|
||
`UseAppHost=true`, `PublishSingleFile=false`, `PublishTrimmed=false`, one MSBuild
|
||
node, and an external task-owned artifacts directory. All runtime packs were
|
||
already cached. No production or test source is patched during publish. The
|
||
launcher requires an empty production/package source diff against the reviewed
|
||
`67bc6123` production tree and
|
||
records the actual published revision separately from the launcher revision.
|
||
The reference was advanced from `7b353d7b` after review of comment-only edits,
|
||
then to `67bc6123` for the independently reviewed/tested HG1.0 adapter/package
|
||
change. Canonical SDK server code is unchanged by that consumer update. Historical
|
||
receipts retain their actual source/reference hashes.
|
||
That is source equivalence, not a claim that separately built Linux and macOS
|
||
assemblies have identical hashes. The full publish file hashes, exact test source
|
||
hash, runtimeconfig and SDK dependency versions are retained.
|
||
|
||
The cached official SDK image
|
||
`mcr.microsoft.com/dotnet/sdk:10.0.103-noble` is resolved to its immutable local
|
||
image ID and must be Linux arm64. The container has `--network none`, no host
|
||
ports and only a read-only application mount plus its fresh task volume.
|
||
`TMPDIR=/task/tmp` and `XDG_DATA_HOME=/task/data` isolate temporary files and PKI;
|
||
`HOME` is not changed. Nothing targets fleet instances, production stores or
|
||
other task fixtures.
|
||
|
||
The image's VSTest runner first executes exactly the two existing descriptor
|
||
controls. A separate invocation executes exactly one lifecycle test. Both TRX
|
||
files must contain the expected exact executed/passed counts, no skips and no
|
||
failure/error/abort/timeout. The published xUnit v3 application must be ELF64
|
||
AArch64. During the live test, `/proc/<testpid>/exe` must point to the published
|
||
application and the mapped CLR must be `/app/libcoreclr.so`, proving that the
|
||
out-of-process test host is the self-contained Linux publish rather than an
|
||
accidentally copied macOS host.
|
||
|
||
The original test resource limits remain **64 MiB retained managed growth,
|
||
128 MiB RSS growth, and 32 additional FDs**, using its original third-cycle
|
||
baseline and original managed trend check. Quiescent sessions, subscriptions,
|
||
items, browse/history points, monitored-node/live-item counts and PKI FDs must be
|
||
zero. Cached certificate stores and address-space shape must remain at baseline.
|
||
The original exact-tag read-only diagnostics and manager/store disposal assertions
|
||
are unchanged. The wrapper verifies the same limits in the header and requires
|
||
one start, one completion, no failed row, one active/stall/quiescent row per
|
||
completed cycle and one restart per four cycles. Full mode requires both requested
|
||
and actual monotonic observation time of at least 7200 seconds and a true final
|
||
`satisfiesTwoHourDuration`. The outer container deadline is duration plus 300
|
||
seconds. Its own container is forcibly removed on timeout/interruption.
|
||
|
||
Artifacts include `publish.log`, `publish-receipt.json`, `source.json`,
|
||
`artifact-manifest.json`, `command.json`, `result.json`, `container.log`, and
|
||
`task/{soak.jsonl,soak.log,descriptor-controls.log,linux-precheck.txt,
|
||
linux-runtime-process.txt,container-dotnet-info.txt,results/*.trx}`. A reused
|
||
artifact keeps its original publish receipt; `source.json` identifies both source
|
||
revisions. `result.json` is passing only after TRX and completion validation.
|
||
|
||
This supplies supported Linux runtime evidence. It does not resolve the observed
|
||
macOS RSS failure or establish its cause. A short Linux pass is not a two-hour
|
||
pass, and a later Linux pass alone cannot explain away the macOS failure. This
|
||
lane makes no Windows, AVEVA, interoperability or complete G4 performance claim.
|
||
|
||
The final checked-in diagnostic completed **91.602 seconds**, 25 cycles, six
|
||
actual server restarts, 25 recovered network stalls, and one passing lifecycle
|
||
TRX result. Both descriptor tests passed: the symlink control observed
|
||
`before=0, opened=1, closed=0`, excluded the sibling path, and the timed-out probe
|
||
was reaped in 267 ms. All quiescent logical-resource counts and PKI FDs were zero.
|
||
Third-cycle to final deltas were **−145,080 managed bytes, +53,739,520 RSS bytes,
|
||
and +2 FDs**, within the unchanged diagnostic limits. These short observations do
|
||
not establish a two-hour resource trend. The attempted full Linux gate failed as recorded below.
|
||
CLI rejection of `--full --duration 90`, rejection of a short completion as a full
|
||
run, and rejection of the wrong TRX executed count were also verified. Selected
|
||
receipts are in `evidence/2026-11-10-diagnostic/`.
|
||
|
||
|
||
The actual full attempt on 2026-11-10 **failed** at 248 cycles, 61 restarts and
|
||
903.508 seconds: RSS353,931,264 exceeded351,850,496 bytes (third-cycle baseline
|
||
+128MiB). Managed memory (~20.5MB), global descriptors189, PKI descriptors0,
|
||
cached stores2 and logical collections remained bounded. This is a combined
|
||
in-process SDK client/server measurement, not standalone server attribution.
|
||
The test/container/launcher returned failure; no completion or two-hour pass
|
||
exists. Selected sanitized source/runtime/hash/TRX/resource receipts are in
|
||
[`server-linux-soak-failed.json`](../../../docs/plans/evidence/2026-11-10-opcua2/server-linux-soak-failed.json).
|
||
[#799](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/799) tracks the
|
||
cross-platform investigation; unchanged thresholds remain mandatory.
|
||
|
||
The canonical test additionally supports explicit `OTOPCUA_SDK2_SOAK_MODE=profile`
|
||
for 60–1800 seconds when invoked directly with the same opt-in/output/build
|
||
environment. This launcher deliberately exposes diagnostic/full modes only.
|
||
Profile mode logs each captured resource tuple before assertions, including
|
||
GC committed/fragmented bytes, JIT method/IL counts, timers and threads. After a
|
||
resource-budget failure it may continue bounded diagnostic collection, labels
|
||
subsequent rows `quiescent-budget-rejected`, and rethrows the first original
|
||
assertion before any completion record. It never changes thresholds, produces a
|
||
two-hour flag, or turns a rejected run into a pass. Full/diagnostic modes retain
|
||
immediate assertion failure. Native/private/file/anonymous RSS profiling and a
|
||
matched original-SDK control are needed before attributing the failed gate.
|