lmxopcua/docs/v2/Galaxy.Performance.md

# Galaxy backend performance

This document covers the performance surface of the in-process
`GalaxyDriver` (the v2 mxgw backend) — the ActivitySource it emits, the
metrics on its EventPump, the soak scenario that validates it, and the
tuning knobs you can reach for when the dev parity rig surfaces a hot
spot.

## Tracing surface (PR 6.1)

The driver emits spans on the `ZB.MOM.WW.OtOpcUa.Driver.Galaxy`
ActivitySource. No package dependency on OpenTelemetry — the host
process picks the listener (OTLP exporter, dotnet-trace, Application
Insights). Wire it via `OpenTelemetry.Trace.AddSource(...)` in the
host's tracing pipeline.

| Span | Source | Tags |
|------|--------|------|
| `galaxy.subscribe_bulk` | `TracedGalaxySubscriber` | `galaxy.client`, `galaxy.tag_count`, `galaxy.buffered_interval_ms`, `galaxy.success_count` |
| `galaxy.unsubscribe_bulk` | `TracedGalaxySubscriber` | `galaxy.client`, `galaxy.tag_count` |
| `galaxy.stream_events` | `TracedGalaxySubscriber` | `galaxy.client`, `galaxy.event_count` (set on stream end) |
| `galaxy.write` | `TracedGalaxyDataWriter` | `galaxy.client`, `galaxy.tag_count`, `galaxy.secured_write_count`, `galaxy.success_count` |
| `galaxy.get_hierarchy` | `TracedGalaxyHierarchySource` | `galaxy.client`, `galaxy.object_count` |

The stream-events span deliberately covers the *entire* stream lifetime
rather than per-event spans — at 50k tags / 1Hz the per-event volume
would dominate the trace pipeline. Per-event visibility flows through
the metrics surface instead.

## Metrics surface (PR 6.2)

`EventPump` publishes three counters on the
`ZB.MOM.WW.OtOpcUa.Driver.Galaxy` meter, each tagged with
`galaxy.client` so multi-driver hosts can split by source:

| Counter | Unit | Meaning |
|---------|------|---------|
| `galaxy.events.received` | `{event}` | MxEvents read from the gateway StreamEvents stream |
| `galaxy.events.dispatched` | `{event}` | MxEvents that made it through the bounded channel into `OnDataChange` |
| `galaxy.events.dropped` | `{event}` | MxEvents discarded because the bounded channel was full (newest-dropped) |
| `galaxy.events.replay_gaps` | `{gap}` | Replay-gap sentinels received — the gateway evicted events we can never see |
| `galaxy.events.undecodable` | `{event}` | Data-change events carrying a value kind this driver cannot represent |

The invariant is `received = dispatched + dropped + replay_gaps +
(in-flight in the channel)`. A replay-gap sentinel is counted as
received but never enters the channel — it is a control signal with no
item payload, handled on the read loop precisely so channel pressure
can never drop it.

Watch the dropped counter — it is the leading indicator of
listener back-pressure. A non-zero dropped rate means a downstream
consumer (DriverNodeManager → UA notification queue → client) is
slower than the gw event stream; investigate that consumer before
raising `EventPump` channel capacity.

`galaxy.events.replay_gaps` should be flat at zero, for a structural
reason rather than a healthy one: the gateway only emits the sentinel on
a stream opened with `after_worker_sequence > 0`, and the driver opens
with `0` ("fresh stream, not a resume"). A non-zero value therefore means
the resume path was enabled — each gap costs a full re-subscribe of every
tag on the driver, so a climbing counter is a sizing problem, not just an
observation.

`galaxy.events.undecodable` should likewise be flat at zero: every
`MxValue` kind the contract declares decodes today. Non-zero means the
gateway shipped a new wire type, and those readings are being published
as **`BadTypeMismatch` with a null value** — visible to clients as bad
data rather than as a plausible null, and logged once per distinct kind.
The remedy is a mapping in `MxValueDecoder`, not a quality change.

### Bounded channel design

The pump runs two background tasks:

1. **Producer** — reads from `IGalaxySubscriber.StreamEventsAsync`,
   increments `events.received`, and `TryWrite`s into a bounded
   `Channel<MxEvent>`. When the channel is full, the producer counts
   the drop and continues reading the gw stream so back-pressure does
   not propagate upstream (which would stall the gw worker and cascade
   to *all* driver instances sharing that worker).
2. **Consumer** — reads from the channel, fans out via
   `SubscriptionRegistry`, increments `events.dispatched`.

Default channel capacity is 50_000 (one second of headroom at 50k
tags / 1Hz). Override via the `EventPump` constructor's
`channelCapacity` parameter; the public-facing wiring path in
`GalaxyDriver.EnsureEventPumpStarted` does not yet expose this through
`GalaxyDriverOptions` because no parity scenario has needed it. Add it
when soak data does.

## Buffered update interval (PR 6.3)

`MxAccess.PublishingIntervalMs` (default 1000) flows through both
subscribe paths:

- `GalaxyDriver.SubscribeAsync` — the caller's `publishingInterval`
  wins when non-zero (the server's UA subscription publishingInterval
  drives this in production). When the caller passes
  `TimeSpan.Zero`, the configured option is the fallback.
- `PerPlatformProbeWatcher` — the watcher passes the configured value
  through `SubscribeBulkAsync` so probe `ScanState` changes publish at
  the deployment's chosen cadence.

A session-level `SetBufferedUpdateInterval` RPC exists in the gw
protocol but the .NET client doesn't expose a typed helper yet —
adjusting an existing subscription's interval mid-flight is a
follow-up. Today's path subscribes once at the right interval, which
covers the common case.

## Soak scenario (PR 6.4)

`SoakScenarioTests.Soak_HoldsSubscription_AndKeepsEventStreamFlowing`
in `Driver.Galaxy.ParityTests` is the long-running validation. It
subscribes a configurable tag count (default 50_000), holds the
subscription for a configurable duration (default 24h), polls the
three counters every minute, and asserts:

- `events.received` continues to grow (gw stream isn't stuck)
- `events.dropped / events.received` stays under the configured
  ceiling (default 0.5%)
- process working-set doesn't grow more than 1 GB above baseline
  (leak guard)

Always skipped unless the operator opts in:

```bash
# Full 24h × 50k soak (production validation)
OTOPCUA_SOAK_RUN=1 dotnet test tests/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.ParityTests/

# Compressed CI-friendly run (10min × 1k tags, 1% drop ceiling)
OTOPCUA_SOAK_RUN=1 OTOPCUA_SOAK_MINUTES=10 OTOPCUA_SOAK_TAGS=1000 \
  OTOPCUA_SOAK_DROP_PCT=1.0 \
  dotnet test tests/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.ParityTests/
```

The scenario writes a per-minute CSV-style row to stdout
(`soak,<minutes>,received=…,dispatched=…,dropped=…,ws_mb=…`) so an
operator can grep the test runner output mid-run.

## Tuned defaults (PR 6.5)

| Option | Default | Source | Notes |
|--------|---------|--------|-------|
| `Gateway.ConnectTimeoutSeconds` | 10 | unchanged | Cold-start network paths fit comfortably; soak never observed >2s |
| `Gateway.DefaultCallTimeoutSeconds` | 30 | **bumped from 5** in PR 6.5 | A 50k-tag `SubscribeBulk` can exceed 5s under MxAccess COM apartment lock contention; 30s leaves headroom while still failing fast on a wedged worker |
| `Gateway.StreamTimeoutSeconds` | 0 (unlimited) | unchanged | The stream must run for the lifetime of the driver |
| `MxAccess.PublishingIntervalMs` | 1000 | unchanged | Matches the legacy `LMXProxyServer` cadence; deployments needing tighter health visibility can dial down |
| `Reconnect.InitialBackoffMs` | 500 | unchanged | First retry shouldn't dogpile a recovering gw |
| `Reconnect.MaxBackoffMs` | 30_000 | unchanged | 30s ceiling so a long-down gw doesn't sit in 5+ min backoff |
| `Repository.DiscoverPageSize` | 5000 | unchanged | One Galaxy page round-trip per ~5k objects; soak hadn't surfaced pressure |
| `EventPump` channel capacity | 50_000 | unchanged | One second of headroom at 50k tags / 1Hz |

The unchanged rows are not "definitely correct" — they are "no live
data argues for changing them." Re-run the soak scenario after every
substantive driver change, and revise this table when the data does.

## Where to look first when something's slow

1. **Slow `Discover`?** Inspect `galaxy.get_hierarchy` span duration
   and `galaxy.object_count`. The gw walks the Galaxy DB serially;
   slow Discovers usually mean a slow ZB SQL.
2. **Subscribe pile-up?** `galaxy.subscribe_bulk` span duration
   correlates with `galaxy.tag_count`. If duration ÷ tag_count starts
   climbing, the gw worker is probably under apartment-lock pressure.
3. **Events stalled?** Watch `galaxy.events.received`. Flat-lined
   means the gw stream is wedged — kick the reconnect supervisor by
   forcing a `ReinitializeAsync`.
4. **Dropped events?** Non-zero `galaxy.events.dropped` means a slow
   downstream consumer. Profile `OnDataChange` handlers in
   `DriverNodeManager` before bumping the channel capacity.
5. **Memory growing?** Confirm with the soak scenario's working-set
   leak guard. Likely culprits: lingering subscription handles in
   `SubscriptionRegistry`, or a downstream consumer retaining
   `DataValueSnapshot` references past their useful life.

## Scripted-alarm engine — hot-path allocation reuse

`ScriptedAlarmEngine` keeps a per-alarm reusable evaluation scratch in `_scratchByAlarmId` — the read-cache `Dictionary<string, DataValueSnapshot>` and the `AlarmPredicateContext` are allocated once per alarm (on first evaluation) and refilled in place across every subsequent predicate evaluation. The hot path no longer allocates a fresh dictionary + context per upstream tag change. (Core.ScriptedAlarms-009)

Safety: reuse is serialised under `_evalGate`, so two threads can never observe the same scratch in a half-refilled state. The context wraps the read-cache by reference, so refilling the dictionary is what the predicate's `ctx.GetTag(path)` calls observe. `LoadAsync` clears `_scratchByAlarmId` alongside `_alarms` so a config-publish drops the prior generation's scratch (a new generation may carry different `Inputs` / `Logger`). Regression tests in `ScriptedAlarmEngineTests` lock the reuse contract:
- `Reevaluation_reuses_the_same_read_cache_dictionary` — asserts dictionary instance identity across two evaluations.
- `Reevaluation_reuses_the_same_predicate_context` — same, for the context.
- `LoadAsync_drops_the_prior_generations_scratch` — asserts a publish resets the scratch.

## Value-path batching (2026-08-15)

The hot value path is batched end to end: one message per driver production burst (a poll cycle, an
inbound message, a stream chunk) instead of one per changed tag at every hop, one SDK-lock acquisition
per batch instead of one per value, and no per-write key allocation in the node manager. Nothing about
*what* is published changed — only how many messages and lock takes it costs. Plan +
task-by-task rationale: [`docs/plans/2026-08-15-value-pipeline-batching.md`](../plans/2026-08-15-value-pipeline-batching.md).

**The seams, in order:**

- **`IBatchSubscribable` / `PollGroupEngine`'s `onBatch`** — an optional driver capability raising
  `OnDataChangeBatch` once per burst. The four engine-based poll drivers (Modbus, S7, AbCip, FOCAS) get
  it from the engine's batch callback. The per-tag `ISubscribable.OnDataChange` is unchanged and still
  raised — Galaxy cold reads and the driver CLIs subscribe it. A consumer subscribes to **exactly one**
  of the two; subscribing both double-delivers.
- **`DriverInstanceActor` → `AttributeValueBatchPublished`** — one per burst, preferring the batch event
  when the driver implements `IBatchSubscribable` and wrapping a non-batch driver's per-tag events as
  single-item batches, so downstream sees one shape. The detach path tears the batch handler down too
  (load-bearing: the `IDriver` outlives the actor across host respawns).
- **`DriverHostActor` → `AttributeValueBatchUpdate`** — resolves the whole batch's `NodeRealmRef`s
  (raw NodeId + every referencing UNS NodeId) and sends **one** message to `OpcUaPublishActor`, plus one
  batch Tell to the dependency mux. The mux keeps its per-item receive — the comms-loss fan-out
  fabricates per-item messages.
- **`OtOpcUaNodeManager.WriteValues`** — the sink's batch override takes the manager-wide `Lock` once per
  chunk of **512** items rather than once per value, with each item isolated so one bad node cannot lose
  the rest of the batch.

**Invariants the batching preserves — do not break them:**

- **#519 status codes ride verbatim.** The full `uint` status code, sub-code included, reaches the SDK
  unchanged at every hop; batching moved the values, it did not re-derive them. Timestamps likewise come
  from the driver — nothing on this path stamps `DateTime.UtcNow`.
- **Metering stays per ITEM, not per batch** (`OpcUaPublishActor`'s write tally + the
  `OtOpcUaTelemetry.OpcUaSinkWrite` meter). Counting per message would have silently dropped the
  dashboards by up to 100×.
- **Deadband is honored on the batch event.** ⚠️ `PollGroupEngine` collects its batch list
  **unconditionally** — it has no knowledge of a driver's publish filter — so `ModbusDriver` ignores the
  engine's list and raises from its own buffer of what actually passed `ShouldPublish`, using `onBatch`
  purely as the end-of-cycle signal. Forwarding the engine's list instead would carry changes the per-tag
  event suppressed, and a consumer preferring the batch event would see deadband silently stop working.
- **Partial failure is typed, not swallowed.** A sink that applies some of a batch and fails the rest
  reports `ValueWriteBatchException` (first failed NodeId + failed/succeeded counts); the publish actor's
  typed catch advances the tally by the **succeeded** count — not zero (the general catch's behaviour) and
  not the whole batch.

**Two supporting changes:** the node manager's variable / folder / alarm / historized-tagname maps are
re-keyed by a `RealmKey(Realm, Id)` struct, so a write no longer allocates `new NodeId(id, ns).ToString()`
per value (the realm still travels explicitly — it is never parsed back out of an id string). And the
pinned `opcua-synchronized-dispatcher`'s `throughput` rose **1 → 100**: the actor keeps its own dedicated
thread (that is what the `PinnedDispatcher` guarantees, not `throughput`), so this only removes a mailbox
re-schedule round-trip per message on the hottest actor in the process, with zero change to concurrency.
Keep it in sync with the inline HOCON copy in `ServiceCollectionExtensions.WithOtOpcUaRuntimeActors`.