mxaccessgw/docs/DesignDecisions.md

# Design Decisions

This document records current v1 choices for the MXAccess gateway design. These
decisions can change, but implementation should follow them until a later design
update says otherwise.

## Source References

Use these local analysis sources when answering MXAccess-specific design or
implementation questions:

```text
C:\Users\dohertj2\Desktop\mxaccess
C:\Users\dohertj2\Desktop\mxaccess\docs\MXAccess-Public-API.md
C:\Users\dohertj2\Desktop\mxaccess\docs\MXAccess-Reverse-Engineering.md
```

Use these local notes for Galaxy Repository SQL metadata:

```text
C:\Users\dohertj2\Desktop\lmxopcua\gr
```

## MXAccess COM Target

Decision: target the installed MXAccess COM interop surface directly from the
x86 worker.

Concrete COM details from the MXAccess analysis:

- Interop assembly:
  `C:\Program Files (x86)\ArchestrA\Framework\Bin\ArchestrA.MXAccess.dll`
- Assembly identity:
  `ArchestrA.MxAccess, Version=3.2.0.0, PublicKeyToken=23106a86e706d0ae`
- COM class:
  `ArchestrA.MxAccess.LMXProxyServerClass`
- CLSID:
  `{C30B52F5-2CB5-4760-AF0A-3A344A7EB5DC}`
- ProgID:
  `LMXProxy.LMXProxyServer.1`
- Version-independent ProgID:
  `LMXProxy.LMXProxyServer`
- Registered server:
  `C:\Program Files (x86)\ArchestrA\Framework\Bin\LmxProxy.dll`
- Registry view:
  `HKCR\Wow6432Node\CLSID\{C30B52F5-2CB5-4760-AF0A-3A344A7EB5DC}`
- Threading model:
  `Apartment`

Rationale: `LMXProxyServer` is a 32-bit in-process COM server, so a .NET 10 x64
gateway cannot instantiate it directly. The x86 sidecar worker is the reliable
parity path.

Implementation guidance:

- Worker should reference `ArchestrA.MXAccess.dll`.
- Worker should instantiate `new LMXProxyServerClass()` on the dedicated STA.
- Worker should expose the resolved class, ProgID, CLSID, interop assembly
  version, and `LmxProxy.dll` path through `GetWorkerInfo` / `WorkerReady`.
- Keep the ProgID/path configurable for diagnostics, but the default should be
  the installed MXAccess class above.

## Session Reconnect

Reconnectable sessions with event replay are shipped and config-gated. The
original "no reconnectable sessions" constraint is superseded.

One `OpenSession` creates one gateway session and one worker process. The
session ends on `CloseSession`, client disconnect policy, lease expiry, worker
fault, gateway shutdown, or — when `DetachGraceSeconds > 0` — detach-grace
expiry after the last external event subscriber drops.

`MxGateway:Sessions:DetachGraceSeconds` (default `30`) controls the retention
window. When positive, a session whose last external gRPC event-stream
subscriber drops stays `Ready` for that many seconds so a client can reconnect
to the same session instead of triggering a new `OpenSession` → worker spawn.
Setting it to `0` reverts to closing only on normal lease expiry.

A reconnecting client issues `StreamEvents` with `after_worker_sequence` set to
the last sequence it observed; the gateway replays retained events newer than
that watermark (capped by `MxGateway:Events:ReplayBufferCapacity` and
`MxGateway:Events:ReplayRetentionSeconds`) then transitions seamlessly to live
delivery. If the requested position precedes the oldest retained event, a
`ReplayGap` sentinel signals the client to re-snapshot. The replay→live handoff
is atomic (no gap, no duplicate). See [Sessions](./Sessions.md) for the full
reconnect and replay protocol.

Decision: event-stream attach is bound to the opening API key. Because the
detach-grace and replay-retention windows are on by default, the reconnect
surface is a trust boundary — a retained session outlives the stream that opened
it. The session records the opening key id (`GatewaySession.OwnerKeyId`) and
every `StreamEvents` attach/reattach is rejected with `PermissionDenied` unless
the caller's key id matches the owner. Gating on the `event` scope alone was
rejected: it would let any `event`-scoped key that learned a session id attach to
another key's retained session and receive its replayed and live data. The check
runs before any subscriber is attached, so a foreign key never touches the
replay ring. Admin-scope override is deferred.

## Event Subscribers

Multi-subscriber fan-out for data-side `StreamEvents` is shipped and
config-gated. The original "one active subscriber per session" constraint is
superseded for deployments that opt in.

`MxGateway:Sessions:AllowMultipleEventSubscribers` (default `false`) controls
the mode. When `false` the session still rejects a second `StreamEvents`
subscriber with `EventSubscriberAlreadyActive`, preserving the original
single-subscriber behavior. When `true`, up to
`MxGateway:Sessions:MaxEventSubscribersPerSession` (default `8`) concurrent
external subscribers may attach; a new attach that would exceed the cap is
rejected with `EventSubscriberLimitReached`. The count-check-and-increment is
atomic under the session lock.

Failure semantics differ by mode: in single-subscriber mode a slow consumer's
channel overflow faults the whole session (`FailFast` backpressure); in
multi-subscriber mode the same condition disconnects only that subscriber so one
slow consumer never faults a session shared by others. The mode is fixed at
session construction and is not changed by a live subscriber-count snapshot.

The gateway-owned internal dashboard mirror subscribes directly on the
distributor with `isInternal: true` and is not counted toward the cap or the
detach-grace subscriber-count in either mode.

See [Sessions](./Sessions.md) for the full event-distributor and backpressure
design.

### Alarms — separate fan-out architecture

The single-subscriber rule never applied to alarms. The gateway runs an
always-on central alarm monitor (`GatewayAlarmMonitor`) that owns one
gateway-managed worker session, caches the active-alarm set, and fans it out to
any number of clients through the session-less `StreamAlarms` RPC.
`AcknowledgeAlarm` is session-less and routes through the monitor. Rationale:
alarm state is gateway-wide, not session-scoped — every client wants the same
current set plus updates, and forcing each to own a worker would multiply AVEVA
polling load for no benefit.

### Alarms — a capped snapshot fetch never implies a clear

Decision (2026-08-15): when the worker's `GetXmlCurrentAlarms2` fetch comes back
holding exactly `MxGateway:Alarms:MaxAlarmsPerFetch` records, the worker treats
the snapshot as **truncated** and merges it into the retained snapshot instead
of replacing it. Alarms the capped reply did carry update normally; alarms it
had no room to mention are retained untouched.

The COM API caps its reply at `maxAlmCnt` and exposes no "more available" flag,
so a reply sitting exactly on the cap is indistinguishable from a galaxy that
happens to hold exactly that many active alarms. Both are treated as truncated,
because the two error directions are not symmetric.

Nothing in the worker emits a Clear transition. The clear is an **inference**:
`WnWrapAlarmConsumer.ComputeTransitions` produces no transition for an alarm
that disappears from the snapshot, and `GatewayAlarmMonitor.ApplyReconcile`
later diffs its cache against `SnapshotActiveAlarms()` and broadcasts a Clear
for every cached alarm the worker no longer reports. Before this decision, a
capped fetch shrank that snapshot, so every alarm past the cap was broadcast as
cleared while still standing — a silent, galaxy-wide false clear on exactly the
alarm floods where the cap is reached.

Consequences, and how this sits with the existing failover/reconcile design:

- **The suppression is an eviction guard, not a transition filter.** It lives in
  the snapshot update inside `PollOnce`, not in `ComputeTransitions`, which was
  never going to emit anything for a disappearance. The reconcile/dedup
  machinery (`_clearedByReconcile` tombstones, the NEXT-03 duplicate-Clear
  suppression) is untouched: it still sees the same shape of snapshot, only
  with the truncated poll's unmentionable alarms still present.
- **It preserves at-least-once, idempotent application.** The failure mode
  becomes bounded staleness — a genuinely cleared alarm can linger until the
  first sub-cap fetch evicts it, and the reconcile then broadcasts its Clear
  late. A late Clear is repaired by the next complete poll; a Clear that never
  happened is broadcast to every `StreamAlarms` subscriber and cannot be taken
  back. Consumers already apply transitions as "set this alarm to this state",
  so a repeated or delayed Clear is absorbed.
- **Under *sustained* truncation, some intermediate history is lost — end state
  is not.** For an alarm that stays outside the fetch window, a full
  clear→re-raise cycle that begins and ends between two sightings emits **no
  transitions at all**: the retained record is identical before and after, so
  the diff sees nothing to report. Consumers that render current state are
  correct; consumers that *count occurrences* lose an event. Likewise, an
  operator acknowledgement of an out-of-window alarm does not reach the feed
  until that alarm re-enters a fetch window, at which point the reconcile
  repairs the acked state. This is a strictly better failure than the
  pre-guard behaviour (which fabricated a Clear for every out-of-window alarm
  on every poll), but it is not lossless, and it is another reason a
  persistently truncating deployment is a configuration defect to fix rather
  than a mode to run in.
- **It does not synthesize anything.** Suppressing an inference is the opposite
  of inventing an event; no transition is fabricated on a truncated poll.
- **Failover is unaffected.** `FailoverAlarmConsumer` selects which
  `IMxAccessAlarmConsumer` is live; the guard is internal to the wnwrap
  consumer's own snapshot bookkeeping and changes neither the failure counting
  that triggers failover nor the subtag standby's snapshot, which is built from
  a bounded watch-list and has no per-fetch cap to hit.
- **Operators get told, weakly.** A truncated poll logs a rate-limited (once
  per minute) `AlarmSnapshotTruncated` warning carrying the cap, the record
  counts, and the running truncated-fetch total — identifiers and counts only,
  never tag names, values, limits, or comments. Be honest about its reach: it
  goes to the worker's console/stderr, which is captured on dev hosts but is
  not a metric, not a dashboard tile, and not part of any session-status or
  alarm-feed payload, so a production deployment can truncate indefinitely
  without anyone noticing. Surfacing truncation as a **structural** degraded
  status (a field on the alarm-provider mode/status surface the dashboard and
  `StreamAlarms` consumers already read) is filed as a follow-up; until it
  lands, the log line is the only signal. A galaxy that truncates persistently
  is a configuration problem: raise `MxGateway:Alarms:MaxAlarmsPerFetch`.

## Session-Resilience Epic Scope

Decision (2026-07-09, archreview TST-04; migrated here 2026-08-07 from the retired
`oldtasks.md` mirror per TST-29): the session-resilience epic
(`docs/plans/2026-06-15-session-resilience.md`, 28 tasks) resolves into three per-phase
decisions rather than one open backlog.

- **Phase 3 (reconnect)** — essentially complete. Task 13 (owner re-validation) shipped
  as archreview **TST-02** (P0, session attach is owner-scoped; see
  [Session Reconnect](#session-reconnect) above). Task 15 (reconnect integration test)
  shipped as **TST-01** (`GatewayEndToEndReconnectReplayTests`). Task 14 (client
  `ReplayGap` handling) shipped as **CLI-15** for four of five clients
  (.NET/Go/Rust/Python); the Java client is the only remainder.
- **Phase 4 (per-session dashboard ACL)** — scoped, not yet built. Tracked as archreview
  **TST-15**. The Viewer-default decision is settled: admin-sees-all, Viewer strictly
  scoped to sessions it owns or is granted — matching the gRPC owner-binding decision in
  [Session Reconnect](#session-reconnect) above, for consistency between the gRPC and
  dashboard surfaces.
- **Phase 5 (orphan-worker reattach)** — deferred, not planned. It would reverse the
  "Gateway restart does not reattach orphan workers" invariant (see CLAUDE.md), adding a
  stable gateway-instance id, an adoption-manifest SQLite store, a worker phone-home
  reconnect protocol, and gateway-side adoption (re-open pipes, nonce-validate, reject
  impostors). It stays deferred unless a concrete requirement appears; the invariant
  stands. **`EnableOrphanReattach` does not exist and must not be referenced anywhere as
  if it does** until that task actually lands.

`docs/plans/2026-06-15-session-resilience.md.tasks.json` remains the sole resume state
for the still-pending Phase 4 tasks (16-19) and the deferred Phase 5 tasks (20-28) — one
authority, no mirror.

## Authentication

Decision: API key authentication for the public gateway.

API keys are stored in a gateway-owned SQLite database. Store hashed API key
secrets only; never store raw key material.

Recommended client format:

```text
authorization: Bearer mxgw_<key-id>_<secret>
```

Recommended SQLite tables:

```sql
CREATE TABLE api_keys (
    key_id TEXT PRIMARY KEY,
    key_prefix TEXT NOT NULL,
    secret_hash BLOB NOT NULL,
    display_name TEXT NOT NULL,
    scopes TEXT NOT NULL,
    created_utc TEXT NOT NULL,
    last_used_utc TEXT NULL,
    revoked_utc TEXT NULL
);

CREATE TABLE api_key_audit (
    audit_id INTEGER PRIMARY KEY AUTOINCREMENT,
    key_id TEXT NULL,
    event_type TEXT NOT NULL,
    remote_address TEXT NULL,
    created_utc TEXT NOT NULL,
    details TEXT NULL
);
```

Recommended scopes:

- `session:open`
- `session:close`
- `invoke:read`
- `invoke:write`
- `invoke:secure`
- `events:read`
- `metadata:read`
- `admin`

Hashing recommendation:

- Use HMAC-SHA256 with a gateway-local secret/pepper stored outside SQLite, or
  use Argon2id if a suitable dependency is already accepted.
- Compare hashes using constant-time comparison.
- Log only the key id or prefix, not the raw key.

Storage recommendation:

- Default SQLite path should be under `ProgramData` or another configured
  gateway data directory.
- Apply restrictive filesystem ACLs for the gateway service identity and
  administrators.
- Require TLS when the gateway is reachable off-machine.

## Audit Pipeline

Decision: audit is asynchronous, bounded, and swept.

The canonical `IAuditWriter` contract has always been best-effort — a failed audit write is
logged and swallowed so it cannot abort the action that produced it. The registered writer is
`ChannelAuditWriter`, which makes the cost of that promise explicit: a producer enqueues onto a
4096-event bounded channel and returns, and `AuditDrainService` commits up to 64 buffered events
per transaction. This exists because constraint denials are emitted per denied tag inside bulk
RPC loops: a partially denied 1,000-tag request previously awaited 1,000 sequential SQLite
inserts — each re-running `CREATE TABLE IF NOT EXISTS` — against the same database file every
authenticated call reads. The schema bootstrap now runs once, from the drain's `StartAsync`.

When the channel is full the newest event is dropped and counted rather than blocking the
producer: a stalled audit database must cost audit completeness, not gateway availability. Drops
are logged once and reported in aggregate on each sweep. Shutdown drains what is buffered under a
2-second cap.

Every other failure mode degrades to synchronous writes rather than to silent loss. The writer
falls back to the direct path whenever nothing is draining: before the drain attaches, after it
detaches, where no hosted service runs at all (the `apikey` admin CLI), and when the channel has
been completed — so no attach/detach sequence can leave producers filling a buffer with no reader.
If the drain loop itself dies it detaches the writer on the way out, which reverts every producer
to the direct path. A batch that will not commit is retried one event at a time, so an unwritable
row costs only itself instead of the up-to-63 good events sharing its transaction.

**All** audit is channelled, including admin and CRUD records — dashboard key create/revoke/rotate,
session Close/Kill, and the library-forwarded API-key lifecycle entries. The alternative considered
was keeping those on the synchronous writer and channelling only high-volume denial audit. It was
rejected because a single dashboard key-create emits two records through two different seams (the
library's `create-key` via `IApiKeyAuditStore`, and the enriching `dashboard-create-key` via
`IAuditWriter`); splitting them across two durability regimes gives an auditor a per-producer
matrix to reason about instead of one rule. The residual exposure is explicit: **if the gateway
process dies between the enqueue and the batch commit, buffered audit events are lost.** The window
is bounded by drain latency — the drain wakes on every write and commits immediately, so it is
sub-millisecond under normal load — and it does not apply to the `apikey` CLI, which writes
synchronously. Audit is a best-effort record of what the gateway did, not a write-ahead log of what
it is about to do; a deployment that needs crash-durable admin audit should ship the events off-box
rather than rely on this table.

`MxGateway:Security:AuditRetentionDays` (default 90, minimum 1) bounds the table: the drain sweeps
at startup and hourly, deleting older rows. Retention cannot be configured off. The sweep compares
through SQLite's `datetime()` rather than on the stored ISO-8601 text. Text comparison is correct
only while every row is UTC-normalized — which the canonical model guarantees for rows written
through the store, but not for rows that entered the table any other way — and on a mixed-format
column it silently deletes live audit, because `2026-05-17T09:00:00-05:00` is two hours after a
`2026-05-17T12:00:00+00:00` cutoff yet sorts before it. Comparing instants is correct however the
text got there, and a timestamp `datetime()` cannot parse yields NULL, so undateable audit is kept
rather than swept.

## Authorization

Decision: start with scope checks by command category.

Suggested mapping:

- `OpenSession`: `session:open`
- `CloseSession`: `session:close`
- `Register`, `Unregister`, `AddItem`, `AddItem2`, `RemoveItem`, `Advise`,
  `UnAdvise`, `AdviseSupervisory`, `AddBufferedItem`,
  `SetBufferedUpdateInterval`, `Suspend`, `Activate`: `invoke:read`
- `Write`, `Write2`: `invoke:write`
- `WriteSecured`, `WriteSecured2`, `AuthenticateUser`,
  `ArchestrAUserToId`: `invoke:secure`
- `StreamEvents`: `events:read`
- Galaxy SQL metadata endpoints if added: `metadata:read`
- worker shutdown diagnostics and key management: `admin`

## Worker Process Identity

Decision: run workers as the gateway service identity for v1.

Rationale: this avoids early COM/DCOM permission failures and keeps the first
implementation focused on MXAccess parity. The worker launcher should keep an
extension point for a restricted service account later.

## Event Backpressure

Decision: fail-fast bounded queues for v1 and parity testing.

If worker or gateway event queues fill, fault the session. Do not silently drop
or coalesce events in parity mode.

Rationale: event drops would hide parity defects. Production coalescing by item
handle can be added later as an explicit opt-in mode once event rates are
measured.

## Event-Rate Target

Decision: do not set a production event-rate target before measurement.

For v1, expose queue depth, event rate, stream send latency, and overflow
metrics. Keep bounded queues and fail-fast behavior. Use observed load from live
systems to set a later coalescing or scaling target.

## Command Batching

Decision: no public command batching for v1.

Use one command per request so replies, HRESULTs, status arrays, event ordering,
and failure behavior are easy to compare against direct MXAccess.

Batch tag registration can be added later if measured setup latency requires it.

## Bulk Command Family

Decision: the gateway exposes a fixed set of *bulk* command kinds —
`AddItemBulk`, `AdviseItemBulk`, `RemoveItemBulk`, `UnAdviseItemBulk`,
`SubscribeBulk`, `UnsubscribeBulk`, `WriteBulk`, `Write2Bulk`,
`WriteSecuredBulk`, `WriteSecured2Bulk`, `ReadBulk` — that carry a list of
entries in one round-trip and return one per-entry result. Each command kind
runs the corresponding single-item MXAccess COM call sequentially on the
worker STA; per-entry failures populate `was_successful = false` with the
underlying HRESULT and never throw. There is no transactional / fail-fast
semantic — bulk here means "one round-trip, per-entry results", not
"atomic".

Rationale: MXAccess COM itself has no native bulk API for any of these
operations. Surfacing the per-entry result list keeps parity transparent —
the caller sees the same per-item HRESULT they would see calling MXAccess
N times directly — while the bulk shape collapses the gateway/IPC overhead
to one round-trip per batch and lets the worker keep the STA hot.

`ReadBulk` is the only bulk command without a 1:1 MXAccess analogue. Two
choices were considered:

1. **Cache-then-snapshot** (chosen): when a requested tag is already in the
   session's item registry AND advised, the worker returns the last cached
   `OnDataChange` value without touching the subscription
   (`was_cached = true`). Otherwise it takes the full `AddItem + Advise +
   wait-for-first-OnDataChange + UnAdvise + RemoveItem` lifecycle itself
   (`was_cached = false`) and leaves the session exactly as it was before
   the call. The cache lives on a per-session `MxAccessValueCache`,
   populated by `MxAccessBaseEventSink` on every `OnDataChange` after the
   event clears the outbound queue.

2. **Always-snapshot**: take the AddItem-through-RemoveItem lifecycle for
   every requested tag. Cleaner conceptually but pays the full lifecycle
   cost on every call and would interfere with existing subscriptions if
   MXAccess reuses item handles.

The chosen behavior matches what callers actually want from "current
value" — a free read of an already-streaming tag, and a one-shot snapshot
otherwise — and never disturbs subscriptions the caller did not create.
The decision intentionally does NOT synthesize an `OnDataChange` event
from the snapshot path: the snapshot value reaches the caller through
`ReadBulk`'s reply payload only, not through the event stream. This
preserves the "Don't synthesize events" rule that scopes the rest of the
worker.

`ReadBulk`'s wait loop pumps Windows messages on the worker STA
(`StaRuntime.PumpPendingMessages`) on every poll iteration so the inbound
MXAccess COM event can dispatch while the bulk executor still holds the
thread — without the pump the OnDataChange would never deliver.

## Graceful Worker Shutdown

Decision: best-effort cleanup before COM release.

During graceful shutdown, the worker should attempt:

1. `UnAdvise` for advised items.
2. `RemoveItem` for active item handles.
3. `Unregister` for active server handles.
4. Event detach.
5. COM release.

Failures during cleanup should be logged and preserved diagnostically, but the
gateway may still kill the worker after shutdown timeout.

## OperationComplete

Decision: model and forward `OperationComplete` only when native MXAccess fires
it. Do not synthesize `OperationComplete` from writes, command replies, ASB
completion queues, or other status frames.

Rationale: the event signature is known, but the MXAccess analysis has not yet
captured the runtime condition that triggers the public event. Synthesizing it
would risk breaking parity.

## Buffered Data Change

Decision: include `OnBufferedDataChange` in the protocol and worker event
model, but treat multi-sample payload conversion as capture-validated work.

The event signature and native path are known. A live buffered sample batch has
not yet been observed. Until then, preserve raw value, quality, timestamp, data
type, and status metadata whenever conversion is incomplete.

## Completion-Only Status Mapping

Decision: preserve completion-only operation-status bytes as raw diagnostic
metadata unless native MXAccess raises a public event or the MXAccess analysis
proves an exact `MXSTATUS_PROXY[]` mapping.

Do not guess status category/source/detail values for frames that MXAccess does
not expose through its public COM events.

## API Key Administration

Decision: v1 API key management is a local administrative CLI/tool, not a
public admin API.

The tool should support:

- initialize auth database,
- create key,
- list keys without showing secrets,
- revoke key,
- rotate key,
- print the raw secret exactly once at creation.

Public gRPC key-management endpoints can be added later only behind `admin`
scope and TLS.

## SQLite Migrations

Decision: use simple startup migrations with a `schema_version` table.

Recommended table:

```sql
CREATE TABLE schema_version (
    id INTEGER PRIMARY KEY CHECK (id = 1),
    version INTEGER NOT NULL,
    applied_utc TEXT NOT NULL
);
```

Migrations should be idempotent, run inside transactions, and fail gateway
startup if the database is newer than the running binary understands.

## Web Dashboard

Decision: host a basic gateway dashboard with Blazor Server and Bootstrap
CSS/JS.

The dashboard should show gateway health, active sessions, worker instances,
basic metrics, queue depths, and recent faults. It should update in real time
through Blazor Server component updates.

Allowed UI stack:

- Blazor Server,
- Bootstrap CSS,
- Bootstrap JavaScript,
- small local CSS.

Do not use MudBlazor or other Blazor UI component libraries for v1.

Dashboard access should require API-key-backed dashboard authentication with
`admin` scope when enabled. For local development, anonymous localhost access
is enabled by default through `Dashboard:AllowAnonymousLocalhost`; the bypass is
limited to loopback requests.

## Lazy Browse Is Wire-Only

Decision: the gateway continues to pull the full Galaxy hierarchy on each
deploy. `BrowseChildren` and the lazy dashboard render only avoid sending and
DOM-materializing the full tree — they do not push laziness into SQL or cache
loading.

Rationale: snapshot persistence and the dashboard summary both depend on a
fully-materialized cache. Lazy SQL would increase per-click latency on a
deployment-heavy box, multiply per-session SQL connections, and complicate the
cold-start path. Wire-side laziness solves the actual pain (oversized gRPC
replies and a heavy DOM) without disturbing the materialization model.

## TLS Auto-Certificate and Lenient Client Trust

Decision: when a Kestrel `https://` endpoint is configured without a certificate
of its own (and no `Kestrel:Certificates:Default` is set), the gateway generates
and persists a self-signed certificate rather than failing to start. Clients
connecting over TLS without a pinned CA accept whatever certificate the server
presents by default; pinning a CA restores full verification.

Rationale: `mxaccessgw` is an internal tool with no PKI to issue or distribute
certificates. The prior behavior — an `https` endpoint with no certificate
fails at startup with Kestrel's opaque "no server certificate was specified"
error — pushed operators toward plaintext (`h2c`), exposing the API key and
request payloads on the wire. Auto-generating a long-lived, persisted, reused
certificate lets TLS "just work" with zero certificate management, while the
lenient client default means clients connect to that self-signed certificate
without a manual trust step. Both choices are deliberate, not oversights:
strict-by-default would force PKI work this tool does not warrant. Plaintext-only
deployments are untouched — no certificate or key material is written for them —
and an operator who supplies a real certificate transparently overrides the
generated one.

Two clients diverge from "accept any certificate" because their gRPC stacks lack
a per-channel skip-verify hook:

- Python uses trust-on-first-use: it fetches the server's presented certificate
  over a separate unverified probe and pins it for the channel, and defaults the
  SNI/target-name override to `localhost` (the generated certificate always
  carries a `localhost` SAN).
- Rust is pin-only: tonic exposes no public hook to inject a custom certificate
  verifier, so TLS over Rust requires either a pinned CA or an explicit opt-in to
  system-trust verification; otherwise connecting returns a clear, actionable
  error.

See [Gateway Configuration — Automatic self-signed certificate](./GatewayConfiguration.md#automatic-self-signed-certificate)
and the per-client READMEs for the as-built behavior.

## Alarm-Manager to Subtag Fallback

Decision: add a second alarm provider (subtag monitoring) that the worker
activates automatically when the native wnwrap alarm manager fails, and fails
back to automatically when the manager recovers.

### Worker-side synthesis

Synthesis of alarm transitions from subtag value changes happens entirely in
the worker (`SubtagAlarmConsumer` / `SubtagAlarmStateMachine`). The gateway
still forwards only events the worker emits and synthesizes nothing itself.
This satisfies the parity rule even though the subtag path is inherently
non-parity: the parity rule governs where synthesis lives, not whether
synthesis is permitted when the native source is unavailable.

### Degraded is explicit

Every subtag-mode transition carries `degraded = true` on the
`OnAlarmTransitionEvent` and `ActiveAlarmSnapshot` proto messages, and the
`AlarmFeedMessage` feed carries an `AlarmProviderStatus` payload on stream
open and on every switch. No client can mistake a subtag-mode alarm for an
authoritative alarmmgr record. Subtag mode has lower fidelity: synthetic
deterministic GUID (SHA-derived from the alarm reference), best-effort
original-raise timestamp, narrower field set. Clients that need full fidelity
must wait for failback.

### Failover trigger

The failover trigger is N consecutive wnwrap COM failures — a `COMException`
thrown by `Subscribe` or `PollOnce`, or a failure HRESULT from
`GetXmlCurrentAlarms2`. A single poll failure does not trigger a switch; the
threshold (default 3, floored at 1) guards against transient COM hiccups. The
counter resets on any clean poll so a flapping provider does not permanently
latch in subtag mode.

### Acknowledge via ack-comment write

In subtag mode, `AcknowledgeAlarm` writes the operator comment to the alarm
attribute's ack-comment subtag (`Fallback:Subtags:AckComment`). The write
performs the native ack in AVEVA. This differs from alarmmgr mode, where
`AlarmAckByName` on `wwAlarmConsumerClass` is called directly. The `AckComment`
subtag name is empty by default; configuring it is required for ack to work in
subtag mode. The exact AVEVA subtag names are not hard-coded — the `Subtags`
config block exists precisely so names are not guessed without validation
against the live MXAccess attribute set.

### Related documentation

- [Gateway Configuration — Alarm Fallback options](./GatewayConfiguration.md#alarm-fallback-options)
- [Alarm Client Discovery — Subtag provider](./AlarmClientDiscovery.md)
- [gRPC Contract — provider_status and degraded fields](./Grpc.md)

## Write Completion Correlation

MXAccess writes are fire-and-forget: the toolkit call returns before the
Galaxy commit, and the per-item outcome only exists in the later
`OnWriteComplete` COM callback. The original unary write reply therefore
proved worker-side command acceptance only, forcing consumers (OtOpcUa's
GalaxyDriver) to report every write as provisionally good.

For the unary write kinds (`Write`/`Write2`/`WriteSecured`/`WriteSecured2`)
the worker now holds the unary reply for a
bounded window (`MxGateway:Worker:WriteCompletionWaitMilliseconds`, default
1.5 s, `0` disables; conveyed to the worker via
`MXGATEWAY_WORKER_WRITE_COMPLETION_WAIT_MS`) and copies the matching
callback's status rows onto `MxCommandReply.statuses`. Key choices, argued in
[the design doc](./plans/2026-08-09-write-completion-correlation-design.md):

- **Pump-wait on the STA, not a parked reply.** The executor holds the STA
  thread but pumps Windows messages each poll — the shipped ReadBulk pattern —
  because commands serialize per session anyway, so freeing the STA during the
  wait buys nothing and a parked reply would change the dispatcher/pipe
  contracts.
- **Version baseline before the COM call** closes the fast-completion edge: a
  callback that dispatches while `WriteSecured` is still on the stack still
  correlates.
- **Timeout returns today's shape** (protocol OK, empty statuses):
  unconfirmed is honest; a synthesized failure row would trigger consumer-side
  write-revert logic on slow-but-successful commits. The 1.5 s default stays
  inside OtOpcUa's 2 s Tier A write-resilience budget.
- **Parity preserved.** `protocol_status`/`hresult` keep describing
  acceptance; the MXAccess outcome (success or failure) rides only in
  `statuses[0]`; the `OnWriteComplete` event still streams unchanged (nothing
  swallowed, nothing synthesized).
- **Scope: all four unary write kinds; bulk writes stay fire-and-forget.**
  The first cut correlated `WriteSecured`/`WriteSecured2` only, but OtOpcUa's
  dominant FreeAccess write path goes out as plain `Write` (2026-08-09 live
  verification, 06/S-1) — a refused plain write was invisible on the reply.
  Plain `Write`/`Write2` now correlate identically. Bulk writes keep
  fire-and-forget replies: waiting per entry would add a device round-trip per
  item to high-rate supervisory loops.
- **Best-effort correlation.** The callback carries only
  `(hItem, statuses)` — no transaction id — so concurrent writes to the same
  item within the window can swap rows; benign for the serialized single-write
  consumer contract.
- **Client cancellation needs no special path**: a caller abandoning the RPC
  mid-wait leaves the worker to finish its bounded wait and reply; the gateway
  discards the reply, the session is never faulted.

## Later Revisit Items

These are explicit post-v1 revisit items, not open blockers:

- restricted worker service account,
- production coalescing by item handle,
- command batching for high-volume tag setup.

The following items were previously listed here and have since shipped:

- **Reconnectable sessions with replay** — shipped, config-gated via
  `MxGateway:Sessions:DetachGraceSeconds` and
  `MxGateway:Events:ReplayBufferCapacity` / `ReplayRetentionSeconds`.
  See [Session Reconnect](#session-reconnect) above and [Sessions](./Sessions.md).
- **Multiple event subscribers per session** — shipped, config-gated via
  `MxGateway:Sessions:AllowMultipleEventSubscribers` and
  `MxGateway:Sessions:MaxEventSubscribersPerSession`.
  See [Event Subscribers](#event-subscribers) above and [Sessions](./Sessions.md).

## Related Documentation

- [Gateway Process Detailed Design](./GatewayProcessDesign.md)
- [MXAccess Worker Instance Detailed Design](./MxAccessWorkerInstanceDesign.md)
- [Authentication](./Authentication.md)
- [Authorization](./Authorization.md)
- [Galaxy Repository](./GalaxyRepository.md)