feat(galaxy): consume the gateway's correlated write statuses; close archreview 06/S-1

The mxaccessgw OnWriteComplete correlation is live (server b948e69, worker
53f69cd — all four unary write kinds), so every write reply's statuses[0] now
carries the real COM commit outcome. Live-gating the cutover on windev found
and fixed three defects:

- StatusCodeMap.FromMxStatus truncated the correlated failure row's MxStatus
  detail code to a byte (1007 & 0xFF = 239 -> OPC DA Good band), mapping a
  REFUSED write to Good. New invariant: a row that is not IsSuccess() never
  maps Good — details > 0xFF (and contradictory Good-band substatuses) route
  by category (SECURITY_ERROR -> BadUserAccessDenied, named const covered by
  StatusCodeParityTests; COMMUNICATION/CONFIGURATION keep their codes).

- The borrow-from-subscription-registry write optimization could never commit:
  a subscription handle is plain-advised, and MXAccess refuses both the
  no-login write through it (SECURITY_ERROR detail 1008, live-captured) and
  the supervisory advise upgrade (E_INVALIDARG) — one advise per HANDLE, not
  per attribute. Removed: writer subscribedHandleSource ctor param + borrow
  branch, GalaxyDriver wiring, SubscriptionRegistry.TryResolveItemHandle +
  its forward index and test file. The writer always AddItems its own
  supervisory-advised handle, which commits fine while the same attribute is
  subscribed (live-proven; pinned by the new write-while-subscribed smoke).

- (gateway-side, fixed there) the correlation initially covered WriteSecured
  only, while the dominant no-login path emits plain Write.

Live gate, all legs PASSED against windev: happy write Good off a confirmed
status row with galaxy.writes.unconfirmed + advise_failed both FLAT; read-only
refusal surfaces Bad-family (the #5 node-revert input Galaxy could never
produce before); WriteCompletionWaitMilliseconds=0 control -> provisional Good
+ metered unconfirmed (override removed after, restoration re-verified). New
env-gated smokes: Live_correlated_write_statuses_confirm_commit_and_surface_refusal,
Live_write_to_subscribed_tag_commits_via_own_handle (replaces the borrow
smoke), GatewayGalaxyWriteMeterLiveTests (run filtered — meters are
process-wide).
This commit is contained in:
Joseph Doherty
2026-08-09 20:11:55 -04:00
parent 1038c0e23c
commit 20a816dbf4
11 changed files with 337 additions and 384 deletions
+2 -2
View File
@@ -643,11 +643,11 @@ The OtOpcUa Client CLI at `src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI` supports `-F
| `otopcua.redundancy.primary_gate_denied` | Counter (`{denial}`) | Operations denied by the Primary data-plane gate (archreview 03/S4). Tagged `site` = `write` \| `ack` \| `alarm-emit` and `reason` = `secondary` \| `detached` \| `role-unknown`. A non-zero `role-unknown` rate on a healthy multi-node cluster flags a stuck boot window (redundancy snapshot not delivering). |
| `otopcua.opcua.apply.failed` | Counter (`{apply}`) | Address-space apply/materialise passes that swallowed at least one sink failure (archreview 01/S-1). Tagged `kind` = `rebuild` (the rebuild threw) \| `nodes` (per-node materialise failures). A non-zero rate means the running server holds a stale or partial address space despite a reported-successful deploy — `OpcUaPublishActor` also logs the degraded apply at **Error**. |
| `galaxy.writes.advise_failed` | Counter (`{write}`) | Galaxy writes short-circuited to `Bad` because `AdviseSupervisory` failed — the value could not have committed, so the write **fails closed** (archreview 06/S-1) and the Bad status fires the #5 node revert instead of leaving a phantom-Good node. On the Galaxy driver meter `ZB.MOM.WW.OtOpcUa.Driver.Galaxy`. |
| `galaxy.writes.unconfirmed` | Counter (`{write}`) | Galaxy writes reported `Good` off an **empty** gateway statuses array (command accepted; the COM-side commit is unconfirmed pending the mxaccessgw `WriteComplete` correlation follow-up). The honest signal for the optimistic-Good rate until that cross-repo fix lands. On the Galaxy driver meter. |
| `galaxy.writes.unconfirmed` | Counter (`{write}`) | Galaxy writes reported `Good` off an **empty** gateway statuses array (worker-side acceptance only — the COM commit is unconfirmed). Against a correlated gateway (mxaccessgw server ≥ `b948e69`, worker ≥ `53f69cd` — every unary write reply carries the worker's `OnWriteComplete` outcome as `statuses[0]`) this stays **flat**; an increment means the bounded completion wait (`MxGateway:Worker:WriteCompletionWaitMilliseconds`, default 1500 ms, deliberately under the 2 s Tier A Write timeout) timed out, or the gateway pre-dates the correlation. On the Galaxy driver meter. |
The redundancy/apply meters are defined on `OtOpcUaTelemetry` (`src/Core/ZB.MOM.WW.OtOpcUa.Commons/Observability/OtOpcUaTelemetry.cs`); the Galaxy write meters hang off the driver's own meter (`EventPump.MeterName`). All surface through whatever OpenTelemetry exporter the host configures.
**Galaxy fail-closed write semantics (archreview 06/S-1):** a non-secured Galaxy write commits only when its item is `AdviseSupervisory`-advised first (the writer runs with no login). If the advise fails, the write is **not** issued — `GatewayGalaxyDataWriter` returns `BadCommunicationError` (a knowingly-lost write is never reported as Good). An empty statuses array on the write reply is still treated as `Good` but metered as `galaxy.writes.unconfirmed`. See [Historian.md](Historian.md) / the Galaxy driver notes for the write path.
**Galaxy fail-closed write semantics (archreview 06/S-1, closed 2026-08-09):** a non-secured Galaxy write commits only when its item is `AdviseSupervisory`-advised first (the writer runs with no login) — live-confirmed: a write through a merely plain-advised handle is refused by the automation object (`SECURITY_ERROR` detail 1008 on the correlated reply). If the advise fails, the write is **not** issued — `GatewayGalaxyDataWriter` returns `BadCommunicationError` (a knowingly-lost write is never reported as Good). With the gateway-side `OnWriteComplete` correlation live, the write reply's `statuses[0]` carries the real COM commit outcome, so a refused write surfaces as a Bad-family code and fires the #5 node revert — the writer always AddItems its **own** supervisory-advised handle (the former borrow-from-subscription-registry optimization was removed: MXAccess enforces one advise per handle, so a subscription's plain-advised handle can neither commit a no-login write nor be upgraded). An empty statuses array (completion-wait timeout / pre-correlation gateway) is still treated as `Good` but metered as `galaxy.writes.unconfirmed`. See [Historian.md](Historian.md) / the Galaxy driver notes for the write path.
## Depth reference