docs(tracking): record WRK-21 review follow-ups (monotonic budget, guarded fallback) with windev evidence

fix(WRK-21): make drain budget monotonic at the reserve boundary; guard the reply-too-large fallback
Review follow-ups on the WRK-21 cluster. 1. ResolveDrainReplyByteBudget was a step function, not a floor: just above the 64 KiB reserve the budget collapsed to a few bytes (at the validator-permitted floor MaxMessageBytes = 1024 + 64 KiB it was exactly 1024), too small to move a byte-heavy event, so DrainEvents truncated on every call and the drain-until- empty loop never terminated. It now takes the max of (frameMax - reserve) and frameMax/2, so the budget is monotonic and never below half the frame max. New test DrainEvents_AtValidatorFloorFrameMax_MakesProgressAndTerminates drives a byte-heavy queue at the exact validator floor and asserts it drains to empty with no head ever reported oversized. 2. The reply-too-large fallback write is now itself size-guarded (WriteReplyTooLargeFallbackAsync, used by both the control and STA reply seams): at a pathologically tiny negotiated max below the gateway's floor the fallback could also throw MessageTooLarge and — uncaught — kill the session, defeating the "no diagnostics command is session-fatal" invariant. It now log-and-swallows; comment notes WRK-24 adds the negotiated-max lower bound that makes it unreachable. 3. Corrected the RepeatedFieldOverheadBytes doc comments: WorkerEvent.CalculateSize() already includes the event's tag and length prefix (the same shape the reply's repeated events field packs), so the 8 bytes is pure slack over an already- conservative estimate, not compensation for a missing wrapper.
2026-08-07 07:13:46 -04:00 · 2026-08-07 07:09:13 -04:00 · 2026-08-07 06:54:24 -04:00 · 2026-08-07 06:50:17 -04:00 · 2026-08-07 06:25:15 -04:00 · 2026-08-07 05:38:23 -04:00
29 changed files with 1490 additions and 92 deletions
@@ -194,15 +194,17 @@ be **deleted**. **Keep** the mxaccessgw-specific ones that exercise behavior the
  `PersistSnapshot`, but the deployments must carry `MxGateway__Galaxy__SnapshotCachePath` and
  `MxGateway__Galaxy__PersistSnapshot` in their NSSM env on redeploy, or snapshot persistence silently
  no-ops in production.
- **Pre-existing NU1903 (unrelated):** adding the package surfaced a transitive `SQLitePCLRaw.lib.e_sqlite3`
+- **Pre-existing NU1903 (unrelated) — ✅ RESOLVED (2026-07-18, commit `2f0cfe3`):** adding the package surfaced a transitive `SQLitePCLRaw.lib.e_sqlite3`
-  2.1.11 advisory (GHSA-2m69-gcr7-jv3q, no upstream patch) that breaks the build under `TreatWarningsAsErrors`
+  2.1.11 advisory (GHSA-2m69-gcr7-jv3q, at the time no upstream patch) that breaks the build under `TreatWarningsAsErrors`
-  — already red on `main`. Resolved with a targeted `NuGetAuditSuppress` in `src/Directory.Build.props`
+  — already red on `main`. Initially resolved with a targeted `NuGetAuditSuppress` in `src/Directory.Build.props`
-  (its own commit). Remove the suppression once a patched e_sqlite3 ships.
+  (its own commit). The patched e_sqlite3 (2.1.12) has since shipped: the suppression was **removed** and the
- **Pre-existing IntegrationTests break (unrelated, NOT fixed here):** `IntegrationTests/WorkerLiveMxAccessSmokeTests.cs`
+  patched native lib pinned intentionally (`src/Directory.Build.props` now documents this in place of the suppression).
-  constructs `EventStreamService` with 6 ctor args, but a prior event-stream refactor reduced that ctor — so
+- **Pre-existing IntegrationTests break (unrelated, NOT fixed here) — ✅ RESOLVED since:** `IntegrationTests/WorkerLiveMxAccessSmokeTests.cs`
-  the IntegrationTests project does not compile (already broken on `main`, independent of Galaxy). The Galaxy
+  constructed `EventStreamService` with 6 ctor args, but a prior event-stream refactor reduced that ctor — so
-  live tests there were rebound to the lib and compile in isolation, but the project won't build until that
+  the IntegrationTests project did not compile (already broken on `main`, independent of Galaxy). The call site
-  unrelated call site is fixed. Track separately.
+  has since been fixed to match the 3-arg ctor (`sessionManager, options, metrics` —
  `Grpc/EventStreamService.cs:11-14`; call site at `WorkerLiveMxAccessSmokeTests.cs:~1580`) and the project compiles
  (verified 2026-08-07).
 - **No republish needed for the lib test additions:** the browse-projector / deploy-notifier / refresh-service
  tests were added to the lib AFTER 0.2.0 was published; tests aren't shipped, so 0.2.0 is unchanged.
@@ -15,6 +15,42 @@ The architecture is a two-process design — read `gateway.md` before making str
 The worker must do all MXAccess COM calls on its dedicated STA thread, and the STA loop must pump Windows messages (`MsgWaitForMultipleObjectsEx` + `PeekMessage`/`DispatchMessage`) so MXAccess events deliver. A plain blocking queue on an STA is not enough.
 ## Sister Projects (scadaproj umbrella)
 `mxaccessgw` is one of a family of related SCADA / OT / Wonderware / OPC UA **sister
 projects** cloned as sibling directories under `~/Desktop/`. They are **separate repos
 and separate processes**, coupled at runtime over wire protocols (gRPC + OPC UA) — not
 by project/compile references — and share the `ZB.MOM.WW.*` product namespace. **mxaccessgw
 is the linchpin**: the only component that loads 32-bit MXAccess COM, so the others depend
 on it over gRPC rather than touching COM directly.
 - **`~/Desktop/scadaproj`** ([`../scadaproj/CLAUDE.md`](../scadaproj/CLAUDE.md)) — the
  umbrella/index workspace that aggregates the whole family (purpose, location, stack,
  primary commands per project) and hosts the shared `ZB.MOM.WW.*` libraries, the dev/test
  GLAuth (`10.100.0.35:3893`, source of truth `scadaproj/infra/glauth/`), and the shared
  **`ZB.MOM.WW.GalaxyRepository`** package this gateway consumes. **This repo is indexed
  there** — see the MxAccessGateway entry in `../scadaproj/CLAUDE.md`.
 - **`~/Desktop/OtOpcUa`** (`lmxopcua`, [`../OtOpcUa/CLAUDE.md`](../OtOpcUa/CLAUDE.md)) — OPC
  UA server whose in-process `GalaxyDriver` **depends on this gateway** for live Galaxy
  read/write/subscribe and Galaxy Repository browse.
 - **`~/Desktop/ScadaBridge`** ([`../ScadaBridge/CLAUDE.md`](../ScadaBridge/CLAUDE.md)) —
  distributed SCADA platform whose Data Connection Layer has a dedicated **MxGateway
  adapter** that talks to this gateway directly (native MxAccess data + A&C alarms),
  bypassing OtOpcUa.
 - **`~/Desktop/HistorianGateway`** (`historiangw`,
  [`../HistorianGateway/CLAUDE.md`](../HistorianGateway/CLAUDE.md)) — single-process gRPC
  sidecar **patterned on this gateway** (session model, dashboard shell, auth interceptor,
  Galaxy SQL browse) but with no COM / no x86 worker. It also consumes the shared
  `ZB.MOM.WW.GalaxyRepository` package — the same library mxaccessgw adopted (2026-06-25),
  so the `galaxy_repository.v1` wire contract is served from one implementation.
 **Propagate cross-repo changes to the umbrella index.** When a fact the index records about
 mxaccessgw changes here — remote/push status, the `.proto` contracts (`mxaccess_gateway.proto`,
 `mxaccess_worker.proto`, `galaxy_repository.proto`), the two-process architecture, shared-lib
 consumption, or per-project commands — update the **MxAccessGateway entry in
 [`../scadaproj/CLAUDE.md`](../scadaproj/CLAUDE.md)** in the same change so the umbrella index
 never drifts from this repo. (Mirrors the same rule in the peer repos.)
 ## Build, Test, Run
 ```powershell
@@ -14,7 +14,7 @@
 | SEC-06 | Done | **Partial** | Production hard-stop verified: `GatewayOptionsValidator.cs:161-165` (`Transport==None` in Production → startup error). Docs: `docs/GatewayConfiguration.md:244-248` (env-var override `MxGateway__Ldap__ServiceAccountPassword` documented). | **The committed dev service-account password is still in the repo at `appsettings.json:29`** (`Ldap.ServiceAccountPassword`) and has not been rotated — the doc itself says it "should be rotated". The transport guard shipped; the credential-removal/rotation half of the remediation did not. → **SEC-36** |
 | SEC-07 | Done | **Yes** | `Security/Authorization/GatewayGrpcScopeResolver.cs:23` (`QueryActiveAlarmsRequest => GatewayScopes.EventsRead`); both tests now construct the real type (`Tests/Security/Authorization/GatewayGrpcAuthorizationInterceptorTests.cs:330,350`). | |
 | SEC-08 | Done | **Yes** | `Security/Authentication/CachingApiKeyVerifier.cs` (15 s success-only TTL cache keyed on SHA-256 of the presented token, `:96-120`; only successes cached `:110-117`); `Security/Authentication/CoalescingMarkApiKeyStore.cs:76-113` (≤1 `last_used` write/key/60 s); wired as decorators in `Security/Authentication/AuthStoreServiceCollectionExtensions.cs:89-98`; invalidation on dashboard revoke/rotate/delete at `Dashboard/DashboardApiKeyManagementService.cs:104,144,190`; per-call constraints JSON deserialize removed via blob cache (`Security/Authentication/GatewayApiKeyIdentityMapper.cs:22-45`). Tests exist (`Tests/Security/Authentication/CachingApiKeyVerifierTests.cs`). | New surface reviewed in depth — see SEC-34 (staleness/race, bounded) below. |
-| SEC-10 | Done | **Yes** | CLI: `--expires` parsed as relative `<N>d`/`<N>h` or absolute ISO-8601 with `AssumeUniversal|AdjustToUniversal` (`Security/Authentication/ApiKeyAdminCommandLineParser.cs:242-274`), threaded into `CreateKeyAsync` (`:85-117`). Dashboard: `DashboardApiKeySummary.cs:13` (`ExpiresUtc`), snapshot projection `Dashboard/DashboardSnapshotService.cs:277`, badge logic compares against `DateTimeOffset.UtcNow` with a 7-day "Expiring" warn window (`Dashboard/Components/Pages/ApiKeysPage.razor:474-497`; `StatusBadge.razor:12-13`). Verifier-side rejection is in the shared `ZB.MOM.WW.Auth.ApiKeys` 0.1.4 (documented `docs/Authentication.md:64-65`; not readable in this repo). | UTC semantics are correct end-to-end on the gateway side. Boundary note: `expiresAt <= now` shows Expired, and relative parse rejects signed values (`NumberStyles.None`). A just-expired key can still authenticate for ≤15 s via the verification cache — see SEC-34. |
+| SEC-10 | Done | **Yes** | CLI: `--expires` parsed as relative `<N>d`/`<N>h` or absolute ISO-8601 with `AssumeUniversal|AdjustToUniversal` (`Security/Authentication/ApiKeyAdminCommandLineParser.cs:242-274`), threaded into `CreateKeyAsync` (`:85-117`). Dashboard: `DashboardApiKeySummary.cs:13` (`ExpiresUtc`), snapshot projection `Dashboard/DashboardSnapshotService.cs:277`, badge logic compares against `DateTimeOffset.UtcNow` with a 7-day "Expiring" warn window (`Dashboard/Components/Pages/ApiKeysPage.razor:474-497`; `StatusBadge.razor:12-13`). Verifier-side rejection is in the shared `ZB.MOM.WW.Auth.ApiKeys` 0.1.5 (pins bumped 0.1.4→0.1.5 in `e107019`, a transitive-dependency security fix only — the expiry enforcement is unchanged; documented `docs/Authentication.md:64-65`; not readable in this repo). | UTC semantics are correct end-to-end on the gateway side. Boundary note: `expiresAt <= now` shows Expired, and relative parse rejects signed values (`NumberStyles.None`). A just-expired key can still authenticate for ≤15 s via the verification cache — see SEC-34. |
 | SEC-11 | Done | **Yes, with new defects** | Login: fixed-window per-remote-IP limiter policy (`Dashboard/DashboardEndpointRouteBuilderExtensions.cs:27-41`), applied to POST `/auth/login` (`:83`), registered + 429 (`GatewayApplication.cs:111-126`), middleware in pipeline (`GatewayApplication.cs:46`), test `Tests/Gateway/Dashboard/DashboardLoginRateLimitTests.cs`. gRPC: `Security/Authorization/ApiKeyFailureLimiter.cs` checked **before** the store read (`GatewayGrpcAuthorizationInterceptor.cs:72-77`), failure recorded `:89`, reset on success `:97`; interceptor test asserts the short-circuit (`GatewayGrpcAuthorizationInterceptorTests.cs:395-402`). | The limiter exists and is enforced, but its key-id partitioning creates an unauthenticated lockout DoS (**SEC-31**) and its LRU eviction is flushable (**SEC-32**). |
 | SEC-12 | Done | **Yes** | `Dashboard/DashboardSessionAdminService.cs`: canonical `AuditEvent`s `dashboard-close-session`/`dashboard-kill-worker` (`:37-40`), written on Denied (`:65,147`), Success (`:89-96,171-178`), and every Failure arm (`:105,116,132,187,198,214`), category `SessionAdmin`, actor/remote/correlation captured (`:238-261`), via `IAuditWriter` (same store as API-key events). | Audit-event completeness is good: denied, not-found, faulted, and unexpected paths all emit. |
 | SEC-20 | Done | **Yes** | `Metrics/GatewayMetrics.cs:374` — `_heartbeatFailuresCounter.Add(1)` with no `session_id` tag (rationale comment `:370-373`). | The in-memory per-session map remains dashboard-only. |
@@ -43,9 +43,9 @@ Sequenced by cluster; a cluster is one change set.
 | GWC-25 | Medium | S | CLI-35/36 (coord) | Not started | Empty-ring ReplayGap sentinel carries `oldest_available_sequence = 0`, dead-streaming a compliant client |
 | CLI-35 | Medium | S | GWC-25 (coord) | Not started | Python CLI `stream-events` crashes on a ReplayGap |
 | CLI-36 | Medium | S | GWC-25 (coord) | Not started | Go CLI `stream-events` silently destroys the ReplayGap signal |
-| WRK-21 | Medium | M | owns IPC-23 fix; WRK-28 same batch | Not started | DrainEvents bound is count-based only; oversized reply kills the session and loses the drained events |
+| WRK-21 | Medium | M | owns IPC-23 fix; WRK-28 same batch | Done | DrainEvents bound is count-based only; oversized reply kills the session and loses the drained events |
-| IPC-23 | Medium | S | WRK-21 | Not started | DrainEvents contract requirements (reply fits negotiated max, no event loss, drain-until-empty) + proto-comment/doc wave |
+| IPC-23 | Medium | S | WRK-21 | In progress — mechanics landed with WRK-21; proto-comment/doc wave pending | DrainEvents contract requirements (reply fits negotiated max, no event loss, drain-until-empty) + proto-comment/doc wave |
-| IPC-30 | Low | M | WRK-21 (same batch) | Not started | Oversized event frame stays session-fatal by design, but the death becomes structured (fault frame + logged identity) |
+| IPC-30 | Low | M | WRK-21 (same batch) | Done | Oversized event frame stays session-fatal by design, but the death becomes structured (fault frame + logged identity) |
 | SEC-31 | Medium | M | — | Not started | Failure limiter partitions on attacker-controlled key id and blocks before verification (lockout DoS) |
 | SEC-32 | Low | S | SEC-31 | Not started | Failure-limiter LRU flushable by junk-token spray; token prefix never validated |
 | IPC-24 | Medium | S | — | Not started | CI's unconditional Java churn-revert masks real drift |
@@ -71,27 +71,27 @@ Full design + implementation for each row lives in the linked domain doc under i
 | ID | Sev | Tier | Eff | Dep | Status | Title |
 |---|---|:-:|:-:|---|---|---|
-| WRK-21 | Medium | P0 | M | IPC-23 (fix owned here); WRK-28 | Not started | DrainEvents bound count-based only; oversized reply kills session and loses drained events |
+| WRK-21 | Medium | P0 | M | IPC-23 (fix owned here); WRK-28 | Done | DrainEvents bound count-based only; oversized reply kills session and loses drained events |
 | WRK-22 | Low | — | S | IPC-26 (fix owned here) | Not started | Cancelled `WriteAsync` leaves its frame queued; it is still written later |
-| WRK-23 | Low | — | S | WRK-21 | Not started | Rejected frames consume sequence numbers, producing wire gaps |
+| WRK-23 | Low | — | S | WRK-21 | Done | Rejected frames consume sequence numbers, producing wire gaps |
 | WRK-24 | Low | — | S | — | Not started | `AdoptNegotiatedMaxMessageBytes` has no lower-bound sanity check |
 | WRK-25 | Low | P2 | S | WRK-22 (shared seam) | Not started | WRK-12 flush coalescing never engages on the event hot path |
 | WRK-26 | Low | P1 | S | WRK-23 (soft); discharges IPC-29 | Not started | Write-priority and overflow doc drift from the WRK-07 change |
 | WRK-27 | Low | — | S | — | Not started | Alarm poll bypasses the watchdog's in-flight suppression (15 s vs 75 s) |
-| WRK-28 | Low | — | S | WRK-21 (same batch) | Not started | 10,000 drain cap is a duplicated magic constant |
+| WRK-28 | Low | — | S | WRK-21 (same batch) | Done | 10,000 drain cap is a duplicated magic constant |
 ### Contracts & IPC — [30-contracts-ipc.md](30-contracts-ipc.md)
 | ID | Sev | Tier | Eff | Dep | Status | Title |
 |---|---|:-:|:-:|---|---|---|
-| IPC-23 | Medium | P0 | S | WRK-21 (mechanics) | Not started | DrainEvents byte-blindness — contract requirements + proto-comment/doc wave |
+| IPC-23 | Medium | P0 | S | WRK-21 (mechanics) | In progress — mechanics landed with WRK-21; proto-comment/doc wave pending | DrainEvents byte-blindness — contract requirements + proto-comment/doc wave |
 | IPC-24 | Medium | P0 | S | — | Not started | CI's unconditional Java churn-revert masks real drift |
 | IPC-25 | Medium | P0 | M | — | Not started | Stale Go/Python worker bindings: regenerate (pinned toolchains) + check-codegen Check 4 |
 | IPC-26 | Low | P2 | S | WRK-22 (mechanics) | Not started | Cancelled write leaves ghost frame — cancelled means never written |
 | IPC-27 | Low | P2 | S | — | Not started | Descriptor-freshness test blind to enums/services/galaxy descriptor |
 | IPC-28 | Low | — | S | — | Not started | docs/Grpc.md missing CommandTooLarge → ResourceExhausted mapping |
 | IPC-29 | Low | — | S | WRK-26 (discharged by) | Not started | WorkerFrameProtocol.md missing write-scheduling/sequencing section |
-| IPC-30 | Low | P0 | M | WRK-21 (same batch) | Not started | Oversized event frame: keep session-fatal, make the death structured |
+| IPC-30 | Low | P0 | M | WRK-21 (same batch) | Done | Oversized event frame: keep session-fatal, make the death structured |
 | IPC-31 | Info | — | — | — | N/A | Gateway creation-time sequence stamping accepted; diagnostic-only, decision recorded |
 | IPC-32 | Info | — | S | IPC-25 (folded in) | Not started | check-codegen banner relabel 1/4…4/4 |
@@ -161,3 +161,5 @@ Sequence these together rather than piecemeal — several are one change set spa
 | 2026-07-13 | Operator bring-up complete: dedicated CI ed25519 key installed in windev `administrators_authorized_keys` (authorized into `dohertj2`, which owns the working MXAccess/toolchain env — a fresh OS account would break the build; the key is independently revocable), Gitea secrets `WINDEV_SSH_KEY`/`WINDEV_SSH_KNOWN_HOSTS` + variable `WINDEV_SSH_USER=dohertj2` stored, runner→`10.100.0.48:22` egress verified on the `traefik` net, issue-write confirmed. **TST-25/TST-26 → `Done`:** credentialed `windows-x86` ran GREEN on `d769244` (Gitea run #37) — Linux runner SSHed windev, checked out the SHA in `C:\build\mxaccessgw-ci` under lock, ran the x86 Worker build + `Worker.Tests`, exit 0; `nightly-windev` correctly skipped on the push event. Branch merged to `main`. Follow-ups (old tracker): revisit **TST-05** (scheduled live smoke — now covered by `nightly-windev`) and **TST-24** (client wire tests) which this unlocks. |
 | 2026-07-13 | Ran the TST-25 acceptance checks (scripts/ci/README.md) — they caught **two real CI defects, both fixed** on `fix/tst-25-ci-key-log-leak`: (1) **CI SSH key leaked in cleartext** in the `windows-x86` step env echo (Gitea's line-oriented masker missed the multiline PEM) — rotated the CI key on windev (old pubkey revoked), stored the key **base64-encoded** so the masker redacts it to `***` (confirmed on run #38), taught `run-windev-ci.sh` to decode, dropped the redundant public known-hosts secret from the job env; (2) **bootstrap lock race** — `run-windev-ci.sh`'s pre-hand-off `git fetch`/`checkout` ran outside the worktree lock, so concurrent runs collided on `.git/index.lock`; the bootstrap now holds the lock (ps1 re-uses it via `MXGW_CI_LOCK_HELD`), retest confirmed clean serialization. Also **deflaked** `SessionManagerTests` fail-fast timing assertions (absolute `<100ms` wall-clock bound flaked under CI load; now anchored to the configured timeout / dropped for the zero-timeout case). Checks passed: unreachable-host fast-fail (exit 255/15s), deliberate-red propagation (Worker.Tests failure → exit 1), lock concurrency (2nd run waits), no-key-in-logs (masked). Merge target `df7e20d` verified GREEN via the local windev path (Worker build + 356 tests); merged to `main` `19cbf7b`. Check 6 (forced-failure nightly issue): issue endpoint+token proven live at bring-up (#124); in-CI forced-failure probe abandoned to shared-runner congestion (residual `if: failure()` gating is standard Actions). |
 | 2026-07-13 | New finding **TST-30** (`Low`/`P2`) added — surfaced during TST-25 acceptance: CI runs on a single shared `gitea-runner` (`maxParallel=1`, co-located `10.100.0.35`) interleaved with `dohertj2/lmxopcua`, and Gitea 1.26 exposes no run cancel/delete, so queue latency is unbounded under cross-repo contention and the runner is a single point of failure. Design: add a second/labelled runner + document the no-cancel reality and the `run-windev-ci.sh` bypass. Roll-ups updated (Testing Low 2→3, total 47→48; P2 9→10). |
 | 2026-08-07 | **WRK-21 + WRK-28 + WRK-23 + IPC-30 → `Done`** (branch `fix/wrk-21-drain-cluster`, commits `33ba612` + test-fixture follow-ups `7c2eaf0`/`a256560`). WRK-21: `MxAccessEventQueue` gains a byte-budgeted `Drain(maxEvents, maxTotalBytes)` returning the new `WorkerEventDrainResult`, sizing inside the queue lock so an event that will not fit is never dequeued; `CreateDrainEventsReply` budgets against the negotiated frame max less a 64 KiB wrapper reserve and reports truncation through the existing `DiagnosticMessage` (no proto change), satisfying IPC-23 R1–R3; both reply-write seams (`HandleControlCommandAsync`, `ProcessCommandAsync`) now catch `MessageTooLarge` and answer the correlation with an `InvalidRequest` reply instead of unwinding/faulting the session. WRK-28: the 10,000 ceiling moved to `GatewayContractInfo.MaxDrainEventsPerCommand`, referenced by the gateway validator and the worker clamp (C# const, no `.proto` change). WRK-23: `WorkerFrameWriter` peek-stamps then commits `Sequence` only immediately before the stream write, so rejections leave no wire gap. IPC-30: an oversized event frame stays session-fatal but writes a `PROTOCOL_VIOLATION` `WorkerFault` with `command_method = EventDrain` naming family/handles/sequence/sizes (never the value) before exiting. Docs same commit: `MxAccessWorkerInstanceDesign.md`, `WorkerFrameProtocol.md`, `gateway.md`. **IPC-23 → `In progress`** — mechanics landed here; the proto-comment/doc wave (and its regen fan-out) is still pending and must not be folded into this branch. **Evidence** — macOS: `dotnet build src/ZB.MOM.WW.MxGateway.NonWindows.slnx` 0 warnings/0 errors, `dotnet test …MxGateway.Tests --filter FullyQualifiedName~MxAccessGrpcRequestValidator` 4/4 passed. windev (`scripts/ci/windev-worker-ci.ps1 -Sha a2565604 -Mode test`, 2026-08-07 06:47): x86 Worker build 0 warnings/0 errors, `Worker.Tests` **367 passed / 0 failed / 11 skipped** (skips are the live-MXAccess/dev-rig opt-ins), script exit 0. **Harness note:** `PipePair` runs both pipe ends in one process with blocking `FlushFileBuffers` per frame, so it wedges on multi-MB frames or after ~85 large round trips; the pipe tests therefore negotiate a 128 KiB frame maximum and walk 1,000 events to empty, while the full 10,000-event drain-to-empty no-loss proof runs at the queue layer (`MxAccessEventQueueTests`). |
 | 2026-08-07 | Code-review follow-ups on the same branch (commit `6bc3f9b`). (1) **Important** — `ResolveDrainReplyByteBudget` was a step, not a floor: just above the 64 KiB reserve the budget collapsed to a few bytes (exactly 1024 at the validator floor `MaxMessageBytes = 1024 + 64 KiB`), so a byte-heavy `DrainEvents` truncated on every call and the drain-until-empty loop never terminated. Now `Math.Max(frameMax - reserve, frameMax / 2)` — monotonic, never below half the frame max. New test `WorkerPipeSessionTests.DrainEvents_AtValidatorFloorFrameMax_MakesProgressAndTerminates` drives a byte-heavy queue at the exact validator floor and asserts drain-to-empty with no head reported oversized. (2) **Hardening** — the reply-too-large fallback write is now itself size-guarded (`WriteReplyTooLargeFallbackAsync`, shared by the control and STA reply seams) so a pathologically tiny negotiated max below the gateway floor (the WRK-24 gap) cannot make even the backstop session-fatal; log-and-swallow, comment points at WRK-24. (3) **Comment** — corrected the `RepeatedFieldOverheadBytes` docs: `WorkerEvent.CalculateSize()` already includes the event's tag+length, so the 8 bytes is pure slack, not wrapper compensation. **Evidence** — macOS build 0/0, validator filter 4/4. windev (`windev-worker-ci.ps1 -Sha 6bc3f9b -Mode test`, 07:07): x86 Worker build 0/0, `Worker.Tests` **368 passed / 0 failed / 11 skipped**, script exit 0. (An earlier run of the same SHA flaked on the pre-existing `RunAsync_WhenStaActivityIsStale_WritesWatchdogFault` — a 5 s CTS timeout under first-run load, untouched by this change; it passed on the clean re-run and in both prior full runs.) |
@@ -16,14 +16,14 @@ members, no positional records). The worker builds and tests only on the Windows
 | ID | Sev | Tier | Eff | Dep | Status | Title |
 |----|-----|------|-----|-----|--------|-------|
-| WRK-21 | Medium | P0 | M | IPC-23 (same defect, fix owned here); WRK-28 (same lines) | Not started | DrainEvents bound is count-based only; an oversized reply still kills the session and loses the drained events |
+| WRK-21 | Medium | P0 | M | IPC-23 (same defect, fix owned here); WRK-28 (same lines) | Done | DrainEvents bound is count-based only; an oversized reply still kills the session and loses the drained events |
 | WRK-22 | Low | — | S | IPC-26 (same defect, fix owned here) | Not started | Cancelled `WriteAsync` leaves its frame queued; it is still written later |
-| WRK-23 | Low | — | S | WRK-21 (rejection path becomes backstop-only) | Not started | Rejected frames consume sequence numbers, producing wire gaps |
+| WRK-23 | Low | — | S | WRK-21 (rejection path becomes backstop-only) | Done | Rejected frames consume sequence numbers, producing wire gaps |
 | WRK-24 | Low | — | S | — | Not started | `AdoptNegotiatedMaxMessageBytes` has no lower-bound sanity check |
 | WRK-25 | Low | P2 | S | WRK-22 (both touch enqueue/dequeue) | Not started | WRK-12 flush coalescing never engages on the event hot path |
 | WRK-26 | Low | P1 | S | WRK-23 (soft — sequence prose); discharges IPC-29 | Not started | Write-priority and overflow doc drift from the WRK-07 change |
 | WRK-27 | Low | — | S | — | Not started | Alarm poll bypasses the watchdog's in-flight suppression (15 s vs 75 s) |
-| WRK-28 | Low | — | S | WRK-21 (land in the same commit cluster) | Not started | 10,000 drain cap is a duplicated magic constant with a comment-only sync contract |
+| WRK-28 | Low | — | S | WRK-21 (land in the same commit cluster) | Done | 10,000 drain cap is a duplicated magic constant with a comment-only sync contract |
 ---
@@ -12,14 +12,14 @@ All `path:line` citations were re-verified against the working tree at `4f5371f`
 | ID | Sev | Tier | Eff | Dep | Status | Title |
 |----|-----|------|-----|-----|--------|-------|
-| IPC-23 | Medium | P0 | S¹ | WRK-21 | Not started | DrainEvents bound is count-based only; byte-heavy queue still builds a session-killing reply frame (contract requirements here; fix mechanics in WRK-21) |
+| IPC-23 | Medium | P0 | S¹ | WRK-21 | In progress — mechanics landed with WRK-21; proto-comment/doc wave pending | DrainEvents bound is count-based only; byte-heavy queue still builds a session-killing reply frame (contract requirements here; fix mechanics in WRK-21) |
 | IPC-24 | Medium | P0 | S | — | Not started | CI's unconditional Java churn-revert masks real generated-code drift for message-level proto changes |
 | IPC-25 | Medium | P0 | M | — | Not started | Committed Go/Python worker bindings are stale at HEAD; no guard covers them |
 | IPC-26 | Low | P2 | S¹ | WRK-22 | Not started | Cancelled write leaves a ghost frame that is still written (contract requirement here; fix mechanics in WRK-22) |
 | IPC-27 | Low | P2 | S | — | Not started | Descriptor freshness test blind to enums, enum values, services/methods, and the Galaxy contract |
 | IPC-28 | Low | — | S | — | Not started | `docs/Grpc.md` omits the `CommandTooLarge` → `ResourceExhausted` mapping |
 | IPC-29 | Low | — | S | — | Not started | Worker writer priority scheduling and write-time sequence stamping undocumented in the frame-protocol doc |
-| IPC-30 | Low | P0 | M | WRK-21 (same file/batch) | Not started | Oversized worker→gateway event frame is session-fatal — make the death deliberate, structured, and diagnosable |
+| IPC-30 | Low | P0 | M | WRK-21 (same file/batch) | Done | Oversized worker→gateway event frame is session-fatal — make the death deliberate, structured, and diagnosable |
 | IPC-31 | Info | — | — | — | N/A | Gateway stamps sequence at creation, worker at write — accepted divergence; sequence is documented diagnostic-only (`gateway.md:328-330`); revisit only if sequence ever becomes load-bearing |
 | IPC-32 | Info | — | S | IPC-25 | Not started | `check-codegen.ps1` check labels miscounted (folded into the IPC-25 script edit) |
@@ -145,7 +145,7 @@ Post-run, verify no new `C:\*` file exists under any `bin/` (manual `find src -n
 **Design.**
- **Expiry (window 2): eliminate, don't document.** The shared `ApiKeyIdentity` (0.1.4) carries `ExpiresUtc`; when caching a success, cap the entry lifetime at the key's expiry: `AbsoluteExpiration = min(now + ttl, ExpiresUtc)` (skip caching entirely if already ≤ now). A cached hit can then never outlive the key. Confirm during implementation that the library verifier populates `ExpiresUtc` on the returned identity; if it does not, fall back to documenting the ≤ TTL window in the remarks and `docs/Authentication.md` and file a donor-library ask.
+- **Expiry (window 2): eliminate, don't document.** The shared `ApiKeyIdentity` (0.1.5 — bumped from 0.1.4 in `e107019`, no API change) carries `ExpiresUtc`; when caching a success, cap the entry lifetime at the key's expiry: `AbsoluteExpiration = min(now + ttl, ExpiresUtc)` (skip caching entirely if already ≤ now). A cached hit can then never outlive the key. Confirm during implementation that the library verifier populates `ExpiresUtc` on the returned identity; if it does not, fall back to documenting the ≤ TTL window in the remarks and `docs/Authentication.md` and file a donor-library ask.
 - **Invalidate race (window 3): per-key generation check.** `ConcurrentDictionary<string, long> _generations`; `Invalidate(keyId)` increments the generation **before** evicting cache keys. `VerifyAsync` parses the key id from the token up front (same split the interceptor does — cheap, no store access), snapshots `g0` before calling the inner verifier, and after a success only `Set`s when the generation still equals `g0` — then re-reads the generation after the `Set` and self-evicts if it moved (bump-before-evict + set-then-recheck closes the remaining interleaving). Unparseable tokens skip caching already (`TryComputeCacheKey`).
 - **CLI window (1): accept and keep documented** — cross-process invalidation is out of scope by design; the TTL is the backstop and the remarks already say so.
@@ -236,7 +236,7 @@ Full design + implementation for each row lives in the linked domain doc under i
 | TST-21 | Low | — | S | — | Not started | Log rotation configured but minimal |
 | TST-22 | Low | — | S | — | Not started | Config-shape JSON block omits documented keys |
 | TST-23 | Low | P2 | S | — | Done | Bidirectional `Session` RPC never built |
-| TST-24 | Low | P2 | M | TST-03 | Not started | Client wire behaviour has no automated verification |
+| TST-24 | Low | P2 | M | TST-03 | Not started | Client wire behaviour has no automated verification. **Gate cleared:** TST-03 CI is Done (live and green 2026-07-10; Windows/x86 tier green 2026-07-13 via the TST-25/TST-26 SSH-driven windev job), so TST-24 is unblocked — deferred by choice now, not CI-gated |
 ## Cross-cutting clusters
@@ -276,7 +276,7 @@ Findings the review flagged as one coordinated design pass — sequence them tog
 | 2026-07-09 | P1 Wave 3 (size/backpressure topology + write ordering). IPC-02/03/04 + WRK-04/07 → `Done`. **Size negotiation (IPC-02):** added `GatewayHello.max_frame_bytes` (regen `Generated/` + `clients/proto` descriptor refresh with pinned protoc 34.1); the gateway sends its negotiated worker-frame max and the worker adopts it (`WorkerFrameProtocolOptions.AdoptNegotiatedMaxMessageBytes`, 0 = keep default, >256 MiB rejected) instead of a hard-coded default. **Headroom (IPC-03):** the pipe frame max now sits `EnvelopeOverheadReserveBytes` (64 KiB) above the public gRPC cap (default `Worker.MaxMessageBytes` 16 MiB→16 MiB+64 KiB), cross-validated at startup; `WorkerClient` pre-checks command envelope size and fails only the offending correlation (`ResourceExhausted`) instead of `SetFaulted`ing the session. **Drain bound (IPC-04):** gateway request validator rejects `DrainEvents max_events` above 10 000; the worker caps each reply at `MaxDrainEventsPerReply` (10 000) and treats `max_events = 0` as that cap, not "drain all". **Sequence (WRK-04):** `WorkerFrameWriter` stamps the envelope `Sequence` at the point of writing under the write lock, so wire order and stamped sequence always agree under concurrent producers. **Priority (WRK-07):** the worker writer is now a cooperative priority scheduler — control frames (reply/fault/heartbeat/shutdown-ack) drain ahead of event frames; per-frame validation/size rejections fail only that frame, a stream failure fails all queued. Docs same-change (GatewayConfiguration, WorkerFrameProtocol, gateway.md). **Verified:** macOS NonWindows build clean + validator/grpc tests green; **windev** x86 worker builds clean, `Worker.Tests` 352 passed / 0 failed / 11 skipped (incl. new monotonic-sequence, control-before-event priority, negotiated-max, drain-bound tests), gateway `Tests` 799 passed / 3 failed — all 3 pre-existing windev-environmental (SelfSigned SAN + 2 `EventStreamServiceTests` timing, both pass in isolation). Commits `c8b3a22` (gateway half), `ebe6aea` (worker half), `309296f` (descriptor + default-expectation refresh). GWC-04 (event-channel decoupling) is the remaining Wave 3 item. |
 | 2026-07-09 | P1 S-misc (dashboard/observability hardening). SEC-02/12/20 → `Done`. SEC-02: `DashboardAuthorizationHandler` restricts the loopback + `Authentication:Mode=Disabled` bypasses to read-only (they satisfy a Viewer-bearing requirement but never `AdminOnly`), closing the policy-layer gap where anonymous localhost was authorized for Admin surfaces. SEC-12: `DashboardSessionAdminService` now emits canonical `AuditEvent`s (`dashboard-close-session`/`dashboard-kill-worker`, category `SessionAdmin`) through `IAuditWriter` on Success/Failure/Denied, so Close/Kill land durable audit rows. SEC-20: dropped the unbounded `session_id` tag from the exported `mxgateway.heartbeats.failed` counter. Docs updated same-change (CLAUDE.md, GatewayDashboardDesign.md, Metrics.md). Server build clean (0 warnings); targeted classes 30/30 pass; broader Dashboard+Security+GatewayApplication+Metrics sweep 295/295 pass. |
 | 2026-07-09 | P1 Wave 2b (security authz+hub). SEC-05/07/08/11 → `Done` (hub-token lifetime; QueryActiveAlarms scope arm; gateway-side verification cache + last-used coalescing; login rate limit + per-peer gRPC failure limiter). Full-suite checkpoint caught + fixed regressions the earlier narrow SEC-01/04/06 filter missed: cross-platform path-rooting, an `IHostEnvironment` fallback for minimal DI containers, a test-assembly `ASPNETCORE_ENVIRONMENT=Development` default, and a platform-correct default-path assertion. Suite: 747 passed / 42 failed, all 42 pre-existing macOS named-pipe-harness env failures. |
-| 2026-07-09 | P1 Wave 2a (security). SEC-01/04/06 → `Done` (config path-rooting + production validator guards; Server build clean, validator+hygiene tests 53/53). SEC-10 → `Done`: the shared `ZB.MOM.WW.Auth.ApiKeys` gained optional `ExpiresUtc` (expired keys rejected, auth DB auto-migrates to schema v3) via a concurrent HistorianGateway-remediation session's "G-2"; this repo consumes it by bumping the four `Auth.*` refs 0.1.2→0.1.4 (commit 197731a). Remaining SEC-10 polish (`apikey create --expires` + dashboard staleness badge) tracked as a small follow-up. |
+| 2026-07-09 | P1 Wave 2a (security). SEC-01/04/06 → `Done` (config path-rooting + production validator guards; Server build clean, validator+hygiene tests 53/53). SEC-10 → `Done`: the shared `ZB.MOM.WW.Auth.ApiKeys` gained optional `ExpiresUtc` (expired keys rejected, auth DB auto-migrates to schema v3) via a concurrent HistorianGateway-remediation session's "G-2"; this repo consumes it by bumping the four `Auth.*` refs 0.1.2→0.1.4 (commit 197731a; since bumped to 0.1.5 in `e107019` — 0.1.4 plus a transitive SQLitePCLRaw security pin, no API change, expiry enforcement retained). Remaining SEC-10 polish (`apikey create --expires` + dashboard staleness badge) tracked as a small follow-up. |
 | 2026-07-09 | P1 Wave 1 (CI + codegen freshness + Rust buildability) via parallel agents. IPC-01/09/19/20, CLI-02 → `Done`; TST-03 → `In review` (CI pipeline authored + YAML/layout-validated, but not yet executed on a Gitea runner — proven on first push). Verified on macOS: NonWindows build clean, `ClientProtoInputTests` 5/5, `publish-client-proto-inputs.ps1 -Check` exit 0, `cargo package` (no `--no-verify`) compiles standalone. Added a vendored-Rust-proto drift guard (Check 3) to `check-codegen.ps1` closing the CLI-02 static-copy risk. IPC-09 script guards not executed end-to-end (need pinned python/JRE toolchains); logic is PATH-resolution + version assertion. |
 | 2026-07-09 | WRK-01 → `Done`. Verified on Windows host (windev) via an isolated `origin/main` worktree: worker builds x86 clean, `StaRuntimeTests`+`WorkerPipeSessionTests` 33/33 pass. Fixed an `xUnit1030` build error (the new worker test used `.ConfigureAwait(false)` in `[Fact]` bodies) that the macOS tree could not surface. Also ran GWC-01's Windows-only `WorkerClientTests` on windev: 18/18 pass (incl. `ReadEventsAsync_SecondEnumerator_Throws`). All 8 P0 findings now `Done`. Not yet committed. |
 | 2026-07-10 | **TST-03 → `Done`: CI is live and green** on branch `fix/ci-selfhosted-tooling`. The pipeline was authored (P1) but had never executed. Root cause it never ran: the co-located `gitea-runner` on the docker host (`10.100.0.35`) spawned job containers on an isolated network (`container.network: ""`) that could not resolve Gitea's internal clone URL `http://gitea:3000`; one-line host fix `container.network: "traefik"` + `docker restart gitea-runner`. The self-hosted `catthehacker` act image also lacks tooling GitHub-hosted runners preinstall — ci.yml now installs pwsh (dotnet global tool, for `check-codegen.ps1`) and Gradle 9.5.1 directly (act can't resolve the `gradle/actions` monorepo action; no gradle wrapper in repo). Driving to green surfaced and fixed **five real latent defects** (TST-03 doing its job): `check-codegen.ps1` `.Trim()`-on-`$null` on a clean tree; **stale vendored rust proto** (`clients/rust/protos/mxaccess_worker.proto` missing canonical `max_frame_bytes`); `OrphanWorkerTerminatorTests` hard-coded `C:\` path failing Linux `Path.GetFullPath` (0 kills); **stale java generated** `MxaccessWorker.java` (missing `max_frame_bytes`, regenerated); a sync python test building a `grpc.aio.Channel` with no current event loop on py3.12 (autouse conftest fixture). Result: `portable` **success** (NonWindows build + codegen freshness + 808/808 gateway tests + .NET/Go/Rust/Python clients) and `java` **success**. `windows` + `live-mxaccess` jobs remain `queued` pending a self-hosted **windev** runner (`10.100.0.48`) with those labels — separate follow-up, does not gate portable/java. |
@@ -123,9 +123,9 @@ The split uses `count: 3` because the secret portion may itself contain undersco
 ### Command value redaction
-> **Not yet implemented.** Command-value logging is *not* wired end-to-end. There is no `MxGateway:Diagnostics:LogCommandValues` (or equivalent) configuration knob, and `RedactCommandValue` / `IsCredentialBearingCommand` have no call sites in the gateway — no command values are logged anywhere today, which is the safest posture. The helpers below exist as the intended redaction seam for a future opt-in value-logging feature; that wiring is deferred until secured-bulk command variants are covered by the redactor's credential list (the `WriteSecuredBulk` / `WriteSecured2Bulk` gap), so enabling value logging cannot leak a secured-bulk payload. Until then, treat this section as describing the planned shape, not current behavior.
+> **Redaction seam wired; value logging still has no opt-in knob** (updated 2026-08-07). Since commit `47c0b64` (2026-07-27), `RedactCommandValue` **is** wired: `GatewayLogRedactorSeam` (`Diagnostics/GatewayLogRedactorSeam.cs`) calls it for any non-null `CommandValue` log property, gated on `CommandMethod`, so a command payload that reaches a log event is masked on every sink through the shared `ILogRedactor` seam. What remains true from the original note: there is still no `MxGateway:Diagnostics:LogCommandValues` (or equivalent) configuration knob — the seam exposes no opt-in, `valueLoggingEnabled` is never passed `true`, so **every** non-null command value is redacted, credential-bearing or not. In practice no log statement currently emits a `CommandValue` property, but one that does can no longer leak a payload in the clear.
-The intended `RedactCommandValue` would enforce the "values are opt-in and redacted by default" rule:
+`RedactCommandValue` enforces the "values are opt-in and redacted by default" rule:
 ```csharp
 public static object? RedactCommandValue(
@@ -1,5 +1,18 @@
 # Galaxy Repository Browse
 > **Adopted as a shared package (2026-06-25, commit `8e196a7`).** The Galaxy browse
 > implementation described in this document no longer lives in this repository. The
 > gateway consumes the shared **`ZB.MOM.WW.GalaxyRepository`** library (pinned at
 > `0.2.0` from the Gitea NuGet feed; sources hosted in the `scadaproj` repo), wired
 > in `GatewayApplication.cs` via `AddZbGalaxyRepository` (registration) and
 > `MapZbGalaxyRepository` (endpoint mapping). The former in-repo `Server/Galaxy/`
 > classes and `Grpc/GalaxyRepositoryGrpcService.cs` / `Grpc/GalaxyProtoMapper.cs`
 > were deleted with the adoption. The wire contract (`galaxy_repository.v1`) is
 > unchanged, so the RPC behavior, filters, paging, caching, and snapshot semantics
 > below still apply as documented — but any `src/ZB.MOM.WW.MxGateway.Server/...`
 > file path cited below is historical; the equivalent class now lives in the
 > shared library.
 The gateway exposes a read-only browse surface over the AVEVA System Platform
 Galaxy Repository (the SQL Server database named `ZB`). Clients use it to
 enumerate the deployed object hierarchy and each object's attributes
@@ -107,7 +120,8 @@ server and dashboard views are consistent.
 ## Hierarchy Cache
 The gateway holds a single shared `IGalaxyHierarchyCache`
-(`src/ZB.MOM.WW.MxGateway.Server/Galaxy/GalaxyHierarchyCache.cs`) — every
+(`GalaxyHierarchyCache`, library-side in `ZB.MOM.WW.GalaxyRepository` — formerly
 `src/ZB.MOM.WW.MxGateway.Server/Galaxy/GalaxyHierarchyCache.cs`) — every
 `DiscoverHierarchy` and `GetLastDeployTime` request reads from this cache
 rather than hitting SQL. Many clients can browse concurrently with at most
 one SQL query in flight.
@@ -174,7 +188,8 @@ record: deleting it only forces the next cold start to wait for live SQL.
 ## Deploy Notifications
 `WatchDeployEvents` is a server-streaming RPC backed by
-`IGalaxyDeployNotifier` (`src/ZB.MOM.WW.MxGateway.Server/Galaxy/GalaxyDeployNotifier.cs`).
+`IGalaxyDeployNotifier` (`GalaxyDeployNotifier`, library-side in
 `ZB.MOM.WW.GalaxyRepository`).
 The notifier maintains a private bounded channel per subscriber so a slow
 client cannot back-pressure other subscribers or the publisher.
@@ -322,7 +337,7 @@ fields cannot express null. Use it to distinguish "no dimension reported" from
 ```text
 gRPC client(s)
-  -> GalaxyRepositoryGrpcService (src/ZB.MOM.WW.MxGateway.Server/Grpc/)
+  -> Galaxy repository gRPC service (library-side; mapped via MapZbGalaxyRepository)
    DiscoverHierarchy, GetLastDeployTime, BrowseChildren -> IGalaxyHierarchyCache.Current
    WatchDeployEvents                                    -> IGalaxyDeployNotifier
    TestConnection                                       -> GalaxyRepository  (direct SQL)
@@ -341,41 +356,44 @@ GalaxyHierarchyRefreshService (BackgroundService)
    -> IGalaxyDeployNotifier.Publish (only on deploy change)
 ```
-Component breakdown:
+Component breakdown (all of these classes are **library-side**, in the shared
 `ZB.MOM.WW.GalaxyRepository` package — the in-repo files formerly at the paths
 below were deleted in commit `8e196a7`):
- `GalaxyRepository` (`src/ZB.MOM.WW.MxGateway.Server/Galaxy/GalaxyRepository.cs`) holds
+- `GalaxyRepository` (formerly `src/ZB.MOM.WW.MxGateway.Server/Galaxy/GalaxyRepository.cs`) holds
  the SQL. Both `HierarchySql` and `AttributesSql` walk template-derivation and
  package-derivation chains via recursive CTEs and pick the most-derived
  override per object. `HierarchySql` still matches the OtOpcUa original;
  `AttributesSql` does not — it additionally enumerates built-in primitive
  attributes (see [Built-in vs configured attributes](#built-in-vs-configured-attributes)).
 - `GalaxyHierarchyCache`
-  (`src/ZB.MOM.WW.MxGateway.Server/Galaxy/GalaxyHierarchyCache.cs`) holds the most
+  (formerly `src/ZB.MOM.WW.MxGateway.Server/Galaxy/GalaxyHierarchyCache.cs`) holds the most
  recent immutable `GalaxyHierarchyCacheEntry` (materialized objects +
  precomputed dashboard summary + counts + status). All gRPC clients share the
  same entry.
 - `GalaxyHierarchyRefreshService`
-  (`src/ZB.MOM.WW.MxGateway.Server/Galaxy/GalaxyHierarchyRefreshService.cs`) is a
+  (formerly `src/ZB.MOM.WW.MxGateway.Server/Galaxy/GalaxyHierarchyRefreshService.cs`) is a
  hosted `BackgroundService` that drives `RefreshAsync` on the configured
  interval, with deploy-time gating to avoid unnecessary heavy queries.
 - `GalaxyDeployNotifier`
-  (`src/ZB.MOM.WW.MxGateway.Server/Galaxy/GalaxyDeployNotifier.cs`) is a thin
+  (formerly `src/ZB.MOM.WW.MxGateway.Server/Galaxy/GalaxyDeployNotifier.cs`) is a thin
  per-subscriber-channel fan-out for streaming clients.
 - `GalaxyProtoMapper`
-  (`src/ZB.MOM.WW.MxGateway.Server/Grpc/GalaxyProtoMapper.cs`) converts row models to
+  (formerly `src/ZB.MOM.WW.MxGateway.Server/Grpc/GalaxyProtoMapper.cs`) converts row models to
  proto messages. Used by the cache during refresh to materialize the reply
  once.
 - `GalaxyBrowseProjector`
-  (`src/ZB.MOM.WW.MxGateway.Server/Galaxy/GalaxyBrowseProjector.cs`) projects one level
+  (formerly `src/ZB.MOM.WW.MxGateway.Server/Galaxy/GalaxyBrowseProjector.cs`) projects one level
  of children out of an immutable cache entry. Memoizes the filtered child list
  per cache-entry instance so repeated paging is an O(pageSize) slice rather than an
  O(siblings) filter scan. The memo is keyed on the cache entry reference, so a new
  entry from the background refresh makes the stale memo unreachable and it is
  collected with it. `DashboardBrowseService` wraps this projector to drive the
  dashboard's lazy-expand tree.
- `GalaxyRepositoryGrpcService`
+- The Galaxy repository gRPC service (formerly
-  (`src/ZB.MOM.WW.MxGateway.Server/Grpc/GalaxyRepositoryGrpcService.cs`) implements
+  `src/ZB.MOM.WW.MxGateway.Server/Grpc/GalaxyRepositoryGrpcService.cs`, deleted
-  the five RPCs.
+  with the adoption) implements the five RPCs; it is now supplied by the library
  and mapped via `MapZbGalaxyRepository`.
 ## Configuration
@@ -378,6 +378,20 @@ If event conversion throws, catch it inside the event handler, record a
 structured `WorkerFault`, and keep the worker alive only if the fault policy
 allows it.
 The event drain loop streams queued events as `WorkerEvent` frames. A single
 event whose envelope exceeds the negotiated frame maximum is **undeliverable end
 to end** — the pipe maximum sits only the envelope-overhead reserve above the
 public gRPC cap, so a frame the pipe rejects would also be rejected on the
 client-facing stream. The session therefore faults on it rather than dropping it
 (a silent drop makes the event stream unfaithful, and a synthesized placeholder
 is barred by the no-synthesized-events rule), but the death is structured: the
 worker logs the event's identity — family, handles, worker sequence, and sizes,
 never the value — writes a `WorkerFault` with category `ProtocolViolation` and
 command method `EventDrain` carrying the same identity, and only then exits.
 Operator remediation is configuration: raise `MxGateway:Worker:MaxMessageBytes`
 for that workload. Other per-frame rejection codes keep their previous behavior
 because they indicate worker bugs, not workload size.
 ## Command Queue
 The pipe reader converts `WorkerCommand` messages into `StaCommand` entries.
@@ -440,6 +454,29 @@ Diagnostics:
 - `DrainEvents`
 - `ShutdownWorker`
 `DrainEvents` is answered on the message-loop thread, not the STA, and its reply
 is bounded on **two** axes because no diagnostics command may be session-fatal:
 - **Count** — `GatewayContractInfo.MaxDrainEventsPerCommand` (10,000) is the
  single home of the ceiling, shared by the gateway's request validator (which
  rejects a larger `max_events` at the public boundary) and this worker clamp
  (which also interprets `max_events = 0`, "as many as available").
 - **Bytes** — the count cap alone is not sufficient: byte-heavy events (large
  string or array `MxValue`s) overshoot the negotiated frame maximum long before
  10,000 events. The drain is therefore byte-budgeted against the negotiated
  maximum less a 64 KiB envelope/reply-wrapper reserve, and the size decision
  happens inside the event queue's lock, so an event is dequeued only once it is
  known to fit. An event that does not fit stays at the head of the queue and is
  never lost.
 Truncation is reported in the reply's existing `DiagnosticMessage`
 ("N events returned, M remain; repeat DrainEvents for the rest") rather than in a
 new field, so the contract is unchanged and callers drain iteratively until a
 reply comes back empty. In the degenerate case where the head event alone exceeds
 the budget, the reply returns whatever fit before it (possibly nothing) and names
 the blocked event's worker sequence so an operator can find the offending tag;
 that event needs a larger `MxGateway:Worker:MaxMessageBytes` to move at all.
 Implement method-specific dispatch instead of a generic string method invoker.
 Parity tests need stable command-specific request and reply shapes.
@@ -623,6 +660,14 @@ queue fills:
 Production coalescing may be added later, but it must be explicit and tested.
 Do not drop or coalesce events in v1.
 No control reply is session-fatal on size. Reply builders size their payloads
 against the negotiated frame maximum, and the two reply-write seams (the
 control-command path and the STA command path) additionally catch a
 `MessageTooLarge` per-frame rejection and answer that correlation with a small
 `InvalidRequest` reply instead of unwinding the session. Oversized *event*
 frames keep the opposite policy — see Event Sink — because an event above the
 frame maximum cannot be delivered to the client at all.
 ## Heartbeat And Watchdog
 `WorkerPipeSession` starts the heartbeat loop after the gateway validates
@@ -29,6 +29,29 @@ default. A `max_frame_bytes` of 0 (an older gateway that never set the field)
 means "use the worker's built-in default". This keeps both ends framing to the
 same limit rather than depending on matched compile-time constants.
 Every worker-to-gateway frame must serialize within this limit, control replies
 included, so reply builders truncate to fit rather than emit a frame the writer
 will reject. `WorkerPipeSession` pre-sizes a `DrainEvents` reply below the
 negotiated maximum (less a fixed envelope/reply-wrapper reserve) and reports the
 truncation in the reply's `DiagnosticMessage`; the caller contract is to repeat
 `DrainEvents` until it returns an empty reply. Should a reply still overshoot,
 `MessageTooLarge` at the reply-write seam is answered with a small
 `InvalidRequest` reply for that correlation, not with session teardown — no
 diagnostics command may kill a session.
 An oversized *event* frame is the deliberate exception. Such an event is
 undeliverable end to end (the pipe maximum sits only the envelope-overhead
 reserve above the public gRPC cap), so the session faults: the worker logs the
 event's identity and sizes — never its value — writes a `WorkerFault` with
 category `ProtocolViolation` and command method `EventDrain`, then exits.
 Remediation is raising `MxGateway:Worker:MaxMessageBytes` for that workload.
 A per-frame rejection does not consume an envelope `sequence`. The writer stamps
 a candidate sequence, runs the empty-payload and size checks against the stamped
 envelope, and commits the counter only immediately before the stream write, so
 the sequences observed on the wire stay contiguous across rejections and an
 operator reading a pipe capture never sees a phantom gap.
 ## Envelope Validation
 `WorkerFrameReader` and `WorkerFrameWriter` validate each envelope against the
@@ -447,10 +447,14 @@ Optional diagnostics:
 - `Ping`
 - `GetSessionState`
 - `GetWorkerInfo`
- `DrainEvents` — diagnostic; `max_events` is bounded (the gateway rejects requests
+- `DrainEvents` — diagnostic; the reply is bounded on two axes so one drain cannot
-  above a public ceiling, and the worker caps each reply at its own per-reply limit,
+  pack an unbounded, session-killing reply frame. By **count**: the gateway rejects
-  treating `max_events = 0` as "the default cap") so one drain cannot pack an
+  requests above the shared ceiling `GatewayContractInfo.MaxDrainEventsPerCommand`
-  unbounded, session-killing reply frame.
+  and the worker clamps to the same value, treating `max_events = 0` as "the default
  cap". By **bytes**: the worker sizes the reply while draining, against the
  negotiated worker-frame maximum, so a byte-heavy queue truncates instead of
  overshooting and events that do not fit stay queued. Truncation is reported in the
  reply's `DiagnosticMessage`; callers drain iteratively until an empty reply.
 - `ShutdownWorker`
 Do not compress MXAccess semantics into generic verbs too early. A command enum
@@ -17,6 +17,21 @@ public static class GatewayContractInfo
    /// <summary>Default backend name identifying the MXAccess worker process type.</summary>
    public const string DefaultBackendName = "mxaccess-worker";
    /// <summary>
    /// Ceiling on how many events one <c>DrainEvents</c> command may move in a single reply.
    /// Shared so the gateway's request-validation ceiling
    /// (<c>MxAccessGrpcRequestValidator</c>, which rejects a larger <c>max_events</c> loudly at
    /// the public boundary) and the worker's per-reply clamp
    /// (<c>WorkerPipeSession.CreateDrainEventsReply</c>, the backstop that also interprets
    /// <c>max_events = 0</c>) cannot drift apart. A count cap alone is necessary but not
    /// sufficient: the worker additionally caps the reply by serialized bytes against the
    /// negotiated frame maximum (WRK-21), so a reply may carry fewer events than this ceiling
    /// and fewer than are queued. Callers drain iteratively until an empty reply.
    /// This is a documented behavioral bound, not wire schema — it is deliberately a C#
    /// constant and not a <c>.proto</c> field.
    /// </summary>
    public const uint MaxDrainEventsPerCommand = 10_000;
    /// <summary>
    /// Environment variable name that opts an xUnit suite into running live
    /// MXAccess COM tests. Single source of truth shared by both
@@ -10,8 +10,15 @@ public sealed class GatewayLogRedactorSeam : ILogRedactor
 {
    private static readonly string[] IdentityKeys = ["ClientIdentity", "authorization", "Authorization"];
    /// <summary>Property name carrying the MXAccess command method, which gates value redaction.</summary>
    private const string CommandMethodProperty = "CommandMethod";
    /// <summary>Property name carrying a command payload value that may bear credentials.</summary>
    private const string CommandValueProperty = "CommandValue";
    /// <summary>
-    /// Masks API-key/credential material in known identity-bearing log properties.
+    /// Masks API-key/credential material in known identity-bearing log properties, and any
    /// command payload value.
    /// </summary>
    /// <param name="properties">The log event property dictionary to redact in place.</param>
    public void Redact(IDictionary<string, object?> properties)
@@ -24,5 +31,26 @@ public sealed class GatewayLogRedactorSeam : ILogRedactor
                properties[key] = GatewayLogRedactor.RedactClientIdentity(s);
            }
        }
        RedactCommandValue(properties);
    }
    /// <summary>
    /// Masks a command payload value. Value logging is off here — the seam exposes no opt-in — so
    /// every non-null value is masked, credential-bearing command or not.
    /// </summary>
    /// <param name="properties">The log event property dictionary to redact in place.</param>
    private static void RedactCommandValue(IDictionary<string, object?> properties)
    {
        if (!properties.TryGetValue(CommandValueProperty, out object? value) || value is null)
        {
            return;
        }
        string? commandMethod = properties.TryGetValue(CommandMethodProperty, out object? method)
            ? method as string
            : null;
        properties[CommandValueProperty] = GatewayLogRedactor.RedactCommandValue(commandMethod, value);
    }
 }
@@ -1,17 +1,11 @@
 using Grpc.Core;
 using ZB.MOM.WW.MxGateway.Contracts;
 using ZB.MOM.WW.MxGateway.Contracts.Proto;
 namespace ZB.MOM.WW.MxGateway.Server.Grpc;
 public sealed class MxAccessGrpcRequestValidator
 {
    // Upper bound on a single DrainEvents request. DrainEvents is a diagnostics RPC that returns
    // buffered events in one non-streaming reply, so an unbounded max_events could pack the whole
    // queue into a session-killing frame. The worker independently caps each reply at its
    // own MaxDrainEventsPerReply; this public bound rejects an obviously-abusive request loudly at
    // the boundary. max_events = 0 is allowed and means "the worker's default batch cap".
    private const uint MaxDrainEventsPerRequest = 10_000;
    /// <summary>Validates an open session request.</summary>
    /// <param name="request">The request to validate.</param>
    public void ValidateOpenSession(OpenSessionRequest request)
@@ -78,10 +72,18 @@ public sealed class MxAccessGrpcRequestValidator
        }
        // The payload case now matches the kind, so command.DrainEvents is non-null here.
-        if (command.Kind is MxCommandKind.DrainEvents && command.DrainEvents.MaxEvents > MaxDrainEventsPerRequest)
+        // DrainEvents is a diagnostics RPC that returns buffered events in one non-streaming
        // reply, so an unbounded max_events could pack the whole queue into a session-killing
        // frame. The worker independently clamps every reply to the same shared ceiling and
        // additionally caps it by serialized bytes; this public bound rejects an obviously-abusive
        // request loudly at the boundary. max_events = 0 is allowed and means "the worker's
        // default batch cap".
        if (command.Kind is MxCommandKind.DrainEvents
            && command.DrainEvents.MaxEvents > GatewayContractInfo.MaxDrainEventsPerCommand)
        {
            throw InvalidArgument(
-                $"DrainEvents max_events ({command.DrainEvents.MaxEvents}) must not exceed {MaxDrainEventsPerRequest}; "
+                $"DrainEvents max_events ({command.DrainEvents.MaxEvents}) must not exceed "
                + $"{GatewayContractInfo.MaxDrainEventsPerCommand}; "
                + "use 0 to request the worker default batch cap.");
        }
    }
@@ -13,13 +13,13 @@
    <PackageReference Include="ZB.MOM.WW.Audit" Version="0.1.0" />
    <PackageReference Include="ZB.MOM.WW.Theme" Version="0.3.1" />
    <PackageReference Include="ZB.MOM.WW.Configuration" Version="0.1.0" />
-    <PackageReference Include="ZB.MOM.WW.Health" Version="0.1.0" />
+    <PackageReference Include="ZB.MOM.WW.Health" Version="0.2.0" />
    <PackageReference Include="ZB.MOM.WW.Telemetry" Version="0.1.0" />
    <PackageReference Include="ZB.MOM.WW.Telemetry.Serilog" Version="0.1.0" />
    <PackageReference Include="ZB.MOM.WW.GalaxyRepository" Version="0.2.0" />
-    <PackageReference Include="ZB.MOM.WW.Secrets" Version="0.2.0" />
+    <PackageReference Include="ZB.MOM.WW.Secrets" Version="0.2.3" />
-    <PackageReference Include="ZB.MOM.WW.Secrets.Abstractions" Version="0.2.0" />
+    <PackageReference Include="ZB.MOM.WW.Secrets.Abstractions" Version="0.2.3" />
-    <PackageReference Include="ZB.MOM.WW.Secrets.Ui" Version="0.2.0" />
+    <PackageReference Include="ZB.MOM.WW.Secrets.Ui" Version="0.2.3" />
    <PackageReference Include="Serilog.AspNetCore" Version="10.0.0" />
    <PackageReference Include="Serilog.Sinks.Console" Version="6.1.1" />
    <PackageReference Include="Serilog.Sinks.File" Version="7.0.0" />
@@ -13,4 +13,75 @@ public sealed class GatewayLogRedactorSeamTests
        redactor.Redact(props);
        Assert.Equal("Bearer mxgw_operator01_[redacted]", props["ClientIdentity"]);
    }
    /// <summary>A command value carried by a credential-bearing command never reaches a sink in the clear.</summary>
    [Fact]
    public void Redact_MasksCommandValueForCredentialBearingCommand()
    {
        GatewayLogRedactorSeam redactor = new();
        Dictionary<string, object?> props = new()
        {
            ["CommandMethod"] = "AuthenticateUser",
            ["CommandValue"] = "operator01:hunter2"
        };
        redactor.Redact(props);
        Assert.Equal(GatewayLogRedactor.RedactedValue, props["CommandValue"]);
    }
    /// <summary>
    /// Value logging is off by default, so an ordinary command value is masked too — the seam
    /// exposes no opt-in, matching <see cref="GatewayLogRedactor.RedactCommandValue"/>'s default.
    /// </summary>
    [Fact]
    public void Redact_MasksCommandValueForOrdinaryCommand()
    {
        GatewayLogRedactorSeam redactor = new();
        Dictionary<string, object?> props = new()
        {
            ["CommandMethod"] = "Read",
            ["CommandValue"] = "plaintext-tag-value"
        };
        redactor.Redact(props);
        Assert.Equal(GatewayLogRedactor.RedactedValue, props["CommandValue"]);
    }
    /// <summary>A command value with no accompanying method is still masked — an unknown method cannot be cleared.</summary>
    [Fact]
    public void Redact_MasksCommandValueWhenCommandMethodAbsent()
    {
        GatewayLogRedactorSeam redactor = new();
        Dictionary<string, object?> props = new() { ["CommandValue"] = "plaintext-tag-value" };
        redactor.Redact(props);
        Assert.Equal(GatewayLogRedactor.RedactedValue, props["CommandValue"]);
    }
    /// <summary>A null command value stays null rather than becoming the redaction placeholder.</summary>
    [Fact]
    public void Redact_LeavesNullCommandValueNull()
    {
        GatewayLogRedactorSeam redactor = new();
        Dictionary<string, object?> props = new() { ["CommandValue"] = null };
        redactor.Redact(props);
        Assert.Null(props["CommandValue"]);
    }
    /// <summary>An event carrying no command value is untouched — the seam must not invent the property.</summary>
    [Fact]
    public void Redact_DoesNotAddCommandValueWhenAbsent()
    {
        GatewayLogRedactorSeam redactor = new();
        Dictionary<string, object?> props = new() { ["ClientIdentity"] = "anonymous" };
        redactor.Redact(props);
        Assert.False(props.ContainsKey("CommandValue"));
    }
 }
@@ -410,6 +410,48 @@ public sealed class WorkerFrameProtocolTests
        }
    }
    /// <summary>
    /// Verifies a per-frame rejection does not burn a sequence number (WRK-23). The sequence is a
    /// diagnostic counter, so a gap breaks nothing functionally — but an operator correlating a pipe
    /// capture reads a gap as a lost frame and chases a bug that does not exist, and the gap-free
    /// guarantee the concurrent-write test asserts would otherwise only hold until the first
    /// rejection.
    /// </summary>
    /// <returns>A task that represents the asynchronous operation.</returns>
    [Fact]
    public async Task WriteAsync_PerFrameRejection_DoesNotConsumeSequence()
    {
        const int maxMessageBytes = 512;
        WorkerFrameProtocolOptions options = new(
            SessionId,
            GatewayContractInfo.WorkerProtocolVersion,
            Nonce,
            maxMessageBytes);
        using MemoryStream stream = new();
        WorkerFrameWriter writer = new(stream, options);
        await writer.WriteAsync(CreateEventEnvelope());
        WorkerEnvelope oversized = CreateGatewayHelloEnvelope();
        oversized.GatewayHello.GatewayVersion = new string('x', maxMessageBytes * 2);
        WorkerFrameProtocolException exception =
            await Assert.ThrowsAsync<WorkerFrameProtocolException>(
                async () => await writer.WriteAsync(oversized));
        Assert.Equal(WorkerFrameProtocolErrorCode.MessageTooLarge, exception.ErrorCode);
        await writer.WriteAsync(CreateEventEnvelope());
        stream.Position = 0;
        WorkerFrameReader reader = new(stream, options);
        WorkerEnvelope first = await reader.ReadAsync();
        WorkerEnvelope second = await reader.ReadAsync();
        // Two frames reached the wire; the rejected frame in between left no gap.
        Assert.Equal(1UL, first.Sequence);
        Assert.Equal(2UL, second.Sequence);
        Assert.Equal(stream.Length, stream.Position);
    }
    /// <summary>Verifies a zero negotiated frame maximum keeps the constructor default.</summary>
    [Fact]
    public void AdoptNegotiatedMaxMessageBytes_WithZero_KeepsDefault()
@@ -19,6 +19,21 @@ public sealed class WorkerPipeSessionTests
    private const string SessionId = "session-1";
    private const string Nonce = "nonce-secret";
    // Byte-heavy drain fixture (WRK-21). 10,000 events at ~1.7 KiB each is ~17 MB of queue — far
    // more than one frame — so DrainEvents must truncate.
    //
    // Two limits below are harness accommodations, not properties of the fix. PipePair runs both
    // ends of a duplex pipe inside one process, with no continuous read pump and with blocking
    // FlushFileBuffers under every frame write, so it tolerates neither multi-megabyte frames nor
    // hundreds of large round trips before both ends wedge waiting on each other. Hence a small
    // negotiated frame maximum, and a smaller queue for the drain-to-empty walk. The byte cap
    // behaves identically at any frame size; exhaustive no-loss over the full 10,000 events is
    // covered without a pipe by MxAccessEventQueueTests.
    private const int ByteHeavyEventCount = 10_000;
    private const int RepeatedDrainEventCount = 1_000;
    private const int ByteHeavyEventPayloadBytes = 1_800;
    private const uint NegotiatedMaxFrameBytes = 128 * 1024;
    /// <summary>Verifies that valid gateway hello triggers worker hello and ready responses.</summary>
    /// <returns>A task that represents the asynchronous operation.</returns>
    [Fact]
@@ -487,6 +502,361 @@ public sealed class WorkerPipeSessionTests
        await SendShutdownAndWaitAsync(pipePair, runTask, cancellation.Token);
    }
    /// <summary>
    ///     The WRK-21 repro. A queue full of byte-heavy events (large string values — the payload
    ///     profile this gateway exists for) used to make <c>DrainEvents max_events = 0</c> build a
    ///     reply above the negotiated frame maximum: the writer rejected the frame, the exception
    ///     unwound the session, and the already-dequeued events were gone. The drain is now
    ///     byte-budgeted, so the reply fits, the truncation is reported in the reply's diagnostic
    ///     message, and the session keeps serving.
    /// </summary>
    /// <returns>A task that represents the asynchronous operation.</returns>
    [Fact]
    public async Task DrainEvents_ByteHeavyQueue_ReplyIsBoundedAndSessionSurvives()
    {
        using CancellationTokenSource cancellation = new(TimeSpan.FromSeconds(60));
        using PipePair pipePair = await PipePair.CreateAsync(cancellation.Token);
        FakeRuntimeSession runtime = new()
        {
            SuppressDrainForBatchSize = 128,
            BackingQueue = CreateByteHeavyQueue(ByteHeavyEventCount, ByteHeavyEventPayloadBytes),
        };
        WorkerPipeSession session = CreatePipeSession(pipePair.WorkerStream, runtime);
        Task runTask = session.RunAsync(cancellation.Token);
        await CompleteGatewayHandshakeAsync(pipePair, NegotiatedMaxFrameBytes, cancellation.Token);
        await pipePair.GatewayWriter
            .WriteAsync(
                CreateControlCommandEnvelope(
                    "drain-heavy-1",
                    MxCommandKind.DrainEvents,
                    command => command.DrainEvents = new DrainEventsCommand { MaxEvents = 0 }),
                cancellation.Token);
        WorkerEnvelope replyEnvelope = await ReadUntilAsync(
            pipePair.GatewayReader,
            WorkerEnvelope.BodyOneofCase.WorkerCommandReply,
            cancellation.Token);
        MxCommandReply reply = replyEnvelope.WorkerCommandReply.Reply;
        Assert.Equal(ProtocolStatusCode.Ok, reply.ProtocolStatus.Code);
        // The whole queue is far larger than one frame, so the reply is a strict subset that fits.
        Assert.True(
            replyEnvelope.CalculateSize() <= NegotiatedMaxFrameBytes,
            $"DrainEvents reply serialized to {replyEnvelope.CalculateSize()} bytes, above the negotiated {NegotiatedMaxFrameBytes}.");
        Assert.InRange(reply.DrainEvents.Events.Count, 1, ByteHeavyEventCount - 1);
        Assert.Contains("remain", reply.DiagnosticMessage);
        Assert.Contains("repeat DrainEvents", reply.DiagnosticMessage);
        // The session is alive: it still answers a ping, and RunAsync has not unwound.
        await pipePair.GatewayWriter
            .WriteAsync(CreatePingCommandEnvelope("ping-after-drain", "still-here"), cancellation.Token);
        WorkerEnvelope pingReply = await ReadUntilAsync(
            pipePair.GatewayReader,
            WorkerEnvelope.BodyOneofCase.WorkerCommandReply,
            envelope => envelope.WorkerCommandReply.Reply.CorrelationId == "ping-after-drain",
            cancellation.Token);
        Assert.Equal("still-here", pingReply.WorkerCommandReply.Reply.DiagnosticMessage);
        Assert.False(runTask.IsCompleted, "The session must survive a byte-heavy DrainEvents.");
        await SendShutdownAndWaitAsync(pipePair, runTask, cancellation.Token);
    }
    /// <summary>
    ///     Verifies the byte-budgeted drain loses nothing: repeating DrainEvents until it comes back
    ///     empty recovers every enqueued event exactly once, in order, across the split replies. The
    ///     pre-fix drain removed events from the queue before sizing the reply, so a rejected frame
    ///     destroyed them — no-loss is the half of the P0 criterion a catch-only fix cannot deliver.
    /// </summary>
    /// <returns>A task that represents the asynchronous operation.</returns>
    [Fact]
    public async Task DrainEvents_RepeatedCalls_RecoverAllEventsWithoutLoss()
    {
        using CancellationTokenSource cancellation = new(TimeSpan.FromSeconds(90));
        using PipePair pipePair = await PipePair.CreateAsync(cancellation.Token);
        FakeRuntimeSession runtime = new()
        {
            SuppressDrainForBatchSize = 128,
            BackingQueue = CreateByteHeavyQueue(RepeatedDrainEventCount, ByteHeavyEventPayloadBytes),
        };
        WorkerPipeSession session = CreatePipeSession(pipePair.WorkerStream, runtime);
        Task runTask = session.RunAsync(cancellation.Token);
        await CompleteGatewayHandshakeAsync(pipePair, NegotiatedMaxFrameBytes, cancellation.Token);
        List<ulong> recovered = new();
        int replyCount = 0;
        while (true)
        {
            string correlationId = $"drain-loop-{replyCount}";
            await pipePair.GatewayWriter
                .WriteAsync(
                    CreateControlCommandEnvelope(
                        correlationId,
                        MxCommandKind.DrainEvents,
                        command => command.DrainEvents = new DrainEventsCommand { MaxEvents = 0 }),
                    cancellation.Token);
            WorkerEnvelope replyEnvelope = await ReadUntilAsync(
                pipePair.GatewayReader,
                WorkerEnvelope.BodyOneofCase.WorkerCommandReply,
                envelope => envelope.WorkerCommandReply.Reply.CorrelationId == correlationId,
                cancellation.Token);
            replyCount++;
            MxCommandReply reply = replyEnvelope.WorkerCommandReply.Reply;
            Assert.True(
                replyEnvelope.CalculateSize() <= NegotiatedMaxFrameBytes,
                $"DrainEvents reply {replyCount} serialized to {replyEnvelope.CalculateSize()} bytes.");
            if (reply.DrainEvents.Events.Count == 0)
            {
                break;
            }
            foreach (MxEvent drained in reply.DrainEvents.Events)
            {
                recovered.Add(drained.WorkerSequence);
            }
            Assert.True(replyCount < 200, "DrainEvents made no progress across 200 replies.");
        }
        // More than one reply proves the drain really split; every event came back exactly once, in
        // enqueue order.
        Assert.True(replyCount > 2, $"Expected the byte cap to split the drain, saw {replyCount} replies.");
        Assert.Equal(RepeatedDrainEventCount, recovered.Count);
        for (int index = 0; index < recovered.Count; index++)
        {
            Assert.Equal((ulong)(index + 1), recovered[index]);
        }
        await SendShutdownAndWaitAsync(pipePair, runTask, cancellation.Token);
    }
    /// <summary>
    ///     Regression for the reserve-boundary budget bug. The gateway validator accepts a Worker
    ///     frame maximum as low as 1024 + 64 KiB, and just above that boundary a naive
    ///     subtract-then-guard budget collapses to ~1024 bytes — too small to move even one
    ///     byte-heavy event, so every drain reports truncation with the same head blocked and the
    ///     drain-until-empty loop never terminates. The budget is now a floor (never below half the
    ///     negotiated maximum), so a byte-heavy queue drains to empty even at the validator floor.
    /// </summary>
    /// <returns>A task that represents the asynchronous operation.</returns>
    [Fact]
    public async Task DrainEvents_AtValidatorFloorFrameMax_MakesProgressAndTerminates()
    {
        // The lowest Worker.MaxMessageBytes GatewayOptionsValidator permits: the public gRPC floor
        // (1024) plus the 64 KiB envelope-overhead reserve. The naive budget would be exactly 1024
        // here; the floored budget is half of the frame max (~33 KiB).
        const uint validatorFloorFrameMax = 1024 + (64 * 1024);
        using CancellationTokenSource cancellation = new(TimeSpan.FromSeconds(60));
        using PipePair pipePair = await PipePair.CreateAsync(cancellation.Token);
        FakeRuntimeSession runtime = new()
        {
            SuppressDrainForBatchSize = 128,
            BackingQueue = CreateByteHeavyQueue(200, ByteHeavyEventPayloadBytes),
        };
        WorkerPipeSession session = CreatePipeSession(pipePair.WorkerStream, runtime);
        Task runTask = session.RunAsync(cancellation.Token);
        await CompleteGatewayHandshakeAsync(pipePair, validatorFloorFrameMax, cancellation.Token);
        int recovered = 0;
        int replyCount = 0;
        while (true)
        {
            string correlationId = $"floor-drain-{replyCount}";
            await pipePair.GatewayWriter
                .WriteAsync(
                    CreateControlCommandEnvelope(
                        correlationId,
                        MxCommandKind.DrainEvents,
                        command => command.DrainEvents = new DrainEventsCommand { MaxEvents = 0 }),
                    cancellation.Token);
            WorkerEnvelope replyEnvelope = await ReadUntilAsync(
                pipePair.GatewayReader,
                WorkerEnvelope.BodyOneofCase.WorkerCommandReply,
                envelope => envelope.WorkerCommandReply.Reply.CorrelationId == correlationId,
                cancellation.Token);
            replyCount++;
            MxCommandReply reply = replyEnvelope.WorkerCommandReply.Reply;
            Assert.True(
                replyEnvelope.CalculateSize() <= validatorFloorFrameMax,
                $"reply {replyCount} serialized to {replyEnvelope.CalculateSize()} bytes.");
            int drainedThisReply = reply.DrainEvents.Events.Count;
            if (drainedThisReply == 0)
            {
                break;
            }
            // The head is never reported as oversized at this frame max: the ~33 KiB floored budget
            // comfortably fits the ~1.8 KiB events, so each reply makes real progress.
            Assert.DoesNotContain("alone exceeds", reply.DiagnosticMessage);
            recovered += drainedThisReply;
            Assert.True(replyCount < 200, "DrainEvents made no progress at the validator floor frame max.");
        }
        Assert.Equal(200, recovered);
        await SendShutdownAndWaitAsync(pipePair, runTask, cancellation.Token);
    }
    /// <summary>
    ///     Verifies the control-reply write seam is not session-fatal on size (WRK-21 backstop). The
    ///     reply builders size their payloads, so this path needs a deliberately budget-blind drain
    ///     to reach — but that is the point: a future command or a sizing bug must degrade to an
    ///     error reply for that correlation, never to a dead session.
    /// </summary>
    /// <returns>A task that represents the asynchronous operation.</returns>
    [Fact]
    public async Task ControlReplyTooLarge_WritesErrorReplyInsteadOfDying()
    {
        const uint tinyMaxFrameBytes = 4096;
        using CancellationTokenSource cancellation = new(TimeSpan.FromSeconds(15));
        using PipePair pipePair = await PipePair.CreateAsync(cancellation.Token);
        FakeRuntimeSession runtime = new()
        {
            SuppressDrainForBatchSize = 128,
            BackingQueue = CreateByteHeavyQueue(eventCount: 1, payloadBytes: 16 * 1024),
            IgnoreDrainByteBudget = true,
        };
        WorkerPipeSession session = CreatePipeSession(pipePair.WorkerStream, runtime);
        Task runTask = session.RunAsync(cancellation.Token);
        await CompleteGatewayHandshakeAsync(pipePair, tinyMaxFrameBytes, cancellation.Token);
        await pipePair.GatewayWriter
            .WriteAsync(
                CreateControlCommandEnvelope(
                    "drain-too-large",
                    MxCommandKind.DrainEvents,
                    command => command.DrainEvents = new DrainEventsCommand { MaxEvents = 0 }),
                cancellation.Token);
        WorkerEnvelope replyEnvelope = await ReadUntilAsync(
            pipePair.GatewayReader,
            WorkerEnvelope.BodyOneofCase.WorkerCommandReply,
            cancellation.Token);
        MxCommandReply reply = replyEnvelope.WorkerCommandReply.Reply;
        Assert.Equal("drain-too-large", reply.CorrelationId);
        Assert.Equal(MxCommandKind.DrainEvents, reply.Kind);
        Assert.Equal(ProtocolStatusCode.InvalidRequest, reply.ProtocolStatus.Code);
        Assert.Contains("frame maximum", reply.ProtocolStatus.Message);
        Assert.False(runTask.IsCompleted, "An oversized control reply must not end the session.");
        await SendShutdownAndWaitAsync(pipePair, runTask, cancellation.Token);
    }
    /// <summary>
    ///     Verifies the same backstop on the STA command path: an oversized reply from a dispatched
    ///     command answers its correlation with an error reply instead of falling into the generic
    ///     catch that faults the whole session with MxaccessCommandFailed.
    /// </summary>
    /// <returns>A task that represents the asynchronous operation.</returns>
    [Fact]
    public async Task CommandReplyTooLarge_WritesErrorReplyInsteadOfFaulting()
    {
        const uint tinyMaxFrameBytes = 4096;
        using CancellationTokenSource cancellation = new(TimeSpan.FromSeconds(15));
        using PipePair pipePair = await PipePair.CreateAsync(cancellation.Token);
        FakeRuntimeSession runtime = new()
        {
            DispatchReplyDiagnosticMessage = new string('x', 16 * 1024),
        };
        WorkerPipeSession session = CreatePipeSession(pipePair.WorkerStream, runtime);
        Task runTask = session.RunAsync(cancellation.Token);
        await CompleteGatewayHandshakeAsync(pipePair, tinyMaxFrameBytes, cancellation.Token);
        await pipePair.GatewayWriter
            .WriteAsync(CreateCommandEnvelope("command-too-large"), cancellation.Token);
        WorkerEnvelope replyEnvelope = await ReadUntilAsync(
            pipePair.GatewayReader,
            WorkerEnvelope.BodyOneofCase.WorkerCommandReply,
            envelope => envelope.WorkerCommandReply.Reply.CorrelationId == "command-too-large",
            cancellation.Token);
        MxCommandReply reply = replyEnvelope.WorkerCommandReply.Reply;
        Assert.Equal(MxCommandKind.Register, reply.Kind);
        Assert.Equal(ProtocolStatusCode.InvalidRequest, reply.ProtocolStatus.Code);
        // No fault, and the session still reports itself Ready rather than Faulted.
        await pipePair.GatewayWriter
            .WriteAsync(
                CreateControlCommandEnvelope(
                    "state-after-too-large",
                    MxCommandKind.GetSessionState,
                    command => command.GetSessionState = new GetSessionStateCommand()),
                cancellation.Token);
        WorkerEnvelope stateEnvelope = await ReadUntilAsync(
            pipePair.GatewayReader,
            WorkerEnvelope.BodyOneofCase.WorkerCommandReply,
            envelope => envelope.WorkerCommandReply.Reply.CorrelationId == "state-after-too-large",
            cancellation.Token);
        Assert.Equal(SessionState.Ready, stateEnvelope.WorkerCommandReply.Reply.SessionState.State);
        Assert.False(runTask.IsCompleted, "An oversized command reply must not fault the session.");
        await SendShutdownAndWaitAsync(pipePair, runTask, cancellation.Token);
    }
    /// <summary>
    ///     IPC-30. An event above the negotiated frame maximum is undeliverable end to end (the pipe
    ///     maximum sits only an envelope reserve above the public gRPC cap), so the session stays
    ///     fatal by design — but the death must be structured: a WorkerFault naming the event, with
    ///     no value payload in it, before the process exits. Silently dropping the event or
    ///     synthesizing a placeholder were both rejected.
    /// </summary>
    /// <returns>A task that represents the asynchronous operation.</returns>
    [Fact]
    public async Task RunAsync_EventFrameTooLarge_WritesStructuredFaultThenEndsSession()
    {
        const uint tinyMaxFrameBytes = 4096;
        using CancellationTokenSource cancellation = new(TimeSpan.FromSeconds(15));
        using PipePair pipePair = await PipePair.CreateAsync(cancellation.Token);
        FakeRuntimeSession runtime = new();
        RecordingWorkerLogger logger = new();
        WorkerPipeSession session = CreatePipeSession(
            pipePair.WorkerStream,
            runtime,
            new WorkerPipeSessionOptions
            {
                HeartbeatInterval = TimeSpan.FromMilliseconds(100),
                HeartbeatGrace = TimeSpan.FromSeconds(5),
            },
            logger);
        runtime.EnqueueEvent(CreateOversizedWorkerEvent(sequence: 77, payloadBytes: 16 * 1024));
        Task runTask = session.RunAsync(cancellation.Token);
        await CompleteGatewayHandshakeAsync(pipePair, tinyMaxFrameBytes, cancellation.Token);
        WorkerEnvelope faultEnvelope = await ReadUntilAsync(
            pipePair.GatewayReader,
            WorkerEnvelope.BodyOneofCase.WorkerFault,
            cancellation.Token);
        WorkerFault fault = faultEnvelope.WorkerFault;
        Assert.Equal(WorkerFaultCategory.ProtocolViolation, fault.Category);
        Assert.Equal("EventDrain", fault.CommandMethod);
        Assert.Contains("77", fault.DiagnosticMessage);
        Assert.Contains("MaxMessageBytes", fault.DiagnosticMessage);
        // The identity is reported; the value payload never is.
        Assert.DoesNotContain(new string('x', 64), fault.DiagnosticMessage);
        Assert.Contains(
            logger.Events,
            entry => entry.EventName == "WorkerEventFrameTooLarge"
                && entry.Fields.TryGetValue("worker_sequence", out object? sequence)
                && sequence is ulong sequenceValue
                && sequenceValue == 77UL);
        // The session ends, and the fault frame parsed cleanly off the same stream above — the
        // rejected event never corrupted the wire.
        Task completedTask = await Task.WhenAny(runTask, Task.Delay(TimeSpan.FromSeconds(5), cancellation.Token));
        Assert.Same(runTask, completedTask);
        await Assert.ThrowsAsync<InvalidOperationException>(async () => await runTask);
    }
    /// <summary>
    ///     Verifies that ShutdownWorker returns its OK reply BEFORE the graceful
    ///     shutdown runs and disposes the runtime session, and that the message
@@ -1241,6 +1611,15 @@ public sealed class WorkerPipeSessionTests
        Stream stream,
        FakeRuntimeSession runtime,
        WorkerPipeSessionOptions sessionOptions)
    {
        return CreatePipeSession(stream, runtime, sessionOptions, logger: null);
    }
    private static WorkerPipeSession CreatePipeSession(
        Stream stream,
        FakeRuntimeSession runtime,
        WorkerPipeSessionOptions sessionOptions,
        ZB.MOM.WW.MxGateway.Worker.Bootstrap.IWorkerLogger? logger)
    {
        WorkerFrameProtocolOptions options = CreateOptions();
        return new WorkerPipeSession(
@@ -1249,7 +1628,8 @@ public sealed class WorkerPipeSessionTests
            options,
            () => 1234,
            sessionOptions,
-            () => runtime);
+            () => runtime,
            logger);
    }
    private static WorkerFrameProtocolOptions CreateOptions()
@@ -1270,7 +1650,8 @@ public sealed class WorkerPipeSessionTests
    private static WorkerEnvelope CreateGatewayHelloEnvelope(
        string nonce = Nonce,
        uint supportedProtocolVersion = GatewayContractInfo.WorkerProtocolVersion,
-        ulong sequence = 1)
+        ulong sequence = 1,
        uint maxFrameBytes = 0)
    {
        return new WorkerEnvelope
        {
@@ -1282,6 +1663,10 @@ public sealed class WorkerPipeSessionTests
                SupportedProtocolVersion = supportedProtocolVersion,
                Nonce = nonce,
                GatewayVersion = "test-gateway",
                // 0 leaves the worker on its compile-time default; a non-zero value is adopted
                // during the handshake and becomes the frame maximum every later assertion is
                // measured against.
                MaxFrameBytes = maxFrameBytes,
            },
        };
    }
@@ -1392,12 +1777,55 @@ public sealed class WorkerPipeSessionTests
        };
    }
-    private static async Task CompleteGatewayHandshakeAsync(
+    private static WorkerEvent CreateOversizedWorkerEvent(ulong sequence, int payloadBytes)
    {
        WorkerEvent workerEvent = CreateWorkerEvent(sequence);
        workerEvent.Event.ItemHandle = 42;
        workerEvent.Event.RawStatus = new string('x', payloadBytes);
        return workerEvent;
    }
    /// <summary>
    /// Fills a real event queue with byte-heavy events — a large string field stands in for the
    /// array/string <c>MxValue</c> payloads that make a count-capped drain overshoot the frame
    /// maximum. The queue is real (not the fake's plain list) so the production byte-budgeting runs.
    /// </summary>
    /// <param name="eventCount">Number of events to enqueue.</param>
    /// <param name="payloadBytes">Size of each event's raw-status payload string.</param>
    /// <returns>The populated queue.</returns>
    private static MxAccessEventQueue CreateByteHeavyQueue(int eventCount, int payloadBytes)
    {
        MxAccessEventQueue queue = new(Math.Max(eventCount, 1));
        string payload = new string('x', payloadBytes);
        for (int index = 0; index < eventCount; index++)
        {
            queue.Enqueue(new MxEvent
            {
                SessionId = SessionId,
                Family = MxEventFamily.OnDataChange,
                ItemHandle = index,
                RawStatus = payload,
                OnDataChange = new OnDataChangeEvent(),
            });
        }
        return queue;
    }
    private static Task CompleteGatewayHandshakeAsync(
        PipePair pipePair,
        CancellationToken cancellationToken)
    {
        return CompleteGatewayHandshakeAsync(pipePair, maxFrameBytes: 0, cancellationToken);
    }
    private static async Task CompleteGatewayHandshakeAsync(
        PipePair pipePair,
        uint maxFrameBytes,
        CancellationToken cancellationToken)
    {
        await pipePair.GatewayWriter
-            .WriteAsync(CreateGatewayHelloEnvelope(), cancellationToken)
+            .WriteAsync(CreateGatewayHelloEnvelope(maxFrameBytes: maxFrameBytes), cancellationToken)
            .ConfigureAwait(false);
        WorkerEnvelope hello = await pipePair.GatewayReader.ReadAsync(cancellationToken).ConfigureAwait(false);
@@ -98,6 +98,157 @@ public sealed class MxAccessEventQueueTests
        Assert.Equal(0, queue.Count);
    }
    /// <summary>
    ///     Verifies the byte-budgeted drain stops before the budget is exceeded, leaves the
    ///     remainder queued in order, and reports the exact remaining count (WRK-21). Events that
    ///     do not fit must never be dequeued — dequeuing them is how the pre-fix drain lost events
    ///     when the reply frame was rejected.
    /// </summary>
    [Fact]
    public void Drain_ByteBudget_StopsBeforeBudgetAndLeavesRemainderQueued()
    {
        MxAccessEventQueue queue = new(capacity: 8);
        for (int itemHandle = 0; itemHandle < 5; itemHandle++)
        {
            queue.Enqueue(CreateEventWithPayload(itemHandle, payloadLength: 512));
        }
        int perEventCost = MeasureDrainCost(payloadLength: 512);
        // Budget for exactly two events (plus a sliver too small for a third).
        IReadOnlyList<WorkerEvent> drained =
            queue.Drain(maxEvents: 0, maxTotalBytes: (perEventCost * 2) + (perEventCost / 2)).Events;
        Assert.Equal(2, drained.Count);
        Assert.Equal(0, drained[0].Event.ItemHandle);
        Assert.Equal(1, drained[1].Event.ItemHandle);
        Assert.Equal(3, queue.Count);
        // The undrained remainder is still present, still in order.
        IReadOnlyList<WorkerEvent> rest = queue.Drain(maxEvents: 0);
        Assert.Equal(new[] { 2, 3, 4 }, new[] { rest[0].Event.ItemHandle, rest[1].Event.ItemHandle, rest[2].Event.ItemHandle });
    }
    /// <summary>
    ///     Verifies the byte-budgeted drain reports truncation and the exact remaining count so the
    ///     DrainEvents reply can tell the caller to drain again.
    /// </summary>
    [Fact]
    public void Drain_ByteBudget_ReportsTruncationAndRemainingCount()
    {
        MxAccessEventQueue queue = new(capacity: 8);
        for (int itemHandle = 0; itemHandle < 4; itemHandle++)
        {
            queue.Enqueue(CreateEventWithPayload(itemHandle, payloadLength: 256));
        }
        // One-and-a-half events' worth of budget: the head fits, the next does not, and the next is
        // comfortably smaller than the whole budget so it is a plain truncation rather than the
        // oversized-head case.
        WorkerEventDrainResult result = queue.Drain(
            maxEvents: 0,
            maxTotalBytes: MeasureDrainCost(payloadLength: 256) * 3 / 2);
        Assert.Single(result.Events);
        Assert.True(result.TruncatedBySize);
        Assert.Equal(3, result.RemainingCount);
        Assert.Equal(0UL, result.OversizedHeadSequence);
    }
    /// <summary>
    ///     Verifies the degenerate case: a head event whose own serialized size exceeds the whole
    ///     budget is not drained (draining it would build an oversized reply or lose the event) and
    ///     its worker sequence is reported so an operator can find the offending tag.
    /// </summary>
    [Fact]
    public void Drain_ByteBudget_OversizedHead_DrainsNothingAndReportsHeadSequence()
    {
        MxAccessEventQueue queue = new(capacity: 8);
        queue.Enqueue(CreateEventWithPayload(itemHandle: 0, payloadLength: 4096));
        queue.Enqueue(CreateEventWithPayload(itemHandle: 1, payloadLength: 8));
        WorkerEventDrainResult result = queue.Drain(maxEvents: 0, maxTotalBytes: 1024);
        Assert.Empty(result.Events);
        Assert.True(result.TruncatedBySize);
        Assert.Equal(2, result.RemainingCount);
        Assert.Equal(1UL, result.OversizedHeadSequence);
        // The blocked event is still queued — it was never removed.
        Assert.Equal(2, queue.Count);
    }
    /// <summary>
    ///     The no-loss half of the WRK-21 acceptance criterion, at full scale. Draining the review's
    ///     10,000 byte-heavy events under a budget that fits only a fraction of them per call must
    ///     return every event exactly once and in order: the pre-fix drain removed events from the
    ///     queue before the reply was sized, so a rejected frame destroyed them. This runs at the
    ///     queue layer because the property is the queue's, and because the pipe harness that covers
    ///     the same walk end to end cannot sustain hundreds of large round trips.
    /// </summary>
    [Fact]
    public void Drain_ByteBudget_RepeatedCalls_RecoverAllEventsInOrderWithoutLoss()
    {
        const int eventCount = 10_000;
        const int payloadLength = 1_800;
        MxAccessEventQueue queue = new(eventCount);
        for (int index = 0; index < eventCount; index++)
        {
            queue.Enqueue(CreateEventWithPayload(index, payloadLength));
        }
        // A budget that fits roughly 35 events, so the walk takes hundreds of calls.
        int budget = MeasureDrainCost(payloadLength) * 35;
        List<ulong> recovered = new();
        int calls = 0;
        while (true)
        {
            WorkerEventDrainResult result = queue.Drain(maxEvents: 0, maxTotalBytes: budget);
            calls++;
            if (result.Events.Count == 0)
            {
                break;
            }
            foreach (WorkerEvent drained in result.Events)
            {
                recovered.Add(drained.Event.WorkerSequence);
            }
            Assert.Equal(eventCount - recovered.Count, result.RemainingCount);
            Assert.True(calls < eventCount, "Drain made no progress.");
        }
        Assert.True(calls > 100, $"Expected the byte budget to split the drain, saw {calls} calls.");
        Assert.Equal(eventCount, recovered.Count);
        for (int index = 0; index < recovered.Count; index++)
        {
            Assert.Equal((ulong)(index + 1), recovered[index]);
        }
        Assert.Equal(0, queue.Count);
    }
    /// <summary>
    ///     Verifies the count cap still binds when the byte budget is generous: the byte cap is an
    ///     additional bound, not a replacement.
    /// </summary>
    [Fact]
    public void Drain_ByteBudget_CountCapStillBinds()
    {
        MxAccessEventQueue queue = new(capacity: 8);
        for (int itemHandle = 0; itemHandle < 5; itemHandle++)
        {
            queue.Enqueue(CreateEventWithPayload(itemHandle, payloadLength: 16));
        }
        WorkerEventDrainResult result = queue.Drain(maxEvents: 2, maxTotalBytes: 1024 * 1024);
        Assert.Equal(2, result.Events.Count);
        Assert.False(result.TruncatedBySize);
        Assert.Equal(3, result.RemainingCount);
    }
    /// <summary>Verifies that Enqueue is rejected after a fault is recorded manually.</summary>
    [Fact]
    public void Enqueue_AfterRecordFault_ThrowsInvalidOperationException()
@@ -149,6 +300,42 @@ public sealed class MxAccessEventQueueTests
        Assert.Equal(WorkerFaultCategory.MxaccessEventConversionFailed, queue.Fault?.Category);
    }
    // Mirrors MxAccessEventQueue's per-event repeated-field allowance. Kept local rather than made
    // public on the queue: the byte-budget tests state their budgets in units of that charge, so a
    // change to it should surface here as a failing bound instead of silently moving with the code.
    private const int RepeatedFieldOverheadBytes = 8;
    /// <summary>
    /// Measures what the queue charges one event of the given payload size against the byte budget:
    /// the serialized <see cref="WorkerEvent"/> as it exists after Enqueue (sequence and timestamp
    /// stamped) plus the repeated-field allowance. The probe uses item handle 0, a proto3 default
    /// that is not serialized, so this is a lower bound on the fixtures' real per-event cost — the
    /// budgets above carry slack rather than assuming byte equality.
    /// </summary>
    /// <param name="payloadLength">Length of the event's raw-status payload string.</param>
    /// <returns>The per-event byte cost.</returns>
    private static int MeasureDrainCost(int payloadLength)
    {
        MxAccessEventQueue probe = new(capacity: 1);
        probe.Enqueue(CreateEventWithPayload(0, payloadLength));
        Assert.True(probe.TryDequeue(out WorkerEvent? probeEvent));
        return probeEvent!.CalculateSize() + RepeatedFieldOverheadBytes;
    }
    /// <summary>
    /// Builds a byte-heavy event: a large string field is the cheapest stand-in for the array/string
    /// <see cref="MxValue"/> payloads that make a count-capped drain overshoot the frame maximum.
    /// </summary>
    /// <param name="itemHandle">Item handle identifying the event in assertions.</param>
    /// <param name="payloadLength">Length of the raw-status payload string.</param>
    /// <returns>The constructed event.</returns>
    private static MxEvent CreateEventWithPayload(int itemHandle, int payloadLength)
    {
        MxEvent mxEvent = CreateEvent(MxEventFamily.OnDataChange, itemHandle);
        mxEvent.RawStatus = new string('x', payloadLength);
        return mxEvent;
    }
    private static MxEvent CreateEvent(
        MxEventFamily family,
        int itemHandle)
@@ -43,6 +43,13 @@ internal sealed class FakeRuntimeSession : IWorkerRuntimeSession
    /// <summary>Gets or sets whether ShutdownGracefullyAsync throws a TimeoutException.</summary>
    public bool ThrowTimeoutOnShutdown { get; set; }
    /// <summary>
    ///     Optional diagnostic message stuffed into every dispatched command reply. A long value
    ///     pushes the STA command reply past a small negotiated frame maximum, which is how a test
    ///     drives the <c>ProcessCommandAsync</c> reply-size backstop.
    /// </summary>
    public string? DispatchReplyDiagnosticMessage { get; set; }
    /// <summary>Gets a value indicating whether Dispose was called.</summary>
    public bool Disposed { get; private set; }
@@ -92,7 +99,7 @@ internal sealed class FakeRuntimeSession : IWorkerRuntimeSession
                    throw new InvalidOperationException("Command failed after shutdown started.");
                }
-                return new MxCommandReply
+                MxCommandReply reply = new()
                {
                    SessionId = command.SessionId,
                    CorrelationId = command.CorrelationId,
@@ -103,6 +110,13 @@ internal sealed class FakeRuntimeSession : IWorkerRuntimeSession
                        Message = "OK",
                    },
                };
                if (DispatchReplyDiagnosticMessage is not null)
                {
                    reply.DiagnosticMessage = DispatchReplyDiagnosticMessage;
                }
                return reply;
            });
    }
@@ -133,6 +147,27 @@ internal sealed class FakeRuntimeSession : IWorkerRuntimeSession
    /// </summary>
    public uint? LastDrainMaxEvents { get; private set; }
    /// <summary>
    ///     Optional real event queue backing the drain paths. When set, both
    ///     <see cref="DrainEvents(uint)"/> and <see cref="DrainEvents(uint, int)"/> delegate to it
    ///     so a test can exercise the production byte-budgeting logic behind the fake session.
    /// </summary>
    public MxAccessEventQueue? BackingQueue { get; set; }
    /// <summary>
    ///     When set, <see cref="DrainEvents(uint, int)"/> ignores the byte budget and drains purely
    ///     by count. Simulates the "sizing bug or future command" case the control-reply size
    ///     backstop exists for, so a test can drive an oversized reply without a real budgeting
    ///     defect.
    /// </summary>
    public bool IgnoreDrainByteBudget { get; set; }
    /// <summary>
    ///     Records the <c>maxTotalBytes</c> argument of the most recent byte-budgeted
    ///     <see cref="DrainEvents(uint, int)"/> call.
    /// </summary>
    public int? LastDrainMaxTotalBytes { get; private set; }
    /// <inheritdoc />
    public IReadOnlyList<WorkerEvent> DrainEvents(uint maxEvents)
    {
@@ -143,6 +178,76 @@ internal sealed class FakeRuntimeSession : IWorkerRuntimeSession
        LastDrainMaxEvents = maxEvents;
        if (BackingQueue is not null)
        {
            return BackingQueue.Drain(maxEvents);
        }
        lock (gate)
        {
            int drainCount = maxEvents == 0
                ? events.Count
                : Math.Min(events.Count, checked((int)Math.Min(maxEvents, int.MaxValue)));
            List<WorkerEvent> drained = new(drainCount);
            for (int index = 0; index < drainCount; index++)
            {
                drained.Add(events.Dequeue());
            }
            return drained;
        }
    }
    /// <inheritdoc />
    public WorkerEventDrainResult DrainEvents(uint maxEvents, int maxTotalBytes)
    {
        if (SuppressDrainForBatchSize is uint suppressed && maxEvents == suppressed)
        {
            return new WorkerEventDrainResult(
                Array.Empty<WorkerEvent>(),
                truncatedBySize: false,
                remainingCount: PendingEventCount,
                oversizedHeadSequence: 0);
        }
        LastDrainMaxEvents = maxEvents;
        LastDrainMaxTotalBytes = maxTotalBytes;
        if (BackingQueue is not null && !IgnoreDrainByteBudget)
        {
            return BackingQueue.Drain(maxEvents, maxTotalBytes);
        }
        // Count-only drain: either no backing queue (the simple fakes) or a deliberately
        // budget-blind drain used to exercise the reply-size backstop.
        IReadOnlyList<WorkerEvent> drained = BackingQueue is not null
            ? BackingQueue.Drain(maxEvents)
            : DrainByCount(maxEvents);
        return new WorkerEventDrainResult(
            drained,
            truncatedBySize: false,
            remainingCount: PendingEventCount,
            oversizedHeadSequence: 0);
    }
    private int PendingEventCount
    {
        get
        {
            if (BackingQueue is not null)
            {
                return BackingQueue.Count;
            }
            lock (gate)
            {
                return events.Count;
            }
        }
    }
    private IReadOnlyList<WorkerEvent> DrainByCount(uint maxEvents)
    {
        lock (gate)
        {
            int drainCount = maxEvents == 0
@@ -43,8 +43,9 @@ public sealed class WorkerFrameWriter
    private readonly Queue<PendingFrame> _eventFrames = new Queue<PendingFrame>();
    // Only ever read/written by the current write-lock holder while draining, so no interlock is
-    // needed. Starts at 0 and is pre-incremented, so the first written frame carries sequence 1
+    // needed. Starts at 0 and is committed only immediately before the stream write, so the first
-    // (matching the previous behaviour).
+    // written frame carries sequence 1 and a per-frame rejection leaves the counter untouched —
    // the next accepted frame reuses the number and the wire sequence stays contiguous.
    private ulong _nextSequence;
    /// <summary>Initializes a new instance of the WorkerFrameWriter class.</summary>
@@ -237,7 +238,14 @@ public sealed class WorkerFrameWriter
        // Stamp the sequence at the actual point of writing, under the write lock, so the wire order
        // and the stamped sequence agree regardless of caller concurrency or priority.
-        envelope.Sequence = unchecked(++_nextSequence);
+        //
        // Peek-stamp-commit (WRK-23): the sequence participates in CalculateSize() (varint width),
        // so it must be stamped before the size checks — but a per-frame rejection must not burn a
        // number, or the wire shows phantom gaps that an operator reads as lost frames. Stamp a
        // candidate, validate the stamped envelope, and commit the counter only once the frame is
        // certain to be written.
        ulong candidateSequence = unchecked(_nextSequence + 1);
        envelope.Sequence = candidateSequence;
        int payloadLength = envelope.CalculateSize();
        if (payloadLength == 0)
@@ -254,6 +262,8 @@ public sealed class WorkerFrameWriter
                $"Worker envelope payload length {payloadLength} exceeds the configured maximum of {_options.MaxMessageBytes} bytes.");
        }
        _nextSequence = candidateSequence;
        // Serialize once into a single buffer that carries the 4-byte length prefix followed by the
        // payload, then issue one stream write. This avoids a second serialization pass, a separate
        // prefix array, and a separate prefix write. The flush is deferred to the end of the drained
@@ -5,6 +5,7 @@ using System.IO;
 using System.Threading;
 using System.Threading.Tasks;
 using Google.Protobuf.WellKnownTypes;
 using ZB.MOM.WW.MxGateway.Contracts;
 using ZB.MOM.WW.MxGateway.Contracts.Proto;
 using ZB.MOM.WW.MxGateway.Worker.Bootstrap;
 using ZB.MOM.WW.MxGateway.Worker.MxAccess;
@@ -18,12 +19,11 @@ public sealed class WorkerPipeSession
    private static readonly TimeSpan BackgroundTaskStopTimeout = TimeSpan.FromSeconds(1);
    private const uint EventDrainBatchSize = 128;
-    // Hard cap on how many events a single DrainEvents diagnostic reply may carry. DrainEvents is a
+    // Headroom subtracted from the negotiated frame maximum when budgeting a DrainEvents reply. It
-    // non-streaming control command, so an unbounded drain (including the max_events = 0 "as many as
+    // covers the WorkerEnvelope/WorkerCommandReply/MxCommandReply wrapper the drained events are
-    // available" request) could pack the whole queue into one session-killing reply frame.
+    // packed into — the same envelope-overhead reserve rationale docs/WorkerFrameProtocol.md
-    // The gateway request validator rejects requests above its public ceiling; this worker-side cap is
+    // records for the frame max itself.
-    // the backstop and defines the effective per-reply maximum. Kept in step with that public ceiling.
+    private const int DrainReplyFrameHeadroomBytes = 64 * 1024;
    private const uint MaxDrainEventsPerReply = 10_000;
    private readonly WorkerFrameProtocolOptions _options;
    private readonly Func<int> _processIdProvider;
@@ -376,13 +376,83 @@ public sealed class WorkerPipeSession
                // Events are the low-priority frame class: the writer holds them behind any pending
                // control frame (reply, fault, heartbeat, shutdown ack) so those are not delayed
                // behind an event backlog.
-                await _writer
+                try
-                    .WriteAsync(CreateEnvelope(workerEvent), WorkerFrameWritePriority.Event, cancellationToken)
+                {
-                    .ConfigureAwait(false);
+                    await _writer
                        .WriteAsync(CreateEnvelope(workerEvent), WorkerFrameWritePriority.Event, cancellationToken)
                        .ConfigureAwait(false);
                }
                catch (WorkerFrameProtocolException exception)
                    when (exception.ErrorCode == WorkerFrameProtocolErrorCode.MessageTooLarge)
                {
                    await FaultOnOversizedEventAsync(workerEvent, exception, cancellationToken)
                        .ConfigureAwait(false);
                }
            }
        }
    }
    /// <summary>
    ///     Ends the session on an event that cannot be framed, but deliberately and diagnosably
    ///     (IPC-30). An event above the negotiated frame maximum is undeliverable end to end — the
    ///     pipe maximum sits only an envelope reserve above the public gRPC cap — so dropping it
    ///     would silently make the event stream unfaithful, and synthesizing a placeholder is barred
    ///     by the no-synthesized-events rule. Instead the worker records which event blocked (never
    ///     its value: the redaction rule), writes a structured fault the gateway and dashboard can
    ///     surface, and then exits as it did before. Remediation is configuration:
    ///     <c>MxGateway:Worker:MaxMessageBytes</c>. Other per-frame rejection codes keep the previous
    ///     behavior — they indicate worker bugs, not workload size.
    /// </summary>
    private async Task FaultOnOversizedEventAsync(
        WorkerEvent workerEvent,
        WorkerFrameProtocolException exception,
        CancellationToken cancellationToken)
    {
        MxEvent? mxEvent = workerEvent.Event;
        string family = (mxEvent?.Family ?? MxEventFamily.Unspecified).ToString();
        ulong workerSequence = mxEvent?.WorkerSequence ?? 0;
        int serverHandle = mxEvent?.ServerHandle ?? 0;
        int itemHandle = mxEvent?.ItemHandle ?? 0;
        _logger?.Error(
            "WorkerEventFrameTooLarge",
            new Dictionary<string, object?>
            {
                ["session_id"] = _options.SessionId,
                ["event_family"] = family,
                ["worker_sequence"] = workerSequence,
                ["server_handle"] = serverHandle,
                ["item_handle"] = itemHandle,
                ["max_message_bytes"] = _options.MaxMessageBytes,
                // Sizes only — the event value never reaches the log.
                ["reason"] = exception.Message,
            });
        string diagnosticMessage =
            $"{family} event for server handle {serverHandle}, item handle {itemHandle} "
            + $"(worker sequence {workerSequence}) exceeds the negotiated frame maximum of "
            + $"{_options.MaxMessageBytes} bytes and cannot be delivered; raise "
            + "MxGateway:Worker:MaxMessageBytes for this workload.";
        _state = WorkerState.Faulted;
        await TryWriteFaultAsync(
            new WorkerFault
            {
                Category = WorkerFaultCategory.ProtocolViolation,
                CommandMethod = "EventDrain",
                ExceptionType = exception.GetType().FullName ?? string.Empty,
                DiagnosticMessage = diagnosticMessage,
                ProtocolStatus = new ProtocolStatus
                {
                    Code = ProtocolStatusCode.ProtocolViolation,
                    Message = diagnosticMessage,
                },
            },
            cancellationToken).ConfigureAwait(false);
        throw new InvalidOperationException(diagnosticMessage, exception);
    }
    private async Task<bool> DispatchGatewayEnvelopeAsync(
        WorkerEnvelope envelope,
        CancellationToken cancellationToken)
@@ -478,7 +548,8 @@ public sealed class WorkerPipeSession
            _ => CreateControlOkReply(correlationId, command.Kind),
        };
-        await WriteControlReplyAsync(reply, cancellationToken).ConfigureAwait(false);
+        await WriteControlReplyWithSizeBackstopAsync(reply, correlationId, command.Kind, cancellationToken)
            .ConfigureAwait(false);
        return true;
    }
@@ -495,6 +566,105 @@ public sealed class WorkerPipeSession
            cancellationToken);
    }
    /// <summary>
    ///     Writes a control reply, answering the correlation with a small error reply instead of
    ///     unwinding the session if the reply does not fit the negotiated frame maximum. Reply
    ///     builders already size their payloads (see <see cref="CreateDrainEventsReply"/>), so this
    ///     is a backstop against a future command or a sizing bug — but without it a single
    ///     oversized diagnostic reply is session-fatal, which no diagnostics command may be.
    /// </summary>
    private async Task WriteControlReplyWithSizeBackstopAsync(
        MxCommandReply reply,
        string correlationId,
        MxCommandKind kind,
        CancellationToken cancellationToken)
    {
        try
        {
            await WriteControlReplyAsync(reply, cancellationToken).ConfigureAwait(false);
        }
        catch (WorkerFrameProtocolException exception)
            when (exception.ErrorCode == WorkerFrameProtocolErrorCode.MessageTooLarge)
        {
            LogControlReplyTooLarge(correlationId, kind, exception);
            await WriteReplyTooLargeFallbackAsync(correlationId, kind, cancellationToken)
                .ConfigureAwait(false);
        }
    }
    /// <summary>
    ///     Writes the small <c>InvalidRequest</c> reply that answers a correlation whose real reply
    ///     overshot the frame maximum. The fallback itself is a handful of bytes, so it fits any
    ///     sane negotiated maximum; the only way it can also throw <c>MessageTooLarge</c> is a
    ///     pathologically tiny negotiated maximum below the gateway's validation floor — the
    ///     pre-existing WRK-24 gap, which adds the negotiated-max lower bound that makes this
    ///     unreachable. Until then, a defensive swallow keeps the "no diagnostics command is
    ///     session-fatal" invariant true even in that degenerate config: the correlation goes
    ///     unanswered and the gateway's own per-command timeout covers it, but the session lives.
    /// </summary>
    private async Task WriteReplyTooLargeFallbackAsync(
        string correlationId,
        MxCommandKind kind,
        CancellationToken cancellationToken)
    {
        try
        {
            await WriteControlReplyAsync(
                CreateReplyTooLargeReply(correlationId, kind),
                cancellationToken).ConfigureAwait(false);
        }
        catch (WorkerFrameProtocolException exception)
            when (exception.ErrorCode == WorkerFrameProtocolErrorCode.MessageTooLarge)
        {
            _logger?.Error(
                "WorkerControlReplyFallbackTooLarge",
                new Dictionary<string, object?>
                {
                    ["correlation_id"] = correlationId,
                    ["command_kind"] = kind.ToString(),
                    ["max_message_bytes"] = _options.MaxMessageBytes,
                    ["reason"] = exception.Message,
                });
        }
    }
    private void LogControlReplyTooLarge(
        string correlationId,
        MxCommandKind kind,
        WorkerFrameProtocolException exception)
    {
        _logger?.Error(
            "WorkerControlReplyTooLarge",
            new Dictionary<string, object?>
            {
                ["correlation_id"] = correlationId,
                ["command_kind"] = kind.ToString(),
                ["max_message_bytes"] = _options.MaxMessageBytes,
                // The writer's message carries the rejected payload length; it names sizes only,
                // never reply content.
                ["reason"] = exception.Message,
            });
    }
    private MxCommandReply CreateReplyTooLargeReply(string correlationId, MxCommandKind kind)
    {
        const string message =
            "Worker reply exceeded the negotiated frame maximum; retry with a smaller request.";
        return new MxCommandReply
        {
            SessionId = _options.SessionId,
            CorrelationId = correlationId,
            Kind = kind,
            Hresult = 0,
            DiagnosticMessage = message,
            ProtocolStatus = new ProtocolStatus
            {
                Code = ProtocolStatusCode.InvalidRequest,
                Message = message,
            },
        };
    }
    private MxCommandReply CreatePingReply(string correlationId, MxCommand command)
    {
        MxCommandReply reply = CreateControlOkReply(correlationId, command.Kind);
@@ -543,24 +713,77 @@ public sealed class WorkerPipeSession
        if (runtimeSession is not null)
        {
            // Bound the diagnostic drain so max_events = 0 ("as many as available") or an over-large
-            // request cannot pack the whole queue into one session-killing reply frame.
+            // request cannot pack the whole queue into one session-killing reply frame. The count cap
            // alone is not enough: byte-heavy events overshoot the negotiated frame maximum long
            // before the count ceiling, so the drain is also byte-budgeted and sizes the reply while
            // draining — an event that does not fit is left queued rather than dequeued and lost.
            uint requested = command.DrainEvents?.MaxEvents ?? 0;
-            uint maxEvents = requested == 0 || requested > MaxDrainEventsPerReply
+            uint maxEvents = requested == 0 || requested > GatewayContractInfo.MaxDrainEventsPerCommand
-                ? MaxDrainEventsPerReply
+                ? GatewayContractInfo.MaxDrainEventsPerCommand
                : requested;
-            foreach (WorkerEvent workerEvent in runtimeSession.DrainEvents(maxEvents))
+            WorkerEventDrainResult drainResult = runtimeSession.DrainEvents(
                maxEvents,
                ResolveDrainReplyByteBudget());
            foreach (WorkerEvent workerEvent in drainResult.Events)
            {
                if (workerEvent.Event is not null)
                {
                    drainReply.Events.Add(workerEvent.Event);
                }
            }
            if (drainResult.TruncatedBySize)
            {
                // DrainEventsReply has no truncation field, and adding one would regenerate every
                // language client for a diagnostic nicety. The reply's existing DiagnosticMessage
                // carries the same information at zero contract cost; the caller contract is to
                // repeat DrainEvents until it comes back empty.
                reply.DiagnosticMessage = CreateDrainTruncationMessage(
                    drainReply.Events.Count,
                    drainResult);
            }
        }
        reply.DrainEvents = drainReply;
        return reply;
    }
    /// <summary>
    ///     Byte budget for the events packed into one DrainEvents reply: the negotiated frame
    ///     maximum less a fixed reserve for the envelope/reply wrapper, but never below half the
    ///     negotiated maximum. The lower bound must be a floor, not a step: a bare
    ///     <c>subtract-then-guard-positive</c> collapses the budget to a handful of bytes just above
    ///     the reserve (e.g. at the validator-permitted floor MaxMessageBytes = 1024 + 64 KiB the
    ///     subtraction leaves 1024, too small to move even one byte-heavy event, so every drain
    ///     truncates and the drain-until-empty caller never terminates). Taking the max with
    ///     half the negotiated maximum keeps the budget monotonic across the reserve boundary while
    ///     still leaving the full reserve for the wrapper whenever the frame max is large enough
    ///     that the reserve is the smaller subtraction — which is every configuration above 128 KiB.
    /// </summary>
    private int ResolveDrainReplyByteBudget()
    {
        return Math.Max(
            _options.MaxMessageBytes - DrainReplyFrameHeadroomBytes,
            _options.MaxMessageBytes / 2);
    }
    private static string CreateDrainTruncationMessage(
        int returnedCount,
        WorkerEventDrainResult drainResult)
    {
        string message =
            $"{returnedCount} events returned, {drainResult.RemainingCount} remain; "
            + "repeat DrainEvents for the rest.";
        if (drainResult.OversizedHeadSequence != 0)
        {
            message +=
                $" The next event (worker sequence {drainResult.OversizedHeadSequence}) alone exceeds "
                + "the negotiated frame maximum and cannot be drained; raise MxGateway:Worker:MaxMessageBytes.";
        }
        return message;
    }
    private MxCommandReply CreateControlOkReply(string correlationId, MxCommandKind kind)
    {
        return new MxCommandReply
@@ -627,15 +850,30 @@ public sealed class WorkerPipeSession
                return;
            }
-            await _writer
+            try
-                .WriteAsync(
+            {
-                    CreateEnvelope(new WorkerCommandReply
+                await _writer
-                    {
+                    .WriteAsync(
-                        Reply = reply,
+                        CreateEnvelope(new WorkerCommandReply
-                        CompletedTimestamp = Timestamp.FromDateTime(DateTime.UtcNow),
+                        {
-                    }),
+                            Reply = reply,
-                    cancellationToken)
+                            CompletedTimestamp = Timestamp.FromDateTime(DateTime.UtcNow),
-                .ConfigureAwait(false);
+                        }),
                        cancellationToken)
                    .ConfigureAwait(false);
            }
            catch (WorkerFrameProtocolException sizeException)
                when (sizeException.ErrorCode == WorkerFrameProtocolErrorCode.MessageTooLarge)
            {
                // An oversized STA command reply is a property of that one command, not of the
                // session. Answer the correlation with an error reply instead of falling into the
                // generic catch below, which would fault the whole session for it. The fallback
                // write is itself size-guarded (see WriteReplyTooLargeFallbackAsync) so a degenerate
                // negotiated maximum cannot make even this backstop session-fatal.
                LogControlReplyTooLarge(envelope.CorrelationId, command.Kind, sizeException);
                await WriteReplyTooLargeFallbackAsync(envelope.CorrelationId, command.Kind, cancellationToken)
                    .ConfigureAwait(false);
            }
        }
        catch (Exception exception) when (exception is not OperationCanceledException)
        {
@@ -44,6 +44,19 @@ public interface IWorkerRuntimeSession : IDisposable
    /// <returns>List of drained events.</returns>
    IReadOnlyList<WorkerEvent> DrainEvents(uint maxEvents);
    /// <summary>
    /// Drains pending events bounded by both a count cap and a byte budget, so a caller building a
    /// single reply frame never removes an event it cannot ship.
    /// </summary>
    /// <remarks>
    ///     Declared as a second method rather than a default interface method: the worker targets
    ///     .NET Framework 4.8, which has no runtime support for default interface members.
    /// </remarks>
    /// <param name="maxEvents">Maximum number of events to drain; 0 means "no count limit".</param>
    /// <param name="maxTotalBytes">Byte budget for the drained events' estimated serialized size.</param>
    /// <returns>The drained events and the truncation facts describing what stayed queued.</returns>
    WorkerEventDrainResult DrainEvents(uint maxEvents, int maxTotalBytes);
    /// <summary>
    /// Drains a pending fault from the queue, if any.
    /// </summary>
@@ -26,6 +26,15 @@ public sealed class MxAccessEventQueue
    /// </summary>
    public const int DefaultCapacity = 10000;
    // Extra per-event slack added to WorkerEvent.CalculateSize() when charging the byte budget in
    // Drain(maxEvents, maxTotalBytes). CalculateSize() already accounts for the event's own tag and
    // length-delimiter (the WorkerEvent wrapper serializes MxEvent as field 1, and the reply packs
    // each MxEvent as DrainEventsReply.events field 1 with the identical tag+length shape), so this
    // is a pure safety margin over an already-conservative estimate — not compensation for a missing
    // wrapper. It keeps the running total strictly ahead of the true serialized size so a rounding
    // edge can never push the packed reply past the frame maximum.
    private const int RepeatedFieldOverheadBytes = 8;
    private readonly int capacity;
    private readonly Queue<WorkerEvent> events;
    private readonly object syncRoot = new();
@@ -209,6 +218,64 @@ public sealed class MxAccessEventQueue
        }
    }
    /// <summary>
    /// Drains from the head while both the count cap and a byte budget allow it, so the caller can
    /// build a reply frame that is guaranteed to fit the negotiated frame maximum.
    /// </summary>
    /// <remarks>
    ///     The size decision happens inside the queue lock, so an event is dequeued only once it is
    ///     known to fit: an event that does not fit stays at the head for the next call and is never
    ///     lost (WRK-21). Per-event cost is <c>WorkerEvent.CalculateSize()</c> — which already
    ///     includes the event's own tag and length prefix, the same shape the reply's
    ///     <c>events</c> repeated field packs it into — plus <see cref="RepeatedFieldOverheadBytes"/>
    ///     of pure slack, so the running total stays strictly ahead of the true serialized size and
    ///     the estimate errs on the safe side.
    /// </remarks>
    /// <param name="maxEvents">Maximum number of events to drain; 0 means "no count limit".</param>
    /// <param name="maxTotalBytes">Byte budget for the drained events' estimated serialized size.</param>
    /// <returns>The drained events plus the truncation facts the caller reports to the gateway.</returns>
    public WorkerEventDrainResult Drain(uint maxEvents, int maxTotalBytes)
    {
        lock (syncRoot)
        {
            int countLimit = maxEvents == 0
                ? int.MaxValue
                : checked((int)Math.Min(maxEvents, int.MaxValue));
            List<WorkerEvent> drained = new();
            int remainingBudget = maxTotalBytes;
            bool truncatedBySize = false;
            ulong oversizedHeadSequence = 0;
            while (drained.Count < countLimit && events.Count > 0)
            {
                WorkerEvent head = events.Peek();
                int cost = head.CalculateSize() + RepeatedFieldOverheadBytes;
                if (cost > remainingBudget)
                {
                    truncatedBySize = true;
                    if (cost > maxTotalBytes)
                    {
                        // The head alone cannot fit this budget, so repeating the call will not
                        // move it either. Report its sequence instead of silently stalling; the
                        // events that did fit are still returned.
                        oversizedHeadSequence = head.Event?.WorkerSequence ?? 0;
                    }
                    break;
                }
                remainingBudget -= cost;
                drained.Add(events.Dequeue());
            }
            return new WorkerEventDrainResult(
                drained,
                truncatedBySize,
                events.Count,
                oversizedHeadSequence);
        }
    }
    /// <summary>
    /// Records a fault if one has not already been recorded.
    /// </summary>
@@ -392,6 +392,12 @@ public sealed class MxAccessStaSession : IWorkerRuntimeSession
        return eventQueue.Drain(maxEvents);
    }
    /// <inheritdoc />
    public WorkerEventDrainResult DrainEvents(uint maxEvents, int maxTotalBytes)
    {
        return eventQueue.Drain(maxEvents, maxTotalBytes);
    }
    /// <inheritdoc />
    public WorkerFault? DrainFault()
    {
@@ -0,0 +1,56 @@
 using System.Collections.Generic;
 using ZB.MOM.WW.MxGateway.Contracts.Proto;
 namespace ZB.MOM.WW.MxGateway.Worker.MxAccess;
 /// <summary>
 /// Outcome of a byte-budgeted drain from the MXAccess outbound event queue.
 /// </summary>
 /// <remarks>
 ///     A count cap alone cannot keep a <c>DrainEvents</c> reply inside the negotiated frame
 ///     maximum: byte-heavy events (large string or array <c>MxValue</c>s) overshoot the frame max
 ///     long before the count ceiling is reached, and the writer's per-frame rejection then
 ///     destroys events that were already removed from the queue. The byte-budgeted drain sizes
 ///     the reply while draining, so an event that does not fit is never dequeued (WRK-21), and
 ///     this result carries the truncation facts the reply's <c>DiagnosticMessage</c> reports —
 ///     no contract change is needed to express them.
 ///     Plain constructor and get-only properties: the worker targets .NET Framework 4.8, which
 ///     has no init-only members or positional records.
 /// </remarks>
 public sealed class WorkerEventDrainResult
 {
    /// <summary>Initializes a new instance of the <see cref="WorkerEventDrainResult"/> class.</summary>
    /// <param name="events">Events removed from the queue, in enqueue order.</param>
    /// <param name="truncatedBySize">Whether the byte budget, not the count cap, ended the drain.</param>
    /// <param name="remainingCount">Number of events still queued after the drain.</param>
    /// <param name="oversizedHeadSequence">
    ///     Worker sequence of a head event whose own serialized size exceeds the whole budget, so
    ///     no future call of the same budget can ship it; 0 when there is no such event.
    /// </param>
    public WorkerEventDrainResult(
        IReadOnlyList<WorkerEvent> events,
        bool truncatedBySize,
        int remainingCount,
        ulong oversizedHeadSequence)
    {
        Events = events;
        TruncatedBySize = truncatedBySize;
        RemainingCount = remainingCount;
        OversizedHeadSequence = oversizedHeadSequence;
    }
    /// <summary>Gets the events removed from the queue, in enqueue order.</summary>
    public IReadOnlyList<WorkerEvent> Events { get; }
    /// <summary>Gets a value indicating whether the byte budget ended the drain early.</summary>
    public bool TruncatedBySize { get; }
    /// <summary>Gets the number of events still queued after the drain.</summary>
    public int RemainingCount { get; }
    /// <summary>
    /// Gets the worker sequence of the head event that alone exceeds the byte budget, or 0 when
    /// no single event blocks the drain. Naming it lets an operator find the offending tag.
    /// </summary>
    public ulong OversizedHeadSequence { get; }
 }
@@ -133,8 +133,8 @@ No placeholder/empty/`Assert.True(true)` tests were found anywhere.
 - 📄 **7.1 D1 plan header stale** — `docs/plans/2026-06-14-deferred-followups.md:4` still says *"Plan only — NOT yet executed,"* but D1 is **done** (`Dashboard/DashboardSnapshotService.cs:198`, commit `4af24b9`). Update the plan status.
 - 📄 **7.2 `AlarmClientDiscovery.md` STA "production fix needed" prose is stale** — `docs/AlarmClientDiscovery.md:765-774` reads as a pending follow-up, but alarms now run through the worker STA / `GatewayAlarmMonitor` (merged). Re-check against current code.
 - 📄 **7.3 EventsHub "publisher side is a follow-up" comment is stale** — `Dashboard/Hubs/EventsHub.cs:9-17`; the `DashboardEventBroadcaster` exists, is DI-registered (`Dashboard/DashboardServiceCollectionExtensions.cs:47`), runs in the live loop (`Grpc/EventStreamService.cs:133`), and `SessionDetailsPage.razor` renders the feed.
- 📄 **7.4 CLAUDE.md project-name drift** — CLAUDE.md uses `src/MxGateway.Server`/`MxGateway.Tests`; the actual tree is `src/ZB.MOM.WW.MxGateway.*`. Misleads path-based work.
+- ✅ **7.4 CLAUDE.md project-name drift — RESOLVED** (verified 2026-08-07): CLAUDE.md now uses the actual `src/ZB.MOM.WW.MxGateway.*` project paths throughout.
- ⚪ **7.5 Dead `MapSqlException` helper** — `Grpc/GalaxyRepositoryGrpcService.cs:350-360`, IDE0051-suppressed, kept for a hypothetical direct-SQL path that doesn't exist.
+- ✅ **7.5 Dead `MapSqlException` helper — OBSOLETE** (noted 2026-08-07): `Grpc/GalaxyRepositoryGrpcService.cs` was deleted wholesale when the gateway adopted the shared `ZB.MOM.WW.GalaxyRepository` 0.2.0 package (2026-06-25, commit `8e196a7`), taking the dead helper with it.
 - **7.6 Accepted code-review gaps (`Won't Fix`, by design):**
  - `Client.Python-012` — `Session.invoke_raw` deliberately skips `ensure_mxaccess_success`, so an embedded MXAccess HRESULT failure surfaces silently (raw-parity inspection). `code-reviews/Client.Python/findings.md:290`.
  - `Contracts-003` — closed as not-a-defect. `code-reviews/Contracts/findings.md`.
Author	SHA1	Message	Date
Joseph Doherty	758277bc62	docs(tracking): record WRK-21 review follow-ups (monotonic budget, guarded fallback) with windev evidence ci / windows-x86 (push) Successful in 1m19s Details ci / nightly-windev (push) Has been skipped Details ci / java (push) Successful in 2m15s Details ci / portable (push) Successful in 9m30s Details	2026-08-07 07:13:46 -04:00
Joseph Doherty	6bc3f9b991	fix(WRK-21): make drain budget monotonic at the reserve boundary; guard the reply-too-large fallback ci / windows-x86 (push) Successful in 1m16s Details ci / nightly-windev (push) Has been skipped Details ci / java (push) Successful in 2m17s Details ci / portable (push) Successful in 9m31s Details Review follow-ups on the WRK-21 cluster. 1. ResolveDrainReplyByteBudget was a step function, not a floor: just above the 64 KiB reserve the budget collapsed to a few bytes (at the validator-permitted floor MaxMessageBytes = 1024 + 64 KiB it was exactly 1024), too small to move a byte-heavy event, so DrainEvents truncated on every call and the drain-until- empty loop never terminated. It now takes the max of (frameMax - reserve) and frameMax/2, so the budget is monotonic and never below half the frame max. New test DrainEvents_AtValidatorFloorFrameMax_MakesProgressAndTerminates drives a byte-heavy queue at the exact validator floor and asserts it drains to empty with no head ever reported oversized. 2. The reply-too-large fallback write is now itself size-guarded (WriteReplyTooLargeFallbackAsync, used by both the control and STA reply seams): at a pathologically tiny negotiated max below the gateway's floor the fallback could also throw MessageTooLarge and — uncaught — kill the session, defeating the "no diagnostics command is session-fatal" invariant. It now log-and-swallows; comment notes WRK-24 adds the negotiated-max lower bound that makes it unreachable. 3. Corrected the RepeatedFieldOverheadBytes doc comments: WorkerEvent.CalculateSize() already includes the event's tag and length prefix (the same shape the reply's repeated events field packs), so the 8 bytes is pure slack over an already- conservative estimate, not compensation for a missing wrapper.	2026-08-07 07:09:13 -04:00
Joseph Doherty	c9925688f5	docs(tracking): record the WRK-21 cluster as Done with windev evidence ci / java (push) Successful in 2m28s Details ci / windows-x86 (push) Failing after 1m28s Details ci / nightly-windev (push) Has been skipped Details ci / portable (push) Successful in 8m38s Details Change-log row for 2026-08-07: what landed for WRK-21/WRK-28/WRK-23/IPC-30, why IPC-23 stays In progress (proto-comment/doc wave pending), and the verification evidence — macOS NonWindows build + validator tests, and the documented windev path (scripts/ci/windev-worker-ci.ps1 -Mode test) at `a256560`: x86 Worker build clean, Worker.Tests 367 passed / 0 failed / 11 skipped.	2026-08-07 06:54:24 -04:00
Joseph Doherty	a2565604df	test(WRK-21): keep the drain-to-empty walk inside the pipe harness envelope ci / nightly-windev (push) Has been skipped Details ci / windows-x86 (push) Successful in 1m14s Details ci / java (push) Successful in 2m12s Details ci / portable (push) Successful in 8m2s Details PipePair runs both ends of a duplex pipe in one process with blocking FlushFileBuffers under every frame write, so it wedges after roughly 85 large round trips. Drain the full 10,000 byte-heavy events to empty at the queue layer, where the no-loss property actually lives, and keep the pipe walk at 1,000 events (29 replies) so it still proves the split end to end. Also give the truncation test's budget slack: item handle 0 is a proto3 default and is not serialized, so the probe measurement is a lower bound on the fixture's per-event cost.	2026-08-07 06:50:17 -04:00
Joseph Doherty	7c2eaf09e2	test(WRK-21): size the byte-heavy drain fixture for the pipe harness ci / java (push) Successful in 2m12s Details ci / portable (push) Successful in 8m23s Details ci / nightly-windev (push) Has been skipped Details ci / windows-x86 (push) Failing after 11m46s Details PipePair has no continuous read pump — the test thread drains the pipe only while it sits in ReadUntilAsync — so multi-megabyte DrainEvents frames interleaved with the heartbeat loop wedge both ends inside FlushFileBuffers, each waiting for the other to read. Negotiate a 128 KiB frame maximum instead: the 10,000 byte-heavy events still overflow it many times over, so every assertion (bounded reply, reported truncation, no event loss across repeated drains, surviving session) is unchanged.	2026-08-07 06:25:15 -04:00
Joseph Doherty	33ba612ddd	fix(WRK-21,WRK-28,WRK-23,IPC-30): byte-budget the DrainEvents reply, stop size errors from killing sessions ci / nightly-windev (push) Has been skipped Details ci / java (push) Successful in 2m8s Details ci / portable (push) Successful in 7m41s Details ci / windows-x86 (push) Failing after 12m32s Details WRK-21 — DrainEvents was bounded by event count only, so a byte-heavy queue (large string/array MxValues) built a reply above the negotiated frame maximum: the writer rejected the frame, the exception unwound the session, and the events already dequeued were destroyed. The drain is now byte-budgeted inside the queue lock, so an event is dequeued only once it is known to fit and one that does not stays at the head. Truncation is reported through the reply's existing DiagnosticMessage (no contract change); callers drain until an empty reply. Both reply-write seams — the control-command path and ProcessCommandAsync — now catch MessageTooLarge and answer the correlation with an InvalidRequest reply instead of unwinding or faulting the session. Satisfies IPC-23 R1-R3. WRK-28 — the 10,000 drain ceiling moves to GatewayContractInfo .MaxDrainEventsPerCommand, referenced by both the gateway request validator and the worker clamp, replacing a comment-only sync contract. C# const only; no .proto change. WRK-23 — WorkerFrameWriter now peek-stamps, validates, then commits the sequence counter immediately before the stream write, so a per-frame rejection leaves no phantom gap on the wire. IPC-30 — an oversized event frame stays session-fatal (it is undeliverable end to end and neither dropping nor synthesizing a replacement is allowed), but the death is structured: the event's identity and sizes are logged (never its value), a WorkerFault with category PROTOCOL_VIOLATION and command method EventDrain is written, then the session exits as before. Docs updated in the same change: MxAccessWorkerInstanceDesign.md (drain byte cap, truncation contract, oversized-head behavior, oversized-event policy, no control reply is session-fatal on size), WorkerFrameProtocol.md (reply pre-sizing, non-fatal reply-size rule, oversized-event policy, rejected frames do not consume sequence numbers), gateway.md (DrainEvents two-axis bound).	2026-08-07 05:38:23 -04:00
Joseph Doherty	ead921cace	docs: truth sweep — Galaxy adoption, Auth 0.1.5, redaction seam, resolved A2 caveats ci / nightly-windev (push) Has been skipped Details ci / windows-x86 (push) Successful in 1m16s Details ci / java (push) Successful in 3m21s Details ci / portable (push) Successful in 8m21s Details Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1	2026-08-07 01:57:01 -04:00
Joseph Doherty	47c0b646a9	fix(logging): redact command values on the shared ILogRedactor seam ci / windows-x86 (push) Failing after 18s Details ci / nightly-windev (push) Has been skipped Details ci / java (push) Successful in 2m13s Details ci / portable (push) Successful in 7m6s Details GatewayLogRedactor.RedactCommandValue had no production caller. Its only one was GatewayLogRedactorAdapter on the abandoned feat/adopt-zb-telemetry-serilog branch; when that work was re-implemented on main as GatewayLogRedactorSeam the identity half was carried over and the command-value half was not. Four unit tests kept the policy green, so it read as wired while masking nothing. No live leak today — no log statement currently emits a CommandValue property — but the next one to do so would have written credential-bearing MXAccess payloads (AuthenticateUser, WriteSecured, WriteSecured2) to every sink in the clear, with passing tests suggesting otherwise. Ports the missing half onto the seam: a non-null CommandValue is masked via the existing policy, gated on CommandMethod. Value logging stays off — the seam exposes no opt-in — so ordinary values are masked too, matching RedactCommandValue's default. A null value stays null rather than becoming the placeholder, and the property is never invented when absent. Five tests added, three of which were red first on the leak itself (operator01:hunter2 reaching the assertion unmasked). The other two pin the null and absent guards. Identity redaction is untouched. Full NonWindows suite: 785 pass, 45 pre-existing macOS NamedPipe-harness failures unchanged from baseline (verified by stashing this change). Build 0 warnings.	2026-07-27 17:01:04 -04:00
Joseph Doherty	aecc50a14b	docs(claude): add Sister Projects section + cross-repo index propagation rule ci / java (push) Successful in 2m2s Details ci / windows-x86 (push) Failing after 29s Details ci / nightly-windev (push) Has been skipped Details ci / portable (push) Successful in 7m20s Details	2026-07-27 15:25:14 -04:00
Joseph Doherty	2d54ace5d9	chore(health): bump ZB.MOM.WW.Health to 0.2.0 Family version-matrix alignment. No behaviour change — mxgw registers no Akka checks, and 0.2.0's per-entry `data` object is emitted only when a check publishes some, so its health payloads are byte-identical. Note: this repo uses inline package pins, NOT central package management (there is no Directory.Packages.props), so the version lives in the Server csproj. Verified: Server builds 0 warnings. Part of scadaproj docs/plans/2026-07-22-overview-dashboard-impl-plan.md Task 2.3.	2026-07-24 05:54:40 -04:00
Joseph Doherty	8f7ee492ba	chore(secrets): bump to Secrets 0.2.3 - visible delete modal (scadaproj#2) ci / windows-x86 (push) Successful in 1m17s Details ci / nightly-windev (push) Has been skipped Details ci / java (push) Successful in 2m29s Details ci / portable (push) Successful in 7m59s Details 0.2.3's Secrets.Ui ships ConfirmDeleteModal's own styles under collision-proof zb-secrets-* class names. This host links no Bootstrap so it never exhibited the invisible-modal defect, but it takes the fixed line for parity; also rides over 0.2.1/0.2.2 (Akka-replicator fixes - inert here, no replicator in use). Tests: 780 pass, 45 fail on macOS both before and after the bump (NamedPipeServerStream multi-instance is Windows-only - the fake-worker pipe harness cannot run on this platform); zero delta from the bump. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts	2026-07-19 01:22:23 -04:00