Compare commits
11 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 758277bc62 | |||
| 6bc3f9b991 | |||
| c9925688f5 | |||
| a2565604df | |||
| 7c2eaf09e2 | |||
| 33ba612ddd | |||
| ead921cace | |||
| 47c0b646a9 | |||
| aecc50a14b | |||
| 2d54ace5d9 | |||
| 8f7ee492ba |
@@ -194,15 +194,17 @@ be **deleted**. **Keep** the mxaccessgw-specific ones that exercise behavior the
|
||||
`PersistSnapshot`, but the deployments must carry `MxGateway__Galaxy__SnapshotCachePath` and
|
||||
`MxGateway__Galaxy__PersistSnapshot` in their NSSM env on redeploy, or snapshot persistence silently
|
||||
no-ops in production.
|
||||
- **Pre-existing NU1903 (unrelated):** adding the package surfaced a transitive `SQLitePCLRaw.lib.e_sqlite3`
|
||||
2.1.11 advisory (GHSA-2m69-gcr7-jv3q, no upstream patch) that breaks the build under `TreatWarningsAsErrors`
|
||||
— already red on `main`. Resolved with a targeted `NuGetAuditSuppress` in `src/Directory.Build.props`
|
||||
(its own commit). Remove the suppression once a patched e_sqlite3 ships.
|
||||
- **Pre-existing IntegrationTests break (unrelated, NOT fixed here):** `IntegrationTests/WorkerLiveMxAccessSmokeTests.cs`
|
||||
constructs `EventStreamService` with 6 ctor args, but a prior event-stream refactor reduced that ctor — so
|
||||
the IntegrationTests project does not compile (already broken on `main`, independent of Galaxy). The Galaxy
|
||||
live tests there were rebound to the lib and compile in isolation, but the project won't build until that
|
||||
unrelated call site is fixed. Track separately.
|
||||
- **Pre-existing NU1903 (unrelated) — ✅ RESOLVED (2026-07-18, commit `2f0cfe3`):** adding the package surfaced a transitive `SQLitePCLRaw.lib.e_sqlite3`
|
||||
2.1.11 advisory (GHSA-2m69-gcr7-jv3q, at the time no upstream patch) that breaks the build under `TreatWarningsAsErrors`
|
||||
— already red on `main`. Initially resolved with a targeted `NuGetAuditSuppress` in `src/Directory.Build.props`
|
||||
(its own commit). The patched e_sqlite3 (2.1.12) has since shipped: the suppression was **removed** and the
|
||||
patched native lib pinned intentionally (`src/Directory.Build.props` now documents this in place of the suppression).
|
||||
- **Pre-existing IntegrationTests break (unrelated, NOT fixed here) — ✅ RESOLVED since:** `IntegrationTests/WorkerLiveMxAccessSmokeTests.cs`
|
||||
constructed `EventStreamService` with 6 ctor args, but a prior event-stream refactor reduced that ctor — so
|
||||
the IntegrationTests project did not compile (already broken on `main`, independent of Galaxy). The call site
|
||||
has since been fixed to match the 3-arg ctor (`sessionManager, options, metrics` —
|
||||
`Grpc/EventStreamService.cs:11-14`; call site at `WorkerLiveMxAccessSmokeTests.cs:~1580`) and the project compiles
|
||||
(verified 2026-08-07).
|
||||
- **No republish needed for the lib test additions:** the browse-projector / deploy-notifier / refresh-service
|
||||
tests were added to the lib AFTER 0.2.0 was published; tests aren't shipped, so 0.2.0 is unchanged.
|
||||
|
||||
|
||||
@@ -15,6 +15,42 @@ The architecture is a two-process design — read `gateway.md` before making str
|
||||
|
||||
The worker must do all MXAccess COM calls on its dedicated STA thread, and the STA loop must pump Windows messages (`MsgWaitForMultipleObjectsEx` + `PeekMessage`/`DispatchMessage`) so MXAccess events deliver. A plain blocking queue on an STA is not enough.
|
||||
|
||||
## Sister Projects (scadaproj umbrella)
|
||||
|
||||
`mxaccessgw` is one of a family of related SCADA / OT / Wonderware / OPC UA **sister
|
||||
projects** cloned as sibling directories under `~/Desktop/`. They are **separate repos
|
||||
and separate processes**, coupled at runtime over wire protocols (gRPC + OPC UA) — not
|
||||
by project/compile references — and share the `ZB.MOM.WW.*` product namespace. **mxaccessgw
|
||||
is the linchpin**: the only component that loads 32-bit MXAccess COM, so the others depend
|
||||
on it over gRPC rather than touching COM directly.
|
||||
|
||||
- **`~/Desktop/scadaproj`** ([`../scadaproj/CLAUDE.md`](../scadaproj/CLAUDE.md)) — the
|
||||
umbrella/index workspace that aggregates the whole family (purpose, location, stack,
|
||||
primary commands per project) and hosts the shared `ZB.MOM.WW.*` libraries, the dev/test
|
||||
GLAuth (`10.100.0.35:3893`, source of truth `scadaproj/infra/glauth/`), and the shared
|
||||
**`ZB.MOM.WW.GalaxyRepository`** package this gateway consumes. **This repo is indexed
|
||||
there** — see the MxAccessGateway entry in `../scadaproj/CLAUDE.md`.
|
||||
- **`~/Desktop/OtOpcUa`** (`lmxopcua`, [`../OtOpcUa/CLAUDE.md`](../OtOpcUa/CLAUDE.md)) — OPC
|
||||
UA server whose in-process `GalaxyDriver` **depends on this gateway** for live Galaxy
|
||||
read/write/subscribe and Galaxy Repository browse.
|
||||
- **`~/Desktop/ScadaBridge`** ([`../ScadaBridge/CLAUDE.md`](../ScadaBridge/CLAUDE.md)) —
|
||||
distributed SCADA platform whose Data Connection Layer has a dedicated **MxGateway
|
||||
adapter** that talks to this gateway directly (native MxAccess data + A&C alarms),
|
||||
bypassing OtOpcUa.
|
||||
- **`~/Desktop/HistorianGateway`** (`historiangw`,
|
||||
[`../HistorianGateway/CLAUDE.md`](../HistorianGateway/CLAUDE.md)) — single-process gRPC
|
||||
sidecar **patterned on this gateway** (session model, dashboard shell, auth interceptor,
|
||||
Galaxy SQL browse) but with no COM / no x86 worker. It also consumes the shared
|
||||
`ZB.MOM.WW.GalaxyRepository` package — the same library mxaccessgw adopted (2026-06-25),
|
||||
so the `galaxy_repository.v1` wire contract is served from one implementation.
|
||||
|
||||
**Propagate cross-repo changes to the umbrella index.** When a fact the index records about
|
||||
mxaccessgw changes here — remote/push status, the `.proto` contracts (`mxaccess_gateway.proto`,
|
||||
`mxaccess_worker.proto`, `galaxy_repository.proto`), the two-process architecture, shared-lib
|
||||
consumption, or per-project commands — update the **MxAccessGateway entry in
|
||||
[`../scadaproj/CLAUDE.md`](../scadaproj/CLAUDE.md)** in the same change so the umbrella index
|
||||
never drifts from this repo. (Mirrors the same rule in the peer repos.)
|
||||
|
||||
## Build, Test, Run
|
||||
|
||||
```powershell
|
||||
|
||||
@@ -14,7 +14,7 @@
|
||||
| SEC-06 | Done | **Partial** | Production hard-stop verified: `GatewayOptionsValidator.cs:161-165` (`Transport==None` in Production → startup error). Docs: `docs/GatewayConfiguration.md:244-248` (env-var override `MxGateway__Ldap__ServiceAccountPassword` documented). | **The committed dev service-account password is still in the repo at `appsettings.json:29`** (`Ldap.ServiceAccountPassword`) and has not been rotated — the doc itself says it "should be rotated". The transport guard shipped; the credential-removal/rotation half of the remediation did not. → **SEC-36** |
|
||||
| SEC-07 | Done | **Yes** | `Security/Authorization/GatewayGrpcScopeResolver.cs:23` (`QueryActiveAlarmsRequest => GatewayScopes.EventsRead`); both tests now construct the real type (`Tests/Security/Authorization/GatewayGrpcAuthorizationInterceptorTests.cs:330,350`). | |
|
||||
| SEC-08 | Done | **Yes** | `Security/Authentication/CachingApiKeyVerifier.cs` (15 s success-only TTL cache keyed on SHA-256 of the presented token, `:96-120`; only successes cached `:110-117`); `Security/Authentication/CoalescingMarkApiKeyStore.cs:76-113` (≤1 `last_used` write/key/60 s); wired as decorators in `Security/Authentication/AuthStoreServiceCollectionExtensions.cs:89-98`; invalidation on dashboard revoke/rotate/delete at `Dashboard/DashboardApiKeyManagementService.cs:104,144,190`; per-call constraints JSON deserialize removed via blob cache (`Security/Authentication/GatewayApiKeyIdentityMapper.cs:22-45`). Tests exist (`Tests/Security/Authentication/CachingApiKeyVerifierTests.cs`). | New surface reviewed in depth — see SEC-34 (staleness/race, bounded) below. |
|
||||
| SEC-10 | Done | **Yes** | CLI: `--expires` parsed as relative `<N>d`/`<N>h` or absolute ISO-8601 with `AssumeUniversal|AdjustToUniversal` (`Security/Authentication/ApiKeyAdminCommandLineParser.cs:242-274`), threaded into `CreateKeyAsync` (`:85-117`). Dashboard: `DashboardApiKeySummary.cs:13` (`ExpiresUtc`), snapshot projection `Dashboard/DashboardSnapshotService.cs:277`, badge logic compares against `DateTimeOffset.UtcNow` with a 7-day "Expiring" warn window (`Dashboard/Components/Pages/ApiKeysPage.razor:474-497`; `StatusBadge.razor:12-13`). Verifier-side rejection is in the shared `ZB.MOM.WW.Auth.ApiKeys` 0.1.4 (documented `docs/Authentication.md:64-65`; not readable in this repo). | UTC semantics are correct end-to-end on the gateway side. Boundary note: `expiresAt <= now` shows Expired, and relative parse rejects signed values (`NumberStyles.None`). A just-expired key can still authenticate for ≤15 s via the verification cache — see SEC-34. |
|
||||
| SEC-10 | Done | **Yes** | CLI: `--expires` parsed as relative `<N>d`/`<N>h` or absolute ISO-8601 with `AssumeUniversal|AdjustToUniversal` (`Security/Authentication/ApiKeyAdminCommandLineParser.cs:242-274`), threaded into `CreateKeyAsync` (`:85-117`). Dashboard: `DashboardApiKeySummary.cs:13` (`ExpiresUtc`), snapshot projection `Dashboard/DashboardSnapshotService.cs:277`, badge logic compares against `DateTimeOffset.UtcNow` with a 7-day "Expiring" warn window (`Dashboard/Components/Pages/ApiKeysPage.razor:474-497`; `StatusBadge.razor:12-13`). Verifier-side rejection is in the shared `ZB.MOM.WW.Auth.ApiKeys` 0.1.5 (pins bumped 0.1.4→0.1.5 in `e107019`, a transitive-dependency security fix only — the expiry enforcement is unchanged; documented `docs/Authentication.md:64-65`; not readable in this repo). | UTC semantics are correct end-to-end on the gateway side. Boundary note: `expiresAt <= now` shows Expired, and relative parse rejects signed values (`NumberStyles.None`). A just-expired key can still authenticate for ≤15 s via the verification cache — see SEC-34. |
|
||||
| SEC-11 | Done | **Yes, with new defects** | Login: fixed-window per-remote-IP limiter policy (`Dashboard/DashboardEndpointRouteBuilderExtensions.cs:27-41`), applied to POST `/auth/login` (`:83`), registered + 429 (`GatewayApplication.cs:111-126`), middleware in pipeline (`GatewayApplication.cs:46`), test `Tests/Gateway/Dashboard/DashboardLoginRateLimitTests.cs`. gRPC: `Security/Authorization/ApiKeyFailureLimiter.cs` checked **before** the store read (`GatewayGrpcAuthorizationInterceptor.cs:72-77`), failure recorded `:89`, reset on success `:97`; interceptor test asserts the short-circuit (`GatewayGrpcAuthorizationInterceptorTests.cs:395-402`). | The limiter exists and is enforced, but its key-id partitioning creates an unauthenticated lockout DoS (**SEC-31**) and its LRU eviction is flushable (**SEC-32**). |
|
||||
| SEC-12 | Done | **Yes** | `Dashboard/DashboardSessionAdminService.cs`: canonical `AuditEvent`s `dashboard-close-session`/`dashboard-kill-worker` (`:37-40`), written on Denied (`:65,147`), Success (`:89-96,171-178`), and every Failure arm (`:105,116,132,187,198,214`), category `SessionAdmin`, actor/remote/correlation captured (`:238-261`), via `IAuditWriter` (same store as API-key events). | Audit-event completeness is good: denied, not-found, faulted, and unexpected paths all emit. |
|
||||
| SEC-20 | Done | **Yes** | `Metrics/GatewayMetrics.cs:374` — `_heartbeatFailuresCounter.Add(1)` with no `session_id` tag (rationale comment `:370-373`). | The in-memory per-session map remains dashboard-only. |
|
||||
|
||||
@@ -43,9 +43,9 @@ Sequenced by cluster; a cluster is one change set.
|
||||
| GWC-25 | Medium | S | CLI-35/36 (coord) | Not started | Empty-ring ReplayGap sentinel carries `oldest_available_sequence = 0`, dead-streaming a compliant client |
|
||||
| CLI-35 | Medium | S | GWC-25 (coord) | Not started | Python CLI `stream-events` crashes on a ReplayGap |
|
||||
| CLI-36 | Medium | S | GWC-25 (coord) | Not started | Go CLI `stream-events` silently destroys the ReplayGap signal |
|
||||
| WRK-21 | Medium | M | owns IPC-23 fix; WRK-28 same batch | Not started | DrainEvents bound is count-based only; oversized reply kills the session and loses the drained events |
|
||||
| IPC-23 | Medium | S | WRK-21 | Not started | DrainEvents contract requirements (reply fits negotiated max, no event loss, drain-until-empty) + proto-comment/doc wave |
|
||||
| IPC-30 | Low | M | WRK-21 (same batch) | Not started | Oversized event frame stays session-fatal by design, but the death becomes structured (fault frame + logged identity) |
|
||||
| WRK-21 | Medium | M | owns IPC-23 fix; WRK-28 same batch | Done | DrainEvents bound is count-based only; oversized reply kills the session and loses the drained events |
|
||||
| IPC-23 | Medium | S | WRK-21 | In progress — mechanics landed with WRK-21; proto-comment/doc wave pending | DrainEvents contract requirements (reply fits negotiated max, no event loss, drain-until-empty) + proto-comment/doc wave |
|
||||
| IPC-30 | Low | M | WRK-21 (same batch) | Done | Oversized event frame stays session-fatal by design, but the death becomes structured (fault frame + logged identity) |
|
||||
| SEC-31 | Medium | M | — | Not started | Failure limiter partitions on attacker-controlled key id and blocks before verification (lockout DoS) |
|
||||
| SEC-32 | Low | S | SEC-31 | Not started | Failure-limiter LRU flushable by junk-token spray; token prefix never validated |
|
||||
| IPC-24 | Medium | S | — | Not started | CI's unconditional Java churn-revert masks real drift |
|
||||
@@ -71,27 +71,27 @@ Full design + implementation for each row lives in the linked domain doc under i
|
||||
|
||||
| ID | Sev | Tier | Eff | Dep | Status | Title |
|
||||
|---|---|:-:|:-:|---|---|---|
|
||||
| WRK-21 | Medium | P0 | M | IPC-23 (fix owned here); WRK-28 | Not started | DrainEvents bound count-based only; oversized reply kills session and loses drained events |
|
||||
| WRK-21 | Medium | P0 | M | IPC-23 (fix owned here); WRK-28 | Done | DrainEvents bound count-based only; oversized reply kills session and loses drained events |
|
||||
| WRK-22 | Low | — | S | IPC-26 (fix owned here) | Not started | Cancelled `WriteAsync` leaves its frame queued; it is still written later |
|
||||
| WRK-23 | Low | — | S | WRK-21 | Not started | Rejected frames consume sequence numbers, producing wire gaps |
|
||||
| WRK-23 | Low | — | S | WRK-21 | Done | Rejected frames consume sequence numbers, producing wire gaps |
|
||||
| WRK-24 | Low | — | S | — | Not started | `AdoptNegotiatedMaxMessageBytes` has no lower-bound sanity check |
|
||||
| WRK-25 | Low | P2 | S | WRK-22 (shared seam) | Not started | WRK-12 flush coalescing never engages on the event hot path |
|
||||
| WRK-26 | Low | P1 | S | WRK-23 (soft); discharges IPC-29 | Not started | Write-priority and overflow doc drift from the WRK-07 change |
|
||||
| WRK-27 | Low | — | S | — | Not started | Alarm poll bypasses the watchdog's in-flight suppression (15 s vs 75 s) |
|
||||
| WRK-28 | Low | — | S | WRK-21 (same batch) | Not started | 10,000 drain cap is a duplicated magic constant |
|
||||
| WRK-28 | Low | — | S | WRK-21 (same batch) | Done | 10,000 drain cap is a duplicated magic constant |
|
||||
|
||||
### Contracts & IPC — [30-contracts-ipc.md](30-contracts-ipc.md)
|
||||
|
||||
| ID | Sev | Tier | Eff | Dep | Status | Title |
|
||||
|---|---|:-:|:-:|---|---|---|
|
||||
| IPC-23 | Medium | P0 | S | WRK-21 (mechanics) | Not started | DrainEvents byte-blindness — contract requirements + proto-comment/doc wave |
|
||||
| IPC-23 | Medium | P0 | S | WRK-21 (mechanics) | In progress — mechanics landed with WRK-21; proto-comment/doc wave pending | DrainEvents byte-blindness — contract requirements + proto-comment/doc wave |
|
||||
| IPC-24 | Medium | P0 | S | — | Not started | CI's unconditional Java churn-revert masks real drift |
|
||||
| IPC-25 | Medium | P0 | M | — | Not started | Stale Go/Python worker bindings: regenerate (pinned toolchains) + check-codegen Check 4 |
|
||||
| IPC-26 | Low | P2 | S | WRK-22 (mechanics) | Not started | Cancelled write leaves ghost frame — cancelled means never written |
|
||||
| IPC-27 | Low | P2 | S | — | Not started | Descriptor-freshness test blind to enums/services/galaxy descriptor |
|
||||
| IPC-28 | Low | — | S | — | Not started | docs/Grpc.md missing CommandTooLarge → ResourceExhausted mapping |
|
||||
| IPC-29 | Low | — | S | WRK-26 (discharged by) | Not started | WorkerFrameProtocol.md missing write-scheduling/sequencing section |
|
||||
| IPC-30 | Low | P0 | M | WRK-21 (same batch) | Not started | Oversized event frame: keep session-fatal, make the death structured |
|
||||
| IPC-30 | Low | P0 | M | WRK-21 (same batch) | Done | Oversized event frame: keep session-fatal, make the death structured |
|
||||
| IPC-31 | Info | — | — | — | N/A | Gateway creation-time sequence stamping accepted; diagnostic-only, decision recorded |
|
||||
| IPC-32 | Info | — | S | IPC-25 (folded in) | Not started | check-codegen banner relabel 1/4…4/4 |
|
||||
|
||||
@@ -161,3 +161,5 @@ Sequence these together rather than piecemeal — several are one change set spa
|
||||
| 2026-07-13 | Operator bring-up complete: dedicated CI ed25519 key installed in windev `administrators_authorized_keys` (authorized into `dohertj2`, which owns the working MXAccess/toolchain env — a fresh OS account would break the build; the key is independently revocable), Gitea secrets `WINDEV_SSH_KEY`/`WINDEV_SSH_KNOWN_HOSTS` + variable `WINDEV_SSH_USER=dohertj2` stored, runner→`10.100.0.48:22` egress verified on the `traefik` net, issue-write confirmed. **TST-25/TST-26 → `Done`:** credentialed `windows-x86` ran GREEN on `d769244` (Gitea run #37) — Linux runner SSHed windev, checked out the SHA in `C:\build\mxaccessgw-ci` under lock, ran the x86 Worker build + `Worker.Tests`, exit 0; `nightly-windev` correctly skipped on the push event. Branch merged to `main`. Follow-ups (old tracker): revisit **TST-05** (scheduled live smoke — now covered by `nightly-windev`) and **TST-24** (client wire tests) which this unlocks. |
|
||||
| 2026-07-13 | Ran the TST-25 acceptance checks (scripts/ci/README.md) — they caught **two real CI defects, both fixed** on `fix/tst-25-ci-key-log-leak`: (1) **CI SSH key leaked in cleartext** in the `windows-x86` step env echo (Gitea's line-oriented masker missed the multiline PEM) — rotated the CI key on windev (old pubkey revoked), stored the key **base64-encoded** so the masker redacts it to `***` (confirmed on run #38), taught `run-windev-ci.sh` to decode, dropped the redundant public known-hosts secret from the job env; (2) **bootstrap lock race** — `run-windev-ci.sh`'s pre-hand-off `git fetch`/`checkout` ran outside the worktree lock, so concurrent runs collided on `.git/index.lock`; the bootstrap now holds the lock (ps1 re-uses it via `MXGW_CI_LOCK_HELD`), retest confirmed clean serialization. Also **deflaked** `SessionManagerTests` fail-fast timing assertions (absolute `<100ms` wall-clock bound flaked under CI load; now anchored to the configured timeout / dropped for the zero-timeout case). Checks passed: unreachable-host fast-fail (exit 255/15s), deliberate-red propagation (Worker.Tests failure → exit 1), lock concurrency (2nd run waits), no-key-in-logs (masked). Merge target `df7e20d` verified GREEN via the local windev path (Worker build + 356 tests); merged to `main` `19cbf7b`. Check 6 (forced-failure nightly issue): issue endpoint+token proven live at bring-up (#124); in-CI forced-failure probe abandoned to shared-runner congestion (residual `if: failure()` gating is standard Actions). |
|
||||
| 2026-07-13 | New finding **TST-30** (`Low`/`P2`) added — surfaced during TST-25 acceptance: CI runs on a single shared `gitea-runner` (`maxParallel=1`, co-located `10.100.0.35`) interleaved with `dohertj2/lmxopcua`, and Gitea 1.26 exposes no run cancel/delete, so queue latency is unbounded under cross-repo contention and the runner is a single point of failure. Design: add a second/labelled runner + document the no-cancel reality and the `run-windev-ci.sh` bypass. Roll-ups updated (Testing Low 2→3, total 47→48; P2 9→10). |
|
||||
| 2026-08-07 | **WRK-21 + WRK-28 + WRK-23 + IPC-30 → `Done`** (branch `fix/wrk-21-drain-cluster`, commits `33ba612` + test-fixture follow-ups `7c2eaf0`/`a256560`). WRK-21: `MxAccessEventQueue` gains a byte-budgeted `Drain(maxEvents, maxTotalBytes)` returning the new `WorkerEventDrainResult`, sizing inside the queue lock so an event that will not fit is never dequeued; `CreateDrainEventsReply` budgets against the negotiated frame max less a 64 KiB wrapper reserve and reports truncation through the existing `DiagnosticMessage` (no proto change), satisfying IPC-23 R1–R3; both reply-write seams (`HandleControlCommandAsync`, `ProcessCommandAsync`) now catch `MessageTooLarge` and answer the correlation with an `InvalidRequest` reply instead of unwinding/faulting the session. WRK-28: the 10,000 ceiling moved to `GatewayContractInfo.MaxDrainEventsPerCommand`, referenced by the gateway validator and the worker clamp (C# const, no `.proto` change). WRK-23: `WorkerFrameWriter` peek-stamps then commits `Sequence` only immediately before the stream write, so rejections leave no wire gap. IPC-30: an oversized event frame stays session-fatal but writes a `PROTOCOL_VIOLATION` `WorkerFault` with `command_method = EventDrain` naming family/handles/sequence/sizes (never the value) before exiting. Docs same commit: `MxAccessWorkerInstanceDesign.md`, `WorkerFrameProtocol.md`, `gateway.md`. **IPC-23 → `In progress`** — mechanics landed here; the proto-comment/doc wave (and its regen fan-out) is still pending and must not be folded into this branch. **Evidence** — macOS: `dotnet build src/ZB.MOM.WW.MxGateway.NonWindows.slnx` 0 warnings/0 errors, `dotnet test …MxGateway.Tests --filter FullyQualifiedName~MxAccessGrpcRequestValidator` 4/4 passed. windev (`scripts/ci/windev-worker-ci.ps1 -Sha a2565604 -Mode test`, 2026-08-07 06:47): x86 Worker build 0 warnings/0 errors, `Worker.Tests` **367 passed / 0 failed / 11 skipped** (skips are the live-MXAccess/dev-rig opt-ins), script exit 0. **Harness note:** `PipePair` runs both pipe ends in one process with blocking `FlushFileBuffers` per frame, so it wedges on multi-MB frames or after ~85 large round trips; the pipe tests therefore negotiate a 128 KiB frame maximum and walk 1,000 events to empty, while the full 10,000-event drain-to-empty no-loss proof runs at the queue layer (`MxAccessEventQueueTests`). |
|
||||
| 2026-08-07 | Code-review follow-ups on the same branch (commit `6bc3f9b`). (1) **Important** — `ResolveDrainReplyByteBudget` was a step, not a floor: just above the 64 KiB reserve the budget collapsed to a few bytes (exactly 1024 at the validator floor `MaxMessageBytes = 1024 + 64 KiB`), so a byte-heavy `DrainEvents` truncated on every call and the drain-until-empty loop never terminated. Now `Math.Max(frameMax - reserve, frameMax / 2)` — monotonic, never below half the frame max. New test `WorkerPipeSessionTests.DrainEvents_AtValidatorFloorFrameMax_MakesProgressAndTerminates` drives a byte-heavy queue at the exact validator floor and asserts drain-to-empty with no head reported oversized. (2) **Hardening** — the reply-too-large fallback write is now itself size-guarded (`WriteReplyTooLargeFallbackAsync`, shared by the control and STA reply seams) so a pathologically tiny negotiated max below the gateway floor (the WRK-24 gap) cannot make even the backstop session-fatal; log-and-swallow, comment points at WRK-24. (3) **Comment** — corrected the `RepeatedFieldOverheadBytes` docs: `WorkerEvent.CalculateSize()` already includes the event's tag+length, so the 8 bytes is pure slack, not wrapper compensation. **Evidence** — macOS build 0/0, validator filter 4/4. windev (`windev-worker-ci.ps1 -Sha 6bc3f9b -Mode test`, 07:07): x86 Worker build 0/0, `Worker.Tests` **368 passed / 0 failed / 11 skipped**, script exit 0. (An earlier run of the same SHA flaked on the pre-existing `RunAsync_WhenStaActivityIsStale_WritesWatchdogFault` — a 5 s CTS timeout under first-run load, untouched by this change; it passed on the clean re-run and in both prior full runs.) |
|
||||
|
||||
@@ -16,14 +16,14 @@ members, no positional records). The worker builds and tests only on the Windows
|
||||
|
||||
| ID | Sev | Tier | Eff | Dep | Status | Title |
|
||||
|----|-----|------|-----|-----|--------|-------|
|
||||
| WRK-21 | Medium | P0 | M | IPC-23 (same defect, fix owned here); WRK-28 (same lines) | Not started | DrainEvents bound is count-based only; an oversized reply still kills the session and loses the drained events |
|
||||
| WRK-21 | Medium | P0 | M | IPC-23 (same defect, fix owned here); WRK-28 (same lines) | Done | DrainEvents bound is count-based only; an oversized reply still kills the session and loses the drained events |
|
||||
| WRK-22 | Low | — | S | IPC-26 (same defect, fix owned here) | Not started | Cancelled `WriteAsync` leaves its frame queued; it is still written later |
|
||||
| WRK-23 | Low | — | S | WRK-21 (rejection path becomes backstop-only) | Not started | Rejected frames consume sequence numbers, producing wire gaps |
|
||||
| WRK-23 | Low | — | S | WRK-21 (rejection path becomes backstop-only) | Done | Rejected frames consume sequence numbers, producing wire gaps |
|
||||
| WRK-24 | Low | — | S | — | Not started | `AdoptNegotiatedMaxMessageBytes` has no lower-bound sanity check |
|
||||
| WRK-25 | Low | P2 | S | WRK-22 (both touch enqueue/dequeue) | Not started | WRK-12 flush coalescing never engages on the event hot path |
|
||||
| WRK-26 | Low | P1 | S | WRK-23 (soft — sequence prose); discharges IPC-29 | Not started | Write-priority and overflow doc drift from the WRK-07 change |
|
||||
| WRK-27 | Low | — | S | — | Not started | Alarm poll bypasses the watchdog's in-flight suppression (15 s vs 75 s) |
|
||||
| WRK-28 | Low | — | S | WRK-21 (land in the same commit cluster) | Not started | 10,000 drain cap is a duplicated magic constant with a comment-only sync contract |
|
||||
| WRK-28 | Low | — | S | WRK-21 (land in the same commit cluster) | Done | 10,000 drain cap is a duplicated magic constant with a comment-only sync contract |
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -12,14 +12,14 @@ All `path:line` citations were re-verified against the working tree at `4f5371f`
|
||||
|
||||
| ID | Sev | Tier | Eff | Dep | Status | Title |
|
||||
|----|-----|------|-----|-----|--------|-------|
|
||||
| IPC-23 | Medium | P0 | S¹ | WRK-21 | Not started | DrainEvents bound is count-based only; byte-heavy queue still builds a session-killing reply frame (contract requirements here; fix mechanics in WRK-21) |
|
||||
| IPC-23 | Medium | P0 | S¹ | WRK-21 | In progress — mechanics landed with WRK-21; proto-comment/doc wave pending | DrainEvents bound is count-based only; byte-heavy queue still builds a session-killing reply frame (contract requirements here; fix mechanics in WRK-21) |
|
||||
| IPC-24 | Medium | P0 | S | — | Not started | CI's unconditional Java churn-revert masks real generated-code drift for message-level proto changes |
|
||||
| IPC-25 | Medium | P0 | M | — | Not started | Committed Go/Python worker bindings are stale at HEAD; no guard covers them |
|
||||
| IPC-26 | Low | P2 | S¹ | WRK-22 | Not started | Cancelled write leaves a ghost frame that is still written (contract requirement here; fix mechanics in WRK-22) |
|
||||
| IPC-27 | Low | P2 | S | — | Not started | Descriptor freshness test blind to enums, enum values, services/methods, and the Galaxy contract |
|
||||
| IPC-28 | Low | — | S | — | Not started | `docs/Grpc.md` omits the `CommandTooLarge` → `ResourceExhausted` mapping |
|
||||
| IPC-29 | Low | — | S | — | Not started | Worker writer priority scheduling and write-time sequence stamping undocumented in the frame-protocol doc |
|
||||
| IPC-30 | Low | P0 | M | WRK-21 (same file/batch) | Not started | Oversized worker→gateway event frame is session-fatal — make the death deliberate, structured, and diagnosable |
|
||||
| IPC-30 | Low | P0 | M | WRK-21 (same file/batch) | Done | Oversized worker→gateway event frame is session-fatal — make the death deliberate, structured, and diagnosable |
|
||||
| IPC-31 | Info | — | — | — | N/A | Gateway stamps sequence at creation, worker at write — accepted divergence; sequence is documented diagnostic-only (`gateway.md:328-330`); revisit only if sequence ever becomes load-bearing |
|
||||
| IPC-32 | Info | — | S | IPC-25 | Not started | `check-codegen.ps1` check labels miscounted (folded into the IPC-25 script edit) |
|
||||
|
||||
|
||||
@@ -145,7 +145,7 @@ Post-run, verify no new `C:\*` file exists under any `bin/` (manual `find src -n
|
||||
|
||||
**Design.**
|
||||
|
||||
- **Expiry (window 2): eliminate, don't document.** The shared `ApiKeyIdentity` (0.1.4) carries `ExpiresUtc`; when caching a success, cap the entry lifetime at the key's expiry: `AbsoluteExpiration = min(now + ttl, ExpiresUtc)` (skip caching entirely if already ≤ now). A cached hit can then never outlive the key. Confirm during implementation that the library verifier populates `ExpiresUtc` on the returned identity; if it does not, fall back to documenting the ≤ TTL window in the remarks and `docs/Authentication.md` and file a donor-library ask.
|
||||
- **Expiry (window 2): eliminate, don't document.** The shared `ApiKeyIdentity` (0.1.5 — bumped from 0.1.4 in `e107019`, no API change) carries `ExpiresUtc`; when caching a success, cap the entry lifetime at the key's expiry: `AbsoluteExpiration = min(now + ttl, ExpiresUtc)` (skip caching entirely if already ≤ now). A cached hit can then never outlive the key. Confirm during implementation that the library verifier populates `ExpiresUtc` on the returned identity; if it does not, fall back to documenting the ≤ TTL window in the remarks and `docs/Authentication.md` and file a donor-library ask.
|
||||
- **Invalidate race (window 3): per-key generation check.** `ConcurrentDictionary<string, long> _generations`; `Invalidate(keyId)` increments the generation **before** evicting cache keys. `VerifyAsync` parses the key id from the token up front (same split the interceptor does — cheap, no store access), snapshots `g0` before calling the inner verifier, and after a success only `Set`s when the generation still equals `g0` — then re-reads the generation after the `Set` and self-evicts if it moved (bump-before-evict + set-then-recheck closes the remaining interleaving). Unparseable tokens skip caching already (`TryComputeCacheKey`).
|
||||
- **CLI window (1): accept and keep documented** — cross-process invalidation is out of scope by design; the TTL is the backstop and the remarks already say so.
|
||||
|
||||
|
||||
@@ -236,7 +236,7 @@ Full design + implementation for each row lives in the linked domain doc under i
|
||||
| TST-21 | Low | — | S | — | Not started | Log rotation configured but minimal |
|
||||
| TST-22 | Low | — | S | — | Not started | Config-shape JSON block omits documented keys |
|
||||
| TST-23 | Low | P2 | S | — | Done | Bidirectional `Session` RPC never built |
|
||||
| TST-24 | Low | P2 | M | TST-03 | Not started | Client wire behaviour has no automated verification |
|
||||
| TST-24 | Low | P2 | M | TST-03 | Not started | Client wire behaviour has no automated verification. **Gate cleared:** TST-03 CI is Done (live and green 2026-07-10; Windows/x86 tier green 2026-07-13 via the TST-25/TST-26 SSH-driven windev job), so TST-24 is unblocked — deferred by choice now, not CI-gated |
|
||||
|
||||
## Cross-cutting clusters
|
||||
|
||||
@@ -276,7 +276,7 @@ Findings the review flagged as one coordinated design pass — sequence them tog
|
||||
| 2026-07-09 | P1 Wave 3 (size/backpressure topology + write ordering). IPC-02/03/04 + WRK-04/07 → `Done`. **Size negotiation (IPC-02):** added `GatewayHello.max_frame_bytes` (regen `Generated/` + `clients/proto` descriptor refresh with pinned protoc 34.1); the gateway sends its negotiated worker-frame max and the worker adopts it (`WorkerFrameProtocolOptions.AdoptNegotiatedMaxMessageBytes`, 0 = keep default, >256 MiB rejected) instead of a hard-coded default. **Headroom (IPC-03):** the pipe frame max now sits `EnvelopeOverheadReserveBytes` (64 KiB) above the public gRPC cap (default `Worker.MaxMessageBytes` 16 MiB→16 MiB+64 KiB), cross-validated at startup; `WorkerClient` pre-checks command envelope size and fails only the offending correlation (`ResourceExhausted`) instead of `SetFaulted`ing the session. **Drain bound (IPC-04):** gateway request validator rejects `DrainEvents max_events` above 10 000; the worker caps each reply at `MaxDrainEventsPerReply` (10 000) and treats `max_events = 0` as that cap, not "drain all". **Sequence (WRK-04):** `WorkerFrameWriter` stamps the envelope `Sequence` at the point of writing under the write lock, so wire order and stamped sequence always agree under concurrent producers. **Priority (WRK-07):** the worker writer is now a cooperative priority scheduler — control frames (reply/fault/heartbeat/shutdown-ack) drain ahead of event frames; per-frame validation/size rejections fail only that frame, a stream failure fails all queued. Docs same-change (GatewayConfiguration, WorkerFrameProtocol, gateway.md). **Verified:** macOS NonWindows build clean + validator/grpc tests green; **windev** x86 worker builds clean, `Worker.Tests` 352 passed / 0 failed / 11 skipped (incl. new monotonic-sequence, control-before-event priority, negotiated-max, drain-bound tests), gateway `Tests` 799 passed / 3 failed — all 3 pre-existing windev-environmental (SelfSigned SAN + 2 `EventStreamServiceTests` timing, both pass in isolation). Commits `c8b3a22` (gateway half), `ebe6aea` (worker half), `309296f` (descriptor + default-expectation refresh). GWC-04 (event-channel decoupling) is the remaining Wave 3 item. |
|
||||
| 2026-07-09 | P1 S-misc (dashboard/observability hardening). SEC-02/12/20 → `Done`. SEC-02: `DashboardAuthorizationHandler` restricts the loopback + `Authentication:Mode=Disabled` bypasses to read-only (they satisfy a Viewer-bearing requirement but never `AdminOnly`), closing the policy-layer gap where anonymous localhost was authorized for Admin surfaces. SEC-12: `DashboardSessionAdminService` now emits canonical `AuditEvent`s (`dashboard-close-session`/`dashboard-kill-worker`, category `SessionAdmin`) through `IAuditWriter` on Success/Failure/Denied, so Close/Kill land durable audit rows. SEC-20: dropped the unbounded `session_id` tag from the exported `mxgateway.heartbeats.failed` counter. Docs updated same-change (CLAUDE.md, GatewayDashboardDesign.md, Metrics.md). Server build clean (0 warnings); targeted classes 30/30 pass; broader Dashboard+Security+GatewayApplication+Metrics sweep 295/295 pass. |
|
||||
| 2026-07-09 | P1 Wave 2b (security authz+hub). SEC-05/07/08/11 → `Done` (hub-token lifetime; QueryActiveAlarms scope arm; gateway-side verification cache + last-used coalescing; login rate limit + per-peer gRPC failure limiter). Full-suite checkpoint caught + fixed regressions the earlier narrow SEC-01/04/06 filter missed: cross-platform path-rooting, an `IHostEnvironment` fallback for minimal DI containers, a test-assembly `ASPNETCORE_ENVIRONMENT=Development` default, and a platform-correct default-path assertion. Suite: 747 passed / 42 failed, all 42 pre-existing macOS named-pipe-harness env failures. |
|
||||
| 2026-07-09 | P1 Wave 2a (security). SEC-01/04/06 → `Done` (config path-rooting + production validator guards; Server build clean, validator+hygiene tests 53/53). SEC-10 → `Done`: the shared `ZB.MOM.WW.Auth.ApiKeys` gained optional `ExpiresUtc` (expired keys rejected, auth DB auto-migrates to schema v3) via a concurrent HistorianGateway-remediation session's "G-2"; this repo consumes it by bumping the four `Auth.*` refs 0.1.2→0.1.4 (commit 197731a). Remaining SEC-10 polish (`apikey create --expires` + dashboard staleness badge) tracked as a small follow-up. |
|
||||
| 2026-07-09 | P1 Wave 2a (security). SEC-01/04/06 → `Done` (config path-rooting + production validator guards; Server build clean, validator+hygiene tests 53/53). SEC-10 → `Done`: the shared `ZB.MOM.WW.Auth.ApiKeys` gained optional `ExpiresUtc` (expired keys rejected, auth DB auto-migrates to schema v3) via a concurrent HistorianGateway-remediation session's "G-2"; this repo consumes it by bumping the four `Auth.*` refs 0.1.2→0.1.4 (commit 197731a; since bumped to 0.1.5 in `e107019` — 0.1.4 plus a transitive SQLitePCLRaw security pin, no API change, expiry enforcement retained). Remaining SEC-10 polish (`apikey create --expires` + dashboard staleness badge) tracked as a small follow-up. |
|
||||
| 2026-07-09 | P1 Wave 1 (CI + codegen freshness + Rust buildability) via parallel agents. IPC-01/09/19/20, CLI-02 → `Done`; TST-03 → `In review` (CI pipeline authored + YAML/layout-validated, but not yet executed on a Gitea runner — proven on first push). Verified on macOS: NonWindows build clean, `ClientProtoInputTests` 5/5, `publish-client-proto-inputs.ps1 -Check` exit 0, `cargo package` (no `--no-verify`) compiles standalone. Added a vendored-Rust-proto drift guard (Check 3) to `check-codegen.ps1` closing the CLI-02 static-copy risk. IPC-09 script guards not executed end-to-end (need pinned python/JRE toolchains); logic is PATH-resolution + version assertion. |
|
||||
| 2026-07-09 | WRK-01 → `Done`. Verified on Windows host (windev) via an isolated `origin/main` worktree: worker builds x86 clean, `StaRuntimeTests`+`WorkerPipeSessionTests` 33/33 pass. Fixed an `xUnit1030` build error (the new worker test used `.ConfigureAwait(false)` in `[Fact]` bodies) that the macOS tree could not surface. Also ran GWC-01's Windows-only `WorkerClientTests` on windev: 18/18 pass (incl. `ReadEventsAsync_SecondEnumerator_Throws`). All 8 P0 findings now `Done`. Not yet committed. |
|
||||
| 2026-07-10 | **TST-03 → `Done`: CI is live and green** on branch `fix/ci-selfhosted-tooling`. The pipeline was authored (P1) but had never executed. Root cause it never ran: the co-located `gitea-runner` on the docker host (`10.100.0.35`) spawned job containers on an isolated network (`container.network: ""`) that could not resolve Gitea's internal clone URL `http://gitea:3000`; one-line host fix `container.network: "traefik"` + `docker restart gitea-runner`. The self-hosted `catthehacker` act image also lacks tooling GitHub-hosted runners preinstall — ci.yml now installs pwsh (dotnet global tool, for `check-codegen.ps1`) and Gradle 9.5.1 directly (act can't resolve the `gradle/actions` monorepo action; no gradle wrapper in repo). Driving to green surfaced and fixed **five real latent defects** (TST-03 doing its job): `check-codegen.ps1` `.Trim()`-on-`$null` on a clean tree; **stale vendored rust proto** (`clients/rust/protos/mxaccess_worker.proto` missing canonical `max_frame_bytes`); `OrphanWorkerTerminatorTests` hard-coded `C:\` path failing Linux `Path.GetFullPath` (0 kills); **stale java generated** `MxaccessWorker.java` (missing `max_frame_bytes`, regenerated); a sync python test building a `grpc.aio.Channel` with no current event loop on py3.12 (autouse conftest fixture). Result: `portable` **success** (NonWindows build + codegen freshness + 808/808 gateway tests + .NET/Go/Rust/Python clients) and `java` **success**. `windows` + `live-mxaccess` jobs remain `queued` pending a self-hosted **windev** runner (`10.100.0.48`) with those labels — separate follow-up, does not gate portable/java. |
|
||||
|
||||
+2
-2
@@ -123,9 +123,9 @@ The split uses `count: 3` because the secret portion may itself contain undersco
|
||||
|
||||
### Command value redaction
|
||||
|
||||
> **Not yet implemented.** Command-value logging is *not* wired end-to-end. There is no `MxGateway:Diagnostics:LogCommandValues` (or equivalent) configuration knob, and `RedactCommandValue` / `IsCredentialBearingCommand` have no call sites in the gateway — no command values are logged anywhere today, which is the safest posture. The helpers below exist as the intended redaction seam for a future opt-in value-logging feature; that wiring is deferred until secured-bulk command variants are covered by the redactor's credential list (the `WriteSecuredBulk` / `WriteSecured2Bulk` gap), so enabling value logging cannot leak a secured-bulk payload. Until then, treat this section as describing the planned shape, not current behavior.
|
||||
> **Redaction seam wired; value logging still has no opt-in knob** (updated 2026-08-07). Since commit `47c0b64` (2026-07-27), `RedactCommandValue` **is** wired: `GatewayLogRedactorSeam` (`Diagnostics/GatewayLogRedactorSeam.cs`) calls it for any non-null `CommandValue` log property, gated on `CommandMethod`, so a command payload that reaches a log event is masked on every sink through the shared `ILogRedactor` seam. What remains true from the original note: there is still no `MxGateway:Diagnostics:LogCommandValues` (or equivalent) configuration knob — the seam exposes no opt-in, `valueLoggingEnabled` is never passed `true`, so **every** non-null command value is redacted, credential-bearing or not. In practice no log statement currently emits a `CommandValue` property, but one that does can no longer leak a payload in the clear.
|
||||
|
||||
The intended `RedactCommandValue` would enforce the "values are opt-in and redacted by default" rule:
|
||||
`RedactCommandValue` enforces the "values are opt-in and redacted by default" rule:
|
||||
|
||||
```csharp
|
||||
public static object? RedactCommandValue(
|
||||
|
||||
+31
-13
@@ -1,5 +1,18 @@
|
||||
# Galaxy Repository Browse
|
||||
|
||||
> **Adopted as a shared package (2026-06-25, commit `8e196a7`).** The Galaxy browse
|
||||
> implementation described in this document no longer lives in this repository. The
|
||||
> gateway consumes the shared **`ZB.MOM.WW.GalaxyRepository`** library (pinned at
|
||||
> `0.2.0` from the Gitea NuGet feed; sources hosted in the `scadaproj` repo), wired
|
||||
> in `GatewayApplication.cs` via `AddZbGalaxyRepository` (registration) and
|
||||
> `MapZbGalaxyRepository` (endpoint mapping). The former in-repo `Server/Galaxy/`
|
||||
> classes and `Grpc/GalaxyRepositoryGrpcService.cs` / `Grpc/GalaxyProtoMapper.cs`
|
||||
> were deleted with the adoption. The wire contract (`galaxy_repository.v1`) is
|
||||
> unchanged, so the RPC behavior, filters, paging, caching, and snapshot semantics
|
||||
> below still apply as documented — but any `src/ZB.MOM.WW.MxGateway.Server/...`
|
||||
> file path cited below is historical; the equivalent class now lives in the
|
||||
> shared library.
|
||||
|
||||
The gateway exposes a read-only browse surface over the AVEVA System Platform
|
||||
Galaxy Repository (the SQL Server database named `ZB`). Clients use it to
|
||||
enumerate the deployed object hierarchy and each object's attributes
|
||||
@@ -107,7 +120,8 @@ server and dashboard views are consistent.
|
||||
## Hierarchy Cache
|
||||
|
||||
The gateway holds a single shared `IGalaxyHierarchyCache`
|
||||
(`src/ZB.MOM.WW.MxGateway.Server/Galaxy/GalaxyHierarchyCache.cs`) — every
|
||||
(`GalaxyHierarchyCache`, library-side in `ZB.MOM.WW.GalaxyRepository` — formerly
|
||||
`src/ZB.MOM.WW.MxGateway.Server/Galaxy/GalaxyHierarchyCache.cs`) — every
|
||||
`DiscoverHierarchy` and `GetLastDeployTime` request reads from this cache
|
||||
rather than hitting SQL. Many clients can browse concurrently with at most
|
||||
one SQL query in flight.
|
||||
@@ -174,7 +188,8 @@ record: deleting it only forces the next cold start to wait for live SQL.
|
||||
## Deploy Notifications
|
||||
|
||||
`WatchDeployEvents` is a server-streaming RPC backed by
|
||||
`IGalaxyDeployNotifier` (`src/ZB.MOM.WW.MxGateway.Server/Galaxy/GalaxyDeployNotifier.cs`).
|
||||
`IGalaxyDeployNotifier` (`GalaxyDeployNotifier`, library-side in
|
||||
`ZB.MOM.WW.GalaxyRepository`).
|
||||
The notifier maintains a private bounded channel per subscriber so a slow
|
||||
client cannot back-pressure other subscribers or the publisher.
|
||||
|
||||
@@ -322,7 +337,7 @@ fields cannot express null. Use it to distinguish "no dimension reported" from
|
||||
|
||||
```text
|
||||
gRPC client(s)
|
||||
-> GalaxyRepositoryGrpcService (src/ZB.MOM.WW.MxGateway.Server/Grpc/)
|
||||
-> Galaxy repository gRPC service (library-side; mapped via MapZbGalaxyRepository)
|
||||
DiscoverHierarchy, GetLastDeployTime, BrowseChildren -> IGalaxyHierarchyCache.Current
|
||||
WatchDeployEvents -> IGalaxyDeployNotifier
|
||||
TestConnection -> GalaxyRepository (direct SQL)
|
||||
@@ -341,41 +356,44 @@ GalaxyHierarchyRefreshService (BackgroundService)
|
||||
-> IGalaxyDeployNotifier.Publish (only on deploy change)
|
||||
```
|
||||
|
||||
Component breakdown:
|
||||
Component breakdown (all of these classes are **library-side**, in the shared
|
||||
`ZB.MOM.WW.GalaxyRepository` package — the in-repo files formerly at the paths
|
||||
below were deleted in commit `8e196a7`):
|
||||
|
||||
- `GalaxyRepository` (`src/ZB.MOM.WW.MxGateway.Server/Galaxy/GalaxyRepository.cs`) holds
|
||||
- `GalaxyRepository` (formerly `src/ZB.MOM.WW.MxGateway.Server/Galaxy/GalaxyRepository.cs`) holds
|
||||
the SQL. Both `HierarchySql` and `AttributesSql` walk template-derivation and
|
||||
package-derivation chains via recursive CTEs and pick the most-derived
|
||||
override per object. `HierarchySql` still matches the OtOpcUa original;
|
||||
`AttributesSql` does not — it additionally enumerates built-in primitive
|
||||
attributes (see [Built-in vs configured attributes](#built-in-vs-configured-attributes)).
|
||||
- `GalaxyHierarchyCache`
|
||||
(`src/ZB.MOM.WW.MxGateway.Server/Galaxy/GalaxyHierarchyCache.cs`) holds the most
|
||||
(formerly `src/ZB.MOM.WW.MxGateway.Server/Galaxy/GalaxyHierarchyCache.cs`) holds the most
|
||||
recent immutable `GalaxyHierarchyCacheEntry` (materialized objects +
|
||||
precomputed dashboard summary + counts + status). All gRPC clients share the
|
||||
same entry.
|
||||
- `GalaxyHierarchyRefreshService`
|
||||
(`src/ZB.MOM.WW.MxGateway.Server/Galaxy/GalaxyHierarchyRefreshService.cs`) is a
|
||||
(formerly `src/ZB.MOM.WW.MxGateway.Server/Galaxy/GalaxyHierarchyRefreshService.cs`) is a
|
||||
hosted `BackgroundService` that drives `RefreshAsync` on the configured
|
||||
interval, with deploy-time gating to avoid unnecessary heavy queries.
|
||||
- `GalaxyDeployNotifier`
|
||||
(`src/ZB.MOM.WW.MxGateway.Server/Galaxy/GalaxyDeployNotifier.cs`) is a thin
|
||||
(formerly `src/ZB.MOM.WW.MxGateway.Server/Galaxy/GalaxyDeployNotifier.cs`) is a thin
|
||||
per-subscriber-channel fan-out for streaming clients.
|
||||
- `GalaxyProtoMapper`
|
||||
(`src/ZB.MOM.WW.MxGateway.Server/Grpc/GalaxyProtoMapper.cs`) converts row models to
|
||||
(formerly `src/ZB.MOM.WW.MxGateway.Server/Grpc/GalaxyProtoMapper.cs`) converts row models to
|
||||
proto messages. Used by the cache during refresh to materialize the reply
|
||||
once.
|
||||
- `GalaxyBrowseProjector`
|
||||
(`src/ZB.MOM.WW.MxGateway.Server/Galaxy/GalaxyBrowseProjector.cs`) projects one level
|
||||
(formerly `src/ZB.MOM.WW.MxGateway.Server/Galaxy/GalaxyBrowseProjector.cs`) projects one level
|
||||
of children out of an immutable cache entry. Memoizes the filtered child list
|
||||
per cache-entry instance so repeated paging is an O(pageSize) slice rather than an
|
||||
O(siblings) filter scan. The memo is keyed on the cache entry reference, so a new
|
||||
entry from the background refresh makes the stale memo unreachable and it is
|
||||
collected with it. `DashboardBrowseService` wraps this projector to drive the
|
||||
dashboard's lazy-expand tree.
|
||||
- `GalaxyRepositoryGrpcService`
|
||||
(`src/ZB.MOM.WW.MxGateway.Server/Grpc/GalaxyRepositoryGrpcService.cs`) implements
|
||||
the five RPCs.
|
||||
- The Galaxy repository gRPC service (formerly
|
||||
`src/ZB.MOM.WW.MxGateway.Server/Grpc/GalaxyRepositoryGrpcService.cs`, deleted
|
||||
with the adoption) implements the five RPCs; it is now supplied by the library
|
||||
and mapped via `MapZbGalaxyRepository`.
|
||||
|
||||
## Configuration
|
||||
|
||||
|
||||
@@ -378,6 +378,20 @@ If event conversion throws, catch it inside the event handler, record a
|
||||
structured `WorkerFault`, and keep the worker alive only if the fault policy
|
||||
allows it.
|
||||
|
||||
The event drain loop streams queued events as `WorkerEvent` frames. A single
|
||||
event whose envelope exceeds the negotiated frame maximum is **undeliverable end
|
||||
to end** — the pipe maximum sits only the envelope-overhead reserve above the
|
||||
public gRPC cap, so a frame the pipe rejects would also be rejected on the
|
||||
client-facing stream. The session therefore faults on it rather than dropping it
|
||||
(a silent drop makes the event stream unfaithful, and a synthesized placeholder
|
||||
is barred by the no-synthesized-events rule), but the death is structured: the
|
||||
worker logs the event's identity — family, handles, worker sequence, and sizes,
|
||||
never the value — writes a `WorkerFault` with category `ProtocolViolation` and
|
||||
command method `EventDrain` carrying the same identity, and only then exits.
|
||||
Operator remediation is configuration: raise `MxGateway:Worker:MaxMessageBytes`
|
||||
for that workload. Other per-frame rejection codes keep their previous behavior
|
||||
because they indicate worker bugs, not workload size.
|
||||
|
||||
## Command Queue
|
||||
|
||||
The pipe reader converts `WorkerCommand` messages into `StaCommand` entries.
|
||||
@@ -440,6 +454,29 @@ Diagnostics:
|
||||
- `DrainEvents`
|
||||
- `ShutdownWorker`
|
||||
|
||||
`DrainEvents` is answered on the message-loop thread, not the STA, and its reply
|
||||
is bounded on **two** axes because no diagnostics command may be session-fatal:
|
||||
|
||||
- **Count** — `GatewayContractInfo.MaxDrainEventsPerCommand` (10,000) is the
|
||||
single home of the ceiling, shared by the gateway's request validator (which
|
||||
rejects a larger `max_events` at the public boundary) and this worker clamp
|
||||
(which also interprets `max_events = 0`, "as many as available").
|
||||
- **Bytes** — the count cap alone is not sufficient: byte-heavy events (large
|
||||
string or array `MxValue`s) overshoot the negotiated frame maximum long before
|
||||
10,000 events. The drain is therefore byte-budgeted against the negotiated
|
||||
maximum less a 64 KiB envelope/reply-wrapper reserve, and the size decision
|
||||
happens inside the event queue's lock, so an event is dequeued only once it is
|
||||
known to fit. An event that does not fit stays at the head of the queue and is
|
||||
never lost.
|
||||
|
||||
Truncation is reported in the reply's existing `DiagnosticMessage`
|
||||
("N events returned, M remain; repeat DrainEvents for the rest") rather than in a
|
||||
new field, so the contract is unchanged and callers drain iteratively until a
|
||||
reply comes back empty. In the degenerate case where the head event alone exceeds
|
||||
the budget, the reply returns whatever fit before it (possibly nothing) and names
|
||||
the blocked event's worker sequence so an operator can find the offending tag;
|
||||
that event needs a larger `MxGateway:Worker:MaxMessageBytes` to move at all.
|
||||
|
||||
Implement method-specific dispatch instead of a generic string method invoker.
|
||||
Parity tests need stable command-specific request and reply shapes.
|
||||
|
||||
@@ -623,6 +660,14 @@ queue fills:
|
||||
Production coalescing may be added later, but it must be explicit and tested.
|
||||
Do not drop or coalesce events in v1.
|
||||
|
||||
No control reply is session-fatal on size. Reply builders size their payloads
|
||||
against the negotiated frame maximum, and the two reply-write seams (the
|
||||
control-command path and the STA command path) additionally catch a
|
||||
`MessageTooLarge` per-frame rejection and answer that correlation with a small
|
||||
`InvalidRequest` reply instead of unwinding the session. Oversized *event*
|
||||
frames keep the opposite policy — see Event Sink — because an event above the
|
||||
frame maximum cannot be delivered to the client at all.
|
||||
|
||||
## Heartbeat And Watchdog
|
||||
|
||||
`WorkerPipeSession` starts the heartbeat loop after the gateway validates
|
||||
|
||||
@@ -29,6 +29,29 @@ default. A `max_frame_bytes` of 0 (an older gateway that never set the field)
|
||||
means "use the worker's built-in default". This keeps both ends framing to the
|
||||
same limit rather than depending on matched compile-time constants.
|
||||
|
||||
Every worker-to-gateway frame must serialize within this limit, control replies
|
||||
included, so reply builders truncate to fit rather than emit a frame the writer
|
||||
will reject. `WorkerPipeSession` pre-sizes a `DrainEvents` reply below the
|
||||
negotiated maximum (less a fixed envelope/reply-wrapper reserve) and reports the
|
||||
truncation in the reply's `DiagnosticMessage`; the caller contract is to repeat
|
||||
`DrainEvents` until it returns an empty reply. Should a reply still overshoot,
|
||||
`MessageTooLarge` at the reply-write seam is answered with a small
|
||||
`InvalidRequest` reply for that correlation, not with session teardown — no
|
||||
diagnostics command may kill a session.
|
||||
|
||||
An oversized *event* frame is the deliberate exception. Such an event is
|
||||
undeliverable end to end (the pipe maximum sits only the envelope-overhead
|
||||
reserve above the public gRPC cap), so the session faults: the worker logs the
|
||||
event's identity and sizes — never its value — writes a `WorkerFault` with
|
||||
category `ProtocolViolation` and command method `EventDrain`, then exits.
|
||||
Remediation is raising `MxGateway:Worker:MaxMessageBytes` for that workload.
|
||||
|
||||
A per-frame rejection does not consume an envelope `sequence`. The writer stamps
|
||||
a candidate sequence, runs the empty-payload and size checks against the stamped
|
||||
envelope, and commits the counter only immediately before the stream write, so
|
||||
the sequences observed on the wire stay contiguous across rejections and an
|
||||
operator reading a pipe capture never sees a phantom gap.
|
||||
|
||||
## Envelope Validation
|
||||
|
||||
`WorkerFrameReader` and `WorkerFrameWriter` validate each envelope against the
|
||||
|
||||
+8
-4
@@ -447,10 +447,14 @@ Optional diagnostics:
|
||||
- `Ping`
|
||||
- `GetSessionState`
|
||||
- `GetWorkerInfo`
|
||||
- `DrainEvents` — diagnostic; `max_events` is bounded (the gateway rejects requests
|
||||
above a public ceiling, and the worker caps each reply at its own per-reply limit,
|
||||
treating `max_events = 0` as "the default cap") so one drain cannot pack an
|
||||
unbounded, session-killing reply frame.
|
||||
- `DrainEvents` — diagnostic; the reply is bounded on two axes so one drain cannot
|
||||
pack an unbounded, session-killing reply frame. By **count**: the gateway rejects
|
||||
requests above the shared ceiling `GatewayContractInfo.MaxDrainEventsPerCommand`
|
||||
and the worker clamps to the same value, treating `max_events = 0` as "the default
|
||||
cap". By **bytes**: the worker sizes the reply while draining, against the
|
||||
negotiated worker-frame maximum, so a byte-heavy queue truncates instead of
|
||||
overshooting and events that do not fit stay queued. Truncation is reported in the
|
||||
reply's `DiagnosticMessage`; callers drain iteratively until an empty reply.
|
||||
- `ShutdownWorker`
|
||||
|
||||
Do not compress MXAccess semantics into generic verbs too early. A command enum
|
||||
|
||||
@@ -17,6 +17,21 @@ public static class GatewayContractInfo
|
||||
/// <summary>Default backend name identifying the MXAccess worker process type.</summary>
|
||||
public const string DefaultBackendName = "mxaccess-worker";
|
||||
|
||||
/// <summary>
|
||||
/// Ceiling on how many events one <c>DrainEvents</c> command may move in a single reply.
|
||||
/// Shared so the gateway's request-validation ceiling
|
||||
/// (<c>MxAccessGrpcRequestValidator</c>, which rejects a larger <c>max_events</c> loudly at
|
||||
/// the public boundary) and the worker's per-reply clamp
|
||||
/// (<c>WorkerPipeSession.CreateDrainEventsReply</c>, the backstop that also interprets
|
||||
/// <c>max_events = 0</c>) cannot drift apart. A count cap alone is necessary but not
|
||||
/// sufficient: the worker additionally caps the reply by serialized bytes against the
|
||||
/// negotiated frame maximum (WRK-21), so a reply may carry fewer events than this ceiling
|
||||
/// and fewer than are queued. Callers drain iteratively until an empty reply.
|
||||
/// This is a documented behavioral bound, not wire schema — it is deliberately a C#
|
||||
/// constant and not a <c>.proto</c> field.
|
||||
/// </summary>
|
||||
public const uint MaxDrainEventsPerCommand = 10_000;
|
||||
|
||||
/// <summary>
|
||||
/// Environment variable name that opts an xUnit suite into running live
|
||||
/// MXAccess COM tests. Single source of truth shared by both
|
||||
|
||||
@@ -10,8 +10,15 @@ public sealed class GatewayLogRedactorSeam : ILogRedactor
|
||||
{
|
||||
private static readonly string[] IdentityKeys = ["ClientIdentity", "authorization", "Authorization"];
|
||||
|
||||
/// <summary>Property name carrying the MXAccess command method, which gates value redaction.</summary>
|
||||
private const string CommandMethodProperty = "CommandMethod";
|
||||
|
||||
/// <summary>Property name carrying a command payload value that may bear credentials.</summary>
|
||||
private const string CommandValueProperty = "CommandValue";
|
||||
|
||||
/// <summary>
|
||||
/// Masks API-key/credential material in known identity-bearing log properties.
|
||||
/// Masks API-key/credential material in known identity-bearing log properties, and any
|
||||
/// command payload value.
|
||||
/// </summary>
|
||||
/// <param name="properties">The log event property dictionary to redact in place.</param>
|
||||
public void Redact(IDictionary<string, object?> properties)
|
||||
@@ -24,5 +31,26 @@ public sealed class GatewayLogRedactorSeam : ILogRedactor
|
||||
properties[key] = GatewayLogRedactor.RedactClientIdentity(s);
|
||||
}
|
||||
}
|
||||
|
||||
RedactCommandValue(properties);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Masks a command payload value. Value logging is off here — the seam exposes no opt-in — so
|
||||
/// every non-null value is masked, credential-bearing command or not.
|
||||
/// </summary>
|
||||
/// <param name="properties">The log event property dictionary to redact in place.</param>
|
||||
private static void RedactCommandValue(IDictionary<string, object?> properties)
|
||||
{
|
||||
if (!properties.TryGetValue(CommandValueProperty, out object? value) || value is null)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
string? commandMethod = properties.TryGetValue(CommandMethodProperty, out object? method)
|
||||
? method as string
|
||||
: null;
|
||||
|
||||
properties[CommandValueProperty] = GatewayLogRedactor.RedactCommandValue(commandMethod, value);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,17 +1,11 @@
|
||||
using Grpc.Core;
|
||||
using ZB.MOM.WW.MxGateway.Contracts;
|
||||
using ZB.MOM.WW.MxGateway.Contracts.Proto;
|
||||
|
||||
namespace ZB.MOM.WW.MxGateway.Server.Grpc;
|
||||
|
||||
public sealed class MxAccessGrpcRequestValidator
|
||||
{
|
||||
// Upper bound on a single DrainEvents request. DrainEvents is a diagnostics RPC that returns
|
||||
// buffered events in one non-streaming reply, so an unbounded max_events could pack the whole
|
||||
// queue into a session-killing frame. The worker independently caps each reply at its
|
||||
// own MaxDrainEventsPerReply; this public bound rejects an obviously-abusive request loudly at
|
||||
// the boundary. max_events = 0 is allowed and means "the worker's default batch cap".
|
||||
private const uint MaxDrainEventsPerRequest = 10_000;
|
||||
|
||||
/// <summary>Validates an open session request.</summary>
|
||||
/// <param name="request">The request to validate.</param>
|
||||
public void ValidateOpenSession(OpenSessionRequest request)
|
||||
@@ -78,10 +72,18 @@ public sealed class MxAccessGrpcRequestValidator
|
||||
}
|
||||
|
||||
// The payload case now matches the kind, so command.DrainEvents is non-null here.
|
||||
if (command.Kind is MxCommandKind.DrainEvents && command.DrainEvents.MaxEvents > MaxDrainEventsPerRequest)
|
||||
// DrainEvents is a diagnostics RPC that returns buffered events in one non-streaming
|
||||
// reply, so an unbounded max_events could pack the whole queue into a session-killing
|
||||
// frame. The worker independently clamps every reply to the same shared ceiling and
|
||||
// additionally caps it by serialized bytes; this public bound rejects an obviously-abusive
|
||||
// request loudly at the boundary. max_events = 0 is allowed and means "the worker's
|
||||
// default batch cap".
|
||||
if (command.Kind is MxCommandKind.DrainEvents
|
||||
&& command.DrainEvents.MaxEvents > GatewayContractInfo.MaxDrainEventsPerCommand)
|
||||
{
|
||||
throw InvalidArgument(
|
||||
$"DrainEvents max_events ({command.DrainEvents.MaxEvents}) must not exceed {MaxDrainEventsPerRequest}; "
|
||||
$"DrainEvents max_events ({command.DrainEvents.MaxEvents}) must not exceed "
|
||||
+ $"{GatewayContractInfo.MaxDrainEventsPerCommand}; "
|
||||
+ "use 0 to request the worker default batch cap.");
|
||||
}
|
||||
}
|
||||
|
||||
@@ -13,13 +13,13 @@
|
||||
<PackageReference Include="ZB.MOM.WW.Audit" Version="0.1.0" />
|
||||
<PackageReference Include="ZB.MOM.WW.Theme" Version="0.3.1" />
|
||||
<PackageReference Include="ZB.MOM.WW.Configuration" Version="0.1.0" />
|
||||
<PackageReference Include="ZB.MOM.WW.Health" Version="0.1.0" />
|
||||
<PackageReference Include="ZB.MOM.WW.Health" Version="0.2.0" />
|
||||
<PackageReference Include="ZB.MOM.WW.Telemetry" Version="0.1.0" />
|
||||
<PackageReference Include="ZB.MOM.WW.Telemetry.Serilog" Version="0.1.0" />
|
||||
<PackageReference Include="ZB.MOM.WW.GalaxyRepository" Version="0.2.0" />
|
||||
<PackageReference Include="ZB.MOM.WW.Secrets" Version="0.2.0" />
|
||||
<PackageReference Include="ZB.MOM.WW.Secrets.Abstractions" Version="0.2.0" />
|
||||
<PackageReference Include="ZB.MOM.WW.Secrets.Ui" Version="0.2.0" />
|
||||
<PackageReference Include="ZB.MOM.WW.Secrets" Version="0.2.3" />
|
||||
<PackageReference Include="ZB.MOM.WW.Secrets.Abstractions" Version="0.2.3" />
|
||||
<PackageReference Include="ZB.MOM.WW.Secrets.Ui" Version="0.2.3" />
|
||||
<PackageReference Include="Serilog.AspNetCore" Version="10.0.0" />
|
||||
<PackageReference Include="Serilog.Sinks.Console" Version="6.1.1" />
|
||||
<PackageReference Include="Serilog.Sinks.File" Version="7.0.0" />
|
||||
|
||||
@@ -13,4 +13,75 @@ public sealed class GatewayLogRedactorSeamTests
|
||||
redactor.Redact(props);
|
||||
Assert.Equal("Bearer mxgw_operator01_[redacted]", props["ClientIdentity"]);
|
||||
}
|
||||
|
||||
/// <summary>A command value carried by a credential-bearing command never reaches a sink in the clear.</summary>
|
||||
[Fact]
|
||||
public void Redact_MasksCommandValueForCredentialBearingCommand()
|
||||
{
|
||||
GatewayLogRedactorSeam redactor = new();
|
||||
Dictionary<string, object?> props = new()
|
||||
{
|
||||
["CommandMethod"] = "AuthenticateUser",
|
||||
["CommandValue"] = "operator01:hunter2"
|
||||
};
|
||||
|
||||
redactor.Redact(props);
|
||||
|
||||
Assert.Equal(GatewayLogRedactor.RedactedValue, props["CommandValue"]);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Value logging is off by default, so an ordinary command value is masked too — the seam
|
||||
/// exposes no opt-in, matching <see cref="GatewayLogRedactor.RedactCommandValue"/>'s default.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void Redact_MasksCommandValueForOrdinaryCommand()
|
||||
{
|
||||
GatewayLogRedactorSeam redactor = new();
|
||||
Dictionary<string, object?> props = new()
|
||||
{
|
||||
["CommandMethod"] = "Read",
|
||||
["CommandValue"] = "plaintext-tag-value"
|
||||
};
|
||||
|
||||
redactor.Redact(props);
|
||||
|
||||
Assert.Equal(GatewayLogRedactor.RedactedValue, props["CommandValue"]);
|
||||
}
|
||||
|
||||
/// <summary>A command value with no accompanying method is still masked — an unknown method cannot be cleared.</summary>
|
||||
[Fact]
|
||||
public void Redact_MasksCommandValueWhenCommandMethodAbsent()
|
||||
{
|
||||
GatewayLogRedactorSeam redactor = new();
|
||||
Dictionary<string, object?> props = new() { ["CommandValue"] = "plaintext-tag-value" };
|
||||
|
||||
redactor.Redact(props);
|
||||
|
||||
Assert.Equal(GatewayLogRedactor.RedactedValue, props["CommandValue"]);
|
||||
}
|
||||
|
||||
/// <summary>A null command value stays null rather than becoming the redaction placeholder.</summary>
|
||||
[Fact]
|
||||
public void Redact_LeavesNullCommandValueNull()
|
||||
{
|
||||
GatewayLogRedactorSeam redactor = new();
|
||||
Dictionary<string, object?> props = new() { ["CommandValue"] = null };
|
||||
|
||||
redactor.Redact(props);
|
||||
|
||||
Assert.Null(props["CommandValue"]);
|
||||
}
|
||||
|
||||
/// <summary>An event carrying no command value is untouched — the seam must not invent the property.</summary>
|
||||
[Fact]
|
||||
public void Redact_DoesNotAddCommandValueWhenAbsent()
|
||||
{
|
||||
GatewayLogRedactorSeam redactor = new();
|
||||
Dictionary<string, object?> props = new() { ["ClientIdentity"] = "anonymous" };
|
||||
|
||||
redactor.Redact(props);
|
||||
|
||||
Assert.False(props.ContainsKey("CommandValue"));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -410,6 +410,48 @@ public sealed class WorkerFrameProtocolTests
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Verifies a per-frame rejection does not burn a sequence number (WRK-23). The sequence is a
|
||||
/// diagnostic counter, so a gap breaks nothing functionally — but an operator correlating a pipe
|
||||
/// capture reads a gap as a lost frame and chases a bug that does not exist, and the gap-free
|
||||
/// guarantee the concurrent-write test asserts would otherwise only hold until the first
|
||||
/// rejection.
|
||||
/// </summary>
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
[Fact]
|
||||
public async Task WriteAsync_PerFrameRejection_DoesNotConsumeSequence()
|
||||
{
|
||||
const int maxMessageBytes = 512;
|
||||
WorkerFrameProtocolOptions options = new(
|
||||
SessionId,
|
||||
GatewayContractInfo.WorkerProtocolVersion,
|
||||
Nonce,
|
||||
maxMessageBytes);
|
||||
using MemoryStream stream = new();
|
||||
WorkerFrameWriter writer = new(stream, options);
|
||||
|
||||
await writer.WriteAsync(CreateEventEnvelope());
|
||||
|
||||
WorkerEnvelope oversized = CreateGatewayHelloEnvelope();
|
||||
oversized.GatewayHello.GatewayVersion = new string('x', maxMessageBytes * 2);
|
||||
WorkerFrameProtocolException exception =
|
||||
await Assert.ThrowsAsync<WorkerFrameProtocolException>(
|
||||
async () => await writer.WriteAsync(oversized));
|
||||
Assert.Equal(WorkerFrameProtocolErrorCode.MessageTooLarge, exception.ErrorCode);
|
||||
|
||||
await writer.WriteAsync(CreateEventEnvelope());
|
||||
|
||||
stream.Position = 0;
|
||||
WorkerFrameReader reader = new(stream, options);
|
||||
WorkerEnvelope first = await reader.ReadAsync();
|
||||
WorkerEnvelope second = await reader.ReadAsync();
|
||||
|
||||
// Two frames reached the wire; the rejected frame in between left no gap.
|
||||
Assert.Equal(1UL, first.Sequence);
|
||||
Assert.Equal(2UL, second.Sequence);
|
||||
Assert.Equal(stream.Length, stream.Position);
|
||||
}
|
||||
|
||||
/// <summary>Verifies a zero negotiated frame maximum keeps the constructor default.</summary>
|
||||
[Fact]
|
||||
public void AdoptNegotiatedMaxMessageBytes_WithZero_KeepsDefault()
|
||||
|
||||
@@ -19,6 +19,21 @@ public sealed class WorkerPipeSessionTests
|
||||
private const string SessionId = "session-1";
|
||||
private const string Nonce = "nonce-secret";
|
||||
|
||||
// Byte-heavy drain fixture (WRK-21). 10,000 events at ~1.7 KiB each is ~17 MB of queue — far
|
||||
// more than one frame — so DrainEvents must truncate.
|
||||
//
|
||||
// Two limits below are harness accommodations, not properties of the fix. PipePair runs both
|
||||
// ends of a duplex pipe inside one process, with no continuous read pump and with blocking
|
||||
// FlushFileBuffers under every frame write, so it tolerates neither multi-megabyte frames nor
|
||||
// hundreds of large round trips before both ends wedge waiting on each other. Hence a small
|
||||
// negotiated frame maximum, and a smaller queue for the drain-to-empty walk. The byte cap
|
||||
// behaves identically at any frame size; exhaustive no-loss over the full 10,000 events is
|
||||
// covered without a pipe by MxAccessEventQueueTests.
|
||||
private const int ByteHeavyEventCount = 10_000;
|
||||
private const int RepeatedDrainEventCount = 1_000;
|
||||
private const int ByteHeavyEventPayloadBytes = 1_800;
|
||||
private const uint NegotiatedMaxFrameBytes = 128 * 1024;
|
||||
|
||||
/// <summary>Verifies that valid gateway hello triggers worker hello and ready responses.</summary>
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
[Fact]
|
||||
@@ -487,6 +502,361 @@ public sealed class WorkerPipeSessionTests
|
||||
await SendShutdownAndWaitAsync(pipePair, runTask, cancellation.Token);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The WRK-21 repro. A queue full of byte-heavy events (large string values — the payload
|
||||
/// profile this gateway exists for) used to make <c>DrainEvents max_events = 0</c> build a
|
||||
/// reply above the negotiated frame maximum: the writer rejected the frame, the exception
|
||||
/// unwound the session, and the already-dequeued events were gone. The drain is now
|
||||
/// byte-budgeted, so the reply fits, the truncation is reported in the reply's diagnostic
|
||||
/// message, and the session keeps serving.
|
||||
/// </summary>
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
[Fact]
|
||||
public async Task DrainEvents_ByteHeavyQueue_ReplyIsBoundedAndSessionSurvives()
|
||||
{
|
||||
using CancellationTokenSource cancellation = new(TimeSpan.FromSeconds(60));
|
||||
using PipePair pipePair = await PipePair.CreateAsync(cancellation.Token);
|
||||
FakeRuntimeSession runtime = new()
|
||||
{
|
||||
SuppressDrainForBatchSize = 128,
|
||||
BackingQueue = CreateByteHeavyQueue(ByteHeavyEventCount, ByteHeavyEventPayloadBytes),
|
||||
};
|
||||
WorkerPipeSession session = CreatePipeSession(pipePair.WorkerStream, runtime);
|
||||
Task runTask = session.RunAsync(cancellation.Token);
|
||||
await CompleteGatewayHandshakeAsync(pipePair, NegotiatedMaxFrameBytes, cancellation.Token);
|
||||
|
||||
await pipePair.GatewayWriter
|
||||
.WriteAsync(
|
||||
CreateControlCommandEnvelope(
|
||||
"drain-heavy-1",
|
||||
MxCommandKind.DrainEvents,
|
||||
command => command.DrainEvents = new DrainEventsCommand { MaxEvents = 0 }),
|
||||
cancellation.Token);
|
||||
|
||||
WorkerEnvelope replyEnvelope = await ReadUntilAsync(
|
||||
pipePair.GatewayReader,
|
||||
WorkerEnvelope.BodyOneofCase.WorkerCommandReply,
|
||||
cancellation.Token);
|
||||
|
||||
MxCommandReply reply = replyEnvelope.WorkerCommandReply.Reply;
|
||||
Assert.Equal(ProtocolStatusCode.Ok, reply.ProtocolStatus.Code);
|
||||
|
||||
// The whole queue is far larger than one frame, so the reply is a strict subset that fits.
|
||||
Assert.True(
|
||||
replyEnvelope.CalculateSize() <= NegotiatedMaxFrameBytes,
|
||||
$"DrainEvents reply serialized to {replyEnvelope.CalculateSize()} bytes, above the negotiated {NegotiatedMaxFrameBytes}.");
|
||||
Assert.InRange(reply.DrainEvents.Events.Count, 1, ByteHeavyEventCount - 1);
|
||||
Assert.Contains("remain", reply.DiagnosticMessage);
|
||||
Assert.Contains("repeat DrainEvents", reply.DiagnosticMessage);
|
||||
|
||||
// The session is alive: it still answers a ping, and RunAsync has not unwound.
|
||||
await pipePair.GatewayWriter
|
||||
.WriteAsync(CreatePingCommandEnvelope("ping-after-drain", "still-here"), cancellation.Token);
|
||||
WorkerEnvelope pingReply = await ReadUntilAsync(
|
||||
pipePair.GatewayReader,
|
||||
WorkerEnvelope.BodyOneofCase.WorkerCommandReply,
|
||||
envelope => envelope.WorkerCommandReply.Reply.CorrelationId == "ping-after-drain",
|
||||
cancellation.Token);
|
||||
Assert.Equal("still-here", pingReply.WorkerCommandReply.Reply.DiagnosticMessage);
|
||||
Assert.False(runTask.IsCompleted, "The session must survive a byte-heavy DrainEvents.");
|
||||
|
||||
await SendShutdownAndWaitAsync(pipePair, runTask, cancellation.Token);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Verifies the byte-budgeted drain loses nothing: repeating DrainEvents until it comes back
|
||||
/// empty recovers every enqueued event exactly once, in order, across the split replies. The
|
||||
/// pre-fix drain removed events from the queue before sizing the reply, so a rejected frame
|
||||
/// destroyed them — no-loss is the half of the P0 criterion a catch-only fix cannot deliver.
|
||||
/// </summary>
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
[Fact]
|
||||
public async Task DrainEvents_RepeatedCalls_RecoverAllEventsWithoutLoss()
|
||||
{
|
||||
using CancellationTokenSource cancellation = new(TimeSpan.FromSeconds(90));
|
||||
using PipePair pipePair = await PipePair.CreateAsync(cancellation.Token);
|
||||
FakeRuntimeSession runtime = new()
|
||||
{
|
||||
SuppressDrainForBatchSize = 128,
|
||||
BackingQueue = CreateByteHeavyQueue(RepeatedDrainEventCount, ByteHeavyEventPayloadBytes),
|
||||
};
|
||||
WorkerPipeSession session = CreatePipeSession(pipePair.WorkerStream, runtime);
|
||||
Task runTask = session.RunAsync(cancellation.Token);
|
||||
await CompleteGatewayHandshakeAsync(pipePair, NegotiatedMaxFrameBytes, cancellation.Token);
|
||||
|
||||
List<ulong> recovered = new();
|
||||
int replyCount = 0;
|
||||
while (true)
|
||||
{
|
||||
string correlationId = $"drain-loop-{replyCount}";
|
||||
await pipePair.GatewayWriter
|
||||
.WriteAsync(
|
||||
CreateControlCommandEnvelope(
|
||||
correlationId,
|
||||
MxCommandKind.DrainEvents,
|
||||
command => command.DrainEvents = new DrainEventsCommand { MaxEvents = 0 }),
|
||||
cancellation.Token);
|
||||
|
||||
WorkerEnvelope replyEnvelope = await ReadUntilAsync(
|
||||
pipePair.GatewayReader,
|
||||
WorkerEnvelope.BodyOneofCase.WorkerCommandReply,
|
||||
envelope => envelope.WorkerCommandReply.Reply.CorrelationId == correlationId,
|
||||
cancellation.Token);
|
||||
replyCount++;
|
||||
|
||||
MxCommandReply reply = replyEnvelope.WorkerCommandReply.Reply;
|
||||
Assert.True(
|
||||
replyEnvelope.CalculateSize() <= NegotiatedMaxFrameBytes,
|
||||
$"DrainEvents reply {replyCount} serialized to {replyEnvelope.CalculateSize()} bytes.");
|
||||
if (reply.DrainEvents.Events.Count == 0)
|
||||
{
|
||||
break;
|
||||
}
|
||||
|
||||
foreach (MxEvent drained in reply.DrainEvents.Events)
|
||||
{
|
||||
recovered.Add(drained.WorkerSequence);
|
||||
}
|
||||
|
||||
Assert.True(replyCount < 200, "DrainEvents made no progress across 200 replies.");
|
||||
}
|
||||
|
||||
// More than one reply proves the drain really split; every event came back exactly once, in
|
||||
// enqueue order.
|
||||
Assert.True(replyCount > 2, $"Expected the byte cap to split the drain, saw {replyCount} replies.");
|
||||
Assert.Equal(RepeatedDrainEventCount, recovered.Count);
|
||||
for (int index = 0; index < recovered.Count; index++)
|
||||
{
|
||||
Assert.Equal((ulong)(index + 1), recovered[index]);
|
||||
}
|
||||
|
||||
await SendShutdownAndWaitAsync(pipePair, runTask, cancellation.Token);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Regression for the reserve-boundary budget bug. The gateway validator accepts a Worker
|
||||
/// frame maximum as low as 1024 + 64 KiB, and just above that boundary a naive
|
||||
/// subtract-then-guard budget collapses to ~1024 bytes — too small to move even one
|
||||
/// byte-heavy event, so every drain reports truncation with the same head blocked and the
|
||||
/// drain-until-empty loop never terminates. The budget is now a floor (never below half the
|
||||
/// negotiated maximum), so a byte-heavy queue drains to empty even at the validator floor.
|
||||
/// </summary>
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
[Fact]
|
||||
public async Task DrainEvents_AtValidatorFloorFrameMax_MakesProgressAndTerminates()
|
||||
{
|
||||
// The lowest Worker.MaxMessageBytes GatewayOptionsValidator permits: the public gRPC floor
|
||||
// (1024) plus the 64 KiB envelope-overhead reserve. The naive budget would be exactly 1024
|
||||
// here; the floored budget is half of the frame max (~33 KiB).
|
||||
const uint validatorFloorFrameMax = 1024 + (64 * 1024);
|
||||
using CancellationTokenSource cancellation = new(TimeSpan.FromSeconds(60));
|
||||
using PipePair pipePair = await PipePair.CreateAsync(cancellation.Token);
|
||||
FakeRuntimeSession runtime = new()
|
||||
{
|
||||
SuppressDrainForBatchSize = 128,
|
||||
BackingQueue = CreateByteHeavyQueue(200, ByteHeavyEventPayloadBytes),
|
||||
};
|
||||
WorkerPipeSession session = CreatePipeSession(pipePair.WorkerStream, runtime);
|
||||
Task runTask = session.RunAsync(cancellation.Token);
|
||||
await CompleteGatewayHandshakeAsync(pipePair, validatorFloorFrameMax, cancellation.Token);
|
||||
|
||||
int recovered = 0;
|
||||
int replyCount = 0;
|
||||
while (true)
|
||||
{
|
||||
string correlationId = $"floor-drain-{replyCount}";
|
||||
await pipePair.GatewayWriter
|
||||
.WriteAsync(
|
||||
CreateControlCommandEnvelope(
|
||||
correlationId,
|
||||
MxCommandKind.DrainEvents,
|
||||
command => command.DrainEvents = new DrainEventsCommand { MaxEvents = 0 }),
|
||||
cancellation.Token);
|
||||
|
||||
WorkerEnvelope replyEnvelope = await ReadUntilAsync(
|
||||
pipePair.GatewayReader,
|
||||
WorkerEnvelope.BodyOneofCase.WorkerCommandReply,
|
||||
envelope => envelope.WorkerCommandReply.Reply.CorrelationId == correlationId,
|
||||
cancellation.Token);
|
||||
replyCount++;
|
||||
|
||||
MxCommandReply reply = replyEnvelope.WorkerCommandReply.Reply;
|
||||
Assert.True(
|
||||
replyEnvelope.CalculateSize() <= validatorFloorFrameMax,
|
||||
$"reply {replyCount} serialized to {replyEnvelope.CalculateSize()} bytes.");
|
||||
|
||||
int drainedThisReply = reply.DrainEvents.Events.Count;
|
||||
if (drainedThisReply == 0)
|
||||
{
|
||||
break;
|
||||
}
|
||||
|
||||
// The head is never reported as oversized at this frame max: the ~33 KiB floored budget
|
||||
// comfortably fits the ~1.8 KiB events, so each reply makes real progress.
|
||||
Assert.DoesNotContain("alone exceeds", reply.DiagnosticMessage);
|
||||
recovered += drainedThisReply;
|
||||
Assert.True(replyCount < 200, "DrainEvents made no progress at the validator floor frame max.");
|
||||
}
|
||||
|
||||
Assert.Equal(200, recovered);
|
||||
|
||||
await SendShutdownAndWaitAsync(pipePair, runTask, cancellation.Token);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Verifies the control-reply write seam is not session-fatal on size (WRK-21 backstop). The
|
||||
/// reply builders size their payloads, so this path needs a deliberately budget-blind drain
|
||||
/// to reach — but that is the point: a future command or a sizing bug must degrade to an
|
||||
/// error reply for that correlation, never to a dead session.
|
||||
/// </summary>
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
[Fact]
|
||||
public async Task ControlReplyTooLarge_WritesErrorReplyInsteadOfDying()
|
||||
{
|
||||
const uint tinyMaxFrameBytes = 4096;
|
||||
using CancellationTokenSource cancellation = new(TimeSpan.FromSeconds(15));
|
||||
using PipePair pipePair = await PipePair.CreateAsync(cancellation.Token);
|
||||
FakeRuntimeSession runtime = new()
|
||||
{
|
||||
SuppressDrainForBatchSize = 128,
|
||||
BackingQueue = CreateByteHeavyQueue(eventCount: 1, payloadBytes: 16 * 1024),
|
||||
IgnoreDrainByteBudget = true,
|
||||
};
|
||||
WorkerPipeSession session = CreatePipeSession(pipePair.WorkerStream, runtime);
|
||||
Task runTask = session.RunAsync(cancellation.Token);
|
||||
await CompleteGatewayHandshakeAsync(pipePair, tinyMaxFrameBytes, cancellation.Token);
|
||||
|
||||
await pipePair.GatewayWriter
|
||||
.WriteAsync(
|
||||
CreateControlCommandEnvelope(
|
||||
"drain-too-large",
|
||||
MxCommandKind.DrainEvents,
|
||||
command => command.DrainEvents = new DrainEventsCommand { MaxEvents = 0 }),
|
||||
cancellation.Token);
|
||||
|
||||
WorkerEnvelope replyEnvelope = await ReadUntilAsync(
|
||||
pipePair.GatewayReader,
|
||||
WorkerEnvelope.BodyOneofCase.WorkerCommandReply,
|
||||
cancellation.Token);
|
||||
|
||||
MxCommandReply reply = replyEnvelope.WorkerCommandReply.Reply;
|
||||
Assert.Equal("drain-too-large", reply.CorrelationId);
|
||||
Assert.Equal(MxCommandKind.DrainEvents, reply.Kind);
|
||||
Assert.Equal(ProtocolStatusCode.InvalidRequest, reply.ProtocolStatus.Code);
|
||||
Assert.Contains("frame maximum", reply.ProtocolStatus.Message);
|
||||
Assert.False(runTask.IsCompleted, "An oversized control reply must not end the session.");
|
||||
|
||||
await SendShutdownAndWaitAsync(pipePair, runTask, cancellation.Token);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Verifies the same backstop on the STA command path: an oversized reply from a dispatched
|
||||
/// command answers its correlation with an error reply instead of falling into the generic
|
||||
/// catch that faults the whole session with MxaccessCommandFailed.
|
||||
/// </summary>
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
[Fact]
|
||||
public async Task CommandReplyTooLarge_WritesErrorReplyInsteadOfFaulting()
|
||||
{
|
||||
const uint tinyMaxFrameBytes = 4096;
|
||||
using CancellationTokenSource cancellation = new(TimeSpan.FromSeconds(15));
|
||||
using PipePair pipePair = await PipePair.CreateAsync(cancellation.Token);
|
||||
FakeRuntimeSession runtime = new()
|
||||
{
|
||||
DispatchReplyDiagnosticMessage = new string('x', 16 * 1024),
|
||||
};
|
||||
WorkerPipeSession session = CreatePipeSession(pipePair.WorkerStream, runtime);
|
||||
Task runTask = session.RunAsync(cancellation.Token);
|
||||
await CompleteGatewayHandshakeAsync(pipePair, tinyMaxFrameBytes, cancellation.Token);
|
||||
|
||||
await pipePair.GatewayWriter
|
||||
.WriteAsync(CreateCommandEnvelope("command-too-large"), cancellation.Token);
|
||||
|
||||
WorkerEnvelope replyEnvelope = await ReadUntilAsync(
|
||||
pipePair.GatewayReader,
|
||||
WorkerEnvelope.BodyOneofCase.WorkerCommandReply,
|
||||
envelope => envelope.WorkerCommandReply.Reply.CorrelationId == "command-too-large",
|
||||
cancellation.Token);
|
||||
|
||||
MxCommandReply reply = replyEnvelope.WorkerCommandReply.Reply;
|
||||
Assert.Equal(MxCommandKind.Register, reply.Kind);
|
||||
Assert.Equal(ProtocolStatusCode.InvalidRequest, reply.ProtocolStatus.Code);
|
||||
|
||||
// No fault, and the session still reports itself Ready rather than Faulted.
|
||||
await pipePair.GatewayWriter
|
||||
.WriteAsync(
|
||||
CreateControlCommandEnvelope(
|
||||
"state-after-too-large",
|
||||
MxCommandKind.GetSessionState,
|
||||
command => command.GetSessionState = new GetSessionStateCommand()),
|
||||
cancellation.Token);
|
||||
WorkerEnvelope stateEnvelope = await ReadUntilAsync(
|
||||
pipePair.GatewayReader,
|
||||
WorkerEnvelope.BodyOneofCase.WorkerCommandReply,
|
||||
envelope => envelope.WorkerCommandReply.Reply.CorrelationId == "state-after-too-large",
|
||||
cancellation.Token);
|
||||
Assert.Equal(SessionState.Ready, stateEnvelope.WorkerCommandReply.Reply.SessionState.State);
|
||||
Assert.False(runTask.IsCompleted, "An oversized command reply must not fault the session.");
|
||||
|
||||
await SendShutdownAndWaitAsync(pipePair, runTask, cancellation.Token);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// IPC-30. An event above the negotiated frame maximum is undeliverable end to end (the pipe
|
||||
/// maximum sits only an envelope reserve above the public gRPC cap), so the session stays
|
||||
/// fatal by design — but the death must be structured: a WorkerFault naming the event, with
|
||||
/// no value payload in it, before the process exits. Silently dropping the event or
|
||||
/// synthesizing a placeholder were both rejected.
|
||||
/// </summary>
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
[Fact]
|
||||
public async Task RunAsync_EventFrameTooLarge_WritesStructuredFaultThenEndsSession()
|
||||
{
|
||||
const uint tinyMaxFrameBytes = 4096;
|
||||
using CancellationTokenSource cancellation = new(TimeSpan.FromSeconds(15));
|
||||
using PipePair pipePair = await PipePair.CreateAsync(cancellation.Token);
|
||||
FakeRuntimeSession runtime = new();
|
||||
RecordingWorkerLogger logger = new();
|
||||
WorkerPipeSession session = CreatePipeSession(
|
||||
pipePair.WorkerStream,
|
||||
runtime,
|
||||
new WorkerPipeSessionOptions
|
||||
{
|
||||
HeartbeatInterval = TimeSpan.FromMilliseconds(100),
|
||||
HeartbeatGrace = TimeSpan.FromSeconds(5),
|
||||
},
|
||||
logger);
|
||||
runtime.EnqueueEvent(CreateOversizedWorkerEvent(sequence: 77, payloadBytes: 16 * 1024));
|
||||
Task runTask = session.RunAsync(cancellation.Token);
|
||||
await CompleteGatewayHandshakeAsync(pipePair, tinyMaxFrameBytes, cancellation.Token);
|
||||
|
||||
WorkerEnvelope faultEnvelope = await ReadUntilAsync(
|
||||
pipePair.GatewayReader,
|
||||
WorkerEnvelope.BodyOneofCase.WorkerFault,
|
||||
cancellation.Token);
|
||||
|
||||
WorkerFault fault = faultEnvelope.WorkerFault;
|
||||
Assert.Equal(WorkerFaultCategory.ProtocolViolation, fault.Category);
|
||||
Assert.Equal("EventDrain", fault.CommandMethod);
|
||||
Assert.Contains("77", fault.DiagnosticMessage);
|
||||
Assert.Contains("MaxMessageBytes", fault.DiagnosticMessage);
|
||||
// The identity is reported; the value payload never is.
|
||||
Assert.DoesNotContain(new string('x', 64), fault.DiagnosticMessage);
|
||||
|
||||
Assert.Contains(
|
||||
logger.Events,
|
||||
entry => entry.EventName == "WorkerEventFrameTooLarge"
|
||||
&& entry.Fields.TryGetValue("worker_sequence", out object? sequence)
|
||||
&& sequence is ulong sequenceValue
|
||||
&& sequenceValue == 77UL);
|
||||
|
||||
// The session ends, and the fault frame parsed cleanly off the same stream above — the
|
||||
// rejected event never corrupted the wire.
|
||||
Task completedTask = await Task.WhenAny(runTask, Task.Delay(TimeSpan.FromSeconds(5), cancellation.Token));
|
||||
Assert.Same(runTask, completedTask);
|
||||
await Assert.ThrowsAsync<InvalidOperationException>(async () => await runTask);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Verifies that ShutdownWorker returns its OK reply BEFORE the graceful
|
||||
/// shutdown runs and disposes the runtime session, and that the message
|
||||
@@ -1241,6 +1611,15 @@ public sealed class WorkerPipeSessionTests
|
||||
Stream stream,
|
||||
FakeRuntimeSession runtime,
|
||||
WorkerPipeSessionOptions sessionOptions)
|
||||
{
|
||||
return CreatePipeSession(stream, runtime, sessionOptions, logger: null);
|
||||
}
|
||||
|
||||
private static WorkerPipeSession CreatePipeSession(
|
||||
Stream stream,
|
||||
FakeRuntimeSession runtime,
|
||||
WorkerPipeSessionOptions sessionOptions,
|
||||
ZB.MOM.WW.MxGateway.Worker.Bootstrap.IWorkerLogger? logger)
|
||||
{
|
||||
WorkerFrameProtocolOptions options = CreateOptions();
|
||||
return new WorkerPipeSession(
|
||||
@@ -1249,7 +1628,8 @@ public sealed class WorkerPipeSessionTests
|
||||
options,
|
||||
() => 1234,
|
||||
sessionOptions,
|
||||
() => runtime);
|
||||
() => runtime,
|
||||
logger);
|
||||
}
|
||||
|
||||
private static WorkerFrameProtocolOptions CreateOptions()
|
||||
@@ -1270,7 +1650,8 @@ public sealed class WorkerPipeSessionTests
|
||||
private static WorkerEnvelope CreateGatewayHelloEnvelope(
|
||||
string nonce = Nonce,
|
||||
uint supportedProtocolVersion = GatewayContractInfo.WorkerProtocolVersion,
|
||||
ulong sequence = 1)
|
||||
ulong sequence = 1,
|
||||
uint maxFrameBytes = 0)
|
||||
{
|
||||
return new WorkerEnvelope
|
||||
{
|
||||
@@ -1282,6 +1663,10 @@ public sealed class WorkerPipeSessionTests
|
||||
SupportedProtocolVersion = supportedProtocolVersion,
|
||||
Nonce = nonce,
|
||||
GatewayVersion = "test-gateway",
|
||||
// 0 leaves the worker on its compile-time default; a non-zero value is adopted
|
||||
// during the handshake and becomes the frame maximum every later assertion is
|
||||
// measured against.
|
||||
MaxFrameBytes = maxFrameBytes,
|
||||
},
|
||||
};
|
||||
}
|
||||
@@ -1392,12 +1777,55 @@ public sealed class WorkerPipeSessionTests
|
||||
};
|
||||
}
|
||||
|
||||
private static async Task CompleteGatewayHandshakeAsync(
|
||||
private static WorkerEvent CreateOversizedWorkerEvent(ulong sequence, int payloadBytes)
|
||||
{
|
||||
WorkerEvent workerEvent = CreateWorkerEvent(sequence);
|
||||
workerEvent.Event.ItemHandle = 42;
|
||||
workerEvent.Event.RawStatus = new string('x', payloadBytes);
|
||||
return workerEvent;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Fills a real event queue with byte-heavy events — a large string field stands in for the
|
||||
/// array/string <c>MxValue</c> payloads that make a count-capped drain overshoot the frame
|
||||
/// maximum. The queue is real (not the fake's plain list) so the production byte-budgeting runs.
|
||||
/// </summary>
|
||||
/// <param name="eventCount">Number of events to enqueue.</param>
|
||||
/// <param name="payloadBytes">Size of each event's raw-status payload string.</param>
|
||||
/// <returns>The populated queue.</returns>
|
||||
private static MxAccessEventQueue CreateByteHeavyQueue(int eventCount, int payloadBytes)
|
||||
{
|
||||
MxAccessEventQueue queue = new(Math.Max(eventCount, 1));
|
||||
string payload = new string('x', payloadBytes);
|
||||
for (int index = 0; index < eventCount; index++)
|
||||
{
|
||||
queue.Enqueue(new MxEvent
|
||||
{
|
||||
SessionId = SessionId,
|
||||
Family = MxEventFamily.OnDataChange,
|
||||
ItemHandle = index,
|
||||
RawStatus = payload,
|
||||
OnDataChange = new OnDataChangeEvent(),
|
||||
});
|
||||
}
|
||||
|
||||
return queue;
|
||||
}
|
||||
|
||||
private static Task CompleteGatewayHandshakeAsync(
|
||||
PipePair pipePair,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
return CompleteGatewayHandshakeAsync(pipePair, maxFrameBytes: 0, cancellationToken);
|
||||
}
|
||||
|
||||
private static async Task CompleteGatewayHandshakeAsync(
|
||||
PipePair pipePair,
|
||||
uint maxFrameBytes,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
await pipePair.GatewayWriter
|
||||
.WriteAsync(CreateGatewayHelloEnvelope(), cancellationToken)
|
||||
.WriteAsync(CreateGatewayHelloEnvelope(maxFrameBytes: maxFrameBytes), cancellationToken)
|
||||
.ConfigureAwait(false);
|
||||
|
||||
WorkerEnvelope hello = await pipePair.GatewayReader.ReadAsync(cancellationToken).ConfigureAwait(false);
|
||||
|
||||
@@ -98,6 +98,157 @@ public sealed class MxAccessEventQueueTests
|
||||
Assert.Equal(0, queue.Count);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Verifies the byte-budgeted drain stops before the budget is exceeded, leaves the
|
||||
/// remainder queued in order, and reports the exact remaining count (WRK-21). Events that
|
||||
/// do not fit must never be dequeued — dequeuing them is how the pre-fix drain lost events
|
||||
/// when the reply frame was rejected.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void Drain_ByteBudget_StopsBeforeBudgetAndLeavesRemainderQueued()
|
||||
{
|
||||
MxAccessEventQueue queue = new(capacity: 8);
|
||||
for (int itemHandle = 0; itemHandle < 5; itemHandle++)
|
||||
{
|
||||
queue.Enqueue(CreateEventWithPayload(itemHandle, payloadLength: 512));
|
||||
}
|
||||
|
||||
int perEventCost = MeasureDrainCost(payloadLength: 512);
|
||||
|
||||
// Budget for exactly two events (plus a sliver too small for a third).
|
||||
IReadOnlyList<WorkerEvent> drained =
|
||||
queue.Drain(maxEvents: 0, maxTotalBytes: (perEventCost * 2) + (perEventCost / 2)).Events;
|
||||
|
||||
Assert.Equal(2, drained.Count);
|
||||
Assert.Equal(0, drained[0].Event.ItemHandle);
|
||||
Assert.Equal(1, drained[1].Event.ItemHandle);
|
||||
Assert.Equal(3, queue.Count);
|
||||
|
||||
// The undrained remainder is still present, still in order.
|
||||
IReadOnlyList<WorkerEvent> rest = queue.Drain(maxEvents: 0);
|
||||
Assert.Equal(new[] { 2, 3, 4 }, new[] { rest[0].Event.ItemHandle, rest[1].Event.ItemHandle, rest[2].Event.ItemHandle });
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Verifies the byte-budgeted drain reports truncation and the exact remaining count so the
|
||||
/// DrainEvents reply can tell the caller to drain again.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void Drain_ByteBudget_ReportsTruncationAndRemainingCount()
|
||||
{
|
||||
MxAccessEventQueue queue = new(capacity: 8);
|
||||
for (int itemHandle = 0; itemHandle < 4; itemHandle++)
|
||||
{
|
||||
queue.Enqueue(CreateEventWithPayload(itemHandle, payloadLength: 256));
|
||||
}
|
||||
|
||||
// One-and-a-half events' worth of budget: the head fits, the next does not, and the next is
|
||||
// comfortably smaller than the whole budget so it is a plain truncation rather than the
|
||||
// oversized-head case.
|
||||
WorkerEventDrainResult result = queue.Drain(
|
||||
maxEvents: 0,
|
||||
maxTotalBytes: MeasureDrainCost(payloadLength: 256) * 3 / 2);
|
||||
|
||||
Assert.Single(result.Events);
|
||||
Assert.True(result.TruncatedBySize);
|
||||
Assert.Equal(3, result.RemainingCount);
|
||||
Assert.Equal(0UL, result.OversizedHeadSequence);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Verifies the degenerate case: a head event whose own serialized size exceeds the whole
|
||||
/// budget is not drained (draining it would build an oversized reply or lose the event) and
|
||||
/// its worker sequence is reported so an operator can find the offending tag.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void Drain_ByteBudget_OversizedHead_DrainsNothingAndReportsHeadSequence()
|
||||
{
|
||||
MxAccessEventQueue queue = new(capacity: 8);
|
||||
queue.Enqueue(CreateEventWithPayload(itemHandle: 0, payloadLength: 4096));
|
||||
queue.Enqueue(CreateEventWithPayload(itemHandle: 1, payloadLength: 8));
|
||||
|
||||
WorkerEventDrainResult result = queue.Drain(maxEvents: 0, maxTotalBytes: 1024);
|
||||
|
||||
Assert.Empty(result.Events);
|
||||
Assert.True(result.TruncatedBySize);
|
||||
Assert.Equal(2, result.RemainingCount);
|
||||
Assert.Equal(1UL, result.OversizedHeadSequence);
|
||||
|
||||
// The blocked event is still queued — it was never removed.
|
||||
Assert.Equal(2, queue.Count);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The no-loss half of the WRK-21 acceptance criterion, at full scale. Draining the review's
|
||||
/// 10,000 byte-heavy events under a budget that fits only a fraction of them per call must
|
||||
/// return every event exactly once and in order: the pre-fix drain removed events from the
|
||||
/// queue before the reply was sized, so a rejected frame destroyed them. This runs at the
|
||||
/// queue layer because the property is the queue's, and because the pipe harness that covers
|
||||
/// the same walk end to end cannot sustain hundreds of large round trips.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void Drain_ByteBudget_RepeatedCalls_RecoverAllEventsInOrderWithoutLoss()
|
||||
{
|
||||
const int eventCount = 10_000;
|
||||
const int payloadLength = 1_800;
|
||||
MxAccessEventQueue queue = new(eventCount);
|
||||
for (int index = 0; index < eventCount; index++)
|
||||
{
|
||||
queue.Enqueue(CreateEventWithPayload(index, payloadLength));
|
||||
}
|
||||
|
||||
// A budget that fits roughly 35 events, so the walk takes hundreds of calls.
|
||||
int budget = MeasureDrainCost(payloadLength) * 35;
|
||||
List<ulong> recovered = new();
|
||||
int calls = 0;
|
||||
while (true)
|
||||
{
|
||||
WorkerEventDrainResult result = queue.Drain(maxEvents: 0, maxTotalBytes: budget);
|
||||
calls++;
|
||||
if (result.Events.Count == 0)
|
||||
{
|
||||
break;
|
||||
}
|
||||
|
||||
foreach (WorkerEvent drained in result.Events)
|
||||
{
|
||||
recovered.Add(drained.Event.WorkerSequence);
|
||||
}
|
||||
|
||||
Assert.Equal(eventCount - recovered.Count, result.RemainingCount);
|
||||
Assert.True(calls < eventCount, "Drain made no progress.");
|
||||
}
|
||||
|
||||
Assert.True(calls > 100, $"Expected the byte budget to split the drain, saw {calls} calls.");
|
||||
Assert.Equal(eventCount, recovered.Count);
|
||||
for (int index = 0; index < recovered.Count; index++)
|
||||
{
|
||||
Assert.Equal((ulong)(index + 1), recovered[index]);
|
||||
}
|
||||
|
||||
Assert.Equal(0, queue.Count);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Verifies the count cap still binds when the byte budget is generous: the byte cap is an
|
||||
/// additional bound, not a replacement.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void Drain_ByteBudget_CountCapStillBinds()
|
||||
{
|
||||
MxAccessEventQueue queue = new(capacity: 8);
|
||||
for (int itemHandle = 0; itemHandle < 5; itemHandle++)
|
||||
{
|
||||
queue.Enqueue(CreateEventWithPayload(itemHandle, payloadLength: 16));
|
||||
}
|
||||
|
||||
WorkerEventDrainResult result = queue.Drain(maxEvents: 2, maxTotalBytes: 1024 * 1024);
|
||||
|
||||
Assert.Equal(2, result.Events.Count);
|
||||
Assert.False(result.TruncatedBySize);
|
||||
Assert.Equal(3, result.RemainingCount);
|
||||
}
|
||||
|
||||
/// <summary>Verifies that Enqueue is rejected after a fault is recorded manually.</summary>
|
||||
[Fact]
|
||||
public void Enqueue_AfterRecordFault_ThrowsInvalidOperationException()
|
||||
@@ -149,6 +300,42 @@ public sealed class MxAccessEventQueueTests
|
||||
Assert.Equal(WorkerFaultCategory.MxaccessEventConversionFailed, queue.Fault?.Category);
|
||||
}
|
||||
|
||||
// Mirrors MxAccessEventQueue's per-event repeated-field allowance. Kept local rather than made
|
||||
// public on the queue: the byte-budget tests state their budgets in units of that charge, so a
|
||||
// change to it should surface here as a failing bound instead of silently moving with the code.
|
||||
private const int RepeatedFieldOverheadBytes = 8;
|
||||
|
||||
/// <summary>
|
||||
/// Measures what the queue charges one event of the given payload size against the byte budget:
|
||||
/// the serialized <see cref="WorkerEvent"/> as it exists after Enqueue (sequence and timestamp
|
||||
/// stamped) plus the repeated-field allowance. The probe uses item handle 0, a proto3 default
|
||||
/// that is not serialized, so this is a lower bound on the fixtures' real per-event cost — the
|
||||
/// budgets above carry slack rather than assuming byte equality.
|
||||
/// </summary>
|
||||
/// <param name="payloadLength">Length of the event's raw-status payload string.</param>
|
||||
/// <returns>The per-event byte cost.</returns>
|
||||
private static int MeasureDrainCost(int payloadLength)
|
||||
{
|
||||
MxAccessEventQueue probe = new(capacity: 1);
|
||||
probe.Enqueue(CreateEventWithPayload(0, payloadLength));
|
||||
Assert.True(probe.TryDequeue(out WorkerEvent? probeEvent));
|
||||
return probeEvent!.CalculateSize() + RepeatedFieldOverheadBytes;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Builds a byte-heavy event: a large string field is the cheapest stand-in for the array/string
|
||||
/// <see cref="MxValue"/> payloads that make a count-capped drain overshoot the frame maximum.
|
||||
/// </summary>
|
||||
/// <param name="itemHandle">Item handle identifying the event in assertions.</param>
|
||||
/// <param name="payloadLength">Length of the raw-status payload string.</param>
|
||||
/// <returns>The constructed event.</returns>
|
||||
private static MxEvent CreateEventWithPayload(int itemHandle, int payloadLength)
|
||||
{
|
||||
MxEvent mxEvent = CreateEvent(MxEventFamily.OnDataChange, itemHandle);
|
||||
mxEvent.RawStatus = new string('x', payloadLength);
|
||||
return mxEvent;
|
||||
}
|
||||
|
||||
private static MxEvent CreateEvent(
|
||||
MxEventFamily family,
|
||||
int itemHandle)
|
||||
|
||||
@@ -43,6 +43,13 @@ internal sealed class FakeRuntimeSession : IWorkerRuntimeSession
|
||||
/// <summary>Gets or sets whether ShutdownGracefullyAsync throws a TimeoutException.</summary>
|
||||
public bool ThrowTimeoutOnShutdown { get; set; }
|
||||
|
||||
/// <summary>
|
||||
/// Optional diagnostic message stuffed into every dispatched command reply. A long value
|
||||
/// pushes the STA command reply past a small negotiated frame maximum, which is how a test
|
||||
/// drives the <c>ProcessCommandAsync</c> reply-size backstop.
|
||||
/// </summary>
|
||||
public string? DispatchReplyDiagnosticMessage { get; set; }
|
||||
|
||||
/// <summary>Gets a value indicating whether Dispose was called.</summary>
|
||||
public bool Disposed { get; private set; }
|
||||
|
||||
@@ -92,7 +99,7 @@ internal sealed class FakeRuntimeSession : IWorkerRuntimeSession
|
||||
throw new InvalidOperationException("Command failed after shutdown started.");
|
||||
}
|
||||
|
||||
return new MxCommandReply
|
||||
MxCommandReply reply = new()
|
||||
{
|
||||
SessionId = command.SessionId,
|
||||
CorrelationId = command.CorrelationId,
|
||||
@@ -103,6 +110,13 @@ internal sealed class FakeRuntimeSession : IWorkerRuntimeSession
|
||||
Message = "OK",
|
||||
},
|
||||
};
|
||||
|
||||
if (DispatchReplyDiagnosticMessage is not null)
|
||||
{
|
||||
reply.DiagnosticMessage = DispatchReplyDiagnosticMessage;
|
||||
}
|
||||
|
||||
return reply;
|
||||
});
|
||||
}
|
||||
|
||||
@@ -133,6 +147,27 @@ internal sealed class FakeRuntimeSession : IWorkerRuntimeSession
|
||||
/// </summary>
|
||||
public uint? LastDrainMaxEvents { get; private set; }
|
||||
|
||||
/// <summary>
|
||||
/// Optional real event queue backing the drain paths. When set, both
|
||||
/// <see cref="DrainEvents(uint)"/> and <see cref="DrainEvents(uint, int)"/> delegate to it
|
||||
/// so a test can exercise the production byte-budgeting logic behind the fake session.
|
||||
/// </summary>
|
||||
public MxAccessEventQueue? BackingQueue { get; set; }
|
||||
|
||||
/// <summary>
|
||||
/// When set, <see cref="DrainEvents(uint, int)"/> ignores the byte budget and drains purely
|
||||
/// by count. Simulates the "sizing bug or future command" case the control-reply size
|
||||
/// backstop exists for, so a test can drive an oversized reply without a real budgeting
|
||||
/// defect.
|
||||
/// </summary>
|
||||
public bool IgnoreDrainByteBudget { get; set; }
|
||||
|
||||
/// <summary>
|
||||
/// Records the <c>maxTotalBytes</c> argument of the most recent byte-budgeted
|
||||
/// <see cref="DrainEvents(uint, int)"/> call.
|
||||
/// </summary>
|
||||
public int? LastDrainMaxTotalBytes { get; private set; }
|
||||
|
||||
/// <inheritdoc />
|
||||
public IReadOnlyList<WorkerEvent> DrainEvents(uint maxEvents)
|
||||
{
|
||||
@@ -143,6 +178,76 @@ internal sealed class FakeRuntimeSession : IWorkerRuntimeSession
|
||||
|
||||
LastDrainMaxEvents = maxEvents;
|
||||
|
||||
if (BackingQueue is not null)
|
||||
{
|
||||
return BackingQueue.Drain(maxEvents);
|
||||
}
|
||||
|
||||
lock (gate)
|
||||
{
|
||||
int drainCount = maxEvents == 0
|
||||
? events.Count
|
||||
: Math.Min(events.Count, checked((int)Math.Min(maxEvents, int.MaxValue)));
|
||||
List<WorkerEvent> drained = new(drainCount);
|
||||
for (int index = 0; index < drainCount; index++)
|
||||
{
|
||||
drained.Add(events.Dequeue());
|
||||
}
|
||||
|
||||
return drained;
|
||||
}
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public WorkerEventDrainResult DrainEvents(uint maxEvents, int maxTotalBytes)
|
||||
{
|
||||
if (SuppressDrainForBatchSize is uint suppressed && maxEvents == suppressed)
|
||||
{
|
||||
return new WorkerEventDrainResult(
|
||||
Array.Empty<WorkerEvent>(),
|
||||
truncatedBySize: false,
|
||||
remainingCount: PendingEventCount,
|
||||
oversizedHeadSequence: 0);
|
||||
}
|
||||
|
||||
LastDrainMaxEvents = maxEvents;
|
||||
LastDrainMaxTotalBytes = maxTotalBytes;
|
||||
|
||||
if (BackingQueue is not null && !IgnoreDrainByteBudget)
|
||||
{
|
||||
return BackingQueue.Drain(maxEvents, maxTotalBytes);
|
||||
}
|
||||
|
||||
// Count-only drain: either no backing queue (the simple fakes) or a deliberately
|
||||
// budget-blind drain used to exercise the reply-size backstop.
|
||||
IReadOnlyList<WorkerEvent> drained = BackingQueue is not null
|
||||
? BackingQueue.Drain(maxEvents)
|
||||
: DrainByCount(maxEvents);
|
||||
return new WorkerEventDrainResult(
|
||||
drained,
|
||||
truncatedBySize: false,
|
||||
remainingCount: PendingEventCount,
|
||||
oversizedHeadSequence: 0);
|
||||
}
|
||||
|
||||
private int PendingEventCount
|
||||
{
|
||||
get
|
||||
{
|
||||
if (BackingQueue is not null)
|
||||
{
|
||||
return BackingQueue.Count;
|
||||
}
|
||||
|
||||
lock (gate)
|
||||
{
|
||||
return events.Count;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private IReadOnlyList<WorkerEvent> DrainByCount(uint maxEvents)
|
||||
{
|
||||
lock (gate)
|
||||
{
|
||||
int drainCount = maxEvents == 0
|
||||
|
||||
@@ -43,8 +43,9 @@ public sealed class WorkerFrameWriter
|
||||
private readonly Queue<PendingFrame> _eventFrames = new Queue<PendingFrame>();
|
||||
|
||||
// Only ever read/written by the current write-lock holder while draining, so no interlock is
|
||||
// needed. Starts at 0 and is pre-incremented, so the first written frame carries sequence 1
|
||||
// (matching the previous behaviour).
|
||||
// needed. Starts at 0 and is committed only immediately before the stream write, so the first
|
||||
// written frame carries sequence 1 and a per-frame rejection leaves the counter untouched —
|
||||
// the next accepted frame reuses the number and the wire sequence stays contiguous.
|
||||
private ulong _nextSequence;
|
||||
|
||||
/// <summary>Initializes a new instance of the WorkerFrameWriter class.</summary>
|
||||
@@ -237,7 +238,14 @@ public sealed class WorkerFrameWriter
|
||||
|
||||
// Stamp the sequence at the actual point of writing, under the write lock, so the wire order
|
||||
// and the stamped sequence agree regardless of caller concurrency or priority.
|
||||
envelope.Sequence = unchecked(++_nextSequence);
|
||||
//
|
||||
// Peek-stamp-commit (WRK-23): the sequence participates in CalculateSize() (varint width),
|
||||
// so it must be stamped before the size checks — but a per-frame rejection must not burn a
|
||||
// number, or the wire shows phantom gaps that an operator reads as lost frames. Stamp a
|
||||
// candidate, validate the stamped envelope, and commit the counter only once the frame is
|
||||
// certain to be written.
|
||||
ulong candidateSequence = unchecked(_nextSequence + 1);
|
||||
envelope.Sequence = candidateSequence;
|
||||
|
||||
int payloadLength = envelope.CalculateSize();
|
||||
if (payloadLength == 0)
|
||||
@@ -254,6 +262,8 @@ public sealed class WorkerFrameWriter
|
||||
$"Worker envelope payload length {payloadLength} exceeds the configured maximum of {_options.MaxMessageBytes} bytes.");
|
||||
}
|
||||
|
||||
_nextSequence = candidateSequence;
|
||||
|
||||
// Serialize once into a single buffer that carries the 4-byte length prefix followed by the
|
||||
// payload, then issue one stream write. This avoids a second serialization pass, a separate
|
||||
// prefix array, and a separate prefix write. The flush is deferred to the end of the drained
|
||||
|
||||
@@ -5,6 +5,7 @@ using System.IO;
|
||||
using System.Threading;
|
||||
using System.Threading.Tasks;
|
||||
using Google.Protobuf.WellKnownTypes;
|
||||
using ZB.MOM.WW.MxGateway.Contracts;
|
||||
using ZB.MOM.WW.MxGateway.Contracts.Proto;
|
||||
using ZB.MOM.WW.MxGateway.Worker.Bootstrap;
|
||||
using ZB.MOM.WW.MxGateway.Worker.MxAccess;
|
||||
@@ -18,12 +19,11 @@ public sealed class WorkerPipeSession
|
||||
private static readonly TimeSpan BackgroundTaskStopTimeout = TimeSpan.FromSeconds(1);
|
||||
private const uint EventDrainBatchSize = 128;
|
||||
|
||||
// Hard cap on how many events a single DrainEvents diagnostic reply may carry. DrainEvents is a
|
||||
// non-streaming control command, so an unbounded drain (including the max_events = 0 "as many as
|
||||
// available" request) could pack the whole queue into one session-killing reply frame.
|
||||
// The gateway request validator rejects requests above its public ceiling; this worker-side cap is
|
||||
// the backstop and defines the effective per-reply maximum. Kept in step with that public ceiling.
|
||||
private const uint MaxDrainEventsPerReply = 10_000;
|
||||
// Headroom subtracted from the negotiated frame maximum when budgeting a DrainEvents reply. It
|
||||
// covers the WorkerEnvelope/WorkerCommandReply/MxCommandReply wrapper the drained events are
|
||||
// packed into — the same envelope-overhead reserve rationale docs/WorkerFrameProtocol.md
|
||||
// records for the frame max itself.
|
||||
private const int DrainReplyFrameHeadroomBytes = 64 * 1024;
|
||||
|
||||
private readonly WorkerFrameProtocolOptions _options;
|
||||
private readonly Func<int> _processIdProvider;
|
||||
@@ -376,13 +376,83 @@ public sealed class WorkerPipeSession
|
||||
// Events are the low-priority frame class: the writer holds them behind any pending
|
||||
// control frame (reply, fault, heartbeat, shutdown ack) so those are not delayed
|
||||
// behind an event backlog.
|
||||
await _writer
|
||||
.WriteAsync(CreateEnvelope(workerEvent), WorkerFrameWritePriority.Event, cancellationToken)
|
||||
.ConfigureAwait(false);
|
||||
try
|
||||
{
|
||||
await _writer
|
||||
.WriteAsync(CreateEnvelope(workerEvent), WorkerFrameWritePriority.Event, cancellationToken)
|
||||
.ConfigureAwait(false);
|
||||
}
|
||||
catch (WorkerFrameProtocolException exception)
|
||||
when (exception.ErrorCode == WorkerFrameProtocolErrorCode.MessageTooLarge)
|
||||
{
|
||||
await FaultOnOversizedEventAsync(workerEvent, exception, cancellationToken)
|
||||
.ConfigureAwait(false);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Ends the session on an event that cannot be framed, but deliberately and diagnosably
|
||||
/// (IPC-30). An event above the negotiated frame maximum is undeliverable end to end — the
|
||||
/// pipe maximum sits only an envelope reserve above the public gRPC cap — so dropping it
|
||||
/// would silently make the event stream unfaithful, and synthesizing a placeholder is barred
|
||||
/// by the no-synthesized-events rule. Instead the worker records which event blocked (never
|
||||
/// its value: the redaction rule), writes a structured fault the gateway and dashboard can
|
||||
/// surface, and then exits as it did before. Remediation is configuration:
|
||||
/// <c>MxGateway:Worker:MaxMessageBytes</c>. Other per-frame rejection codes keep the previous
|
||||
/// behavior — they indicate worker bugs, not workload size.
|
||||
/// </summary>
|
||||
private async Task FaultOnOversizedEventAsync(
|
||||
WorkerEvent workerEvent,
|
||||
WorkerFrameProtocolException exception,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
MxEvent? mxEvent = workerEvent.Event;
|
||||
string family = (mxEvent?.Family ?? MxEventFamily.Unspecified).ToString();
|
||||
ulong workerSequence = mxEvent?.WorkerSequence ?? 0;
|
||||
int serverHandle = mxEvent?.ServerHandle ?? 0;
|
||||
int itemHandle = mxEvent?.ItemHandle ?? 0;
|
||||
|
||||
_logger?.Error(
|
||||
"WorkerEventFrameTooLarge",
|
||||
new Dictionary<string, object?>
|
||||
{
|
||||
["session_id"] = _options.SessionId,
|
||||
["event_family"] = family,
|
||||
["worker_sequence"] = workerSequence,
|
||||
["server_handle"] = serverHandle,
|
||||
["item_handle"] = itemHandle,
|
||||
["max_message_bytes"] = _options.MaxMessageBytes,
|
||||
// Sizes only — the event value never reaches the log.
|
||||
["reason"] = exception.Message,
|
||||
});
|
||||
|
||||
string diagnosticMessage =
|
||||
$"{family} event for server handle {serverHandle}, item handle {itemHandle} "
|
||||
+ $"(worker sequence {workerSequence}) exceeds the negotiated frame maximum of "
|
||||
+ $"{_options.MaxMessageBytes} bytes and cannot be delivered; raise "
|
||||
+ "MxGateway:Worker:MaxMessageBytes for this workload.";
|
||||
|
||||
_state = WorkerState.Faulted;
|
||||
await TryWriteFaultAsync(
|
||||
new WorkerFault
|
||||
{
|
||||
Category = WorkerFaultCategory.ProtocolViolation,
|
||||
CommandMethod = "EventDrain",
|
||||
ExceptionType = exception.GetType().FullName ?? string.Empty,
|
||||
DiagnosticMessage = diagnosticMessage,
|
||||
ProtocolStatus = new ProtocolStatus
|
||||
{
|
||||
Code = ProtocolStatusCode.ProtocolViolation,
|
||||
Message = diagnosticMessage,
|
||||
},
|
||||
},
|
||||
cancellationToken).ConfigureAwait(false);
|
||||
|
||||
throw new InvalidOperationException(diagnosticMessage, exception);
|
||||
}
|
||||
|
||||
private async Task<bool> DispatchGatewayEnvelopeAsync(
|
||||
WorkerEnvelope envelope,
|
||||
CancellationToken cancellationToken)
|
||||
@@ -478,7 +548,8 @@ public sealed class WorkerPipeSession
|
||||
_ => CreateControlOkReply(correlationId, command.Kind),
|
||||
};
|
||||
|
||||
await WriteControlReplyAsync(reply, cancellationToken).ConfigureAwait(false);
|
||||
await WriteControlReplyWithSizeBackstopAsync(reply, correlationId, command.Kind, cancellationToken)
|
||||
.ConfigureAwait(false);
|
||||
return true;
|
||||
}
|
||||
|
||||
@@ -495,6 +566,105 @@ public sealed class WorkerPipeSession
|
||||
cancellationToken);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Writes a control reply, answering the correlation with a small error reply instead of
|
||||
/// unwinding the session if the reply does not fit the negotiated frame maximum. Reply
|
||||
/// builders already size their payloads (see <see cref="CreateDrainEventsReply"/>), so this
|
||||
/// is a backstop against a future command or a sizing bug — but without it a single
|
||||
/// oversized diagnostic reply is session-fatal, which no diagnostics command may be.
|
||||
/// </summary>
|
||||
private async Task WriteControlReplyWithSizeBackstopAsync(
|
||||
MxCommandReply reply,
|
||||
string correlationId,
|
||||
MxCommandKind kind,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
try
|
||||
{
|
||||
await WriteControlReplyAsync(reply, cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
catch (WorkerFrameProtocolException exception)
|
||||
when (exception.ErrorCode == WorkerFrameProtocolErrorCode.MessageTooLarge)
|
||||
{
|
||||
LogControlReplyTooLarge(correlationId, kind, exception);
|
||||
await WriteReplyTooLargeFallbackAsync(correlationId, kind, cancellationToken)
|
||||
.ConfigureAwait(false);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Writes the small <c>InvalidRequest</c> reply that answers a correlation whose real reply
|
||||
/// overshot the frame maximum. The fallback itself is a handful of bytes, so it fits any
|
||||
/// sane negotiated maximum; the only way it can also throw <c>MessageTooLarge</c> is a
|
||||
/// pathologically tiny negotiated maximum below the gateway's validation floor — the
|
||||
/// pre-existing WRK-24 gap, which adds the negotiated-max lower bound that makes this
|
||||
/// unreachable. Until then, a defensive swallow keeps the "no diagnostics command is
|
||||
/// session-fatal" invariant true even in that degenerate config: the correlation goes
|
||||
/// unanswered and the gateway's own per-command timeout covers it, but the session lives.
|
||||
/// </summary>
|
||||
private async Task WriteReplyTooLargeFallbackAsync(
|
||||
string correlationId,
|
||||
MxCommandKind kind,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
try
|
||||
{
|
||||
await WriteControlReplyAsync(
|
||||
CreateReplyTooLargeReply(correlationId, kind),
|
||||
cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
catch (WorkerFrameProtocolException exception)
|
||||
when (exception.ErrorCode == WorkerFrameProtocolErrorCode.MessageTooLarge)
|
||||
{
|
||||
_logger?.Error(
|
||||
"WorkerControlReplyFallbackTooLarge",
|
||||
new Dictionary<string, object?>
|
||||
{
|
||||
["correlation_id"] = correlationId,
|
||||
["command_kind"] = kind.ToString(),
|
||||
["max_message_bytes"] = _options.MaxMessageBytes,
|
||||
["reason"] = exception.Message,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
private void LogControlReplyTooLarge(
|
||||
string correlationId,
|
||||
MxCommandKind kind,
|
||||
WorkerFrameProtocolException exception)
|
||||
{
|
||||
_logger?.Error(
|
||||
"WorkerControlReplyTooLarge",
|
||||
new Dictionary<string, object?>
|
||||
{
|
||||
["correlation_id"] = correlationId,
|
||||
["command_kind"] = kind.ToString(),
|
||||
["max_message_bytes"] = _options.MaxMessageBytes,
|
||||
// The writer's message carries the rejected payload length; it names sizes only,
|
||||
// never reply content.
|
||||
["reason"] = exception.Message,
|
||||
});
|
||||
}
|
||||
|
||||
private MxCommandReply CreateReplyTooLargeReply(string correlationId, MxCommandKind kind)
|
||||
{
|
||||
const string message =
|
||||
"Worker reply exceeded the negotiated frame maximum; retry with a smaller request.";
|
||||
return new MxCommandReply
|
||||
{
|
||||
SessionId = _options.SessionId,
|
||||
CorrelationId = correlationId,
|
||||
Kind = kind,
|
||||
Hresult = 0,
|
||||
DiagnosticMessage = message,
|
||||
ProtocolStatus = new ProtocolStatus
|
||||
{
|
||||
Code = ProtocolStatusCode.InvalidRequest,
|
||||
Message = message,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
private MxCommandReply CreatePingReply(string correlationId, MxCommand command)
|
||||
{
|
||||
MxCommandReply reply = CreateControlOkReply(correlationId, command.Kind);
|
||||
@@ -543,24 +713,77 @@ public sealed class WorkerPipeSession
|
||||
if (runtimeSession is not null)
|
||||
{
|
||||
// Bound the diagnostic drain so max_events = 0 ("as many as available") or an over-large
|
||||
// request cannot pack the whole queue into one session-killing reply frame.
|
||||
// request cannot pack the whole queue into one session-killing reply frame. The count cap
|
||||
// alone is not enough: byte-heavy events overshoot the negotiated frame maximum long
|
||||
// before the count ceiling, so the drain is also byte-budgeted and sizes the reply while
|
||||
// draining — an event that does not fit is left queued rather than dequeued and lost.
|
||||
uint requested = command.DrainEvents?.MaxEvents ?? 0;
|
||||
uint maxEvents = requested == 0 || requested > MaxDrainEventsPerReply
|
||||
? MaxDrainEventsPerReply
|
||||
uint maxEvents = requested == 0 || requested > GatewayContractInfo.MaxDrainEventsPerCommand
|
||||
? GatewayContractInfo.MaxDrainEventsPerCommand
|
||||
: requested;
|
||||
foreach (WorkerEvent workerEvent in runtimeSession.DrainEvents(maxEvents))
|
||||
WorkerEventDrainResult drainResult = runtimeSession.DrainEvents(
|
||||
maxEvents,
|
||||
ResolveDrainReplyByteBudget());
|
||||
foreach (WorkerEvent workerEvent in drainResult.Events)
|
||||
{
|
||||
if (workerEvent.Event is not null)
|
||||
{
|
||||
drainReply.Events.Add(workerEvent.Event);
|
||||
}
|
||||
}
|
||||
|
||||
if (drainResult.TruncatedBySize)
|
||||
{
|
||||
// DrainEventsReply has no truncation field, and adding one would regenerate every
|
||||
// language client for a diagnostic nicety. The reply's existing DiagnosticMessage
|
||||
// carries the same information at zero contract cost; the caller contract is to
|
||||
// repeat DrainEvents until it comes back empty.
|
||||
reply.DiagnosticMessage = CreateDrainTruncationMessage(
|
||||
drainReply.Events.Count,
|
||||
drainResult);
|
||||
}
|
||||
}
|
||||
|
||||
reply.DrainEvents = drainReply;
|
||||
return reply;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Byte budget for the events packed into one DrainEvents reply: the negotiated frame
|
||||
/// maximum less a fixed reserve for the envelope/reply wrapper, but never below half the
|
||||
/// negotiated maximum. The lower bound must be a floor, not a step: a bare
|
||||
/// <c>subtract-then-guard-positive</c> collapses the budget to a handful of bytes just above
|
||||
/// the reserve (e.g. at the validator-permitted floor MaxMessageBytes = 1024 + 64 KiB the
|
||||
/// subtraction leaves 1024, too small to move even one byte-heavy event, so every drain
|
||||
/// truncates and the drain-until-empty caller never terminates). Taking the max with
|
||||
/// half the negotiated maximum keeps the budget monotonic across the reserve boundary while
|
||||
/// still leaving the full reserve for the wrapper whenever the frame max is large enough
|
||||
/// that the reserve is the smaller subtraction — which is every configuration above 128 KiB.
|
||||
/// </summary>
|
||||
private int ResolveDrainReplyByteBudget()
|
||||
{
|
||||
return Math.Max(
|
||||
_options.MaxMessageBytes - DrainReplyFrameHeadroomBytes,
|
||||
_options.MaxMessageBytes / 2);
|
||||
}
|
||||
|
||||
private static string CreateDrainTruncationMessage(
|
||||
int returnedCount,
|
||||
WorkerEventDrainResult drainResult)
|
||||
{
|
||||
string message =
|
||||
$"{returnedCount} events returned, {drainResult.RemainingCount} remain; "
|
||||
+ "repeat DrainEvents for the rest.";
|
||||
if (drainResult.OversizedHeadSequence != 0)
|
||||
{
|
||||
message +=
|
||||
$" The next event (worker sequence {drainResult.OversizedHeadSequence}) alone exceeds "
|
||||
+ "the negotiated frame maximum and cannot be drained; raise MxGateway:Worker:MaxMessageBytes.";
|
||||
}
|
||||
|
||||
return message;
|
||||
}
|
||||
|
||||
private MxCommandReply CreateControlOkReply(string correlationId, MxCommandKind kind)
|
||||
{
|
||||
return new MxCommandReply
|
||||
@@ -627,15 +850,30 @@ public sealed class WorkerPipeSession
|
||||
return;
|
||||
}
|
||||
|
||||
await _writer
|
||||
.WriteAsync(
|
||||
CreateEnvelope(new WorkerCommandReply
|
||||
{
|
||||
Reply = reply,
|
||||
CompletedTimestamp = Timestamp.FromDateTime(DateTime.UtcNow),
|
||||
}),
|
||||
cancellationToken)
|
||||
.ConfigureAwait(false);
|
||||
try
|
||||
{
|
||||
await _writer
|
||||
.WriteAsync(
|
||||
CreateEnvelope(new WorkerCommandReply
|
||||
{
|
||||
Reply = reply,
|
||||
CompletedTimestamp = Timestamp.FromDateTime(DateTime.UtcNow),
|
||||
}),
|
||||
cancellationToken)
|
||||
.ConfigureAwait(false);
|
||||
}
|
||||
catch (WorkerFrameProtocolException sizeException)
|
||||
when (sizeException.ErrorCode == WorkerFrameProtocolErrorCode.MessageTooLarge)
|
||||
{
|
||||
// An oversized STA command reply is a property of that one command, not of the
|
||||
// session. Answer the correlation with an error reply instead of falling into the
|
||||
// generic catch below, which would fault the whole session for it. The fallback
|
||||
// write is itself size-guarded (see WriteReplyTooLargeFallbackAsync) so a degenerate
|
||||
// negotiated maximum cannot make even this backstop session-fatal.
|
||||
LogControlReplyTooLarge(envelope.CorrelationId, command.Kind, sizeException);
|
||||
await WriteReplyTooLargeFallbackAsync(envelope.CorrelationId, command.Kind, cancellationToken)
|
||||
.ConfigureAwait(false);
|
||||
}
|
||||
}
|
||||
catch (Exception exception) when (exception is not OperationCanceledException)
|
||||
{
|
||||
|
||||
@@ -44,6 +44,19 @@ public interface IWorkerRuntimeSession : IDisposable
|
||||
/// <returns>List of drained events.</returns>
|
||||
IReadOnlyList<WorkerEvent> DrainEvents(uint maxEvents);
|
||||
|
||||
/// <summary>
|
||||
/// Drains pending events bounded by both a count cap and a byte budget, so a caller building a
|
||||
/// single reply frame never removes an event it cannot ship.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Declared as a second method rather than a default interface method: the worker targets
|
||||
/// .NET Framework 4.8, which has no runtime support for default interface members.
|
||||
/// </remarks>
|
||||
/// <param name="maxEvents">Maximum number of events to drain; 0 means "no count limit".</param>
|
||||
/// <param name="maxTotalBytes">Byte budget for the drained events' estimated serialized size.</param>
|
||||
/// <returns>The drained events and the truncation facts describing what stayed queued.</returns>
|
||||
WorkerEventDrainResult DrainEvents(uint maxEvents, int maxTotalBytes);
|
||||
|
||||
/// <summary>
|
||||
/// Drains a pending fault from the queue, if any.
|
||||
/// </summary>
|
||||
|
||||
@@ -26,6 +26,15 @@ public sealed class MxAccessEventQueue
|
||||
/// </summary>
|
||||
public const int DefaultCapacity = 10000;
|
||||
|
||||
// Extra per-event slack added to WorkerEvent.CalculateSize() when charging the byte budget in
|
||||
// Drain(maxEvents, maxTotalBytes). CalculateSize() already accounts for the event's own tag and
|
||||
// length-delimiter (the WorkerEvent wrapper serializes MxEvent as field 1, and the reply packs
|
||||
// each MxEvent as DrainEventsReply.events field 1 with the identical tag+length shape), so this
|
||||
// is a pure safety margin over an already-conservative estimate — not compensation for a missing
|
||||
// wrapper. It keeps the running total strictly ahead of the true serialized size so a rounding
|
||||
// edge can never push the packed reply past the frame maximum.
|
||||
private const int RepeatedFieldOverheadBytes = 8;
|
||||
|
||||
private readonly int capacity;
|
||||
private readonly Queue<WorkerEvent> events;
|
||||
private readonly object syncRoot = new();
|
||||
@@ -209,6 +218,64 @@ public sealed class MxAccessEventQueue
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Drains from the head while both the count cap and a byte budget allow it, so the caller can
|
||||
/// build a reply frame that is guaranteed to fit the negotiated frame maximum.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// The size decision happens inside the queue lock, so an event is dequeued only once it is
|
||||
/// known to fit: an event that does not fit stays at the head for the next call and is never
|
||||
/// lost (WRK-21). Per-event cost is <c>WorkerEvent.CalculateSize()</c> — which already
|
||||
/// includes the event's own tag and length prefix, the same shape the reply's
|
||||
/// <c>events</c> repeated field packs it into — plus <see cref="RepeatedFieldOverheadBytes"/>
|
||||
/// of pure slack, so the running total stays strictly ahead of the true serialized size and
|
||||
/// the estimate errs on the safe side.
|
||||
/// </remarks>
|
||||
/// <param name="maxEvents">Maximum number of events to drain; 0 means "no count limit".</param>
|
||||
/// <param name="maxTotalBytes">Byte budget for the drained events' estimated serialized size.</param>
|
||||
/// <returns>The drained events plus the truncation facts the caller reports to the gateway.</returns>
|
||||
public WorkerEventDrainResult Drain(uint maxEvents, int maxTotalBytes)
|
||||
{
|
||||
lock (syncRoot)
|
||||
{
|
||||
int countLimit = maxEvents == 0
|
||||
? int.MaxValue
|
||||
: checked((int)Math.Min(maxEvents, int.MaxValue));
|
||||
List<WorkerEvent> drained = new();
|
||||
int remainingBudget = maxTotalBytes;
|
||||
bool truncatedBySize = false;
|
||||
ulong oversizedHeadSequence = 0;
|
||||
|
||||
while (drained.Count < countLimit && events.Count > 0)
|
||||
{
|
||||
WorkerEvent head = events.Peek();
|
||||
int cost = head.CalculateSize() + RepeatedFieldOverheadBytes;
|
||||
if (cost > remainingBudget)
|
||||
{
|
||||
truncatedBySize = true;
|
||||
if (cost > maxTotalBytes)
|
||||
{
|
||||
// The head alone cannot fit this budget, so repeating the call will not
|
||||
// move it either. Report its sequence instead of silently stalling; the
|
||||
// events that did fit are still returned.
|
||||
oversizedHeadSequence = head.Event?.WorkerSequence ?? 0;
|
||||
}
|
||||
|
||||
break;
|
||||
}
|
||||
|
||||
remainingBudget -= cost;
|
||||
drained.Add(events.Dequeue());
|
||||
}
|
||||
|
||||
return new WorkerEventDrainResult(
|
||||
drained,
|
||||
truncatedBySize,
|
||||
events.Count,
|
||||
oversizedHeadSequence);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Records a fault if one has not already been recorded.
|
||||
/// </summary>
|
||||
|
||||
@@ -392,6 +392,12 @@ public sealed class MxAccessStaSession : IWorkerRuntimeSession
|
||||
return eventQueue.Drain(maxEvents);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public WorkerEventDrainResult DrainEvents(uint maxEvents, int maxTotalBytes)
|
||||
{
|
||||
return eventQueue.Drain(maxEvents, maxTotalBytes);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public WorkerFault? DrainFault()
|
||||
{
|
||||
|
||||
@@ -0,0 +1,56 @@
|
||||
using System.Collections.Generic;
|
||||
using ZB.MOM.WW.MxGateway.Contracts.Proto;
|
||||
|
||||
namespace ZB.MOM.WW.MxGateway.Worker.MxAccess;
|
||||
|
||||
/// <summary>
|
||||
/// Outcome of a byte-budgeted drain from the MXAccess outbound event queue.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// A count cap alone cannot keep a <c>DrainEvents</c> reply inside the negotiated frame
|
||||
/// maximum: byte-heavy events (large string or array <c>MxValue</c>s) overshoot the frame max
|
||||
/// long before the count ceiling is reached, and the writer's per-frame rejection then
|
||||
/// destroys events that were already removed from the queue. The byte-budgeted drain sizes
|
||||
/// the reply while draining, so an event that does not fit is never dequeued (WRK-21), and
|
||||
/// this result carries the truncation facts the reply's <c>DiagnosticMessage</c> reports —
|
||||
/// no contract change is needed to express them.
|
||||
/// Plain constructor and get-only properties: the worker targets .NET Framework 4.8, which
|
||||
/// has no init-only members or positional records.
|
||||
/// </remarks>
|
||||
public sealed class WorkerEventDrainResult
|
||||
{
|
||||
/// <summary>Initializes a new instance of the <see cref="WorkerEventDrainResult"/> class.</summary>
|
||||
/// <param name="events">Events removed from the queue, in enqueue order.</param>
|
||||
/// <param name="truncatedBySize">Whether the byte budget, not the count cap, ended the drain.</param>
|
||||
/// <param name="remainingCount">Number of events still queued after the drain.</param>
|
||||
/// <param name="oversizedHeadSequence">
|
||||
/// Worker sequence of a head event whose own serialized size exceeds the whole budget, so
|
||||
/// no future call of the same budget can ship it; 0 when there is no such event.
|
||||
/// </param>
|
||||
public WorkerEventDrainResult(
|
||||
IReadOnlyList<WorkerEvent> events,
|
||||
bool truncatedBySize,
|
||||
int remainingCount,
|
||||
ulong oversizedHeadSequence)
|
||||
{
|
||||
Events = events;
|
||||
TruncatedBySize = truncatedBySize;
|
||||
RemainingCount = remainingCount;
|
||||
OversizedHeadSequence = oversizedHeadSequence;
|
||||
}
|
||||
|
||||
/// <summary>Gets the events removed from the queue, in enqueue order.</summary>
|
||||
public IReadOnlyList<WorkerEvent> Events { get; }
|
||||
|
||||
/// <summary>Gets a value indicating whether the byte budget ended the drain early.</summary>
|
||||
public bool TruncatedBySize { get; }
|
||||
|
||||
/// <summary>Gets the number of events still queued after the drain.</summary>
|
||||
public int RemainingCount { get; }
|
||||
|
||||
/// <summary>
|
||||
/// Gets the worker sequence of the head event that alone exceeds the byte budget, or 0 when
|
||||
/// no single event blocks the drain. Naming it lets an operator find the offending tag.
|
||||
/// </summary>
|
||||
public ulong OversizedHeadSequence { get; }
|
||||
}
|
||||
+2
-2
@@ -133,8 +133,8 @@ No placeholder/empty/`Assert.True(true)` tests were found anywhere.
|
||||
- 📄 **7.1 D1 plan header stale** — `docs/plans/2026-06-14-deferred-followups.md:4` still says *"Plan only — NOT yet executed,"* but D1 is **done** (`Dashboard/DashboardSnapshotService.cs:198`, commit `4af24b9`). Update the plan status.
|
||||
- 📄 **7.2 `AlarmClientDiscovery.md` STA "production fix needed" prose is stale** — `docs/AlarmClientDiscovery.md:765-774` reads as a pending follow-up, but alarms now run through the worker STA / `GatewayAlarmMonitor` (merged). Re-check against current code.
|
||||
- 📄 **7.3 EventsHub "publisher side is a follow-up" comment is stale** — `Dashboard/Hubs/EventsHub.cs:9-17`; the `DashboardEventBroadcaster` exists, is DI-registered (`Dashboard/DashboardServiceCollectionExtensions.cs:47`), runs in the live loop (`Grpc/EventStreamService.cs:133`), and `SessionDetailsPage.razor` renders the feed.
|
||||
- 📄 **7.4 CLAUDE.md project-name drift** — CLAUDE.md uses `src/MxGateway.Server`/`MxGateway.Tests`; the actual tree is `src/ZB.MOM.WW.MxGateway.*`. Misleads path-based work.
|
||||
- ⚪ **7.5 Dead `MapSqlException` helper** — `Grpc/GalaxyRepositoryGrpcService.cs:350-360`, IDE0051-suppressed, kept for a hypothetical direct-SQL path that doesn't exist.
|
||||
- ✅ **7.4 CLAUDE.md project-name drift — RESOLVED** (verified 2026-08-07): CLAUDE.md now uses the actual `src/ZB.MOM.WW.MxGateway.*` project paths throughout.
|
||||
- ✅ **7.5 Dead `MapSqlException` helper — OBSOLETE** (noted 2026-08-07): `Grpc/GalaxyRepositoryGrpcService.cs` was deleted wholesale when the gateway adopted the shared `ZB.MOM.WW.GalaxyRepository` 0.2.0 package (2026-06-25, commit `8e196a7`), taking the dead helper with it.
|
||||
- **7.6 Accepted code-review gaps (`Won't Fix`, by design):**
|
||||
- `Client.Python-012` — `Session.invoke_raw` deliberately skips `ensure_mxaccess_success`, so an embedded MXAccess HRESULT failure surfaces silently (raw-parity inspection). `code-reviews/Client.Python/findings.md:290`.
|
||||
- `Contracts-003` — closed as not-a-defect. `code-reviews/Contracts/findings.md`.
|
||||
|
||||
Reference in New Issue
Block a user