diff --git a/archreview/2026-07-12/remediation/00-tracking.md b/archreview/2026-07-12/remediation/00-tracking.md index 8c588d8..72d76f1 100644 --- a/archreview/2026-07-12/remediation/00-tracking.md +++ b/archreview/2026-07-12/remediation/00-tracking.md @@ -174,5 +174,5 @@ Sequence these together rather than piecemeal — several are one change set spa | 2026-08-07 | **GWC-28, GWC-29, GWC-30, TST-28 → `Done`** (branch `fix/gwc-28-29-30-polish`). GWC-28: `WorkerClient.WriteLoopAsync` now stamps `envelope.Sequence = unchecked(++_nextSequence)` immediately before `_writer.WriteAsync`, and `CreateEnvelope` leaves it unset; `_nextSequence` dropped from `long` + `Interlocked` to a plain `ulong` touched only by the write loop (the channel's single consumer, `SingleReader = true`), so wire order and sequence order are the same thing by construction. Mirrors the worker's WRK-04 stamping, which the gateway half had never received; `gateway.md`'s envelope-sequence rule now states that both sides stamp at write inside their single write path and that inbound enforcement (still open, old **GWC-10**) would rely on it. New `WorkerClientTests.ConcurrentInvokesEmitStrictlyIncreasingSequencesOnTheWire` (32 parallel invokes, sequences asserted strictly increasing in wire order) failed 3/3 pre-fix. GWC-29: added `MxAccessGrpcMapper.MapCommand(MxCommand)`; `Invoke` no longer deep-clones the whole `MxCommandRequest` just to overwrite and discard its command. The one clone inside `MapCommand` stays and is documented as required — `commandToInvoke` may be the gRPC-owned `request.Command` and is read again after dispatch by `TrackCommandReply`, so it is what keeps `CreateCommandEnvelope`'s no-aliasing invariant true. New `MxAccessGrpcMapperTests.MapCommandFromCommandClonesPayload` (isolation + both overloads equal under a `FakeTimeProvider`). GWC-30: `WorkerFrameReader` reuses a per-instance `_lengthPrefix` scratch buffer instead of allocating 4 bytes per frame, with a class remark that `ReadAsync` is not reentrant (single read loop per `WorkerClient`; handshake reads complete before the loop starts); guarded by new `WorkerFrameProtocolTests.ReadAsync_WithMultipleFramesOnOneReader_ParsesEveryFrame` (5 frames, varying payload lengths, one reader). TST-28: new `[Theory] WorkerClientTests.StartAsync_SendsGatewayHelloWithConfiguredMaxFrameBytes` over the default and a 2 MiB override via `FakeWorkerHarness.CreateConnectedPairAsync(maxMessageBytes:)` — test-only, and the mutation check (hard-code `MaxFrameBytes = 0`) failed both cases before being reverted. Verification: `NonWindows.slnx` 0 warnings/0 errors; `WorkerClientTests` 25 passed, `WorkerFrameProtocolTests` 11 passed, `MxAccessGrpcMapperTests` 6 passed, `MxAccessGatewayService*` 29 passed, full gateway suite 844 passed / 0 failed (`TMPDIR=/tmp` on macOS). | | 2026-08-07 | **WRK-21 + WRK-28 + WRK-23 + IPC-30 → `Done`** (branch `fix/wrk-21-drain-cluster`, commits `33ba612` + test-fixture follow-ups `7c2eaf0`/`a256560`). WRK-21: `MxAccessEventQueue` gains a byte-budgeted `Drain(maxEvents, maxTotalBytes)` returning the new `WorkerEventDrainResult`, sizing inside the queue lock so an event that will not fit is never dequeued; `CreateDrainEventsReply` budgets against the negotiated frame max less a 64 KiB wrapper reserve and reports truncation through the existing `DiagnosticMessage` (no proto change), satisfying IPC-23 R1–R3; both reply-write seams (`HandleControlCommandAsync`, `ProcessCommandAsync`) now catch `MessageTooLarge` and answer the correlation with an `InvalidRequest` reply instead of unwinding/faulting the session. WRK-28: the 10,000 ceiling moved to `GatewayContractInfo.MaxDrainEventsPerCommand`, referenced by the gateway validator and the worker clamp (C# const, no `.proto` change). WRK-23: `WorkerFrameWriter` peek-stamps then commits `Sequence` only immediately before the stream write, so rejections leave no wire gap. IPC-30: an oversized event frame stays session-fatal but writes a `PROTOCOL_VIOLATION` `WorkerFault` with `command_method = EventDrain` naming family/handles/sequence/sizes (never the value) before exiting. Docs same commit: `MxAccessWorkerInstanceDesign.md`, `WorkerFrameProtocol.md`, `gateway.md`. **IPC-23 → `In progress`** — mechanics landed here; the proto-comment/doc wave (and its regen fan-out) is still pending and must not be folded into this branch. **Evidence** — macOS: `dotnet build src/ZB.MOM.WW.MxGateway.NonWindows.slnx` 0 warnings/0 errors, `dotnet test …MxGateway.Tests --filter FullyQualifiedName~MxAccessGrpcRequestValidator` 4/4 passed. windev (`scripts/ci/windev-worker-ci.ps1 -Sha a2565604 -Mode test`, 2026-08-07 06:47): x86 Worker build 0 warnings/0 errors, `Worker.Tests` **367 passed / 0 failed / 11 skipped** (skips are the live-MXAccess/dev-rig opt-ins), script exit 0. **Harness note:** `PipePair` runs both pipe ends in one process with blocking `FlushFileBuffers` per frame, so it wedges on multi-MB frames or after ~85 large round trips; the pipe tests therefore negotiate a 128 KiB frame maximum and walk 1,000 events to empty, while the full 10,000-event drain-to-empty no-loss proof runs at the queue layer (`MxAccessEventQueueTests`). | | 2026-08-07 | Code-review follow-ups on the same branch (commit `6bc3f9b`). (1) **Important** — `ResolveDrainReplyByteBudget` was a step, not a floor: just above the 64 KiB reserve the budget collapsed to a few bytes (exactly 1024 at the validator floor `MaxMessageBytes = 1024 + 64 KiB`), so a byte-heavy `DrainEvents` truncated on every call and the drain-until-empty loop never terminated. Now `Math.Max(frameMax - reserve, frameMax / 2)` — monotonic, never below half the frame max. New test `WorkerPipeSessionTests.DrainEvents_AtValidatorFloorFrameMax_MakesProgressAndTerminates` drives a byte-heavy queue at the exact validator floor and asserts drain-to-empty with no head reported oversized. (2) **Hardening** — the reply-too-large fallback write is now itself size-guarded (`WriteReplyTooLargeFallbackAsync`, shared by the control and STA reply seams) so a pathologically tiny negotiated max below the gateway floor (the WRK-24 gap) cannot make even the backstop session-fatal; log-and-swallow, comment points at WRK-24. (3) **Comment** — corrected the `RepeatedFieldOverheadBytes` docs: `WorkerEvent.CalculateSize()` already includes the event's tag+length, so the 8 bytes is pure slack, not wrapper compensation. **Evidence** — macOS build 0/0, validator filter 4/4. windev (`windev-worker-ci.ps1 -Sha 6bc3f9b -Mode test`, 07:07): x86 Worker build 0/0, `Worker.Tests` **368 passed / 0 failed / 11 skipped**, script exit 0. (An earlier run of the same SHA flaked on the pre-existing `RunAsync_WhenStaActivityIsStale_WritesWatchdogFault` — a 5 s CTS timeout under first-run load, untouched by this change; it passed on the clean re-run and in both prior full runs.) | -| 2026-08-07 | **Worker-seam batch → `Done`: WRK-22 (mechanics for IPC-26), WRK-24, WRK-25, WRK-27** (branch `fix/wrk-22-25-seam`). **WRK-22/IPC-26**: `WorkerFrameWriter.PendingFrame` gained a `Claimed` field; a `WriteAsync`/`WriteBatchAsync` cancelled while waiting for the write lock tombstones its still-unclaimed frame (`TrySetCanceled` under `_gate`) and `DequeueNext` skips cancelled frames and marks the one it returns `Claimed`, so a cancelled write never reaches the wire — except the documented, by-design residual where a lock-holder claimed the frame first (mid-write, cannot be recalled; caller still observes cancellation). **WRK-25**: new `WriteBatchAsync(IReadOnlyList, priority, ct)` enqueues a whole batch under one `_gate` acquisition, takes the lock once, drains, then observes every completion (surfacing the first per-frame rejection); `RunEventDrainLoopAsync` now submits the drained event batch through it, so a burst of N events costs one flush not N — the WRK-12 coalescing now engages on the event hot path. IPC-30's oversized-event structured fault is preserved (`FindOversizedEvent` maps the batch rejection back to the offending event). **WRK-24**: `WorkerFrameProtocolOptions.MinNegotiableFrameBytes = 1024` (matches `GatewayOptionsValidator.MinimumMaxMessageBytes`); `AdoptNegotiatedMaxMessageBytes` now faults a below-floor negotiated value at the handshake, closing the [1024, 256 MiB] accepted range. **WRK-27**: alarm poll runs outside the dispatcher, so `MxAccessStaSession` sets a `volatile staAlarmPollInProgress` around the `PollOnce` COM call and surfaces it on the new `WorkerRuntimeHeartbeatSnapshot.StaCallInProgress`; `ReportWatchdogFaultIfNeededAsync` honors it alongside `CurrentCommandCorrelationId`, so a healthy-but-slow poll gets grace-to-ceiling suppression (not the 15 s grace) but still faults past the 75 s ceiling. Docs same commit: `docs/WorkerFrameProtocol.md` (accepted-range paragraph, flush-coalescing sentence flipped to coalesced-on-drain, cancellation-tombstone contract replacing the WRK-26 "pending" placeholder), `docs/MxAccessWorkerInstanceDesign.md` (watchdog alarm-poll paragraph). New tests: `WorkerFrameProtocolTests.{WriteAsync_CancelledWhileWaitingForLock_FrameIsNeverWritten, WriteAsync_CancelledEventFrame_DoesNotTrailShutdownAck, WriteBatchAsync_FlushesOnceAndPreservesOrder, WriteBatchAsync_ControlFrameQueuedDuringBatch_JumpsRemainingEvents, AdoptNegotiatedMaxMessageBytes_BelowFloor_ThrowsInvalidConfiguration}`, `WorkerPipeSessionTests.{Watchdog_StaCallInProgress_SuppressedUntilCeiling, EventBurst_DrainLoopCoalescesFlushes, Handshake_GatewayHelloWithTinyMaxFrameBytes_FaultsAtHandshake}`, `MxAccessStaSessionTests.CaptureHeartbeat_DuringAlarmPoll_ReportsStaCallInProgress` (+ `FakeRuntimeSession.EnqueueEvents` bulk helper, `staCallInProgress` snapshot ctor param). **IPC-26 → `Done`** (mechanics owned here). **Evidence** — macOS: `dotnet build src/ZB.MOM.WW.MxGateway.NonWindows.slnx` . windev: . | +| 2026-08-07 | **Worker-seam batch → `Done`: WRK-22 (mechanics for IPC-26), WRK-24, WRK-25, WRK-27** (branch `fix/wrk-22-25-seam`). **WRK-22/IPC-26**: `WorkerFrameWriter.PendingFrame` gained a `Claimed` field; a `WriteAsync`/`WriteBatchAsync` cancelled while waiting for the write lock tombstones its still-unclaimed frame (`TrySetCanceled` under `_gate`) and `DequeueNext` skips cancelled frames and marks the one it returns `Claimed`, so a cancelled write never reaches the wire — except the documented, by-design residual where a lock-holder claimed the frame first (mid-write, cannot be recalled; caller still observes cancellation). **WRK-25**: new `WriteBatchAsync(IReadOnlyList, priority, ct)` enqueues a whole batch under one `_gate` acquisition, takes the lock once, drains, then observes every completion (surfacing the first per-frame rejection); `RunEventDrainLoopAsync` now submits the drained event batch through it, so a burst of N events costs one flush not N — the WRK-12 coalescing now engages on the event hot path. IPC-30's oversized-event structured fault is preserved (`FindOversizedEvent` maps the batch rejection back to the offending event). **WRK-24**: `WorkerFrameProtocolOptions.MinNegotiableFrameBytes = 1024` (matches `GatewayOptionsValidator.MinimumMaxMessageBytes`); `AdoptNegotiatedMaxMessageBytes` now faults a below-floor negotiated value at the handshake, closing the [1024, 256 MiB] accepted range. **WRK-27**: alarm poll runs outside the dispatcher, so `MxAccessStaSession` sets a `volatile staAlarmPollInProgress` around the `PollOnce` COM call and surfaces it on the new `WorkerRuntimeHeartbeatSnapshot.StaCallInProgress`; `ReportWatchdogFaultIfNeededAsync` honors it alongside `CurrentCommandCorrelationId`, so a healthy-but-slow poll gets grace-to-ceiling suppression (not the 15 s grace) but still faults past the 75 s ceiling. Docs same commit: `docs/WorkerFrameProtocol.md` (accepted-range paragraph, flush-coalescing sentence flipped to coalesced-on-drain, cancellation-tombstone contract replacing the WRK-26 "pending" placeholder), `docs/MxAccessWorkerInstanceDesign.md` (watchdog alarm-poll paragraph). New tests: `WorkerFrameProtocolTests.{WriteAsync_CancelledWhileWaitingForLock_FrameIsNeverWritten, WriteAsync_CancelledEventFrame_DoesNotTrailShutdownAck, WriteBatchAsync_FlushesOnceAndPreservesOrder, WriteBatchAsync_ControlFrameQueuedDuringBatch_JumpsRemainingEvents, AdoptNegotiatedMaxMessageBytes_BelowFloor_ThrowsInvalidConfiguration}`, `WorkerPipeSessionTests.{Watchdog_StaCallInProgress_SuppressedUntilCeiling, EventBurst_DrainLoopCoalescesFlushes, Handshake_GatewayHelloWithTinyMaxFrameBytes_FaultsAtHandshake}`, `MxAccessStaSessionTests.CaptureHeartbeat_DuringAlarmPoll_ReportsStaCallInProgress` (+ `FakeRuntimeSession.EnqueueEvents` bulk helper, `staCallInProgress` snapshot ctor param). **IPC-26 → `Done`** (mechanics owned here). **Evidence** — macOS: `dotnet build src/ZB.MOM.WW.MxGateway.NonWindows.slnx` 0 warnings / 0 errors (worker excluded there; proves the shared/server side is intact). windev (`scripts/ci/windev-worker-ci.ps1 -Sha 8df35cd -Mode test`, 2026-08-07 07:45): x86 Worker build 0 warnings / 0 errors, `Worker.Tests` **377 passed / 0 failed / 11 skipped** (skips are the live-MXAccess/dev-rig opt-ins; +9 over the prior 368 = the nine new tests, all green), script exit 0. | | 2026-08-07 | **P1 doc-drift batch → `Done`: TST-27, WRK-26 (discharges IPC-29), CLI-42, CLI-43, IPC-28** (branch `fix/doc-drift-batch`). Doc-only; no source, proto, or test changes — cross-checked against HEAD in this worktree. **TST-27**: `docs/GatewayConfiguration.md`'s `ShowTagValues` row no longer says "Reserved" — it now states what `false` (default) does (`DashboardEventBroadcaster` blanks tag values from a deep-cloned `MxEvent` before the SignalR events-hub mirror, metadata still renders), the security relevance (the per-session hub ACL, SEC-25 roadmap item 12, still does not exist, so this redaction is the only thing between a low-trust Viewer and other sessions' tag values), and the honest scope limit (the flag does **not** cover `/browse`). **WRK-26** (discharges **IPC-29**): `docs/MxAccessWorkerInstanceDesign.md`'s "Outbound Queues" section rewritten from the stale five-level priority list to the two-class `Control`/`Event` scheduler actually shipped (`WorkerFrameWriter`/`WorkerFrameWritePriority.cs`), with the collapsed-decision rationale recorded, and the overflow paragraph rewritten to the implemented fail-fast (`WorkerFault` category `QueueOverflow` → fault frame written → `RunAsync` unwinds → generic `WorkerExitCode.UnexpectedFailure`, dedicated code still open). `docs/WorkerFrameProtocol.md` gained a new "Write Scheduling And Sequencing" section: the two priority classes, enqueue-then-contend/single-lock-holder-drains-all, write-time peek-stamp-commit sequencing, per-frame-rejection vs. stream-failure semantics, and flush coalescing — stated truthfully as landed (WRK-23's peek-stamp-commit is live at HEAD) or not (the drain loop still awaits each event `WriteAsync` individually, so WRK-25's N-events-one-flush batching has **not** landed and the section says so explicitly). Cancellation is deliberately **not** documented as a firm contract — a one-paragraph placeholder notes it is pending WRK-22, which has not landed (confirmed by reading `WorkerFrameWriter.cs`: no `Claimed`/tombstone machinery exists yet). `gateway.md:328-330` was cross-checked and left unchanged — its sequence prose (both sides stamp at write, per GWC-28) already reads true. **CLI-42**: `clients/rust/README.md` and `docs/ClientPackaging.md`'s Rust section now document the vendored proto layout matching `clients/rust/build.rs` exactly — repo-path-first resolution (`../../src/ZB.MOM.WW.MxGateway.Contracts/Protos`) falling back to `clients/rust/protos/` when the canonical path is absent (published-tarball case), the same-commit refresh rule enforced by `scripts/check-codegen.ps1` Check 3, and why `cargo package`/`cargo publish` run without `--no-verify` (matches `scripts/pack-clients.ps1:190-192`). **CLI-43**: `docs/style-guides/JavaStyleGuide.md` line 8 now says "Target Java 17 (the Ignition 8.3 baseline...)" mirroring the CLI-12 wording, matching the shipped `clients/java/build.gradle` toolchain-17 build. **IPC-28**: `docs/Grpc.md`'s exception-mapping prose gained `CommandTooLarge` → `ResourceExhausted` (verified against the live `switch` in `Grpc/MxAccessGatewayService.cs:950-960`), and the `Invoke` section gained one sentence on the oversized-payload path (`WorkerClient.InvokeAsync` rejects at the enqueue boundary per-correlation, session not faulted — verified against `WorkerClient.cs:220-234`), cross-referencing the headroom rule already documented in `docs/GatewayConfiguration.md:120-129`. Did not touch the DrainEvents-truncation row or the proto/`Generated/` trees — those belong to a parallel codegen task per the handoff note. **Source files cross-read for accuracy** (no edits): `src/ZB.MOM.WW.MxGateway.Worker/Ipc/WorkerFrameWriter.cs`, `.../WorkerFrameWritePriority.cs`, `.../WorkerPipeSession.cs` (confirmed two-class scheduler, WRK-21/23/28/30 landed, WRK-25/WRK-22 not landed), `src/ZB.MOM.WW.MxGateway.Worker/WorkerApplication.cs` (exit-code mapping), `src/ZB.MOM.WW.MxGateway.Worker/MxAccess/MxAccessEventQueue.cs` (overflow fault path), `src/ZB.MOM.WW.MxGateway.Server/Dashboard/Hubs/DashboardEventBroadcaster.cs` + `Configuration/DashboardOptions.cs` + `docs/GatewayDashboardDesign.md:170` (ShowTagValues), `src/ZB.MOM.WW.MxGateway.Server/Grpc/MxAccessGatewayService.cs:940-963` + `Workers/WorkerClient.cs:205-244` + `Workers/WorkerClientErrorCode.cs` (CommandTooLarge mapping), `clients/rust/build.rs`, `clients/rust/Cargo.toml`, `scripts/check-codegen.ps1`, `scripts/pack-clients.ps1` (Rust vendoring), `gateway.md:326-360` (sequence-prose cross-check). Verification (greps, doc-only — no build required): `grep -n 'Reserved' docs/GatewayConfiguration.md` no longer matches the `ShowTagValues` row; `grep -n 'faults' docs/MxAccessWorkerInstanceDesign.md` shows no remaining five-level list; `grep -n 'scheduling' docs/WorkerFrameProtocol.md` finds the new section; `grep -rn 'Java 21' docs/style-guides/` empty; `grep -i vendored docs/ClientPackaging.md clients/rust/README.md` non-empty in both; `grep -n 'CommandTooLarge' docs/Grpc.md` shows the mapping. |