Tracking IDs are review bookkeeping that rots in shipped source; the rationale they tagged is kept. Comment-only change (CommentChecker 458 -> 16 false positives).
IPC-08 closed with WorkerCancel sent from the gateway's own timeout arm only.
Its rationale -- the gateway has stopped waiting but the worker has not stopped
working, so a queued-but-not-started command should be dropped instead of
occupying the single STA behind a caller that is already gone -- applies exactly
as well to caller cancellation, which is the common abandonment path: gRPC
clients routinely set per-call deadlines shorter than the gateway's
CommandTimeout, and a client disconnect cancels the RPC's token. Both landed in
arms that removed the pending entry and returned silently, leaving the
correlation to execute in its turn. Under a retry storm against a slow provider
the STA works through dead commands while live callers queue behind them --
precisely the backlog the cancel exists to shorten.
- TrySendCancelForTimedOutCommand is now TrySendCancelForAbandonedCommand and
takes the reason string, since it serves both arms.
- Both caller-cancellation arms route through AbandonCommandOnCallerCancellation,
which sends the cancel under the same `removed` ownership gate as the timeout
arm (so a reply that won the race is never chased) and skips it once _stopCts
has fired, where the outbound channel is being completed and every pending
command is failed anyway.
- New WorkerClientErrorCode.CallerCancelled replaces the GatewayShutdown label
those arms used, so metrics and logs stop recording a shutdown that did not
happen. It maps to StatusCode.Cancelled, the same status GatewayShutdown maps
to, so nothing changes for clients.
Docs: docs/Grpc.md mapping line, gateway.md cancellation rules,
docs/GatewayProcessDesign.md pending-command section.
Tests: InvokeAsync_WhenCallerCancels_SendsWorkerCancelForThatCorrelation (twin
of the timeout test, driven by the caller token; verified failing without the
send) and InvokeAsync_WhenClientStopsUnderInFlightCommand_DoesNotSendWorkerCancel.
Targeted re-read of the 203-file fca978d sweep (docs(src): add missing
XML docs and strip tracking-ID comments): a mechanical pre-pass narrowed
1,383 deletions to 68 files / 816 residual prose lines, and a judged
review of every one found 17 collateral deletions across 10 files —
rationale prose deleted alongside resolved markers with no equivalent
surviving anywhere in the tree. Restored (markers stay stripped, per the
sweep's intent):
- SessionManager: the three metrics-accounting invariants (kill-path
gauge decrement safety, shutdown kill-fallback registry guard vs
double bookkeeping, SessionClosed-not-SessionRemoved on failed close)
- SessionManagerTests: the matching accounting expectation note and the
reason-string propagation pins (test summary + FakeWorkerClient.LastKillReason)
- MxAccessGatewayService.AcknowledgeAlarm: the routing remarks (GUID vs
Provider!Group.Tag vs InvalidRequest; session-less via IGatewayAlarmService)
— inheritdoc resolves to nothing (proto-generated base is undocumented)
- HubTokenService.Validate: why the hollow-token guard exists
(non-empty AuthenticationType alone satisfies IsAuthenticated)
- DashboardSessionAdminService: why both broad catches exist (keep raw
teardown exceptions out of Blazor's error boundary), Close + Kill paths
- WorkerPipeSession.RunAsync: why the factory result throws instead of
NREing (unambiguous failure; finally-block Dispose can't no-op)
- LmxSubtagAlarmSource: Advise idempotency; Write is always unsecured
(user id 0), never WriteSecured semantics
- WnWrapAlarmConsumer: the v1-prefix path is what WIN-911-style code uses
- DashboardSnapshotPublisherTests: what the 10ms slack absorbs
(Task.Delay's coarse Windows timer quantum)
- DashboardBrowseAndAlarmModelTests: why the label text is pinned, not
just the CSS class
Everything else flagged verified benign: inheritdoc replacements resolve
to equal-or-richer interface docs, or the substance survives relocated.
NonWindows slnx 0W/0E; touched gateway test classes 65/65.
Adds a dashboard event-visibility tag to ApiKeyConstraints, riding in the existing
constraints JSON blob so no auth-store schema migration is needed (design
docs/plans/2026-07-10-dashboard-session-acl-tst15.md sections 3/3.1, open call
settled per its own recommendation). The tag is visibility-only: no read, write,
browse, or subscribe path consults it, and HasRead/HasWriteConstraints ignore it.
GatewaySession gains an immutable, ordinal-ignore-case Tags set stamped at
construction from the owning API key, forwarded by MxAccessGatewayService.OpenSession
from the resolved ApiKeyIdentity — never from the wire request, so a client cannot
label its own session with another tenant's tag. ISessionManager gains a tag-carrying
OpenSessionAsync overload whose default implementation forwards to the tagless one, so
an implementation that does not model tags opens an untagged (least visible) session.
apikey create-key gains --dashboard-tags team-a,team-b (repeatable, trimmed,
de-duplicated; an empty segment is rejected rather than dropped) and list-keys prints
the tags column. No enforcement yet — the EventsHub ACL that consumes the tag is a
later change.
The remediation reviews approved every task but left a tail of small notes.
This lands the gateway-side half of them.
Hardening (behavior changes, all narrow):
- BuildFilteredWriteBulkCommand's unreachable default case failed OPEN: a
fifth bulk-write kind added upstream without a filter case here would have
shipped the DENIED entries to the worker while reporting them denied to the
caller. It now throws UnreachableException.
- SqliteCanonicalAuditStore.ListRecentAsync no longer throws on a row it
cannot date. The retention sweep deliberately preserves such rows (SQLite's
datetime() yields NULL, so the DELETE never matches), which guaranteed the
dashboard's recent-audit view would meet one eventually and lose the whole
page to it. The row is now reported at DateTimeOffset.MinValue with every
other column intact, behind an optional logger.
- The audit drain loop's finally now completes the channel writer alongside
detaching the drain, so a producer that raced past the attached check takes
the write-through branch instead of stranding its event in a buffer nobody
reads until shutdown. TryComplete is idempotent, so StopAsync is unaffected.
Tests:
- MapCommandReply ownership (Assert.Same on the inner reply), mirroring the
existing MapEvent ownership test.
- Redactor key-id length boundary at exactly 64 and 65 characters, pinning
which way it fails. Nothing validates key-id length at creation, so
docs/Diagnostics.md's "which no issued key id does" is now stated as the
heuristic it is.
- ApiKeyFailureLimiter.Reset with a PartitionResolution whose partition was
evicted between the Check and the Reset: inert, and clears nobody else's
block.
- Constraint-cache concurrency stress: the cap is enforced by the inserting
thread, so overshoot must be transient and proportional to the in-flight
inserters, and the cache must settle at or under the cap.
- ListRecentAsync against a raw-SQL undateable row.
Comment/doc accuracy:
- EventsHubViewerRegistry.ReleaseConnection records that it relies on
SignalR's default sequential per-connection dispatch
(MaximumParallelInvocationsPerClient = 1).
- A PERF(followup) note on Invoke's double session resolve and why removing it
needs a SessionManager overload.
- SessionEventDistributor: the volatile-field comment named the pump as the
lock-free reader, but the pump's single capture point is inside _replayLock;
the genuinely lock-free reader is SubscriberCount. OnSubscriberOverflow's
"cannot be observed here" now excepts the DisposeAsync abandon path. The
churn test names its ConcurrentDictionary bucket-order assumption and that a
violation surfaces as a read timeout, not a silent pass.
- The two "restores the sequential drain's behavior" claims (SessionManager,
docs/Sessions.md) were wrong: the sequential drain leaked too, because
KillWorkerAsync's entry ThrowIfCancellationRequested aborted the whole loop
on the first session for zero kills. Reworded to "fixes a leak the
sequential drain also had", with the sweep-bound/shutdown-unbound
ParallelOptions asymmetry explained.
- ISessionManager.ShutdownAsync's token doc: it degrades the drain to a kill
sweep rather than cancelling it, with the bounded overrun stated.
SessionShutdownHostedService.StopAsync records that its cancellation-logging
branch is now unreachable.
Invoke deep-cloned the whole MxCommandRequest — including its command
payload, potentially a large bulk-write graph — only to overwrite the
cloned command with commandToInvoke and discard it. MapCommand then did
the one clone actually needed. Net cost: a full wasted command deep-clone
per Invoke, worst for exactly the bulk writes that are largest.
Adds a MapCommand(MxCommand) overload (MapCommand reads nothing else off
the request) and has Invoke pass commandToInvoke directly; the request
overload delegates so other callers are untouched.
The remaining clone inside MapCommand stays and is now documented as
required rather than incidental: commandToInvoke may be the gRPC-owned
request.Command, and the caller reads it again after dispatch via
TrackCommandReply, so ownership transfer (à la GWC-07) is not safe here.
That clone is what keeps WorkerClient.CreateCommandEnvelope's no-aliasing
invariant true.
Tests: MxAccessGrpcMapperTests.MapCommandFromCommandClonesPayload
(mutating the input leaves the mapped command untouched; both overloads
produce equal results under a fixed TimeProvider).
Behavior-preserving allocation cuts on the per-event/per-command path:
- GWC-06: StreamEvents send-timing uses Stopwatch.GetTimestamp() +
GetElapsedTime() instead of a per-event Stopwatch allocation (same measured
span).
- GWC-07/IPC-05 (event): MapEvent transfers ownership of the inner MxEvent
instead of .Clone()-ing it. Safe: WorkerEvent is parsed fresh per pipe frame
with the distributor pump as its single consumer (GWC-01), MapEvent runs once
before fan-out, and every downstream consumer (subscribers, replay ring)
only READS the event (WorkerSequence is stamped upstream; verified no
post-mapping mutation). Comment documents the invariant + restore-clone
caveat if a second consumer is added.
- IPC-05 (command): CreateCommandEnvelope no longer re-clones; MapCommand
already isolated the graph from the caller-owned gRPC message.
- GWC-15: grpc_stream_queue.depth converts from a per-event push counter to an
ObservableGauge summing registered channel sources at scrape time only
(name/semantics unchanged); removes all per-event .Count/lock work.
Kept every load-bearing isolation clone (MapCommand, Invoke, bulk filters,
MapCommandReply). Server build clean (0 warnings); EventStream/Metrics/
Distributor/Mapper tests 62/62 (incl. formerly-flaky queue-depth tests, now
green under the lazy gauge, + 2 new MapEvent ownership tests). Docs: Metrics.md,
Grpc.md.
Claude-Session: https://claude.ai/code/session_01DMXXvNuPekkkrTEyPNxEkW
Proto foundation + gateway-side of the size/backpressure pass:
- IPC-02: add GatewayHello.max_frame_bytes (regen Generated/); gateway sends
its negotiated worker-frame max in the handshake so the worker can adopt it
instead of a hard-coded default. Worker read-half lands separately.
- IPC-03: give the pipe frame max envelope-overhead headroom above the public
gRPC cap (WorkerFrameProtocolOptions.EnvelopeOverheadReserveBytes = 64 KiB;
default Worker.MaxMessageBytes bumped to 16 MiB + reserve), cross-validate the
headroom at startup, and pre-check command envelope size in WorkerClient so an
oversized command fails only that correlation (ResourceExhausted) instead of
faulting the whole session.
- IPC-04: reject DrainEvents max_events above a public ceiling in the request
validator (worker per-reply cap lands with the worker half).
Docs: GatewayConfiguration, WorkerFrameProtocol, gateway.md.
Tests: headroom validation, DrainEvents bound, oversized-command per-command
failure (pipe-harness, verified on windev).
Interlocking changes across the gateway server (shared GatewaySession.cs /
SessionManager.cs), committed together:
- GWC-01 (Critical): alarm monitor now attaches as an internal
(non-counted) distributor subscriber instead of a second raw drain of the
single worker event channel; WorkerClient._events -> SingleReader with a
claimed-once guard so a future dual-consumer regression throws loudly.
- GWC-02 (High): faulted sessions are swept in CloseExpiredLeasesAsync
(IsFaultedReapable + FaultedReason); new FaultedGraceSeconds (default 0).
- GWC-03 (High): configurable MaxSparseArrayLength (default 1_000_000)
enforced before allocation.
- TST-02 (High, security): StreamEvents attach now enforces the opening key
id -> PermissionDenied on owner mismatch.
- TST-12 (Medium): CLAUDE.md retention-defaults sentence corrected.
Verified: NonWindows build clean; targeted tests 135/135 on macOS, plus
WorkerClientTests 18/18 on the Windows host.
Add a nullable string? OwnerKeyId property to GatewaySession that captures
the API key identifier (KeyId) of the authenticated caller that opened the
session. Wire it through ISessionManager.OpenSessionAsync → SessionManager
→ GatewaySession constructor. The gRPC service passes identityAccessor
.Current?.KeyId; internal callers (GatewayAlarmMonitor, DashboardLiveDataService)
pass null. Covers the positive and null cases with two new TDD-first tests.
Adds missing <summary>/<param> XML docs across 99 server, worker, and test
files so CommentChecker reports zero issues (TreatWarningsAsErrors needs the
analyzer clean). Bundles in WIP dashboard work: NavSection extraction,
MainLayout/site.css/js styling alignment, and DashboardOptions/Auth tweaks.
Rename across every client surface using each language's idiomatic convention:
* .NET clients/dotnet/MxGateway.Client[.Cli|.Tests]/
-> clients/dotnet/ZB.MOM.WW.MxGateway.Client[.Cli|.Tests]/
namespaces -> ZB.MOM.WW.MxGateway.Client[.Cli|.Tests]
contracts ProjectReference repointed to ZB.MOM.WW.MxGateway.Contracts
sln migrated to slnx (dotnet sln migrate)
* Python src/mxgateway -> src/zb_mom_ww_mxgateway
src/mxgateway_cli -> src/zb_mom_ww_mxgateway_cli
distribution: mxaccess-gateway-client -> zb-mom-ww-mxaccess-gateway-client
* Rust crate: mxgateway-client -> zb-mom-ww-mxgateway-client
build.rs proto path repointed
* Java subprojects: mxgateway-{client,cli} -> zb-mom-ww-mxgateway-{client,cli}
packages com.dohertylan.mxgateway -> com.zb.mom.ww.mxgateway
group com.dohertylan.mxgateway -> com.zb.mom.ww.mxgateway
rootProject mxaccessgw-java -> zb-mom-ww-mxaccessgw-java
* Go generate-proto.ps1 proto path repointed; module path and
package mxgateway kept (Go convention).
* proto-inputs.json: generatedOutputs.python updated to new package path.
* scripts/run-client-e2e-tests.ps1: Java CLI install path + gradle task
updated to zb-mom-ww-mxgateway-cli.
CLI binary names (mxgw, mxgw-py, mxgw-go, mxgateway-cli) and wire-level
identifiers (MXGATEWAY_* env vars, the mxgw_<id>_<secret> API key
prefix, protobuf package names like mxaccess_gateway.v1, all MXAccess
references) intentionally NOT renamed.
Fix pre-existing alarms-over-gateway breaks unblocked by the rename:
* mxaccess_gateway.proto: add missing public message QueryActiveAlarmsRequest
{session_id, client_correlation_id, alarm_filter_prefix} and missing
rpc QueryActiveAlarms(QueryActiveAlarmsRequest) returns
(stream ActiveAlarmSnapshot). All four typed clients referenced
these but they were absent from the proto.
* MxAccessGatewayService.QueryActiveAlarms: implement the new RPC on
the server, streaming from IGatewayAlarmService.CurrentAlarms with
optional alarm_filter_prefix filter.
* clients/dotnet/.../DiscoverHierarchyOptions.cs: add the hand-written
.NET POCO that wraps DiscoverHierarchyRequest (referenced by
GalaxyRepositoryClient.DiscoverHierarchyAsync but never authored).
* Drop retired session_id field references from
AcknowledgeAlarmRequest/AcknowledgeAlarmReply test fixtures across
.NET, Rust, Go, and Python clients.
* Rust integration test: add the missing stream_alarms impl on the
fake MxAccessGateway server (the trait gained the method, fake
didn't).
* Rust CLI test: bump expected gatewayProtocolVersion 2 -> 3.
Regenerated artifacts updated in this commit:
* src/ZB.MOM.WW.MxGateway.Contracts/Generated/{MxaccessGateway,MxaccessGatewayGrpc}.cs
* clients/python/src/zb_mom_ww_mxgateway/generated/*_pb2{,_grpc}.py
* clients/go/internal/generated/*.pb.go
(C# regenerated by Grpc.Tools on contracts build; Python and Go via
their generate-proto.ps1 scripts; Rust regenerates from .proto via
tonic-build at compile time so no checked-in artefact.)
Verification: 472 server tests, 275 worker tests (9 dev-rig skipped),
18 integration tests (live MxAccess + LDAP + Galaxy), 57 .NET client
tests, 32 Rust workspace tests, 39 Python tests, all Go packages, and
gradle build for Java all pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Apply the ZB.MOM.WW. prefix to all gateway-side projects, folders,
.csproj/.sln contents, C# namespaces, using directives, generated proto
C# (csharp_namespace + checked-in generated files), InternalsVisibleTo
attributes, project-name string literals (LoadProject, .sln lookups,
worker exe paths, staticwebassets manifest), and the install/script/doc
references that point at any of the above. Migrate the solution from
.sln to .slnx via `dotnet sln migrate` and delete the old file.
External-runtime identifiers are intentionally NOT prefixed so external
configuration keeps working:
- GatewayMetrics.cs MeterName ("MxGateway.Server")
- DashboardAuthenticationDefaults Scheme/Policy ("MxGateway.Dashboard")
- GatewayRequestLoggingMiddleware logger category ("MxGateway.Request")
- StaRuntime thread name ("MxGateway.Worker.STA")
- appsettings.json root section "MxGateway" + env-var prefix
MxGateway__... and secret-name MxGateway:ApiKeyPepper
- C:\ProgramData\MxGateway\ data dir paths
Also fixes two tests that were not rename-related but became visible
while validating the rename:
- WorkerLiveMxAccessSmokeTests.ShutDownAsync: cancellation that the
gateway service correctly maps to RpcException(Cancelled) per gRPC
convention was being misclassified as a stream fault. Added a sibling
catch on RpcException with StatusCode.Cancelled.
- IntegrationTestEnvironment.ResolveRepositoryRoot: extracted IsRepositoryRoot
and made it accept either a .git marker OR a .sln/.slnx next to src/
so the worker-exe walker works in non-git working copies.
clients/proto/proto-inputs.json's protoRoot updated to point at
src/ZB.MOM.WW.MxGateway.Contracts/Protos.
Verified by `dotnet build` and a full `dotnet test` of the .slnx with
MXGATEWAY_RUN_LIVE_{MXACCESS,LDAP,GALAXY}_TESTS=1:
Tests: 472/472 pass
Worker.Tests: 280/280 pass (4 dev-rig [Fact(Skip=...)] skipped)
IntegrationTests: 18/18 pass
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Joseph Doherty