Code review of the CLI-39 branch caught an Important gap: Contracts.csproj
was left at the already-published 0.1.2 while the .NET Client moved to
0.2.0. Invoke-PackDotnet in scripts/pack-clients.ps1 packs and publishes
both ZB.MOM.WW.MxGateway.Contracts and .Client through the same -Publish
loop, and the new collision guard runs every nupkg it finds through
Assert-GiteaPackageNotPublished. Left as-is, the next real .NET publish
would pack Contracts at 0.1.2, the guard would correctly refuse to
republish it, and the loop would abort mid-way with Client (alphabetically
first) possibly already pushed -- the two packages permanently out of
lockstep.
- src/ZB.MOM.WW.MxGateway.Contracts/ZB.MOM.WW.MxGateway.Contracts.csproj:
<Version> 0.1.2 -> 0.2.0, matching the .NET Client (they have always
released together).
- src/Directory.Build.props: corrected a comment that was now stale --
it claimed the repo-wide 0.1.2 default was kept to match the Contracts
package, which is no longer true now that Contracts.csproj overrides it.
The <Version> value itself is unchanged; Server/Worker/Tests staying at
0.1.2 is a separate, not-yet-made decision, out of scope for CLI-39.
- docs/ClientPackaging.md: Contracts.csproj added as a fifth manifest in
the Versioning section, with the near-miss recorded.
Also hardened scripts/pack-clients.ps1 per the same review: the Python
(pyproject.toml) and Rust (Cargo.toml) version-extraction regexes now
scope to the [project]/[package] section header instead of matching the
first "version = ..." line anywhere in the file (Cargo.toml has an
identical second one under [workspace.package] -- matching whichever came
first was luck of ordering, not correctness). One-line comment added on
the nuget filename-parse regex.
Verified live against the real Gitea registry: Contracts and Client both
still refuse at 0.1.2 and both now pass at 0.2.0, including running the
actual Invoke-PackDotnet filename-parse-then-guard logic against two
freshly packed real .nupkg files. dotnet build of Contracts.csproj and the
client slnx both clean. No publish performed.
WRK-22/IPC-26: tombstone a WriteAsync/WriteBatchAsync cancelled while
waiting for the write lock (PendingFrame.Claimed under _gate; DequeueNext
skips cancelled, claims the frame it returns) so a cancelled write never
reaches the wire unless already claimed mid-write (documented residual).
WRK-25: add WriteBatchAsync; RunEventDrainLoopAsync submits the drained
event batch through it, so a burst of N events costs one flush not N.
IPC-30 oversized-event structured fault preserved via FindOversizedEvent.
WRK-24: reject a below-1024 negotiated frame maximum at the handshake
(MinNegotiableFrameBytes, matching GatewayOptionsValidator floor).
WRK-27: alarm poll advertises StaCallInProgress on the heartbeat snapshot
so the watchdog suppresses to the ceiling, not the grace.
Docs (WorkerFrameProtocol.md, MxAccessWorkerInstanceDesign.md) and the
2026-07-12 remediation registers/change-log updated in the same commit.
ClientProtoInputTests.Descriptor_ContainsEveryContractMessageAndField only
compared messages and fields, and only enumerated the gateway/worker
descriptors, so a new enum value, a new RPC, or any galaxy_repository.proto-only
change would not redden the test even though it is documented as the primary
protoc-free CI gate.
Rename to Descriptor_ContainsEveryContractSymbol and extend the reflection walk
on both sides (published protoset and in-process contract) to also collect
enums/enum values ({enumFullName}, {enumFullName}/{valueName}) and
services/methods ({serviceFullName}, {serviceFullName}/{methodName}), and add
GalaxyRepositoryReflection.Descriptor to the enumerated files. The comparison
stays a flat, order-insensitive string-set diff with no protoc dependency.
Update docs/ClientProtoGeneration.md and docs/Contracts.md prose from
"message or field" to the full symbol coverage.
Red-path proof: pointed the test at the pre-IPC-01 stale protoset and confirmed
it failed naming max_frame_bytes, several MxCommandKind/AlarmProviderMode enum
values, MxAccessGateway/StreamAlarms and GalaxyRepository/BrowseChildren, and
the galaxy_repository.v1.* surface; restored the real path and re-ran green.
Flips IPC-27 to Done in the 2026-07-12 remediation tracker and register.
Proto comments (comment-only, no wire change):
- mxaccess_worker.proto GatewayHello.max_frame_bytes: every worker->gateway frame
must serialize within the negotiated max; reply builders truncate (IPC-23).
- mxaccess_gateway.proto DrainEventsReply: count-cap + byte-cap, drain-until-empty
caller contract (IPC-23).
- mxaccess_gateway.proto ReplayGap.oldest_available_sequence: empty-ring value is
highest-observed+1, oldest-1 resume formula stays valid (GWC-25 deferred amendment).
Regen wave: Contracts/Generated (C# XML doc), rust vendored protos (byte-copy),
Go bindings (worker binding was genuinely stale - lacked MaxFrameBytes entirely),
Python worker _pb2 (real descriptor delta), Java aggregates (javadoc, zero
protobuf-version churn under the pinned toolchain), client descriptor set.
IPC-24: pinned Java toolchain regenerates with no gencode-version churn, so the
unconditional churn-revert step in ci.yml is a fossil - deleted it; git diff is
now a true message-level drift gate for the single-file Java aggregates.
IPC-25: pin protoc-gen-go v1.36.11 / protoc-gen-go-grpc 1.6.2 in the Go generate
script (+ fix a latent pwsh-7 parse bug); add Check 4 to check-codegen.ps1
(regenerate Go+Python bindings, fail on diff, tool-missing fails not skips); add
the pinned-generator installs to the portable CI job.
IPC-32: relabel check-codegen banners 1/4..4/4 (folded into the Check 4 edit).
Docs: ClientProtoGeneration.md, Contracts.md, GatewayTesting.md, build.gradle
checkGeneratedClean caveat. Tracking: IPC-23/24/25/32 -> Done, GWC-25 proto note
resolved, change-log 2026-08-07.
Repo-side half of SEC-36. The appsettings.json plaintext was already discharged
before this branch (HEAD ships the fail-closed ${secret:ldap/mxgateway/bind}
store reference), so the residual leak was the literal value in glauth.md,
docs/GatewayTesting.md, and the historical archreview SEC-06 evidence -- all
scrubbed to <service-account-password> placeholders pointing at the source of
truth scadaproj/infra/glauth/.
- csproj: add <UserSecretsId>mxaccessgw-server</UserSecretsId> (dev channel)
- GatewayOptionsValidator: blank-password message now names both channels
(dev user-secrets, deployed MxGateway__Ldap__ServiceAccountPassword)
- test: assert the message names both channels
- docs: GatewayConfiguration.md (three channels + rotation note), glauth.md
(placeholders + rotation-required + runbook pointer), GatewayTesting.md
- new operator runbook docs/runbooks/SEC-36-ldap-credential-rotation.md
(live rotation + NSSM staging remain operator-pending)
- tracking: SEC-36 -> Done (repo-side) in both registers + change-log
Deviation: kept the ${secret:} reference in appsettings.json rather than
deleting it (spec step 2 assumed the stale plaintext baseline); deleting it
would regress the shipped/documented/tested secret-store channel.
git grep -i for the old value is empty across all tracked files.
Review follow-ups on the WRK-21 cluster.
1. ResolveDrainReplyByteBudget was a step function, not a floor: just above the
64 KiB reserve the budget collapsed to a few bytes (at the validator-permitted
floor MaxMessageBytes = 1024 + 64 KiB it was exactly 1024), too small to move a
byte-heavy event, so DrainEvents truncated on every call and the drain-until-
empty loop never terminated. It now takes the max of (frameMax - reserve) and
frameMax/2, so the budget is monotonic and never below half the frame max. New
test DrainEvents_AtValidatorFloorFrameMax_MakesProgressAndTerminates drives a
byte-heavy queue at the exact validator floor and asserts it drains to empty
with no head ever reported oversized.
2. The reply-too-large fallback write is now itself size-guarded
(WriteReplyTooLargeFallbackAsync, used by both the control and STA reply seams):
at a pathologically tiny negotiated max below the gateway's floor the fallback
could also throw MessageTooLarge and — uncaught — kill the session, defeating the
"no diagnostics command is session-fatal" invariant. It now log-and-swallows;
comment notes WRK-24 adds the negotiated-max lower bound that makes it unreachable.
3. Corrected the RepeatedFieldOverheadBytes doc comments: WorkerEvent.CalculateSize()
already includes the event's tag and length prefix (the same shape the reply's
repeated events field packs), so the 8 bytes is pure slack over an already-
conservative estimate, not compensation for a missing wrapper.
Change-log row for 2026-08-07: what landed for WRK-21/WRK-28/WRK-23/IPC-30, why
IPC-23 stays In progress (proto-comment/doc wave pending), and the verification
evidence — macOS NonWindows build + validator tests, and the documented windev
path (scripts/ci/windev-worker-ci.ps1 -Mode test) at a256560: x86 Worker build
clean, Worker.Tests 367 passed / 0 failed / 11 skipped.
PipePair runs both ends of a duplex pipe in one process with blocking
FlushFileBuffers under every frame write, so it wedges after roughly 85 large
round trips. Drain the full 10,000 byte-heavy events to empty at the queue layer,
where the no-loss property actually lives, and keep the pipe walk at 1,000 events
(29 replies) so it still proves the split end to end. Also give the truncation
test's budget slack: item handle 0 is a proto3 default and is not serialized, so
the probe measurement is a lower bound on the fixture's per-event cost.
Same-commit docs rule (were missed in the prior commit):
- docs/GalaxyRepository.md: SnapshotCachePath now documents the per-OS derived
default and the GalaxyRepositoryOptionsValidator rooting/validity enforcement.
- A2-galaxyrepository-adoption-handoff.md: correct the now-inaccurate NSSM caveat
(SnapshotCachePath override is optional, not required; blank seeds a rooted host
default, no silent no-op) and repoint the option-validation item at the new
GalaxyRepositoryOptionsValidator.
SEC-34 guard confirmed and documented: TryParseKeyId's '_' split cannot truncate a
key id because both — and the only — gateway key-creation paths
(ApiKeyAdminCommandLineParser.IsValidKeyId, DashboardApiKeyManagementService.ValidateKeyId)
restrict key ids to IsAsciiLetterOrDigit || '.' || '-', and key ids are never
library-generated. Added a citing comment; no behavior change.
Test consolidation: moved the three host-start SqlitePath overrides into
TestHostEnvironmentInitializer (per-process temp store, mirroring Secrets__SqlitePath)
so future host-start tests auto-cover.
SEC-33: make rooting host-meaningful and stop shipping foreign-platform literals.
- Delete IsRootedForAnyPlatform; AddIfNotRooted now uses Path.IsPathRooted (current OS).
- Promote AddIfNotRooted/AddIfInvalidPath to shared GatewayConfigPathRules so the new
Galaxy validator reuses them and the two validators cannot drift.
- Remove Authentication:SqlitePath and Galaxy:SnapshotCachePath Windows literals from
appsettings.json; the CommonApplicationData-derived code defaults take over. The
Galaxy default is seeded as a configuration value before AddZbGalaxyRepository
(SnapshotCachePath is init-only, so a PostConfigure mutation cannot compile).
- New GalaxyRepositoryOptionsValidator (ValidateOnStart) enforces a valid, host-rooted
SnapshotCachePath when PersistSnapshot is true.
- Root-cause the stray junk-named auth DB: host start eagerly builds
AuthSqliteConnectionFactory; under the non-rooted Windows literal on macOS SQLite
wrote it relative to the test bin CWD. The three real-host-start tests now pin
SqlitePath to a temp path.
SEC-34: verification cache Invalidate-vs-in-flight-repopulation race closed with a
per-key generation counter (bump-before-evict, snapshot-then-recheck). The expiry
cap (window 2) takes the documented fallback: the library verification identity
carries no ExpiresUtc, so the cache cannot cap at the key's expiry (donor-library ask).
GWC-24 rider: cap MxGateway:Events:QueueCapacity at int.MaxValue/2 so the derived
checked(2 * EventChannelCapacity) in WorkerClient cannot overflow at session creation.
SEC-35 (doc-only): note IsProduction() env-name semantics in GatewayConfiguration.md.
Docs updated same commit (GatewayConfiguration.md, Authentication.md) and tracking
registers/change-log flipped (00-tracking.md, 40-security-dashboard.md).
PipePair has no continuous read pump — the test thread drains the pipe only
while it sits in ReadUntilAsync — so multi-megabyte DrainEvents frames
interleaved with the heartbeat loop wedge both ends inside FlushFileBuffers,
each waiting for the other to read. Negotiate a 128 KiB frame maximum instead:
the 10,000 byte-heavy events still overflow it many times over, so every
assertion (bounded reply, reported truncation, no event loss across repeated
drains, surviving session) is unchanged.
ReadAsync allocated a fresh 4-byte array per inbound frame; the GWC-08
pass pooled the payload buffer but left the prefix. Replaced with a
per-instance scratch field — the reader is single-consumer by
construction (one read loop per WorkerClient, handshake reads complete
before the loop starts), so a per-instance buffer is safe and the
non-reentrancy that makes it safe is now stated on the class. Pooling
four bytes via ArrayPool would cost more than the allocation it saves.
Tests: WorkerFrameProtocolTests.ReadAsync_WithMultipleFramesOnOneReader_
ParsesEveryFrame reads five frames of differing payload length through
one reader, so a stale prefix carried between calls would misparse.
Invoke deep-cloned the whole MxCommandRequest — including its command
payload, potentially a large bulk-write graph — only to overwrite the
cloned command with commandToInvoke and discard it. MapCommand then did
the one clone actually needed. Net cost: a full wasted command deep-clone
per Invoke, worst for exactly the bulk writes that are largest.
Adds a MapCommand(MxCommand) overload (MapCommand reads nothing else off
the request) and has Invoke pass commandToInvoke directly; the request
overload delegates so other callers are untouched.
The remaining clone inside MapCommand stays and is now documented as
required rather than incidental: commandToInvoke may be the gRPC-owned
request.Command, and the caller reads it again after dispatch via
TrackCommandReply, so ownership transfer (à la GWC-07) is not safe here.
That clone is what keeps WorkerClient.CreateCommandEnvelope's no-aliasing
invariant true.
Tests: MxAccessGrpcMapperTests.MapCommandFromCommandClonesPayload
(mutating the input leaves the mapped command untouched; both overloads
produce equal results under a fixed TimeProvider).
CreateEnvelope stamped Sequence with an interlocked increment when the
envelope was built, so two concurrent InvokeAsync callers could take 1
and 2 and then enqueue in the order 2, 1 — non-monotonic on the wire,
breaking gateway.md's "monotonic per sender" contract. Benign today
(neither side validates inbound sequence, old GWC-10 still open) but it
would fault healthy sessions the moment worker-side validation lands.
WriteLoopAsync now stamps immediately before _writer.WriteAsync. It is
the outbound channel's single consumer (SingleReader = true), so wire
order and stamp order are the same thing by construction and
_nextSequence drops to a plain ulong with no interlocking. This mirrors
the worker's WRK-04 fix, which the gateway half never received.
Also adds TST-28: a [Theory] pinning that GatewayHello.MaxFrameBytes
carries the configured worker-frame maximum (default + 2 MiB override).
The adoption half is asserted only in the Windows-only worker suite, so
a regression to sending 0 — "older gateway, use default" to the worker —
would silently downgrade the negotiated IPC-02 limit with every CI test
still green. Mutation-checked (hard-coded 0 fails both cases).
Tests: WorkerClientTests.ConcurrentInvokesEmitStrictlyIncreasingSequences
OnTheWire (32 parallel invokes; failed 3/3 pre-fix) and
.StartAsync_SendsGatewayHelloWithConfiguredMaxFrameBytes.
ReleaseProbe recognised its own reservation by comparing NextProbeAtTicks to
now + _probeIntervalTicks. RecordInto's rearm-on-trip writes that identical
expression, so a concurrent RecordFailure on the same WindowState whose `now`
lands on the claimer's tick — routine at ~1 ms clock resolution under load — was
mistaken for the caller's own claim. The release then stomped the legitimate
fresh re-arm back to the stale previousProbeAtTicks, which is already due, handing
the next arrival a free probe the re-arm had just closed.
WindowState gains a monotonic ProbeVersion bumped by every writer of
NextProbeAtTicks (TryConsumeProbe's claim and RecordInto's re-arm alike).
TryConsumeProbe returns the stamp it set as part of a ProbeClaim; ReleaseProbe
restores the previous value only while the state's version still equals that
stamp, checking and restoring in one lock(state) section and bumping the version
again on restore so no other stale release can match either.
Test: ProbeSlotRestore_DoesNotStompConcurrentRearmAtSameTick, with the clock held
still so the claim and the interleaved failure necessarily share a tick. Making it
deterministic needed a seam — the claim-to-release window is a few nanoseconds and
racing threads do not hit it (an earlier thread-based attempt passed against the
defective guard three runs out of three, and its end state was ordering-dependent
rather than correctness-dependent, so it was dropped rather than shipped as
theatre). The seam is an internal ProbeReleaseInterleaveHook, null in production,
costing one null check on the already-refused path. Verified as a genuine red
against the timestamp guard: Expected ThrottledByPeer, Actual ProbeAdmitted.
Code-review follow-up on fix/gwc-26-27-alarm-attach.
ApplyReconcile's snapshot-derived feed repairs are at-least-once, not
exactly-once: a reconcile reads the worker's current state while the matching
live transition may still be buffered in the monitor's lease, so both broadcast
and the duplicates are indistinguishable on the alarm feed. This pre-dates the
acked-state delta — the Raise/Clear presence repair has always had it, since
nothing serializes a reconcile pass against the in-flight live stream — so
closing it (serialization or timestamp dedup) stays out of scope for a P2 fix.
Documented instead, with the consumer contract stated explicitly (apply
transitions idempotently, never as an increment or toggle):
- ApplyReconcile gains a "Delivery semantics" comment.
- gateway.md softens the "defense in depth" prose to state the semantics.
- docs/Sessions.md carries the same caveat on the alarm-feed description.
- Tracker change-log records it as a known pre-existing characteristic and a
candidate finding for the next review cycle.
Also hoists the ChannelWorkerClient fake — duplicated across the three alarm
test files — into TestSupport/, dropping the usings it took with it.
Two defects found in code review of the limiter rework.
Probe admission was check-then-act across two lock scopes: Check() read
"probe due" under lock(state), released it, then re-acquired to advance
NextProbeAtTicks. A burst of requests arriving together at an interval boundary
could therefore all observe the slot as due and all be admitted, handing the
verifier the very burst the interval exists to bound. The claim is now a single
critical section (TryConsumeProbe). The two layers are still claimed one at a
time — holding two per-state locks at once would need a global lock ordering to
stay deadlock-free — so a slot claimed on the composite partition is compensated
via ReleaseProbe when the aggregate then refuses, which otherwise silently spent
the partition's next slot and pushed the legitimate holder out by a full
interval.
Reset() removed whatever partition the caller resolved to, including the
address's shared fallback partition when the caller's key id had been collapsed
into it by the per-peer cap (or when the token was junk-shaped). That bucket also
carries failures contributed by other key ids from the same address, so one
successful authentication became a reset button for an in-progress spray. Reset
now clears only a partition the caller owns (effectiveKeyId == presented key id);
the shared bucket decays by window expiry instead, and the caller still recovers
through probe admission. The key's aggregate is cleared either way, as designed.
Also applied from the review: closure-free GetOrAdd overload on _partitions, and
a remarks paragraph acknowledging the best-effort O(n) eviction scan under
sustained overflow. Threading the resolved partition key from Check through to
RecordFailure/Reset was declined: Check resolves with mint:false and RecordFailure
with mint:true, and the two can legitimately differ when a concurrent caller fills
the per-peer cap in between — reusing Check's key would record into the wrong
partition and bypass the cap, which is not worth saving one string concat.
Tests (limiter suite 11 -> 14): ProbeAdmission_UnderConcurrentArrivals_
GrantsExactlyOneSlot (200 rounds x 8 barrier-released threads at the boundary),
ProbeAdmission_WhenAggregateRefuses_ReturnsTheClaimedPeerSlot, and
Reset_WithOverCapKeyId_DoesNotClearSharedFallbackPartition. The latter two were
confirmed as genuine reds against the unfixed code; the concurrency test is a
guard — it is deterministically green on the fixed structure but did not
reproduce the original nanosecond-wide window on its own.
Seeding the cache with a live Raise transition raced the first reconcile: when
the reconcile snapshot populated the cache first, the still-buffered live Raise
was applied — and broadcast — after the test's feed subscriber had registered,
so the exactly-one-transition assertion saw two. Seed through a reconcile pass
instead (forced by a provider-mode probe, as the acked step already did) so no
live transition is ever in flight. Verified: 5 consecutive full alarm-monitor
runs green, and the test still fails (timeout) with the ApplyReconcile
acked-delta branch removed.
An empty replay ring reported oldest_available_sequence = 0 even when gap was
true. Clients follow the documented after_worker_sequence = oldest - 1 formula,
so an unsigned client computed ulong.MaxValue: the follow-up resume replayed
nothing, reported no gap, and the live filter dropped every subsequent event —
a silently dead stream in the headline detach-and-resume scenario, reachable on
default config once ReplayRetentionSeconds (300) age-evicts the ring.
GWC-25: SessionEventDistributor.RegisterWithReplay's empty-ring branch now
reports _highestSequenceSeen + 1 — the next sequence that can possibly be
delivered — when gap is true, so oldest - 1 lands exactly on the highest
observed sequence and the resume delivers everything newer. Still 0 when there
is no gap, where the field is meaningless and never emitted. Nothing is lost:
the evicted interval was unrecoverable either way, and the sentinel's job is to
say "re-snapshot".
CLI-35: the Python CLI fed every stream item into MessageToDict, which raised on
the ReplayGap dataclass and aborted the command after consuming the stream. A
new _event_row helper renders a gap as {"replayGap": {...}} — the same camelCase
shape the Rust CLI emits — and leaves proto events on the existing path.
CLI-36: the Go CLI formatted result.Event on every row, but the library
deliberately clears Event on a gap, so text mode printed
"0 MX_EVENT_FAMILY_UNSPECIFIED" and JSON mode an empty object, discarding the
resume cursors. The loop now branches on result.IsReplayGap() and renders the
typed row in both modes, counting it toward -limit like any other row. The JSON
row's cursors are typed by hand rather than marshalled with protojson: the
proto3 JSON mapping renders 64-bit integers as strings ("7") while the Rust and
Python CLIs emit numbers (7), so going through protojson would have made Go the
only canonical CLI with a different value type.
Docs in the same change: docs/Sessions.md documents the empty-ring sentinel
value and that oldest - 1 is the universal resume formula in both the retained
and fully-evicted cases; docs/CrossLanguageSmokeMatrix.md gains a per-CLI
gap-rendering table covering both client findings, and records exactly what is
and is not comparable across CLIs (same keys and numeric cursors for Rust/Go/
Python; quoted cursors for .NET/Java; differing key order, whitespace, and
container), so a matrix runner compares parsed values rather than raw bytes.
Tests, all written red first and each reproducing its defect verbatim:
- SessionEventDistributorTests: RegisterWithReplayReportsNextDeliverableSequence
WhenRingEmptiedByAge, ...WithRetentionDisabled, and
ResumeUsingSentinelFormulaAfterEmptyRingGapDeliversLiveEvents.
- GatewayEndToEndReconnectReplayTests.ReconnectAfterFullAgeEvictionResumesWith
SentinelFormula — fake-worker e2e resume walk on a fake clock; the fixture now
takes a retention window and a TimeProvider.
- clients/python test_stream_events_renders_replay_gap.
- clients/go TestRunStreamEventsPrintsReplayGap.
GWC-25's ReplayGap.oldest_available_sequence proto-comment amendment is
deliberately deferred to the later codegen wave (see the tracker change log): it
is comment-only but triggers the full five-client regen fan-out.
RunMonitorAsync issued SubscribeAlarms and the first reconcile before the
internal distributor subscriber was attached (via ISessionManager
.ReadAlarmEventsAsync). The pump has been running since MarkReady started the
dashboard mirror and only fans to subscribers registered at fan-out time, so
every transition raised in that two-round-trip window bypassed the alarm feed —
and a missed Acknowledge was never repaired, because ApplyReconcile broadcast
presence deltas only.
- The monitor now takes the internal lease directly from its session BEFORE
SubscribeAlarms and drains it after the first reconcile; window transitions
buffer in the lease's bounded channel. Processing them after ApplyReconcile is
order-safe (ApplyTransition handles alarms the snapshot already placed).
- ISessionManager.ReadAlarmEventsAsync removed — zero remaining callers.
- ApplyReconcile broadcasts an Acknowledge feed transition when a both-present
alarm's state advanced to ActiveAcked. This is a feed-level repair on the
AlarmFeedMessage/StreamAlarms surface rebuilt from the worker's own snapshot,
not MxEvent emission, so the "never synthesize events" rule is untouched;
the reasoning is recorded on ApplyReconcile.
The alarm-monitor test fakes now hand the monitor a real Ready GatewaySession
with a dashboard mirror, which is what makes the window reproducible.
Docs: docs/Sessions.md and gateway.md alarm-monitor ordering notes.
Refs: archreview/2026-07-12/remediation/10-gateway-core.md GWC-26
The gRPC auth failure limiter partitioned on the key id parsed out of the
*unauthenticated* token and rejected with ResourceExhausted before VerifyAsync
ran. Key ids are not secret — they ride in every token and are listed on the
dashboard — so any network peer could send 10 garbage-secret requests per minute
and deny that key indefinitely: the legitimate holder's correct secret was
refused before it was ever checked, and the success-path Reset that would clear
the block sat behind the verification the block prevented (SEC-31). The tracked
map was also flushable — any `a_b_c`-shaped junk minted a fresh partition (the
`mxgw` literal was never compared), so ~4096 throwaway tokens evicted a blocked
entry and reset the window (SEC-32).
ApiKeyFailureLimiter moves from IsBlocked/RecordFailure/Reset(string peer) to a
partition-pair API: Check/RecordFailure/Reset(ApiKeyThrottlePartition) with an
ApiKeyThrottleDecision result. Two layers share one sliding window — a composite
(transport peer, key id) partition at ApiKeyFailureLimit, and a per-key-id
aggregate across all peers at the new ApiKeyFailureAggregateLimit (default 30)
that bounds a source-rotating sprayer. An over-limit state is now a valve rather
than a wall: one request per the new ApiKeyFailureProbeIntervalSeconds (default
5) is admitted through to the real verifier, so the correct secret always reaches
the constant-time compare and resets both layers. Guarantees preserved: guessing
stays bounded per window, and the failure path still spends no store read per
attempt.
SEC-32 rides the same change set: the interceptor validates token shape (literal
`mxgw` prefix, >= 3 non-empty `_` segments, key id <= 64 chars) before minting a
key-id partition, each transport peer may mint at most 32 of them before the
overflow collapses onto its fallback partition, and eviction prefers fully
expired windows and never drops an over-limit partition below a 2x transient
overshoot ceiling. Throttled attempts increment mxgateway.auth.throttled, tagged
stage=peer|aggregate only — /metrics is unauthenticated (open SEC-14), so no key
material may appear there.
Docs in the same commit: GatewayConfiguration limiter rows plus the two new keys,
the Authentication hot-path paragraph, the Authorization SEC-11 section, and the
limiter / SecurityOptions XML remarks (the old NAT rationale described the
defective keying). Tracking rows flipped to Done with a change-log entry.
Tests: new ApiKeyFailureLimiterTests (11) covering window pruning, composite vs
aggregate trip points, probe cadence, absolute-block mode, reset across both
layers, junk-spray eviction resistance, the per-peer cap, and expired-window
eviction preference; GatewayGrpcAuthorizationInterceptorTests gains the four
SEC-31 contract tests plus NonMxgwToken_FallsBackToTransportPeerPartition (20
total); GatewayOptionsValidatorTests covers both new keys including 0 as a
supported disable value (66 total).
WRK-21 — DrainEvents was bounded by event count only, so a byte-heavy queue
(large string/array MxValues) built a reply above the negotiated frame maximum:
the writer rejected the frame, the exception unwound the session, and the events
already dequeued were destroyed. The drain is now byte-budgeted inside the queue
lock, so an event is dequeued only once it is known to fit and one that does not
stays at the head. Truncation is reported through the reply's existing
DiagnosticMessage (no contract change); callers drain until an empty reply. Both
reply-write seams — the control-command path and ProcessCommandAsync — now catch
MessageTooLarge and answer the correlation with an InvalidRequest reply instead
of unwinding or faulting the session. Satisfies IPC-23 R1-R3.
WRK-28 — the 10,000 drain ceiling moves to GatewayContractInfo
.MaxDrainEventsPerCommand, referenced by both the gateway request validator and
the worker clamp, replacing a comment-only sync contract. C# const only; no
.proto change.
WRK-23 — WorkerFrameWriter now peek-stamps, validates, then commits the sequence
counter immediately before the stream write, so a per-frame rejection leaves no
phantom gap on the wire.
IPC-30 — an oversized event frame stays session-fatal (it is undeliverable end to
end and neither dropping nor synthesizing a replacement is allowed), but the
death is structured: the event's identity and sizes are logged (never its value),
a WorkerFault with category PROTOCOL_VIOLATION and command method EventDrain is
written, then the session exits as before.
Docs updated in the same change: MxAccessWorkerInstanceDesign.md (drain byte cap,
truncation contract, oversized-head behavior, oversized-event policy, no control
reply is session-fatal on size), WorkerFrameProtocol.md (reply pre-sizing,
non-fatal reply-size rule, oversized-event policy, rejected frames do not consume
sequence numbers), gateway.md (DrainEvents two-axis bound).
The GWC-04 remediation decoupled the read loop from event backpressure by
staging events into an unbounded channel, so its TryWrite always succeeded and
the only overflow fault was a single timed WriteAsync exceeding
EventChannelFullModeTimeout. A consumer draining slower than the worker
produces — each individual write still completing inside the window — therefore
grew gateway memory without bound, without a fault, and without a metric: the
queue-depth gauge counted only the bounded consumer channel, so staged events
were invisible.
Bound _eventStaging at 2 x EventChannelCapacity (Wait, single reader/writer, no
synchronous continuations). A rejected staging TryWrite is the sustained
slow-drain signal and faults the client ProtocolViolation with
QueueOverflow("worker-event-staging"), guarded by IsTerminalState() so a
completed channel during shutdown stays a silent drop. SetFaulted is
non-blocking, so the read loop still never awaits behind events. The timed-write
fault is unchanged and still catches the full-stall case earlier.
Move the queue-depth increment from EnqueueWorkerEventAsync to StageWorkerEvent
so the single counter reports total undelivered events (staged + queued); the
decrement at consumer read was already correct. No new configuration key: the
bound is derived, and gateway-side buffering per session is now at most
3 x MxGateway:Events:QueueCapacity. Coordination with still-open GWC-21
(EventChannelFullModeTimeout configurability) remains open and was not blocked
on.
Tests: StagingChannelOverflowFaultsWorkerWithoutWaitingForFullModeTimeout (5-min
full-mode timeout so only the staging bound can fire; asserts an interleaved
command reply still completes) and WorkerEventQueueDepthGaugeCountsStagedEvents.
Docs updated in the same change: GatewayProcessDesign, MxAccessWorkerInstanceDesign,
GatewayConfiguration, Metrics. GWC-24 flipped to Done in both trackers.
AttachInternalEventSubscriber ran EnsureDistributorCreated / Register /
StartPumpIfRequested with no state check, unlike AttachEventSubscriber. A
premature attach would start the pump against a not-yet-Ready worker; the pump
source throws SessionNotReady, PumpAsync completes every subscriber with that
error and latches the distributor, and _eventDistributorStarted is never reset —
so the session would reach Ready with permanently dead event streaming.
Mirror AttachEventSubscriber's gate: check _state/_workerClient.State under
_syncRoot and throw SessionManagerException(SessionNotReady) before the
distributor is created, keeping the distributor calls outside the lock.
Refs: archreview/2026-07-12/remediation/10-gateway-core.md GWC-27
GatewayLogRedactor.RedactCommandValue had no production caller. Its only
one was GatewayLogRedactorAdapter on the abandoned
feat/adopt-zb-telemetry-serilog branch; when that work was re-implemented
on main as GatewayLogRedactorSeam the identity half was carried over and
the command-value half was not. Four unit tests kept the policy green, so
it read as wired while masking nothing.
No live leak today — no log statement currently emits a CommandValue
property — but the next one to do so would have written credential-bearing
MXAccess payloads (AuthenticateUser, WriteSecured, WriteSecured2) to every
sink in the clear, with passing tests suggesting otherwise.
Ports the missing half onto the seam: a non-null CommandValue is masked via
the existing policy, gated on CommandMethod. Value logging stays off — the
seam exposes no opt-in — so ordinary values are masked too, matching
RedactCommandValue's default. A null value stays null rather than becoming
the placeholder, and the property is never invented when absent.
Five tests added, three of which were red first on the leak itself
(operator01:hunter2 reaching the assertion unmasked). The other two pin
the null and absent guards. Identity redaction is untouched.
Full NonWindows suite: 785 pass, 45 pre-existing macOS NamedPipe-harness
failures unchanged from baseline (verified by stashing this change).
Build 0 warnings.
Family version-matrix alignment. No behaviour change — mxgw registers no Akka
checks, and 0.2.0's per-entry `data` object is emitted only when a check
publishes some, so its health payloads are byte-identical.
Note: this repo uses inline package pins, NOT central package management (there is
no Directory.Packages.props), so the version lives in the Server csproj.
Verified: Server builds 0 warnings.
Part of scadaproj docs/plans/2026-07-22-overview-dashboard-impl-plan.md Task 2.3.
0.2.3's Secrets.Ui ships ConfirmDeleteModal's own styles under
collision-proof zb-secrets-* class names. This host links no Bootstrap so
it never exhibited the invisible-modal defect, but it takes the fixed
line for parity; also rides over 0.2.1/0.2.2 (Akka-replicator fixes -
inert here, no replicator in use). Tests: 780 pass, 45 fail on macOS both
before and after the bump (NamedPipeServerStream multi-instance is
Windows-only - the fake-worker pipe harness cannot run on this platform);
zero delta from the bump.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
The suppression's own comment said 'Remove once an upstream fix ships'. It has:
SQLitePCLRaw.lib.e_sqlite3 2.1.12 patches this advisory within the 2.1.x line, so
the premise that no patched release existed is no longer true.
The Server project already resolved 2.1.12, but only incidentally - transitively via
ZB.MOM.WW.Auth.ApiKeys. An explicit PackageReference makes that floor intentional, so
a change to the Auth dependency graph cannot silently regress to the vulnerable 2.1.11.
Removing the suppression restores auditing for exactly that case.
Verified: forced restore of the NonWindows solution reports no NU1903; build is
0 warnings / 0 errors; test results unchanged from baseline (781 passed, 44 failed -
all 44 pre-existing macOS Unix-domain-socket path-length failures in the fake-worker
harness, identical count before and after).
Version hygiene + picks up the G-8 KEK-rotation surface.
NOT a security fix for this repo. An earlier version of this message claimed it
closed GHSA-2m69-gcr7-jv3q; that was wrong. A/B against the 0.1.2 baseline shows
SQLitePCLRaw.lib.e_sqlite3 already resolved 2.1.12, supplied transitively by the
pre-existing ZB.MOM.WW.Auth.ApiKeys 0.1.5 reference.
Note: src/Directory.Build.props still suppresses GHSA-2m69-gcr7-jv3q on the
now-outdated rationale that no patched e_sqlite3 exists. 2.1.12 is patched and
already resolving, so that suppression looks removable - verify separately.
Auth.ApiKeys 0.1.4 pulled SQLitePCLRaw.lib.e_sqlite3 2.1.11, which carries
high-severity advisory GHSA-2m69-gcr7-jv3q. This gateway was genuinely exposed --
verified 2.1.11 resolving before the bump and 2.1.12 after, with the vulnerability
scan now clean.
Auth 0.1.5 is 0.1.4 plus a transitive pin, so there is no API change here.
Suite unchanged from the documented baseline: 781 pass / 44 pre-existing worker-COM
failures on macOS, none Auth/ApiKey/SQLite related.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
The nightly (run #47) portable job failed CreateAsync_WhenFakeWorkerNeverSendsReady
_TimesOutAndKillsWorker: it asserts the factory's TimeoutException carries 'did not
complete startup' (thrown at the session's 1s startup timeout), but the test's
.WaitAsync(TestTimeout=5s) anti-hang net tripped first under CI load and surfaced
.NET's generic 'The operation has timed out.' — same commit passed on the push run
(#46), so it's a load-sensitive timeout race, not a regression. Raising runner
capacity to 4 makes concurrent-load flakes like this more likely.
Fix: give the two CreateAsync failure/timeout tests a dedicated 30s HangGuardTimeout,
far above the 1s semantic timeout under test, so the factory's own exception is
always the one observed. The net still catches a genuine hang.
Note: this class cannot run on macOS (named pipes -> Unix domain sockets fail on
the pipe path); verification is via CI Linux.
CI (portable, run #38) intermittently failed SessionManagerTests
.InvokeAsync_WhenTimeoutZero_FailsFastUnchanged with 'expected immediate
fail-fast but took 155ms'. The fail-fast paths throw synchronously under the
lock (GatewaySession.GetReadyWorkerClientAsync) and never enter the poll loop,
so the absolute <100ms wall-clock bound measured only host load, not behavior,
and flaked under CI contention.
- WhenWorkerFaulted_FailsFastWithBothStates: anchor the bound to a large (5000ms)
ready-wait timeout and assert fail-fast returns in < timeout/3 — a regression
that burned the timeout is still caught, but scheduling jitter can't trip it.
- WhenTimeoutZero_FailsFastUnchanged: drop the wall-clock assertion entirely
(a zero timeout has no wait window to burn); the error code, both-states
message, and InvokeCount == 0 already pin the immediate fail-fast.
Verified: the 3 SessionManager timing tests pass (net10.0).
Resolve all CommentChecker findings across the gateway server, worker, tests,
and .NET client (314 -> 0 real issues): add missing <returns>/<summary>/<param>
on public and test members, convert Stream/interface overrides to <inheritdoc/>,
and remove internal task/issue tracking IDs (SEC-*, IPC-*, WRK-*, GWC-*, TST-*,
Client.Dotnet-*) from shipped code documentation while preserving the design
rationale prose. Shipped comments should not carry internal bookkeeping, and
complete XML docs keep the analyzer/TreatWarningsAsErrors gate and generated API
docs clean. The 6 remaining flags are heuristic false positives (MD5, UTC-4,
capacity-1, near-1601) left intact so real documentation is not corrupted.
Claude-Session: https://claude.ai/code/session_01DMXXvNuPekkkrTEyPNxEkW
Joseph Doherty