WRK-22/IPC-26: tombstone a WriteAsync/WriteBatchAsync cancelled while
waiting for the write lock (PendingFrame.Claimed under _gate; DequeueNext
skips cancelled, claims the frame it returns) so a cancelled write never
reaches the wire unless already claimed mid-write (documented residual).
WRK-25: add WriteBatchAsync; RunEventDrainLoopAsync submits the drained
event batch through it, so a burst of N events costs one flush not N.
IPC-30 oversized-event structured fault preserved via FindOversizedEvent.
WRK-24: reject a below-1024 negotiated frame maximum at the handshake
(MinNegotiableFrameBytes, matching GatewayOptionsValidator floor).
WRK-27: alarm poll advertises StaCallInProgress on the heartbeat snapshot
so the watchdog suppresses to the ceiling, not the grace.
Docs (WorkerFrameProtocol.md, MxAccessWorkerInstanceDesign.md) and the
2026-07-12 remediation registers/change-log updated in the same commit.
Doc half of TST-30 (single shared Gitea runner is a CI throughput/availability
bottleneck): docs/GatewayTesting.md's Continuous Integration section gains a
"Runner capacity is shared and finite" subsection covering the maxParallel=1
instance-level runner shared with dohertj2/lmxopcua, the ~20-30 min queue
latency observed under cross-repo contention, and Gitea 1.26's missing run
cancel/delete API. The existing "windev tier down" degraded-mode paragraph now
also covers "runner contended" as a reason to bypass the queue via
CI_SHA=<sha> scripts/ci/run-windev-ci.sh <mode> or the manual windev worktree
flow, generalizing it per the finding's design note.
New operator runbook docs/runbooks/TST-30-second-ci-runner.md carries the
actual runner registration (option a: second act_runner instance on
10.100.0.35 with the same container.network: traefik config, recommended;
option b: dedicated labelled runner, escalation only; option c: runner on
windev, rejected) plus verification steps and the no-cancel caveat. The
optional workflow-level concurrency group is documented as unverified --
framed as "verify before relying on it" -- and left unimplemented in ci.yml,
since registering the runner and any runs-on gating is operator/infra work
outside this repo's tree.
Tracking: TST-30 -> Done (doc half; runner registration operator-pending) in
both registers + change-log row.
ClientProtoInputTests.Descriptor_ContainsEveryContractMessageAndField only
compared messages and fields, and only enumerated the gateway/worker
descriptors, so a new enum value, a new RPC, or any galaxy_repository.proto-only
change would not redden the test even though it is documented as the primary
protoc-free CI gate.
Rename to Descriptor_ContainsEveryContractSymbol and extend the reflection walk
on both sides (published protoset and in-process contract) to also collect
enums/enum values ({enumFullName}, {enumFullName}/{valueName}) and
services/methods ({serviceFullName}, {serviceFullName}/{methodName}), and add
GalaxyRepositoryReflection.Descriptor to the enumerated files. The comparison
stays a flat, order-insensitive string-set diff with no protoc dependency.
Update docs/ClientProtoGeneration.md and docs/Contracts.md prose from
"message or field" to the full symbol coverage.
Red-path proof: pointed the test at the pre-IPC-01 stale protoset and confirmed
it failed naming max_frame_bytes, several MxCommandKind/AlarmProviderMode enum
values, MxAccessGateway/StreamAlarms and GalaxyRepository/BrowseChildren, and
the galaxy_repository.v1.* surface; restored the real path and re-ran green.
Flips IPC-27 to Done in the 2026-07-12 remediation tracker and register.
Proto comments (comment-only, no wire change):
- mxaccess_worker.proto GatewayHello.max_frame_bytes: every worker->gateway frame
must serialize within the negotiated max; reply builders truncate (IPC-23).
- mxaccess_gateway.proto DrainEventsReply: count-cap + byte-cap, drain-until-empty
caller contract (IPC-23).
- mxaccess_gateway.proto ReplayGap.oldest_available_sequence: empty-ring value is
highest-observed+1, oldest-1 resume formula stays valid (GWC-25 deferred amendment).
Regen wave: Contracts/Generated (C# XML doc), rust vendored protos (byte-copy),
Go bindings (worker binding was genuinely stale - lacked MaxFrameBytes entirely),
Python worker _pb2 (real descriptor delta), Java aggregates (javadoc, zero
protobuf-version churn under the pinned toolchain), client descriptor set.
IPC-24: pinned Java toolchain regenerates with no gencode-version churn, so the
unconditional churn-revert step in ci.yml is a fossil - deleted it; git diff is
now a true message-level drift gate for the single-file Java aggregates.
IPC-25: pin protoc-gen-go v1.36.11 / protoc-gen-go-grpc 1.6.2 in the Go generate
script (+ fix a latent pwsh-7 parse bug); add Check 4 to check-codegen.ps1
(regenerate Go+Python bindings, fail on diff, tool-missing fails not skips); add
the pinned-generator installs to the portable CI job.
IPC-32: relabel check-codegen banners 1/4..4/4 (folded into the Check 4 edit).
Docs: ClientProtoGeneration.md, Contracts.md, GatewayTesting.md, build.gradle
checkGeneratedClean caveat. Tracking: IPC-23/24/25/32 -> Done, GWC-25 proto note
resolved, change-log 2026-08-07.
Repo-side half of SEC-36. The appsettings.json plaintext was already discharged
before this branch (HEAD ships the fail-closed ${secret:ldap/mxgateway/bind}
store reference), so the residual leak was the literal value in glauth.md,
docs/GatewayTesting.md, and the historical archreview SEC-06 evidence -- all
scrubbed to <service-account-password> placeholders pointing at the source of
truth scadaproj/infra/glauth/.
- csproj: add <UserSecretsId>mxaccessgw-server</UserSecretsId> (dev channel)
- GatewayOptionsValidator: blank-password message now names both channels
(dev user-secrets, deployed MxGateway__Ldap__ServiceAccountPassword)
- test: assert the message names both channels
- docs: GatewayConfiguration.md (three channels + rotation note), glauth.md
(placeholders + rotation-required + runbook pointer), GatewayTesting.md
- new operator runbook docs/runbooks/SEC-36-ldap-credential-rotation.md
(live rotation + NSSM staging remain operator-pending)
- tracking: SEC-36 -> Done (repo-side) in both registers + change-log
Deviation: kept the ${secret:} reference in appsettings.json rather than
deleting it (spec step 2 assumed the stale plaintext baseline); deleting it
would regress the shipped/documented/tested secret-store channel.
git grep -i for the old value is empty across all tracked files.
TST-27: docs/GatewayConfiguration.md's ShowTagValues row no longer says
"Reserved" — it now states what false (default) does (DashboardEventBroadcaster
blanks tag values from a deep-cloned MxEvent before the SignalR events-hub
mirror), the security relevance (no per-session hub ACL yet, so this
redaction is the only thing between a low-trust Viewer and other sessions'
tag values), and the honest scope limit (does not cover /browse).
WRK-26 (discharges IPC-29): docs/MxAccessWorkerInstanceDesign.md's "Outbound
Queues" section rewritten from the stale five-level priority list to the
two-class Control/Event scheduler actually shipped, with the collapsed-
decision rationale, and the overflow paragraph rewritten to the implemented
fail-fast. docs/WorkerFrameProtocol.md gained a "Write Scheduling And
Sequencing" section describing HEAD truthfully: WRK-23's peek-stamp-commit
sequencing is live, WRK-25's event-batch flush coalescing is not (the drain
loop still awaits each event write individually), and WRK-22's cancellation
tombstone is not yet defined (noted as pending, not documented as shipped).
CLI-42: clients/rust/README.md and docs/ClientPackaging.md document the
vendored Rust proto layout matching build.rs — repo-path-first resolution
falling back to clients/rust/protos/, the check-codegen.ps1 Check 3 refresh
rule, and why cargo package/publish run without --no-verify.
CLI-43: docs/style-guides/JavaStyleGuide.md now says Java 17 (Ignition 8.3
baseline), mirroring CLI-12's wording, matching the shipped build.gradle.
IPC-28: docs/Grpc.md's exception-mapping prose gained CommandTooLarge ->
ResourceExhausted, and the Invoke section gained the oversized-payload
sentence, cross-referencing GatewayConfiguration.md's headroom rule.
Tracking: TST-27, WRK-26, CLI-42, CLI-43, IPC-28 flipped to Done and IPC-29
marked discharged-by-WRK-26 in 00-tracking.md and the 20/30/50/60 domain
registers, with a 2026-08-07 change-log entry.
Doc-only change; no source, proto, or test edits.
Code-review follow-up on the CLI-40/41/44 branch.
ISSUE 1 (all five, critical): the message-only scrub still leaked the
server-echoed credential through the redacted error's structured reply accessor
(.NET Reply/Statuses, Java reply()/protocolStatus(), Go MxAccessError.Reply via
errors.As, Rust reply()/into_reply(), Python raw_reply). The redacted error now
carries a scrubbed clone of the reply (protocol_status.message,
diagnostic_message, statuses[].diagnostic_text), with per-language tests asserting
the reply accessor no longer contains the credential.
ISSUE 2 (Rust, critical): ensure_command_success routed MXACCESS_FAILURE to
Error::Command (unlike the other four clients), bypassing attach_secrets and
leaking via derived Debug/Display. MXACCESS_FAILURE now routes to Error::MxAccess,
fixing the cross-client inconsistency.
ISSUE 3 (Go, important): the CLI-44 terminal send was unconditionally
non-blocking, dropping a genuine terminal error under a full buffer on the
never-drop SubscribeEvents path. It is now reserved-slot-non-blocking only for the
cancel-on-overflow path and blocking for the never-drop path.
New shared fixture authenticate-user.echoed-credential-mxaccess-failure.reply.json
wired into all five suites. Minors: whitespace-secret guard on .NET/Java redact
helpers; Java preserves exception subtype on redaction; redaction-helper unit
tests (Go/Java/.NET). Docs (ClientBehaviorFixtures.md, ClientLibrariesDesign.md)
updated to make the structured-field claim true.
Same-commit docs rule (were missed in the prior commit):
- docs/GalaxyRepository.md: SnapshotCachePath now documents the per-OS derived
default and the GalaxyRepositoryOptionsValidator rooting/validity enforcement.
- A2-galaxyrepository-adoption-handoff.md: correct the now-inaccurate NSSM caveat
(SnapshotCachePath override is optional, not required; blank seeds a rooted host
default, no silent no-op) and repoint the option-validation item at the new
GalaxyRepositoryOptionsValidator.
SEC-34 guard confirmed and documented: TryParseKeyId's '_' split cannot truncate a
key id because both — and the only — gateway key-creation paths
(ApiKeyAdminCommandLineParser.IsValidKeyId, DashboardApiKeyManagementService.ValidateKeyId)
restrict key ids to IsAsciiLetterOrDigit || '.' || '-', and key ids are never
library-generated. Added a citing comment; no behavior change.
Test consolidation: moved the three host-start SqlitePath overrides into
TestHostEnvironmentInitializer (per-process temp store, mirroring Secrets__SqlitePath)
so future host-start tests auto-cover.
CLI-40: port the exact-secret credential scrub to Rust/Java/.NET (Go/Python
already did it). AuthenticateUser/WriteSecured(2) helpers now redact the exact
caller-supplied secret from any surfaced error, as defense-in-depth on top of the
by-construction guarantee. Rust hand-writes a redacting Debug (derived Debug would
leak the reply); Java/.NET rebuild the same exception type with the redacted
message and do not carry the secret-bearing original forward (so ToString/stack
traces stay clean too).
CLI-41: uniform malformed-reply contract for AuthenticateUser/ArchestrAUserToId/
AddBufferedItem across all five clients — typed payload, else a present int32
return_value, else a typed malformed-reply error. Fixes Go/Java silent-0, .NET
NRE, and Rust's own internal inconsistency.
CLI-44: the Go event goroutine's Recv-error path now uses a non-blocking
sendTerminalEventResult on the reserved slot, so a genuine terminal stream error
is reported as itself instead of being mislabeled ErrSlowConsumer under overflow.
Riders from the CLI-37/38 review: (a) .NET ToDiagnosticSummary and Python
_mxaccess_message surface the raw success member (diagnostics-only parity with
Rust); (b) the status-conversion fixture carries an independent wantSuccess
boolean and the Go/.NET fixture tests assert against it instead of recomputing
the formula under test.
Shared fixtures (authenticate-user.{echoed-credential,missing-payload,
return-value-only}.reply.json) + manifest + ClientBehaviorFixtures.md +
ClientLibrariesDesign.md updated in the same change. Tracking: CLI-40/41/44 -> Done.
SEC-33: make rooting host-meaningful and stop shipping foreign-platform literals.
- Delete IsRootedForAnyPlatform; AddIfNotRooted now uses Path.IsPathRooted (current OS).
- Promote AddIfNotRooted/AddIfInvalidPath to shared GatewayConfigPathRules so the new
Galaxy validator reuses them and the two validators cannot drift.
- Remove Authentication:SqlitePath and Galaxy:SnapshotCachePath Windows literals from
appsettings.json; the CommonApplicationData-derived code defaults take over. The
Galaxy default is seeded as a configuration value before AddZbGalaxyRepository
(SnapshotCachePath is init-only, so a PostConfigure mutation cannot compile).
- New GalaxyRepositoryOptionsValidator (ValidateOnStart) enforces a valid, host-rooted
SnapshotCachePath when PersistSnapshot is true.
- Root-cause the stray junk-named auth DB: host start eagerly builds
AuthSqliteConnectionFactory; under the non-rooted Windows literal on macOS SQLite
wrote it relative to the test bin CWD. The three real-host-start tests now pin
SqlitePath to a temp path.
SEC-34: verification cache Invalidate-vs-in-flight-repopulation race closed with a
per-key generation counter (bump-before-evict, snapshot-then-recheck). The expiry
cap (window 2) takes the documented fallback: the library verification identity
carries no ExpiresUtc, so the cache cannot cap at the key's expiry (donor-library ask).
GWC-24 rider: cap MxGateway:Events:QueueCapacity at int.MaxValue/2 so the derived
checked(2 * EventChannelCapacity) in WorkerClient cannot overflow at session creation.
SEC-35 (doc-only): note IsProduction() env-name semantics in GatewayConfiguration.md.
Docs updated same commit (GatewayConfiguration.md, Authentication.md) and tracking
registers/change-log flipped (00-tracking.md, 40-security-dashboard.md).
ReleaseProbe recognised its own reservation by comparing NextProbeAtTicks to
now + _probeIntervalTicks. RecordInto's rearm-on-trip writes that identical
expression, so a concurrent RecordFailure on the same WindowState whose `now`
lands on the claimer's tick — routine at ~1 ms clock resolution under load — was
mistaken for the caller's own claim. The release then stomped the legitimate
fresh re-arm back to the stale previousProbeAtTicks, which is already due, handing
the next arrival a free probe the re-arm had just closed.
WindowState gains a monotonic ProbeVersion bumped by every writer of
NextProbeAtTicks (TryConsumeProbe's claim and RecordInto's re-arm alike).
TryConsumeProbe returns the stamp it set as part of a ProbeClaim; ReleaseProbe
restores the previous value only while the state's version still equals that
stamp, checking and restoring in one lock(state) section and bumping the version
again on restore so no other stale release can match either.
Test: ProbeSlotRestore_DoesNotStompConcurrentRearmAtSameTick, with the clock held
still so the claim and the interleaved failure necessarily share a tick. Making it
deterministic needed a seam — the claim-to-release window is a few nanoseconds and
racing threads do not hit it (an earlier thread-based attempt passed against the
defective guard three runs out of three, and its end state was ordering-dependent
rather than correctness-dependent, so it was dropped rather than shipped as
theatre). The seam is an internal ProbeReleaseInterleaveHook, null in production,
costing one null check on the already-refused path. Verified as a genuine red
against the timestamp guard: Expected ThrottledByPeer, Actual ProbeAdmitted.
All five client CLIs now share one credential contract for `authenticate-user`:
flags `--password` / `--password-env` (Go: `-password` / `-password-env`) with
default env `MXGATEWAY_VERIFY_PASSWORD`, resolution flag-then-env, and a resolved
credential that is missing *or empty* is a usage error naming the flag and the
variable. The value is never echoed and never reaches the wire.
Go and Java previously sent an empty credential when the variable was unset,
turning a misconfigured environment into a real MXAccess authentication attempt.
Go now returns the guard error before dialing; Java throws a picocli
ParameterException instead of falling back to "". Python's `--password-env`
gained the canonical default and its UsageError names the resolved variable.
Rust treats an empty flag or env value as missing, with the resolution extracted
into a testable `resolve_verify_user_password`. .NET adopts the canonical flags
and keeps `--verify-user-password`, `--verify-user-password-env`, and
MXGATEWAY_VERIFY_USER_PASSWORD as deprecated aliases for one release.
Docs same commit: CrossLanguageSmokeMatrix.md gains the credential contract and
the per-CLI subcommand-coverage table (the documented-not-fixed half of the
finding); all five READMEs name the canonical variable and the fail-fast rule,
and the .NET README carries the deprecation note. Tracking flipped to Done in
both remediation registers with a change-log row.
No .proto changed; no generated code regenerated.
One cross-client conformance pass; also closes first-cycle CLI-08.
CLI-37: an MxStatusProxy entry is a failure iff `category !=
MX_STATUS_CATEGORY_OK`. The proto contract has always said so — `success` is
the raw 16-bit COM member carried verbatim for diagnostics, not a boolean — but
four clients branched on `success` alone and .NET required both, so the same
gateway reply produced opposite verdicts per language. An absent entry stays
success; a present entry with an UNSPECIFIED category is a failure, because the
worker always maps a category and an unmapped one is not proven OK.
CLI-38: a reply fails on HRESULT iff `hresult` is present and negative, so
positive COM success codes such as S_FALSE (1) pass. .NET/Go/Java used `!= 0`,
which errored on a parity-preserving S_FALSE that Python and Rust accepted.
This makes the existing ClientLibrariesDesign.md claim true rather than
rewriting the doc to describe the divergence.
Four shared fixtures pin both rules cross-client, and each language suite also
carries a table test for the two edges a fixture cannot express (absent entry,
UNSPECIFIED category). A Java test fake that built a status with a bare
`setSuccess(1)` and no category is fixed — under the category rule that reply
was never a success.
Code-review follow-up on fix/gwc-26-27-alarm-attach.
ApplyReconcile's snapshot-derived feed repairs are at-least-once, not
exactly-once: a reconcile reads the worker's current state while the matching
live transition may still be buffered in the monitor's lease, so both broadcast
and the duplicates are indistinguishable on the alarm feed. This pre-dates the
acked-state delta — the Raise/Clear presence repair has always had it, since
nothing serializes a reconcile pass against the in-flight live stream — so
closing it (serialization or timestamp dedup) stays out of scope for a P2 fix.
Documented instead, with the consumer contract stated explicitly (apply
transitions idempotently, never as an increment or toggle):
- ApplyReconcile gains a "Delivery semantics" comment.
- gateway.md softens the "defense in depth" prose to state the semantics.
- docs/Sessions.md carries the same caveat on the alarm-feed description.
- Tracker change-log records it as a known pre-existing characteristic and a
candidate finding for the next review cycle.
Also hoists the ChannelWorkerClient fake — duplicated across the three alarm
test files — into TestSupport/, dropping the usings it took with it.
Two defects found in code review of the limiter rework.
Probe admission was check-then-act across two lock scopes: Check() read
"probe due" under lock(state), released it, then re-acquired to advance
NextProbeAtTicks. A burst of requests arriving together at an interval boundary
could therefore all observe the slot as due and all be admitted, handing the
verifier the very burst the interval exists to bound. The claim is now a single
critical section (TryConsumeProbe). The two layers are still claimed one at a
time — holding two per-state locks at once would need a global lock ordering to
stay deadlock-free — so a slot claimed on the composite partition is compensated
via ReleaseProbe when the aggregate then refuses, which otherwise silently spent
the partition's next slot and pushed the legitimate holder out by a full
interval.
Reset() removed whatever partition the caller resolved to, including the
address's shared fallback partition when the caller's key id had been collapsed
into it by the per-peer cap (or when the token was junk-shaped). That bucket also
carries failures contributed by other key ids from the same address, so one
successful authentication became a reset button for an in-progress spray. Reset
now clears only a partition the caller owns (effectiveKeyId == presented key id);
the shared bucket decays by window expiry instead, and the caller still recovers
through probe admission. The key's aggregate is cleared either way, as designed.
Also applied from the review: closure-free GetOrAdd overload on _partitions, and
a remarks paragraph acknowledging the best-effort O(n) eviction scan under
sustained overflow. Threading the resolved partition key from Check through to
RecordFailure/Reset was declined: Check resolves with mint:false and RecordFailure
with mint:true, and the two can legitimately differ when a concurrent caller fills
the per-peer cap in between — reusing Check's key would record into the wrong
partition and bypass the cap, which is not worth saving one string concat.
Tests (limiter suite 11 -> 14): ProbeAdmission_UnderConcurrentArrivals_
GrantsExactlyOneSlot (200 rounds x 8 barrier-released threads at the boundary),
ProbeAdmission_WhenAggregateRefuses_ReturnsTheClaimedPeerSlot, and
Reset_WithOverCapKeyId_DoesNotClearSharedFallbackPartition. The latter two were
confirmed as genuine reds against the unfixed code; the concurrency test is a
guard — it is deterministically green on the fixed structure but did not
reproduce the original nanosecond-wide window on its own.
An empty replay ring reported oldest_available_sequence = 0 even when gap was
true. Clients follow the documented after_worker_sequence = oldest - 1 formula,
so an unsigned client computed ulong.MaxValue: the follow-up resume replayed
nothing, reported no gap, and the live filter dropped every subsequent event —
a silently dead stream in the headline detach-and-resume scenario, reachable on
default config once ReplayRetentionSeconds (300) age-evicts the ring.
GWC-25: SessionEventDistributor.RegisterWithReplay's empty-ring branch now
reports _highestSequenceSeen + 1 — the next sequence that can possibly be
delivered — when gap is true, so oldest - 1 lands exactly on the highest
observed sequence and the resume delivers everything newer. Still 0 when there
is no gap, where the field is meaningless and never emitted. Nothing is lost:
the evicted interval was unrecoverable either way, and the sentinel's job is to
say "re-snapshot".
CLI-35: the Python CLI fed every stream item into MessageToDict, which raised on
the ReplayGap dataclass and aborted the command after consuming the stream. A
new _event_row helper renders a gap as {"replayGap": {...}} — the same camelCase
shape the Rust CLI emits — and leaves proto events on the existing path.
CLI-36: the Go CLI formatted result.Event on every row, but the library
deliberately clears Event on a gap, so text mode printed
"0 MX_EVENT_FAMILY_UNSPECIFIED" and JSON mode an empty object, discarding the
resume cursors. The loop now branches on result.IsReplayGap() and renders the
typed row in both modes, counting it toward -limit like any other row. The JSON
row's cursors are typed by hand rather than marshalled with protojson: the
proto3 JSON mapping renders 64-bit integers as strings ("7") while the Rust and
Python CLIs emit numbers (7), so going through protojson would have made Go the
only canonical CLI with a different value type.
Docs in the same change: docs/Sessions.md documents the empty-ring sentinel
value and that oldest - 1 is the universal resume formula in both the retained
and fully-evicted cases; docs/CrossLanguageSmokeMatrix.md gains a per-CLI
gap-rendering table covering both client findings, and records exactly what is
and is not comparable across CLIs (same keys and numeric cursors for Rust/Go/
Python; quoted cursors for .NET/Java; differing key order, whitespace, and
container), so a matrix runner compares parsed values rather than raw bytes.
Tests, all written red first and each reproducing its defect verbatim:
- SessionEventDistributorTests: RegisterWithReplayReportsNextDeliverableSequence
WhenRingEmptiedByAge, ...WithRetentionDisabled, and
ResumeUsingSentinelFormulaAfterEmptyRingGapDeliversLiveEvents.
- GatewayEndToEndReconnectReplayTests.ReconnectAfterFullAgeEvictionResumesWith
SentinelFormula — fake-worker e2e resume walk on a fake clock; the fixture now
takes a retention window and a TimeProvider.
- clients/python test_stream_events_renders_replay_gap.
- clients/go TestRunStreamEventsPrintsReplayGap.
GWC-25's ReplayGap.oldest_available_sequence proto-comment amendment is
deliberately deferred to the later codegen wave (see the tracker change log): it
is comment-only but triggers the full five-client regen fan-out.
RunMonitorAsync issued SubscribeAlarms and the first reconcile before the
internal distributor subscriber was attached (via ISessionManager
.ReadAlarmEventsAsync). The pump has been running since MarkReady started the
dashboard mirror and only fans to subscribers registered at fan-out time, so
every transition raised in that two-round-trip window bypassed the alarm feed —
and a missed Acknowledge was never repaired, because ApplyReconcile broadcast
presence deltas only.
- The monitor now takes the internal lease directly from its session BEFORE
SubscribeAlarms and drains it after the first reconcile; window transitions
buffer in the lease's bounded channel. Processing them after ApplyReconcile is
order-safe (ApplyTransition handles alarms the snapshot already placed).
- ISessionManager.ReadAlarmEventsAsync removed — zero remaining callers.
- ApplyReconcile broadcasts an Acknowledge feed transition when a both-present
alarm's state advanced to ActiveAcked. This is a feed-level repair on the
AlarmFeedMessage/StreamAlarms surface rebuilt from the worker's own snapshot,
not MxEvent emission, so the "never synthesize events" rule is untouched;
the reasoning is recorded on ApplyReconcile.
The alarm-monitor test fakes now hand the monitor a real Ready GatewaySession
with a dashboard mirror, which is what makes the window reproducible.
Docs: docs/Sessions.md and gateway.md alarm-monitor ordering notes.
Refs: archreview/2026-07-12/remediation/10-gateway-core.md GWC-26
The gRPC auth failure limiter partitioned on the key id parsed out of the
*unauthenticated* token and rejected with ResourceExhausted before VerifyAsync
ran. Key ids are not secret — they ride in every token and are listed on the
dashboard — so any network peer could send 10 garbage-secret requests per minute
and deny that key indefinitely: the legitimate holder's correct secret was
refused before it was ever checked, and the success-path Reset that would clear
the block sat behind the verification the block prevented (SEC-31). The tracked
map was also flushable — any `a_b_c`-shaped junk minted a fresh partition (the
`mxgw` literal was never compared), so ~4096 throwaway tokens evicted a blocked
entry and reset the window (SEC-32).
ApiKeyFailureLimiter moves from IsBlocked/RecordFailure/Reset(string peer) to a
partition-pair API: Check/RecordFailure/Reset(ApiKeyThrottlePartition) with an
ApiKeyThrottleDecision result. Two layers share one sliding window — a composite
(transport peer, key id) partition at ApiKeyFailureLimit, and a per-key-id
aggregate across all peers at the new ApiKeyFailureAggregateLimit (default 30)
that bounds a source-rotating sprayer. An over-limit state is now a valve rather
than a wall: one request per the new ApiKeyFailureProbeIntervalSeconds (default
5) is admitted through to the real verifier, so the correct secret always reaches
the constant-time compare and resets both layers. Guarantees preserved: guessing
stays bounded per window, and the failure path still spends no store read per
attempt.
SEC-32 rides the same change set: the interceptor validates token shape (literal
`mxgw` prefix, >= 3 non-empty `_` segments, key id <= 64 chars) before minting a
key-id partition, each transport peer may mint at most 32 of them before the
overflow collapses onto its fallback partition, and eviction prefers fully
expired windows and never drops an over-limit partition below a 2x transient
overshoot ceiling. Throttled attempts increment mxgateway.auth.throttled, tagged
stage=peer|aggregate only — /metrics is unauthenticated (open SEC-14), so no key
material may appear there.
Docs in the same commit: GatewayConfiguration limiter rows plus the two new keys,
the Authentication hot-path paragraph, the Authorization SEC-11 section, and the
limiter / SecurityOptions XML remarks (the old NAT rationale described the
defective keying). Tracking rows flipped to Done with a change-log entry.
Tests: new ApiKeyFailureLimiterTests (11) covering window pruning, composite vs
aggregate trip points, probe cadence, absolute-block mode, reset across both
layers, junk-spray eviction resistance, the per-peer cap, and expired-window
eviction preference; GatewayGrpcAuthorizationInterceptorTests gains the four
SEC-31 contract tests plus NonMxgwToken_FallsBackToTransportPeerPartition (20
total); GatewayOptionsValidatorTests covers both new keys including 0 as a
supported disable value (66 total).
WRK-21 — DrainEvents was bounded by event count only, so a byte-heavy queue
(large string/array MxValues) built a reply above the negotiated frame maximum:
the writer rejected the frame, the exception unwound the session, and the events
already dequeued were destroyed. The drain is now byte-budgeted inside the queue
lock, so an event is dequeued only once it is known to fit and one that does not
stays at the head. Truncation is reported through the reply's existing
DiagnosticMessage (no contract change); callers drain until an empty reply. Both
reply-write seams — the control-command path and ProcessCommandAsync — now catch
MessageTooLarge and answer the correlation with an InvalidRequest reply instead
of unwinding or faulting the session. Satisfies IPC-23 R1-R3.
WRK-28 — the 10,000 drain ceiling moves to GatewayContractInfo
.MaxDrainEventsPerCommand, referenced by both the gateway request validator and
the worker clamp, replacing a comment-only sync contract. C# const only; no
.proto change.
WRK-23 — WorkerFrameWriter now peek-stamps, validates, then commits the sequence
counter immediately before the stream write, so a per-frame rejection leaves no
phantom gap on the wire.
IPC-30 — an oversized event frame stays session-fatal (it is undeliverable end to
end and neither dropping nor synthesizing a replacement is allowed), but the
death is structured: the event's identity and sizes are logged (never its value),
a WorkerFault with category PROTOCOL_VIOLATION and command method EventDrain is
written, then the session exits as before.
Docs updated in the same change: MxAccessWorkerInstanceDesign.md (drain byte cap,
truncation contract, oversized-head behavior, oversized-event policy, no control
reply is session-fatal on size), WorkerFrameProtocol.md (reply pre-sizing,
non-fatal reply-size rule, oversized-event policy, rejected frames do not consume
sequence numbers), gateway.md (DrainEvents two-axis bound).
The GWC-04 remediation decoupled the read loop from event backpressure by
staging events into an unbounded channel, so its TryWrite always succeeded and
the only overflow fault was a single timed WriteAsync exceeding
EventChannelFullModeTimeout. A consumer draining slower than the worker
produces — each individual write still completing inside the window — therefore
grew gateway memory without bound, without a fault, and without a metric: the
queue-depth gauge counted only the bounded consumer channel, so staged events
were invisible.
Bound _eventStaging at 2 x EventChannelCapacity (Wait, single reader/writer, no
synchronous continuations). A rejected staging TryWrite is the sustained
slow-drain signal and faults the client ProtocolViolation with
QueueOverflow("worker-event-staging"), guarded by IsTerminalState() so a
completed channel during shutdown stays a silent drop. SetFaulted is
non-blocking, so the read loop still never awaits behind events. The timed-write
fault is unchanged and still catches the full-stall case earlier.
Move the queue-depth increment from EnqueueWorkerEventAsync to StageWorkerEvent
so the single counter reports total undelivered events (staged + queued); the
decrement at consumer read was already correct. No new configuration key: the
bound is derived, and gateway-side buffering per session is now at most
3 x MxGateway:Events:QueueCapacity. Coordination with still-open GWC-21
(EventChannelFullModeTimeout configurability) remains open and was not blocked
on.
Tests: StagingChannelOverflowFaultsWorkerWithoutWaitingForFullModeTimeout (5-min
full-mode timeout so only the staging bound can fire; asserts an interleaved
command reply still completes) and WorkerEventQueueDepthGaugeCountsStagedEvents.
Docs updated in the same change: GatewayProcessDesign, MxAccessWorkerInstanceDesign,
GatewayConfiguration, Metrics. GWC-24 flipped to Done in both trackers.
Migrate the durable session-resilience governance record (Phase 5
orphan-worker reattach deferred-not-planned, EnableOrphanReattach
does not yet exist, settled Phase-4 Viewer-default decision) from
oldtasks.md into a new "Session-Resilience Epic Scope" entry in
docs/DesignDecisions.md, repoint CLAUDE.md and stillpending.md's
oldtasks.md references to the new home / tasks.json, and git rm
oldtasks.md now that it has no unique content left. Flip TST-29 to
Done in the archreview tracking registers.
The five untracked root docs-review artifacts (MxAccessGateway-docs-*,
MxGatewayClient-docs-*) are absent from this worktree; they must be
deleted from the main working tree separately (gitignored, no repo
impact).
The `windows`/`live-mxaccess` CI jobs were removed in abb0930 because Gitea
act_runner host-mode on Windows is broken and a runs-on gate with no runner
wedges the queue. This left the entire x86/net48 Worker + Worker.Tests tier
(and the live-MXAccess smoke) unguarded by CI.
Restore it without a native Windows runner: Linux jobs on ubuntu-latest (which
always schedule) SSH to windev (10.100.0.48), fetch+checkout the SHA under test
in an isolated clone C:\build\mxaccessgw-ci, run the x86 build/tests there, and
propagate the remote exit code back — so a Worker regression turns the job red
and an unreachable host fails loud (never stuck).
- scripts/ci/run-windev-ci.sh: Linux driver (key/known-hosts, UTF-16LE base64
EncodedCommand bootstrap, ssh, exit-code passthrough).
- scripts/ci/windev-worker-ci.ps1: windev stage — worktree lock, re-fetch/
checkout under lock, build|test|live modes; PS 5.1-safe, checks $LASTEXITCODE.
- scripts/ci/windev.known_hosts: pinned host keys for StrictHostKeyChecking.
- scripts/ci/README.md: operator bring-up + acceptance checklist.
- ci.yml: windows-x86 (per-push `test`) + nightly-windev (scheduled `live` +
on-failure Gitea issue); drop the removal note; fix header/cron comments.
TST-26 (same commit, by rule): correct docs/scripts that still describe the
removed jobs — GatewayTesting.md, Contracts.md, check-codegen.ps1 — and
reattribute the Generated/-commit guard (primary = check-codegen diff in the
portable job; secondary = windows-x86 net48 compile).
Mechanism hand-verified on windev: build->0, bogus-SHA->nonzero (lock released),
test->356 passed/0 failed in ~50s, ssh exit-code propagation confirmed. Merge
requires operator bring-up first (dedicated ci@ key + Gitea secrets) or the
per-push job is red every push — see scripts/ci/README.md.
Add docs/plans/2026-07-10-dashboard-session-acl-tst15.md: the fleshed-out
Phase-4 design for the deferred TST-15 finding. Resolves the crux the deferral
left open — the dashboard authenticates LDAP users (Admin/Viewer) while sessions
are API-key-owned (OwnerKeyId), two disjoint identity domains — via a session tag
sourced from the owning API key (carried in the existing ApiKeyConstraints JSON
blob, no SQLite migration). Admin-sees-all; a Viewer may SubscribeSession iff
session.Tags intersects the Viewer's granted tags (new Dashboard:GroupToTag map
-> hub-token tag claims); untagged sessions Admin-only by default. Includes the
enforcement path, task breakdown (epic Tasks 16-19), test plan incl. live-LDAP,
and rejected alternatives.
Design only — TST-15 stays Not started (no implementation). The tracker and the
60-testing-docs-gaps TST-15 section point at the design doc; the change-log also
records the TST-03 finding (zero registered runners; needs a runner co-located on
the gitea Docker network).
Claude-Session: https://claude.ai/code/session_01DMXXvNuPekkkrTEyPNxEkW
Completes both findings across all five clients — done locally on the Mac now
that homebrew openjdk@17 is available (JAVA_HOME=/opt/homebrew/opt/openjdk@17).
CLI-15: new MxEventStreamItem record + MxEventStream.nextItem() surfaces the
gateway's ReplayGap sentinel as a typed, non-terminal signal (isReplayGap()/
replayGap()/event()); existing Iterator<MxEvent> path unchanged, sentinel never
swallowed/synthesized. Javadoc covers gap semantics + resume contract.
CLI-04: typed single-item helpers on MxGatewaySession — Phase 1
adviseSupervisory/writeSecured/writeSecured2/authenticateUser/archestrAUserToId,
Phase 2 addBufferedItem/setBufferedUpdateInterval/suspend/activate (unregister
already present). Each routes through invokeCommand -> ensureProtocolSuccess +
ensureMxAccessSuccess (same validation as bulk). MXAccess parity preserved.
Credentials flow only into the request proto; exceptions carry only the reply and
gRPC status text is scrubbed via MxGatewaySecrets.redactCredentials — tests
assert the password/secured value is absent from getMessage()/toString()/CLI
output. New CLI subcommands write-secured/authenticate-user (credential via
--password/--password-env, prints only the user id).
gradle test: 106 tests, 0 failures (58 client + 48 cli); no generated churn.
Shared docs: ClientLibrariesDesign + CLAUDE.md updated to "all five clients".
Claude-Session: https://claude.ai/code/session_01DMXXvNuPekkkrTEyPNxEkW
Resolve the session-resilience epic's shipped-vs-planned entanglement:
- Phase 3 (reconnect) finished: Task 13 = TST-02 (owner-scoped attach, P0),
Task 15 = TST-01 (reconnect integration test), Task 14 = CLI-15 for 4/5
clients (Java pending, windev batch).
- Phase 4 (per-session dashboard ACL) scoped as TST-15; the open Viewer-default
decision is settled: admin-sees-all, Viewer strict per owned/granted session
(matches TST-02's gRPC owner binding).
- Phase 5 (orphan-worker reattach) marked DEFERRED, not planned. The
EnableOrphanReattach flag does not exist and must not be referenced as if it
does. The CLAUDE.md "gateway restart does not reattach orphan workers"
invariant stands.
Updates oldtasks.md, the tasks.json mirror (statuses + governance note), and the
CLAUDE.md reconnect paragraph (clients now consume ReplayGap; reattach deferred).
Claude-Session: https://claude.ai/code/session_01DMXXvNuPekkkrTEyPNxEkW
The gateway emits a ReplayGap sentinel MxEvent at the head of a StreamEvents
stream resumed via after_worker_sequence when the requested cursor predates the
oldest retained event. Clients previously ignored it, silently mis-treating a
lossy resume as continuous. Each client now surfaces the sentinel as a distinct,
typed, non-terminal signal (never synthesized, never swallowed) so a consumer can
detect the gap and re-snapshot; resume contract is
after_worker_sequence = oldest_available_sequence - 1.
- .NET: MxEventStreamItem (IsReplayGap/ReplayGap/Event) via new StreamEventItemsAsync
+ AsStreamItemsAsync extension. Build clean, 87 passed.
- Go: EventResult.ReplayGap field + IsReplayGap(); ReplayGap type alias. build/vet/test clean.
- Rust: EventItem enum (Event/ReplayGap); EventStream now yields Result<EventItem, Error>;
CLI renders REPLAY_GAP line / replayGap JSON. fmt/check/test/clippy clean.
- Python: ReplayGap dataclass; stream_events yields pb.MxEvent | ReplayGap. 131 passed.
- Shared docs: ClientLibrariesDesign non-goals reframed (reconnect-replay protocol is
consumable; auto-reconnect stays a non-goal); CrossLanguageSmokeMatrix resume-gap note.
Java client is deferred to the windev batch (no local JRE); CLI-15 stays open until it lands.
Claude-Session: https://claude.ai/code/session_01DMXXvNuPekkkrTEyPNxEkW
TST-11: single-source the .NET-side version in src/Directory.Build.props so
Server/Worker/Contracts/tests stamp 0.1.2 (was the SDK default 1.0.0) and
InformationalVersion carries the git short SHA (0.1.2+<sha>) for support
correlation; the git query is guarded (ContinueOnError) so a build outside a
git checkout still succeeds. Verified: Server assembly stamps 0.1.2+579282f,
build clean. Kept at 0.1.2 (matches Contracts + aligned Python/Rust/Go
clients); Java leads at 0.2.0 post JDK-17 retarget. Convergence to a single
0.2.0 cadence left as a release decision, not forced here.
WRK-15 (docs-only, no x86 worker build needed): correct the STA thread name in
docs/WorkerSta.md + docs/MxAccessWorkerInstanceDesign.md to the actual
MxGateway.Worker.STA (was the FQ ZB.MOM.WW.MxGateway.Worker.STA), and fix the
stale heartbeat-counter note - CaptureHeartbeat now populates event queue depth
(eventQueue.Count) and sequence (LastEventSequence) from the live queue.
Claude-Session: https://claude.ai/code/session_01DMXXvNuPekkkrTEyPNxEkW
SEC-25 (near-term hardening; full per-session EventsHub ACL stays deferred to
roadmap item 12): DashboardEventBroadcaster now redacts tag values from a deep
CLONE of the event before mirroring to SignalR when Dashboard:ShowTagValues is
false (the default). Clears MxEvent.value (covers OnDataChange/OnWriteComplete/
OperationComplete/OnBufferedDataChange — their bodies are empty discriminators,
values ride in the top-level field incl. buffered arrays) and the alarm body's
current_value/limit_value. Source event never mutated (shared with the gRPC
path + replay ring) - verified by a source-not-mutated test. Makes the formerly
dead ShowTagValues flag live for the mirror. EventsHub TODO(per-session-acl)
kept and tied to roadmap item 12. Tests: DashboardEventBroadcasterTests (3).
SEC-30: trim docs/Diagnostics.md to mark opt-in command-value logging as NOT
YET IMPLEMENTED (no LogCommandValues knob, RedactCommandValue has no call
sites, no values logged); wiring deferred pending secured-bulk redaction
coverage (SEC-13). No option/call sites added.
Server build clean (0 warnings); Dashboard tests 152/152.
Claude-Session: https://claude.ai/code/session_01DMXXvNuPekkkrTEyPNxEkW
Behavior-preserving allocation cuts on the per-event/per-command path:
- GWC-06: StreamEvents send-timing uses Stopwatch.GetTimestamp() +
GetElapsedTime() instead of a per-event Stopwatch allocation (same measured
span).
- GWC-07/IPC-05 (event): MapEvent transfers ownership of the inner MxEvent
instead of .Clone()-ing it. Safe: WorkerEvent is parsed fresh per pipe frame
with the distributor pump as its single consumer (GWC-01), MapEvent runs once
before fan-out, and every downstream consumer (subscribers, replay ring)
only READS the event (WorkerSequence is stamped upstream; verified no
post-mapping mutation). Comment documents the invariant + restore-clone
caveat if a second consumer is added.
- IPC-05 (command): CreateCommandEnvelope no longer re-clones; MapCommand
already isolated the graph from the caller-owned gRPC message.
- GWC-15: grpc_stream_queue.depth converts from a per-event push counter to an
ObservableGauge summing registered channel sources at scrape time only
(name/semantics unchanged); removes all per-event .Count/lock work.
Kept every load-bearing isolation clone (MapCommand, Invoke, bulk filters,
MapCommandReply). Server build clean (0 warnings); EventStream/Metrics/
Distributor/Mapper tests 62/62 (incl. formerly-flaky queue-depth tests, now
green under the lazy gauge, + 2 new MapEvent ownership tests). Docs: Metrics.md,
Grpc.md.
Claude-Session: https://claude.ai/code/session_01DMXXvNuPekkkrTEyPNxEkW
Restore the __Host- browser guarantees for the default secure deployment
without breaking plaintext/dev:
- DashboardServiceCollectionExtensions PostConfigure now resolves the cookie
name as: explicit MxGateway:Dashboard:CookieName override wins; else
__Host-MxGatewayDashboard when SecurePolicy==Always (RequireHttpsCookie
true); else the plain MxGatewayDashboard default. Guard: never apply the
__Host- prefix without Secure (browsers silently drop it).
- DashboardAuthenticationDefaults: add SecureCookieName const; keep the plain
CookieName as the non-secure fallback.
- Docs corrected to the actual conditional contract (five stale claims):
gateway.md, GatewayProcessDesign.md, ImplementationPlanGateway.md,
GatewayDashboardDesign.md, CLAUDE.md; GatewayConfiguration.md phrasing
tightened.
- Test: DashboardCookieOptionsTests asserts the name flips with
RequireHttpsCookie and that an explicit override wins.
Server build clean (0 warnings); Dashboard tests 149/149.
Claude-Session: https://claude.ai/code/session_01DMXXvNuPekkkrTEyPNxEkW
Reconcile load-bearing docs with shipped behavior:
- IPC-06: gateway.md Worker Envelope sketch -> points to mxaccess_worker.proto
as source of truth (string correlation_id, real oneof arms incl.
worker_shutdown_ack/worker_ready).
- IPC-07: docs/Grpc.md six RPCs -> seven; document QueryActiveAlarms handler
+ validation row.
- IPC-21: gateway.md Session RPC moved from live API into a 'Future work: not
implemented' subsection.
- TST-13: drop stale design-era sketches from gateway.md; correct the
single-subscriber-default (config-gated fan-out) note.
- SEC-09: dashboard GroupToRole sample GwAdmin:Admin -> Administrator so it
passes GatewayOptionsValidator; clarify Administrator is the canonical role.
- SEC-22: rewrite docs/Authentication.md to the pipeline that actually ships
(ZB.MOM.WW.Auth.ApiKeys package + gateway-owned CachingApiKeyVerifier,
CoalescingMarkApiKeyStore, CanonicalForwardingApiKeyAuditStore, etc.);
remove 18 stale type names (grep-verified absent).
- IPC-17: correct wrong Python generated dir (mxgateway -> zb_mom_ww_mxgateway)
in CLAUDE.md + 3 docs.
- CLI-12: Java docs Java 21 -> Java 17 (JDK17 retarget for Ignition 8.3).
- CLI-16: docs/ClientPackaging.md reconciled with real .slnx, Python package
name, and gradle project names; fix stale generateProto task name.
Docs-only; type/path/version claims verified against source.
Claude-Session: https://claude.ai/code/session_01DMXXvNuPekkkrTEyPNxEkW
Gateway-side follow-up to the shared auth-lib expiry core (delivered via G-2):
- apikey create-key gains optional --expires — absolute ISO-8601 UTC or relative
<N>d/<N>h from now; omitted means non-expiring (opt-in, unchanged default).
Threaded ApiKeyAdminCommand -> parser -> runner into the library's
CreateKeyAsync(..., expiresUtc, ...). list-keys shows an expiry column and an
active/expired/revoked status.
- Dashboard API Keys page surfaces expiry: an Expires column (Never when unset)
and a status badge reading Expired (red) / Expiring (<=7d, amber) / Revoked /
Active. DashboardApiKeySummary.ExpiresUtc projected in DashboardSnapshotService;
StatusBadge maps the new states.
Tests: parser (absolute/relative/invalid/none) + end-to-end past-expiry rejection
through the live verifier; dashboard summary suite green. Docs: Authentication.md
(verification-flow expiry step, CLI table + examples, dashboard badge).
Closes the tracked SEC-10 polish (SEC-10 core was already Done).
Joseph Doherty