ReleaseProbe recognised its own reservation by comparing NextProbeAtTicks to
now + _probeIntervalTicks. RecordInto's rearm-on-trip writes that identical
expression, so a concurrent RecordFailure on the same WindowState whose `now`
lands on the claimer's tick — routine at ~1 ms clock resolution under load — was
mistaken for the caller's own claim. The release then stomped the legitimate
fresh re-arm back to the stale previousProbeAtTicks, which is already due, handing
the next arrival a free probe the re-arm had just closed.
WindowState gains a monotonic ProbeVersion bumped by every writer of
NextProbeAtTicks (TryConsumeProbe's claim and RecordInto's re-arm alike).
TryConsumeProbe returns the stamp it set as part of a ProbeClaim; ReleaseProbe
restores the previous value only while the state's version still equals that
stamp, checking and restoring in one lock(state) section and bumping the version
again on restore so no other stale release can match either.
Test: ProbeSlotRestore_DoesNotStompConcurrentRearmAtSameTick, with the clock held
still so the claim and the interleaved failure necessarily share a tick. Making it
deterministic needed a seam — the claim-to-release window is a few nanoseconds and
racing threads do not hit it (an earlier thread-based attempt passed against the
defective guard three runs out of three, and its end state was ordering-dependent
rather than correctness-dependent, so it was dropped rather than shipped as
theatre). The seam is an internal ProbeReleaseInterleaveHook, null in production,
costing one null check on the already-refused path. Verified as a genuine red
against the timestamp guard: Expected ThrottledByPeer, Actual ProbeAdmitted.
Two defects found in code review of the limiter rework.
Probe admission was check-then-act across two lock scopes: Check() read
"probe due" under lock(state), released it, then re-acquired to advance
NextProbeAtTicks. A burst of requests arriving together at an interval boundary
could therefore all observe the slot as due and all be admitted, handing the
verifier the very burst the interval exists to bound. The claim is now a single
critical section (TryConsumeProbe). The two layers are still claimed one at a
time — holding two per-state locks at once would need a global lock ordering to
stay deadlock-free — so a slot claimed on the composite partition is compensated
via ReleaseProbe when the aggregate then refuses, which otherwise silently spent
the partition's next slot and pushed the legitimate holder out by a full
interval.
Reset() removed whatever partition the caller resolved to, including the
address's shared fallback partition when the caller's key id had been collapsed
into it by the per-peer cap (or when the token was junk-shaped). That bucket also
carries failures contributed by other key ids from the same address, so one
successful authentication became a reset button for an in-progress spray. Reset
now clears only a partition the caller owns (effectiveKeyId == presented key id);
the shared bucket decays by window expiry instead, and the caller still recovers
through probe admission. The key's aggregate is cleared either way, as designed.
Also applied from the review: closure-free GetOrAdd overload on _partitions, and
a remarks paragraph acknowledging the best-effort O(n) eviction scan under
sustained overflow. Threading the resolved partition key from Check through to
RecordFailure/Reset was declined: Check resolves with mint:false and RecordFailure
with mint:true, and the two can legitimately differ when a concurrent caller fills
the per-peer cap in between — reusing Check's key would record into the wrong
partition and bypass the cap, which is not worth saving one string concat.
Tests (limiter suite 11 -> 14): ProbeAdmission_UnderConcurrentArrivals_
GrantsExactlyOneSlot (200 rounds x 8 barrier-released threads at the boundary),
ProbeAdmission_WhenAggregateRefuses_ReturnsTheClaimedPeerSlot, and
Reset_WithOverCapKeyId_DoesNotClearSharedFallbackPartition. The latter two were
confirmed as genuine reds against the unfixed code; the concurrency test is a
guard — it is deterministically green on the fixed structure but did not
reproduce the original nanosecond-wide window on its own.
The gRPC auth failure limiter partitioned on the key id parsed out of the
*unauthenticated* token and rejected with ResourceExhausted before VerifyAsync
ran. Key ids are not secret — they ride in every token and are listed on the
dashboard — so any network peer could send 10 garbage-secret requests per minute
and deny that key indefinitely: the legitimate holder's correct secret was
refused before it was ever checked, and the success-path Reset that would clear
the block sat behind the verification the block prevented (SEC-31). The tracked
map was also flushable — any `a_b_c`-shaped junk minted a fresh partition (the
`mxgw` literal was never compared), so ~4096 throwaway tokens evicted a blocked
entry and reset the window (SEC-32).
ApiKeyFailureLimiter moves from IsBlocked/RecordFailure/Reset(string peer) to a
partition-pair API: Check/RecordFailure/Reset(ApiKeyThrottlePartition) with an
ApiKeyThrottleDecision result. Two layers share one sliding window — a composite
(transport peer, key id) partition at ApiKeyFailureLimit, and a per-key-id
aggregate across all peers at the new ApiKeyFailureAggregateLimit (default 30)
that bounds a source-rotating sprayer. An over-limit state is now a valve rather
than a wall: one request per the new ApiKeyFailureProbeIntervalSeconds (default
5) is admitted through to the real verifier, so the correct secret always reaches
the constant-time compare and resets both layers. Guarantees preserved: guessing
stays bounded per window, and the failure path still spends no store read per
attempt.
SEC-32 rides the same change set: the interceptor validates token shape (literal
`mxgw` prefix, >= 3 non-empty `_` segments, key id <= 64 chars) before minting a
key-id partition, each transport peer may mint at most 32 of them before the
overflow collapses onto its fallback partition, and eviction prefers fully
expired windows and never drops an over-limit partition below a 2x transient
overshoot ceiling. Throttled attempts increment mxgateway.auth.throttled, tagged
stage=peer|aggregate only — /metrics is unauthenticated (open SEC-14), so no key
material may appear there.
Docs in the same commit: GatewayConfiguration limiter rows plus the two new keys,
the Authentication hot-path paragraph, the Authorization SEC-11 section, and the
limiter / SecurityOptions XML remarks (the old NAT rationale described the
defective keying). Tracking rows flipped to Done with a change-log entry.
Tests: new ApiKeyFailureLimiterTests (11) covering window pruning, composite vs
aggregate trip points, probe cadence, absolute-block mode, reset across both
layers, junk-spray eviction resistance, the per-peer cap, and expired-window
eviction preference; GatewayGrpcAuthorizationInterceptorTests gains the four
SEC-31 contract tests plus NonMxgwToken_FallsBackToTransportPeerPartition (20
total); GatewayOptionsValidatorTests covers both new keys including 0 as a
supported disable value (66 total).
Surfaced during TST-25 acceptance verification: all CI runs on one shared
gitea-runner (maxParallel=1, co-located 10.100.0.35) interleaved with
dohertj2/lmxopcua, and Gitea 1.26 has no run cancel/delete API, so queue
latency is unbounded under cross-repo contention and the runner is a single
point of failure. Design: add a second/labelled runner; document the no-cancel
reality and the run-windev-ci.sh queue-bypass. Low/P2. Roll-ups updated.
Operator bring-up complete and the SSH-driven Windows/x86 CI tier passed
green end-to-end under Gitea Actions run #37: the Linux runner SSHed windev,
checked out the SHA in the isolated C:\build\mxaccessgw-ci clone under lock,
built the x86 Worker and ran Worker.Tests (exit 0); nightly-windev correctly
skipped on the push event. Flip TST-25/TST-26 status rows to Done and record
the bring-up + green-run evidence in the change log.
The `windows`/`live-mxaccess` CI jobs were removed in abb0930 because Gitea
act_runner host-mode on Windows is broken and a runs-on gate with no runner
wedges the queue. This left the entire x86/net48 Worker + Worker.Tests tier
(and the live-MXAccess smoke) unguarded by CI.
Restore it without a native Windows runner: Linux jobs on ubuntu-latest (which
always schedule) SSH to windev (10.100.0.48), fetch+checkout the SHA under test
in an isolated clone C:\build\mxaccessgw-ci, run the x86 build/tests there, and
propagate the remote exit code back — so a Worker regression turns the job red
and an unreachable host fails loud (never stuck).
- scripts/ci/run-windev-ci.sh: Linux driver (key/known-hosts, UTF-16LE base64
EncodedCommand bootstrap, ssh, exit-code passthrough).
- scripts/ci/windev-worker-ci.ps1: windev stage — worktree lock, re-fetch/
checkout under lock, build|test|live modes; PS 5.1-safe, checks $LASTEXITCODE.
- scripts/ci/windev.known_hosts: pinned host keys for StrictHostKeyChecking.
- scripts/ci/README.md: operator bring-up + acceptance checklist.
- ci.yml: windows-x86 (per-push `test`) + nightly-windev (scheduled `live` +
on-failure Gitea issue); drop the removal note; fix header/cron comments.
TST-26 (same commit, by rule): correct docs/scripts that still describe the
removed jobs — GatewayTesting.md, Contracts.md, check-codegen.ps1 — and
reattribute the Generated/-commit guard (primary = check-codegen diff in the
portable job; secondary = windows-x86 net48 compile).
Mechanism hand-verified on windev: build->0, bogus-SHA->nonzero (lock released),
test->356 passed/0 failed in ~50s, ssh exit-code propagation confirmed. Merge
requires operator bring-up first (dedicated ci@ key + Gitea secrets) or the
per-push job is red every push — see scripts/ci/README.md.
Six-domain re-review at the P2 merge (4f5371f): all prior Done claims
verified (none false, 10 partial), 47 new findings (1 High, 14 Medium),
with per-finding design/implementation plans and a new tracking register.
Make the authored CI pipeline execute and go green on the co-located runner (portable + java), fix the five real latent defects the first real run surfaced (codegen-check null, stale rust proto, orphan-terminator Linux path, stale java worker codegen, py3.12 event loop), provision pwsh + Gradle for the self-hosted act image, and disable the Windows jobs (act host-mode broken). TST-03 -> Done.
https://claude.ai/code/session_01DMXXvNuPekkkrTEyPNxEkW
Add docs/plans/2026-07-10-dashboard-session-acl-tst15.md: the fleshed-out
Phase-4 design for the deferred TST-15 finding. Resolves the crux the deferral
left open — the dashboard authenticates LDAP users (Admin/Viewer) while sessions
are API-key-owned (OwnerKeyId), two disjoint identity domains — via a session tag
sourced from the owning API key (carried in the existing ApiKeyConstraints JSON
blob, no SQLite migration). Admin-sees-all; a Viewer may SubscribeSession iff
session.Tags intersects the Viewer's granted tags (new Dashboard:GroupToTag map
-> hub-token tag claims); untagged sessions Admin-only by default. Includes the
enforcement path, task breakdown (epic Tasks 16-19), test plan incl. live-LDAP,
and rejected alternatives.
Design only — TST-15 stays Not started (no implementation). The tracker and the
60-testing-docs-gaps TST-15 section point at the design doc; the change-log also
records the TST-03 finding (zero registered runners; needs a runner co-located on
the gitea Docker network).
Claude-Session: https://claude.ai/code/session_01DMXXvNuPekkkrTEyPNxEkW
Java client toolchain (homebrew openjdk@17) works on the Mac, so the remaining
Java halves of CLI-15/CLI-04 are done locally this session, not batched to windev.
Claude-Session: https://claude.ai/code/session_01DMXXvNuPekkkrTEyPNxEkW
Gateway-side follow-up to the shared auth-lib expiry core (delivered via G-2):
- apikey create-key gains optional --expires — absolute ISO-8601 UTC or relative
<N>d/<N>h from now; omitted means non-expiring (opt-in, unchanged default).
Threaded ApiKeyAdminCommand -> parser -> runner into the library's
CreateKeyAsync(..., expiresUtc, ...). list-keys shows an expiry column and an
active/expired/revoked status.
- Dashboard API Keys page surfaces expiry: an Expires column (Never when unset)
and a status badge reading Expired (red) / Expiring (<=7d, amber) / Revoked /
Active. DashboardApiKeySummary.ExpiresUtc projected in DashboardSnapshotService;
StatusBadge maps the new states.
Tests: parser (absolute/relative/invalid/none) + end-to-end past-expiry rejection
through the live verifier; dashboard summary suite green. Docs: Authentication.md
(verification-flow expiry step, CLI table + examples, dashboard badge).
Closes the tracked SEC-10 polish (SEC-10 core was already Done).
Empirically verified the full ZB.MOM.WW.MxGateway.Tests suite exits cleanly:
- macOS: 0 surviving testhost after a full run.
- windev (origin/main worktree): 0 new testhost and 0 new worker processes
after a full run, isolated by process StartTime.
Static audit confirms the prime-suspect fixtures already dispose deterministically
(WorkerClient cancels _stopCts + awaits its read/write/heartbeat tasks with a 5s
timeout; GatewaySession disposes the client; harness/E2E fixtures use await using).
Drop the stale 'leaves orphaned testhost processes' rationale from CLAUDE.md's
Source Update Workflow; keep filtered-run guidance as a speed guideline only.
Separately discovered (tracked in 00-tracking.md change log, NOT part of TST-08):
Windows-only temp-file-lock flakiness in ~4 full-host test classes (Sqlite
connection-pool retaining a temp .db handle; SelfSignedCertificateProvider's
fixed-name gw.pfx.tmp racing teardown) that macOS never surfaces.
- SEC-02: DashboardAuthorizationHandler restricts the loopback and
Authentication:Mode=Disabled bypasses to read-only. They now satisfy only a
Viewer-bearing requirement (AnyDashboardRole), never AdminOnly, so anonymous
localhost can view the dashboard but cannot reach API-key CRUD or session
Close/Kill at the policy layer (previously guarded only by service re-checks).
- SEC-12: DashboardSessionAdminService emits canonical AuditEvents through
IAuditWriter (actions dashboard-close-session / dashboard-kill-worker, category
SessionAdmin) on Success/Failure/Denied, mirroring the API-key audit path so
destructive session actions leave durable, queryable rows.
- SEC-20: drop the unbounded session_id tag from the exported
mxgateway.heartbeats.failed counter (per-session detail stays in the snapshot/log).
Docs updated same-change: CLAUDE.md (read-only loopback + 5-min bearer),
GatewayDashboardDesign.md (bypass scoping + session-admin audit), Metrics.md.
Server build clean; 30/30 targeted + 295/295 Dashboard/Security/App/Metrics sweep.
Adds the 2026-07-08 architecture review (00-overall + six domain reports)
and a remediation/ tree: one design+implementation doc per domain covering
every finding, plus 00-tracking.md as the master progress tracker.
- 153 findings with stable IDs (GWC/WRK/IPC/SEC/CLI/TST), each with
design rationale, implementation steps, tests, docs, and verification.
- Tracker rolls findings up by severity and P0/P1/P2 roadmap tier, records
cross-cutting clusters and per-finding status (all Not started).
- Planning docs only; no source changes.
Joseph Doherty
dohertj2