From 0152180929aa1d3af84bcbffd22e21d68fd384de Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Mon, 10 Aug 2026 05:49:44 -0400 Subject: [PATCH 1/6] build(tst-11): stop Windows builds stamping git stderr into InformationalVersion (NEXT-09) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit MSBuildThisFileDirectory ends in a backslash, which escaped the closing quote of the Exec command on Windows; git then failed and, because the target runs with ContinueOnError + ConsoleToMSBuild (which mixes stderr into ConsoleOutput), the failure text was stamped as the source revision — an observed stamp read '0.1.2+fatal: cannot change to ...'. Append '.' to the quoted path so the trailing separator can no longer escape the quote, and gate SourceRevisionId on a short-SHA shape so no future git failure text can become the revision either. Windows verification runs on windev with the rest of this batch; macOS stamp confirmed unchanged (0.2.0+75c71ad). --- src/Directory.Build.props | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/src/Directory.Build.props b/src/Directory.Build.props index d390d73..0b809f2 100644 --- a/src/Directory.Build.props +++ b/src/Directory.Build.props @@ -26,7 +26,10 @@ - + - $(_StampedGitSha.Trim()) + + $(_StampedGitSha.Trim()) From 8769ee97658315fd830766563e4c77d97b0c1ebf Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Mon, 10 Aug 2026 05:54:11 -0400 Subject: [PATCH 2/6] fix(sessions): keep named-pipe socket paths inside the macOS sun_path limit (NEXT-01) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The pipe name mxaccess-gateway-{pid}-session-{32hex} plus .NET's CoreFxPipe_ prefix overflowed the 104-byte Unix-domain-socket path limit under the default per-user macOS TMPDIR (~49 chars), so every test that opened a real pipe threw ArgumentOutOfRangeException at pipe creation unless TMPDIR=/tmp was exported. Rename to mxgw-{pid}-{sessionUid} (the session guid hex without the session- prefix; worst-case 43 chars) and shorten the three test-fixture names the same way. Uniqueness is unchanged: gateway pid + full session guid. The worker receives the pipe name via its launch command line, so mixed Server/Worker deploy SHAs are unaffected. Docs updated in the same change (gateway.md, GatewayProcessDesign, GatewayConfiguration, Sessions, CLAUDE.md); new regression test pins the format and the length budget. Verified: SessionManagerTests 39/39; SessionWorkerClientFactory, GatewayEndToEndFakeWorkerSmoke, WorkerClient, and ReconnectReplay suites 33/33 under the default macOS TMPDIR — this also retires the previously-misdiagnosed 'macOS pipe-timeout test failures': they were this path-length throw, not a timeout-message defect. --- CLAUDE.md | 2 +- docs/GatewayConfiguration.md | 2 +- docs/GatewayProcessDesign.md | 7 +- docs/Sessions.md | 2 +- ...7-followups-windev-ldapfixtures-runners.md | 261 +++++++++++ ...-windev-ldapfixtures-runners.md.tasks.json | 13 + ...-08-07-live-actions-sec36-tst30-publish.md | 420 ++++++++++++++++++ ...-actions-sec36-tst30-publish.md.tasks.json | 18 + gateway.md | 9 +- .../Sessions/SessionManager.cs | 14 +- .../Gateway/Sessions/SessionManagerTests.cs | 30 ++ ...ssionWorkerClientFactoryFakeWorkerTests.cs | 2 +- .../Workers/Fakes/FakeWorkerHarness.cs | 2 +- .../Gateway/Workers/WorkerClientTests.cs | 2 +- 14 files changed, 769 insertions(+), 15 deletions(-) create mode 100644 docs/plans/2026-08-07-followups-windev-ldapfixtures-runners.md create mode 100644 docs/plans/2026-08-07-followups-windev-ldapfixtures-runners.md.tasks.json create mode 100644 docs/plans/2026-08-07-live-actions-sec36-tst30-publish.md create mode 100644 docs/plans/2026-08-07-live-actions-sec36-tst30-publish.md.tasks.json diff --git a/CLAUDE.md b/CLAUDE.md index 205accb..d7409f1 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -10,7 +10,7 @@ The architecture is a two-process design — read `gateway.md` before making str - **Gateway** (`src/ZB.MOM.WW.MxGateway.Server`, .NET 10, x64): ASP.NET Core gRPC server. Owns the public API, sessions, auth, the Blazor dashboard, and the Galaxy Repository SQL browse RPCs. The Galaxy-browse implementation comes from the shared **`ZB.MOM.WW.GalaxyRepository`** package (`AddZbGalaxyRepository`/`MapZbGalaxyRepository`), not inline code; mxaccessgw adds `GatewayBrowseScopeProvider` (per-key browse-subtree scoping) and a host-side dashboard summary projector. See `A2-galaxyrepository-adoption-handoff.md`. **Never instantiates MXAccess COM directly.** - **Worker** (`src/ZB.MOM.WW.MxGateway.Worker`, .NET Framework 4.8, **x86**): one process per session. Owns one MXAccess COM instance on a dedicated STA, pumps Windows messages, and converts COM events to protobuf. -- **IPC**: gateway↔worker uses one bidirectional named pipe per worker (`mxaccess-gateway-{gatewayPid}-{sessionId}`) with length-prefixed `WorkerEnvelope` protobuf frames. Gateway hosts the pipe server and launches the worker. **gRPC is not used inside the worker** — .NET Framework 4.8 doesn't have a first-class gRPC stack. +- **IPC**: gateway↔worker uses one bidirectional named pipe per worker (`mxgw-{gatewayPid}-{sessionUid}` — kept short so the macOS/Linux test matrix's Unix-domain-socket path fits the 104-byte macOS `sun_path` limit) with length-prefixed `WorkerEnvelope` protobuf frames. Gateway hosts the pipe server and launches the worker. **gRPC is not used inside the worker** — .NET Framework 4.8 doesn't have a first-class gRPC stack. - **Contracts** (`src/ZB.MOM.WW.MxGateway.Contracts`): multi-targets `net10.0;net48` and owns the `.proto` files (`mxaccess_gateway.proto`, `mxaccess_worker.proto`, `galaxy_repository.proto`). All other projects consume the generated types from here. Do not hand-edit anything under `Generated/`. Note `galaxy_repository.proto` is intentionally kept here as the generation source for the language clients even though the gateway server consumes the wire-identical Galaxy types from the `ZB.MOM.WW.GalaxyRepository` package — it is not dead code; deleting it breaks all five clients. The worker must do all MXAccess COM calls on its dedicated STA thread, and the STA loop must pump Windows messages (`MsgWaitForMultipleObjectsEx` + `PeekMessage`/`DispatchMessage`) so MXAccess events deliver. A plain blocking queue on an STA is not enough. diff --git a/docs/GatewayConfiguration.md b/docs/GatewayConfiguration.md index 781c50f..7529d09 100644 --- a/docs/GatewayConfiguration.md +++ b/docs/GatewayConfiguration.md @@ -663,7 +663,7 @@ See each client README for the as-built behavior. Transport security here applies only to the public gRPC channel. The gateway↔worker link is a per-session **named pipe** -(`mxaccess-gateway-{gatewayPid}-{sessionId}`), not a network socket. It is not +(`mxgw-{gatewayPid}-{sessionUid}`), not a network socket. It is not TLS-encrypted and does not need to be: it never leaves the local Windows host and is secured by the OS pipe ACL. See [Worker Frame Protocol](./WorkerFrameProtocol.md). diff --git a/docs/GatewayProcessDesign.md b/docs/GatewayProcessDesign.md index 4a1d3af..6fd2ddd 100644 --- a/docs/GatewayProcessDesign.md +++ b/docs/GatewayProcessDesign.md @@ -418,9 +418,14 @@ The gateway creates the pipe server before launching the worker. Pipe name: ```text -mxaccess-gateway-{gatewayProcessId}-{sessionId} +mxgw-{gatewayProcessId}-{sessionUid} ``` +`sessionUid` is the session id's guid hex without the `session-` prefix. The +short form keeps the Unix-domain-socket path .NET uses for named pipes on +macOS/Linux (`$TMPDIR/CoreFxPipe_{name}`) inside the 104-byte macOS `sun_path` +limit under the default per-user `TMPDIR`. + Message framing: ```text diff --git a/docs/Sessions.md b/docs/Sessions.md index 466f46a..6f511f4 100644 --- a/docs/Sessions.md +++ b/docs/Sessions.md @@ -14,7 +14,7 @@ All four interfaces (`ISessionManager`, `ISessionRegistry`, `ISessionWorkerClien `GatewaySession` is a sealed class that holds the identity, configured timeouts, worker client reference, and current `SessionState` for one session. State is protected by a private `_syncRoot` lock so that property reads and transitions are observed atomically by concurrent gRPC calls and the lease sweeper. -The session id is an opaque string in the form `session-{guid:N}` and the per-session pipe name is `mxaccess-gateway-{ProcessId}-{SessionId}`. Encoding the gateway PID into the pipe name avoids collisions when an old gateway process leaks pipes that the OS has not yet reclaimed. +The session id is an opaque string in the form `session-{guid:N}` and the per-session pipe name is `mxgw-{ProcessId}-{guid:N}` (the same guid hex, without the `session-` prefix). Encoding the gateway PID into the pipe name avoids collisions when an old gateway process leaks pipes that the OS has not yet reclaimed. The name is kept short because .NET named pipes on Unix-like hosts are Unix domain sockets at `$TMPDIR/CoreFxPipe_{name}`, and macOS caps that path at 104 bytes while its default per-user `TMPDIR` is already ~49 — the old `mxaccess-gateway-{pid}-{sessionId}` form overflowed it and broke the fake-worker/e2e tests on macOS. `SessionState` itself is the protobuf-generated enum from `ZB.MOM.WW.MxGateway.Contracts.Proto`, so it is shared between the gateway and clients on the wire. diff --git a/docs/plans/2026-08-07-followups-windev-ldapfixtures-runners.md b/docs/plans/2026-08-07-followups-windev-ldapfixtures-runners.md new file mode 100644 index 0000000..10c3fdf --- /dev/null +++ b/docs/plans/2026-08-07-followups-windev-ldapfixtures-runners.md @@ -0,0 +1,261 @@ +# Follow-Ups: windev Redeploy, LDAP Test Fixtures, Runner Hygiene — Implementation Plan + +> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:subagent-driven-development +> (opus implementers; controller verifies ops evidence; final review pass). + +**Goal:** Close the five items surfaced by the 2026-08-07 live-actions cycle: repair windev's +crash-looping gateway and finish the deferred SEC-36 dashboard verification (NEXT-07), fix the +DashboardLdapLiveTests fixture drift (NEXT-06), resolve the unexpected macOS instance runner, +harden runner-1's plaintext registration token, and verify the cargo Bearer fix. + +**Architecture:** Three independent live streams (windev serial: 1→2→3; repo test fix: 4; +Gitea/runner hygiene: 5, 6) run concurrently; task 7 closes out docs/trackers. No contract, +gateway-logic, or client changes — one test-file edit (Task 4) plus live ops plus docs. + +**Tech Stack:** SSH + PowerShell `-EncodedCommand` (windev 10.100.0.48), SSH + docker compose +(10.100.0.35), Gitea admin API (`gitea.dohertylan.com`, token via `~/.zshenv` `GITEA_TOKEN`), +NSSM, xUnit live-LDAP suite, GLAuth at `10.100.0.35:3893`. + +--- + +## Preflight facts (verified before planning) + +- Unpushed local mxaccessgw commits: `0566716`, `9760497`, `5b153da`, `41e8648` (all docs-only). + **`origin/main` = `a346d51`** — contains all current code, so windev can build from + `origin/main` without any push. +- windev (`10.100.0.48`): NSSM service `MxAccessGw`; deployed Server build of 2026-06-25 + (Auth 0.1.2.0, supports auth-DB schema 2) crash-loops on + `C:\ProgramData\MxGateway\gateway-auth.db` migrated to schema 3 on 2026-07-15 + (`AuthStoreMigrationException`, ~3.8k–10k Hosting-failed events/day). The NEW LDAP secret is + already staged as the 10th `AppEnvironmentExtra` entry (SEC-36 Task 3) — preserve it. +- `DashboardLdapLiveTests.cs` (`src/ZB.MOM.WW.MxGateway.IntegrationTests/`): uses + `admin`/`admin123` (3 tests) and `readonly`/`readonly123`. Directory reality + (`scadaproj/infra/glauth/config.toml`): `admin` exists, password is the standard dev test + password (`password`, hash `5e884898…42d8` — same as `multi-role`), and IS in GwAdmin + (othergroups `[5610, 5701]`); `readonly` does not exist; `gw-viewer` (primarygroup 5611 = + GwReader, NOT GwAdmin) is the natural not-an-admin fixture. Test binds + `MxGateway:Ldap` from `appsettings.json` (**`Server: localhost`**) + env overrides — so the + live run needs `MxGateway__Ldap__Server=10.100.0.35` as well as + `MxGateway__Ldap__ServiceAccountPassword` (from Mac user-secrets, never printed). +- Gitea instance runners (`GET /api/v1/admin/actions/runners`): id 1 `gitea-runner` (cap 4), + id 4 `macos-local-Josephs-MacBook-Pro` (**unexpected, online, labels overlap + ubuntu-latest**), id 5 `gitea-runner-2` (cap 2). +- `10.100.0.35:/opt/gitea/docker-compose.yml` (+ `docker-compose.yml.bak-tst30`): runner-1's + registration token inline in plaintext env, file world-readable. runner-2 uses + `GITEA_RUNNER_REGISTRATION_TOKEN_FILE: /run/secrets/runner_token` ← 0600 + `/opt/gitea/runner_token`. runner-1 data volume `/opt/gitea/runner:/data` (its `.runner` + credential persists — the registration env is only needed for first registration). +- Cargo Bearer fix already applied (`~/.zshenv`, backup `~/.zshenv.bak-cli39`) and documented + (`docs/ClientPackaging.md`, commit `5b153da`). Task 7 verifies; no further action expected. + +## Secret hygiene (binding, all tasks) + +- Never print the LDAP service-account password, `GITEA_TOKEN`, cargo token, runner + registration tokens, or API keys — not in commands, logs, commits, or reports. Read the LDAP + password from `dotnet user-secrets list` into an env var without echoing + (e.g. `export MxGateway__Ldap__ServiceAccountPassword="$(dotnet user-secrets list --project src/ZB.MOM.WW.MxGateway.Server | awk -F' = ' '/ServiceAccountPassword/ {print $2}')"`). +- Documented dev **test users** (`multi-role`/`password`, `admin`/`password`, + `gw-viewer`/`password`) are NOT secrets — glauth.md publishes them; fine in code/commits. +- SSH→windev PowerShell: always `powershell -NoProfile -EncodedCommand `; + never put secrets inside EncodedCommand blobs or argv. + +--- + +### Task 1: NEXT-07 — Recon windev deployment layout + schema support + +**Classification:** standard — read-only recon, but its output gates a service redeploy +**Estimated implement time:** ~5 min +**Parallelizable with:** Task 4, Task 5, Task 6 + +**Files:** none edited. SSH recon on `10.100.0.48` + repo/scadaproj reads on the Mac. + +Determine everything Task 2 needs, and confirm the fresh-deploy path is safe: + +1. `nssm get MxAccessGw Application`, `AppDirectory`, `AppParameters`, + `AppEnvironmentExtra` (count entries; do NOT print values of secret-bearing entries — + names only). +2. Inventory the deployed dir: path, `ZB.MOM.WW.MxGateway.Server.exe` timestamp, whether + `appsettings.json`/`appsettings.Production.json` in the deploy dir differ from repo + `origin/main` (diff; windev-specific config must survive the redeploy). +3. Confirm build feasibility on windev: `dotnet --list-sdks` (need 10.x), locate an existing + mxaccessgw checkout/worktree (CI uses `scripts/ci/windev-worker-ci.ps1` — find its + worktree path) or pick a fresh clone location. Confirm `git fetch` reaches `origin/main` + = `a346d514dd24e775640e5667aa7cd8e561fec68a`. +4. Confirm current code supports auth-DB schema 3: find the auth-store supported-schema + constant (ZB.MOM.WW.Auth packages — check the package version the Server at `origin/main` + references, and/or the migration code in the shared scadaproj libs) and state the + evidence. **If current code does NOT support schema 3, STOP — report, do not deploy.** +5. Gateway endpoints for verification: bound URLs/ports (from deployed config/env), dashboard + scheme (http vs https → cookie will be `MxGatewayDashboard` vs `__Host-…`). +6. Check what migrated the DB to schema 3 on 2026-07-15 (event log / file timestamps) — only + to confirm schema 3 is the shared-lib current version, not an anomaly. + +**Step: report** all findings as structured text (no secrets); no changes, no commits. + +### Task 2: NEXT-07 — Build current Server on windev and redeploy the service + +**Classification:** high-risk — replaces a running (crash-looping) service's binaries +**Estimated implement time:** ~10 min +**Parallelizable with:** none (needs Task 1) + +**Files:** none in repo. windev filesystem + NSSM only. + +Using Task 1's facts: + +1. On windev, fetch/checkout `origin/main` (`a346d51…`) in the build worktree/clone. +2. `dotnet publish src/ZB.MOM.WW.MxGateway.Server -c Release` (match deployed layout/RID from + Task 1; framework-dependent vs self-contained must match what NSSM `Application` points at). +3. Stop the service (`nssm stop MxAccessGw`), confirm process exited. +4. Backup: deployed dir → sibling `*.bak-next07` copy; copy + `C:\ProgramData\MxGateway\gateway-auth.db` (+ `-wal`/`-shm` if present) to + `gateway-auth.db.bak-next07`. **Never delete the live DB.** +5. Deploy publish output over the deploy dir, then restore any windev-specific config files + identified in Task 1 (do not clobber live overrides; NSSM env entries are untouched by + file copies but verify count unchanged after start). +6. `nssm start MxAccessGw`; verify: service state RUNNING and stable ≥60 s (no restart + cycle), Application event log shows clean host start and **zero new + `AuthStoreMigrationException` / `Hosting failed to start`** after the start timestamp, + bound port answers (e.g. dashboard root or health endpoint returns HTTP). +7. Rollback if unhealthy: stop, restore `*.bak-next07` dir, start, report. + +**Step: report** deployed SHA, verification evidence, backup paths. No repo commits. + +### Task 3: SEC-36 deferred verification + NEXT-07/runbook closeout + +**Classification:** standard +**Estimated implement time:** ~6 min +**Parallelizable with:** none (needs Task 2) + +**Files:** +- Modify: `docs/runbooks/SEC-36-ldap-credential-rotation.md` (Correction 3 — mark the + deferred dashboard check done, dated) +- Modify: `archreview/2026-07-12/remediation/90-candidate-findings-next-cycle.md` (NEXT-07 → + resolved 2026-08-07, evidence one-liner) + +1. Complete SEC-36 step 4 against the repaired windev gateway: log in to the dashboard as + `multi-role`/`password` end-to-end. Preferred: `curl` flow — GET `/login` (capture + antiforgery token + cookie), POST credentials, expect success redirect + auth cookie + (name per Task 1 scheme). If the login page resists scripting (Blazor circuit), report + exactly why and fall back to asserting a fresh `DashboardLdapLiveTests` green run + (Task 4) plus windev log evidence of successful LDAP bind on a manual attempt. +2. Update the two docs; commit locally (`docs(sec-36,next-07): …`), do NOT push. + +### Task 4: NEXT-06 — Fix DashboardLdapLiveTests fixtures to match the shared directory + +**Classification:** small — one test file, but must go green against live GLAuth +**Estimated implement time:** ~6 min +**Parallelizable with:** Task 1, Task 5, Task 6 + +**Files:** +- Modify: `src/ZB.MOM.WW.MxGateway.IntegrationTests/DashboardLdapLiveTests.cs` +- Modify: `archreview/2026-07-12/remediation/90-candidate-findings-next-cycle.md` (NEXT-06 → + resolved) +- Possibly modify: `docs/GatewayTesting.md` (live-LDAP opt-in row: document the + `MxGateway__Ldap__Server` override needed when GLAuth is not localhost) + +1. Read `DashboardAuthenticator` first: confirm a user who binds successfully but maps to no + role yields `Succeeded == false` (drives the gw-viewer fixture). +2. Fix fixtures: `admin`/`admin123` → `admin`/`password` (positive + wrong-password + + unreachable tests); `readonly`/`readonly123` → `gw-viewer`/`password` (exercises + user-binds-but-lacks-GwAdmin; keep the no-password-leak assertion, updating the asserted + literal). Update XML doc comments to match. Keep MXAccess-repo style rules + (TreatWarningsAsErrors). +3. Build: `dotnet build src/ZB.MOM.WW.MxGateway.IntegrationTests` (macOS OK — net10.0). +4. Live run (env only, never echo the password): + `MXGATEWAY_RUN_LIVE_LDAP_TESTS=1 MxGateway__Ldap__Server=10.100.0.35 MxGateway__Ldap__ServiceAccountPassword= dotnet test … --filter FullyQualifiedName~DashboardLdapLiveTests` + → expect **5/5 passed** (this is also positive live proof of the SEC-36 service-account + bind). +5. Update tracker row (+ GatewayTesting.md if the Server-override note is missing); commit + locally (`test(ldap): …`), do NOT push. + +### Task 5: Resolve the unexpected macOS instance runner (id 4) + +**Classification:** standard — evidence-gated removal of a live runner registration +**Estimated implement time:** ~6 min +**Parallelizable with:** Task 1, Task 4, Task 6 + +**Files:** +- Modify: `docs/runbooks/TST-30-second-ci-runner.md` (the Correction paragraph mentions + "id 4 — an unrelated local macOS runner" — update to final state) + +1. Evidence, local: `pgrep -fl act_runner`, `launchctl list | grep -i act`, + `brew services list | grep -i act`, look for `~/.runner`/act_runner config dirs. Evidence, + Gitea (token from `~/.zshenv`, never printed): runner detail for id 4 (labels, last + online), and whether any recent runs' jobs report `runner_id == 4` + (`GET /repos/{owner}/{repo}/actions/runs?…` → `…/runs/{id}/jobs` for both `mxaccessgw` + and `lmxopcua` recent runs). +2. Decision rule: the runner advertises ubuntu labels from a macOS host, so it can steal + Linux container jobs → **remove it** unless evidence shows it deliberately serves jobs + the docker runners cannot (none expected). Removal = stop the local act_runner process + AND disable its autostart (launchd/brew), then `DELETE /api/v1/admin/actions/runners/4`. + Keep the local config file (renamed `*.disabled-2026-08-07`) so re-registering with + mac-specific labels stays easy; note the re-registration recipe in the runbook edit. +3. Verify: admin runner list shows only ids 1 and 5, both online; no act_runner process + locally; a `pgrep` after 60 s still empty (nothing respawned). +4. Update the TST-30 runbook correction paragraph; commit locally, do NOT push. + +### Task 6: Harden runner-1's registration token on 10.100.0.35 + +**Classification:** standard — touches the live CI stack's compose file +**Estimated implement time:** ~7 min +**Parallelizable with:** Task 1, Task 4, Task 5 + +**Files:** none in repo (host `/opt/gitea/` only; runbook note lands in Task 7 if needed). + +1. Preconditions on the host: confirm runner-1's `/data/.runner` exists in its volume + (registration credential persists → the registration env var is no longer needed); + confirm both runners idle (no `act_runner`-spawned job containers, no in-progress runs + via API) before recreating. +2. Edit `/opt/gitea/docker-compose.yml` (backup first → `docker-compose.yml.bak-tst30b`): + replace runner-1's inline `GITEA_RUNNER_REGISTRATION_TOKEN: ` with the same + `_FILE`/secrets pattern runner-2 uses (`/opt/gitea/runner_token`, 0600). Do NOT touch the + `gitea` service definition. +3. `docker compose up -d --no-deps` the runner-1 service only; verify it comes back online + in the admin runner list and its `.runner` identity is unchanged (still id 1). +4. Tighten perms: `chmod 600 /opt/gitea/docker-compose.yml docker-compose.yml.bak-tst30 docker-compose.yml.bak-tst30b` + (verify compose stack still operable by the deploy user). +5. Rotate the leaked registration token if the deployment allows: + `docker exec … gitea actions generate-runner-token` (or admin API) — if Gitea offers no + invalidation of the old value, say so explicitly in the report (residual risk: LAN actor + could register a rogue runner until rotation) rather than claiming it rotated. +6. Verify CI still works: trigger nothing; just confirm both runners online and the token + file perms; a real push lands naturally later. Report evidence. + +### Task 7: Closeout — cargo Bearer verification, docs/tracker sync, commits + +**Classification:** small +**Estimated implement time:** ~5 min +**Parallelizable with:** none (needs Tasks 3, 4, 5, 6) + +**Files:** +- Modify: `archreview/2026-07-12/remediation/90-candidate-findings-next-cycle.md` (final + state of NEXT-06/NEXT-07 rows if Tasks 3/4 left anything) +- Possibly modify: `docs/GatewayTesting.md` / TST-30 runbook (runner topology now ids 1+5 + only; token-hardening note) + +1. Cargo Bearer verification (no printing): assert `~/.zshenv` line matches + `CARGO_REGISTRIES_DOHERTJ2_GITEA_TOKEN="Bearer …"` via `grep -c`, confirm + `docs/ClientPackaging.md` note present (commit `5b153da`); check no other credential + location (CI secrets, windev profiles) publishes to cargo — expected none. +2. Sweep: every doc touched this cycle consistent (runbooks, trackers, GatewayTesting.md); + `git grep` for stale phrases ("crash-loop… pending", "id 4", "admin123") and fix. +3. Commit remaining doc changes locally; do NOT push. List the full unpushed stack in the + report. + +--- + +## Out of scope + +- Pushing any mxaccessgw commits (user decides; stack listed at closeout). +- The five next-cycle candidate findings other than NEXT-06/NEXT-07. +- Auth-DB restore path for windev (fresh deploy chosen — preserves schema-3 data). +- `ci.yml` changes (labels, concurrency groups). + +## Dependency graph + +``` +{1} → 2 → 3 ┐ +{4} ├→ 7 +{5} │ +{6} ─────────┘ +``` diff --git a/docs/plans/2026-08-07-followups-windev-ldapfixtures-runners.md.tasks.json b/docs/plans/2026-08-07-followups-windev-ldapfixtures-runners.md.tasks.json new file mode 100644 index 0000000..06d790d --- /dev/null +++ b/docs/plans/2026-08-07-followups-windev-ldapfixtures-runners.md.tasks.json @@ -0,0 +1,13 @@ +{ + "planPath": "docs/plans/2026-08-07-followups-windev-ldapfixtures-runners.md", + "tasks": [ + {"id": 1, "subject": "Task 1: NEXT-07 — Recon windev deployment layout + schema support", "status": "completed"}, + {"id": 2, "subject": "Task 2: NEXT-07 — Build current Server on windev and redeploy the service", "status": "completed", "blockedBy": [1]}, + {"id": 3, "subject": "Task 3: SEC-36 deferred verification + NEXT-07/runbook closeout", "status": "completed", "blockedBy": [2]}, + {"id": 4, "subject": "Task 4: NEXT-06 — Fix DashboardLdapLiveTests fixtures to match the shared directory", "status": "completed"}, + {"id": 5, "subject": "Task 5: Resolve the unexpected macOS instance runner (id 4)", "status": "completed"}, + {"id": 6, "subject": "Task 6: Harden runner-1's registration token on 10.100.0.35", "status": "completed"}, + {"id": 7, "subject": "Task 7: Closeout — cargo Bearer verification, docs/tracker sync, commits", "status": "completed", "blockedBy": [3, 4, 5, 6]} + ], + "lastUpdated": "2026-08-07 (all tasks executed; SEC-36 verification done during Task 2's foreground smoke test; 8 commits local on main, not pushed; one pending operator action: Gitea registration-token UI reset)" +} diff --git a/docs/plans/2026-08-07-live-actions-sec36-tst30-publish.md b/docs/plans/2026-08-07-live-actions-sec36-tst30-publish.md new file mode 100644 index 0000000..2ba6f94 --- /dev/null +++ b/docs/plans/2026-08-07-live-actions-sec36-tst30-publish.md @@ -0,0 +1,420 @@ +# Live Actions: SEC-36 Rotation, TST-30 Second Runner, Client Publish — Implementation Plan + +> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task (or subagent-driven-development in-session). + +**Goal:** Execute the three repo-complete-but-live-pending operator actions: rotate the dev GLAuth service-account credential (SEC-36), register a second Gitea Actions runner (TST-30), and publish the five client packages at 0.2.0 (Java 0.2.1). + +**Architecture:** Three independent workstreams executed by subagents. SEC-36 is a strictly ordered cutover (pre-stage hosts → flip GLAuth → verify → finalize) with secret-hygiene rules. TST-30 is infra work on docker host 10.100.0.35 plus a concurrency verification. Publish runs the existing guarded `pack-clients.ps1 -Publish` + `tag-go-module.ps1` locally on macOS. + +**Tech Stack:** ssh (BatchMode works to 10.100.0.35 and 10.100.0.48), PowerShell/nssm on windev, docker compose on 10.100.0.35, Gitea API (`~/.zshenv` has admin-scoped `GITEA_USERNAME`/`GITEA_TOKEN`), pwsh 7 on macOS. + +--- + +## Preflight facts (verified 2026-08-07 from this macOS box) + +- `ssh 10.100.0.35` OK. GLAuth container is **`zb-shared-glauth`**, compose working dir **`/home/dohertj2/zb-glauth`** (NOT the runbook's `~/Desktop/scadaproj/infra/glauth` — that path does not exist on the host; the runbook must be corrected in Task 5). Runner container **`gitea-runner`**, compose working dir **`/opt/gitea`**. +- `ssh 10.100.0.48` (windev) OK; `powershell -NoProfile` works; `nssm` at `C:\Users\dohertj2\AppData\Local\Microsoft\WinGet\Links\nssm.exe`. +- `wonder-app-vd03` does NOT resolve from macOS — check it from windev (Task 2). +- Gitea API: token valid (`/api/v1/user` → 200), admin (`/api/v1/admin/users` → 200, `POST /api/v1/admin/actions/runners/registration-token` → 200). +- Local `~/Desktop/scadaproj/infra/glauth/config.toml` exists (14 `passsha256` entries) — the git source of truth. +- `pwsh` at `/usr/local/bin/pwsh`. + +## Secret hygiene (SEC-36, binding for every task) + +- The new plaintext password lives ONLY in `$SECRET_FILE = /private/tmp/claude-501/-Users-dohertj2-Desktop-MxAccessGateway/67849767-a07c-4afa-94e2-3ce4a39d8d23/scratchpad/sec36-new-secret` (chmod 600), created in Task 1 and shredded in Task 5. +- **Never echo/cat the plaintext to stdout, never put it in a commit, a repo file, a log line, or a command whose text is captured verbatim.** Always load it into a shell variable from the file (`val=$(cat "$SECRET_FILE")`) and pass it via stdin or remote-side expansion, never inline in an `ssh "...literal..."` string where avoidable. +- The `passsha256` hash MAY appear in `config.toml` commits — that is the established pattern (14 existing entries). +- The OLD password must never be printed either. Its only uses are: GLAuth keeps honoring it until Task 4, and the single old-bind-must-fail probe in Task 4. + +--- + +### Task 1: SEC-36 — Generate secret, stage GLAuth config change (repo + host copy diff) + +**Classification:** high-risk +**Estimated implement time:** ~5 min +**Parallelizable with:** Task 2, Task 6, Task 9 + +**Files:** +- Modify: `~/Desktop/scadaproj/infra/glauth/config.toml` (the `serviceaccount` user's `passsha256`) — DO NOT commit yet (Task 5 commits) +- Create: `$SECRET_FILE` (scratchpad, chmod 600) + +**Step 1: Generate the new secret and its hash** + +```bash +SECRET_FILE="/private/tmp/claude-501/-Users-dohertj2-Desktop-MxAccessGateway/67849767-a07c-4afa-94e2-3ce4a39d8d23/scratchpad/sec36-new-secret" +umask 077 +openssl rand -base64 24 | tr -d '\n' > "$SECRET_FILE" +chmod 600 "$SECRET_FILE" +NEW_SHA=$(cat "$SECRET_FILE" | tr -d '\n' | shasum -a 256 | awk '{print $1}') +echo "$NEW_SHA" # hash only — safe to display +``` +Cross-check the hash recipe against `glauth.md` ("Generate `passsha256` from a plaintext password") in this repo and follow that recipe if it differs. + +**Step 2: Diff host deployment config vs repo source of truth** + +```bash +ssh 10.100.0.35 'cat /home/dohertj2/zb-glauth/config.toml' > /tmp/host-glauth-config.toml 2>/dev/null || true +diff ~/Desktop/scadaproj/infra/glauth/config.toml /tmp/host-glauth-config.toml +``` +Small drift (comments, ports) is fine — note it. If the `serviceaccount` stanza differs structurally, STOP and surface before editing. + +**Step 3: Edit the repo source of truth** + +In `~/Desktop/scadaproj/infra/glauth/config.toml`, replace the `passsha256` value of the `[[users]]` entry whose `name`/`cn` is `serviceaccount` with `$NEW_SHA`. Edit ONLY that line. Do not `docker compose up` anything yet. + +**Step 4: Record findings** + +Report: hash staged (show hash, never plaintext), drift summary from step 2, and confirm `$SECRET_FILE` exists with mode 600. + +--- + +### Task 2: SEC-36 — Determine wonder-app-vd03 LDAP status (via windev) + +**Classification:** small +**Estimated implement time:** ~3 min +**Parallelizable with:** Task 1, Task 6, Task 9 + +**Step 1: Try to reach vd03 from windev** + +```bash +ssh 10.100.0.48 'powershell -NoProfile -Command "Test-Connection wonder-app-vd03 -Count 1 -Quiet"' +``` + +**Step 2: If reachable, read its gateway config for `MxGateway:Ldap:Enabled`** + +Try (in order, stop at first success): `ssh` hop from windev; reading `\\wonder-app-vd03\c$\...` appsettings/environment via PowerShell remoting (`Invoke-Command -ComputerName wonder-app-vd03`); or `nssm get MxAccessGw AppEnvironmentExtra` remotely. Look for `MxGateway__Ldap__Enabled` / appsettings `Ldap:Enabled`. + +**Step 3: Decide and record** + +- `Enabled=false` or host unreachable/no gateway service → vd03 is OUT of scope; record why (runbook says its dashboard is disabled — `false` is the expected answer). +- `Enabled=true` → vd03 is IN scope for Task 3 pre-staging; record the connection method that worked. + +--- + +### Task 3: SEC-36 — Pre-stage the NEW value on LDAP-enabled deployed hosts + +**Classification:** high-risk +**Estimated implement time:** ~4 min +**Parallelizable with:** none (blocked by Tasks 1, 2) + +**Step 1: Pre-stage windev (10.100.0.48)** + +Load the secret locally, then set the env var remotely without leaking it into logged command text more than unavoidable (ssh arguments are not logged remotely by default; do NOT echo the value): + +```bash +SECRET_FILE="/private/tmp/claude-501/-Users-dohertj2-Desktop-MxAccessGateway/67849767-a07c-4afa-94e2-3ce4a39d8d23/scratchpad/sec36-new-secret" +val=$(cat "$SECRET_FILE") +ssh 10.100.0.48 'powershell -NoProfile -Command "$v = [Console]::In.ReadLine(); $cur = (& nssm get MxAccessGw AppEnvironmentExtra) -join \"`n\"; Write-Output (\"CURRENT: \" + ($cur -replace \"Password=.*\", \"Password=<redacted>\")); & nssm set MxAccessGw AppEnvironmentExtra (\"MxGateway__Ldap__ServiceAccountPassword=\" + $v)"' <<< "$val" +``` +**CAUTION:** `nssm set AppEnvironmentExtra` REPLACES the whole extra-environment block. First inspect `nssm get MxAccessGw AppEnvironmentExtra` (redacting any `Password=` values); if other variables exist, preserve them in the new value (newline-separated). Adapt quoting as needed — verify with a redacted `nssm get` afterwards. + +**Step 2: Restart the service** + +```bash +ssh 10.100.0.48 'nssm restart MxAccessGw' +``` +Expected: service restarts. Binds against GLAuth now fail (old directory, new client value) — expected and brief; proceed immediately to Task 4. + +**Step 3: vd03 (only if Task 2 said IN scope)** — same pre-stage + restart via the method Task 2 found. + +--- + +### Task 4: SEC-36 — Rotate GLAuth and verify end-to-end + +**Classification:** high-risk +**Estimated implement time:** ~5 min +**Parallelizable with:** none (blocked by Task 3) + +**Step 1: Back up current host config, sync the staged config, recreate** + +```bash +ssh 10.100.0.35 'cp /home/dohertj2/zb-glauth/config.toml /home/dohertj2/zb-glauth/config.toml.bak-sec36' +scp ~/Desktop/scadaproj/infra/glauth/config.toml 10.100.0.35:/home/dohertj2/zb-glauth/config.toml +``` +**If Task 1's diff showed host-vs-repo drift beyond the serviceaccount line:** do NOT wholesale-copy — instead edit only the serviceaccount `passsha256` line in the host copy (sed on the host), so unrelated host-local drift is preserved. + +```bash +ssh 10.100.0.35 'cd /home/dohertj2/zb-glauth && docker compose up -d --force-recreate && sleep 3 && docker compose logs --tail 30' +``` +Expected: clean startup, no TOML parse error. On parse error: restore `.bak-sec36`, recreate, STOP, surface. + +**Step 2: Verify new credential binds (from the glauth host, ldapsearch or python)** + +```bash +SECRET_FILE=".../sec36-new-secret" # full scratchpad path +val=$(cat "$SECRET_FILE") +ssh 10.100.0.35 'ldapsearch -x -H ldap://localhost:3893 -D "cn=serviceaccount,dc=zb,dc=local" -w "$(cat -)" -b "dc=zb,dc=local" "(cn=multi-role)" cn' <<< "$val" +``` +Expected: search returns the `multi-role` entry. (If ldapsearch is missing on the host, run the equivalent from macOS against `10.100.0.35:3893`, or use `docker exec`.) Adjust the bind DN to match the actual `serviceaccount` DN in config.toml. + +**Step 3: Verify the OLD value is dead — exactly ONE probe, from 10.100.0.35 itself** + +One deliberately failing bind with the old password must return invalid credentials. **Only one attempt** (3-fail/10-min per-IP lockout; never probe from a shared-NAT box). The old value: recover it transiently from `config.toml.bak-sec36`'s hash? No — hash is not the plaintext. Instead: skip the plaintext probe if the old plaintext is not already known out-of-band; the hash replacement in config.toml is itself proof GLAuth no longer honors the old value (GLAuth compares against `passsha256` only). Record that reasoning instead of probing blind. + +**Step 4: Verify dashboard login end-to-end on windev** + +```bash +curl -sk -o /dev/null -w '%{http_code}' -c /tmp/mxgw-cookies.txt https://10.100.0.48:5001/login +``` +Find the actual dashboard port from windev config first (`nssm get`/appsettings; likely https). Then POST the login form as `multi-role`/`password` (the GLAuth TEST USER password, not the service account) and expect a redirect + `__Host-MxGatewayDashboard` (or `MxGatewayDashboard`) cookie: + +```bash +curl -sk -o /dev/null -w '%{http_code}\n' -b /tmp/mxgw-cookies.txt -c /tmp/mxgw-cookies.txt -d 'username=multi-role&password=password' <dashboard-base>/login +grep -i mxgatewaydashboard /tmp/mxgw-cookies.txt +``` +Inspect the login page HTML first for real form field names / antiforgery token; adapt. A successful `multi-role` login proves the service-account search bind works with the new credential end-to-end. If HTTP verification proves impractical (antiforgery), fall back to grepping the gateway log on windev for a successful LDAP bind/login line after attempting — or run the live-LDAP integration test from macOS: + +```bash +export MXGATEWAY_RUN_LIVE_LDAP_TESTS=1 +export MxGateway__Ldap__ServiceAccountPassword="$(cat "$SECRET_FILE")" +dotnet test src/ZB.MOM.WW.MxGateway.IntegrationTests/ZB.MOM.WW.MxGateway.IntegrationTests.csproj --filter FullyQualifiedName~DashboardLdapLiveTests +``` +Expected: green. (This binds from macOS to 10.100.0.35:3893 directly — it verifies the credential, and the curl/log check verifies windev.) + +**Step 5: Rollback (only on failure)** — restore `.bak-sec36` on the host, `docker compose up -d --force-recreate`, re-point windev's env var back (old value from where it was before — if unknown, STOP and surface), `nssm restart MxAccessGw`. + +--- + +### Task 5: SEC-36 — Finalize: commit source of truth, dev secrets, runbook fix, tracker, cleanup + +**Classification:** standard +**Estimated implement time:** ~5 min +**Parallelizable with:** none (blocked by Task 4) + +**Step 1: Commit and push the scadaproj glauth change (glauth paths ONLY)** + +```bash +cd ~/Desktop/scadaproj +git add infra/glauth/config.toml +git commit -m "sec(glauth): rotate serviceaccount passsha256 (mxaccessgw SEC-36)" +git push +``` +(`scadaproj` is a shared monorepo — stage only this path. If the worktree has unrelated staged changes, use `git commit -- infra/glauth/config.toml` style isolation.) + +**Step 2: Set dev user-secrets on this macOS box** + +```bash +cd ~/Desktop/MxAccessGateway +cat "$SECRET_FILE" | tr -d '\n' | dotnet user-secrets set "MxGateway:Ldap:ServiceAccountPassword" --project src/ZB.MOM.WW.MxGateway.Server/ZB.MOM.WW.MxGateway.Server.csproj +``` +(Check `dotnet user-secrets set -h` for stdin support; if unsupported, pass via `"$(cat "$SECRET_FILE")"` — acceptable, it's a local process arg.) + +**Step 3: Correct the runbook + flip tracker rows (mxaccessgw repo)** + +- `docs/runbooks/SEC-36-ldap-credential-rotation.md`: fix the host deployment path (`/home/dohertj2/zb-glauth`, container `zb-shared-glauth`; repo source of truth remains `scadaproj/infra/glauth/`), and note vd03's actual status per Task 2. +- Grep `archreview/2026-07-12/remediation/` for SEC-36 pending-operator rows; flip to Done citing the runbook + today's date. + +```bash +cd ~/Desktop/MxAccessGateway +grep -rn "SEC-36" archreview/2026-07-12/remediation/ docs/ | grep -iv binary +# edit the rows, then: +git add -A docs archreview && git commit -m "docs(sec-36): record live rotation done; correct runbook host paths" +``` + +**Step 4: Shred the secret file** + +```bash +rm -P "$SECRET_FILE" 2>/dev/null || rm "$SECRET_FILE" +``` + +**Step 5: Done-criteria check** — walk the runbook's Done criteria list; report each as met/not-met. + +--- + +### Task 6: TST-30 — Recon existing runner config on 10.100.0.35 + +**Classification:** small +**Estimated implement time:** ~4 min +**Parallelizable with:** Task 1, Task 2, Task 9 + +**Step 1: Inspect the existing runner** + +```bash +ssh 10.100.0.35 'cat /opt/gitea/docker-compose.yml 2>/dev/null || sudo cat /opt/gitea/docker-compose.yml; ls /opt/gitea' +ssh 10.100.0.35 'docker inspect gitea-runner --format "{{json .Mounts}}"; docker exec gitea-runner cat /config.yaml 2>/dev/null || true' +``` +Find: image/version, config file location (look for `container.network: traefik` and `capacity`/`maxParallel`), data volume, registration state file, docker socket mount, labels. + +**Step 2: Check host capacity** + +```bash +ssh 10.100.0.35 'nproc; free -h; df -h / | tail -1' +``` + +**Step 3: Decide (a)-variant** — second container vs raising `capacity` on the existing runner. Runbook prefers a second instance; if the existing runner's config shows a simple `capacity: 1` and resources are tight, raising capacity is the smaller change — but a second registered instance is the runbook default and survives one-runner wedge. Record the chosen variant, the exact compose/config snippets to reuse, and where the registration token goes. + +--- + +### Task 7: TST-30 — Register and start the second runner + +**Classification:** high-risk +**Estimated implement time:** ~5 min +**Parallelizable with:** none (blocked by Task 6) + +**Step 1: Mint an instance-level registration token** + +```bash +source ~/.zshenv +curl -s -X POST -u "$GITEA_USERNAME:$GITEA_TOKEN" 'https://gitea.dohertylan.com/api/v1/admin/actions/runners/registration-token' +``` +(Returns `{"token": "..."}` — a registration token, not a secret credential of lasting value; still avoid committing it.) + +**Step 2: Create the second runner instance per Task 6's plan** + +E.g. add a `gitea-runner-2` service to the compose (distinct name + data volume, same image, same `container.network: traefik`, same socket mount), inject the token via the runner's registration env (`GITEA_RUNNER_REGISTRATION_TOKEN`) or `act_runner register --no-interactive`, then `docker compose up -d gitea-runner-2` from `/opt/gitea`. Back up the compose file first (`cp docker-compose.yml docker-compose.yml.bak-tst30`). Do NOT touch the existing `gitea-runner` service definition. + +**Step 3: Confirm both runners online** + +```bash +curl -s -u "$GITEA_USERNAME:$GITEA_TOKEN" 'https://gitea.dohertylan.com/api/v1/admin/actions/runners' | python3 -m json.tool +``` +Expected: ≥2 runners, both online. Also check `docker logs` of the new container for a clean registration + poll loop. + +--- + +### Task 8: TST-30 — Verify concurrency, gitea:3000 resolution, tracker + +**Classification:** standard +**Estimated implement time:** ~5 min +**Parallelizable with:** none (blocked by Task 7) + +**Step 1: Trigger two concurrent runs** + +Push two scratch branches to `mxaccessgw` back-to-back (empty commits off `main`, branch names `scratch/tst30-a`, `scratch/tst30-b`): + +```bash +cd ~/Desktop/MxAccessGateway +git push origin main:refs/heads/scratch/tst30-a +git commit --allow-empty -m "tst30 concurrency probe" && git push origin HEAD:refs/heads/scratch/tst30-b && git reset --hard HEAD~1 +``` +(Adapt: any two pushes that fan out jobs. Clean up branches after: `git push origin :scratch/tst30-a :scratch/tst30-b`.) + +**Step 2: Confirm parallel execution** + +Poll the runs API/UI: the second run's jobs must START before the first run finishes. + +```bash +curl -s -u "$GITEA_USERNAME:$GITEA_TOKEN" 'https://gitea.dohertylan.com/api/v1/repos/dohertj2/mxaccessgw/actions/tasks' | python3 -m json.tool | head -60 +``` + +**Step 3: Confirm `gitea:3000` resolves on the new runner** — verify a job scheduled on runner-2 succeeds at checkout (checkout hits `gitea:3000` over the traefik network); identify which runner took each job from the runs UI/API or runner logs. + +**Step 4: Flip TST-30 tracker rows** in `archreview/2026-07-12/remediation/` (grep `TST-30`) to Done with today's date; confirm `docs/GatewayTesting.md` prose is still accurate (it should be — it already describes the bypass as valid regardless of runner count). Commit. + +--- + +### Task 9: Publish — Preflight audit (versions, registry collisions, toolchains) + +**Classification:** small +**Estimated implement time:** ~4 min +**Parallelizable with:** Task 1, Task 2, Task 6 + +**Step 1: Audit source versions** + +```bash +cd ~/Desktop/MxAccessGateway +grep -n 'version' clients/rust/Cargo.toml | head -5 +grep -n 'version' clients/python/pyproject.toml clients/python/src/zb_mom_ww_mxgateway/version.py +grep -n 'ClientVersion' clients/go/mxgateway/version.go +grep -n '<Version>' clients/dotnet/ZB.MOM.WW.MxGateway.Client/ZB.MOM.WW.MxGateway.Client.csproj src/ZB.MOM.WW.MxGateway.Contracts/ZB.MOM.WW.MxGateway.Contracts.csproj +grep -n 'version' clients/java/build.gradle | head -5 +grep -rn 'CLIENT_VERSION' clients/java --include=*.java | grep -i mxgatewayclientversion +``` +Expected: Rust/Python/Go/.NET/Contracts = 0.2.0; Java build.gradle AND `MxGatewayClientVersion.CLIENT_VERSION` = 0.2.1. Any mismatch → STOP, surface (do not bump versions yourself; that's a scope change). + +**Step 2: Query live registry for collisions** + +```bash +source ~/.zshenv +for u in 'nuget/ZB.MOM.WW.MxGateway.Client/0.2.0' 'nuget/ZB.MOM.WW.MxGateway.Contracts/0.2.0' 'pypi/zb-mom-ww-mxaccess-gateway-client/0.2.0' 'cargo/zb-mom-ww-mxgateway-client/0.2.0' 'maven/com.zb.mom.ww.mxgateway-zb-mom-ww-mxgateway-client/0.2.1'; do + echo "$u => $(curl -s -o /dev/null -w '%{http_code}' -u "$GITEA_USERNAME:$GITEA_TOKEN" "https://gitea.dohertylan.com/api/v1/packages/dohertj2/$u")" +done +``` +Expected: 404 for every target (unclaimed). Check the exact maven path convention against `pack-clients.ps1`'s own guard code and use its convention. 200 anywhere → STOP, surface. + +**Step 3: Toolchain + workspace check** + +```bash +git -C ~/Desktop/MxAccessGateway status --porcelain # must be clean (publish from a clean tree at origin/main) +for t in dotnet cargo go python3 gradle pwsh; do which $t; done +``` +Also confirm `clients/go` module tag `clients/go/v0.2.0` does NOT already exist: `git ls-remote --tags origin 'clients/go/v*'`. + +--- + +### Task 10: Publish — Run the guarded pack-and-publish + +**Classification:** high-risk +**Estimated implement time:** ~5 min dispatch (script runtime longer) +**Parallelizable with:** none (blocked by Task 9) + +**Step 1: Run pack-clients with publish** + +```bash +cd ~/Desktop/MxAccessGateway +source ~/.zshenv +pwsh -NoProfile -File scripts/pack-clients.ps1 -Publish 2>&1 | tee /private/tmp/claude-501/-Users-dohertj2-Desktop-MxAccessGateway/67849767-a07c-4afa-94e2-3ce4a39d8d23/scratchpad/pack-clients-publish.log +``` +Expected: per-language build+test+pack, collision guard prints "safe to publish" per artifact, uploads succeed. Timeout generously (Bash timeout 600000). If any language fails MID-loop, record exactly which artifacts pushed and which didn't — partial publish is the known failure mode; do not re-run blindly (re-run is safe only because the guard skips? NO — the guard ABORTS on existing versions. A re-run after partial publish will abort on the already-pushed artifact. If that happens, surface with the log; per-language `-Languages` selective re-run is the fix). +If macOS cannot build a language (e.g. gradle/java env), use `-Languages` to publish what builds and surface the remainder — do not fake success. + +**Step 2: Verify each artifact now exists (200)** — re-run Task 9 step 2's loop; expected 200 everywhere published. + +--- + +### Task 11: Publish — Go module tag + +**Classification:** small +**Estimated implement time:** ~3 min +**Parallelizable with:** Task 10 (blocked by Task 9) + +**Step 1: Tag via the guarded script** + +```bash +cd ~/Desktop/MxAccessGateway +pwsh -NoProfile -File scripts/tag-go-module.ps1 -Version 0.2.0 +``` +Read the script's param block first (`-Version` name may differ; it validates semver and that `version.go` matches, then creates+pushes `clients/go/v0.2.0`). Expected: tag created and pushed to origin. + +**Step 2: Verify** + +```bash +git ls-remote --tags origin 'clients/go/v0.2.0*' +``` +Expected: exactly one tag. Optionally `GOPROXY=direct go list -m gitea.dohertylan.com/dohertj2/mxaccessgw/clients/go@v0.2.0` from a temp dir. + +--- + +### Task 12: Publish — Docs/tracker closeout + +**Classification:** small +**Estimated implement time:** ~4 min +**Parallelizable with:** none (blocked by Tasks 10, 11) + +**Step 1:** Update `docs/ClientPackaging.md`'s versioning narrative if it claims 0.2.0/0.2.1 are unpublished (it currently records the maven 0.2.1 exception; add a dated line that 0.2.0 (Java 0.2.1) published on 2026-08-07). Grep `archreview/2026-07-12/remediation/` for publish/CLI-39 pending-operator rows and flip to Done. + +**Step 2:** Commit: + +```bash +cd ~/Desktop/MxAccessGateway +git add docs archreview && git commit -m "docs(clients): record 0.2.0/0.2.1 publish + close operator actions" +``` + +**Step 3:** Report the full publish matrix (artifact → version → registry HTTP status). + +--- + +## Dependency graph + +``` +{T1, T2} ──▶ T3 ──▶ T4 ──▶ T5 (SEC-36, strictly serial after recon) + T6 ──▶ T7 ──▶ T8 (TST-30) + T9 ──▶ {T10, T11} ──▶ T12 (Publish) +``` +The three streams are mutually independent and run concurrently. All subagents run with model=opus per operator instruction. + +## Out of scope (explicitly) + +- Option (b)/(c) runner topologies and the `concurrency:` ci.yml experiment (TST-30 runbook marks them escalation/optional). +- The five next-cycle candidate findings in `archreview/2026-07-12/remediation/90-candidate-findings-next-cycle.md`. +- Any client version bumps (versions are already landed; a mismatch is a STOP-and-surface). diff --git a/docs/plans/2026-08-07-live-actions-sec36-tst30-publish.md.tasks.json b/docs/plans/2026-08-07-live-actions-sec36-tst30-publish.md.tasks.json new file mode 100644 index 0000000..48caa2e --- /dev/null +++ b/docs/plans/2026-08-07-live-actions-sec36-tst30-publish.md.tasks.json @@ -0,0 +1,18 @@ +{ + "planPath": "docs/plans/2026-08-07-live-actions-sec36-tst30-publish.md", + "tasks": [ + {"id": 1, "subject": "Task 1: SEC-36 — Generate secret, stage GLAuth config change", "status": "completed"}, + {"id": 2, "subject": "Task 2: SEC-36 — Determine wonder-app-vd03 LDAP status via windev", "status": "completed"}, + {"id": 3, "subject": "Task 3: SEC-36 — Pre-stage NEW value on LDAP-enabled hosts (nssm + restart)", "status": "completed", "blockedBy": [1, 2]}, + {"id": 4, "subject": "Task 4: SEC-36 — Rotate GLAuth and verify end-to-end", "status": "completed", "blockedBy": [3]}, + {"id": 5, "subject": "Task 5: SEC-36 — Finalize: commit, dev secrets, runbook fix, tracker, cleanup", "status": "completed", "blockedBy": [4]}, + {"id": 6, "subject": "Task 6: TST-30 — Recon existing runner config on 10.100.0.35", "status": "completed"}, + {"id": 7, "subject": "Task 7: TST-30 — Register and start the second runner", "status": "completed", "blockedBy": [6]}, + {"id": 8, "subject": "Task 8: TST-30 — Verify concurrency + gitea:3000 + tracker", "status": "completed", "blockedBy": [7]}, + {"id": 9, "subject": "Task 9: Publish — Preflight audit (versions, collisions, toolchains)", "status": "completed"}, + {"id": 10, "subject": "Task 10: Publish — Run pack-clients.ps1 -Publish", "status": "completed", "blockedBy": [9]}, + {"id": 11, "subject": "Task 11: Publish — Go module tag clients/go/v0.2.0", "status": "completed", "blockedBy": [9]}, + {"id": 12, "subject": "Task 12: Publish — Docs/tracker closeout", "status": "completed", "blockedBy": [10, 11]} + ], + "lastUpdated": "2026-08-07 (all tasks executed; 4 closeout commits local on main, not pushed)" +} diff --git a/gateway.md b/gateway.md index cc7047d..6c59f51 100644 --- a/gateway.md +++ b/gateway.md @@ -299,9 +299,16 @@ Default transport: one bidirectional named pipe per worker. Pipe name: ```text -mxaccess-gateway-{gatewayProcessId}-{sessionId} +mxgw-{gatewayProcessId}-{sessionUid} ``` +`sessionUid` is the session id without its `session-` prefix (the raw guid hex). +The name is deliberately short: on Unix-like hosts (the macOS/Linux test +matrix), .NET named pipes are Unix domain sockets at +`$TMPDIR/CoreFxPipe_{name}`, and macOS caps the socket path at 104 bytes while +its default per-user `TMPDIR` already spends ~49 of them. The gateway PID keeps +the name collision-free across gateway restarts. + Message framing: ```text diff --git a/src/ZB.MOM.WW.MxGateway.Server/Sessions/SessionManager.cs b/src/ZB.MOM.WW.MxGateway.Server/Sessions/SessionManager.cs index 23abece..f5c9a6d 100644 --- a/src/ZB.MOM.WW.MxGateway.Server/Sessions/SessionManager.cs +++ b/src/ZB.MOM.WW.MxGateway.Server/Sessions/SessionManager.cs @@ -417,7 +417,8 @@ public sealed class SessionManager : ISessionManager string? clientIdentity, string? ownerKeyId) { - string sessionId = CreateSessionId(); + string sessionUid = Guid.NewGuid().ToString("N"); + string sessionId = $"session-{sessionUid}"; string backendName = string.IsNullOrWhiteSpace(request.RequestedBackend) ? GatewayContractInfo.DefaultBackendName : request.RequestedBackend!; @@ -425,7 +426,11 @@ public sealed class SessionManager : ISessionManager TimeSpan startupTimeout = TimeSpan.FromSeconds(_options.Worker.StartupTimeoutSeconds); TimeSpan shutdownTimeout = TimeSpan.FromSeconds(_options.Worker.ShutdownTimeoutSeconds); TimeSpan leaseDuration = TimeSpan.FromSeconds(_options.Sessions.DefaultLeaseSeconds); - string pipeName = $"mxaccess-gateway-{Environment.ProcessId}-{sessionId}"; + // The short prefix and bare guid keep the pipe's Unix-domain-socket path + // (TMPDIR + "CoreFxPipe_" + name) inside the 104-byte sun_path limit on + // macOS, whose default per-user TMPDIR is ~49 chars; the gateway PID keeps + // the name collision-free across gateway restarts (NEXT-01). + string pipeName = $"mxgw-{Environment.ProcessId}-{sessionUid}"; string nonce = CreateNonce(); DateTimeOffset openedAt = _timeProvider.GetUtcNow(); string clientCorrelationId = CreateClientCorrelationId(request.ClientSessionName, sessionId); @@ -484,11 +489,6 @@ public sealed class SessionManager : ISessionManager : timeout; } - private static string CreateSessionId() - { - return $"session-{Guid.NewGuid():N}"; - } - private static string CreateNonce() { Span<byte> bytes = stackalloc byte[32]; diff --git a/src/ZB.MOM.WW.MxGateway.Tests/Gateway/Sessions/SessionManagerTests.cs b/src/ZB.MOM.WW.MxGateway.Tests/Gateway/Sessions/SessionManagerTests.cs index 1ac81c7..397de15 100644 --- a/src/ZB.MOM.WW.MxGateway.Tests/Gateway/Sessions/SessionManagerTests.cs +++ b/src/ZB.MOM.WW.MxGateway.Tests/Gateway/Sessions/SessionManagerTests.cs @@ -37,6 +37,36 @@ public sealed class SessionManagerTests Assert.Equal(1, metrics.GetSnapshot().SessionsOpened); } + /// <summary> + /// Verifies the pipe name stays short enough that its Unix-domain-socket path + /// (TMPDIR + "CoreFxPipe_" + name) fits the 104-byte macOS sun_path limit under the + /// default per-user TMPDIR (~49 chars), and keeps the pid + session-guid uniqueness + /// contract (NEXT-01). + /// </summary> + /// <returns>A task that represents the asynchronous operation.</returns> + [Fact] + public async Task OpenSessionAsync_PipeNameIsShortAndUniquePerPidAndSession() + { + FakeWorkerClient workerClient = new(); + FakeSessionWorkerClientFactory factory = new(workerClient) + { + ApplyLifecycleTransitions = true, + }; + SessionManager manager = CreateManager(factory); + + GatewaySession session = await manager.OpenSessionAsync(CreateOpenRequest(), "client-1", ownerKeyId: null, CancellationToken.None); + + Assert.Matches($"^mxgw-{Environment.ProcessId}-[0-9a-f]{{32}}$", session.PipeName); + Assert.EndsWith(session.SessionId["session-".Length..], session.PipeName, StringComparison.Ordinal); + + // 104-byte sun_path − NUL − ~49-char default macOS TMPDIR − "CoreFxPipe_". + const int MaxPipeNameLength = 104 - 1 - 49 - 11; + int worstCasePidDigits = 5 - Environment.ProcessId.ToString(System.Globalization.CultureInfo.InvariantCulture).Length; + Assert.True( + session.PipeName.Length + Math.Max(0, worstCasePidDigits) <= MaxPipeNameLength, + $"Pipe name '{session.PipeName}' would overflow the macOS socket-path budget at a 5-digit pid."); + } + /// <summary>Verifies that a session opened by an authenticated caller records that caller's API key id in OwnerKeyId.</summary> /// <returns>A task that represents the asynchronous operation.</returns> [Fact] diff --git a/src/ZB.MOM.WW.MxGateway.Tests/Gateway/Sessions/SessionWorkerClientFactoryFakeWorkerTests.cs b/src/ZB.MOM.WW.MxGateway.Tests/Gateway/Sessions/SessionWorkerClientFactoryFakeWorkerTests.cs index ed808bc..baeb902 100644 --- a/src/ZB.MOM.WW.MxGateway.Tests/Gateway/Sessions/SessionWorkerClientFactoryFakeWorkerTests.cs +++ b/src/ZB.MOM.WW.MxGateway.Tests/Gateway/Sessions/SessionWorkerClientFactoryFakeWorkerTests.cs @@ -138,7 +138,7 @@ public sealed class SessionWorkerClientFactoryFakeWorkerTests : IAsyncDisposable return new GatewaySession( FakeWorkerHarness.DefaultSessionId, GatewayContractInfo.DefaultBackendName, - $"mxaccessgw-session-fake-worker-{Guid.NewGuid():N}", + $"mxgw-sf-{Guid.NewGuid():N}", FakeWorkerHarness.DefaultNonce, "test-client", "fake-worker-session-test", diff --git a/src/ZB.MOM.WW.MxGateway.Tests/Gateway/Workers/Fakes/FakeWorkerHarness.cs b/src/ZB.MOM.WW.MxGateway.Tests/Gateway/Workers/Fakes/FakeWorkerHarness.cs index 10c5d05..64bfc2a 100644 --- a/src/ZB.MOM.WW.MxGateway.Tests/Gateway/Workers/Fakes/FakeWorkerHarness.cs +++ b/src/ZB.MOM.WW.MxGateway.Tests/Gateway/Workers/Fakes/FakeWorkerHarness.cs @@ -60,7 +60,7 @@ public sealed class FakeWorkerHarness : IAsyncDisposable int maxMessageBytes = WorkerFrameProtocolOptions.DefaultMaxMessageBytes, CancellationToken cancellationToken = default) { - string pipeName = $"mxaccessgw-fake-worker-{Guid.NewGuid():N}"; + string pipeName = $"mxgw-fw-{Guid.NewGuid():N}"; NamedPipeServerStream gatewayStream = new( pipeName, PipeDirection.InOut, diff --git a/src/ZB.MOM.WW.MxGateway.Tests/Gateway/Workers/WorkerClientTests.cs b/src/ZB.MOM.WW.MxGateway.Tests/Gateway/Workers/WorkerClientTests.cs index 3be1668..f8b3796 100644 --- a/src/ZB.MOM.WW.MxGateway.Tests/Gateway/Workers/WorkerClientTests.cs +++ b/src/ZB.MOM.WW.MxGateway.Tests/Gateway/Workers/WorkerClientTests.cs @@ -1069,7 +1069,7 @@ public sealed class WorkerClientTests /// <returns>The connected <see cref="PipePair"/>.</returns> public static async Task<PipePair> CreateAsync() { - string pipeName = $"mxaccessgw-workerclient-tests-{Guid.NewGuid():N}"; + string pipeName = $"mxgw-wc-{Guid.NewGuid():N}"; NamedPipeServerStream gatewayStream = new( pipeName, PipeDirection.InOut, From 84dbf20a43288c311b65c35bc0f44a1d4f0d4f5d Mon Sep 17 00:00:00 2001 From: Joseph Doherty <dohejw01@gmail.com> Date: Mon, 10 Aug 2026 05:57:53 -0400 Subject: [PATCH 3/6] fix(worker): observe faults on frames abandoned by cancellation (NEXT-04, NEXT-05 decision) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A WriteAsync/WriteBatchAsync caller cancelled after the draining lock-holder claimed its frame unwinds without awaiting that frame's completion; the same holds for a frame already faulted by a concurrent FailAllQueued, where TrySetCanceled loses. A later wire-write failure then lands TrySetException on a task with no awaiter and surfaces as TaskScheduler.UnobservedTaskException. The tombstone helpers now attach a fault-observing continuation to every frame in the cancelled call (a cancelled task never fires OnlyOnFaulted, so unconditional attach is safe), outside _gate because an already-faulted task runs the continuation inline. NEXT-05 is resolved as a documented decision, not a code change: tombstoned entries keep their lazy DequeueNext purge — any subsequent write drains both queues to empty and the heartbeat loop bounds residency to one interval, while eager Queue<T> rebuilds under _gate would add ordering-invariant surface for no gain. Rationale recorded in docs/WorkerFrameProtocol.md alongside the WRK-22 residual-window contract. New regression test drives the exact abandonment: gated stream holds writer A mid-write, the queued event frame is claimed and blocked mid-write, its caller is cancelled, the write then faults with a marker exception, and the test asserts the marker never reaches UnobservedTaskException after a forced GC. net48 x86 build/test runs on windev with the rest of this batch. --- docs/WorkerFrameProtocol.md | 13 ++ .../Ipc/WorkerFrameProtocolTests.cs | 127 ++++++++++++++++++ .../Ipc/WorkerFrameWriter.cs | 32 ++++- 3 files changed, 171 insertions(+), 1 deletion(-) diff --git a/docs/WorkerFrameProtocol.md b/docs/WorkerFrameProtocol.md index a23631b..ea1aa59 100644 --- a/docs/WorkerFrameProtocol.md +++ b/docs/WorkerFrameProtocol.md @@ -147,6 +147,19 @@ so the caller observes `OperationCanceledException` while that one frame still reaches the wire. That residual window is by design: blocking the canceller behind the very write it is abandoning would defeat the point of cancellation. +Two hygiene notes on that residual (NEXT-04/NEXT-05). First, a frame the +cancelled caller abandons — claimed mid-write, or already faulted by a +concurrent queue-wide failure — completes on a task nobody awaits; the +tombstone path attaches a fault-observing continuation to it so a later write +failure never surfaces as a `TaskScheduler.UnobservedTaskException`. Second, +tombstoned entries stay in the class queues until a future `DequeueNext` pops +and skips them; that lazy purge is deliberate. Eagerly rebuilding a `Queue<T>` +under `_gate` on every cancellation would add ordering-invariant surface next +to the claim/cancel interlock for no real gain: any subsequent write of either +class drains both queues to empty, and the heartbeat loop guarantees one +arrives within a heartbeat interval, so worst-case residency is a few envelope +references for seconds — not a leak. + ## Verification The frame protocol lives in `ZB.MOM.WW.MxGateway.Worker.Ipc` (`WorkerFrameReader`, diff --git a/src/ZB.MOM.WW.MxGateway.Worker.Tests/Ipc/WorkerFrameProtocolTests.cs b/src/ZB.MOM.WW.MxGateway.Worker.Tests/Ipc/WorkerFrameProtocolTests.cs index 334e4a9..bdeea32 100644 --- a/src/ZB.MOM.WW.MxGateway.Worker.Tests/Ipc/WorkerFrameProtocolTests.cs +++ b/src/ZB.MOM.WW.MxGateway.Worker.Tests/Ipc/WorkerFrameProtocolTests.cs @@ -545,6 +545,70 @@ public sealed class WorkerFrameProtocolTests Assert.Equal(2UL, frame2.Sequence); } + /// <summary> + /// NEXT-04. A frame claimed by the draining lock-holder before its caller's cancellation lands + /// is abandoned — the cancelled caller never awaits its completion. If the wire write then + /// faults, the tombstone path's fault-observing continuation must still observe the exception + /// so it never surfaces as <see cref="TaskScheduler.UnobservedTaskException"/>. + /// </summary> + /// <returns>A task that represents the asynchronous operation.</returns> + [Fact] + public async Task WriteAsync_ClaimedFrameAbandonedByCancellation_FaultIsObserved() + { + string marker = $"NEXT-04-{Guid.NewGuid():N}"; + WorkerFrameProtocolOptions options = CreateOptions(); + bool sawUnobservedMarkerFault = false; + EventHandler<UnobservedTaskExceptionEventArgs> handler = (sender, args) => + { + if (args.Exception.ToString().Contains(marker)) + { + sawUnobservedMarkerFault = true; + } + }; + + TaskScheduler.UnobservedTaskException += handler; + try + { + using (SecondWriteFaultingGatedStream stream = new SecondWriteFaultingGatedStream(marker)) + { + WorkerFrameWriter writer = new WorkerFrameWriter(stream, options); + + // Writer A holds the lock, blocked mid-write of its own frame. + Task firstWrite = writer.WriteAsync(CreateGatewayHelloEnvelope(), WorkerFrameWritePriority.Control); + await AwaitWithTimeoutAsync(stream.FirstWriteStarted); + + using (CancellationTokenSource cts = new CancellationTokenSource()) + { + // Queue the doomed event write behind A, release A so its drain claims the + // event frame and blocks mid-write of it, then cancel the queued caller — + // the frame is claimed, so the caller unwinds without an awaiter for it. + Task abandonedWrite = writer.WriteAsync(CreateEventEnvelope(), WorkerFrameWritePriority.Event, cts.Token); + stream.ReleaseFirstWrite(); + await AwaitWithTimeoutAsync(stream.SecondWriteStarted); + cts.Cancel(); + await Assert.ThrowsAnyAsync<OperationCanceledException>(async () => await abandonedWrite); + } + + // Fault the abandoned frame's wire write; observe writer A's own outcome so only + // the abandoned frame's completion could ever raise the marker unobserved. + stream.ReleaseSecondWrite(); + _ = await Record.ExceptionAsync(async () => await firstWrite); + } + + GC.Collect(); + GC.WaitForPendingFinalizers(); + GC.Collect(); + } + finally + { + TaskScheduler.UnobservedTaskException -= handler; + } + + Assert.False( + sawUnobservedMarkerFault, + "The abandoned frame's write fault surfaced as an unobserved-task exception."); + } + /// <summary> /// WRK-22 / IPC-26, the review's shutdown scenario. A cancelled event frame queued before a /// shutdown-ack control frame must not trail the ack on the wire: the tombstone rule plus the @@ -749,6 +813,69 @@ public sealed class WorkerFrameProtocolTests } } + // A MemoryStream whose first write blocks until released and whose second write blocks until + // released and then throws, so a test can abandon a claimed frame by cancellation and fault its + // wire write afterwards (NEXT-04). + private sealed class SecondWriteFaultingGatedStream : MemoryStream + { + private readonly SemaphoreSlim _firstRelease = new SemaphoreSlim(0); + private readonly SemaphoreSlim _secondRelease = new SemaphoreSlim(0); + private readonly TaskCompletionSource<bool> _firstWriteStarted = + new TaskCompletionSource<bool>(TaskCreationOptions.RunContinuationsAsynchronously); + private readonly TaskCompletionSource<bool> _secondWriteStarted = + new TaskCompletionSource<bool>(TaskCreationOptions.RunContinuationsAsynchronously); + private readonly string _faultMessage; + private int _writeCount; + + public SecondWriteFaultingGatedStream(string faultMessage) + { + _faultMessage = faultMessage; + } + + /// <summary>Gets a task that completes once the first <see cref="WriteAsync"/> call has started blocking.</summary> + public Task FirstWriteStarted => _firstWriteStarted.Task; + + /// <summary>Gets a task that completes once the second <see cref="WriteAsync"/> call has started blocking.</summary> + public Task SecondWriteStarted => _secondWriteStarted.Task; + + /// <summary>Releases the first blocked write so it can complete.</summary> + public void ReleaseFirstWrite() => _firstRelease.Release(); + + /// <summary>Releases the second blocked write so it can throw.</summary> + public void ReleaseSecondWrite() => _secondRelease.Release(); + + /// <inheritdoc /> + public override async Task WriteAsync(byte[] buffer, int offset, int count, CancellationToken cancellationToken) + { + int writeIndex = Interlocked.Increment(ref _writeCount); + if (writeIndex == 1) + { + _firstWriteStarted.TrySetResult(true); + await _firstRelease.WaitAsync(cancellationToken); + } + else if (writeIndex == 2) + { + _secondWriteStarted.TrySetResult(true); + await _secondRelease.WaitAsync(cancellationToken); + throw new IOException(_faultMessage); + } + + await base.WriteAsync(buffer, offset, count, cancellationToken); + } + + /// <inheritdoc /> + protected override void Dispose(bool disposing) + { + if (disposing) + { + _firstRelease.Dispose(); + _secondRelease.Dispose(); + } + + base.Dispose(disposing); + } + } + // A MemoryStream whose first WriteAsync blocks until released, so a test can queue additional frames // behind an in-progress write and observe the writer's priority ordering. private sealed class GatedWriteStream : MemoryStream diff --git a/src/ZB.MOM.WW.MxGateway.Worker/Ipc/WorkerFrameWriter.cs b/src/ZB.MOM.WW.MxGateway.Worker/Ipc/WorkerFrameWriter.cs index 6061f00..7fab801 100644 --- a/src/ZB.MOM.WW.MxGateway.Worker/Ipc/WorkerFrameWriter.cs +++ b/src/ZB.MOM.WW.MxGateway.Worker/Ipc/WorkerFrameWriter.cs @@ -92,6 +92,8 @@ public sealed class WorkerFrameWriter /// already claimed it, in which case the frame may still reach the wire even though this call /// observes <see cref="OperationCanceledException"/>. That residual window is by design: blocking /// the canceller behind the very write it is abandoning would defeat the point of cancellation. + /// The abandoned frame's completion gets a fault-observing continuation so a write failure after + /// the caller unwinds never raises an unobserved-task exception (NEXT-04). /// </remarks> public async Task WriteAsync( WorkerEnvelope envelope, @@ -162,7 +164,9 @@ public sealed class WorkerFrameWriter /// awaited completions as its <see cref="WorkerFrameProtocolException"/>; the remaining frames are /// still observed so none faults unobserved. Cancellation while waiting for the lock tombstones /// every still-unclaimed frame in the batch, per the WRK-22 contract on - /// <see cref="WriteAsync(WorkerEnvelope, WorkerFrameWritePriority, CancellationToken)"/>. + /// <see cref="WriteAsync(WorkerEnvelope, WorkerFrameWritePriority, CancellationToken)"/>; frames + /// the cancelled caller abandons (claimed mid-write, or already faulted) get a fault-observing + /// continuation so a later write failure never raises an unobserved-task exception (NEXT-04). /// </remarks> public async Task WriteBatchAsync( IReadOnlyList<WorkerEnvelope> envelopes, @@ -245,6 +249,8 @@ public sealed class WorkerFrameWriter frame.Completion.TrySetCanceled(cancellationToken); } } + + ObserveAbandonedFault(frame); } private void TombstoneUnclaimed(PendingFrame[] frames, CancellationToken cancellationToken) @@ -259,6 +265,30 @@ public sealed class WorkerFrameWriter } } } + + foreach (PendingFrame frame in frames) + { + ObserveAbandonedFault(frame); + } + } + + /// <summary> + /// Observes any fault on a frame the cancelled caller stops awaiting (NEXT-04). A frame + /// claimed by a draining lock-holder — or already faulted by a concurrent + /// <c>FailAllQueued</c> — completes on a task nobody awaits after cancellation unwinds the + /// caller; a later write failure would then surface as an unobserved-task exception. A + /// cancelled task never triggers the faulted continuation, so attaching unconditionally is + /// safe. Attached outside <c>_gate</c> because an already-faulted task runs the + /// continuation inline. + /// </summary> + /// <param name="frame">Frame whose completion may fault without an awaiter.</param> + private static void ObserveAbandonedFault(PendingFrame frame) + { + _ = frame.Completion.Task.ContinueWith( + task => _ = task.Exception, + CancellationToken.None, + TaskContinuationOptions.OnlyOnFaulted | TaskContinuationOptions.ExecuteSynchronously, + TaskScheduler.Default); } // Runs only under _writeLock. Drains control frames before event frames, stamping and writing each. From 8624e21372167678d72389605c71320f6a466264 Mon Sep 17 00:00:00 2001 From: Joseph Doherty <dohejw01@gmail.com> Date: Mon, 10 Aug 2026 06:01:44 -0400 Subject: [PATCH 4/6] fix(clients): render ReplayGap as the typed cross-CLI row in the .NET and Java CLIs (NEXT-02) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The .NET and Java stream-events commands handed the raw ReplayGap sentinel MxEvent to their protobuf JSON formatters (Java text mode printed '0 MX_EVENT_FAMILY_UNSPECIFIED'), while the Go/Python/Rust CLIs already emit the typed row (CLI-35/36). Both now branch on the sentinel: Java text mode prints 'REPLAY_GAP requested_after=<n> oldest_available=<n>' and JSON mode a hand-built {"replayGap":{...}} line via nextItem()/isReplayGap(); the .NET CLI emits the same hand-built row in jsonl/text (its text mode is JSON-per-line) and inside the --json events array. Rows are hand-built so the cursors are JSON numbers like the other three CLIs, not the proto3 JSON mapping's quoted uint64 strings — the CrossLanguageSmokeMatrix divergence table collapses to a single converged contract. Tests: .NET MxGatewayClientCliTests 35/35 (new RendersReplayGapAsTypedRow covers jsonl + aggregate); Java gradle test 52/52 (new streamEventsRendersReplayGapAsTypedRow covers --json + text over the in-process harness). No generated-file churn. --- .../MxGatewayClientCli.cs | 38 +++++++-- .../MxGatewayClientCliTests.cs | 79 +++++++++++++++++++ .../zb/mom/ww/mxgateway/cli/MxGatewayCli.java | 29 ++++++- .../ww/mxgateway/cli/MxGatewayCliTests.java | 54 +++++++++++++ docs/CrossLanguageSmokeMatrix.md | 25 +++--- 5 files changed, 201 insertions(+), 24 deletions(-) diff --git a/clients/dotnet/ZB.MOM.WW.MxGateway.Client.Cli/MxGatewayClientCli.cs b/clients/dotnet/ZB.MOM.WW.MxGateway.Client.Cli/MxGatewayClientCli.cs index 5320e3c..cdd6e7d 100644 --- a/clients/dotnet/ZB.MOM.WW.MxGateway.Client.Cli/MxGatewayClientCli.cs +++ b/clients/dotnet/ZB.MOM.WW.MxGateway.Client.Cli/MxGatewayClientCli.cs @@ -1418,14 +1418,16 @@ public static class MxGatewayClientCli .WithCancellation(cancellationToken) .ConfigureAwait(false)) { - if (jsonLines) - { - output.WriteLine(ProtobufJsonFormatter.Format(gatewayEvent)); - } - else if (json) + if (json && !jsonLines) { events.Add(gatewayEvent); } + else if (gatewayEvent.ReplayGap is { } replayGap) + { + // Render the ReplayGap sentinel as the typed cross-CLI row instead of the raw + // sentinel MxEvent (NEXT-02, mirroring the Go/Python/Rust CLIs). + output.WriteLine(FormatReplayGapRow(replayGap)); + } else { output.WriteLine(ProtobufJsonFormatter.Format(gatewayEvent)); @@ -1835,7 +1837,31 @@ public static class MxGatewayClientCli private static JsonElement EventToJsonElement(MxEvent gatewayEvent) { - return JsonDocument.Parse(ProtobufJsonFormatter.Format(gatewayEvent)).RootElement.Clone(); + string row = gatewayEvent.ReplayGap is { } replayGap + ? FormatReplayGapRow(replayGap) + : ProtobufJsonFormatter.Format(gatewayEvent); + return JsonDocument.Parse(row).RootElement.Clone(); + } + + /// <summary> + /// Formats the typed ReplayGap row shared by the CLIs (NEXT-02). Hand-built so the + /// cursors are JSON numbers like the Go/Python/Rust rows, not the protobuf JSON + /// formatter's quoted uint64 strings. + /// </summary> + /// <param name="replayGap">Replay gap sentinel payload.</param> + /// <returns>A single-line JSON row describing the gap.</returns> + private static string FormatReplayGapRow(ReplayGap replayGap) + { + return JsonSerializer.Serialize( + new + { + replayGap = new + { + requestedAfterSequence = replayGap.RequestedAfterSequence, + oldestAvailableSequence = replayGap.OldestAvailableSequence, + }, + }, + JsonOptions); } private static MxValue ParseValue(CliArguments arguments) diff --git a/clients/dotnet/ZB.MOM.WW.MxGateway.Client.Tests/MxGatewayClientCliTests.cs b/clients/dotnet/ZB.MOM.WW.MxGateway.Client.Tests/MxGatewayClientCliTests.cs index 2a98d75..726f591 100644 --- a/clients/dotnet/ZB.MOM.WW.MxGateway.Client.Tests/MxGatewayClientCliTests.cs +++ b/clients/dotnet/ZB.MOM.WW.MxGateway.Client.Tests/MxGatewayClientCliTests.cs @@ -1,3 +1,4 @@ +using System.Text.Json; using Google.Protobuf.WellKnownTypes; using ZB.MOM.WW.MxGateway.Client.Cli; using ZB.MOM.WW.MxGateway.Contracts.Proto; @@ -585,6 +586,84 @@ public sealed class MxGatewayClientCliTests Assert.DoesNotContain("ON_WRITE_COMPLETE", output.ToString()); } + /// <summary> + /// Verifies stream-events renders the ReplayGap sentinel as the typed cross-CLI row — + /// numeric cursors under a replayGap key — instead of the raw sentinel MxEvent (NEXT-02). + /// </summary> + /// <returns>A task that represents the asynchronous operation.</returns> + [Fact] + public async Task RunAsync_StreamEvents_RendersReplayGapAsTypedRow() + { + using var output = new StringWriter(); + using var error = new StringWriter(); + FakeCliClient fakeClient = new(); + fakeClient.Events.Add(new MxEvent + { + ReplayGap = new ReplayGap + { + RequestedAfterSequence = 7, + OldestAvailableSequence = 42, + }, + }); + fakeClient.Events.Add(new MxEvent + { + SessionId = "session-fixture", + Family = MxEventFamily.OnDataChange, + WorkerSequence = 43, + }); + + int exitCode = await MxGatewayClientCli.RunAsync( + [ + "stream-events", + "--endpoint", + "http://localhost:5000", + "--api-key", + "test-api-key", + "--session-id", + "session-fixture", + "--max-events", + "2", + ], + output, + error, + _ => fakeClient); + + Assert.Equal(0, exitCode); + string[] rows = output.ToString().Split(Environment.NewLine, StringSplitOptions.RemoveEmptyEntries); + Assert.Equal(2, rows.Length); + using JsonDocument gapRow = JsonDocument.Parse(rows[0]); + JsonElement gap = gapRow.RootElement.GetProperty("replayGap"); + Assert.Equal(7UL, gap.GetProperty("requestedAfterSequence").GetUInt64()); + Assert.Equal(42UL, gap.GetProperty("oldestAvailableSequence").GetUInt64()); + Assert.Equal(JsonValueKind.Number, gap.GetProperty("requestedAfterSequence").ValueKind); + Assert.DoesNotContain("MX_EVENT_FAMILY_UNSPECIFIED", rows[0], StringComparison.Ordinal); + Assert.Contains("workerSequence", rows[1], StringComparison.Ordinal); + + // The aggregate --json shape carries the same typed row inside the events array. + using var aggregateOutput = new StringWriter(); + int aggregateExit = await MxGatewayClientCli.RunAsync( + [ + "stream-events", + "--endpoint", + "http://localhost:5000", + "--api-key", + "test-api-key", + "--session-id", + "session-fixture", + "--max-events", + "2", + "--json", + ], + aggregateOutput, + error, + _ => fakeClient); + + Assert.Equal(0, aggregateExit); + using JsonDocument aggregate = JsonDocument.Parse(aggregateOutput.ToString()); + JsonElement firstRow = aggregate.RootElement.GetProperty("events")[0]; + Assert.Equal(42UL, firstRow.GetProperty("replayGap").GetProperty("oldestAvailableSequence").GetUInt64()); + } + /// <summary>Verifies that stream-alarms with --max-events stops output and distinguishes payload cases.</summary> /// <returns>A task that represents the asynchronous operation.</returns> diff --git a/clients/java/zb-mom-ww-mxgateway-cli/src/main/java/com/zb/mom/ww/mxgateway/cli/MxGatewayCli.java b/clients/java/zb-mom-ww-mxgateway-cli/src/main/java/com/zb/mom/ww/mxgateway/cli/MxGatewayCli.java index 9bb0f43..3fe632e 100644 --- a/clients/java/zb-mom-ww-mxgateway-cli/src/main/java/com/zb/mom/ww/mxgateway/cli/MxGatewayCli.java +++ b/clients/java/zb-mom-ww-mxgateway-cli/src/main/java/com/zb/mom/ww/mxgateway/cli/MxGatewayCli.java @@ -5,6 +5,7 @@ import com.zb.mom.ww.mxgateway.client.DeployEventStream; import com.zb.mom.ww.mxgateway.client.GalaxyRepositoryClient; import com.zb.mom.ww.mxgateway.client.LazyBrowseNode; import com.zb.mom.ww.mxgateway.client.MxEventStream; +import com.zb.mom.ww.mxgateway.client.MxEventStreamItem; import com.zb.mom.ww.mxgateway.client.MxGatewayAlarmFeedSubscription; import com.zb.mom.ww.mxgateway.client.MxGatewayClient; import com.zb.mom.ww.mxgateway.client.MxGatewayClientOptions; @@ -59,6 +60,7 @@ import mxaccess_gateway.v1.MxaccessGateway.MxValue; import mxaccess_gateway.v1.MxaccessGateway.OnAlarmTransitionEvent; import mxaccess_gateway.v1.MxaccessGateway.OpenSessionRequest; import mxaccess_gateway.v1.MxaccessGateway.PingCommand; +import mxaccess_gateway.v1.MxaccessGateway.ReplayGap; import mxaccess_gateway.v1.MxaccessGateway.StreamAlarmsRequest; import mxaccess_gateway.v1.MxaccessGateway.SubscribeResult; import mxaccess_gateway.v1.MxaccessGateway.Write2BulkEntry; @@ -1654,11 +1656,30 @@ public final class MxGatewayCli implements Callable<Integer> { MxEventStream events = client.session(sessionId).streamEventsAfter(afterWorkerSequence)) { int count = 0; while (events.hasNext()) { - MxEvent event = events.next(); - if (json) { - client.out().println(protoJson(event)); + MxEventStreamItem item = events.nextItem(); + if (item.isReplayGap()) { + // Render the ReplayGap sentinel as the typed cross-CLI row (NEXT-02, + // mirroring the Go/Python/Rust/.NET CLIs) instead of the raw sentinel + // event, whose text form printed "0 MX_EVENT_FAMILY_UNSPECIFIED". + ReplayGap gap = item.replayGap(); + if (json) { + client.out().printf( + "{\"replayGap\":{\"requestedAfterSequence\":%s,\"oldestAvailableSequence\":%s}}%n", + Long.toUnsignedString(gap.getRequestedAfterSequence()), + Long.toUnsignedString(gap.getOldestAvailableSequence())); + } else { + client.out().printf( + "REPLAY_GAP requested_after=%s oldest_available=%s%n", + Long.toUnsignedString(gap.getRequestedAfterSequence()), + Long.toUnsignedString(gap.getOldestAvailableSequence())); + } } else { - client.out().printf("%d %s%n", event.getWorkerSequence(), event.getFamily()); + MxEvent event = item.event(); + if (json) { + client.out().println(protoJson(event)); + } else { + client.out().printf("%d %s%n", event.getWorkerSequence(), event.getFamily()); + } } count++; if (limit > 0 && count >= limit) { diff --git a/clients/java/zb-mom-ww-mxgateway-cli/src/test/java/com/zb/mom/ww/mxgateway/cli/MxGatewayCliTests.java b/clients/java/zb-mom-ww-mxgateway-cli/src/test/java/com/zb/mom/ww/mxgateway/cli/MxGatewayCliTests.java index 09610bd..c3dd35d 100644 --- a/clients/java/zb-mom-ww-mxgateway-cli/src/test/java/com/zb/mom/ww/mxgateway/cli/MxGatewayCliTests.java +++ b/clients/java/zb-mom-ww-mxgateway-cli/src/test/java/com/zb/mom/ww/mxgateway/cli/MxGatewayCliTests.java @@ -43,6 +43,7 @@ import mxaccess_gateway.v1.MxaccessGateway.OpenSessionRequest; import mxaccess_gateway.v1.MxaccessGateway.ProtocolStatus; import mxaccess_gateway.v1.MxaccessGateway.ProtocolStatusCode; import mxaccess_gateway.v1.MxaccessGateway.RegisterReply; +import mxaccess_gateway.v1.MxaccessGateway.ReplayGap; import mxaccess_gateway.v1.MxaccessGateway.SessionState; import mxaccess_gateway.v1.MxaccessGateway.StreamAlarmsRequest; import mxaccess_gateway.v1.MxaccessGateway.SubscribeResult; @@ -902,6 +903,59 @@ final class MxGatewayCliTests { } } + @Test + void streamEventsRendersReplayGapAsTypedRow() { + // NEXT-02: the ReplayGap sentinel must render as the typed cross-CLI + // row (numeric cursors under a replayGap key in --json, a REPLAY_GAP + // line in text mode), never as the raw sentinel event — text mode + // used to print "0 MX_EVENT_FAMILY_UNSPECIFIED". + MxEvent gap = MxEvent.newBuilder() + .setReplayGap(ReplayGap.newBuilder() + .setRequestedAfterSequence(7L) + .setOldestAvailableSequence(42L) + .build()) + .build(); + MxEvent dataChange = MxEvent.newBuilder() + .setFamily(MxEventFamily.MX_EVENT_FAMILY_ON_DATA_CHANGE) + .setSessionId("session-cli") + .setWorkerSequence(43L) + .build(); + + try (InProcessGatewayHarness harness = new InProcessGatewayHarness()) { + harness.setScriptedEvents(List.of(gap, dataChange)); + CliRun jsonRun = execute( + new HarnessClientFactory(harness), + "stream-events", + "--session-id", + "session-cli", + "--json"); + + assertEquals(0, jsonRun.exitCode(), "errors:\n" + jsonRun.errors()); + String jsonOut = jsonRun.output(); + assertTrue( + jsonOut.contains( + "{\"replayGap\":{\"requestedAfterSequence\":7,\"oldestAvailableSequence\":42}}"), + jsonOut); + assertTrue(jsonOut.contains("\"family\":\"MX_EVENT_FAMILY_ON_DATA_CHANGE\""), jsonOut); + assertFalse(jsonOut.contains("MX_EVENT_FAMILY_UNSPECIFIED"), jsonOut); + } + + try (InProcessGatewayHarness harness = new InProcessGatewayHarness()) { + harness.setScriptedEvents(List.of(gap, dataChange)); + CliRun textRun = execute( + new HarnessClientFactory(harness), + "stream-events", + "--session-id", + "session-cli"); + + assertEquals(0, textRun.exitCode(), "errors:\n" + textRun.errors()); + String textOut = textRun.output(); + assertTrue(textOut.contains("REPLAY_GAP requested_after=7 oldest_available=42"), textOut); + assertFalse(textOut.contains("MX_EVENT_FAMILY_UNSPECIFIED"), textOut); + assertTrue(textOut.contains("43 MX_EVENT_FAMILY_ON_DATA_CHANGE"), textOut); + } + } + // ---- galaxy-discover / galaxy-watch over the in-process harness (Task 6) ---- @Test diff --git a/docs/CrossLanguageSmokeMatrix.md b/docs/CrossLanguageSmokeMatrix.md index acd1a37..4a7e0bb 100644 --- a/docs/CrossLanguageSmokeMatrix.md +++ b/docs/CrossLanguageSmokeMatrix.md @@ -40,27 +40,24 @@ reports the next deliverable sequence rather than `0` (see [Sessions](Sessions.m The default smoke sequence opens a fresh stream (no cursor) and does not exercise the gap path; a resume-with-gap fixture case is tracked separately (TST-24). -The CLIs differ in how they *print* that library-level signal. Three of them consume -the typed gap and emit a dedicated row rather than a degenerate event row; the other -two hand the raw sentinel `MxEvent` straight to the formatter, so they print the -sentinel itself, whose `replayGap` field carries the same cursors: +All five CLIs consume the typed gap and emit a dedicated row rather than a +degenerate event row (the .NET and Java halves were the last to convert — NEXT-02): | CLI | Text mode | JSON mode | |-----|-----------|-----------| | `mxgw-rs` (Rust, canonical) | `REPLAY_GAP requested_after=<n> oldest_available=<n>` | `{"replayGap": {"requestedAfterSequence": <n>, "oldestAvailableSequence": <n>}}` as one entry of the `events` array | | `mxgw-go` (Go) | `REPLAY_GAP requested_after=<n> oldest_available=<n>` | one `{"replayGap": {"requestedAfterSequence": <n>, "oldestAvailableSequence": <n>}}` line, counted toward `-limit` like any other row | | `mxgw-py` (Python) | same JSON dump as `--json` | `{"replayGap": {"requestedAfterSequence": <n>, "oldestAvailableSequence": <n>}}` as one entry of the `events` array | -| `mxgw-dotnet` (.NET) | the raw sentinel `MxEvent` as protobuf JSON, including its `replayGap` field | same, as one entry of the `events` array | -| `mxgw-java` (Java) | the sentinel's `worker_sequence` and `family` (`0 MX_EVENT_FAMILY_UNSPECIFIED`) | the raw sentinel `MxEvent` as protobuf JSON, including its `replayGap` field | +| `mxgw-dotnet` (.NET) | one `{"replayGap": {...}}` line (its "text" mode is JSON-per-line) | the same row — per line with `--jsonl`, as one entry of the `events` array with `--json` | +| `mxgw-java` (Java) | `REPLAY_GAP requested_after=<n> oldest_available=<n>` | one `{"replayGap": {"requestedAfterSequence": <n>, "oldestAvailableSequence": <n>}}` line | -Rust, Go, and Python emit the same two key names and, deliberately, the same JSON -value **types**: the cursors are JSON numbers (`7`), not strings. That is why the -Go CLI types the row by hand instead of marshalling `ReplayGap` with `protojson` — -the proto3 JSON mapping renders 64-bit integers as strings (`"7"`), which is also -why the .NET and Java rows, which pass the sentinel through a protobuf JSON -formatter, carry **quoted** cursors. A matrix runner must therefore compare parsed -values, not raw bytes, and must not assume the same value type across all five -CLIs. +All five emit the same two key names and, deliberately, the same JSON value +**types**: the cursors are JSON numbers (`7`), not strings. That is why every +CLI types the row by hand instead of marshalling `ReplayGap` through its +protobuf JSON formatter — the proto3 JSON mapping renders 64-bit integers as +strings (`"7"`). Normal event rows still come from the protobuf formatters, so +a matrix runner must still compare parsed values, not raw bytes, when it mixes +gap rows with event rows. Two further formatting differences among the three canonical CLIs, none of them semantic: Python sorts object keys and uses `", "` / `": "` separators From 2c03e0a6848a40c1e5395da473ea23d95fc5b55a Mon Sep 17 00:00:00 2001 From: Joseph Doherty <dohejw01@gmail.com> Date: Mon, 10 Aug 2026 06:06:12 -0400 Subject: [PATCH 5/6] fix(alarms): dedup reconcile/live duplicate broadcasts on the alarm feed (NEXT-03) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A periodic reconcile can synthesize a repair transition whose matching live transition is still buffered in the alarm lease; both then broadcast as indistinguishable duplicates on StreamAlarms and the dashboard hub. Nothing serializes the two paths, and a correct serialization needs a worker-side high-water mark on QueryActiveAlarms (proto + worker change + a stall path), so this closes the common case with a local best-effort dedup instead: both paths already carry the same worker-derived identity — the worker stamps record.TransitionTimestampUtc into both OnAlarmTransitionEvent's transition_timestamp and ActiveAlarmSnapshot.last_transition_timestamp — so ApplyTransition suppresses a live transition whose (timestamp, resulting state) the cache already carries from a repair. The Clear leg has no cache entry left to compare, so ApplyReconcile tombstones each synthesized Clear by the instance's original_raise_timestamp for one reconcile generation; a matching live Clear consumes the tombstone, while a new raise/clear cycle carries a newer raise timestamp and passes. Suppression fires only on a positive marker match — unset timestamps keep today's behavior — so the documented at-least-once consumer contract stands (gateway.md, Sessions.md updated in the same change). Tests: two new regressions drive the exact race through the GWC-26 harness (repair-then-buffered-live for Raise and for Clear, each with a genuine follow-up transition proving no over-suppression); GatewayAlarmMonitor suites 18/18. --- docs/Sessions.md | 2 +- gateway.md | 12 +- .../Alarms/GatewayAlarmMonitor.cs | 77 ++++++++- .../GatewayAlarmMonitorAttachOrderTests.cs | 152 ++++++++++++++++++ 4 files changed, 232 insertions(+), 11 deletions(-) diff --git a/docs/Sessions.md b/docs/Sessions.md index 6f511f4..4ebb1ca 100644 --- a/docs/Sessions.md +++ b/docs/Sessions.md @@ -201,7 +201,7 @@ The single worker event channel has exactly one direct reader: the `SessionEvent The monitor takes that lease **before** it issues `SubscribeAlarms`, so no transition window is missed: the pump has been running since `MarkReady` started the dashboard mirror, and the distributor only fans to subscribers registered at fan-out time, so a lease taken after the subscribe + first-reconcile round trips would lose every transition raised inside that window. Transitions arriving while the monitor subscribes and reconciles buffer in the lease's bounded channel and are applied after the reconcile, which is order-safe because a live transition can update an alarm the reconciled snapshot already holds. -The repair transitions the monitor's reconcile broadcasts on the alarm feed (Raise/Clear presence deltas and the acked-state delta) are **at-least-once**: a reconcile reads the worker's current state while the matching live transition may still be buffered in the lease, so both can be broadcast and the duplicates are indistinguishable — alarm-feed consumers must apply transitions idempotently. This is a property of the reconcile design, not of the buffering above. +The repair transitions the monitor's reconcile broadcasts on the alarm feed (Raise/Clear presence deltas and the acked-state delta) are **at-least-once**: a reconcile reads the worker's current state while the matching live transition may still be buffered in the lease, so both can be broadcast and the duplicates are indistinguishable — alarm-feed consumers must apply transitions idempotently. This is a property of the reconcile design, not of the buffering above. The monitor dedups the common case best-effort (NEXT-03): a buffered live transition that positively matches the cache's worker timestamp and resulting state — or, for Clear, a tombstone keyed on the cleared instance's original raise timestamp — was already broadcast as a repair and is suppressed; unset timestamps never suppress, so the consumer contract is unchanged. `AttachInternalEventSubscriber` enforces the same readiness gate as `AttachEventSubscriber` — a session (or worker) that is not `Ready` throws `SessionNotReady` *before* the distributor is constructed. A premature attach would otherwise start the pump against a source that throws, completing every subscriber with that error and latching the distributor for the session's whole lifetime. diff --git a/gateway.md b/gateway.md index 6c59f51..5ab5d27 100644 --- a/gateway.md +++ b/gateway.md @@ -173,9 +173,15 @@ the worker's current state while the corresponding live transition may still be buffered in the monitor's lease, so both can broadcast and the two are indistinguishable on the feed. This applies to the acked-state delta and equally to the older Raise/Clear presence repair: nothing serializes a reconcile pass -against the in-flight live stream. Alarm-feed consumers (`StreamAlarms` clients -and the dashboard alarm hub) must apply transitions idempotently — treat one as -"set this alarm to this state", never as an increment or a toggle. +against the in-flight live stream. The monitor narrows that window with a +best-effort dedup (NEXT-03): a buffered live transition whose worker timestamp +and resulting state the cache already carries from a repair — or whose Clear +matches a one-reconcile-generation tombstone keyed on the instance's original +raise timestamp — is suppressed instead of re-broadcast. The dedup fires only on +a positive marker match (unset timestamps never suppress), so the contract stays +at-least-once: alarm-feed consumers (`StreamAlarms` clients and the dashboard +alarm hub) must apply transitions idempotently — treat one as "set this alarm to +this state", never as an increment or a toggle. ### Alarm providers and failover diff --git a/src/ZB.MOM.WW.MxGateway.Server/Alarms/GatewayAlarmMonitor.cs b/src/ZB.MOM.WW.MxGateway.Server/Alarms/GatewayAlarmMonitor.cs index 0f1e1cc..59bb783 100644 --- a/src/ZB.MOM.WW.MxGateway.Server/Alarms/GatewayAlarmMonitor.cs +++ b/src/ZB.MOM.WW.MxGateway.Server/Alarms/GatewayAlarmMonitor.cs @@ -34,6 +34,14 @@ public sealed class GatewayAlarmMonitor : BackgroundService, IGatewayAlarmServic private readonly Dictionary<string, ActiveAlarmSnapshot> _alarms = new(StringComparer.Ordinal); private readonly List<Subscriber> _subscribers = []; + // NEXT-03 dedup tombstones, guarded by _sync: alarm instances whose Clear was synthesized by + // the most recent reconcile pass, keyed by reference with the instance's original raise + // timestamp as the identity marker. A buffered live Clear for the same instance is a duplicate + // of the repair and is suppressed. One generation deep: each reconcile pass replaces the map, + // so a tombstone lives at least one reconcile interval — far longer than the lease buffer the + // duplicate would be sitting in — and the map stays bounded by the feed's churn per interval. + private readonly Dictionary<string, Timestamp> _clearedByReconcile = new(StringComparer.Ordinal); + // Current provider status (mode + degraded + reason + since), guarded by _sync. // Initialized to the alarm-manager, not-degraded baseline so a late joiner sees // a sensible status even before any OnAlarmProviderModeChanged event arrives. @@ -413,17 +421,60 @@ public sealed class GatewayAlarmMonitor : BackgroundService, IGatewayAlarmServic { if (transition.TransitionKind == AlarmTransitionKind.Clear) { - _alarms.Remove(reference); + bool wasKnown = _alarms.Remove(reference); + if (!wasKnown && IsDuplicateOfReconcileClear(reference, transition)) + { + return; + } } else { - _alarms[reference] = SnapshotFromTransition(transition); + ActiveAlarmSnapshot snapshot = SnapshotFromTransition(transition); + bool duplicate = _alarms.TryGetValue(reference, out ActiveAlarmSnapshot? existing) + && IsDuplicateOfCachedState(existing, snapshot); + _alarms[reference] = snapshot; + if (duplicate) + { + return; + } } Broadcast(new AlarmFeedMessage { Transition = transition }, reference); } } + // NEXT-03: best-effort dedup of the reconcile/live race. A reconcile that already synthesized + // this transition as a feed repair left the cache carrying the worker's transition timestamp + // and resulting state — both derived from the same worker-side value the live transition + // carries — so an exact (timestamp, state) match means this live transition's outcome has + // already been broadcast. Suppress only on a positive match: an unset timestamp on either + // side keeps today's at-least-once behavior. + private static bool IsDuplicateOfCachedState(ActiveAlarmSnapshot existing, ActiveAlarmSnapshot incoming) + { + return existing.LastTransitionTimestamp is not null + && incoming.LastTransitionTimestamp is not null + && existing.LastTransitionTimestamp.Equals(incoming.LastTransitionTimestamp) + && existing.CurrentState == incoming.CurrentState; + } + + // NEXT-03, the Clear leg. A reconcile Clear repair removes the cache entry before the buffered + // live Clear drains, so there is no cached state to compare against; the tombstone recorded by + // ApplyReconcile identifies the cleared instance by its original raise timestamp instead. The + // match consumes the tombstone, so a genuinely new raise/clear cycle (which carries a newer + // original raise timestamp) is never swallowed. Caller holds _sync. + private bool IsDuplicateOfReconcileClear(string reference, OnAlarmTransitionEvent transition) + { + if (transition.OriginalRaiseTimestamp is not null + && _clearedByReconcile.TryGetValue(reference, out Timestamp? clearedInstance) + && clearedInstance.Equals(transition.OriginalRaiseTimestamp)) + { + _clearedByReconcile.Remove(reference); + return true; + } + + return false; + } + // Handles the worker's provider-mode-change event: updates the stored provider // status, broadcasts it to every subscriber (provider status is global, not // alarm-scoped), records the switch metric, and forces a cache reconcile so the @@ -533,11 +584,14 @@ public sealed class GatewayAlarmMonitor : BackgroundService, IGatewayAlarmServic // // Delivery semantics: feed repair transitions are AT-LEAST-ONCE, not exactly-once. A reconcile // reads the worker's current state while the corresponding live transition may still be - // buffered in the alarm lease's channel; both then broadcast, and the two are indistinguishable - // on the feed. This is inherent to the reconcile design and pre-dates the acked-state delta - // (the Raise/Clear repair has always had it), since nothing serializes a reconcile against the - // in-flight live stream. Consumers must therefore treat alarm state idempotently — apply a - // transition as "set the alarm to this state", never as an increment or a toggle. + // buffered in the alarm lease's channel; both would then broadcast, and the two are + // indistinguishable on the feed, since nothing serializes a reconcile against the in-flight + // live stream. ApplyTransition narrows that window with a best-effort dedup (NEXT-03): a live + // transition whose worker timestamp and resulting state the cache already carries — or whose + // Clear matches a tombstone recorded below — was already broadcast as a repair and is + // suppressed. The dedup fires only on a positive marker match, so the contract stays + // at-least-once: consumers must still treat alarm state idempotently — apply a transition as + // "set the alarm to this state", never as an increment or a toggle. private void ApplyReconcile(IEnumerable<ActiveAlarmSnapshot> snapshots) { Dictionary<string, ActiveAlarmSnapshot> next = new(StringComparer.Ordinal); @@ -551,10 +605,19 @@ public sealed class GatewayAlarmMonitor : BackgroundService, IGatewayAlarmServic lock (_sync) { + // Previous-generation tombstones have outlived the buffered live transitions they + // guard against (one full reconcile interval); start this pass's generation fresh. + _clearedByReconcile.Clear(); + foreach (KeyValuePair<string, ActiveAlarmSnapshot> existing in _alarms) { if (!next.ContainsKey(existing.Key)) { + if (existing.Value.OriginalRaiseTimestamp is not null) + { + _clearedByReconcile[existing.Key] = existing.Value.OriginalRaiseTimestamp; + } + Broadcast( new AlarmFeedMessage { Transition = TransitionFromSnapshot(existing.Value, AlarmTransitionKind.Clear) }, existing.Key); diff --git a/src/ZB.MOM.WW.MxGateway.Tests/Alarms/GatewayAlarmMonitorAttachOrderTests.cs b/src/ZB.MOM.WW.MxGateway.Tests/Alarms/GatewayAlarmMonitorAttachOrderTests.cs index 061931b..d262154 100644 --- a/src/ZB.MOM.WW.MxGateway.Tests/Alarms/GatewayAlarmMonitorAttachOrderTests.cs +++ b/src/ZB.MOM.WW.MxGateway.Tests/Alarms/GatewayAlarmMonitorAttachOrderTests.cs @@ -155,6 +155,133 @@ public sealed class GatewayAlarmMonitorAttachOrderTests await monitor.StopAsync(CancellationToken.None); } + /// <summary> + /// NEXT-03. A reconcile Raise repair applied while the matching live Raise is still + /// buffered must not double-broadcast: the live transition carrying the same worker + /// timestamp and resulting state the cache already holds is a duplicate and is suppressed. + /// </summary> + /// <returns>A task that represents the asynchronous operation.</returns> + [Fact] + public async Task LiveTransitionMatchingReconcileRepair_IsSuppressed() + { + using GatewayMetrics metrics = new(); + await using FakeSessionManager sessions = new(); + using GatewayAlarmMonitor monitor = CreateMonitor(sessions, metrics); + + using CancellationTokenSource cts = new(); + await monitor.StartAsync(cts.Token); + await sessions.WaitForSubscribeStartAsync(WaitTimeout); + + // The reconcile sees the raised alarm first (its Raise repair broadcasts before this + // reader attaches) and stamps the cache with the worker's transition timestamp. + Timestamp raiseTime = Timestamp.FromDateTimeOffset(new DateTimeOffset(2026, 8, 10, 12, 0, 0, TimeSpan.Zero)); + sessions.SetReconcileSnapshot(SnapshotAt(AlarmConditionState.Active, raiseTime, raiseTime)); + sessions.EmitEvent(ProviderModeProbe(1)); + await WaitUntilAsync( + () => monitor.CurrentAlarms.Any(alarm => alarm.AlarmFullReference == AlarmReference), + WaitTimeout); + + List<AlarmFeedMessage> received = []; + TaskCompletionSource snapshotComplete = new(TaskCreationOptions.RunContinuationsAsynchronously); + using CancellationTokenSource streamCts = new(); + Task reader = ReadFeedAsync(monitor, received, snapshotComplete, streamCts.Token, untilSnapshotComplete: true); + await snapshotComplete.Task.WaitAsync(WaitTimeout); + + // The buffered live Raise drains with the same worker timestamp — a duplicate of the + // repair. The follow-up Acknowledge with a newer timestamp is genuine and must pass. + sessions.EmitEvent(TransitionAt(2, AlarmTransitionKind.Raise, raiseTime, raiseTime)); + Timestamp ackTime = Timestamp.FromDateTimeOffset(new DateTimeOffset(2026, 8, 10, 12, 0, 5, TimeSpan.Zero)); + sessions.EmitEvent(TransitionAt(3, AlarmTransitionKind.Acknowledge, ackTime, raiseTime)); + + await WaitForAsync( + received, + m => m.PayloadCase == AlarmFeedMessage.PayloadOneofCase.Transition + && m.Transition.TransitionKind == AlarmTransitionKind.Acknowledge, + WaitTimeout); + + lock (received) + { + AlarmFeedMessage[] transitions = received + .Where(m => m.PayloadCase == AlarmFeedMessage.PayloadOneofCase.Transition) + .ToArray(); + AlarmFeedMessage single = Assert.Single(transitions); + Assert.Equal(AlarmTransitionKind.Acknowledge, single.Transition.TransitionKind); + } + + await streamCts.CancelAsync(); + await reader; + await cts.CancelAsync(); + await monitor.StopAsync(CancellationToken.None); + } + + /// <summary> + /// NEXT-03, the Clear leg. A reconcile Clear repair removes the cache entry, so the + /// buffered live Clear is deduped through the tombstone keyed on the instance's original + /// raise timestamp — and a genuinely new raise/clear cycle is never swallowed. + /// </summary> + /// <returns>A task that represents the asynchronous operation.</returns> + [Fact] + public async Task LiveClearMatchingReconcileClearRepair_IsSuppressed() + { + using GatewayMetrics metrics = new(); + await using FakeSessionManager sessions = new(); + using GatewayAlarmMonitor monitor = CreateMonitor(sessions, metrics); + + using CancellationTokenSource cts = new(); + await monitor.StartAsync(cts.Token); + await sessions.WaitForSubscribeStartAsync(WaitTimeout); + + Timestamp raiseTime = Timestamp.FromDateTimeOffset(new DateTimeOffset(2026, 8, 10, 13, 0, 0, TimeSpan.Zero)); + sessions.SetReconcileSnapshot(SnapshotAt(AlarmConditionState.Active, raiseTime, raiseTime)); + sessions.EmitEvent(ProviderModeProbe(1)); + await WaitUntilAsync( + () => monitor.CurrentAlarms.Any(alarm => alarm.AlarmFullReference == AlarmReference), + WaitTimeout); + + List<AlarmFeedMessage> received = []; + TaskCompletionSource snapshotComplete = new(TaskCreationOptions.RunContinuationsAsynchronously); + using CancellationTokenSource streamCts = new(); + Task reader = ReadFeedAsync(monitor, received, snapshotComplete, streamCts.Token, untilSnapshotComplete: true); + await snapshotComplete.Task.WaitAsync(WaitTimeout); + + // The worker no longer reports the alarm: the reconcile synthesizes the Clear repair and + // tombstones the instance by its original raise timestamp. + sessions.SetReconcileSnapshot(); + sessions.EmitEvent(ProviderModeProbe(2)); + await WaitForAsync( + received, + m => m.PayloadCase == AlarmFeedMessage.PayloadOneofCase.Transition + && m.Transition.TransitionKind == AlarmTransitionKind.Clear, + WaitTimeout); + + // The buffered live Clear for the SAME instance is a duplicate of the repair; the Raise + // that follows starts a new instance and must pass. + Timestamp clearTime = Timestamp.FromDateTimeOffset(new DateTimeOffset(2026, 8, 10, 13, 0, 10, TimeSpan.Zero)); + sessions.EmitEvent(TransitionAt(3, AlarmTransitionKind.Clear, clearTime, raiseTime)); + Timestamp newRaiseTime = Timestamp.FromDateTimeOffset(new DateTimeOffset(2026, 8, 10, 13, 0, 20, TimeSpan.Zero)); + sessions.EmitEvent(TransitionAt(4, AlarmTransitionKind.Raise, newRaiseTime, newRaiseTime)); + + await WaitForAsync( + received, + m => m.PayloadCase == AlarmFeedMessage.PayloadOneofCase.Transition + && m.Transition.TransitionKind == AlarmTransitionKind.Raise, + WaitTimeout); + + lock (received) + { + AlarmTransitionKind[] kinds = received + .Where(m => m.PayloadCase == AlarmFeedMessage.PayloadOneofCase.Transition) + .Select(m => m.Transition.TransitionKind) + .ToArray(); + Assert.Equal([AlarmTransitionKind.Clear, AlarmTransitionKind.Raise], kinds); + } + + await streamCts.CancelAsync(); + await reader; + await cts.CancelAsync(); + await monitor.StopAsync(CancellationToken.None); + } + private static GatewayAlarmMonitor CreateMonitor(FakeSessionManager sessions, GatewayMetrics metrics) { AlarmsOptions options = new() @@ -254,6 +381,31 @@ public sealed class GatewayAlarmMonitorAttachOrderTests SourceProvider = AlarmProviderMode.Alarmmgr, }; + // Snapshot carrying the worker-side identity markers the NEXT-03 dedup compares on. + private static ActiveAlarmSnapshot SnapshotAt( + AlarmConditionState state, + Timestamp lastTransition, + Timestamp originalRaise) + { + ActiveAlarmSnapshot snapshot = Snapshot(state); + snapshot.LastTransitionTimestamp = lastTransition; + snapshot.OriginalRaiseTimestamp = originalRaise; + return snapshot; + } + + // Live transition with explicit worker timestamps, for driving the NEXT-03 dedup. + private static MxEvent TransitionAt( + ulong sequence, + AlarmTransitionKind kind, + Timestamp transitionTimestamp, + Timestamp originalRaise) + { + MxEvent mxEvent = Transition(sequence, kind); + mxEvent.OnAlarmTransition.TransitionTimestamp = transitionTimestamp; + mxEvent.OnAlarmTransition.OriginalRaiseTimestamp = originalRaise; + return mxEvent; + } + private static async Task<AlarmFeedMessage> WaitForAsync( List<AlarmFeedMessage> received, Func<AlarmFeedMessage, bool> predicate, From 5fe74db971d11e36d6b6aaa9e5843e237860de55 Mon Sep 17 00:00:00 2001 From: Joseph Doherty <dohejw01@gmail.com> Date: Mon, 10 Aug 2026 06:07:00 -0400 Subject: [PATCH 6/6] docs(tracking): strike NEXT-01/02/03/04/05/09 as resolved 2026-08-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Six of the eight open next-cycle candidates are closed in this batch (fix/next-cycle-batch): macOS pipe-path length, .NET/Java CLI ReplayGap rendering, alarm reconcile/live dedup, frame-writer unobserved-fault hygiene (plus the NEXT-05 lazy-purge decision), and the Windows InformationalVersion stamp. NEXT-08 (GLAuth TLS posture) and NEXT-10 (glauth.md user-table reconciliation entangled with the OPC-UA group taxonomy) remain open by design — both need cross-repo/posture decisions, not gateway code. --- .../remediation/90-candidate-findings-next-cycle.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/archreview/2026-07-12/remediation/90-candidate-findings-next-cycle.md b/archreview/2026-07-12/remediation/90-candidate-findings-next-cycle.md index 4003d04..476412f 100644 --- a/archreview/2026-07-12/remediation/90-candidate-findings-next-cycle.md +++ b/archreview/2026-07-12/remediation/90-candidate-findings-next-cycle.md @@ -4,15 +4,15 @@ These were discovered while remediating the 2026-07-12 backlog but were **out of | ID (proposed) | Area | Severity (est.) | Summary | |---|---|---|---| -| NEXT-01 | Testing / macOS | Low | Fake-worker/e2e gateway tests fail on macOS under the default `TMPDIR` because the `CoreFxPipe_mxaccess-gateway-{pid}-{sessionId}` path exceeds the 104-char Unix-domain-socket `sun_path` limit under `/var/folders/…/T/`. Workaround today is `TMPDIR=/tmp`. Fix options: shorten the pipe name, or document the `TMPDIR=/tmp` requirement in `docs/GatewayTesting.md`. Surfaced independently by multiple remediation agents. | -| NEXT-02 | Clients (.NET, Java) | Low | The .NET and Java CLIs render the raw `ReplayGap` sentinel `MxEvent` on `stream-events` instead of a typed gap row — Java text mode prints `0 MX_EVENT_FAMILY_UNSPECIFIED`. Same defect class as CLI-36 (Go) / CLI-35 (Python), which were fixed this cycle; the .NET/Java halves were out of scope. The cross-language smoke matrix now records this divergence honestly. | -| NEXT-03 | Gateway alarms | Low | `GatewayAlarmMonitor.ApplyReconcile` feed-repair broadcasts (the new acked-delta from GWC-26 **and** the pre-existing Raise/Clear repair) are **at-least-once, not exactly-once**: a periodic reconcile can synthesize a transition whose matching live transition is still buffered in the alarm lease, so both broadcast as indistinguishable duplicates on the alarm feed (StreamAlarms + dashboard hub). Pre-existing (the Raise/Clear repair always had it); GWC-26 documented the at-least-once contract rather than closing the race. Closing it needs reconcile/live serialization or a monotonic dedup marker. | -| NEXT-04 | Worker frame writer | Low | WRK-22/WRK-25 cancellation path: a frame `Claimed` by a concurrent lock-holder just before its caller's cancellation races in is never awaited by that caller; if the write then faults, `TrySetException` lands on a `Task` nobody observes (unobserved-task-exception). By-design residual, non-crash (no `UnobservedTaskException` handler registered), pre-existing to single-frame WRK-22 and amplified per-batch by WRK-25. Hygiene fix: attach a fault-observing continuation to abandoned/tombstoned frame completions. | -| NEXT-05 | Worker frame writer | Info | A batch whose remaining frames are tombstoned by cancellation leaves dead `PendingFrame` entries in `_eventFrames`/`_controlFrames` until a future `DequeueNext` pops and skips them. Same pre-existing behavior as single-frame WRK-22, amplified per-batch; in practice heartbeats purge them promptly, so not a real leak. | +| ~~NEXT-01~~ | Testing / macOS | Low | **Resolved 2026-08-10** — the pipe name is now `mxgw-{pid}-{sessionUid}` (session guid hex, worst-case 43 chars), which fits the 104-byte `sun_path` budget under the default macOS `TMPDIR`; the three test-fixture pipe names were shortened the same way, and a `SessionManagerTests` regression pins the format and length budget. All previously failing suites (SessionWorkerClientFactory, e2e fake-worker smoke, WorkerClient, reconnect-replay) pass 33/33 under the default `TMPDIR` — this also retired the separately-remembered "macOS pipe-timeout test failures", which were this throw misread. Docs updated (gateway.md, GatewayProcessDesign, GatewayConfiguration, Sessions, CLAUDE.md). Original finding: Fake-worker/e2e gateway tests fail on macOS under the default `TMPDIR` because the `CoreFxPipe_mxaccess-gateway-{pid}-{sessionId}` path exceeds the 104-char Unix-domain-socket `sun_path` limit under `/var/folders/…/T/`. Workaround today is `TMPDIR=/tmp`. Fix options: shorten the pipe name, or document the `TMPDIR=/tmp` requirement in `docs/GatewayTesting.md`. Surfaced independently by multiple remediation agents. | +| ~~NEXT-02~~ | Clients (.NET, Java) | Low | **Resolved 2026-08-10** — both CLIs now branch on the sentinel and emit the typed cross-CLI row with numeric cursors (Java text mode prints `REPLAY_GAP requested_after=<n> oldest_available=<n>`; .NET emits the `{"replayGap":{…}}` row in jsonl/text and inside the `--json` events array). CrossLanguageSmokeMatrix.md's divergence table collapsed to one converged contract. New CLI regressions in both languages (.NET 35/35, Java 52/52). Original finding: The .NET and Java CLIs render the raw `ReplayGap` sentinel `MxEvent` on `stream-events` instead of a typed gap row — Java text mode prints `0 MX_EVENT_FAMILY_UNSPECIFIED`. Same defect class as CLI-36 (Go) / CLI-35 (Python), which were fixed this cycle; the .NET/Java halves were out of scope. The cross-language smoke matrix now records this divergence honestly. | +| ~~NEXT-03~~ | Gateway alarms | Low | **Resolved 2026-08-10** — best-effort dedup in `GatewayAlarmMonitor`: a buffered live transition whose worker timestamp + resulting state the cache already carries from a repair is suppressed, and reconcile Clear repairs tombstone the instance by `original_raise_timestamp` for one reconcile generation so the buffered live Clear dedups too. Positive-match only (unset timestamps never suppress), so the documented at-least-once consumer contract stands; serialization was rejected as the larger change that still needs a worker-side high-water mark to be correct. Two new race-driving regressions; alarm suites 18/18. Original finding: `GatewayAlarmMonitor.ApplyReconcile` feed-repair broadcasts (the new acked-delta from GWC-26 **and** the pre-existing Raise/Clear repair) are **at-least-once, not exactly-once**: a periodic reconcile can synthesize a transition whose matching live transition is still buffered in the alarm lease, so both broadcast as indistinguishable duplicates on the alarm feed (StreamAlarms + dashboard hub). Pre-existing (the Raise/Clear repair always had it); GWC-26 documented the at-least-once contract rather than closing the race. Closing it needs reconcile/live serialization or a monotonic dedup marker. | +| ~~NEXT-04~~ | Worker frame writer | Low | **Resolved 2026-08-10** — the tombstone helpers now attach a fault-observing continuation to every frame of a cancelled call (a cancelled task never fires `OnlyOnFaulted`, so unconditional attach is safe; covers both the claimed-mid-write frame and the already-faulted-by-`FailAllQueued` frame where `TrySetCanceled` loses). New regression drives the exact abandonment and asserts a marker exception never reaches `TaskScheduler.UnobservedTaskException`. Original finding: WRK-22/WRK-25 cancellation path: a frame `Claimed` by a concurrent lock-holder just before its caller's cancellation races in is never awaited by that caller; if the write then faults, `TrySetException` lands on a `Task` nobody observes (unobserved-task-exception). By-design residual, non-crash (no `UnobservedTaskException` handler registered), pre-existing to single-frame WRK-22 and amplified per-batch by WRK-25. Hygiene fix: attach a fault-observing continuation to abandoned/tombstoned frame completions. | +| ~~NEXT-05~~ | Worker frame writer | Info | **Resolved 2026-08-10 as a documented decision** — the lazy `DequeueNext` purge stays: any subsequent write drains both queues to empty and the heartbeat loop bounds tombstone residency to one interval, while eager `Queue<T>` rebuilds under `_gate` would add ordering-invariant surface next to the WRK-22 interlock for no real gain. Rationale recorded in `docs/WorkerFrameProtocol.md`. Original finding: A batch whose remaining frames are tombstoned by cancellation leaves dead `PendingFrame` entries in `_eventFrames`/`_controlFrames` until a future `DequeueNext` pops and skips them. Same pre-existing behavior as single-frame WRK-22, amplified per-batch; in practice heartbeats purge them promptly, so not a real leak. | | ~~NEXT-06~~ | Testing / live LDAP | Medium | **Resolved 2026-08-07** — fixtures realigned to the shared directory (`admin`/`password` for the GwAdmin success path, `gw-viewer`/`password` for the bind-succeeds-but-no-role path); verified `Failed: 0, Passed: 5` live against the shared GLAuth at `10.100.0.35:3893`, so the success-path assertion (GwAdmin group claim + Admin role claim) now fails if the service-account credential is wrong. Original finding: `DashboardLdapLiveTests` fixtures have drifted from the shared GLAuth directory, leaving the suite with **no positive-proof coverage of the service-account bind**. Its only success-path test, `AuthenticateAsync_AdminInGwAdminGroup_Succeeds`, binds `admin`/`admin123`, but the directory's `admin` user carries the standard dev password (`scadaproj/infra/glauth/config.toml`), so that assertion cannot pass. `AuthenticateAsync_ReadOnlyUserMissingGwAdminGroup_Fails` binds fixture user `readonly`, which **does not exist** in the GLAuth config at all — it passes for the wrong reason (user-not-found rather than the group-missing branch it names; the `readonly` name is in fact barred by the README's user/group case-collision rule). The three remaining tests are negative assertions that pass whether or not the service account can bind. Net effect: a green `DashboardLdapLiveTests` run proves nothing about the bind credential — surfaced during SEC-36, where the suite was considered as a substitute for the deferred dashboard-login check and rejected. Fix: realign the fixtures to real directory users (e.g. `multi-role`/`gw-viewer`) or add the missing users to the GLAuth config, and add one test that fails when the service-account credential is wrong. | | ~~NEXT-07~~ | Deployment / windev | High | **Resolved 2026-08-07** — a fresh portable framework-dependent publish of `origin/main` (`a346d51`) was built in a clean clone at `C:\build\mxgw-redeploy`, deployed to `C:\publish\mxaccessgw\Server-20260807`, and the `MxAccessGw` NSSM service repointed at it; the service now holds a stable PID with both `5120`/`5130` listening, a worker spawned, the Galaxy snapshot restored (129 objects / 56,731 attributes) and a clean event log. Root cause confirmed as the version skew this row predicted: the deployed 2026-06-25 build carried `ZB.MOM.WW.Auth.ApiKeys` 0.1.2.0, which supports auth-DB schema 2, against a `gateway-auth.db` stamped at schema 3 on 2026-07-15 by an ephemeral run of newer code — schema 3 is the current shared-lib version (`SqliteAuthSchema.CurrentVersion=3` in Auth 0.1.5), so the redeploy is the forward fix and the DB was left alone. Rollback artifacts kept: `C:\ProgramData\MxGateway\gateway-auth.db.bak-next07` (with `-wal`/`-shm`) and the previous `C:\publish\mxaccessgw\Server` directory. Two side effects worth recording: the old deploy's `appsettings.json` held the LDAP bind password in **plaintext on disk**, while the new one keeps the repo's `${secret:ldap/mxgateway/bind}` token with the NSSM environment supplying the value, so no plaintext LDAP secret remains on that host; and the redeploy tripped the SEC-06 `Ldap:Transport=None` production hard-stop (`GatewayOptionsValidator.cs:178`), resolved by relabelling the host — windev runs `Dashboard:DisableLogin=true`, which this repo's own docs mark dev/test-only, so its `Production` label contradicted its configuration and `DOTNET_ENVIRONMENT` was changed to `Staging` (that one NSSM environment entry only; the other nine preserved byte-identical). SEC-06 is untouched for genuinely production hosts — see NEXT-08 for the posture problem that relabelling defers. Original finding: The `10.100.0.48` (windev) gateway deployment is **stale and crash-looping**, and has been since at least 2026-08-06 (~10k Hosting-failed events/day). The deployed Server binary dates to 2026-06-25 and predates the auth-DB migration of 2026-07-15: it opens a schema-version-3 `gateway-auth.db` that it supports only at version 2 and aborts at startup, so the `MxAccessGw` service never reaches a listening state. Not a code defect in the current tree — a deploy-drift/operations gap — but it means the repo's only deployed host has been dark for over a day and any host-level verification (including SEC-36's dashboard-login check) is blocked until it is repaired. Fix: deploy a current Server build to windev, or restore/downgrade the auth DB to schema 2 if the old binary must stand. Worth asking separately why a service in a permanent restart loop raised no alert. Discovered during SEC-36. | | NEXT-08 | Security / LDAP posture | Medium | **The shared GLAuth offers no TLS, so SEC-06 makes it undeployable from a `Production`-labelled host.** `GatewayOptionsValidator` (`src/ZB.MOM.WW.MxGateway.Server/Configuration/GatewayOptionsValidator.cs:178`) refuses to start when `Ldap:Transport=None` in the `Production` environment, and `docs/GatewayConfiguration.md`'s `Transport` row states "Deployed hosts must set `Ldaps` or `StartTls`" — but the shared instance at `10.100.0.35:3893` has `[ldaps] enabled=false`, port `3894` closed, and answers StartTLS with `protocolError`, so neither value can work against it. That instruction is currently unsatisfiable for every host that authenticates there. windev sidestepped it on 2026-08-07 by moving to the `Staging` environment name (NEXT-07), which is honest for a dev/test rig but is not available to a real production host. Resolution needs either LDAPS/StartTLS on the shared GLAuth (certificate plus a trust story on each gateway host) or an explicit written posture decision that production gateways bind a different, TLS-capable directory. Surfaced during the NEXT-07 redeploy. | -| NEXT-09 | Build / versioning | Low | **Windows builds stamp git's error text into `InformationalVersion`.** `src/Directory.Build.props:29` runs `git -C "$(MSBuildThisFileDirectory)" …`; MSBuild's directory property ends in a backslash, which escapes the closing quote, so the command is malformed on Windows. The target carries `ContinueOnError`, so the failure is silent and git's stderr is captured as the revision — an observed stamp reads `0.1.2+fatal: cannot change to …`. Any Windows build without a preset `SourceRevisionId` therefore ships a binary that cannot be correlated back to a commit, defeating the point of TST-11. Not reproducible on macOS/Linux, where the separator is `/`. Fix sketch: append `.` to the path or trim the trailing separator before quoting. Surfaced while identifying the deployed binary during NEXT-07. | +| ~~NEXT-09~~ | Build / versioning | Low | **Resolved 2026-08-10** — the Exec path now appends `.` so the trailing backslash can no longer escape the closing quote, and `SourceRevisionId` is additionally gated on a short-SHA regex so no future git failure text can be stamped either. macOS stamp verified unchanged; Windows stamp verified on windev with this batch. Original finding: **Windows builds stamp git's error text into `InformationalVersion`.** `src/Directory.Build.props:29` runs `git -C "$(MSBuildThisFileDirectory)" …`; MSBuild's directory property ends in a backslash, which escapes the closing quote, so the command is malformed on Windows. The target carries `ContinueOnError`, so the failure is silent and git's stderr is captured as the revision — an observed stamp reads `0.1.2+fatal: cannot change to …`. Any Windows build without a preset `SourceRevisionId` therefore ships a binary that cannot be correlated back to a commit, defeating the point of TST-11. Not reproducible on macOS/Linux, where the separator is `/`. Fix sketch: append `.` to the path or trim the trailing separator before quoting. Surfaced while identifying the deployed binary during NEXT-07. | | NEXT-10 | Docs / glauth | Medium | **`glauth.md`'s "Pre-provisioned users" table contradicts both the directory and the rest of its own file.** It documents `readonly`/`readonly123` and `admin`/`admin123`, neither of which matches `scadaproj/infra/glauth/config.toml` (`readonly` does not exist there; `admin` carries the standard dev password), and lists the `ReadOnly` gid as `5501` against an actual `5601`. Its dashboard section, by contrast, is correct — so the file is internally inconsistent and a reader cannot tell which half to trust. This table was the **root cause of the NEXT-06 fixture drift**, and it has propagated further: `docs/GatewayTesting.md`'s `MXGATEWAY_LIVE_MXACCESS_WRITE_SECURED_PASSWORD` default and the matching literal in `WorkerLiveMxAccessSmokeTests` both take `admin123` from it. Deliberately **not** fixed in the 2026-08-07 pass: the table is entangled with the OPC-UA group taxonomy (gids, role mapping, and the sister-repo consumers of the same directory), so reconciling it means sweeping that taxonomy as one unit rather than patching two rows. | ## Operator actions still pending (from this cycle's runbooks)