Compare commits

..

23 Commits

Author SHA1 Message Date
Joseph Doherty 5776021970 ci(rust): allow generated result size lint
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m18s
ci / java (push) Successful in 2m24s
ci / portable (push) Successful in 9m31s
2026-08-25 12:28:12 -04:00
Joseph Doherty a11786cbea docs: record batch N portable gates
ci / portable (push) Failing after 17m19s
ci / java (push) Successful in 2m29s
ci / windows-x86 (push) Successful in 5m55s
ci / nightly-windev (push) Has been skipped
2026-08-25 12:09:40 -04:00
Joseph Doherty aea2c74fdf docs: start batch N release gate
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m18s
ci / java (push) Successful in 2m11s
ci / portable (push) Failing after 8m43s
2026-08-25 12:06:34 -04:00
Joseph Doherty 8b147a32a2 docs: record batch M verification
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m19s
ci / java (push) Successful in 2m26s
ci / portable (push) Failing after 5m27s
2026-08-25 12:02:19 -04:00
Joseph Doherty cc143b52cf ci(TST-35): run gateway suite on windev nightly
ci / nightly-windev (push) Has been skipped
ci / java (push) Successful in 2m44s
ci / windows-x86 (push) Successful in 5m56s
ci / portable (push) Failing after 9m21s
2026-08-25 11:56:35 -04:00
Joseph Doherty ad79238c84 docs: record batch L verification
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m17s
ci / java (push) Successful in 2m28s
ci / portable (push) Failing after 10m12s
2026-08-25 11:50:48 -04:00
Joseph Doherty 986c6c0ce4 fix(IPC-36): import obsolete attribute namespace
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 2m4s
ci / java (push) Successful in 2m22s
ci / portable (push) Failing after 9m44s
2026-08-25 11:49:02 -04:00
Joseph Doherty 484028e721 feat(GWC-37,WRK-32): refresh live alarm watch lists
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Failing after 51s
ci / java (push) Successful in 2m8s
ci / portable (push) Failing after 1h1m54s
2026-08-25 11:46:07 -04:00
Joseph Doherty 1b11e89f29 docs: record batch K verification 2026-08-25 11:26:12 -04:00
Joseph Doherty 5cdb3da8c0 fix(CLI-47,CLI-48,CLI-49,CLI-50): align alarm feed clients 2026-08-25 11:25:47 -04:00
Joseph Doherty feadcf5740 docs: record batch J verification 2026-08-25 11:14:33 -04:00
Joseph Doherty d6c5a11ae6 fix(IPC-36,IPC-37): guard secondary contracts 2026-08-25 11:14:14 -04:00
Joseph Doherty f3827c71c9 docs: record batch I verification 2026-08-25 11:01:38 -04:00
Joseph Doherty 1875426e4b fix(SEC-38,SEC-39): preserve aggregate and fallback throttles 2026-08-25 11:00:29 -04:00
Joseph Doherty 780720165a fix(GWC-35): repair re-raised alarm snapshots 2026-08-25 10:58:21 -04:00
Joseph Doherty b0156c685e fix(GWC-33): unregister terminal worker queue gauge 2026-08-25 10:56:23 -04:00
Joseph Doherty f2090fe1b5 docs: record batch H verification 2026-08-25 10:53:03 -04:00
Joseph Doherty 92aae28f07 chore(GWC-36,IPC-38,TST-33,TST-34,TST-36): close cycle-3 currency gaps 2026-08-25 10:52:39 -04:00
Joseph Doherty 4614080e24 docs: normalize continuation design formatting 2026-08-25 10:41:22 -04:00
Joseph Doherty 8e30ec5be9 docs: plan and track cycle-3 P2 continuation 2026-08-25 10:40:55 -04:00
Joseph Doherty 30548da71c docs: design cycle-3 P2 continuation 2026-08-25 10:37:30 -04:00
Joseph Doherty 914495a9f3 chore: ignore project-local worktrees 2026-08-20 12:14:23 -04:00
Joseph Doherty fa364ce139 chore(archreview): cycle-3 remediation integration gate recorded (gateway 1183/1183, windev worker 524/524)
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m23s
ci / java (push) Successful in 2m27s
ci / portable (push) Successful in 9m25s
2026-08-20 09:41:49 -04:00
75 changed files with 5532 additions and 2044 deletions
+9 -6
View File
@@ -4,16 +4,16 @@
# .NET Framework 4.8 / x86 and need MXAccess COM installed, so they build ONLY on a Windows host.
# They are out of scope for this Linux `portable` job — the `windows-x86` job below builds and
# tests them on windev (10.100.0.48) over SSH (see scripts/ci/run-windev-ci.sh), and the nightly
# `nightly-windev` job additionally runs the live-MXAccess smoke, which is scheduled-only so it
# never gates a push.
# `nightly-windev` job additionally runs the full solution build, gateway suite, and live-MXAccess
# smoke, which are scheduled-only so they never gate a push.
name: ci
on:
push:
pull_request:
schedule:
# 06:00 UTC nightly: the `nightly-windev` job re-runs the x86 Worker build + full Worker.Tests on
# windev, then the live-MXAccess smoke (MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1) and a full-slnx build.
# 06:00 UTC nightly: x86 Worker build + Worker.Tests, full-slnx build, gateway tests, then the
# live-MXAccess smoke (MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1), serialized on windev's host lock.
# Push/PR runs skip it — the job is gated `if: github.event_name == 'schedule'`.
- cron: '0 6 * * *'
@@ -91,6 +91,9 @@ jobs:
shell: pwsh
run: ./scripts/check-client-versions.ps1
- name: Code-reviews README consistency
run: python code-reviews/regen-readme.py --check
- name: Gateway fake-worker tests
run: dotnet test src/ZB.MOM.WW.MxGateway.Tests/ZB.MOM.WW.MxGateway.Tests.csproj -c Release --no-build
@@ -179,7 +182,7 @@ jobs:
run: ./scripts/ci/run-windev-ci.sh test
nightly-windev:
# TST-25: scheduled deep run — full Worker.Tests + live-MXAccess smoke + full-slnx build on windev.
# TST-35: scheduled deep run — Worker, full-slnx, gateway, and live-smoke tiers on windev.
# Gated to the schedule event via `if:` (not an unsatisfiable `runs-on`), so push/PR runs mark it
# skipped, not queued. Never gates a push. On failure it opens a Gitea issue so a red nightly is
# visible even though nobody watches the Actions page.
@@ -195,7 +198,7 @@ jobs:
PUBLIC_SERVER_URL: https://gitea.dohertylan.com
steps:
- uses: actions/checkout@v4
- name: Full Worker.Tests + live-MXAccess smoke on windev
- name: Worker + gateway tests + live-MXAccess smoke on windev
env:
# WINDEV_SSH_KEY is stored base64-encoded (single line) so Gitea's line-oriented secret
# masker redacts it in the step env echo; run-windev-ci.sh decodes it. A raw multiline PEM
+1
View File
@@ -155,3 +155,4 @@ generated-scratch/
# Agent worktrees (subagent isolation) — never commit
.claude/worktrees/
.worktrees/
+1 -1
View File
@@ -157,7 +157,7 @@ External analysis sources referenced by design docs:
Gateway gRPC clients authenticate with an API key in metadata: `authorization: Bearer mxgw_<key-id>_<secret>`. Keys are stored hashed (with a peppered SHA) in a gateway-owned SQLite DB (default `C:\ProgramData\MxGateway\gateway-auth.db`). Scopes (`session:open`, `session:close`, `invoke:read`, `invoke:write`, `invoke:secure`, `events:read`, `metadata:read`, `admin`) gate specific RPCs; missing → `Unauthenticated`, insufficient → `PermissionDenied`. The `apikey` subcommand on the server exe manages keys; see `src/ZB.MOM.WW.MxGateway.Server/Security/Authentication/`.
Session event streaming is **owner-scoped**: the API key that opened a session is recorded on the session, and every `StreamEvents` attach/reattach is rejected with `PermissionDenied` unless the caller's key id matches the owner. Possessing the `event` scope and knowing a session id is not sufficient — this closes the reconnect/fan-out trust boundary (detach-grace and replay retention are on by default) so an `event`-scoped key cannot attach to another key's retained session.
Session event streaming is **owner-scoped**: the API key that opened a session is recorded on the session, and every `StreamEvents` attach/reattach is rejected with `PermissionDenied` unless the caller's key id matches the owner. Possessing the `events:read` scope and knowing a session id is not sufficient — this closes the reconnect/fan-out trust boundary (detach-grace and replay retention are on by default) so an `events:read`-scoped key cannot attach to another key's retained session.
Dashboard auth is LDAP-backed (separate from the gRPC API-key model). `/login` binds against `MxGateway:Ldap` and maps the user's LDAP groups to `Admin` or `Viewer` via `MxGateway:Dashboard:GroupToRole`, then issues an HTTP-only secure cookie named `__Host-MxGatewayDashboard` when `Dashboard:RequireHttpsCookie` is true (default) and no `Dashboard:CookieName` override is set, else the plain `MxGatewayDashboard` (the `__Host-` prefix requires a Secure cookie). SignalR hubs at `/hubs/{snapshot,alarms,events}` accept either the cookie or a 5-minute bearer minted at `/hubs/token`; `/hubs/alarms` and the `/alarms` page additionally require a **real authenticated dashboard role** (`MxGateway.Dashboard.Alarms`, a requirement type the environment-bypass handler does not handle), because the alarm feed is gateway-central with no per-session provenance and so cannot be tag-scoped — its metadata stays galaxy-wide for authenticated Viewers, but anonymous localhost and `Authentication:Mode=Disabled` do not receive it at all. Dashboard event visibility is **tag-scoped per session** (`IDashboardSessionAcl`, gating both the events hub and the session-details page's in-process subscribe): an Administrator sees every session, while any other caller sees a session only when its tags — inherited from the owning API key's `apikey --dashboard-tags`, never from the client's request — intersect the tags their LDAP groups grant via `Dashboard:GroupToTag`. That grant is **re-resolved from the caller's LDAP group claims against current options on every subscribe** (not read from the tag claims stamped at login, which would outlive a config-revoked grant for the cookie's 8-hour sliding life); login-stamped tag claims are the fallback only for a principal carrying no group claims — i.e. the hub-token principal, whose grant the mint already re-resolved. Untagged sessions follow `Dashboard:UntaggedSessionVisibility` (default `AdminOnly`), and a principal with neither claim kind (anonymous localhost included) is an empty-grant Viewer. `Dashboard:AllowAnonymousLocalhost` (default `true`) grants **read-only** loopback access — it satisfies the Viewer requirement but never the Admin-only requirement, so anonymous localhost can view the dashboard but not reach API-key CRUD, session Close/Kill, or the alarm feed (`Authentication:Mode=Disabled` is scoped the same way). `Dashboard:DisableLogin` (default `false`) auto-authenticates every dashboard request — including remote browsers — as `Dashboard:AutoLoginUser` (default `multi-role`) with both Admin and Viewer roles; dev/test only, never enable in production.
+3 -2
View File
@@ -133,8 +133,9 @@ finding carries an unrecognised Status value. The PowerShell wrapper
`scripts/check-code-reviews-readme.ps1` runs that check and is the intended hook
for CI or a pre-commit step.
> The repo's installed `python` is the real interpreter; the bare `python3`
> alias resolves to the Windows Store stub and fails. Use `python`.
> On Windows, the installed interpreter may be available only as `python`
> because `python3` can resolve to the Windows Store stub. macOS and Linux
> normally use `python3`; CI uses the configured `python` executable.
The per-module `findings.md` files are the source of truth; `README.md` is the
aggregated index and must always agree with them — which the script guarantees.
@@ -155,6 +155,14 @@ Sequence these together rather than piecemeal — several are one change set spa
- ~~When CLI-38 lands, close old **CLI-08** with a pointer here.~~ Done 2026-08-07: CLI-38 landed and old CLI-08 is now `Done` in the first-cycle tracker, pointing at [CLI-38](50-clients.md#cli-38--align-netgojava-on-hresult--0-lands-prior-cli-08-cures-the-doc-drift---medium--p1).
- ~~When WRK-26 lands, its doc section also discharges the WorkerFrameProtocol gap~~ Done 2026-08-07: WRK-26 landed and **IPC-29** is discharged, pointing at [WRK-26](20-worker.md#wrk-26--write-priority-and-overflow-doc-drift-from-the-wrk-07-change---low--p1). ~~When TST-25 lands, revisit old **TST-05** (scheduled live smoke) and **TST-24** (client wire tests), which it unlocks.~~ TST-05 revisited 2026-08-10: `Partially done` in the first-cycle tracker — the `nightly-windev` job closes the scheduled-cadence half, but the finding's coverage-audit half stays open (the live suite reaches all six late-added COM commands and none of the five control commands). TST-24 revisited in the same change and closed `Done`: Go/Rust/Java already had real-server wire tests, and the two genuine gaps (.NET, Python) now have them plus a CI step.
## 2026-08-25 documentation correction
The 2026-08-07 IPC-23 change-log entry's `docs/Grpc.md` deviation rationale is inaccurate:
`DrainEvents` is not private worker IPC. It is a public `MxCommandKind` accepted by the
`Invoke` RPC. Cycle-3 IPC-38 adds the missing public count-cap, byte-cap,
drain-until-empty, and destructive-stream interaction contract to `docs/Grpc.md` while
preserving the historical entry below.
## Change log
| Date | Change |
@@ -17,8 +17,8 @@ This plan is executed as **seven parallel batches**, each by one subagent workin
| B | `fix/c3-glauth-doc` | TST-31 | sonnet | Reconcile a table against one TOML source of truth, re-derive one default, change an assert to a log | Done |
| C | `fix/c3-tracker-hygiene` | GWC-31 + housekeeping (close GWC-16/IPC-08/IPC-31, refresh TST-06/TST-17, register WRK-31) | sonnet | Cross-file comment/ID sweep needs care about what each ID *means*, but no logic changes | Done (`3619152`) |
| D | `fix/c3-dashboard-acl` | SEC-40, SEC-41, GWC-38 | opus | Authorization semantics, fail-closed decisions, claims re-resolution; each needs a regression test | **Done** (`b177636` GWC-38, `2ef7b93` SEC-41+SEC-40, `8257a03` comment fixup; branch not merged/pushed) |
| E | `fix/c3-ipc-boundaries` | IPC-33, IPC-39, IPC-34, IPC-35 | opus | Changes the WorkerClient command lifecycle, the options validator, and the worker drain budget; race-sensitive | **Done** (`577a0db` IPC-33, `a71663a` IPC-39, `b27035b` IPC-34, `ab41362` IPC-35; branch `fix/c3-ipc-boundaries`, not merged/pushed) — worker half pending windev (`WorkerPipeSession.ResolveDrainReplyByteBudget` and the `WorkerPipeSessionTests` floor-test comment cannot be built on macOS) |
| F | `fix/c3-worker-residuals` | WRK-32, WRK-33 (+WRK-34 opportunistic) | sonnet | Small, precisely located worker edits + doc prose; cannot be built on macOS so the agent's job is a careful, minimal diff | Done-pending-windev |
| E | `fix/c3-ipc-boundaries` | IPC-33, IPC-39, IPC-34, IPC-35 | opus | Changes the WorkerClient command lifecycle, the options validator, and the worker drain budget; race-sensitive | **Done** (`577a0db` IPC-33, `a71663a` IPC-39, `b27035b` IPC-34, `ab41362` IPC-35; branch `fix/c3-ipc-boundaries`, not merged/pushed) — worker half verified on windev (`f2ab9cc`, Worker.Tests 524/524) |
| F | `fix/c3-worker-residuals` | WRK-32, WRK-33 (+WRK-34 opportunistic) | sonnet | Small, precisely located worker edits + doc prose; cannot be built on macOS so the agent's job is a careful, minimal diff | Done (windev x86 build + Worker.Tests 524/524 green on `f2ab9cc`) |
| G | `fix/c3-session-lifecycle` | GWC-32, GWC-34, GWC-37 | opus | Distributor/dispose lifecycle races and an alarm-monitor re-resolve path; needs lock-discipline reasoning | Done (`058d8d5`, `282a0e7`, `1932217`) |
**Out of scope for this execution (recorded, not dropped):**
@@ -27,7 +27,10 @@ This plan is executed as **seven parallel batches**, each by one subagent workin
|---|---|---|
| TST-32 | Blocked (external) | Gitea runner registration-token reset is a web-UI action on the Gitea admin page; no repo change. Operator to-do. |
| SEC-37 | Blocked (external) | Family-wide: TLS on the shared GLAuth lives in `scadaproj/infra/glauth/` (separate repo + docker host). Plan separately; after it lands, move the deployed service to `Production`. |
| CLI-47/48/49/50, IPC-36, IPC-37, TST-33/34/35/36, GWC-33/35/36, SEC-38/39, IPC-38, WRK-34 | Deferred (P2) | Roadmap tier 10; WRK-34 may be picked up by batch F if trivial. |
| CLI-47/48/49/50, IPC-36/37, TST-35 | Not started | Activated by the 2026-08-25 P2 continuation below. |
| GWC-33/35, SEC-38/39 | Done | Closed by continuation batch I; regression counts and commits are recorded below. |
| GWC-36, IPC-38, TST-33/34/36 | Done | Closed by continuation batch H; verification and commit are recorded below. |
| WRK-34 | Done | Completed opportunistically in batch F (`3679e59`): watchdog faults use `CommandMethod=Watchdog` and carry the stuck correlation ID in the diagnostic message. Tracker correction recorded by the continuation audit. |
## Batch specifications
@@ -87,3 +90,149 @@ This plan is executed as **seven parallel batches**, each by one subagent workin
## Merge order
C is merged **last** (its comment sweep touches the same files E and G edit). Order: A, B, F, D, G, E, C. After merge: full `NonWindows` build + one full gateway test run; then windev worker CI for E+F.
## Integration gate (2026-08-20)
All seven branches merged into `main` (A `ee68cc3`, B `f59c34a`, C `f2ab9cc`, D `9f0c3fd`, E `cfbc995`, F `aa07497`, G `c6650c1`). Merged tree: `NonWindows.slnx` 0 warnings; full gateway suite **1183/1183** passed; windev `windev-worker-ci.ps1 -Sha f2ab9cc -Mode test`: x86 Worker build succeeded, `Worker.Tests` **524 passed / 11 skipped / 0 failed**. Follow-ups surfaced by the batches: a worker command to refresh a live alarm subscription's watch-list (would let GWC-37's re-resolve be applied, not just reported); WRK-32's proto reply-field comment deferred until the next proto regen wave; the five clients are bumped but not yet published.
## P2 continuation (planned 2026-08-25)
Current-state audit: refreshed `origin/main` and re-checked every deferred finding and integration-gate follow-up. Sixteen of the seventeen P2 findings remain open; WRK-34 was already completed by `3679e59`. All three follow-ups remain open. Exact target client coordinates are still absent from Gitea (HTTP 404), and `clients/go/v0.2.1` does not exist.
Approved design: [`../../../docs/plans/2026-08-25-cycle-3-p2-followups-design.md`](../../../docs/plans/2026-08-25-cycle-3-p2-followups-design.md) (`30548da`).
Execution plan: [`../../../docs/plans/2026-08-25-cycle-3-p2-followups.md`](../../../docs/plans/2026-08-25-cycle-3-p2-followups.md).
The continuation runs serially in the isolated `.worktrees/c3-p2-followups` worktree on `codex/c3-p2-followups`, with a tracker update at every batch boundary. Codex equivalents are selected from the implementation risk: `gpt-5.6-terra`/medium for contained documentation, CI, and client work; `gpt-5.6-sol`/high for concurrency, authorization, protocol, and live alarm lifecycle changes.
| Batch | Findings / follow-ups | Codex model | Status | Exit gate |
|---|---|---|---|---|
| H — tracker and currency | WRK-34 record, GWC-36, IPC-38, TST-33/34/36 | `gpt-5.6-terra` medium | Done | Currency greps clean; README check in portable CI; NonWindows build clean |
| I — gateway correctness/security | GWC-33/35, SEC-38/39 | `gpt-5.6-sol` high | Done | Focused regression tests green; NonWindows build clean |
| J — secondary contracts | IPC-36/37 | `gpt-5.6-sol` high | Done | Shared launch constants and descriptor parity tests green |
| K — client alarm parity | CLI-47/48/49/50 | `gpt-5.6-terra` high | Done | All five client suites green |
| L — live watch-list/proto wave | watch-list refresh follow-up + WRK-32 proto comment | `gpt-5.6-sol` high | Done | Codegen clean; portable alarm/contract tests green; windev Worker.Tests green |
| M — Windows gateway CI | TST-35 | `gpt-5.6-terra` medium | Done | windev `live` mode proves Worker, gateway, full-slnx, and live smoke tiers |
| N — integration and release | publish five clients + Contracts | `gpt-5.6-sol` high | In progress | Full local/windev gates; merged `main`; registry coordinates and Go tag verified |
### Continuation tasklist
#### H — tracker and currency
- [x] Correct WRK-34 from Deferred to Done with commit `3679e59`.
- [x] GWC-36: correct `EventSubscriberReplayAttachment.OldestAvailableSequence` semantics.
- [x] IPC-38: add public `DrainEvents` behavior to `docs/Grpc.md` and correct the cycle-2 omission rationale.
- [x] TST-33: reconcile the five stale/self-contradictory `stillpending.md` claims and mark its live-status boundary.
- [x] TST-34: run `code-reviews/regen-readme.py --check` in portable CI and qualify the interpreter note.
- [x] TST-36: replace prose ``event`` scope references with canonical ``events:read``.
- [x] Record batch-H commit and verification evidence here.
Batch-H verification (2026-08-25): `python3 code-reviews/regen-readme.py --check`
reported the generated README current and consistent; the TST-33 and live-document
TST-36 stale-pattern greps returned no matches; `docs/Grpc.md` contains the public
`DrainEvents` contract; and `dotnet build src/ZB.MOM.WW.MxGateway.NonWindows.slnx`
succeeded with **0 warnings / 0 errors** in 1.59 s. Batch commit: `92aae28`.
#### I — gateway correctness and security
- [x] GWC-33: unregister terminal clients from the event-queue-depth gauge without shortening fault retention.
- [x] GWC-35: repair acknowledged-to-active snapshot deltas with a Raise transition and instance identity.
- [x] SEC-38: feed/consult the aggregate for valid key IDs that overflow the peer partition cap.
- [x] SEC-39: prove a null effective key ID cannot reset the shared fallback partition.
- [x] Run focused tests and clean NonWindows build; record counts and commits here.
Batch-I verification (2026-08-25): the combined `WorkerClientTests`,
`GatewayAlarmMonitorAttachOrderTests`, and `ApiKeyFailureLimiterTests` filter passed
**55/55** tests in 12 s; `dotnet build src/ZB.MOM.WW.MxGateway.NonWindows.slnx`
succeeded with **0 warnings / 0 errors** in 3.42 s. Finding commits: GWC-33
`b0156c6`, GWC-35 `7807201`, and SEC-38/SEC-39 `1875426`.
#### J — secondary contracts
- [x] IPC-36: centralize all six worker-launch environment names in Contracts and consume them on both sides.
- [x] IPC-36: document variable/config/default mapping and test launcher completeness.
- [x] IPC-37: add normalized retained-vs-served Galaxy descriptor parity test.
- [x] Run focused tests and clean NonWindows build; record counts and commits here.
Batch-J verification (2026-08-25): the combined `WorkerProcessLauncherTests` and
`GalaxyContractParityTests` filter passed **10/10** tests in 1 s; the descriptor
mutation proof identified the changed protobuf message; and `dotnet build
src/ZB.MOM.WW.MxGateway.NonWindows.slnx` succeeded with **0 warnings / 0
errors** in 2.20 s. IPC-36/IPC-37 commit: `d6c5a11`.
#### K — client alarm-feed parity
- [x] CLI-47: make .NET text status flags explicit and document deliberate JSON differences.
- [x] CLI-48: test provider/snapshot status arms in .NET, Go, and Java; add Go stream-alarm scaffold.
- [x] CLI-49: export Go aliases for `AlarmProviderStatus` and `AlarmSnapshotStatus`; fix five-arm documentation.
- [x] CLI-50: stream Python alarm rows incrementally with an unbounded default and bounded opt-in.
- [x] Run .NET, Go, Python, Rust, and Java client suites; record counts and commits here.
Batch-K verification (2026-08-25): `dotnet test
clients/dotnet/ZB.MOM.WW.MxGateway.Client.slnx` passed **134/134** portable
tests (1 live test skipped); `go test ./...` passed both tested packages;
`python -m pytest` passed **169/169** portable tests (1 opt-in TLS test
skipped); `cargo test --workspace` passed **108/108** tests; and
`JAVA_HOME=/opt/homebrew/opt/openjdk@17 gradle test` passed **132/132** tests.
CLI-47/CLI-48/CLI-49/CLI-50 commit: `5cdb3da`.
#### L — live watch-list refresh and protobuf wave
- [x] Add explicit fallback-enabled subscription semantics for an initially empty discovered list.
- [x] Add and round-trip `UpdateAlarmWatchListCommand` and its `MxCommandKind`.
- [x] Route the command through the worker executor and alarm handler on the STA.
- [x] Build/subscribe/attach a replacement standby before swap; preserve the old standby on failure.
- [x] Preserve current primary/subtag mode during a successful replacement.
- [x] Re-resolve on the existing gateway reconciliation cadence and send only material changes/recovery.
- [x] Refresh immediately at failover edge, update acknowledged fingerprint only after worker OK, and reconcile after success.
- [x] Expand WRK-32's public reply-field comment to the sequential stale-callback/recency case.
- [x] Replace IPC-36's internal env-var name in the proto comment with the public configuration key.
- [x] Regenerate Contracts and all client generated trees once; run codegen and all affected suites.
- [x] Verify x86 Worker build and Worker.Tests on windev; record SHA/counts here.
Batch-L verification (2026-08-25): `pwsh scripts/check-codegen.ps1` passed all
four descriptor/C#/Rust/Go/Python freshness checks, and Java
`checkGeneratedClean` passed. The exact portable alarm/contract/descriptor
filter passed **77/77**; `NonWindows.slnx` built with **0 warnings / 0 errors**;
the .NET, Go, Python, Rust, and Java client suites were green (respectively
**134 passed / 1 live skip**, both Go packages, **169 passed / 1 opt-in TLS
skip**, **108 passed**, and **132 passed**). Windev `test` mode at
`986c6c0ce4f02c62b4f59ec44780de5af6d291b8` built the x86/net48 Worker with
**0 warnings / 0 errors** and passed Worker.Tests **529 passed / 11 opt-in
skips / 0 failed**. Feature/proto commit: `484028e`; Batch-J x86 compile
follow-up: `986c6c0`.
#### M — Windows gateway CI
- [x] TST-35: add the serialized net10 gateway suite to windev `live` mode.
- [x] Update `docs/GatewayTesting.md` and `scripts/ci/README.md` with order/failure semantics.
- [x] Execute `live` mode on windev and record Worker, gateway, full-slnx, and live-smoke evidence here.
Batch-M verification (2026-08-25): windev `live` mode at exact commit
`cc143b52cf04be0b2f08350922a987b2e250bc1d` completed the serialized sequence in
**4m45s** with remote exit code 0. The x86/net48 Worker build succeeded with **0
warnings / 0 errors** in 22.59 s; Worker.Tests passed **529**, skipped **11**
opt-in probes, and failed **0**; the full Windows `slnx` build succeeded with **0
warnings / 0 errors** in 43.24 s; the net10 gateway suite passed **1193/1193**
in 53 s; and the live-MXAccess smoke passed **8/8** in 1m46s. TST-35 commit:
`cc143b5`.
#### N — integration and release
- [x] Run `git diff --check`, codegen, client-version, README, NonWindows build, and full gateway suite gates.
- [x] Run all five client suites on the final candidate.
- [ ] Run final windev `live` gate at the exact candidate SHA.
- [ ] Merge/push the verified continuation to `main`; wait for green CI.
- [ ] Re-query all package coordinates and Go tag immediately before publish; abort on collision.
- [ ] Publish .NET Client + Contracts 0.2.1, Python 0.2.1, Rust 0.2.1, Java 0.2.2.
- [ ] Create/push `clients/go/v0.2.1` at the exact release commit.
- [ ] Verify every registry coordinate and Go module/tag, update `docs/ClientPackaging.md`, and record immutable release evidence here.
Batch-N portable verification (2026-08-25): `git diff --check`, codegen freshness,
client-version freshness, and the generated README check all passed without changing
the worktree. `NonWindows.slnx` built with **0 warnings / 0 errors**; the gateway
suite passed **1193/1193**, and the portable integration suite passed **5** with
**19** live-only skips. The final .NET, Go, Python, Rust, and Java client gates were
green (respectively **134 passed / 1 live skip**, all Go packages built/tested with
clean `gofmt`, **169 passed / 1 opt-in TLS skip** from a clean virtual environment,
**108 passed** plus clean `cargo fmt`/Clippy, and **132 passed / 0 failed** with the
generated Java tree unchanged).
+6 -1
View File
@@ -166,6 +166,11 @@ when the gateway's alarm monitor restarts and drops a truncated verdict. Track
it if you need set completeness on a live feed without polling
`QueryActiveAlarmsAsync`.
The CLI's text rows always spell out `degraded=false` and `truncated=false` so
healthy and clearing frames remain visible. Its protobuf-JSON modes retain the
standard proto3 omission rule, so those same false-valued payloads appear as
`{"providerStatus": {}}` and `{"snapshotStatus": {}}`; absent means false.
`MxGatewaySession.CloseAsync` is explicit and idempotent. Repeated calls return
the first `CloseSessionReply` instead of sending another close request.
@@ -285,7 +290,7 @@ dotnet run --project clients/dotnet/ZB.MOM.WW.MxGateway.Client.Cli -- advise --s
dotnet run --project clients/dotnet/ZB.MOM.WW.MxGateway.Client.Cli -- write --session-id <id> --server-handle 1 --item-handle 1 --type int32 --value 123 --json
dotnet run --project clients/dotnet/ZB.MOM.WW.MxGateway.Client.Cli -- write2 --session-id <id> --server-handle 1 --item-handle 1 --type int32 --value 123 --timestamp 2026-01-01T00:00:00Z --json
dotnet run --project clients/dotnet/ZB.MOM.WW.MxGateway.Client.Cli -- stream-events --session-id <id> --max-events 1 --json
dotnet run --project clients/dotnet/ZB.MOM.WW.MxGateway.Client.Cli -- stream-alarms --filter-prefix Area001 --max-events 1 --json
dotnet run --project clients/dotnet/ZB.MOM.WW.MxGateway.Client.Cli -- stream-alarms --filter-prefix Area001 --max-events 4 --json
dotnet run --project clients/dotnet/ZB.MOM.WW.MxGateway.Client.Cli -- acknowledge-alarm --reference "\\Galaxy\Area001.Pump001.PumpFault" --comment "ack from cli" --operator operator1 --json
dotnet run --project clients/dotnet/ZB.MOM.WW.MxGateway.Client.Cli -- smoke --endpoint http://localhost:5000 --api-key-env MXGATEWAY_API_KEY --item Area001.Pump001.Speed --json
```
@@ -1561,9 +1561,9 @@ public static class MxGatewayClientCli
AlarmFeedMessage.PayloadOneofCase.Transition =>
$"transition {ProtobufJsonFormatter.Format(feedMessage.Transition)}",
AlarmFeedMessage.PayloadOneofCase.ProviderStatus =>
$"provider-status {ProtobufJsonFormatter.Format(feedMessage.ProviderStatus)}",
$"provider-status mode={feedMessage.ProviderStatus.Mode} degraded={feedMessage.ProviderStatus.Degraded.ToString().ToLowerInvariant()} reason=\"{feedMessage.ProviderStatus.Reason}\"",
AlarmFeedMessage.PayloadOneofCase.SnapshotStatus =>
$"snapshot-status {ProtobufJsonFormatter.Format(feedMessage.SnapshotStatus)}",
$"snapshot-status truncated={feedMessage.SnapshotStatus.Truncated.ToString().ToLowerInvariant()}",
_ => $"unknown-payload {feedMessage.PayloadCase}",
};
}
@@ -705,6 +705,50 @@ public sealed class MxGatewayClientCliTests
Assert.Equal(string.Empty, error.ToString());
}
/// <summary>Verifies that false alarm-feed status flags remain explicit in text while protobuf JSON omits defaults.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task RunAsync_StreamAlarms_StatusFramesDistinguishTextFromProtobufJson()
{
FakeCliClient fakeClient = new();
fakeClient.AlarmFeedMessages.Add(new AlarmFeedMessage
{
ProviderStatus = new AlarmProviderStatus { Degraded = false },
});
fakeClient.AlarmFeedMessages.Add(new AlarmFeedMessage
{
SnapshotStatus = new AlarmSnapshotStatus { Truncated = false },
});
using var textOutput = new StringWriter();
using var textError = new StringWriter();
int textExit = await MxGatewayClientCli.RunAsync(
["stream-alarms", "--endpoint", "http://localhost:5000", "--api-key", "test-api-key", "--max-events", "2"],
textOutput,
textError,
_ => fakeClient);
Assert.Equal(0, textExit);
Assert.Equal(
["provider-status mode=Unspecified degraded=false reason=\"\"", "snapshot-status truncated=false"],
textOutput.ToString().Split(Environment.NewLine, StringSplitOptions.RemoveEmptyEntries));
Assert.Equal(string.Empty, textError.ToString());
using var jsonOutput = new StringWriter();
using var jsonError = new StringWriter();
int jsonExit = await MxGatewayClientCli.RunAsync(
["stream-alarms", "--endpoint", "http://localhost:5000", "--api-key", "test-api-key", "--max-events", "2", "--jsonl"],
jsonOutput,
jsonError,
_ => fakeClient);
Assert.Equal(0, jsonExit);
Assert.Equal(
["{ \"providerStatus\": { } }", "{ \"snapshotStatus\": { } }"],
jsonOutput.ToString().Split(Environment.NewLine, StringSplitOptions.RemoveEmptyEntries));
Assert.Equal(string.Empty, jsonError.ToString());
}
/// <summary>Verifies that acknowledge-alarm builds a request and prints the JSON reply.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
+6 -1
View File
@@ -162,6 +162,11 @@ on every verdict change — including the clearing frame sent when the gateway's
alarm monitor restarts and drops a truncated verdict. Track it if you need set
completeness on a live feed without polling `QueryActiveAlarms`.
The CLI's text rows always spell out `degraded=false` and `truncated=false`.
Its protobuf-JSON mode retains the standard proto3 omission rule, so those
false-valued payloads appear as `{"providerStatus":{}}` and
`{"snapshotStatus":{}}`; absent means false.
## Write Semantics And Common Pitfalls
These are MXAccess parity behaviors that surprise new callers. The gateway
@@ -439,7 +444,7 @@ go run ./cmd/mxgw-go write -session-id <id> -server-handle 1 -item-handle 1 -typ
go run ./cmd/mxgw-go write-bulk -session-id <id> -server-handle 1 -item-handles 1,2 -values 10,20 -type int32 -plaintext -json
go run ./cmd/mxgw-go read-bulk -session-id <id> -item-handles 1,2 -plaintext -json
go run ./cmd/mxgw-go stream-events -session-id <id> -plaintext -json
go run ./cmd/mxgw-go stream-alarms -plaintext -json
go run ./cmd/mxgw-go stream-alarms -plaintext -limit 4 -json
go run ./cmd/mxgw-go smoke -item Area001.Tag.Value -plaintext -json
go run ./cmd/mxgw-go galaxy-test-connection -plaintext -json
go run ./cmd/mxgw-go galaxy-last-deploy -plaintext -json
+62
View File
@@ -283,6 +283,68 @@ type pingFakeGateway struct {
pb.UnimplementedMxAccessGatewayServer
}
// alarmStatusFakeGateway emits both status arms with their proto3 boolean
// defaults so the CLI contract covers values that protobuf JSON omits.
type alarmStatusFakeGateway struct {
pb.UnimplementedMxAccessGatewayServer
}
func (g *alarmStatusFakeGateway) StreamAlarms(
_ *pb.StreamAlarmsRequest,
stream grpc.ServerStreamingServer[pb.AlarmFeedMessage],
) error {
if err := stream.Send(&pb.AlarmFeedMessage{
Payload: &pb.AlarmFeedMessage_ProviderStatus{
ProviderStatus: &pb.AlarmProviderStatus{Degraded: false},
},
}); err != nil {
return err
}
return stream.Send(&pb.AlarmFeedMessage{
Payload: &pb.AlarmFeedMessage_SnapshotStatus{
SnapshotStatus: &pb.AlarmSnapshotStatus{Truncated: false},
},
})
}
func startAlarmStatusGateway(t *testing.T) string {
t.Helper()
listener, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatalf("listen: %v", err)
}
server := grpc.NewServer()
pb.RegisterMxAccessGatewayServer(server, &alarmStatusFakeGateway{})
go func() { _ = server.Serve(listener) }()
t.Cleanup(func() {
server.Stop()
_ = listener.Close()
})
return listener.Addr().String()
}
func TestRunStreamAlarmsPrintsStatusArmsInTextAndProtobufJSON(t *testing.T) {
endpoint := startAlarmStatusGateway(t)
baseArgs := []string{"stream-alarms", "-endpoint", endpoint, "-plaintext", "-api-key", "test", "-limit", "2"}
var textOut, textErr bytes.Buffer
if err := runWithIO(t.Context(), baseArgs, &textOut, &textErr); err != nil {
t.Fatalf("text stream-alarms error = %v; stderr = %s", err, textErr.String())
}
if got, want := strings.TrimSpace(textOut.String()), "provider-status mode=ALARM_PROVIDER_MODE_UNSPECIFIED degraded=false reason=\"\"\nsnapshot-status truncated=false"; got != want {
t.Fatalf("text stream-alarms output = %q, want %q", got, want)
}
var jsonOut, jsonErr bytes.Buffer
jsonArgs := append(append([]string{}, baseArgs...), "-json")
if err := runWithIO(t.Context(), jsonArgs, &jsonOut, &jsonErr); err != nil {
t.Fatalf("JSON stream-alarms error = %v; stderr = %s", err, jsonErr.String())
}
if got, want := strings.TrimSpace(jsonOut.String()), "{\"providerStatus\":{}}\n{\"snapshotStatus\":{}}"; got != want {
t.Fatalf("JSON stream-alarms output = %q, want protobuf JSON %q", got, want)
}
}
func (g *pingFakeGateway) Invoke(_ context.Context, req *pb.MxCommandRequest) (*pb.MxCommandReply, error) {
echo := "pong:" + req.GetCommand().GetPing().GetMessage()
return &pb.MxCommandReply{
File diff suppressed because it is too large Load Diff
+7 -2
View File
@@ -127,9 +127,14 @@ type (
QueryActiveAlarmsRequest = pb.QueryActiveAlarmsRequest
// StreamAlarmsRequest is the gateway StreamAlarms request message.
StreamAlarmsRequest = pb.StreamAlarmsRequest
// AlarmFeedMessage is one message on the StreamAlarms feed — an
// active-alarm snapshot row, a snapshot-complete sentinel, or a transition.
// AlarmFeedMessage is one message on the StreamAlarms feed — an active-alarm
// snapshot row, snapshot-complete sentinel, transition, provider status, or
// snapshot-completeness status.
AlarmFeedMessage = pb.AlarmFeedMessage
// AlarmProviderStatus reports the alarm provider mode and degradation state.
AlarmProviderStatus = pb.AlarmProviderStatus
// AlarmSnapshotStatus reports whether the cached alarm snapshot is truncated.
AlarmSnapshotStatus = pb.AlarmSnapshotStatus
// ActiveAlarmSnapshot is one row in a ConditionRefresh stream.
ActiveAlarmSnapshot = pb.ActiveAlarmSnapshot
// OnAlarmTransitionEvent is the body carried by alarm-transition MxEvents.
+17
View File
@@ -0,0 +1,17 @@
package mxgateway_test
import (
"testing"
"gitea.dohertylan.com/dohertj2/mxaccessgw/clients/go/mxgateway"
)
// TestAlarmFeedStatusAliasesArePublic is a compile-time consumer contract for
// the two status payloads carried by AlarmFeedMessage.
func TestAlarmFeedStatusAliasesArePublic(t *testing.T) {
t.Helper()
var provider mxgateway.AlarmProviderStatus
var snapshot mxgateway.AlarmSnapshotStatus
_ = provider
_ = snapshot
}
+6 -1
View File
@@ -135,6 +135,11 @@ the clearing frame sent when the gateway's alarm monitor restarts and drops a
truncated verdict. Track it if you need set completeness on a live feed without
polling `queryActiveAlarms`.
The CLI's text rows always spell out `degraded=false` and `truncated=false`.
Its protobuf-JSON mode retains the standard proto3 omission rule, so those
false-valued payloads appear as `{"providerStatus":{}}` and
`{"snapshotStatus":{}}`; absent means false.
## Write Semantics And Common Pitfalls
These are MXAccess parity behaviors that surprise new callers. The gateway
@@ -411,7 +416,7 @@ gradle :zb-mom-ww-mxgateway-cli:run --args="advise-supervisory --endpoint localh
gradle :zb-mom-ww-mxgateway-cli:run --args="authenticate-user --endpoint localhost:5000 --api-key-env MXGATEWAY_API_KEY --plaintext --session-id <id> --server-handle 1 --verify-user operator --password-env MXGATEWAY_VERIFY_PASSWORD --json"
gradle :zb-mom-ww-mxgateway-cli:run --args="write-secured --endpoint localhost:5000 --api-key-env MXGATEWAY_API_KEY --plaintext --session-id <id> --server-handle 1 --item-handle 1 --type int32 --value 123 --current-user-id 100 --verifier-user-id 100 --json"
gradle :zb-mom-ww-mxgateway-cli:run --args="stream-events --endpoint localhost:5000 --api-key-env MXGATEWAY_API_KEY --plaintext --session-id <id> --limit 1 --json"
gradle :zb-mom-ww-mxgateway-cli:run --args="stream-alarms --endpoint localhost:5000 --api-key-env MXGATEWAY_API_KEY --plaintext --filter-prefix Galaxy --limit 1 --json"
gradle :zb-mom-ww-mxgateway-cli:run --args="stream-alarms --endpoint localhost:5000 --api-key-env MXGATEWAY_API_KEY --plaintext --filter-prefix Galaxy --limit 4 --json"
gradle :zb-mom-ww-mxgateway-cli:run --args="acknowledge-alarm --endpoint localhost:5000 --api-key-env MXGATEWAY_API_KEY --plaintext --reference \"\\Galaxy\Area001.Pump001.PumpFault\" --json"
gradle :zb-mom-ww-mxgateway-cli:run --args="smoke --endpoint localhost:5000 --api-key-env MXGATEWAY_API_KEY --plaintext --item TestObject.TestInt --json"
```
File diff suppressed because it is too large Load Diff
@@ -27,6 +27,8 @@ import mxaccess_gateway.v1.MxaccessGateway.ActiveAlarmSnapshot;
import mxaccess_gateway.v1.MxaccessGateway.AddItemReply;
import mxaccess_gateway.v1.MxaccessGateway.AlarmConditionState;
import mxaccess_gateway.v1.MxaccessGateway.AlarmFeedMessage;
import mxaccess_gateway.v1.MxaccessGateway.AlarmProviderStatus;
import mxaccess_gateway.v1.MxaccessGateway.AlarmSnapshotStatus;
import mxaccess_gateway.v1.MxaccessGateway.AlarmTransitionKind;
import mxaccess_gateway.v1.MxaccessGateway.BulkReadResult;
import mxaccess_gateway.v1.MxaccessGateway.BulkWriteResult;
@@ -685,6 +687,21 @@ final class MxGatewayCliTests {
assertTrue(run.output().contains("\"snapshotComplete\""), run.output());
}
@Test
void streamAlarmsCommandPrintsStatusArmsInTextAndProtobufJson() {
FakeClientFactory factory = new FakeClientFactory();
CliRun text = execute(factory, "stream-alarms");
List<String> textRows = text.output().lines().filter(line -> !line.isBlank()).toList();
assertEquals("provider-status mode=ALARM_PROVIDER_MODE_UNSPECIFIED degraded=false reason=", textRows.get(3));
assertEquals("snapshot-status truncated=false", textRows.get(4));
CliRun json = execute(factory, "stream-alarms", "--json");
List<String> jsonRows = json.output().lines().filter(line -> !line.isBlank()).toList();
assertEquals("{\"providerStatus\":{}}", jsonRows.get(3));
assertEquals("{\"snapshotStatus\":{}}", jsonRows.get(4));
}
@Test
void acknowledgeAlarmCommandForwardsOptionsAndPrintsReply() {
FakeClientFactory factory = new FakeClientFactory();
@@ -1409,6 +1426,12 @@ final class MxGatewayCliTests {
.setTransitionKind(AlarmTransitionKind.ALARM_TRANSITION_KIND_ACKNOWLEDGE)
.setSeverity(700))
.build());
observer.onNext(AlarmFeedMessage.newBuilder()
.setProviderStatus(AlarmProviderStatus.newBuilder().setDegraded(false))
.build());
observer.onNext(AlarmFeedMessage.newBuilder()
.setSnapshotStatus(AlarmSnapshotStatus.newBuilder().setTruncated(false))
.build());
observer.onCompleted();
return new MxGatewayAlarmFeedSubscription();
}
+9 -1
View File
@@ -132,6 +132,14 @@ sent when the gateway's alarm monitor restarts and drops a truncated verdict.
Track it if you need set completeness on a live feed without polling
`query_active_alarms`.
`mxgw-py stream-alarms` writes one JSON row as each feed message arrives. The
default `--max-messages 0` is unbounded; use a positive value for a finite
sample, remembering that the provider-status and snapshot-status preamble both
count toward it. `--timeout` ends the command after an idle wait and the stream
is closed in every exit path. Protobuf JSON omits false-valued scalar defaults,
so healthy/clearing frames appear as `{"providerStatus": {}}` and
`{"snapshotStatus": {}}`; absent means false.
Canceling a Python task cancels the client-side gRPC call or stream wait. It
does not abort an in-flight MXAccess COM call inside the worker process.
@@ -448,7 +456,7 @@ mxgw-py register --session-id <id> --client-name python-client --json
mxgw-py add-item --session-id <id> --server-handle 1 --item Object.Attribute --json
mxgw-py advise --session-id <id> --server-handle 1 --item-handle 2 --json
mxgw-py stream-events --session-id <id> --max-events 1 --json
mxgw-py stream-alarms --max-messages 1 --json
mxgw-py stream-alarms --max-messages 4 --json
mxgw-py acknowledge-alarm --reference "\\Galaxy\Area001.Pump001.PumpFault" --json
mxgw-py write --session-id <id> --server-handle 1 --item-handle 2 --type int32 --value 123 --json
```
File diff suppressed because one or more lines are too long
@@ -511,16 +511,15 @@ def stream_events(**kwargs: Any) -> None:
@main.command("stream-alarms")
@gateway_options
@click.option("--filter-prefix", default="", help="Alarm-reference prefix filter.")
@click.option("--max-messages", default=1, type=int, show_default=True)
@click.option("--max-messages", default=0, type=int, show_default=True, help="Maximum rows to emit; 0 means unbounded.")
@click.option("--timeout", default=5.0, type=float, show_default=True)
@click.option("--correlation-id", default="", help="Client correlation id.")
@click.option("--json", "output_json", is_flag=True, help="Emit JSON output.")
def stream_alarms(**kwargs: Any) -> None:
"""Stream a bounded number of messages from the gateway's central alarm feed."""
"""Stream messages from the gateway's central alarm feed as they arrive."""
_run(
_run_streaming(
_stream_alarms(**kwargs),
output_json=kwargs["output_json"],
secrets=_secrets(kwargs),
)
@@ -1119,19 +1118,39 @@ async def _stream_events(**kwargs: Any) -> dict[str, Any]:
return {"events": [_event_row(event) for event in events]}
async def _stream_alarms(**kwargs: Any) -> dict[str, Any]:
async with await _connect(kwargs) as client:
messages = await _collect_alarm_messages(
client.stream_alarms(
pb.StreamAlarmsRequest(
client_correlation_id=kwargs["correlation_id"],
alarm_filter_prefix=kwargs["filter_prefix"],
),
),
max_messages=kwargs["max_messages"],
timeout=kwargs["timeout"],
async def _stream_alarms(**kwargs: Any) -> None:
max_messages = int(kwargs["max_messages"])
if max_messages > MAX_AGGREGATE_EVENTS:
raise click.BadParameter(
f"must be less than or equal to {MAX_AGGREGATE_EVENTS}",
param_hint="--max-messages",
)
return {"messages": [_message_dict(message) for message in messages]}
async with await _connect(kwargs) as client:
messages = client.stream_alarms(
pb.StreamAlarmsRequest(
client_correlation_id=kwargs["correlation_id"],
alarm_filter_prefix=kwargs["filter_prefix"],
),
)
iterator = messages.__aiter__()
emitted = 0
try:
while max_messages == 0 or emitted < max_messages:
try:
message = await asyncio.wait_for(
iterator.__anext__(),
timeout=kwargs["timeout"],
)
except (StopAsyncIteration, asyncio.TimeoutError):
break
_emit(_message_dict(message), output_json=kwargs["output_json"])
sys.stdout.flush()
emitted += 1
finally:
close = getattr(iterator, "aclose", None)
if close is not None:
await close()
async def _acknowledge_alarm(**kwargs: Any) -> dict[str, Any]:
@@ -1489,6 +1508,17 @@ def _run(
_emit(payload, output_json=output_json)
def _run_streaming(
awaitable: Awaitable[None],
*,
secrets: list[str | None],
) -> None:
try:
asyncio.run(awaitable)
except MxGatewayError as error:
raise click.ClickException(redact_secret(str(error), secrets)) from error
def _emit(
payload: dict[str, Any],
*,
@@ -1536,34 +1566,6 @@ async def _collect_events(
return collected
async def _collect_alarm_messages(
messages: Any,
*,
max_messages: int,
timeout: float,
) -> list[pb.AlarmFeedMessage]:
if max_messages > MAX_AGGREGATE_EVENTS:
raise click.BadParameter(
f"must be less than or equal to {MAX_AGGREGATE_EVENTS}",
param_hint="--max-messages",
)
collected: list[pb.AlarmFeedMessage] = []
iterator = messages.__aiter__()
try:
while len(collected) < max_messages:
collected.append(await asyncio.wait_for(iterator.__anext__(), timeout=timeout))
except StopAsyncIteration:
pass
except asyncio.TimeoutError:
pass
finally:
close = getattr(iterator, "aclose", None)
if close is not None:
await close()
return collected
async def _collect_deploy_events(
events: Any,
*,
@@ -7,6 +7,7 @@ fixed source.
from __future__ import annotations
import asyncio
import json
import re
import time as _time_module_ref
@@ -296,6 +297,22 @@ class _AlarmFakeStream:
self.cancelled = True
class _IncrementalAlarmFakeStream(_AlarmFakeStream):
def __init__(self, messages: list[pb.AlarmFeedMessage], emitted: list[str]) -> None:
super().__init__(messages)
self._emitted = emitted
self._yielded = 0
async def __anext__(self) -> pb.AlarmFeedMessage:
if self._yielded > 0:
assert len(self._emitted) >= self._yielded, "previous alarm row was buffered"
if not self._messages:
await asyncio.Event().wait()
raise AssertionError("unreachable")
self._yielded += 1
return self._messages.pop(0)
class _BulkFakeUnary:
def __init__(self, replies: list[Any]) -> None:
self.replies = replies
@@ -667,8 +684,15 @@ def test_cli_write_bulk_happy_path(monkeypatch) -> None:
assert cmd.kind == pb.MX_COMMAND_KIND_WRITE_BULK
def test_cli_stream_alarms_happy_path(monkeypatch) -> None:
transitions = [
def test_cli_stream_alarms_default_streams_status_cached_and_live_rows_incrementally(
monkeypatch,
) -> None:
messages = [
pb.AlarmFeedMessage(provider_status=pb.AlarmProviderStatus(degraded=False)),
pb.AlarmFeedMessage(snapshot_status=pb.AlarmSnapshotStatus(truncated=False)),
pb.AlarmFeedMessage(
active_alarm=pb.ActiveAlarmSnapshot(alarm_full_reference="Tank01.Level.HiHi"),
),
pb.AlarmFeedMessage(
transition=pb.OnAlarmTransitionEvent(
alarm_full_reference="Tank01.Level.HiHi",
@@ -676,7 +700,15 @@ def test_cli_stream_alarms_happy_path(monkeypatch) -> None:
),
),
]
stream = _AlarmFakeStream(transitions)
emitted: list[str] = []
real_echo = cli_commands.click.echo
def recording_echo(message=None, **kwargs) -> None:
emitted.append(str(message))
real_echo(message, **kwargs)
monkeypatch.setattr(cli_commands.click, "echo", recording_echo)
stream = _IncrementalAlarmFakeStream(messages, emitted)
stub = _BulkFakeStub()
stub.set_alarm_stream(stream)
_install_fake_connect(monkeypatch, stub)
@@ -689,10 +721,8 @@ def test_cli_stream_alarms_happy_path(monkeypatch) -> None:
"--endpoint",
"localhost:5000",
"--plaintext",
"--max-messages",
"1",
"--timeout",
"5.0",
"0.01",
"--filter-prefix",
"Tank01.",
"--json",
@@ -700,8 +730,44 @@ def test_cli_stream_alarms_happy_path(monkeypatch) -> None:
)
assert result.exit_code == 0, result.output
payload = json.loads(result.output)
assert payload["messages"][0]["transition"]["alarmFullReference"] == "Tank01.Level.HiHi"
rows = [json.loads(line) for line in result.output.splitlines()]
assert [next(iter(row)) for row in rows] == [
"providerStatus",
"snapshotStatus",
"activeAlarm",
"transition",
]
assert stream.cancelled
def test_cli_stream_alarms_positive_max_messages_remains_finite(monkeypatch) -> None:
messages = [
pb.AlarmFeedMessage(provider_status=pb.AlarmProviderStatus()),
pb.AlarmFeedMessage(snapshot_status=pb.AlarmSnapshotStatus()),
pb.AlarmFeedMessage(active_alarm=pb.ActiveAlarmSnapshot()),
]
stream = _AlarmFakeStream(messages)
stub = _BulkFakeStub()
stub.set_alarm_stream(stream)
_install_fake_connect(monkeypatch, stub)
result = CliRunner().invoke(
main,
[
"stream-alarms",
"--endpoint",
"localhost:5000",
"--plaintext",
"--max-messages",
"2",
"--json",
],
)
assert result.exit_code == 0, result.output
rows = [json.loads(line) for line in result.output.splitlines()]
assert [next(iter(row)) for row in rows] == ["providerStatus", "snapshotStatus"]
assert stream.cancelled
def test_cli_acknowledge_alarm_happy_path(monkeypatch) -> None:
+6 -1
View File
@@ -77,7 +77,7 @@ cargo run -p mxgw-cli -- register --session-id <session-id> --client-name mxgw-r
cargo run -p mxgw-cli -- add-item --session-id <session-id> --server-handle 1 --item TestChildObject.TestInt --json
cargo run -p mxgw-cli -- advise --session-id <session-id> --server-handle 1 --item-handle 1 --json
cargo run -p mxgw-cli -- stream-events --session-id <session-id> --max-events 1 --json
cargo run -p mxgw-cli -- stream-alarms --session-id <session-id> --max-messages 1 --json
cargo run -p mxgw-cli -- stream-alarms --session-id <session-id> --max-messages 4 --json
cargo run -p mxgw-cli -- acknowledge-alarm --session-id <session-id> --alarm-reference "\\Galaxy\Area001.Pump001.PumpFault" --json
cargo run -p mxgw-cli -- write --session-id <session-id> --server-handle 1 --item-handle 1 --value-type int32 --value 123 --json
```
@@ -141,6 +141,11 @@ every verdict change — including the clearing frame sent when the gateway's
alarm monitor restarts and drops a truncated verdict. Track it if you need set
completeness on a live feed without polling `query_active_alarms`.
Unlike the protobuf-JSON formatters used by the other CLIs, the Rust CLI's
hand-built JSON status rows always include `"degraded": false` and
`"truncated": false`. This deliberate representational difference is
documented in the cross-language smoke matrix; both shapes mean the same thing.
The session helpers keep MXAccess handles visible:
```rust
+20 -4
View File
@@ -130,6 +130,7 @@ message MxCommand {
WriteSecuredBulkCommand write_secured_bulk = 41;
WriteSecured2BulkCommand write_secured2_bulk = 42;
ReadBulkCommand read_bulk = 43;
UpdateAlarmWatchListCommand update_alarm_watch_list = 44;
PingCommand ping = 100;
GetSessionStateCommand get_session_state = 101;
GetWorkerInfoCommand get_worker_info = 102;
@@ -174,6 +175,7 @@ enum MxCommandKind {
MX_COMMAND_KIND_WRITE_SECURED_BULK = 32;
MX_COMMAND_KIND_WRITE_SECURED2_BULK = 33;
MX_COMMAND_KIND_READ_BULK = 34;
MX_COMMAND_KIND_UPDATE_ALARM_WATCH_LIST = 35;
MX_COMMAND_KIND_PING = 100;
MX_COMMAND_KIND_GET_SESSION_STATE = 101;
MX_COMMAND_KIND_GET_WORKER_INFO = 102;
@@ -344,8 +346,8 @@ enum AlarmProviderMode {
// Subscription expression follows the canonical
// `\\<machine>\Galaxy!<area>` format (literal "Galaxy" provider). The
// worker spins up a wnwrapConsumer-backed subscription on its STA on
// first call; subsequent calls are an error (use UnsubscribeAlarms then
// SubscribeAlarms to reconfigure).
// first call; subsequent calls are an error. Use UpdateAlarmWatchList to
// replace the standby targets without restarting the primary subscription.
message SubscribeAlarmsCommand {
string subscription_expression = 1;
// UNSPECIFIED = auto-failover/failback. ALARMMGR/SUBTAG force one provider.
@@ -354,6 +356,17 @@ message SubscribeAlarmsCommand {
// alarmmgr mode; in subtag mode it bounds what the consumer can observe.
repeated AlarmSubtagTarget watch_list = 3;
AlarmFailoverConfig failover = 4;
// True when auto mode should retain subtag-fallback capability even if the
// gateway's initial discovery produced an empty watch list. False leaves an
// empty list as an intentional alarm-manager-only subscription.
bool fallback_enabled = 5;
}
// Replace the live alarm subscription's subtag standby targets. The worker
// constructs and attaches the replacement before swapping it into service, so
// a setup failure leaves the existing standby and active provider unchanged.
message UpdateAlarmWatchListCommand {
repeated AlarmSubtagTarget watch_list = 1;
}
// Tear down the worker's alarm consumer. No-op if no subscription is
@@ -537,11 +550,14 @@ message MxCommandReply {
MxValue return_value = 6;
// Correlated per-item outcome rows. For WRITE / WRITE2 / WRITE_SECURED /
// WRITE_SECURED2 replies the worker holds the reply for a bounded window
// (default 1.5 s, MXGATEWAY_WORKER_WRITE_COMPLETION_WAIT_MS) waiting for
// (default 1.5 s, MxGateway:Worker:WriteCompletionWaitMilliseconds) waiting for
// the matching MXAccess OnWriteComplete callback and copies its status rows
// here, so statuses[0] carries the real MXAccess commit outcome (success OR
// failure) while protocol_status/hresult still describe command acceptance
// only. Empty statuses on a write reply means the completion did not arrive
// only. Callback correlation is recency-based per item handle: after the
// bounded wait, a sequential write's callback supersedes a timed-out
// predecessor, and a later stale callback for that predecessor is ignored.
// Empty statuses on a write reply means the completion did not arrive
// within the window — the write is unconfirmed, not failed. Correlation is
// best-effort per (server_handle, item_handle): MXAccess's callback carries
// no transaction id, so concurrent writes to the same item within the
+3 -3
View File
@@ -13,7 +13,7 @@ pub mod mxaccess_gateway {
/// `mxaccess_gateway.v1` package — the v1 wire contract surfaced by the
/// gateway to language clients.
pub mod v1 {
#![allow(clippy::large_enum_variant)]
#![allow(clippy::large_enum_variant, clippy::result_large_err)]
tonic::include_proto!("mxaccess_gateway.v1");
}
@@ -24,7 +24,7 @@ pub mod mxaccess_worker {
/// `mxaccess_worker.v1` package — frame and envelope types used inside
/// the named-pipe transport between gateway and worker.
pub mod v1 {
#![allow(clippy::large_enum_variant)]
#![allow(clippy::large_enum_variant, clippy::result_large_err)]
tonic::include_proto!("mxaccess_worker.v1");
}
@@ -35,7 +35,7 @@ pub mod galaxy_repository {
/// `galaxy_repository.v1` package — types for the Galaxy hierarchy
/// discovery and deploy-event watch RPCs.
pub mod v1 {
#![allow(clippy::large_enum_variant)]
#![allow(clippy::large_enum_variant, clippy::result_large_err)]
tonic::include_proto!("galaxy_repository.v1");
}
+18 -13
View File
@@ -849,15 +849,20 @@ MXAccess item addresses for the `InAlarm`, `Acked`, `AckMsg`, and `Priority`
subtags (confirmed AVEVA `AlarmExtension` field names, verified against the live
ZB Galaxy `attribute_definition` rows).
**The list is fixed for the life of the subscription.** The worker builds its
subtag consumer from the targets `SubscribeAlarms` carried and keeps them until
the subscription ends; a second `SubscribeAlarms` on a subscribed session is
rejected, and the only way to replace the list — `UnsubscribeAlarms` then
`SubscribeAlarms` — rebuilds `FailoverAlarmConsumer`, restarting it on the
primary and discarding the warm standby state. Re-sending on a failover would
therefore flap the feed rather than repair it. A monitor restart (a worker fault,
or a gateway restart) re-resolves and re-sends, and that is the supported way to
pick up Galaxy objects added since the monitor started.
The gateway refreshes this list on the existing alarm reconciliation cadence
and immediately before publishing a failover-to-subtag status. It compares a
canonical fingerprint of all six target fields and sends
`UpdateAlarmWatchListCommand` only for a material change. The worker constructs,
subscribes, and attaches the replacement standby on its STA before swapping it
in; a setup failure leaves the previous standby and the current primary/subtag
mode unchanged. The gateway advances its acknowledged fingerprint only after an
OK reply, so a rejected update remains degraded and is retried.
`SubscribeAlarmsCommand.fallback_enabled` distinguishes an initially empty but
discovery-enabled fallback from an intentionally alarm-manager-only
subscription. In the former case the worker creates an empty standby that can be
hot-swapped after discovery recovers. `ForceAlarmManager` leaves the field false
and does not create a standby.
Because the list is fixed, the resolution is **graded** rather than merely
logged (GWC-37). `ResolveAsync` returns an `AlarmWatchListResolution` carrying
@@ -871,10 +876,10 @@ monitor folds the verdict into the alarm feed's `AlarmProviderStatus`:
`AlarmProviderReasons.SubtagDiscoveryFailed` and the failure message, so a
Galaxy Repository outage at startup is a standing status rather than one log
line emitted possibly weeks before the failover that needs the list.
- On the failover edge (`ApplyProviderModeChangeAsync` with `toMode == Subtag`)
the monitor re-resolves and compares against what it sent. A difference is
reported as `AlarmProviderReasons.SubtagWatchListStale` with both counts, on
the same status frame as the failover, because it cannot be applied.
- If the worker rejects a material update, the monitor retains the previous
acknowledged fingerprint and reports
`AlarmProviderReasons.SubtagWatchListStale` with the rejection diagnostic.
The next reconciliation or failover edge retries the same resolved targets.
`degraded` on the status is therefore the OR of "the subtag provider is the one
serving" and "the subtag watch-list cannot cover anything". In `Auto` the second
+3 -1
View File
@@ -138,7 +138,9 @@ would otherwise let any peer throttle a key it does not hold — and it joins th
partition only when the presented token is validly shaped
(`mxgw_<keyId>_<secret>`, key id at most 64 characters), with at most 32 key-id
partitions per address before the overflow collapses onto that address's fallback
partition. An over-limit state admits one probe per
partition. That peer-layer collapse does not exempt a validly shaped key id from
the cross-peer aggregate: its failures still feed and consult the key-id aggregate,
whose own cardinality is bounded by the configured LRU cap. An over-limit state admits one probe per
`ApiKeyFailureProbeIntervalSeconds` through to the real verifier and refuses
everything else with `ResourceExhausted` before the store read, so a legitimate
holder presenting the correct secret always reaches the constant-time compare and
+25
View File
@@ -40,6 +40,15 @@ the corresponding MXAccess `AddItem`, `Advise`, `UnAdvise`, and `RemoveItem`
calls sequentially on the session STA and preserves input order in the result
list.
The shared alarm command model includes `SubscribeAlarms` and
`UpdateAlarmWatchList`. `SubscribeAlarmsCommand.fallback_enabled` makes an empty
initial watch list explicitly fallback-capable, allowing discovery recovery to
populate a standby without restarting the primary alarm-manager subscription.
`UpdateAlarmWatchListCommand.watch_list` is the complete replacement list. The
worker prepares the replacement on its STA before swapping it into the existing
failover consumer and returns a normal `MxCommandReply`; callers must treat only
`protocol_status = OK` as acknowledgement of the new list.
The command model also includes bulk write/read command kinds:
`WriteBulk`, `Write2Bulk`, `WriteSecuredBulk`, `WriteSecured2Bulk`, and
`ReadBulk`. They are unary `Invoke` payloads on the same `MxAccessGateway`
@@ -94,6 +103,15 @@ only and does not share types with `mxaccess_gateway.proto`. See
[Galaxy Repository Browse](./GalaxyRepository.md) for the RPC catalog and
behavior.
The retained proto remains the client-generation source of truth while the
gateway serves the wire-identical descriptor from the
`ZB.MOM.WW.GalaxyRepository` package. `GalaxyContractParityTests` converts both
loaded descriptors to `FileDescriptorProto`, removes source locations, and
normalizes the C# namespace and file name before comparing deterministic bytes.
Its mutation test also requires drift failures to identify the first differing
protobuf message. Any parity failure must be resolved before publishing either
Contracts or clients.
Generated C# output is written to `src/ZB.MOM.WW.MxGateway.Contracts/Generated/`. Do not
hand-edit generated files.
@@ -140,6 +158,13 @@ Run the focused contract tests after changing either `.proto` file:
dotnet test src/ZB.MOM.WW.MxGateway.Tests/ZB.MOM.WW.MxGateway.Tests.csproj --filter ProtobufContractRoundTripTests
```
When changing `galaxy_repository.proto` or the served Galaxy Repository
package, also run the retained-vs-served parity guard:
```bash
dotnet test src/ZB.MOM.WW.MxGateway.Tests/ZB.MOM.WW.MxGateway.Tests.csproj --filter GalaxyContractParityTests
```
The full solution build also regenerates the C# contracts before compiling
gateway and test projects:
+17
View File
@@ -66,6 +66,23 @@ declaration-ordered JSON; and the row sits alone on its own line for Go and for
Rust's `--jsonl`, but inside an `events` array for Python and for Rust's
aggregate `--json`.
Alarm-feed status rows deliberately preserve each CLI's established JSON
mapping while making false values explicit in every text formatter:
| CLI | Healthy provider text | Clearing snapshot text | JSON false-value shape |
|-----|-----------------------|------------------------|------------------------|
| `mxgw-dotnet` | `provider-status mode=Unspecified degraded=false reason=""` | `snapshot-status truncated=false` | `{"providerStatus": {}}` / `{"snapshotStatus": {}}` |
| `mxgw-go` | `provider-status mode=ALARM_PROVIDER_MODE_UNSPECIFIED degraded=false reason=""` | `snapshot-status truncated=false` | `{"providerStatus":{}}` / `{"snapshotStatus":{}}` |
| `mxgw-py` | same JSON row as `--json` | same JSON row as `--json` | `{"providerStatus": {}}` / `{"snapshotStatus": {}}`, emitted one row at a time |
| `mxgw-java` | `provider-status mode=ALARM_PROVIDER_MODE_UNSPECIFIED degraded=false reason=` | `snapshot-status truncated=false` | `{"providerStatus":{}}` / `{"snapshotStatus":{}}` |
| `mxgw-rs` | explicit `degraded=false` summary | explicit `truncated=false` summary | always includes `"degraded":false` / `"truncated":false` |
For the protobuf-JSON rows, an absent proto3 boolean means `false`; Rust's
always-explicit booleans are semantically equivalent. Python `stream-alarms`
defaults to an unbounded feed and emits each row immediately; a positive
`--max-messages` value provides a bounded sample and includes both status
preamble rows in the count.
## Integration Gate
Cross-language smoke execution is opt-in. Runners should skip the matrix unless
+3 -3
View File
@@ -90,9 +90,9 @@ detach-grace and replay-retention windows are on by default, the reconnect
surface is a trust boundary — a retained session outlives the stream that opened
it. The session records the opening key id (`GatewaySession.OwnerKeyId`) and
every `StreamEvents` attach/reattach is rejected with `PermissionDenied` unless
the caller's key id matches the owner. Gating on the `event` scope alone was
rejected: it would let any `event`-scoped key that learned a session id attach to
another key's retained session and receive its replayed and live data. The check
the caller's key id matches the owner. Gating on the `events:read` scope alone was
rejected: it would let any `events:read`-scoped key that learned a session id
attach to another key's retained session and receive its replayed and live data. The check
runs before any subscriber is attached, so a foreign key never touches the
replay ring. Admin-scope override is deferred.
+1 -1
View File
@@ -439,7 +439,7 @@ behavior.
| `MxGateway:Alarms:Enabled` | `false` | Gates the gateway's always-on central alarm monitor. When `true`, the gateway opens one gateway-owned worker session dedicated to alarms, caches the active-alarm set, and fans it out to every client through the `StreamAlarms` RPC — no client opens its own session to see alarms. |
| `MxGateway:Alarms:SubscriptionExpression` | _(empty)_ | AVEVA alarm-subscription expression the monitor subscribes on startup, in canonical `\\<machine>\Galaxy!<area>` form. The literal `Galaxy` provider is correct regardless of the Galaxy database name. When empty and `Enabled` is `true`, the gateway falls back to `\\<MachineName>\Galaxy!<DefaultArea>` if `DefaultArea` is set. |
| `MxGateway:Alarms:DefaultArea` | _(empty)_ | Area name used to compose a default subscription when `SubscriptionExpression` is empty. If both are empty while `Enabled` is `true`, the monitor faults with a configuration diagnostic. |
| `MxGateway:Alarms:ReconcileIntervalSeconds` | `30` | How often the monitor reconciles its in-process alarm cache against the worker's authoritative active-alarm snapshot, catching transitions the live poll-and-diff feed missed. Floored at 5 seconds. |
| `MxGateway:Alarms:ReconcileIntervalSeconds` | `30` | How often the monitor re-resolves and, on material change, hot-swaps the subtag fallback watch-list, then reconciles its in-process alarm cache against the worker's authoritative active-alarm snapshot. The acknowledged watch-list fingerprint advances only after an OK worker reply. Floored at 5 seconds. |
| `MxGateway:Alarms:PollIntervalMilliseconds` | `500` | Cadence at which the worker's STA polls the AVEVA alarm consumer (`GetXmlCurrentAlarms2`) for the active-alarm snapshot the live feed diffs. Must be between `100` and `3600000` (one hour): every poll is a COM call plus an XML parse on the same STA that serves reads and writes, so a tighter cadence starves the command path, while a value above an hour stops being a cadence and silently disables alarm polling. The gateway conveys the value to the worker via the `MXGATEWAY_ALARM_POLL_INTERVAL_MS` environment variable; a missing or out-of-range value leaves the worker on the 500 ms default rather than failing the session. |
| `MxGateway:Alarms:MaxAlarmsPerFetch` | `1024` | Cap the worker passes to `GetXmlCurrentAlarms2`'s `maxAlmCnt`. Must be between `64` and `65536` — the worker is a 32-bit process that materializes each reply as one BSTR plus a full `XmlDocument`, so an unbounded cap faults the STA with an out-of-memory rather than merely slowing it. It doubles as the **truncation threshold**: a fetch returning exactly this many records is treated as truncated, because the COM API caps its reply with no "more available" flag. On a truncated poll the worker retains the alarms the capped reply could not mention instead of letting their absence read as a clear, and logs a rate-limited `AlarmSnapshotTruncated` warning to its stderr (identifiers and counts only). **Remediation when you see that warning: raise this value** so the steady-state active-alarm count fits inside one fetch. A galaxy permanently above the cap holds stale entries in the snapshot until a sub-cap poll, and loses clear→re-raise cycles that happen entirely out of window (see `docs/DesignDecisions.md`). Conveyed to the worker via the `MXGATEWAY_ALARM_MAX_ALARMS_PER_FETCH` environment variable; a missing or out-of-range value leaves the worker on the 1024 default. |
+7 -5
View File
@@ -650,11 +650,13 @@ runtime because the x86 Worker cannot build on Linux:
the tier is down, not the change**: an unreachable windev fails the job (never skips), so
if it goes red without a related code change, check the tier — `ssh windev`, read
`C:\build\mxaccessgw-ci\ci.log` — before assuming a real test failure.
- **`nightly-windev`** (Linux runner, scheduled `0 6 * * *`) — re-runs the full x86 build +
Worker.Tests on windev, then the live-MXAccess smoke (`MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1`)
and a full `slnx` build (which surfaces Windows-only analyzer diagnostics such as
`xUnit1030`). Gated `if: github.event_name == 'schedule'`, so it never gates a push. On
failure it opens a Gitea issue via the Actions token, since nobody watches the Actions page.
- **`nightly-windev`** (Linux runner, scheduled `0 6 * * *`) — runs, in order, the x86 Worker
build, `Worker.Tests`, a full `slnx` build (which surfaces Windows-only analyzer diagnostics
such as `xUnit1030`), the net10 gateway suite, and the live-MXAccess smoke
(`MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1`). The checkout and all five tiers share the windev
host lock, and the first nonzero native exit stops the sequence and propagates to the CI job.
Gated `if: github.event_name == 'schedule'`, so it never gates a push. On failure it opens a
Gitea issue via the Actions token, since nobody watches the Actions page.
### Runner capacity is shared and finite
+7 -1
View File
@@ -84,6 +84,12 @@ Carrying the enqueue timestamp into the worker layer is what lets queue-wait tim
An accepted gRPC command payload can still be too large for the worker pipe: the envelope built around it must fit `MxGateway:Worker:MaxMessageBytes`, which is validated at startup to sit at least a fixed envelope-overhead reserve above `MaxGrpcMessageBytes` (see the headroom rule in [Gateway Configuration](./GatewayConfiguration.md)) so this should not occur for a conformant payload, but if it does, `WorkerClient` raises `WorkerClientException(CommandTooLarge)` and `Invoke` reports `ResourceExhausted` for that command — the session is not faulted, so a client can retry with a smaller payload without reopening the session.
Selected control commands have additional public behavior beyond their protobuf shape:
| Command | Public behavior |
|---|---|
| `DrainEvents` | Returns a destructive diagnostic snapshot of the worker's outbound event queue. `max_events = 0` selects the default cap; positive values may not exceed `GatewayContractInfo.MaxDrainEventsPerCommand` (10,000). The worker also applies a serialized-byte budget derived from the negotiated frame limit, so a reply may contain fewer events than the requested count. When either cap leaves queued events, `MxCommandReply.diagnostic_message` reports the truncation and callers repeat `DrainEvents` until an empty reply. Events included in a drain are removed from the worker queue and therefore do not later appear on the normal session `StreamEvents` path; the background stream drain may concurrently consume other events first. |
### `StreamEvents`
`StreamEvents` is a server-streaming RPC. The handler delegates the full pipeline to `IEventStreamService` and just forwards each `MxEvent` onto the response stream. Keeping the channel and producer/consumer machinery out of the handler means cancellation, exception mapping, and metric bookkeeping live in one place.
@@ -100,7 +106,7 @@ An `OK` response means the alarm provider accepted the acknowledgement, not that
### `QueryActiveAlarms`
`QueryActiveAlarms` is a server-streaming, **session-less** RPC that returns a point-in-time snapshot of the currently-active alarm set. Unlike `StreamAlarms`, it does not stay attached for live transitions — it writes the snapshot and completes, which clients use after a reconnect to reseed alarm state or to reconcile alarms that may have been missed during a transport blip. It is served from the always-on `GatewayAlarmMonitor` cache (`IGatewayAlarmService.CurrentAlarms`), so no worker session is opened. The handler does not run through `MxAccessGrpcRequestValidator`; it only null-checks the request inline, then streams one `ActiveAlarmSnapshot` per cached alarm, honouring cancellation between writes. When `alarm_filter_prefix` is non-empty, only alarms whose `alarm_full_reference` starts with that prefix (ordinal `StartsWith`) are emitted; an empty prefix returns the full set. Exceptions are translated by the same `MapException` path as the other handlers. The RPC requires the `event` scope (`GatewayScopes.EventsRead`), the same scope as `StreamEvents` and `StreamAlarms`.
`QueryActiveAlarms` is a server-streaming, **session-less** RPC that returns a point-in-time snapshot of the currently-active alarm set. Unlike `StreamAlarms`, it does not stay attached for live transitions — it writes the snapshot and completes, which clients use after a reconnect to reseed alarm state or to reconcile alarms that may have been missed during a transport blip. It is served from the always-on `GatewayAlarmMonitor` cache (`IGatewayAlarmService.CurrentAlarms`), so no worker session is opened. The handler does not run through `MxAccessGrpcRequestValidator`; it only null-checks the request inline, then streams one `ActiveAlarmSnapshot` per cached alarm, honouring cancellation between writes. When `alarm_filter_prefix` is non-empty, only alarms whose `alarm_full_reference` starts with that prefix (ordinal `StartsWith`) are emitted; an empty prefix returns the full set. Exceptions are translated by the same `MapException` path as the other handlers. The RPC requires the `events:read` scope (`GatewayScopes.EventsRead`), the same scope as `StreamEvents` and `StreamAlarms`.
#### Provider status on the alarm feed
+25 -3
View File
@@ -90,13 +90,22 @@ Expected command-line arguments:
--protocol-version <version>
```
Expected protected environment values:
Expected launch environment values (the shared names are defined by
`WorkerLaunchEnvironment` in Contracts):
```text
MXGATEWAY_WORKER_NONCE=<random nonce>
MXGATEWAY_WORKER_LOG_CONTEXT=<optional context>
MXGATEWAY_WORKER_PIPE_CONNECT_ATTEMPT_TIMEOUT_MS=<milliseconds>
MXGATEWAY_WORKER_WRITE_COMPLETION_WAIT_MS=<milliseconds>
MXGATEWAY_EVENT_QUEUE_CAPACITY=<events>
MXGATEWAY_ALARM_POLL_INTERVAL_MS=<milliseconds>
MXGATEWAY_ALARM_MAX_ALARMS_PER_FETCH=<alarms>
```
The nonce is protected bootstrap material. The remaining values are derived
from the public `MxGateway:Worker:*` and `MxGateway:Alarms:*` configuration
keys documented in [Worker Process Launcher](./WorkerProcessLauncher.md).
Startup sequence:
1. Parse command-line arguments.
@@ -312,7 +321,8 @@ write wait still yields the empty-`statuses` unconfirmed reply, and an expired
per-tag `ReadBulk` wait still reports its own timeout.
The write wait's budget is `MxGateway:Worker:WriteCompletionWaitMilliseconds`
(default 1500). It is a bounded hold on the STA per unary write, so deployments
(default 1500). It applies to `Write`, `Write2`, `WriteSecured`, and
`WriteSecured2` and is a bounded hold on the STA per single-item write, so deployments
whose write workload is effectively fire-and-forget — no consumer reads the
reply's `statuses` — can lower it, or set `0` to skip the wait entirely and
reply on acceptance alone.
@@ -341,6 +351,18 @@ native client correlating by handle has the identical ambiguity). The exposure
is bounded: `statuses` is an enrichment field, and `Hresult`/`ProtocolStatus`
stay acceptance-only regardless of which completion (if any) was attributed.
## Live alarm watch-list replacement
An auto-mode `SubscribeAlarms` with `fallback_enabled = true` always builds a
`FailoverAlarmConsumer`, even when initial discovery supplied no targets. The
handler retains the subscription expression so `UpdateAlarmWatchList` can build
and subscribe a new subtag consumer entirely on the session STA. The failover
consumer attaches the replacement's transition callback before swapping the
standby reference, then detaches and disposes the old standby. If construction,
subscription, or attachment fails, it disposes the replacement and preserves
the old standby. The active-provider selector is never reset, so primary mode
stays primary and subtag mode immediately delegates to the new standby.
## COM Creation
The MXAccess analysis source at `C:\Users\dohertj2\Desktop\mxaccess` identifies
+1 -1
View File
@@ -247,7 +247,7 @@ A session that faults (for example, a slow single-subscriber client that overflo
A client that drops mid-stream reconnects by re-issuing `StreamEvents` with `StreamEventsRequest.after_worker_sequence` set to the last `worker_sequence` it observed. A non-zero `after_worker_sequence` means *resume*; `0` means *fresh stream* and behaves exactly as a first-time subscribe — no replay, no sentinel.
**Owner-scoped attach (security control).** `OpenSession` records the caller's API key id on the session as `GatewaySession.OwnerKeyId`. Every `StreamEvents` attach and reattach is owner-checked: `EventStreamService.StreamEventsAsync` compares the caller's key id (threaded from `GatewayGrpcAuthorizationInterceptor` via `MxAccessGatewayService.StreamEvents`) to the session owner and throws `SessionManagerException(PermissionDenied)` — surfaced as gRPC `PermissionDenied` — when they differ, before any subscriber is attached and before any replayed or live event is delivered. Possessing the `event` scope and knowing a session id is not enough. This is what makes the default-on detach-grace and replay-retention windows safe: without the check, any `event`-scoped key that learned a session id could attach to another key's retained session and receive its replayed and live data. A `null` owner (session opened with authentication disabled) matches only a `null` caller key.
**Owner-scoped attach (security control).** `OpenSession` records the caller's API key id on the session as `GatewaySession.OwnerKeyId`. Every `StreamEvents` attach and reattach is owner-checked: `EventStreamService.StreamEventsAsync` compares the caller's key id (threaded from `GatewayGrpcAuthorizationInterceptor` via `MxAccessGatewayService.StreamEvents`) to the session owner and throws `SessionManagerException(PermissionDenied)` — surfaced as gRPC `PermissionDenied` — when they differ, before any subscriber is attached and before any replayed or live event is delivered. Possessing the `events:read` scope and knowing a session id is not enough. This is what makes the default-on detach-grace and replay-retention windows safe: without the check, any `events:read`-scoped key that learned a session id could attach to another key's retained session and receive its replayed and live data. A `null` owner (session opened with authentication disabled) matches only a `null` caller key.
On a resume, `EventStreamService.StreamEventsAsync` attaches through `GatewaySession.AttachEventSubscriberWithReplay`, which calls `SessionEventDistributor.RegisterWithReplay`. That method snapshots the session's replay ring for events newer than `after_worker_sequence` **and** registers the live subscriber inside a single `_replayLock` critical section. This atomicity is what makes the replay→live handoff free of gaps and duplicates: the pump appends each event to the replay ring (under `_replayLock`) before fanning it to subscriber channels, so relative to that one critical section every event is either in the replay snapshot or fanned into the freshly-registered live channel — never both observably, never neither.
+19 -5
View File
@@ -26,6 +26,23 @@ The launcher sets the nonce through the `MXGATEWAY_WORKER_NONCE` environment
variable. The nonce is not included in `WorkerProcessCommandLine` so logs and
diagnostics can report the launch command without exposing the secret.
All launch-variable names live in the shared Contracts
`WorkerLaunchEnvironment` class and are consumed by both the launcher and the
worker. The launcher always exports all six values:
| Environment variable | Gateway source | Worker default when missing/invalid |
|---|---|---|
| `MXGATEWAY_WORKER_NONCE` | `WorkerProcessLaunchRequest.Nonce` | None; bootstrap fails because the nonce is required |
| `MXGATEWAY_WORKER_PIPE_CONNECT_ATTEMPT_TIMEOUT_MS` | `MxGateway:Worker:PipeConnectAttemptTimeoutMilliseconds` | `2000` ms |
| `MXGATEWAY_WORKER_WRITE_COMPLETION_WAIT_MS` | `MxGateway:Worker:WriteCompletionWaitMilliseconds` | `1500` ms |
| `MXGATEWAY_EVENT_QUEUE_CAPACITY` | `MxGateway:Worker:EventQueueCapacity` | `10000` events |
| `MXGATEWAY_ALARM_POLL_INTERVAL_MS` | `MxGateway:Alarms:PollIntervalMilliseconds` | `500` ms |
| `MXGATEWAY_ALARM_MAX_ALARMS_PER_FETCH` | `MxGateway:Alarms:MaxAlarmsPerFetch` | `1024` alarms |
The write-completion wait applies to the four single-item write variants:
`Write`, `Write2`, `WriteSecured`, and `WriteSecured2`. Bulk writes report COM
acceptance per entry and do not wait for `OnWriteComplete` rows.
## Validation And Cleanup
Before starting the process, the launcher validates that the configured worker
@@ -44,11 +61,8 @@ process remains alive. The policy is configured by
`WorkerOptions.StartupProbeRetryDelayMilliseconds`; the retry counter is
recorded as `mxgateway.retries.attempted` with `area=worker_startup`.
The launcher also passes
`MXGATEWAY_WORKER_PIPE_CONNECT_ATTEMPT_TIMEOUT_MS` to the worker process from
`WorkerOptions.PipeConnectAttemptTimeoutMilliseconds`. The worker uses that
value as the per-attempt named-pipe connect timeout inside its own bounded
Polly retry loop.
The worker uses the exported pipe-attempt value inside its own bounded Polly
retry loop.
If startup fails or exceeds `WorkerOptions.StartupTimeoutSeconds`, the launcher
kills the worker process tree, disposes the process handle, disposes the
@@ -0,0 +1,119 @@
# Cycle 3 P2 And Follow-Up Design
**Date:** 2026-08-25
**Source:** `archreview/2026-08-19/remediation/00-tracking.md`
**Scope:** The 17 findings recorded as deferred P2 plus the three follow-ups recorded by the cycle-3 integration gate. `SEC-37` and `TST-32` remain external actions and are not part of this repository change.
## Current-State Reconciliation
The 2026-08-25 audit refreshed `origin/main` and checked every finding against the current tree. Sixteen P2 findings remain open. `WRK-34` is already complete in commit `3679e59` and needs a tracker correction rather than another code change. All three integration-gate follow-ups remain open:
- a live worker command and gateway path for refreshing an alarm subscription's subtag watch-list;
- the full WRK-32 recency limitation in the public reply-field comment;
- publication of the already-bumped client release train.
## Design Principles
1. Preserve the existing worker STA and COM-affinity rules. Worker alarm-consumer construction, subscription changes, and disposal remain on the STA.
2. Preserve the primary alarm-manager subscription during watch-list repair. A refresh must never unsubscribe and recreate the whole alarm feed.
3. Make behavioral changes test-first and keep documentation in the same commit as the behavior it describes.
4. Perform one protobuf regeneration wave for the watch-list command, the WRK-32 reply-field clarification, and the IPC-36 operator-facing comment correction.
5. Publish clients only after every client-affecting change is integrated, all five client suites pass, the release commit is on `main`, and registry collision checks pass.
6. Keep the canonical archreview tracker current at every batch boundary, including commit IDs and verification counts.
## Workstreams
### 1. Tracker And Low-Risk Currency
Correct WRK-34 to Done, repair the stale replay-watermark documentation (GWC-36), document the public `DrainEvents` command (IPC-38), reconcile `stillpending.md` (TST-33), wire the code-review README check into portable CI (TST-34), and replace the nonexistent `event` scope spelling with `events:read` (TST-36).
These edits form the first batch because they have a small conflict surface and establish an accurate tracker before code changes begin.
### 2. Gateway Correctness And Security
- **GWC-33:** stop faulted or closed worker clients from contributing frozen queue depth while retaining fault records for the configured grace period.
- **GWC-35:** make snapshot reconciliation emit a Raise when a cached acknowledged alarm is replaced by an active unacknowledged instance.
- **SEC-38:** consult and feed the cross-peer aggregate for every validly-shaped key ID, even when the per-peer key-partition cap routes the request through the fallback bucket. Aggregate cardinality remains bounded by the aggregate cache's existing eviction policy.
- **SEC-39:** never let a null effective key ID clear the shared fallback partition.
Each change gets a focused regression test that first reproduces the filed failure mode.
### 3. Secondary Contracts
- **IPC-36:** move all six worker-launch environment-variable names into one net48-safe Contracts class, consume those constants on both sides, enumerate the launch contract in the launcher and worker design documentation, and test that the launcher sets every declared variable.
- **IPC-37:** compare the retained gateway Galaxy descriptor with the descriptor served by the shared `ZB.MOM.WW.GalaxyRepository` package after normalizing source-only metadata.
This work makes the two non-protobuf peer contracts executable invariants rather than comments.
### 4. Client Alarm-Feed Parity
- **CLI-47:** explicitly render false-valued provider and snapshot status fields in .NET text output; document the deliberate protobuf-JSON omission behavior and Rust's explicit JSON shape.
- **CLI-48:** add provider-status and clearing snapshot-status coverage in .NET, Go, and Java, including Go's missing `stream-alarms` test scaffold.
- **CLI-49:** expose the two Go status payload aliases and update the alarm-feed alias documentation.
- **CLI-50:** change Python `stream-alarms` from a one-message buffered sample to incremental streaming with an unbounded-by-default limit, while retaining timeout and explicit finite-limit behavior.
The Python change uses the same arrival-order streaming model as the other four CLIs. Documentation examples will request a finite bound when a bounded demonstration is desirable.
### 5. Live Alarm Watch-List Refresh
The complete periodic hot-swap design is selected over failover-edge-only refresh and unsubscribe/resubscribe.
#### Contract
Add an explicit worker command for updating an active alarm subscription's watch-list. Add an explicit subscription flag that distinguishes these cases when the initial list is empty:
- alarm-manager-only deployment with no fallback requested;
- auto fallback configured, but discovery currently produced no targets.
The command is gateway-owned even though it travels through the shared `MxCommand` envelope. It returns a normal `MxCommandReply`; invalid mode/state or replacement-subscription failure returns a structured failure without destroying the existing subscription.
#### Worker
In auto mode, retain a failover composite even when fallback is enabled with an initially empty list. The composite retains its subscription expression and supports replacing its standby consumer on the STA:
1. construct the replacement standby;
2. subscribe and warm it before altering the current child;
3. attach transition forwarding;
4. atomically replace the standby reference;
5. dispose the previous standby;
6. if standby is currently active, keep the mode on subtag and expose the replacement snapshot immediately.
If any pre-swap step fails, dispose the replacement and keep the old standby untouched. Forced alarm-manager mode rejects refresh as inapplicable. Forced subtag mode may replace its sole consumer through the same build-first/swap-second rule.
#### Gateway
Reuse the existing alarm reconciliation cadence; do not add another timer or configuration option. Resolve only when fallback can use subtags. Compare a canonical target fingerprint with the list last acknowledged by the worker, and send an update only after material change or recovery from an empty/failed resolution.
The failover edge still performs an immediate refresh attempt before publishing its coverage verdict. A successful update changes the recorded sent fingerprint, clears stale-coverage status, and triggers reconciliation. A failed update retains the previous fingerprint and publishes/logs the degraded reason.
This design repairs the original startup-outage case before failover whenever Galaxy Repository recovers during normal monitor operation, while preserving the warm primary and current provider mode.
### 6. Windows Gateway Coverage
**TST-35** extends windev `live` mode to run the net10 gateway suite after the serialized Worker tier and full solution build. The existing host lock remains the load-sensitivity control. Logs record the observed count rather than hard-coding the historical 879 or 1183 count as an invariant.
### 7. Protobuf Regeneration And Release
The watch-list command lands in the same regeneration wave as:
- WRK-32's explicit sequential stale-callback/recency limitation;
- IPC-36's replacement of the internal environment-variable name with the public configuration key.
Regenerate Contracts and all five client trees according to `docs/Contracts.md` and `docs/ClientPackaging.md`, then run the codegen cleanliness checks and every affected suite.
After integration to `main`, publish .NET/Contracts, Python, Rust, and Go as `0.2.1`, and Java as `0.2.2`. Query each exact coordinate before publishing, never overwrite, create and push the Go module tag at the exact release commit, then query every registry coordinate again and update the packaging record and tracker with immutable release evidence.
## Verification Strategy
- macOS: clean `NonWindows.slnx` build and full gateway suite.
- Focused gateway tests after each behavioral batch.
- All five client build/test commands from `docs/ClientPackaging.md`.
- `scripts/check-codegen.ps1`, client-version guard, and code-review README check.
- windev: x86 Worker build plus Worker.Tests for worker/proto changes; `live` mode for TST-35 and the final integration gate.
- Release: exact registry-coordinate checks and Go module resolution from the pushed tag.
## Completion Criteria
The continuation is complete only when all 16 open P2 findings and all three follow-ups are Done or explicitly blocked with evidence, WRK-34 is corrected in the tracker, every required verification is green, client releases are confirmed at their registries, and `archreview/2026-08-19/remediation/00-tracking.md` contains the commit and test evidence for every batch.
@@ -0,0 +1,405 @@
# Cycle 3 P2 And Follow-Up Implementation Plan
> **For Codex:** REQUIRED SUB-SKILL: Use `executeplan` to implement this plan task-by-task.
**Goal:** Close the 16 still-open cycle-3 P2 findings, correct WRK-34's tracker status, implement the three integration-gate follow-ups, and publish the completed client release train.
**Architecture:** Work proceeds in seven reviewable batches. Small tracker and documentation corrections land first; focused gateway, security, and secondary-contract fixes follow; client parity is completed before a single protobuf regeneration wave adds live alarm watch-list refresh and the remaining public comments; Windows CI is extended before the final windev gate; packages are published only from the integrated `main` release commit.
**Tech Stack:** .NET 10/C#, .NET Framework 4.8 x86, protobuf/Grpc.Tools, PowerShell 5.1/7, Go, Python/pytest, Rust/Cargo, Java/Gradle, Gitea Actions and package registries.
---
## Preconditions
- Work only in `.worktrees/c3-p2-followups` on `codex/c3-p2-followups`.
- Preserve the clean baseline recorded on 2026-08-20/25: `NonWindows.slnx` builds with zero warnings and the gateway suite passes 1183/1183.
- Follow red-green-refactor for every behavior change. Documentation-only changes use an explicit failing grep/check as their red gate.
- After every batch, update `archreview/2026-08-19/remediation/00-tracking.md` with status, commits, and exact verification output before committing.
- Worker/protobuf changes require windev x86 verification before Done.
### Task 1: Batch H — Tracker And Documentation Currency
**Findings:** WRK-34 tracker correction, GWC-36, IPC-38, TST-33, TST-34, TST-36
**Files:**
- Modify: `archreview/2026-08-19/remediation/00-tracking.md`
- Modify: `src/ZB.MOM.WW.MxGateway.Server/Sessions/EventSubscriberReplayAttachment.cs`
- Modify: `docs/Grpc.md`
- Modify: `stillpending.md`
- Modify: `.gitea/workflows/ci.yml`
- Modify: `REVIEW-PROCESS.md`
- Modify: `CLAUDE.md`
- Modify: `docs/Sessions.md`
- Modify: `docs/DesignDecisions.md`
**Step 1: Capture the failing currency checks**
Run:
```bash
git grep -n '`event` scope' -- CLAUDE.md docs/
rg -n 'not safe to rely|Phase 5, not started|6 \[LiveMxAccessFact\]|wire behavior is unverified|still scheduled.*Phase 4' stillpending.md
rg -n 'regen-readme.py|check-code-reviews-readme' .gitea/workflows/ci.yml
rg -n 'DrainEvents' docs/Grpc.md
```
Expected: stale scope and pending claims are found; CI and `DrainEvents` checks find no intended entry.
**Step 2: Correct GWC-36 and TST-36**
Change `OldestAvailableSequence` to document retained-oldest or `highest observed + 1` when an empty ring has a gap, with `0` only when `Gap` is false. Replace prose references to the nonexistent ``event`` scope with ``events:read`` without renaming identifiers such as `GatewayScopes.EventsRead`.
**Step 3: Correct IPC-38 and TST-33**
Add a `DrainEvents` command row to `docs/Grpc.md` covering the 10,000-event count cap, byte cap, truncation diagnostic, drain-until-empty contract, and removal from the normal session event stream. Reconcile the five stale `stillpending.md` claims and stamp the file as a historical inventory whose live status comes from archreview trackers.
**Step 4: Wire TST-34**
Add this portable-job step next to the existing repository consistency checks:
```yaml
- name: Code-reviews README consistency
run: python code-reviews/regen-readme.py --check
```
Qualify the interpreter note in `REVIEW-PROCESS.md`: Windows may require `python`; macOS/Linux normally use `python3`, while CI uses its configured `python`.
**Step 5: Re-run the checks**
Run:
```bash
python3 code-reviews/regen-readme.py --check
git grep -n '`event` scope' -- CLAUDE.md docs/ || true
rg -n 'not safe to rely|Phase 5, not started|6 \[LiveMxAccessFact\]|wire behavior is unverified|still scheduled.*Phase 4' stillpending.md || true
rg -n 'DrainEvents' docs/Grpc.md
dotnet build src/ZB.MOM.WW.MxGateway.NonWindows.slnx
```
Expected: README check passes; stale greps return nothing; DrainEvents is documented; build has zero warnings/errors.
**Step 6: Update tracker and commit**
Mark the five P2 findings Done and WRK-34 Done (`3679e59`), record verification, then commit:
```bash
git add .gitea/workflows/ci.yml REVIEW-PROCESS.md CLAUDE.md stillpending.md docs/Grpc.md docs/Sessions.md docs/DesignDecisions.md src/ZB.MOM.WW.MxGateway.Server/Sessions/EventSubscriberReplayAttachment.cs archreview/2026-08-19/remediation/00-tracking.md
git commit -m "chore(GWC-36,IPC-38,TST-33,TST-34,TST-36): close cycle-3 currency gaps"
```
### Task 2: Batch I — Gateway Correctness And Security
**Findings:** GWC-33, GWC-35, SEC-38, SEC-39
**Files:**
- Modify: `src/ZB.MOM.WW.MxGateway.Server/Workers/WorkerClient.cs`
- Test: `src/ZB.MOM.WW.MxGateway.Tests/Gateway/Workers/WorkerClientTests.cs`
- Modify: `src/ZB.MOM.WW.MxGateway.Server/Alarms/GatewayAlarmMonitor.cs`
- Test: `src/ZB.MOM.WW.MxGateway.Tests/Alarms/GatewayAlarmMonitorAttachOrderTests.cs`
- Modify: `src/ZB.MOM.WW.MxGateway.Server/Security/Authorization/ApiKeyFailureLimiter.cs`
- Test: `src/ZB.MOM.WW.MxGateway.Tests/Security/Authorization/ApiKeyFailureLimiterTests.cs`
- Modify: `docs/Authentication.md`
- Modify: `archreview/2026-08-19/remediation/00-tracking.md`
**Step 1: RED for GWC-33**
Add a worker-client test that stages an event, faults the client without disposing it, and asserts the aggregate queue-depth observable stops counting that client while the faulted object remains alive. Run:
```bash
dotnet test src/ZB.MOM.WW.MxGateway.Tests/ZB.MOM.WW.MxGateway.Tests.csproj --filter FullyQualifiedName~WorkerClientTests
```
Expected: the new assertion fails because the registration is disposed only by `DisposeAsync`.
**Step 2: GREEN for GWC-33**
Make queue-depth registration disposal idempotent and call it on both terminal transitions and disposal. Do not clear the faulted client's diagnostic state or shorten `FaultedGraceSeconds` retention. Re-run the WorkerClient filter and commit the focused change.
**Step 3: RED/GREEN for GWC-35**
Add a monitor test whose first reconcile contains `ActiveAcked` and whose next reconcile carries a newer `Active` instance. Assert one `Raise` repair carrying the incoming instance identity. Verify red, then generalize the both-present state delta:
```csharp
AlarmTransitionKind kind = incoming.Value.CurrentState == AlarmConditionState.ActiveAcked
? AlarmTransitionKind.Acknowledge
: AlarmTransitionKind.Raise;
```
Keep the existing duplicate suppression keyed by original raise timestamp. Re-run `GatewayAlarmMonitorAttachOrderTests`.
**Step 4: RED/GREEN for SEC-38**
Add a limiter test that fills all 32 peer key-ID partitions, then attacks one valid victim key ID through the fallback from multiple peers. Assert the victim aggregate trips. Change resolution/recording so every validly-shaped presented key ID consults and feeds its aggregate even if its peer partition fell back; continue using the resolved partition key for the peer layer and the existing capped aggregate eviction for cardinality. Update class remarks and `docs/Authentication.md`.
**Step 5: RED/GREEN for SEC-39**
Add a test that records fallback failures, calls `Reset` with a null effective key ID, and asserts the fallback remains tracked. Tighten ownership:
```csharp
if (effectiveKeyId is not null
&& string.Equals(effectiveKeyId, partition.KeyId, StringComparison.Ordinal))
{
RemovePartition(partitionKey);
}
```
**Step 6: Verify, track, and commit**
Run:
```bash
dotnet test src/ZB.MOM.WW.MxGateway.Tests/ZB.MOM.WW.MxGateway.Tests.csproj --filter "FullyQualifiedName~WorkerClientTests|FullyQualifiedName~GatewayAlarmMonitorAttachOrderTests|FullyQualifiedName~ApiKeyFailureLimiterTests"
dotnet build src/ZB.MOM.WW.MxGateway.NonWindows.slnx
```
Expected: all filtered tests pass; zero warnings/errors. Update tracker with per-finding commits and counts.
### Task 3: Batch J — Secondary Contract Guards
**Findings:** IPC-36, IPC-37
**Files:**
- Create: `src/ZB.MOM.WW.MxGateway.Contracts/WorkerLaunchEnvironment.cs`
- Modify: `src/ZB.MOM.WW.MxGateway.Server/Workers/WorkerProcessLauncher.cs`
- Modify: worker consumers under `src/ZB.MOM.WW.MxGateway.Worker/Bootstrap`, `Ipc`, and `MxAccess`
- Test: `src/ZB.MOM.WW.MxGateway.Tests/Gateway/Workers/WorkerProcessLauncherTests.cs`
- Create: `src/ZB.MOM.WW.MxGateway.Tests/Contracts/GalaxyContractParityTests.cs`
- Modify: `src/ZB.MOM.WW.MxGateway.Contracts/ZB.MOM.WW.MxGateway.Contracts.csproj`
- Modify: `docs/WorkerProcessLauncher.md`
- Modify: `docs/MxAccessWorkerInstanceDesign.md`
- Modify: `docs/Contracts.md`
- Modify: `archreview/2026-08-19/remediation/00-tracking.md`
**Step 1: RED for IPC-36**
Change the launcher test to enumerate the six constants expected on `WorkerLaunchEnvironment` and assert each is present in `ProcessStartInfo.Environment`. Run the test and confirm compilation fails because the shared class does not exist.
**Step 2: GREEN for IPC-36**
Create a net48-safe static class containing six `public const string` names. Replace both gateway and worker literals with these constants; remove the duplicate public constants from `WorkerProcessLauncher` unless compatibility requires obsolete aliases. Add a launch-environment table with variable, source configuration key, and worker default. Refresh Write/Write2 wording.
**Step 3: RED for IPC-37**
Add `GalaxyContractParityTests` that converts both loaded descriptors with `ToProto()`, clears `SourceCodeInfo`, normalizes `Options.CsharpNamespace` and file `Name`, then compares deterministic bytes. Before the assertion, mutate a clone in a local test helper to prove the comparison reports the differing symbol/message clearly; the real descriptor assertion must pass at HEAD.
**Step 4: Document and verify**
Record the parity guard in the Contracts project comment and `docs/Contracts.md`. Run:
```bash
dotnet test src/ZB.MOM.WW.MxGateway.Tests/ZB.MOM.WW.MxGateway.Tests.csproj --filter "FullyQualifiedName~WorkerProcessLauncherTests|FullyQualifiedName~GalaxyContractParityTests"
dotnet build src/ZB.MOM.WW.MxGateway.NonWindows.slnx
```
Then update tracker and commit IPC-36/37.
### Task 4: Batch K — Client Alarm-Feed Parity
**Findings:** CLI-47, CLI-48, CLI-49, CLI-50
**Files:**
- Modify: `clients/dotnet/ZB.MOM.WW.MxGateway.Client.Cli/MxGatewayClientCli.cs`
- Test: `clients/dotnet/ZB.MOM.WW.MxGateway.Client.Tests/MxGatewayClientCliTests.cs`
- Test/modify: `clients/go/cmd/mxgw-go/main.go`, `main_test.go`
- Modify: `clients/go/mxgateway/types.go`
- Test/modify: Java CLI and `MxGatewayCliTests.java`
- Modify/test: `clients/python/src/zb_mom_ww_mxgateway_cli/commands.py`, `clients/python/tests/`
- Modify: all affected client READMEs and `docs/CrossLanguageSmokeMatrix.md`
- Modify: `archreview/2026-08-19/remediation/00-tracking.md`
**Step 1: RED/GREEN for CLI-47 and CLI-48**
In .NET, Go, and Java tests, feed `provider_status { degraded: false }` and `snapshot_status { truncated: false }`; assert exact text rows and protobuf-JSON behavior. Confirm .NET text fails as `{}` and Go lacks a stream-alarm harness. Hand-format .NET text to match the explicit rows; add the missing Go and Java coverage without changing their correct output.
**Step 2: RED/GREEN for CLI-49**
Add a Go compile-time consumer test that declares variables of `mxgateway.AlarmProviderStatus` and `mxgateway.AlarmSnapshotStatus`. Confirm red, then add aliases and update the five-arm `AlarmFeedMessage` comment.
**Step 3: RED/GREEN for CLI-50**
Add Python CLI tests proving default `stream-alarms` emits provider status, snapshot status, cached alarms, and later live messages incrementally until timeout/cancellation; add a finite `--max-messages` test. Change the default to `0` (unbounded) and print rows as consumed instead of collecting first. Keep the maximum bound for positive finite values and ensure iterator cleanup occurs in `finally`.
**Step 4: Verify all clients**
Run the authoritative commands from `docs/ClientPackaging.md`:
```bash
dotnet test clients/dotnet/ZB.MOM.WW.MxGateway.Client.slnx
(cd clients/go && go test ./...)
(cd clients/python && python -m pytest)
(cd clients/rust && cargo test --workspace)
(cd clients/java && JAVA_HOME=/opt/homebrew/opt/openjdk@17 gradle test)
```
Revert only proven generator churn; do not revert source changes. Update tracker and commit CLI-47–50.
### Task 5: Batch L — Live Watch-List Refresh And Protobuf Wave
**Follow-ups:** complete GWC-37 application path, WRK-32 public comment
**Files:**
- Modify: `src/ZB.MOM.WW.MxGateway.Contracts/Protos/mxaccess_gateway.proto`
- Regenerate: Contracts and every client generated tree required by `docs/Contracts.md`
- Modify/test: worker alarm command executor, handler, failover/subtag consumers, and Worker.Tests
- Modify/test: `src/ZB.MOM.WW.MxGateway.Server/Alarms/GatewayAlarmMonitor.cs` and alarm monitor tests
- Modify: alarm, worker protocol, configuration, and client contract documentation
- Modify: `archreview/2026-08-19/remediation/00-tracking.md`
**Step 1: RED — contract round trip**
Add a protobuf round-trip test for:
```proto
message UpdateAlarmWatchListCommand {
repeated AlarmSubtagTarget watch_list = 1;
}
```
Add a new non-conflicting `MxCommand` oneof field and `MxCommandKind`, plus an explicit `fallback_enabled` field on `SubscribeAlarmsCommand`. Confirm the test cannot compile before editing the proto.
**Step 2: RED — worker routing and failure preservation**
Add Worker.Tests covering command routing, rejection without an active/fallback-capable subscription, replacement setup failure preserving the previous standby, primary-mode replacement keeping primary active, and standby-mode replacement keeping subtag active.
**Step 3: GREEN — worker hot-swap**
Implement `IAlarmCommandHandler.UpdateWatchList`. Retain the subscription expression. Refactor `FailoverAlarmConsumer` so standby replacement performs build/subscribe/attach before swap and disposes the old standby after swap. No COM operation may leave the STA. In auto mode, build the composite with an empty standby when `fallback_enabled` is true; leave an intentionally alarm-manager-only subscription unchanged.
**Step 4: RED — gateway periodic update**
Extend `GatewayAlarmMonitorProviderModeTests`/`AlarmFailoverEndToEndTests` with these scenarios:
- startup discovery fails/empty, later reconcile resolves targets, exactly one update command is sent;
- unchanged targets send no update;
- changed targets update the sent fingerprint only after an OK reply;
- worker rejection retains the prior fingerprint and degraded coverage reason;
- failover edge performs an immediate refresh before publishing status.
**Step 5: GREEN — gateway refresh path**
Reuse `ReconcileIntervalSeconds`. Resolve only when mode can use subtags, compare canonical full-target fingerprints, invoke the new command on material change/recovery, and reconcile after success. Do not add a timer or option.
**Step 6: Complete the shared proto comments**
In the same proto edit:
- replace the IPC-36 internal environment-variable name with `MxGateway:Worker:WriteCompletionWaitMilliseconds`;
- describe WRK-32 correlation as recency-based per handle and name the sequential timed-out-predecessor stale-callback case, not only concurrent writes.
**Step 7: Regenerate once and run portable checks**
Follow `docs/Contracts.md` exactly, then run:
```bash
pwsh scripts/check-codegen.ps1
dotnet build src/ZB.MOM.WW.MxGateway.NonWindows.slnx
dotnet test src/ZB.MOM.WW.MxGateway.Tests/ZB.MOM.WW.MxGateway.Tests.csproj --filter "FullyQualifiedName~ProtobufContractRoundTripTests|FullyQualifiedName~AlarmFailoverEndToEndTests|FullyQualifiedName~GatewayAlarmMonitorProviderModeTests|FullyQualifiedName~ClientProtoInputTests"
```
Run all five client suites again because generated trees changed.
**Step 8: Verify worker on windev**
Commit the candidate, then run the repository's windev test mode at that SHA. Required evidence: x86 Worker build success and Worker.Tests with zero failures. Only then mark the watch-list/WRK-32 follow-ups Done in the tracker and commit any evidence-only tracker update.
### Task 6: Batch M — Windows Gateway Suite In Nightly CI
**Finding:** TST-35
**Files:**
- Modify: `scripts/ci/windev-worker-ci.ps1`
- Modify: `.gitea/workflows/ci.yml` comments if needed
- Modify: `docs/GatewayTesting.md`
- Modify: `scripts/ci/README.md`
- Modify: `archreview/2026-08-19/remediation/00-tracking.md`
**Step 1: Add an acceptance check before behavior**
Document the expected `live` order and add a script-level/static assertion or dry-run seam showing gateway tests are absent before the change.
**Step 2: Extend live mode**
After Worker.Tests and full-slnx build, run:
```powershell
Invoke-Native 'gateway tests on Windows' {
dotnet test 'src/ZB.MOM.WW.MxGateway.Tests/ZB.MOM.WW.MxGateway.Tests.csproj' -c Release --no-build
}
```
Keep it before the opt-in live-MXAccess smoke, under the existing host lock. Log the actual result; do not enforce a historical count.
**Step 3: Verify on windev**
Run `live` mode at the committed SHA. Require Worker.Tests, gateway tests, full solution build, and live smoke to complete successfully. Update docs/tracker with date, SHA, counts, and duration; commit TST-35.
### Task 7: Batch N — Full Integration And Release
**Follow-up:** publish the five clients and Contracts
**Files:**
- Modify after publication: `docs/ClientPackaging.md`
- Modify: `archreview/2026-08-19/remediation/00-tracking.md`
**Step 1: Full local integration gate**
Run:
```bash
git diff --check
pwsh scripts/check-codegen.ps1
pwsh scripts/check-client-versions.ps1
python3 code-reviews/regen-readme.py --check
dotnet build src/ZB.MOM.WW.MxGateway.NonWindows.slnx
dotnet test src/ZB.MOM.WW.MxGateway.NonWindows.slnx --no-build --no-restore
```
Run all five client suites. Require zero unexpected working-tree changes.
**Step 2: Final windev gate**
Run `windev-worker-ci.ps1 -Mode live` at the exact candidate SHA. Record Worker.Tests, gateway tests, live smoke, and full solution build evidence.
**Step 3: Integrate before release**
Use `superpowers:finishing-a-development-branch`. Merge/push the verified branch to `main`, confirm CI green, and resolve the exact release commit SHA. Do not publish artifacts from an unmerged feature commit.
**Step 4: Preflight exact coordinates**
Confirm these are absent (HTTP 404) immediately before publishing:
- NuGet `ZB.MOM.WW.MxGateway.Client` 0.2.1
- NuGet `ZB.MOM.WW.MxGateway.Contracts` 0.2.1
- PyPI `zb-mom-ww-mxaccess-gateway-client` 0.2.1
- Cargo `zb-mom-ww-mxgateway-client` 0.2.1
- Maven `com.zb.mom.ww.mxgateway:zb-mom-ww-mxgateway-client` 0.2.2
- Git tag `clients/go/v0.2.1`
Abort rather than overwrite if any coordinate exists unexpectedly.
**Step 5: Publish from clean `main`**
Run the documented `scripts/pack-clients.ps1 -Publish` commands for .NET, Python, Rust, and Java with required credentials, then:
```powershell
pwsh scripts/tag-go-module.ps1 -Version v0.2.1 -Push
```
The Go tag must point to the exact integrated release commit.
**Step 6: Verify release and close tracker**
Query every exact registry coordinate for HTTP 200 and resolve the Go module/tag remotely. Update `docs/ClientPackaging.md` and the tracker with release SHA, coordinates, and verification date. Run the doc checks, commit the release record, and push it.
## Final Tasklist Exit Gate
- [ ] All 16 open P2 findings are Done with commit/test evidence.
- [ ] WRK-34 is corrected to Done (`3679e59`).
- [ ] Live watch-list refresh passes portable and windev tests.
- [ ] WRK-32 public proto comment is regenerated through every client surface.
- [ ] TST-35 nightly Windows gateway suite is proven on windev.
- [ ] Full NonWindows gateway suite and all five client suites pass.
- [ ] Worker.Tests and Windows gateway suite pass at the integrated release SHA.
- [ ] Five client releases plus Contracts are confirmed in registries; Go tag resolves.
- [ ] `archreview/2026-08-19/remediation/00-tracking.md` contains no stale Deferred row for completed work.
+6 -2
View File
@@ -11,11 +11,15 @@ code back so a regression turns the job red.
| File | Runs on | Role |
|---|---|---|
| `run-windev-ci.sh <build\|test\|live>` | Linux CI job | Writes the SSH key/known-hosts, base64-encodes a bootstrap, SSHes to windev, returns the remote exit code. |
| `windev-worker-ci.ps1 -Sha -Mode` | windev | Locks the CI clone, checks out the SHA, runs the x86 build / `Worker.Tests` / live smoke. |
| `windev-worker-ci.ps1 -Sha -Mode` | windev | Locks the CI clone, checks out the SHA, and runs the requested cumulative Windows tiers. |
| `windev.known_hosts` | committed | Pinned windev host keys so `StrictHostKeyChecking=yes` has no TOFU prompt. Host keys are public; committing is intentional. |
Modes are cumulative: `build` = x86 Worker build; `test` = build + `Worker.Tests`;
`live` = test + full-`slnx` build + live-MXAccess smoke (`MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1`).
`live` = test + full-`slnx` build + the net10 gateway suite + live-MXAccess smoke
(`MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1`). The `live` order is x86 Worker build,
`Worker.Tests`, full solution build, gateway tests, then live smoke. One host lock covers
checkout and every tier, so concurrent runs cannot overlap; each native command is checked
immediately and the first nonzero exit aborts the remaining tiers and propagates to CI.
CI jobs (`.gitea/workflows/ci.yml`): `windows-x86` runs `test` per push/PR; `nightly-windev`
runs `live` on the `0 6 * * *` schedule and opens a Gitea issue on failure.
+7 -2
View File
@@ -16,8 +16,8 @@
Modes (cumulative):
build x86 Worker build only (net48/CS0246/x86 compile guard — always fast).
test build + Worker.Tests (x86). The per-push default.
live test + the live-MXAccess smoke (MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1) and a full
slnx build (surfaces Windows-only analyzer diagnostics, e.g. xUnit1030). Nightly.
live test + full slnx build + the net10 gateway suite + the live-MXAccess smoke
(MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1). Nightly.
Runs under Windows PowerShell 5.1 (`powershell.exe`), so it avoids PS7-only syntax and
checks $LASTEXITCODE after every native command rather than relying on stderr-as-error
@@ -112,6 +112,7 @@ try {
$workerProj = 'src/ZB.MOM.WW.MxGateway.Worker/ZB.MOM.WW.MxGateway.Worker.csproj'
$workerTests = 'src/ZB.MOM.WW.MxGateway.Worker.Tests/ZB.MOM.WW.MxGateway.Worker.Tests.csproj'
$gatewayTests = 'src/ZB.MOM.WW.MxGateway.Tests/ZB.MOM.WW.MxGateway.Tests.csproj'
$integrationTests = 'src/ZB.MOM.WW.MxGateway.IntegrationTests/ZB.MOM.WW.MxGateway.IntegrationTests.csproj'
# build: always. The x86/net48 compile is the floor guard (catches CS0246 on
@@ -127,6 +128,10 @@ try {
# that never fire on the macOS/Linux NonWindows build.
Invoke-Native 'full slnx build' { dotnet build src/ZB.MOM.WW.MxGateway.slnx -c Release }
Invoke-Native 'gateway tests on Windows' {
dotnet test $gatewayTests -c Release --no-build
}
$env:MXGATEWAY_RUN_LIVE_MXACCESS_TESTS = '1'
try {
Invoke-Native 'live-MXAccess smoke' {
File diff suppressed because it is too large Load Diff
@@ -130,6 +130,7 @@ message MxCommand {
WriteSecuredBulkCommand write_secured_bulk = 41;
WriteSecured2BulkCommand write_secured2_bulk = 42;
ReadBulkCommand read_bulk = 43;
UpdateAlarmWatchListCommand update_alarm_watch_list = 44;
PingCommand ping = 100;
GetSessionStateCommand get_session_state = 101;
GetWorkerInfoCommand get_worker_info = 102;
@@ -174,6 +175,7 @@ enum MxCommandKind {
MX_COMMAND_KIND_WRITE_SECURED_BULK = 32;
MX_COMMAND_KIND_WRITE_SECURED2_BULK = 33;
MX_COMMAND_KIND_READ_BULK = 34;
MX_COMMAND_KIND_UPDATE_ALARM_WATCH_LIST = 35;
MX_COMMAND_KIND_PING = 100;
MX_COMMAND_KIND_GET_SESSION_STATE = 101;
MX_COMMAND_KIND_GET_WORKER_INFO = 102;
@@ -344,8 +346,8 @@ enum AlarmProviderMode {
// Subscription expression follows the canonical
// `\\<machine>\Galaxy!<area>` format (literal "Galaxy" provider). The
// worker spins up a wnwrapConsumer-backed subscription on its STA on
// first call; subsequent calls are an error (use UnsubscribeAlarms then
// SubscribeAlarms to reconfigure).
// first call; subsequent calls are an error. Use UpdateAlarmWatchList to
// replace the standby targets without restarting the primary subscription.
message SubscribeAlarmsCommand {
string subscription_expression = 1;
// UNSPECIFIED = auto-failover/failback. ALARMMGR/SUBTAG force one provider.
@@ -354,6 +356,17 @@ message SubscribeAlarmsCommand {
// alarmmgr mode; in subtag mode it bounds what the consumer can observe.
repeated AlarmSubtagTarget watch_list = 3;
AlarmFailoverConfig failover = 4;
// True when auto mode should retain subtag-fallback capability even if the
// gateway's initial discovery produced an empty watch list. False leaves an
// empty list as an intentional alarm-manager-only subscription.
bool fallback_enabled = 5;
}
// Replace the live alarm subscription's subtag standby targets. The worker
// constructs and attaches the replacement before swapping it into service, so
// a setup failure leaves the existing standby and active provider unchanged.
message UpdateAlarmWatchListCommand {
repeated AlarmSubtagTarget watch_list = 1;
}
// Tear down the worker's alarm consumer. No-op if no subscription is
@@ -537,11 +550,14 @@ message MxCommandReply {
MxValue return_value = 6;
// Correlated per-item outcome rows. For WRITE / WRITE2 / WRITE_SECURED /
// WRITE_SECURED2 replies the worker holds the reply for a bounded window
// (default 1.5 s, MXGATEWAY_WORKER_WRITE_COMPLETION_WAIT_MS) waiting for
// (default 1.5 s, MxGateway:Worker:WriteCompletionWaitMilliseconds) waiting for
// the matching MXAccess OnWriteComplete callback and copies its status rows
// here, so statuses[0] carries the real MXAccess commit outcome (success OR
// failure) while protocol_status/hresult still describe command acceptance
// only. Empty statuses on a write reply means the completion did not arrive
// only. Callback correlation is recency-based per item handle: after the
// bounded wait, a sequential write's callback supersedes a timed-out
// predecessor, and a later stale callback for that predecessor is ignored.
// Empty statuses on a write reply means the completion did not arrive
// within the window — the write is unconfirmed, not failed. Correlation is
// best-effort per (server_handle, item_handle): MXAccess's callback carries
// no transaction id, so concurrent writes to the same item within the
@@ -0,0 +1,25 @@
namespace ZB.MOM.WW.MxGateway.Contracts;
/// <summary>Names of environment variables shared by the gateway worker launcher and worker process.</summary>
public static class WorkerLaunchEnvironment
{
/// <summary>Bootstrap nonce used to authenticate the worker handshake.</summary>
public const string WorkerNonce = "MXGATEWAY_WORKER_NONCE";
/// <summary>Per-attempt named-pipe connection timeout in milliseconds.</summary>
public const string PipeConnectAttemptTimeoutMilliseconds =
"MXGATEWAY_WORKER_PIPE_CONNECT_ATTEMPT_TIMEOUT_MS";
/// <summary>Bounded wait for a Write/Write2/WriteSecured/WriteSecured2 completion callback in milliseconds.</summary>
public const string WriteCompletionWaitMilliseconds =
"MXGATEWAY_WORKER_WRITE_COMPLETION_WAIT_MS";
/// <summary>Capacity of the worker's outbound MXAccess event queue.</summary>
public const string EventQueueCapacity = "MXGATEWAY_EVENT_QUEUE_CAPACITY";
/// <summary>Worker alarm polling cadence in milliseconds.</summary>
public const string AlarmPollIntervalMilliseconds = "MXGATEWAY_ALARM_POLL_INTERVAL_MS";
/// <summary>Maximum alarm records requested in one current-alarm fetch.</summary>
public const string MaxAlarmsPerFetch = "MXGATEWAY_ALARM_MAX_ALARMS_PER_FETCH";
}
@@ -37,7 +37,9 @@
the wire-identical Galaxy types from the ZB.MOM.WW.GalaxyRepository package. This proto is
still the generation source of truth for the language clients (Go/Rust/Java/Python compile it
by path; the .NET client consumes Contracts.Proto.Galaxy.* via this project reference) and for
clients/proto/proto-inputs.json. Removing it breaks all five clients — do not delete. -->
clients/proto/proto-inputs.json. GalaxyContractParityTests normalizes its descriptor against
the descriptor served by the package and fails on wire drift. Removing it breaks all five
clients — do not delete. -->
<Protobuf Include="Protos\galaxy_repository.proto" ProtoRoot="Protos" OutputDir="Generated" GrpcOutputDir="Generated" GrpcServices="Both" />
</ItemGroup>
@@ -66,9 +66,10 @@ public sealed class GatewayAlarmMonitor : BackgroundService, IGatewayAlarmServic
private bool _subtagCoverageMissing;
private string _subtagCoverageReason = string.Empty;
// The watch-list references actually sent to the worker in SubscribeAlarms, guarded by _sync.
// The worker holds that list for the life of the subscription, so a later re-resolution is
// compared against this to report drift rather than silently diverging.
// The full canonical watch-list fingerprint acknowledged by the worker and its alarm
// references, guarded by _sync. The fingerprint advances only after an OK subscribe/update
// reply, so a rejected replacement remains material and is retried on the next refresh edge.
private string _acknowledgedWatchListFingerprint = string.Empty;
private IReadOnlyCollection<string> _sentWatchListReferences = [];
// Whether the worker's most recent reconcile fetch was capped, guarded by _sync.
@@ -356,9 +357,6 @@ public sealed class GatewayAlarmMonitor : BackgroundService, IGatewayAlarmServic
// feed subscribers outlive the monitor's worker session and may already be attached.
lock (_sync)
{
_sentWatchListReferences = watchList
.Select(target => target.AlarmFullReference)
.ToHashSet(StringComparer.OrdinalIgnoreCase);
SetSubtagCoverage(GradeSubtagCoverage(forcedMode, resolution, staleAgainstSent: false), broadcast: true);
}
@@ -374,6 +372,7 @@ public sealed class GatewayAlarmMonitor : BackgroundService, IGatewayAlarmServic
{
SubscriptionExpression = subscription,
ForcedMode = forcedMode,
FallbackEnabled = CanUseSubtagFallback(forcedMode),
Failover = new AlarmFailoverConfig
{
ConsecutiveFailureThreshold = _options.Fallback.ConsecutiveFailureThreshold,
@@ -404,6 +403,11 @@ public sealed class GatewayAlarmMonitor : BackgroundService, IGatewayAlarmServic
?? $"status {code}";
throw new InvalidOperationException($"Worker rejected SubscribeAlarms: {diagnostic}");
}
lock (_sync)
{
AcknowledgeWatchList(watchList);
}
}
private async Task ReconcileLoopAsync(string sessionId, CancellationToken cancellationToken)
@@ -416,6 +420,8 @@ public sealed class GatewayAlarmMonitor : BackgroundService, IGatewayAlarmServic
{
try
{
await RefreshAlarmWatchListAsync(sessionId, broadcast: true, cancellationToken)
.ConfigureAwait(false);
await ReconcileAsync(sessionId, cancellationToken).ConfigureAwait(false);
}
catch (OperationCanceledException)
@@ -545,18 +551,14 @@ public sealed class GatewayAlarmMonitor : BackgroundService, IGatewayAlarmServic
AlarmProviderMode toMode = change.Mode;
string reason = change.Reason ?? string.Empty;
// GWC-37: this is the moment the subtag watch-list stops being theoretical — the provider
// that is about to serve is the one built from it. Re-resolve BEFORE the status is
// published so the frame the operator sees on the failover carries the coverage verdict
// rather than a second frame later, and so a watch-list that has drifted since
// SubscribeAlarms is reported instead of silently diverging. The re-resolution is NOT
// re-sent: the worker rejects a second SubscribeAlarms while subscribed, and the only way
// to replace the list — Unsubscribe then Subscribe — rebuilds the FailoverAlarmConsumer,
// which restarts on the primary that just failed and discards the warm standby state, so
// it would flap the feed rather than repair it. A monitor restart re-resolves and re-sends.
// The failover edge is the moment the subtag watch-list stops being theoretical. Refresh
// it before publishing the status frame so recovered discovery is applied to the already
// active standby and the operator sees the acknowledged coverage verdict atomically with
// the provider switch.
if (toMode == AlarmProviderMode.Subtag)
{
await RefreshSubtagCoverageAsync(cancellationToken).ConfigureAwait(false);
await RefreshAlarmWatchListAsync(sessionId, broadcast: false, cancellationToken)
.ConfigureAwait(false);
}
AlarmProviderStatus status;
@@ -606,14 +608,17 @@ public sealed class GatewayAlarmMonitor : BackgroundService, IGatewayAlarmServic
}
}
// GWC-37: re-resolves the subtag watch-list and re-grades the coverage verdict without
// touching the worker's subscription. Best-effort: a resolver that throws (only cancellation
// and programming errors do — discovery failure is reported in the resolution) leaves the
// verdict from the last subscribe in place, because a stale verdict is better than none.
// Does not broadcast: the callers publish a status frame of their own immediately after.
private async Task RefreshSubtagCoverageAsync(CancellationToken cancellationToken)
private async Task RefreshAlarmWatchListAsync(
string sessionId,
bool broadcast,
CancellationToken cancellationToken)
{
AlarmProviderMode forcedMode = MapForcedMode(_options.Fallback.Mode);
if (!CanUseSubtagFallback(forcedMode))
{
return;
}
AlarmWatchListResolution resolution;
try
{
@@ -627,18 +632,96 @@ public sealed class GatewayAlarmMonitor : BackgroundService, IGatewayAlarmServic
{
_logger.LogWarning(
exception,
"Re-resolving the alarm subtag watch-list failed; keeping the coverage verdict from the last subscribe.");
"Re-resolving the alarm subtag watch-list failed; keeping the last acknowledged watch list.");
return;
}
string resolvedFingerprint = CanonicalWatchListFingerprint(resolution.Targets);
string acknowledgedFingerprint;
lock (_sync)
{
bool stale = !resolution.Targets
.Select(target => target.AlarmFullReference)
.ToHashSet(StringComparer.OrdinalIgnoreCase)
.SetEquals(_sentWatchListReferences);
SetSubtagCoverage(GradeSubtagCoverage(forcedMode, resolution, stale), broadcast: false);
acknowledgedFingerprint = _acknowledgedWatchListFingerprint;
}
if (string.Equals(resolvedFingerprint, acknowledgedFingerprint, StringComparison.Ordinal))
{
lock (_sync)
{
SetSubtagCoverage(
GradeSubtagCoverage(forcedMode, resolution, staleAgainstSent: false),
broadcast);
}
return;
}
UpdateAlarmWatchListCommand update = new();
update.WatchList.AddRange(resolution.Targets);
WorkerCommandReply reply = await _sessionManager.InvokeAsync(
sessionId,
new WorkerCommand
{
Command = new MxCommand
{
Kind = MxCommandKind.UpdateAlarmWatchList,
UpdateAlarmWatchList = update,
},
},
cancellationToken)
.ConfigureAwait(false);
ProtocolStatusCode? code = reply.Reply?.ProtocolStatus?.Code;
lock (_sync)
{
if (code == ProtocolStatusCode.Ok)
{
AcknowledgeWatchList(resolution.Targets);
SetSubtagCoverage(
GradeSubtagCoverage(forcedMode, resolution, staleAgainstSent: false),
broadcast);
return;
}
string diagnostic = reply.Reply?.DiagnosticMessage
?? reply.Reply?.ProtocolStatus?.Message
?? $"status {code}";
string reason = GradeSubtagCoverage(forcedMode, resolution, staleAgainstSent: true);
SetSubtagCoverage($"{reason}; worker rejected watch-list update ({diagnostic})", broadcast);
}
}
private bool CanUseSubtagFallback(AlarmProviderMode forcedMode) =>
forcedMode != AlarmProviderMode.Alarmmgr
&& (forcedMode == AlarmProviderMode.Subtag
|| _options.Fallback.Discovery.UseGalaxyRepository
|| _options.Fallback.Discovery.IncludeAttributes.Length > 0);
// Caller holds _sync.
private void AcknowledgeWatchList(IReadOnlyList<AlarmSubtagTarget> watchList)
{
_acknowledgedWatchListFingerprint = CanonicalWatchListFingerprint(watchList);
_sentWatchListReferences = watchList
.Select(target => target.AlarmFullReference)
.ToHashSet(StringComparer.OrdinalIgnoreCase);
}
private static string CanonicalWatchListFingerprint(IEnumerable<AlarmSubtagTarget> watchList) =>
string.Join(
"\n",
watchList
.Select(target => string.Join(
"|",
FingerprintPart(target.AlarmFullReference),
FingerprintPart(target.SourceObjectReference),
FingerprintPart(target.ActiveSubtag),
FingerprintPart(target.AckedSubtag),
FingerprintPart(target.AckCommentSubtag),
FingerprintPart(target.PrioritySubtag)))
.Order(StringComparer.Ordinal));
private static string FingerprintPart(string? value)
{
string normalized = value ?? string.Empty;
return $"{normalized.Length}:{normalized}";
}
// Grades one resolution against the effective mode. Returns the reason the subtag coverage is
@@ -684,8 +767,7 @@ public sealed class GatewayAlarmMonitor : BackgroundService, IGatewayAlarmServic
{
reasons.Add(
$"{AlarmProviderReasons.SubtagWatchListStale} ({resolution.Targets.Count} target(s) now "
+ $"resolve against {_sentWatchListReferences.Count} sent at subscribe); restart the alarm "
+ "monitor to re-send it");
+ $"resolve against {_sentWatchListReferences.Count} acknowledged by the worker)");
}
return string.Join("; ", reasons);
@@ -839,15 +921,17 @@ public sealed class GatewayAlarmMonitor : BackgroundService, IGatewayAlarmServic
new AlarmFeedMessage { Transition = TransitionFromSnapshot(incoming.Value, AlarmTransitionKind.Raise) },
incoming.Key);
}
else if (existing.CurrentState != incoming.Value.CurrentState
&& incoming.Value.CurrentState == AlarmConditionState.ActiveAcked)
else if (existing.CurrentState != incoming.Value.CurrentState)
{
// The alarm was already known but the worker now reports it acknowledged: the
// live Acknowledge transition never reached the feed. Without this the acked
// state is absorbed silently by the snapshot replace below and subscribers show
// the alarm unacked until it clears.
// The alarm was already known but the worker now reports a different state. An
// ActiveAcked snapshot repairs a missed Acknowledge. Any other incoming active
// state repairs a missed Raise for the incoming instance (including an
// acknowledged alarm reference that was cleared and re-raised between passes).
AlarmTransitionKind kind = incoming.Value.CurrentState == AlarmConditionState.ActiveAcked
? AlarmTransitionKind.Acknowledge
: AlarmTransitionKind.Raise;
Broadcast(
new AlarmFeedMessage { Transition = TransitionFromSnapshot(incoming.Value, AlarmTransitionKind.Acknowledge) },
new AlarmFeedMessage { Transition = TransitionFromSnapshot(incoming.Value, kind) },
incoming.Key);
}
}
@@ -28,8 +28,11 @@ namespace ZB.MOM.WW.MxGateway.Server.Security.Authorization;
/// <para>
/// The tracked partition set is a bounded LRU (<see cref="SecurityOptions.ApiKeyFailureTrackedPeers"/>).
/// Because only a validly shaped token mints a key-id partition and each transport peer may mint at
/// most <see cref="MaxKeyIdPartitionsPerPeer"/> of them (the overflow collapses onto that address's
/// fallback partition), the map bounds memory <em>and</em> resists a junk-token flush (SEC-32):
/// most <see cref="MaxKeyIdPartitionsPerPeer"/> of them (the peer-layer overflow collapses onto that
/// address's fallback partition), the map bounds memory <em>and</em> resists a junk-token flush
/// (SEC-32). A validly shaped presented key id still feeds its cross-peer aggregate when its peer
/// layer uses the fallback, so exhausting the per-peer cap cannot bypass distributed-spray
/// protection; the aggregate map remains bounded by the same capped eviction policy:
/// eviction prefers fully expired windows, never removes an over-limit partition below a transient
/// overshoot ceiling of twice the cap, and only then falls back to least-recently-active. Successful
/// peers are removed on reset, so in steady state the map holds only partitions with recent failures
@@ -155,9 +158,9 @@ public sealed class ApiKeyFailureLimiter
WindowState? peerState = _partitions.TryGetValue(partitionKey, out WindowState? tracked) ? tracked : null;
WindowState? aggregateState = null;
if (effectiveKeyId is not null
if (partition.KeyId is not null
&& _aggregateLimit > 0
&& _aggregates.TryGetValue(effectiveKeyId, out WindowState? aggregate))
&& _aggregates.TryGetValue(partition.KeyId, out WindowState? aggregate))
{
aggregateState = aggregate;
}
@@ -222,11 +225,13 @@ public sealed class ApiKeyFailureLimiter
(Peer: peer, KeyId: effectiveKeyId));
RecordInto(state, now, _limit);
// Only a key id that earned its own partition feeds the aggregate: an id squeezed out by the
// per-peer cap is spray, and letting it through would make aggregate cardinality unbounded.
if (effectiveKeyId is not null && _aggregateLimit > 0)
// The aggregate follows the validly shaped presented key id independently of the peer
// layer's storage choice. Otherwise an attacker could exhaust each peer's dedicated-key cap
// and route a victim id through the fallback to bypass cross-peer throttling. Aggregate
// cardinality is still bounded by EvictIfOverCapacity below.
if (partition.KeyId is not null && _aggregateLimit > 0)
{
WindowState aggregate = _aggregates.GetOrAdd(effectiveKeyId, static _ => new WindowState(null, null));
WindowState aggregate = _aggregates.GetOrAdd(partition.KeyId, static _ => new WindowState(null, null));
RecordInto(aggregate, now, _aggregateLimit);
}
@@ -261,7 +266,8 @@ public sealed class ApiKeyFailureLimiter
// contributed by other key ids — and by junk-shaped tokens — from the same address, so
// removing it would let one successful authentication wipe an in-progress spray. The shared
// bucket decays by window expiry instead; the caller still recovers through probe admission.
if (string.Equals(effectiveKeyId, partition.KeyId, StringComparison.Ordinal))
if (effectiveKeyId is not null
&& string.Equals(effectiveKeyId, partition.KeyId, StringComparison.Ordinal))
{
RemovePartition(partitionKey);
}
@@ -24,9 +24,11 @@ namespace ZB.MOM.WW.MxGateway.Server.Sessions;
/// replay batch.
/// </param>
/// <param name="OldestAvailableSequence">
/// The oldest worker sequence still retained and replayable; <c>0</c> when nothing is
/// retained. Populates the <c>ReplayGap.oldest_available_sequence</c> field. Meaningful
/// only when <paramref name="Gap"/> is <see langword="true"/>.
/// Populates the <c>ReplayGap.oldest_available_sequence</c> field. When
/// <paramref name="Gap"/> is <see langword="true"/>, this is the oldest worker sequence
/// still retained and replayable, or the highest observed sequence plus one when the ring
/// is empty. It is <c>0</c> only when <paramref name="Gap"/> is
/// <see langword="false"/>, where the value is not emitted.
/// </param>
/// <param name="LiveResumeSequence">
/// The worker sequence the live channel must resume strictly after: the highest replayed
@@ -444,11 +444,10 @@ public sealed class WorkerClient : IWorkerClient
{
_state = WorkerClientState.Closed;
}
UnregisterEventQueueDepthSource();
}
// Drop out of the worker queue-depth gauge before teardown: whatever this client still holds
// is about to be discarded, and the sum must not keep counting a client that is going away.
_eventQueueDepthRegistration?.Dispose();
KillOwnedProcess("Dispose");
_stopCts.Cancel();
_outboundEnvelopes.Writer.TryComplete();
@@ -1092,6 +1091,7 @@ public sealed class WorkerClient : IWorkerClient
}
_state = WorkerClientState.Closed;
UnregisterEventQueueDepthSource();
}
_stopCts.Cancel();
@@ -1126,6 +1126,7 @@ public sealed class WorkerClient : IWorkerClient
}
_state = WorkerClientState.Faulted;
UnregisterEventQueueDepthSource();
}
_stopCts.Cancel();
@@ -1139,6 +1140,16 @@ public sealed class WorkerClient : IWorkerClient
_logger.LogWarning(exception, "Worker client faulted for session {SessionId}: {Message}", SessionId, message);
}
/// <summary>
/// Removes this terminal client's abandoned backlog from the gateway-wide queue-depth
/// gauge. The registration handle is idempotent, so terminal transitions and later disposal
/// may all call this safely while the faulted client remains retained for diagnostics.
/// </summary>
private void UnregisterEventQueueDepthSource()
{
_eventQueueDepthRegistration?.Dispose();
}
private void KillOwnedProcess(string reason)
{
WorkerProcessHandle? processHandle = _connection.ProcessHandle;
@@ -2,6 +2,7 @@ using System.Diagnostics;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Options;
using ZB.MOM.WW.MxGateway.Contracts;
using ZB.MOM.WW.MxGateway.Server.Configuration;
using ZB.MOM.WW.MxGateway.Server.Metrics;
using Polly;
@@ -15,41 +16,47 @@ namespace ZB.MOM.WW.MxGateway.Server.Workers;
public sealed class WorkerProcessLauncher : IWorkerProcessLauncher
{
/// <summary>Environment variable for worker nonce.</summary>
public const string WorkerNonceEnvironmentVariableName = "MXGATEWAY_WORKER_NONCE";
[Obsolete("Use WorkerLaunchEnvironment.WorkerNonce.")]
public const string WorkerNonceEnvironmentVariableName = WorkerLaunchEnvironment.WorkerNonce;
/// <summary>Environment variable for worker pipe connect attempt timeout.</summary>
[Obsolete("Use WorkerLaunchEnvironment.PipeConnectAttemptTimeoutMilliseconds.")]
public const string WorkerPipeConnectAttemptTimeoutEnvironmentVariableName =
"MXGATEWAY_WORKER_PIPE_CONNECT_ATTEMPT_TIMEOUT_MS";
WorkerLaunchEnvironment.PipeConnectAttemptTimeoutMilliseconds;
/// <summary>
/// Conveys MxGateway:Worker:WriteCompletionWaitMilliseconds to the worker:
/// the bounded wait for the OnWriteComplete callback on
/// WriteSecured/WriteSecured2 replies. 0 disables the wait.
/// the bounded wait for the OnWriteComplete callback on Write, Write2,
/// WriteSecured, and WriteSecured2 replies. 0 disables the wait.
/// </summary>
[Obsolete("Use WorkerLaunchEnvironment.WriteCompletionWaitMilliseconds.")]
public const string WorkerWriteCompletionWaitEnvironmentVariableName =
"MXGATEWAY_WORKER_WRITE_COMPLETION_WAIT_MS";
WorkerLaunchEnvironment.WriteCompletionWaitMilliseconds;
/// <summary>
/// Conveys MxGateway:Alarms:PollIntervalMilliseconds to the worker: the
/// cadence at which the worker's STA polls the AVEVA alarm consumer.
/// </summary>
[Obsolete("Use WorkerLaunchEnvironment.AlarmPollIntervalMilliseconds.")]
public const string WorkerAlarmPollIntervalEnvironmentVariableName =
"MXGATEWAY_ALARM_POLL_INTERVAL_MS";
WorkerLaunchEnvironment.AlarmPollIntervalMilliseconds;
/// <summary>
/// Conveys MxGateway:Alarms:MaxAlarmsPerFetch to the worker: the cap
/// passed to GetXmlCurrentAlarms2, which is also the record count at
/// which the worker treats a fetch as truncated.
/// </summary>
[Obsolete("Use WorkerLaunchEnvironment.MaxAlarmsPerFetch.")]
public const string WorkerMaxAlarmsPerFetchEnvironmentVariableName =
"MXGATEWAY_ALARM_MAX_ALARMS_PER_FETCH";
WorkerLaunchEnvironment.MaxAlarmsPerFetch;
/// <summary>
/// Conveys MxGateway:Worker:EventQueueCapacity to the worker: the capacity
/// of the outbound MXAccess event queue, whose overflow faults the session.
/// </summary>
[Obsolete("Use WorkerLaunchEnvironment.EventQueueCapacity.")]
public const string WorkerEventQueueCapacityEnvironmentVariableName =
"MXGATEWAY_EVENT_QUEUE_CAPACITY";
WorkerLaunchEnvironment.EventQueueCapacity;
private readonly IWorkerProcessFactory _processFactory;
private readonly IWorkerStartupProbe _startupProbe;
@@ -204,16 +211,16 @@ public sealed class WorkerProcessLauncher : IWorkerProcessLauncher
startInfo.ArgumentList.Add(argument);
}
startInfo.Environment[WorkerNonceEnvironmentVariableName] = request.Nonce;
startInfo.Environment[WorkerPipeConnectAttemptTimeoutEnvironmentVariableName] =
startInfo.Environment[WorkerLaunchEnvironment.WorkerNonce] = request.Nonce;
startInfo.Environment[WorkerLaunchEnvironment.PipeConnectAttemptTimeoutMilliseconds] =
_workerOptions.PipeConnectAttemptTimeoutMilliseconds.ToString(System.Globalization.CultureInfo.InvariantCulture);
startInfo.Environment[WorkerWriteCompletionWaitEnvironmentVariableName] =
startInfo.Environment[WorkerLaunchEnvironment.WriteCompletionWaitMilliseconds] =
_workerOptions.WriteCompletionWaitMilliseconds.ToString(System.Globalization.CultureInfo.InvariantCulture);
startInfo.Environment[WorkerEventQueueCapacityEnvironmentVariableName] =
startInfo.Environment[WorkerLaunchEnvironment.EventQueueCapacity] =
_workerOptions.EventQueueCapacity.ToString(System.Globalization.CultureInfo.InvariantCulture);
startInfo.Environment[WorkerAlarmPollIntervalEnvironmentVariableName] =
startInfo.Environment[WorkerLaunchEnvironment.AlarmPollIntervalMilliseconds] =
_alarmsOptions.PollIntervalMilliseconds.ToString(System.Globalization.CultureInfo.InvariantCulture);
startInfo.Environment[WorkerMaxAlarmsPerFetchEnvironmentVariableName] =
startInfo.Environment[WorkerLaunchEnvironment.MaxAlarmsPerFetch] =
_alarmsOptions.MaxAlarmsPerFetch.ToString(System.Globalization.CultureInfo.InvariantCulture);
commandLine = new WorkerProcessCommandLine(executablePath, arguments);
@@ -155,6 +155,66 @@ public sealed class GatewayAlarmMonitorAttachOrderTests
await monitor.StopAsync(CancellationToken.None);
}
/// <summary>
/// GWC-35: an acknowledged cached alarm followed by an active snapshot with a newer
/// original-raise timestamp is a new alarm instance, not an acknowledgement reversal. The
/// reconcile repair must broadcast a Raise carrying the incoming instance identity.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task ApplyReconcile_WhenAcknowledgedAlarmHasNewActiveInstance_BroadcastsRaise()
{
using GatewayMetrics metrics = new();
await using FakeSessionManager sessions = new();
using GatewayAlarmMonitor monitor = CreateMonitor(sessions, metrics);
using CancellationTokenSource cts = new();
await monitor.StartAsync(cts.Token);
await sessions.WaitForSubscribeStartAsync(WaitTimeout);
Timestamp oldRaise = Timestamp.FromDateTimeOffset(
new DateTimeOffset(2026, 8, 10, 13, 0, 0, TimeSpan.Zero));
sessions.SetReconcileSnapshot(SnapshotAt(AlarmConditionState.ActiveAcked, oldRaise, oldRaise));
sessions.EmitEvent(ProviderModeProbe(1));
await WaitUntilAsync(
() => monitor.CurrentAlarms.Any(alarm => alarm.AlarmFullReference == AlarmReference
&& alarm.CurrentState == AlarmConditionState.ActiveAcked),
WaitTimeout);
List<AlarmFeedMessage> received = [];
TaskCompletionSource snapshotComplete = new(TaskCreationOptions.RunContinuationsAsynchronously);
using CancellationTokenSource streamCts = new();
Task reader = ReadFeedAsync(monitor, received, snapshotComplete, streamCts.Token, untilSnapshotComplete: true);
await snapshotComplete.Task.WaitAsync(WaitTimeout);
Timestamp newRaise = Timestamp.FromDateTimeOffset(
new DateTimeOffset(2026, 8, 10, 13, 0, 20, TimeSpan.Zero));
sessions.SetReconcileSnapshot(SnapshotAt(AlarmConditionState.Active, newRaise, newRaise));
sessions.EmitEvent(ProviderModeProbe(2));
AlarmFeedMessage raise = await WaitForAsync(
received,
m => m.PayloadCase == AlarmFeedMessage.PayloadOneofCase.Transition
&& m.Transition.TransitionKind == AlarmTransitionKind.Raise,
WaitTimeout);
Assert.Equal(AlarmReference, raise.Transition.AlarmFullReference);
Assert.Equal(newRaise, raise.Transition.OriginalRaiseTimestamp);
Assert.Equal(newRaise, raise.Transition.TransitionTimestamp);
lock (received)
{
AlarmFeedMessage[] transitions = received
.Where(m => m.PayloadCase == AlarmFeedMessage.PayloadOneofCase.Transition)
.ToArray();
Assert.Single(transitions);
}
await streamCts.CancelAsync();
await reader;
await cts.CancelAsync();
await monitor.StopAsync(CancellationToken.None);
}
/// <summary>
/// NEXT-03. A reconcile Raise repair applied while the matching live Raise is still
/// buffered must not double-broadcast: the live transition carrying the same worker
@@ -646,6 +646,7 @@ public sealed class GatewayAlarmMonitorProviderModeTests
// The empty list still went to the worker — the verdict is a report, not a behaviour change.
Assert.Empty(sessions.LastSubscribeCommand!.WatchList);
Assert.True(sessions.LastSubscribeCommand.FallbackEnabled);
AlarmProviderStatus status = await ReadFirstProviderStatusAsync(monitor);
@@ -661,16 +662,14 @@ public sealed class GatewayAlarmMonitorProviderModeTests
/// <summary>
/// GWC-37: the failover edge is the moment the watch-list stops being theoretical, so the
/// monitor re-resolves it there and publishes the fresh verdict on the same status frame
/// the failover produces. Discovery that has recovered since <c>SubscribeAlarms</c> is
/// reported as drift rather than applied — the worker rejects a second <c>SubscribeAlarms</c>
/// while subscribed, and the only way to replace the list rebuilds the failover consumer on
/// the primary that just failed. The status names both the fresh count and the sent count so
/// an operator knows a monitor restart will pick the new list up.
/// monitor re-resolves it there, hot-swaps the recovered list through
/// <c>UpdateAlarmWatchList</c>, and publishes the acknowledged verdict on the same status
/// frame the failover produces. The update must precede the status frame so the frame never
/// claims recovered coverage before the worker has accepted it.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task SubtagCoverage_WhenDiscoveryRecoversByFailover_ReResolvesAndReportsDrift()
public async Task SubtagCoverage_WhenDiscoveryRecoversByFailover_UpdatesWorkerBeforePublishingStatus()
{
using GatewayMetrics metrics = new();
FakeSessionManager sessions = new();
@@ -734,13 +733,11 @@ public sealed class GatewayAlarmMonitorProviderModeTests
// The re-resolve happened on the failover edge, before the status was published.
Assert.Equal(2, resolver.Calls);
Assert.Single(sessions.UpdateCommands);
Assert.Equal(2, sessions.UpdateCommands[0].WatchList.Count);
string reason = degraded.ProviderStatus.Reason;
Assert.Contains("alarmmgr failed", reason, StringComparison.Ordinal);
Assert.Contains(AlarmProviderReasons.SubtagWatchListStale, reason, StringComparison.Ordinal);
Assert.Contains("2 target(s) now resolve against 0 sent", reason, StringComparison.Ordinal);
// The stale-list warning replaced the empty/failed pair: discovery works again, so the only
// remaining problem is that the worker is still using the list it was handed.
Assert.DoesNotContain(AlarmProviderReasons.SubtagWatchListStale, reason, StringComparison.Ordinal);
Assert.DoesNotContain(AlarmProviderReasons.SubtagWatchListEmpty, reason, StringComparison.Ordinal);
Assert.DoesNotContain(AlarmProviderReasons.SubtagDiscoveryFailed, reason, StringComparison.Ordinal);
@@ -750,6 +747,69 @@ public sealed class GatewayAlarmMonitorProviderModeTests
await monitor.StopAsync(CancellationToken.None);
}
/// <summary>Periodic reconciliation applies a recovered watch list once and suppresses an unchanged resolution on the next cadence.</summary>
[Fact]
public async Task WatchListRefresh_PeriodicRecovery_SendsExactlyOneMaterialUpdate()
{
using GatewayMetrics metrics = new();
FakeSessionManager sessions = new();
AlarmWatchListResolution recovered = new(
[Target("Galaxy!Area.Tank01.Level.HiHi")],
DiscoveryAttempted: true,
DiscoveryFailed: false,
DiscoveryFailureMessage: null);
ScriptedWatchListResolver resolver = new(DiscoveryFailedResolution(), recovered, recovered);
using GatewayAlarmMonitor monitor = CreateMonitor(sessions, metrics, resolver, reconcileIntervalSeconds: 5);
using CancellationTokenSource cts = new();
await monitor.StartAsync(cts.Token);
await sessions.WaitForUpdateCountAsync(1, WaitTimeout);
await WaitUntilAsync(() => resolver.Calls >= 3, WaitTimeout);
Assert.Single(sessions.UpdateCommands);
Assert.Single(sessions.UpdateCommands[0].WatchList);
await cts.CancelAsync();
await monitor.StopAsync(CancellationToken.None);
}
/// <summary>A rejected refresh keeps the old fingerprint and degraded verdict so the same material change is retried.</summary>
[Fact]
public async Task WatchListRefresh_WorkerRejects_RetainsFingerprintAndRetriesSameTargets()
{
using GatewayMetrics metrics = new();
FakeSessionManager sessions = new();
sessions.UpdateReplyCodes.Enqueue(ProtocolStatusCode.MxaccessFailure);
sessions.UpdateReplyCodes.Enqueue(ProtocolStatusCode.Ok);
AlarmWatchListResolution recovered = new(
[Target("Galaxy!Area.Tank01.Level.HiHi")],
DiscoveryAttempted: true,
DiscoveryFailed: false,
DiscoveryFailureMessage: null);
ScriptedWatchListResolver resolver = new(DiscoveryFailedResolution(), recovered, recovered);
using GatewayAlarmMonitor monitor = CreateMonitor(sessions, metrics, resolver);
using CancellationTokenSource cts = new();
await monitor.StartAsync(cts.Token);
await sessions.WaitForSubscribeAsync(WaitTimeout);
sessions.EmitModeChange(AlarmProviderMode.Subtag, "alarmmgr failed");
await sessions.WaitForUpdateCountAsync(1, WaitTimeout);
AlarmProviderStatus rejected = await ReadFirstProviderStatusAsync(monitor);
Assert.True(rejected.Degraded);
Assert.Contains(AlarmProviderReasons.SubtagWatchListStale, rejected.Reason, StringComparison.Ordinal);
sessions.EmitModeChange(AlarmProviderMode.Subtag, "alarmmgr still failed");
await sessions.WaitForUpdateCountAsync(2, WaitTimeout);
Assert.Equal(
sessions.UpdateCommands[0].WatchList[0].AlarmFullReference,
sessions.UpdateCommands[1].WatchList[0].AlarmFullReference);
await cts.CancelAsync();
await monitor.StopAsync(CancellationToken.None);
}
/// <summary>Reads the provider-status frame every stream opens with, then closes the stream.</summary>
/// <param name="monitor">The monitor to read from.</param>
/// <returns>The provider status carried by the first frame.</returns>
@@ -786,12 +846,14 @@ public sealed class GatewayAlarmMonitorProviderModeTests
private static GatewayAlarmMonitor CreateMonitor(
FakeSessionManager sessions,
GatewayMetrics metrics,
IAlarmWatchListResolver resolver)
IAlarmWatchListResolver resolver,
int reconcileIntervalSeconds = 30)
{
AlarmsOptions options = new()
{
Enabled = true,
SubscriptionExpression = @"\\NODE\Galaxy!Area",
ReconcileIntervalSeconds = reconcileIntervalSeconds,
};
return new GatewayAlarmMonitor(
sessions,
@@ -923,11 +985,34 @@ public sealed class GatewayAlarmMonitorProviderModeTests
/// <summary>The most recent SubscribeAlarms command the monitor sent.</summary>
public SubscribeAlarmsCommand? LastSubscribeCommand { get; private set; }
/// <summary>Gets each live replacement command in invocation order.</summary>
public List<UpdateAlarmWatchListCommand> UpdateCommands { get; } = [];
/// <summary>Scripted protocol statuses for replacement commands.</summary>
public Queue<ProtocolStatusCode> UpdateReplyCodes { get; } = new();
/// <summary>Pushes a worker event onto the monitor's event stream.</summary>
/// <param name="mxEvent">The worker event to push.</param>
public void EmitEvent(MxEvent mxEvent) =>
_events.Writer.TryWrite(new WorkerEvent { Event = mxEvent });
/// <summary>Pushes a provider-mode change onto the worker event stream.</summary>
public void EmitModeChange(AlarmProviderMode mode, string reason) => EmitEvent(new MxEvent
{
OnAlarmProviderModeChanged = new OnAlarmProviderModeChangedEvent
{
Mode = mode,
Reason = reason,
At = Timestamp.FromDateTimeOffset(DateTimeOffset.UtcNow),
},
});
/// <summary>Waits until at least the requested number of update commands have arrived.</summary>
public async Task WaitForUpdateCountAsync(int count, TimeSpan timeout)
{
await WaitUntilAsync(() => UpdateCommands.Count >= count, timeout);
}
/// <summary>Completes once the monitor has issued its SubscribeAlarms command.</summary>
/// <param name="timeout">The maximum time to wait.</param>
/// <returns>A task that represents the asynchronous operation.</returns>
@@ -972,9 +1057,20 @@ public sealed class GatewayAlarmMonitorProviderModeTests
_subscribed.TrySetResult();
}
if (command.Command?.Kind == MxCommandKind.UpdateAlarmWatchList)
{
UpdateCommands.Add(command.Command.UpdateAlarmWatchList.Clone());
}
MxCommandReply reply = new()
{
ProtocolStatus = new ProtocolStatus { Code = ProtocolStatusCode.Ok },
ProtocolStatus = new ProtocolStatus
{
Code = command.Command?.Kind == MxCommandKind.UpdateAlarmWatchList
&& UpdateReplyCodes.TryDequeue(out ProtocolStatusCode updateCode)
? updateCode
: ProtocolStatusCode.Ok,
},
};
if (command.Command?.Kind == MxCommandKind.QueryActiveAlarms)
{
@@ -0,0 +1,100 @@
using Google.Protobuf;
using Google.Protobuf.Reflection;
using RetainedGalaxyReflection = ZB.MOM.WW.MxGateway.Contracts.Proto.Galaxy.GalaxyRepositoryReflection;
using ServedGalaxyReflection = ZB.MOM.WW.GalaxyRepository.Grpc.GalaxyRepositoryReflection;
namespace ZB.MOM.WW.MxGateway.Tests.Contracts;
public sealed class GalaxyContractParityTests
{
/// <summary>Guards the retained client proto against drift from the descriptor served by the gateway.</summary>
[Fact]
public void RetainedClientDescriptor_MatchesServedGatewayDescriptor()
{
FileDescriptorProto retained = Normalize(RetainedGalaxyReflection.Descriptor);
FileDescriptorProto served = Normalize(ServedGalaxyReflection.Descriptor);
AssertEquivalentDescriptors(retained, served);
}
/// <summary>Proves descriptor drift diagnostics name the first differing protobuf message.</summary>
[Fact]
public void DescriptorComparison_WhenMessageDiffers_ReportsDifferingMessage()
{
FileDescriptorProto retained = Normalize(RetainedGalaxyReflection.Descriptor);
FileDescriptorProto changed = retained.Clone();
DescriptorProto changedMessage = changed.MessageType.First(message => message.Field.Count > 0);
changedMessage.Field[0].Name += "_changed";
Xunit.Sdk.XunitException exception = Assert.Throws<Xunit.Sdk.XunitException>(
() => AssertEquivalentDescriptors(retained, changed));
Assert.Contains(changedMessage.Name, exception.Message, StringComparison.Ordinal);
}
private static FileDescriptorProto Normalize(FileDescriptor descriptor)
{
FileDescriptorProto proto = descriptor.ToProto();
proto.SourceCodeInfo = null;
proto.Name = string.Empty;
if (proto.Options is not null)
{
proto.Options.CsharpNamespace = string.Empty;
}
return proto;
}
private static void AssertEquivalentDescriptors(FileDescriptorProto expected, FileDescriptorProto actual)
{
byte[] expectedBytes = SerializeDeterministically(expected);
byte[] actualBytes = SerializeDeterministically(actual);
if (expectedBytes.AsSpan().SequenceEqual(actualBytes))
{
return;
}
throw new Xunit.Sdk.XunitException(
$"Galaxy Repository descriptors differ at {DescribeFirstDifference(expected, actual)}.");
}
private static string DescribeFirstDifference(FileDescriptorProto expected, FileDescriptorProto actual)
{
int comparableMessageCount = Math.Min(expected.MessageType.Count, actual.MessageType.Count);
for (int index = 0; index < comparableMessageCount; index++)
{
DescriptorProto expectedMessage = expected.MessageType[index];
DescriptorProto actualMessage = actual.MessageType[index];
if (!SerializeDeterministically(expectedMessage).AsSpan()
.SequenceEqual(SerializeDeterministically(actualMessage)))
{
string messageName = string.IsNullOrEmpty(expectedMessage.Name)
? actualMessage.Name
: expectedMessage.Name;
return $"message '{Qualify(expected.Package, messageName)}'";
}
}
if (expected.MessageType.Count != actual.MessageType.Count)
{
return "the top-level message list";
}
return "the normalized file descriptor";
}
private static string Qualify(string package, string symbol) =>
string.IsNullOrEmpty(package) ? symbol : package + "." + symbol;
private static byte[] SerializeDeterministically(IMessage message)
{
using MemoryStream stream = new();
using CodedOutputStream output = new(stream, leaveOpen: true)
{
Deterministic = true,
};
message.WriteTo(output);
output.Flush();
return stream.ToArray();
}
}
@@ -1507,6 +1507,52 @@ public sealed class ProtobufContractRoundTripTests
Assert.Equal(5, parsed.Failover.FailbackStableProbes);
}
/// <summary>
/// Verifies the live alarm watch-list replacement command and the
/// subscription's explicit empty-list fallback intent retain their
/// command kind, oneof arm, and payload across protobuf serialization.
/// </summary>
[Fact]
public void AlarmWatchListCommands_RoundTripFallbackIntentAndReplacementPayload()
{
var subscribe = new SubscribeAlarmsCommand
{
SubscriptionExpression = @"\\node\Galaxy!Area",
FallbackEnabled = true,
};
var parsedSubscribe = SubscribeAlarmsCommand.Parser.ParseFrom(subscribe.ToByteArray());
Assert.True(parsedSubscribe.FallbackEnabled);
Assert.Empty(parsedSubscribe.WatchList);
var original = new MxCommand
{
Kind = MxCommandKind.UpdateAlarmWatchList,
UpdateAlarmWatchList = new UpdateAlarmWatchListCommand
{
WatchList =
{
new AlarmSubtagTarget
{
AlarmFullReference = "Galaxy!Area.Tank01.Level.HiHi",
SourceObjectReference = "Tank01",
ActiveSubtag = "Tank01.Level.HiHi.InAlarm",
AckedSubtag = "Tank01.Level.HiHi.Acked",
AckCommentSubtag = "Tank01.Level.HiHi.AckMsg",
PrioritySubtag = "Tank01.Level.HiHi.Priority",
},
},
},
};
var parsed = MxCommand.Parser.ParseFrom(original.ToByteArray());
Assert.Equal(original, parsed);
Assert.Equal(MxCommandKind.UpdateAlarmWatchList, parsed.Kind);
Assert.Equal(MxCommand.PayloadOneofCase.UpdateAlarmWatchList, parsed.PayloadCase);
Assert.Single(parsed.UpdateAlarmWatchList.WatchList);
}
/// <summary>
/// Verifies that an <see cref="MxEvent"/> carrying an
/// <see cref="OnAlarmProviderModeChangedEvent"/> body (the
@@ -1147,6 +1147,41 @@ public sealed class WorkerClientTests
Assert.Equal(0, metrics.GetSnapshot().WorkerEventQueueDepth);
}
/// <summary>
/// GWC-33: a faulted client can remain retained for diagnostics, but its abandoned event
/// backlog must stop contributing to the gateway-wide queue-depth gauge as soon as the
/// client enters a terminal state rather than waiting for disposal.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task WorkerEventQueueDepthGauge_WhenClientFaults_StopsCountingRetainedClient()
{
using GatewayMetrics metrics = new();
await using PipePair pipePair = await PipePair.CreateAsync();
await using WorkerClient client = CreateClient(
pipePair,
new WorkerClientOptions
{
EventChannelCapacity = 4,
EventChannelFullModeTimeout = TimeSpan.FromMinutes(5),
HeartbeatGrace = TimeSpan.FromSeconds(30),
HeartbeatCheckInterval = TimeSpan.FromSeconds(30),
},
metrics: metrics);
await CompleteHandshakeAsync(client, pipePair);
await pipePair.WriteAsync(CreateEventEnvelope(sequence: 1, MxEventFamily.OnDataChange));
await WaitUntilAsync(
() => metrics.GetSnapshot().WorkerEventQueueDepth == 1,
TestTimeout);
await pipePair.WriteAsync(CreateWorkerFaultEnvelope("scripted retained-client fault"));
await WaitUntilAsync(() => client.State == WorkerClientState.Faulted, TestTimeout);
Assert.Equal(WorkerClientState.Faulted, client.State);
Assert.Equal(0, metrics.GetSnapshot().WorkerEventQueueDepth);
}
private static WorkerClient CreateClient(
PipePair pipePair,
WorkerClientOptions? options = null,
@@ -14,6 +14,16 @@ public sealed class WorkerProcessLauncherTests
private const string PipeName = "mxaccess-gateway-123-session-1";
private const string Nonce = "super-secret-nonce";
private static readonly string[] RequiredLaunchEnvironmentVariables =
[
WorkerLaunchEnvironment.WorkerNonce,
WorkerLaunchEnvironment.PipeConnectAttemptTimeoutMilliseconds,
WorkerLaunchEnvironment.WriteCompletionWaitMilliseconds,
WorkerLaunchEnvironment.EventQueueCapacity,
WorkerLaunchEnvironment.AlarmPollIntervalMilliseconds,
WorkerLaunchEnvironment.MaxAlarmsPerFetch,
];
/// <summary>Verifies that a valid worker executable starts with correct bootstrap arguments and nonce environment variable.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
@@ -38,33 +48,38 @@ public sealed class WorkerProcessLauncherTests
Assert.Equal(
["--session-id", SessionId, "--pipe-name", PipeName, "--protocol-version", "1"],
processFactory.LastStartInfo.ArgumentList);
Assert.Equal(Nonce, processFactory.LastStartInfo.Environment[WorkerProcessLauncher.WorkerNonceEnvironmentVariableName]);
Assert.Equal(Nonce, processFactory.LastStartInfo.Environment[WorkerLaunchEnvironment.WorkerNonce]);
Assert.Equal(
"2000",
processFactory.LastStartInfo.Environment[
WorkerProcessLauncher.WorkerPipeConnectAttemptTimeoutEnvironmentVariableName]);
WorkerLaunchEnvironment.PipeConnectAttemptTimeoutMilliseconds]);
Assert.Equal(
"1500",
processFactory.LastStartInfo.Environment[
WorkerProcessLauncher.WorkerWriteCompletionWaitEnvironmentVariableName]);
WorkerLaunchEnvironment.WriteCompletionWaitMilliseconds]);
// The worker sizes its outbound event queue from the launch environment;
// the queue has no drop policy, so this capacity is the session's burst
// headroom rather than a throttle.
Assert.Equal(
"10000",
processFactory.LastStartInfo.Environment[
WorkerProcessLauncher.WorkerEventQueueCapacityEnvironmentVariableName]);
WorkerLaunchEnvironment.EventQueueCapacity]);
// MxGateway:Alarms defaults reach the worker's alarm poll loop and its
// GetXmlCurrentAlarms2 cap (which is also its truncation threshold)
// through the launch environment, not the command line.
Assert.Equal(
"500",
processFactory.LastStartInfo.Environment[
WorkerProcessLauncher.WorkerAlarmPollIntervalEnvironmentVariableName]);
WorkerLaunchEnvironment.AlarmPollIntervalMilliseconds]);
Assert.Equal(
"1024",
processFactory.LastStartInfo.Environment[
WorkerProcessLauncher.WorkerMaxAlarmsPerFetchEnvironmentVariableName]);
WorkerLaunchEnvironment.MaxAlarmsPerFetch]);
Assert.All(
RequiredLaunchEnvironmentVariables,
variable => Assert.True(
processFactory.LastStartInfo.Environment.ContainsKey(variable),
$"Worker launch environment is missing required variable '{variable}'."));
Assert.DoesNotContain(Nonce, handle.CommandLine.ToString(), StringComparison.Ordinal);
Assert.DoesNotContain(Nonce, string.Join(" ", handle.CommandLine.Arguments), StringComparison.Ordinal);
Assert.False(pipeReservation.DisposeCalled);
@@ -94,7 +109,7 @@ public sealed class WorkerProcessLauncherTests
Assert.Equal(
"65536",
processFactory.LastStartInfo.Environment[
WorkerProcessLauncher.WorkerEventQueueCapacityEnvironmentVariableName]);
WorkerLaunchEnvironment.EventQueueCapacity]);
}
/// <summary>Verifies that a failed startup probe kills and disposes the worker process.</summary>
@@ -0,0 +1,53 @@
namespace ZB.MOM.WW.MxGateway.Tests.ProjectStructure;
public sealed class WindevCiContractTests
{
[Fact]
public void LiveMode_RunsWindowsTiersInDocumentedOrder()
{
DirectoryInfo repositoryRoot = FindRepositoryRoot();
string script = File.ReadAllText(
Path.Combine(repositoryRoot.FullName, "scripts", "ci", "windev-worker-ci.ps1"));
string[] orderedSteps =
[
"Invoke-Native 'x86 Worker build'",
"Invoke-Native 'x86 Worker.Tests'",
"Invoke-Native 'full slnx build'",
"Invoke-Native 'gateway tests on Windows'",
"Invoke-Native 'live-MXAccess smoke'",
];
int previousIndex = -1;
foreach (string step in orderedSteps)
{
int currentIndex = script.IndexOf(step, StringComparison.Ordinal);
Assert.True(currentIndex >= 0, $"Missing windev live-mode step: {step}");
Assert.True(currentIndex > previousIndex, $"Windev live-mode step is out of order: {step}");
previousIndex = currentIndex;
}
Assert.Contains(
"dotnet test $gatewayTests -c Release --no-build",
script,
StringComparison.Ordinal);
}
private static DirectoryInfo FindRepositoryRoot()
{
DirectoryInfo? current = new(AppContext.BaseDirectory);
while (current is not null)
{
if (File.Exists(Path.Combine(current.FullName, "CLAUDE.md"))
&& Directory.Exists(Path.Combine(current.FullName, "src"))
&& Directory.Exists(Path.Combine(current.FullName, "scripts")))
{
return current;
}
current = current.Parent;
}
throw new InvalidOperationException("Could not locate the repository root from the test output directory.");
}
}
@@ -101,6 +101,37 @@ public sealed class ApiKeyFailureLimiterTests
limiter.Check(new ApiKeyThrottlePartition("ipv4:10.0.9.9:1", "other-key")));
}
/// <summary>
/// SEC-38: a valid key id still feeds and consults its cross-peer aggregate when each
/// attacking peer has exhausted its 32 dedicated key-id partitions and therefore records
/// the peer layer through the fallback bucket.
/// </summary>
[Fact]
public void AggregateLayer_TripsForValidKeyIdCollapsedToPeerFallback()
{
ManualTimeProvider clock = new(DateTimeOffset.UnixEpoch);
ApiKeyFailureLimiter limiter = CreateLimiter(
clock,
limit: 100,
aggregateLimit: 3,
maxPartitions: 4096);
for (int peerIndex = 0; peerIndex < 3; peerIndex++)
{
string peer = $"ipv4:10.0.0.{peerIndex + 1}:1";
for (int keyIndex = 0; keyIndex < ApiKeyFailureLimiter.MaxKeyIdPartitionsPerPeer; keyIndex++)
{
limiter.RecordFailure(new ApiKeyThrottlePartition(peer, $"prefill-{peerIndex}-{keyIndex}"));
}
limiter.RecordFailure(new ApiKeyThrottlePartition(peer, "victim"));
}
Assert.Equal(
ApiKeyThrottleDecision.ThrottledByAggregate,
limiter.Check(new ApiKeyThrottlePartition("ipv4:10.0.9.9:1", "victim")));
}
/// <summary>
/// A throttled partition is a valve, not a wall: exactly one request per probe interval is
/// admitted to the real verifier, so the holder of the correct secret can always get through.
@@ -340,6 +371,29 @@ public sealed class ApiKeyFailureLimiterTests
Assert.Equal(ApiKeyThrottleDecision.ThrottledByPeer, limiter.Check(overCap));
}
/// <summary>
/// SEC-39: a request with no effective key id does not own the peer's shared fallback
/// partition. Resetting after such a request must leave other fallback failures tracked.
/// </summary>
[Fact]
public void Reset_WithNullEffectiveKeyId_DoesNotClearSharedFallbackPartition()
{
ManualTimeProvider clock = new(DateTimeOffset.UnixEpoch);
ApiKeyFailureLimiter limiter = CreateLimiter(clock, limit: 3, aggregateLimit: 100);
ApiKeyThrottlePartition fallback = new("ipv4:10.0.0.1:1", KeyId: null);
RecordFailures(limiter, fallback, 3);
Assert.Equal(
ApiKeyThrottleDecision.ThrottledByPeer,
limiter.Check(fallback, out ApiKeyFailureLimiter.PartitionResolution resolution));
Assert.Null(resolution.EffectiveKeyId);
limiter.Reset(fallback, resolution);
Assert.True(limiter.IsTracked(fallback));
Assert.Equal(ApiKeyThrottleDecision.ThrottledByPeer, limiter.Check(fallback));
}
/// <summary>
/// SEC-32: a spray of unique junk partitions (junk tokens resolve to the sender's fallback
/// partition, one per address) must not evict a partition that is currently throttled — the
@@ -137,7 +137,7 @@ public sealed class WorkerApplicationTests
private static MemoryWorkerEnvironment CreateEnvironment(string nonce)
{
MemoryWorkerEnvironment environment = new();
environment.Set(WorkerOptions.NonceEnvironmentVariableName, nonce);
environment.Set(WorkerLaunchEnvironment.WorkerNonce, nonce);
return environment;
}
@@ -115,7 +115,7 @@ public sealed class WorkerOptionsParserTests
private static MemoryWorkerEnvironment CreateEnvironment(string nonce)
{
MemoryWorkerEnvironment environment = new();
environment.Set(WorkerOptions.NonceEnvironmentVariableName, nonce);
environment.Set(WorkerLaunchEnvironment.WorkerNonce, nonce);
return environment;
}
}
@@ -91,6 +91,27 @@ public sealed class AlarmCommandExecutorTests
Assert.Equal(ProtocolStatusCode.InvalidRequest, reply.ProtocolStatus.Code);
}
/// <summary>Verifies that a live watch-list replacement routes to the alarm handler on the executor thread.</summary>
[Fact]
public void UpdateAlarmWatchList_WithHandler_RoutesToHandlerAndReturnsOk()
{
FakeAlarmHandler handler = new FakeAlarmHandler();
MxAccessCommandExecutor executor = NewExecutor(handler);
var target = new AlarmSubtagTarget { AlarmFullReference = "Galaxy!Area.Tank01.Alarm" };
MxCommandReply reply = executor.Execute(new StaCommand(
SessionId,
CorrelationId,
new MxCommand
{
Kind = MxCommandKind.UpdateAlarmWatchList,
UpdateAlarmWatchList = new UpdateAlarmWatchListCommand { WatchList = { target } },
}));
Assert.Equal(ProtocolStatusCode.Ok, reply.ProtocolStatus.Code);
Assert.Equal(target, Assert.Single(handler.LastWatchList!));
}
/// <summary>Verifies that acknowledge routes native status into hresult and payload.</summary>
[Fact]
public void AcknowledgeAlarm_WithHandler_RoutesNativeStatusIntoHresultAndPayload()
@@ -447,6 +468,15 @@ public sealed class AlarmCommandExecutorTests
LastSessionId = sessionId;
}
/// <summary>Gets the most recent replacement watch list.</summary>
public IReadOnlyList<AlarmSubtagTarget>? LastWatchList { get; private set; }
/// <inheritdoc />
public void UpdateWatchList(IReadOnlyList<AlarmSubtagTarget> watchList)
{
LastWatchList = watchList;
}
/// <inheritdoc />
public void Unsubscribe()
{
@@ -287,8 +287,12 @@ public sealed class AlarmCommandHandlerTests
handler.PollOnce();
Assert.Equal(5, guardInvocations);
handler.Unsubscribe();
Assert.Throws<InvalidOperationException>(
() => handler.UpdateWatchList(Array.Empty<AlarmSubtagTarget>()));
Assert.Equal(6, guardInvocations);
handler.Unsubscribe();
Assert.Equal(7, guardInvocations);
}
/// <summary>
@@ -324,6 +328,8 @@ public sealed class AlarmCommandHandlerTests
() => handler.AcknowledgeByName("", "", "", "", "", "", "", ""));
Assert.Throws<InvalidOperationException>(() => handler.QueryActive(null, out _));
Assert.Throws<InvalidOperationException>(() => handler.PollOnce());
Assert.Throws<InvalidOperationException>(
() => handler.UpdateWatchList(Array.Empty<AlarmSubtagTarget>()));
Assert.Throws<InvalidOperationException>(() => handler.Unsubscribe());
}
@@ -412,6 +418,114 @@ public sealed class AlarmCommandHandlerTests
Assert.Equal(AlarmProviderMode.Subtag, evt.OnAlarmProviderModeChanged.Mode);
}
/// <summary>An explicitly fallback-capable auto subscription builds a replaceable standby even when discovery is initially empty.</summary>
[Fact]
public void Subscribe_AutoModeWithEmptyWatchListAndFallbackEnabled_BuildsReplaceableStandby()
{
FakeConsumer primary = new FakeConsumer { AcknowledgeReturn = 11 };
FakeConsumer initialStandby = new FakeConsumer { AcknowledgeReturn = 22 };
FakeConsumer replacement = new FakeConsumer { AcknowledgeReturn = 33 };
int buildCount = 0;
AlarmCommandHandler handler = new AlarmCommandHandler(
new MxAccessEventQueue(),
() => primary,
threadAffinityCheck: null,
comFactory: null,
standbyFactory: _ => ++buildCount == 1 ? initialStandby : replacement);
handler.Subscribe(new SubscribeAlarmsCommand
{
SubscriptionExpression = @"\\HOST\Galaxy!Area",
FallbackEnabled = true,
}, "s1");
handler.UpdateWatchList(new[] { NewTarget("Galaxy!Area.Tank01.Alarm") });
Assert.Equal(2, buildCount);
Assert.Equal(@"\\HOST\Galaxy!Area", replacement.LastSubscription);
Assert.True(initialStandby.Disposed);
Assert.Equal(11, handler.Acknowledge(Guid.Empty, "", "", "", "", ""));
}
/// <summary>A replacement is rejected when no fallback-capable subscription is active.</summary>
[Fact]
public void UpdateWatchList_WithoutFallbackCapableSubscription_Throws()
{
AlarmCommandHandler handler = new AlarmCommandHandler(
new MxAccessEventQueue(),
() => new FakeConsumer());
Assert.Throws<InvalidOperationException>(
() => handler.UpdateWatchList(new[] { NewTarget("Galaxy!Area.Tank01.Alarm") }));
handler.Subscribe(new SubscribeAlarmsCommand
{
SubscriptionExpression = @"\\HOST\Galaxy!Area",
ForcedMode = AlarmProviderMode.Alarmmgr,
}, "s1");
Assert.Throws<InvalidOperationException>(
() => handler.UpdateWatchList(new[] { NewTarget("Galaxy!Area.Tank02.Alarm") }));
}
/// <summary>A replacement setup failure leaves the old standby and primary mode intact.</summary>
[Fact]
public void UpdateWatchList_WhenReplacementSubscribeFails_PreservesOldStandbyAndPrimaryMode()
{
FakeConsumer primary = new FakeConsumer { AcknowledgeReturn = 11 };
FakeConsumer oldStandby = new FakeConsumer { AcknowledgeReturn = 22 };
FakeConsumer failedReplacement = new FakeConsumer { ThrowOnSubscribe = true, AcknowledgeReturn = 33 };
int buildCount = 0;
AlarmCommandHandler handler = new AlarmCommandHandler(
new MxAccessEventQueue(),
() => primary,
null,
null,
_ => ++buildCount == 1 ? oldStandby : failedReplacement);
SubscribeAlarmsCommand subscribe = new SubscribeAlarmsCommand
{
SubscriptionExpression = @"\\HOST\Galaxy!Area",
FallbackEnabled = true,
};
subscribe.WatchList.Add(NewTarget("Galaxy!Area.Tank01.Alarm"));
handler.Subscribe(subscribe, "s1");
Assert.Throws<InvalidOperationException>(
() => handler.UpdateWatchList(new[] { NewTarget("Galaxy!Area.Tank02.Alarm") }));
Assert.False(oldStandby.Disposed);
Assert.True(failedReplacement.Disposed);
Assert.Equal(11, handler.Acknowledge(Guid.Empty, "", "", "", "", ""));
}
/// <summary>A successful replacement while failed over keeps subtag mode active and delegates to the new standby.</summary>
[Fact]
public void UpdateWatchList_WhileSubtagActive_PreservesModeAndUsesReplacement()
{
FakeConsumer primary = new FakeConsumer { ThrowOnSubscribe = true, AcknowledgeReturn = 11 };
FakeConsumer oldStandby = new FakeConsumer { AcknowledgeReturn = 22 };
FakeConsumer replacement = new FakeConsumer { AcknowledgeReturn = 33 };
int buildCount = 0;
AlarmCommandHandler handler = new AlarmCommandHandler(
new MxAccessEventQueue(),
() => primary,
null,
null,
_ => ++buildCount == 1 ? oldStandby : replacement);
SubscribeAlarmsCommand subscribe = new SubscribeAlarmsCommand
{
SubscriptionExpression = @"\\HOST\Galaxy!Area",
FallbackEnabled = true,
Failover = new AlarmFailoverConfig { ConsecutiveFailureThreshold = 1 },
};
subscribe.WatchList.Add(NewTarget("Galaxy!Area.Tank01.Alarm"));
handler.Subscribe(subscribe, "s1");
handler.UpdateWatchList(new[] { NewTarget("Galaxy!Area.Tank02.Alarm") });
Assert.True(oldStandby.Disposed);
Assert.Equal(33, handler.Acknowledge(Guid.Empty, "", "", "", "", ""));
}
/// <summary>
/// A non-failover subscribe (alarmmgr-only) never enqueues a
/// provider-mode-changed event, and a subsequent Unsubscribe detaches
@@ -467,6 +581,14 @@ public sealed class AlarmCommandHandlerTests
};
}
private static AlarmSubtagTarget NewTarget(string alarmFullReference) => new AlarmSubtagTarget
{
AlarmFullReference = alarmFullReference,
ActiveSubtag = alarmFullReference + ".InAlarm",
AckedSubtag = alarmFullReference + ".Acked",
AckCommentSubtag = alarmFullReference + ".AckMsg",
};
private sealed class FakeConsumer : IMxAccessAlarmConsumer
{
#pragma warning disable CS0067 // Event never invoked — fake; AlarmCommandHandler tests don't drive transitions.
@@ -4,6 +4,7 @@ using System.Diagnostics;
using System.Threading;
using System.Threading.Tasks;
using Google.Protobuf.WellKnownTypes;
using ZB.MOM.WW.MxGateway.Contracts;
using ZB.MOM.WW.MxGateway.Contracts.Proto;
using ZB.MOM.WW.MxGateway.Worker.MxAccess;
@@ -609,11 +610,11 @@ public sealed class MxAccessEventQueueTests
int expected)
{
string? original = Environment.GetEnvironmentVariable(
MxAccessEventQueue.CapacityEnvironmentVariableName);
WorkerLaunchEnvironment.EventQueueCapacity);
try
{
Environment.SetEnvironmentVariable(
MxAccessEventQueue.CapacityEnvironmentVariableName,
WorkerLaunchEnvironment.EventQueueCapacity,
environmentValue);
Assert.Equal(expected, MxAccessEventQueue.ResolveCapacity());
@@ -621,7 +622,7 @@ public sealed class MxAccessEventQueueTests
finally
{
Environment.SetEnvironmentVariable(
MxAccessEventQueue.CapacityEnvironmentVariableName,
WorkerLaunchEnvironment.EventQueueCapacity,
original);
}
}
@@ -635,11 +636,11 @@ public sealed class MxAccessEventQueueTests
public void Constructor_WithResolvedCapacity_UsesEnvironmentValue()
{
string? original = Environment.GetEnvironmentVariable(
MxAccessEventQueue.CapacityEnvironmentVariableName);
WorkerLaunchEnvironment.EventQueueCapacity);
try
{
Environment.SetEnvironmentVariable(
MxAccessEventQueue.CapacityEnvironmentVariableName,
WorkerLaunchEnvironment.EventQueueCapacity,
"2500");
MxAccessEventQueue queue = new(MxAccessEventQueue.ResolveCapacity());
@@ -649,7 +650,7 @@ public sealed class MxAccessEventQueueTests
finally
{
Environment.SetEnvironmentVariable(
MxAccessEventQueue.CapacityEnvironmentVariableName,
WorkerLaunchEnvironment.EventQueueCapacity,
original);
}
}
@@ -3,6 +3,7 @@ using System.Collections.Generic;
using System.Runtime.InteropServices;
using System.Threading;
using System.Threading.Tasks;
using ZB.MOM.WW.MxGateway.Contracts;
using ZB.MOM.WW.MxGateway.Contracts.Proto;
using ZB.MOM.WW.MxGateway.Worker.MxAccess;
using ZB.MOM.WW.MxGateway.Worker.Sta;
@@ -35,11 +36,11 @@ public sealed class MxAccessStaSessionTests
public void ResolveWriteCompletionTimeout_ParsesEnvironmentValue(string? rawValue, int? expectedMilliseconds)
{
string? original = Environment.GetEnvironmentVariable(
MxAccessStaSession.WriteCompletionWaitEnvironmentVariableName);
WorkerLaunchEnvironment.WriteCompletionWaitMilliseconds);
try
{
Environment.SetEnvironmentVariable(
MxAccessStaSession.WriteCompletionWaitEnvironmentVariableName,
WorkerLaunchEnvironment.WriteCompletionWaitMilliseconds,
rawValue);
TimeSpan resolved = MxAccessStaSession.ResolveWriteCompletionTimeout();
@@ -52,7 +53,7 @@ public sealed class MxAccessStaSessionTests
finally
{
Environment.SetEnvironmentVariable(
MxAccessStaSession.WriteCompletionWaitEnvironmentVariableName,
WorkerLaunchEnvironment.WriteCompletionWaitMilliseconds,
original);
}
}
@@ -83,11 +84,11 @@ public sealed class MxAccessStaSessionTests
int expectedMilliseconds)
{
string? original = Environment.GetEnvironmentVariable(
MxAccessStaSession.AlarmPollIntervalEnvironmentVariableName);
WorkerLaunchEnvironment.AlarmPollIntervalMilliseconds);
try
{
Environment.SetEnvironmentVariable(
MxAccessStaSession.AlarmPollIntervalEnvironmentVariableName,
WorkerLaunchEnvironment.AlarmPollIntervalMilliseconds,
environmentValue);
Assert.Equal(
@@ -97,7 +98,7 @@ public sealed class MxAccessStaSessionTests
finally
{
Environment.SetEnvironmentVariable(
MxAccessStaSession.AlarmPollIntervalEnvironmentVariableName,
WorkerLaunchEnvironment.AlarmPollIntervalMilliseconds,
original);
}
}
@@ -650,6 +651,11 @@ public sealed class MxAccessStaSessionTests
LastSubscription = command.SubscriptionExpression;
}
/// <inheritdoc />
public void UpdateWatchList(IReadOnlyList<AlarmSubtagTarget> watchList)
{
}
/// <inheritdoc />
public void Unsubscribe()
{
@@ -3,6 +3,7 @@ using System.Collections.Generic;
using System.Linq;
using System.Reflection;
using System.Threading;
using ZB.MOM.WW.MxGateway.Contracts;
using ZB.MOM.WW.MxGateway.Worker.MxAccess;
namespace ZB.MOM.WW.MxGateway.Worker.Tests.MxAccess;
@@ -506,11 +507,11 @@ public sealed class WnWrapAlarmConsumerXmlTests
int expected)
{
string? original = Environment.GetEnvironmentVariable(
WnWrapAlarmConsumer.MaxAlarmsPerFetchEnvironmentVariableName);
WorkerLaunchEnvironment.MaxAlarmsPerFetch);
try
{
Environment.SetEnvironmentVariable(
WnWrapAlarmConsumer.MaxAlarmsPerFetchEnvironmentVariableName,
WorkerLaunchEnvironment.MaxAlarmsPerFetch,
environmentValue);
Assert.Equal(expected, WnWrapAlarmConsumer.ResolveMaxAlarmsPerFetch());
@@ -518,7 +519,7 @@ public sealed class WnWrapAlarmConsumerXmlTests
finally
{
Environment.SetEnvironmentVariable(
WnWrapAlarmConsumer.MaxAlarmsPerFetchEnvironmentVariableName,
WorkerLaunchEnvironment.MaxAlarmsPerFetch,
original);
}
}
@@ -1,10 +1,14 @@
namespace ZB.MOM.WW.MxGateway.Worker.Bootstrap;
using System;
using ZB.MOM.WW.MxGateway.Contracts;
/// <summary>Worker bootstrap options passed via environment variables and named pipes.</summary>
public sealed class WorkerOptions
{
/// <summary>Environment variable name for the worker nonce.</summary>
public const string NonceEnvironmentVariableName = "MXGATEWAY_WORKER_NONCE";
[Obsolete("Use WorkerLaunchEnvironment.WorkerNonce.")]
public const string NonceEnvironmentVariableName = WorkerLaunchEnvironment.WorkerNonce;
/// <summary>Initializes worker options from a bootstrap handshake.</summary>
/// <param name="sessionId">Identifier of the session.</param>
@@ -75,7 +75,7 @@ public sealed class WorkerOptionsParser
[$"Unsupported protocol version '{protocolVersionText}'."]);
}
string? nonce = _environment.GetEnvironmentVariable(WorkerOptions.NonceEnvironmentVariableName);
string? nonce = _environment.GetEnvironmentVariable(WorkerLaunchEnvironment.WorkerNonce);
if (string.IsNullOrWhiteSpace(nonce))
{
@@ -4,6 +4,7 @@ using System.IO;
using System.IO.Pipes;
using System.Threading;
using System.Threading.Tasks;
using ZB.MOM.WW.MxGateway.Contracts;
using ZB.MOM.WW.MxGateway.Worker.Bootstrap;
using Polly;
using Polly.Retry;
@@ -22,8 +23,9 @@ public sealed class WorkerPipeClient : IWorkerPipeClient
public const int DefaultConnectAttemptTimeoutMilliseconds = 2000;
/// <summary>Environment variable for overriding the per-attempt connection timeout.</summary>
[Obsolete("Use WorkerLaunchEnvironment.PipeConnectAttemptTimeoutMilliseconds.")]
public const string ConnectAttemptTimeoutEnvironmentVariableName =
"MXGATEWAY_WORKER_PIPE_CONNECT_ATTEMPT_TIMEOUT_MS";
WorkerLaunchEnvironment.PipeConnectAttemptTimeoutMilliseconds;
private readonly int _connectTimeoutMilliseconds;
private readonly int _connectAttemptTimeoutMilliseconds;
@@ -255,7 +257,8 @@ public sealed class WorkerPipeClient : IWorkerPipeClient
private static int ResolveDefaultConnectAttemptTimeoutMilliseconds()
{
string? configuredValue = Environment.GetEnvironmentVariable(ConnectAttemptTimeoutEnvironmentVariableName);
string? configuredValue = Environment.GetEnvironmentVariable(
WorkerLaunchEnvironment.PipeConnectAttemptTimeoutMilliseconds);
return int.TryParse(configuredValue, out int milliseconds) && milliseconds > 0
? milliseconds
: DefaultConnectAttemptTimeoutMilliseconds;
@@ -45,6 +45,7 @@ public sealed class AlarmCommandHandler : IAlarmCommandHandler
private FailoverAlarmConsumer? failoverConsumer;
private EventHandler<AlarmProviderModeChange>? providerModeChangedHandler;
private string subscribeSessionId = string.Empty;
private string subscriptionExpression = string.Empty;
private bool disposed;
/// <summary>Initializes a new alarm command handler with the given event queue.</summary>
@@ -153,6 +154,7 @@ public sealed class AlarmCommandHandler : IAlarmCommandHandler
}
subscribeSessionId = sessionId ?? string.Empty;
subscriptionExpression = command.SubscriptionExpression ?? string.Empty;
IMxAccessAlarmConsumer consumer = BuildConsumer(command)
?? throw new InvalidOperationException("Alarm consumer factory returned null.");
@@ -171,7 +173,7 @@ public sealed class AlarmCommandHandler : IAlarmCommandHandler
dispatcher = new AlarmDispatcher(consumer, sink, subscribeSessionId);
try
{
dispatcher.Subscribe(command.SubscriptionExpression ?? string.Empty);
dispatcher.Subscribe(subscriptionExpression);
}
catch
{
@@ -212,7 +214,8 @@ public sealed class AlarmCommandHandler : IAlarmCommandHandler
return BuildStandby(watchList);
}
if (command.ForcedMode == AlarmProviderMode.Unspecified && watchList.Count > 0)
if (command.ForcedMode == AlarmProviderMode.Unspecified
&& (command.FallbackEnabled || watchList.Count > 0))
{
IMxAccessAlarmConsumer primary = consumerFactory()
?? throw new InvalidOperationException("Alarm consumer factory returned null.");
@@ -230,6 +233,26 @@ public sealed class AlarmCommandHandler : IAlarmCommandHandler
?? throw new InvalidOperationException("Alarm consumer factory returned null.");
}
/// <inheritdoc />
public void UpdateWatchList(IReadOnlyList<AlarmSubtagTarget> watchList)
{
if (disposed) throw new ObjectDisposedException(nameof(AlarmCommandHandler));
if (watchList is null) throw new ArgumentNullException(nameof(watchList));
threadAffinityCheck?.Invoke();
lock (syncRoot)
{
if (dispatcher is null || failoverConsumer is null)
{
throw new InvalidOperationException(
"UpdateAlarmWatchList requires an active fallback-capable alarm subscription.");
}
IMxAccessAlarmConsumer replacement = BuildStandby(watchList);
failoverConsumer.ReplaceStandby(replacement, subscriptionExpression);
}
}
private IMxAccessAlarmConsumer BuildStandby(IReadOnlyList<AlarmSubtagTarget> watchList)
{
if (standbyFactory is not null)
@@ -276,6 +299,7 @@ public sealed class AlarmCommandHandler : IAlarmCommandHandler
DetachProviderModeChanged();
toDispose = dispatcher;
dispatcher = null;
subscriptionExpression = string.Empty;
}
toDispose?.Dispose();
}
@@ -55,7 +55,7 @@ public sealed class FailoverAlarmConsumer : IMxAccessAlarmConsumer
}
private readonly IMxAccessAlarmConsumer primary;
private readonly IMxAccessAlarmConsumer standby;
private IMxAccessAlarmConsumer standby;
private readonly FailoverSettings settings;
private Active active = Active.Primary;
@@ -103,6 +103,36 @@ public sealed class FailoverAlarmConsumer : IMxAccessAlarmConsumer
/// </summary>
public AlarmProviderMode Mode => mode;
/// <summary>
/// Arms and attaches a replacement standby before atomically swapping
/// it into the current provider mode. Setup failure disposes only the
/// replacement and leaves the existing standby untouched.
/// </summary>
/// <param name="replacement">The newly built standby consumer.</param>
/// <param name="subscription">The retained subscription expression.</param>
public void ReplaceStandby(IMxAccessAlarmConsumer replacement, string subscription)
{
if (disposed) throw new ObjectDisposedException(nameof(FailoverAlarmConsumer));
if (replacement is null) throw new ArgumentNullException(nameof(replacement));
try
{
replacement.Subscribe(subscription ?? string.Empty);
replacement.AlarmTransitionEmitted += OnChildTransition;
}
catch
{
try { replacement.Dispose(); } catch { /* preserve setup failure */ }
throw;
}
IMxAccessAlarmConsumer previous = standby;
standby = replacement;
try { previous.AlarmTransitionEmitted -= OnChildTransition; } catch { /* best effort */ }
try { previous.Dispose(); } catch { /* replacement is already active */ }
}
/// <inheritdoc />
public void Subscribe(string subscription)
{
@@ -7,7 +7,8 @@ namespace ZB.MOM.WW.MxGateway.Worker.MxAccess;
/// <summary>
/// Per-session interface routing the worker's alarm IPC commands —
/// <c>SubscribeAlarmsCommand</c>, <c>AcknowledgeAlarmCommand</c>,
/// <c>QueryActiveAlarmsCommand</c>, <c>UnsubscribeAlarmsCommand</c> —
/// <c>UpdateAlarmWatchListCommand</c>, <c>QueryActiveAlarmsCommand</c>,
/// <c>UnsubscribeAlarmsCommand</c> —
/// to the underlying <see cref="AlarmDispatcher"/>. Production binding
/// is <see cref="AlarmCommandHandler"/>; tests substitute a fake.
/// </summary>
@@ -24,6 +25,13 @@ public interface IAlarmCommandHandler : IDisposable
/// <param name="sessionId">The session identifier.</param>
void Subscribe(SubscribeAlarmsCommand command, string sessionId);
/// <summary>
/// Replace the active auto-failover subscription's subtag standby
/// targets without restarting its primary provider.
/// </summary>
/// <param name="watchList">The complete replacement watch list.</param>
void UpdateWatchList(IReadOnlyList<AlarmSubtagTarget> watchList);
/// <summary>Tear down the active subscription. No-op if not subscribed.</summary>
void Unsubscribe();
@@ -158,6 +158,7 @@ public sealed class MxAccessCommandExecutor : IStaCommandExecutor
MxCommandKind.WriteSecured2Bulk => ExecuteWriteSecured2Bulk(command),
MxCommandKind.ReadBulk => ExecuteReadBulk(command),
MxCommandKind.SubscribeAlarms => ExecuteSubscribeAlarms(command),
MxCommandKind.UpdateAlarmWatchList => ExecuteUpdateAlarmWatchList(command),
MxCommandKind.UnsubscribeAlarms => ExecuteUnsubscribeAlarms(command),
MxCommandKind.AcknowledgeAlarm => ExecuteAcknowledgeAlarm(command),
MxCommandKind.AcknowledgeAlarmByName => ExecuteAcknowledgeAlarmByName(command),
@@ -866,6 +867,30 @@ public sealed class MxAccessCommandExecutor : IStaCommandExecutor
}
}
private MxCommandReply ExecuteUpdateAlarmWatchList(StaCommand command)
{
if (command.Command.PayloadCase != MxCommand.PayloadOneofCase.UpdateAlarmWatchList)
{
return CreateInvalidRequestReply(command, "UpdateAlarmWatchList command payload is required.");
}
if (alarmCommandHandler is null)
{
return CreateInvalidRequestReply(
command,
"UpdateAlarmWatchList requires an alarm command handler; the worker was constructed without one.");
}
try
{
alarmCommandHandler.UpdateWatchList(command.Command.UpdateAlarmWatchList.WatchList);
return CreateOkReply(command);
}
catch (Exception ex)
{
return CreateAlarmFailureReply(command, ex);
}
}
private MxCommandReply ExecuteAcknowledgeAlarm(StaCommand command)
{
if (command.Command.PayloadCase != MxCommand.PayloadOneofCase.AcknowledgeAlarmCommand)
@@ -5,6 +5,7 @@ using System.Threading;
using System.Threading.Tasks;
using Google.Protobuf;
using Google.Protobuf.WellKnownTypes;
using ZB.MOM.WW.MxGateway.Contracts;
using ZB.MOM.WW.MxGateway.Contracts.Proto;
namespace ZB.MOM.WW.MxGateway.Worker.MxAccess;
@@ -45,8 +46,6 @@ public sealed class MxAccessEventQueue
/// <see cref="DefaultCapacity"/> — a bad environment value must never
/// keep the session from starting.
/// </summary>
internal const string CapacityEnvironmentVariableName = "MXGATEWAY_EVENT_QUEUE_CAPACITY";
/// <summary>
/// Floor on the resolved capacity. Mirrors the gateway-side
/// <c>MxGateway:Worker:EventQueueCapacity</c> minimum so a value that
@@ -140,7 +139,7 @@ public sealed class MxAccessEventQueue
/// <returns>The capacity to construct the outbound event queue with.</returns>
internal static int ResolveCapacity()
{
string? value = Environment.GetEnvironmentVariable(CapacityEnvironmentVariableName);
string? value = Environment.GetEnvironmentVariable(WorkerLaunchEnvironment.EventQueueCapacity);
return int.TryParse(
value,
NumberStyles.Integer,
@@ -3,6 +3,7 @@ using System.Collections.Generic;
using System.Diagnostics;
using System.Threading;
using System.Threading.Tasks;
using ZB.MOM.WW.MxGateway.Contracts;
using ZB.MOM.WW.MxGateway.Contracts.Proto;
using ZB.MOM.WW.MxGateway.Worker.Conversion;
using ZB.MOM.WW.MxGateway.Worker.Sta;
@@ -16,17 +17,11 @@ public sealed class MxAccessStaSession : IWorkerRuntimeSession
/// MxGateway:Worker:WriteCompletionWaitMilliseconds. 0 disables the
/// write-completion wait (pure fire-and-forget replies).
/// </summary>
internal const string WriteCompletionWaitEnvironmentVariableName =
"MXGATEWAY_WORKER_WRITE_COMPLETION_WAIT_MS";
/// <summary>
/// Environment variable the gateway's WorkerProcessLauncher sets from
/// MxGateway:Alarms:PollIntervalMilliseconds. A missing or invalid
/// value falls back to <see cref="DefaultAlarmPollInterval"/>.
/// </summary>
internal const string AlarmPollIntervalEnvironmentVariableName =
"MXGATEWAY_ALARM_POLL_INTERVAL_MS";
/// <summary>
/// Floor on the resolved alarm poll cadence. Mirrors the gateway-side
/// MxGateway:Alarms:PollIntervalMilliseconds minimum so a value that
@@ -211,7 +206,7 @@ public sealed class MxAccessStaSession : IWorkerRuntimeSession
/// </summary>
internal static TimeSpan ResolveWriteCompletionTimeout()
{
string? value = Environment.GetEnvironmentVariable(WriteCompletionWaitEnvironmentVariableName);
string? value = Environment.GetEnvironmentVariable(WorkerLaunchEnvironment.WriteCompletionWaitMilliseconds);
return int.TryParse(
value,
System.Globalization.NumberStyles.Integer,
@@ -233,7 +228,7 @@ public sealed class MxAccessStaSession : IWorkerRuntimeSession
/// <returns>The cadence the alarm poll loop waits between polls.</returns>
internal static TimeSpan ResolveAlarmPollInterval()
{
string? value = Environment.GetEnvironmentVariable(AlarmPollIntervalEnvironmentVariableName);
string? value = Environment.GetEnvironmentVariable(WorkerLaunchEnvironment.AlarmPollIntervalMilliseconds);
if (!int.TryParse(
value,
System.Globalization.NumberStyles.Integer,
@@ -5,6 +5,7 @@ using System.Globalization;
using System.Runtime.InteropServices;
using System.Xml;
using WNWRAPCONSUMERLib;
using ZB.MOM.WW.MxGateway.Contracts;
namespace ZB.MOM.WW.MxGateway.Worker.MxAccess;
@@ -75,9 +76,6 @@ public sealed class WnWrapAlarmConsumer : IMxAccessAlarmConsumer
/// <see cref="DefaultMaxAlarmsPerFetch"/> — a bad environment value
/// must never keep the alarm consumer from starting.
/// </summary>
internal const string MaxAlarmsPerFetchEnvironmentVariableName =
"MXGATEWAY_ALARM_MAX_ALARMS_PER_FETCH";
/// <summary>
/// Minimum gap between truncated-fetch warnings. A galaxy parked above
/// the cap truncates on *every* poll, so an unthrottled warning would
@@ -156,7 +154,7 @@ public sealed class WnWrapAlarmConsumer : IMxAccessAlarmConsumer
/// <returns>The cap passed to <c>GetXmlCurrentAlarms2</c>.</returns>
internal static int ResolveMaxAlarmsPerFetch()
{
string? value = Environment.GetEnvironmentVariable(MaxAlarmsPerFetchEnvironmentVariableName);
string? value = Environment.GetEnvironmentVariable(WorkerLaunchEnvironment.MaxAlarmsPerFetch);
return int.TryParse(
value,
NumberStyles.Integer,
+10 -8
View File
@@ -2,9 +2,11 @@
**Generated:** 2026-06-15 · **Commit:** `c7f754c` (main) · **Method:** six parallel read-only audits (Server, Worker, Contracts/proto, all five clients, docs/design/plans, tests + review backlog). Every item cites a verified `file:line`.
> **Historical inventory:** This file preserves the June 2026 audit and its resolution annotations. It is not the live remediation register. Current finding status and verification evidence live under `archreview/`, including the active cycle tracker at `archreview/2026-08-19/remediation/00-tracking.md`.
> **Resolution update (2026-06-15, branch `feat/stillpending-completion`):** The actionable items were implemented and verified per `docs/plans/2026-06-15-stillpending-completion.md`. **§1.1** (all 11 worker command kinds), **§1.2** (audit CorrelationId), and the **§4** client CLI/helper parity gaps are **Resolved** — see per-item annotations below. Worker COM commands are live-verified on the dev rig (`efd9971`, `f7ada90`). Remaining open items are the documented residuals (**§1.3**, **§1.4**, the **§3** vendor/capture-gated questions incl. the new **§3.2** multi-sample buffered residual) and the deliberate v1 scope of **§2**. Zero `.proto` changes were needed (all reply messages already existed).
>
> **Resolution update (2026-06-16, `main`):** The **session-resilience epic** (`docs/plans/2026-06-15-session-resilience.md`) landed its first **12 of 28 tasks** on `main` (through merge `c446bef`), which moves several **§2** items off "deferred": **multi-subscriber fan-out is now Resolved**, and **reconnectable sessions** has its server-side core (detach-grace window + replay-on-reconnect + a new `ReplayGap` signal on `MxEvent`). Still pending in that epic: reconnect Tasks 13–15 (owner re-validation, client `ReplayGap` handling, integration test), **Phase 4** per-session dashboard ACL (Tasks 16–19 — the §2/§7.6/§8 EventsHub-ACL item), and **Phase 5** orphan-worker reattach (Tasks 20–28). Separately, a `MxGateway:Dashboard:DisableLogin` dev flag shipped (`ca443b1`) — auto-authenticates the dashboard as a multi-role admin; default off, **enabled on the 10.100.0.48 deployment**. Per-item §2 status annotated below; remaining epic tasks tracked in `docs/plans/2026-06-15-session-resilience.md.tasks.json`.
> **Resolution update (2026-08-25, `main`):** The **session-resilience epic** (`docs/plans/2026-06-15-session-resilience.md`) subsequently completed reconnect owner validation, typed `ReplayGap` handling in all five clients, the reconnect integration test, and the Phase 4 per-session dashboard ACL. Orphan-worker reattach remains deliberately deferred and is not planned. Separately, a `MxGateway:Dashboard:DisableLogin` dev flag shipped (`ca443b1`) — auto-authenticates the dashboard as a multi-role admin; default off, **enabled on the 10.100.0.48 deployment**. See the current archreview tracker for live status.
## How to read this
@@ -51,9 +53,9 @@ Items are graded by what they actually are, because most "pending" surface in th
These are documented, deliberate, and mostly enforced. Listed so the deferred surface is in one place — **none are bugs.** Canonical register: `docs/DesignDecisions.md:466-474` ("Later Revisit Items") + `gateway.md` "Post-v1 revisit items".
- 🟡 **Reconnectable sessions — server-side core landed, not yet complete (2026-06-16, `main`).** Epic Phase 3 added the detach-grace retention window, replay-on-reconnect from the bounded replay ring, and the `ReplayGap` signal on `MxEvent` (Tasks 10–12, merged). Still pending: owner re-validation on reconnect (Task 13), client `ReplayGap` handling across all five clients (Task 14), and the fake-worker reconnect integration test (Task 15) — so reconnect is **not** safe to rely on end-to-end yet. Overturns `docs/DesignDecisions.md:63-73`. See `docs/plans/2026-06-15-session-resilience.md` Phase 3.
- ✅ **Reconnectable sessions — RESOLVED.** Detach-grace retention, replay-on-reconnect, the `ReplayGap` signal, owner re-validation, typed handling in all five official clients, and fake-worker end-to-end coverage have shipped. See `docs/DesignDecisions.md`, `docs/Sessions.md`, and `GatewayEndToEndReconnectReplayTests` for the current contract.
- ✅ **Multi-event-subscriber fan-out — RESOLVED (2026-06-16, `main`).** Epic Phase 2 (Tasks 7–9) removed the `GatewayOptionsValidator` rejection, added a `MaxEventSubscribersPerSession` cap (default 8), and built fan-out on the new `SessionEventDistributor` (single pump → N bounded per-subscriber channels with per-subscriber backpressure), with FakeWorkerHarness end-to-end coverage. Was: validator-blocked. `docs/plans/2026-06-15-session-resilience.md` Phase 2.
- 🟡 **Gateway restart does not reattach orphan workers — still true on `main`, planned (epic Phase 5, not started).** Workers are still terminated on startup. Epic Phase 5 (Tasks 20–28) is designed to reverse this — stable pipe naming, a SQLite adoption manifest, a worker adopt/reconnect proto frame, and nonce-validated re-adoption behind an `EnableOrphanReattach` flag (default off). Reverses the hard rule in `docs/DesignDecisions.md:65-69` and `CLAUDE.md`. `docs/plans/2026-06-15-session-resilience.md` Phase 5.
- 🔵 **Gateway restart does not reattach orphan workers — deliberately deferred, not planned.** Workers are terminated on startup and no code path may assume adoption after a gateway restart. The old epic Phase 5 sketch remains historical context only; `docs/DesignDecisions.md` and `CLAUDE.md` are authoritative.
- 🔵 **Workers run as the gateway service identity** — restricted service account is a reserved extension point. `docs/DesignDecisions.md:179-184`.
- 🔵 **Fail-fast event backpressure, no coalescing** — opt-in coalescing is post-v1. `docs/DesignDecisions.md:187-203`.
- 🔵 **No public command batching** — `docs/DesignDecisions.md:206-212`.
@@ -108,14 +110,14 @@ Library RPC surface is at **full parity**: all gateway + GalaxyRepository RPCs a
All live/integration paths are opt-in; the default unit suites do not exercise them.
- ⚪ **Live MXAccess COM + STA + message pump** — `Worker.Tests/MxAccess/MxAccessLiveComCreationTests.cs` (5 `[LiveMxAccessFact]`), gated `MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1`.
- ⚪ **Live gateway↔worker↔MXAccess round-trip** — `IntegrationTests/WorkerLiveMxAccessSmokeTests.cs` (6 `[LiveMxAccessFact]`).
- ⚪ **Live gateway↔worker↔MXAccess round-trip** — `IntegrationTests/WorkerLiveMxAccessSmokeTests.cs` (8 `[LiveMxAccessFact]`).
- ⚪ **Live Galaxy Repository SQL** — `IntegrationTests/Galaxy/GalaxyRepositoryLiveTests.cs` (4 `[LiveGalaxyRepositoryFact]`), gated `MXGATEWAY_RUN_LIVE_GALAXY_TESTS=1`.
- ⚪ **Live LDAP dashboard auth** — `IntegrationTests/DashboardLdapLiveTests.cs` (5 `[LiveLdapFact]`), gated `MXGATEWAY_RUN_LIVE_LDAP_TESTS=1`.
- ⚪ **Alarm runtime/discovery probes (dev-rig)** — `Worker.Tests/Probes/{WnWrapConsumerProbeTests,AlarmClientWmProbeTests}.cs`, `AlarmClientDiscoveryTests.cs` — hard `[Fact(Skip=...)]`.
- ⚪ **Live alarm + subtag-fallback smoke** — `Worker.Tests/Probes/{AlarmSubtagLiveSmokeTests,AlarmsLiveSmokeTests}.cs` — `Skip` + one `[LiveMxAccessFact]`; Clear path remains undrivable even when enabled.
- ⚪ **Python loopback TLS** — `clients/python/tests/test_tls.py:111-112` — gated `MXGATEWAY_RUN_TLS_TESTS=1` + openssl; only cert-config parsing runs by default.
- ⚪ **.NET client live browse smoke** — `clients/dotnet/.../BrowseChildrenSmokeTests.cs:17-18` — hard `[Fact(Skip=...)]`.
- ⚪ **Cross-language client↔gateway wire behavior** — no per-client integration unit tests; only `scripts/run-client-e2e-tests.ps1` against a live gateway (`MXGATEWAY_INTEGRATION=1`). All client wire behavior is unverified in default unit runs.
- ✅ **Cross-language client↔gateway wire behavior — RESOLVED (TST-24).** All five clients run real-server transport tests in their default suites: .NET and Python use loopback servers, while Go (`bufconn`), Rust, and Java use in-process servers/channels. The opt-in live matrix in `scripts/run-client-e2e-tests.ps1` (`MXGATEWAY_INTEGRATION=1`) remains the separate deployed-gateway check.
No placeholder/empty/`Assert.True(true)` tests were found anywhere.
@@ -144,10 +146,10 @@ No placeholder/empty/`Assert.True(true)` tests were found anywhere.
## 8. Deferred test-coverage follow-ups (noted in resolutions, never filed as findings)
> **Resolution update (2026-06-16, branch `feat/stillpending-section8`):** Three of the four items below are **Resolved** per `docs/plans/2026-06-16-stillpending-section8.md` (Java CLI coverage, Worker-Ready wait, duplicated `FakeWorkerProcess`). Java work verified on the windev toolchain (full cli suite green); C# verified locally. The remaining item (per-session-ACL TODO) was correctly out of scope — it is session-resilience epic Phase 4.
> **Resolution update (2026-08-25):** All four items below are **Resolved**. The first three landed through `docs/plans/2026-06-16-stillpending-section8.md`; the per-session dashboard ACL subsequently shipped as archreview TST-15.
- ✅ **Java CLI bulk-subcommand coverage — RESOLVED.** All 10 previously-untested subcommands now have CLI-layer round-trip tests in `clients/java/zb-mom-ww-mxgateway-cli/src/test/java/com/zb/mom/ww/mxgateway/cli/MxGatewayCliTests.java`: the bulk family (`read-bulk`, `write-bulk`, `write2-bulk`, `write-secured-bulk`, `write-secured2-bulk`, `bench-read-bulk`) + `close-session` via the `FakeSession` seam; `stream-events`, `galaxy-discover`, `galaxy-watch` via a new in-process gRPC harness (`InProcessGatewayHarness`) that drives the real client over an in-process channel. A `GalaxyClientFactory` seam was added to `MxGatewayCli` (behavior-preserving default) to make the galaxy commands testable. Was: `code-reviews/Client.Java/findings.md:495` (Client.Java-026).
- 🟡 **Per-session-ACL TODO** at `Server/Dashboard/Hubs/EventsHub.cs` (`code-reviews/Server/findings.md:765`) — still scheduled as session-resilience epic Phase 4 (Tasks 16–19); not yet started. (Out of scope for the §8 completion branch.)
- ✅ **Per-session dashboard ACL — RESOLVED (archreview TST-15).** Both the Events hub and session-details in-process subscription enforce `IDashboardSessionAcl`; the stale `TODO(per-session-acl)` is gone. See `docs/Authorization.md`.
- ✅ **Worker-Ready retry race — RESOLVED.** `GatewaySession.GetReadyWorkerClient` is now an opt-in bounded ready-wait (`GetReadyWorkerClientAsync` + `MxGateway:Sessions:WorkerReadyWaitTimeoutMs`, **default `0` = unchanged fail-fast**): a session that is `Ready` while its worker is transiently `Handshaking`/`Created` waits up to the configured window for `Ready`; terminal worker states fail fast immediately with the both-states diagnostic preserved. Was: `code-reviews/Server/findings.md:611`.
- ✅ **Duplicated `FakeWorkerProcess` harness — RESOLVED.** The three private nested `FakeWorkerProcess` copies were consolidated onto the canonical `src/ZB.MOM.WW.MxGateway.Tests/TestSupport/FakeWorkerProcess.cs` (one definition remains). Was a latent regression vector — `code-reviews/Tests/findings.md:463`.
@@ -162,6 +164,6 @@ No placeholder/empty/`Assert.True(true)` tests were found anywhere.
- **§1.4 / §3.4 / §3.5** — the AVEVA 8-arg `AlarmAckByName` is a vendor stub (−55) and `AlarmAckByGUID` is `E_NOTIMPL`; the `domain`/`full_name` fields stay forward-compat-only until AVEVA implements them.
- **§3.2** — buffered commands work and the empty bootstrap converts cleanly live, but a multi-sample buffered batch is undrivable on the rig (unit-tested only).
- **§3.1 / §3.3 / §3.6 / §3.7** — await live MXAccess captures.
- **§2** — mostly deliberate v1 scope, but the session-resilience epic (12/28 tasks merged to `main`) has since **resolved multi-subscriber fan-out** and landed the **reconnect server-side core**; reconnect Tasks 13–15, per-session ACL (Phase 4), and orphan-worker reattach (Phase 5) remain (see `docs/plans/2026-06-15-session-resilience.md`, `docs/plans/2026-06-15-session-resilience.md.tasks.json`). **§5** — opt-in verification gates. **§7.6** — accepted `Won't Fix` review findings.
- **§2** — deliberate v1 scope. Multi-subscriber fan-out, reconnect, and the per-session dashboard ACL are resolved; orphan-worker reattach remains deliberately deferred and not planned. **§5** — opt-in verification gates. **§7.6** — accepted `Won't Fix` review findings.
MXAccess **event/data/value/write** mapping, the **Galaxy** RPC surface, and now the **full command surface** are complete; no `NotImplementedException`s, stubbed RPC bodies, or empty tests remain in the production paths.