Compare commits
29 Commits
577a0db862
...
f2ab9cc879
| Author | SHA1 | Date | |
|---|---|---|---|
| f2ab9cc879 | |||
| cb820bb666 | |||
| c6650c156b | |||
| 193221773d | |||
| 06cd8b844c | |||
| cfbc99581b | |||
| ab41362a5c | |||
| b27035b3a6 | |||
| 9f0c3fd9fa | |||
| a71663a2a2 | |||
| 8257a0333e | |||
| 2ef7b93761 | |||
| b177636d7a | |||
| 282a0e70ad | |||
| 3619152b9f | |||
| ee68cc3f72 | |||
| 7200023bcc | |||
| 058d8d529f | |||
| aa07497cba | |||
| f1353dda86 | |||
| 3679e59eed | |||
| f59c34a544 | |||
| 20a3eefb1e | |||
| 22d5d52c14 | |||
| 93edb1b74d | |||
| ddfd41b06d | |||
| 7c5eb062e3 | |||
| b75f8b1704 | |||
| 6c784e3db7 |
@@ -23,7 +23,12 @@ jobs:
|
||||
# tests, the codegen/descriptor freshness guards, and the clients that build on Linux.
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
# fetch-depth: 0 (full history + tags): the CLI-46 client-version-freshness check below
|
||||
# diffs each client's source against its last publish tag (e.g. clients/go/vX.Y.Z), which
|
||||
# a shallow, tagless checkout cannot resolve.
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
fetch-depth: 0
|
||||
|
||||
- uses: actions/setup-dotnet@v4
|
||||
with:
|
||||
@@ -77,6 +82,15 @@ jobs:
|
||||
shell: pwsh
|
||||
run: ./scripts/check-codegen.ps1
|
||||
|
||||
# CLI-46: catches a client whose source changed since its last publish
|
||||
# tag while the version constant still reads that tag's already-shipped
|
||||
# number — the drift the pack-clients.ps1/tag-go-module.ps1 collision
|
||||
# guards only catch at actual publish time. Needs tag history, hence
|
||||
# `fetch-depth: 0` on this job's checkout above.
|
||||
- name: Client version freshness
|
||||
shell: pwsh
|
||||
run: ./scripts/check-client-versions.ps1
|
||||
|
||||
- name: Gateway fake-worker tests
|
||||
run: dotnet test src/ZB.MOM.WW.MxGateway.Tests/ZB.MOM.WW.MxGateway.Tests.csproj -c Release --no-build
|
||||
|
||||
|
||||
@@ -159,7 +159,7 @@ Gateway gRPC clients authenticate with an API key in metadata: `authorization: B
|
||||
|
||||
Session event streaming is **owner-scoped**: the API key that opened a session is recorded on the session, and every `StreamEvents` attach/reattach is rejected with `PermissionDenied` unless the caller's key id matches the owner. Possessing the `event` scope and knowing a session id is not sufficient — this closes the reconnect/fan-out trust boundary (detach-grace and replay retention are on by default) so an `event`-scoped key cannot attach to another key's retained session.
|
||||
|
||||
Dashboard auth is LDAP-backed (separate from the gRPC API-key model). `/login` binds against `MxGateway:Ldap` and maps the user's LDAP groups to `Admin` or `Viewer` via `MxGateway:Dashboard:GroupToRole`, then issues an HTTP-only secure cookie named `__Host-MxGatewayDashboard` when `Dashboard:RequireHttpsCookie` is true (default) and no `Dashboard:CookieName` override is set, else the plain `MxGatewayDashboard` (the `__Host-` prefix requires a Secure cookie). SignalR hubs at `/hubs/{snapshot,alarms,events}` accept either the cookie or a 5-minute bearer minted at `/hubs/token`. Dashboard event visibility is **tag-scoped per session** (`IDashboardSessionAcl`, gating both the events hub and the session-details page's in-process subscribe): an Administrator sees every session, while any other caller sees a session only when its tags — inherited from the owning API key's `apikey --dashboard-tags`, never from the client's request — intersect the tags their LDAP groups grant via `Dashboard:GroupToTag`; untagged sessions follow `Dashboard:UntaggedSessionVisibility` (default `AdminOnly`), and a principal with no tag claims (anonymous localhost included) is an empty-grant Viewer. `Dashboard:AllowAnonymousLocalhost` (default `true`) grants **read-only** loopback access — it satisfies the Viewer requirement but never the Admin-only requirement, so anonymous localhost can view the dashboard but not reach API-key CRUD or session Close/Kill (`Authentication:Mode=Disabled` is scoped the same way). `Dashboard:DisableLogin` (default `false`) auto-authenticates every dashboard request — including remote browsers — as `Dashboard:AutoLoginUser` (default `multi-role`) with both Admin and Viewer roles; dev/test only, never enable in production.
|
||||
Dashboard auth is LDAP-backed (separate from the gRPC API-key model). `/login` binds against `MxGateway:Ldap` and maps the user's LDAP groups to `Admin` or `Viewer` via `MxGateway:Dashboard:GroupToRole`, then issues an HTTP-only secure cookie named `__Host-MxGatewayDashboard` when `Dashboard:RequireHttpsCookie` is true (default) and no `Dashboard:CookieName` override is set, else the plain `MxGatewayDashboard` (the `__Host-` prefix requires a Secure cookie). SignalR hubs at `/hubs/{snapshot,alarms,events}` accept either the cookie or a 5-minute bearer minted at `/hubs/token`; `/hubs/alarms` and the `/alarms` page additionally require a **real authenticated dashboard role** (`MxGateway.Dashboard.Alarms`, a requirement type the environment-bypass handler does not handle), because the alarm feed is gateway-central with no per-session provenance and so cannot be tag-scoped — its metadata stays galaxy-wide for authenticated Viewers, but anonymous localhost and `Authentication:Mode=Disabled` do not receive it at all. Dashboard event visibility is **tag-scoped per session** (`IDashboardSessionAcl`, gating both the events hub and the session-details page's in-process subscribe): an Administrator sees every session, while any other caller sees a session only when its tags — inherited from the owning API key's `apikey --dashboard-tags`, never from the client's request — intersect the tags their LDAP groups grant via `Dashboard:GroupToTag`. That grant is **re-resolved from the caller's LDAP group claims against current options on every subscribe** (not read from the tag claims stamped at login, which would outlive a config-revoked grant for the cookie's 8-hour sliding life); login-stamped tag claims are the fallback only for a principal carrying no group claims — i.e. the hub-token principal, whose grant the mint already re-resolved. Untagged sessions follow `Dashboard:UntaggedSessionVisibility` (default `AdminOnly`), and a principal with neither claim kind (anonymous localhost included) is an empty-grant Viewer. `Dashboard:AllowAnonymousLocalhost` (default `true`) grants **read-only** loopback access — it satisfies the Viewer requirement but never the Admin-only requirement, so anonymous localhost can view the dashboard but not reach API-key CRUD, session Close/Kill, or the alarm feed (`Authentication:Mode=Disabled` is scoped the same way). `Dashboard:DisableLogin` (default `false`) auto-authenticates every dashboard request — including remote browsers — as `Dashboard:AutoLoginUser` (default `multi-role`) with both Admin and Viewer roles; dev/test only, never enable in production.
|
||||
|
||||
## Process / Platform Notes
|
||||
|
||||
|
||||
@@ -79,6 +79,12 @@ Add one entry per finding to the `## Findings` section of the module's
|
||||
|
||||
- **Finding ID** — `<Module>-NNN`, numbered sequentially within the module and
|
||||
never reused (e.g. `Worker-001`). IDs are permanent even after resolution.
|
||||
The `GWC`/`WRK`/`IPC`/`SEC`/`CLI`/`TST` prefixes are reserved for the
|
||||
`archreview/` domain registers (a separate, whole-repo review series with its
|
||||
own tracking documents) — never assign one of these prefixes to a
|
||||
`code-reviews/` finding, and any other work stream (perf passes, ad hoc
|
||||
remediation plans, etc.) that wants a stable per-item ID must mint its own
|
||||
distinct prefix rather than borrowing one of these six.
|
||||
- **Severity:**
|
||||
- **Critical** — data loss, security breach, crash/deadlock, or outage.
|
||||
- **High** — incorrect behaviour with significant impact; no safe workaround.
|
||||
|
||||
@@ -79,6 +79,9 @@ Full design + implementation for each row lives in the linked domain doc under i
|
||||
| WRK-26 | Low | P1 | S | WRK-23 (soft); discharges IPC-29 | Done | Write-priority and overflow doc drift from the WRK-07 change |
|
||||
| WRK-27 | Low | — | S | — | Done | Alarm poll bypasses the watchdog's in-flight suppression (15 s vs 75 s) |
|
||||
| WRK-28 | Low | — | S | WRK-21 (same batch) | Done | 10,000 drain cap is a duplicated magic constant |
|
||||
| WRK-29 | — | — | — | — | Reserved (do not reuse) | Interim ID space possibly burned alongside WRK-31 by `e913dab`'s ad hoc numbering; not confirmed in the repo or its history, held back defensively |
|
||||
| WRK-30 | — | — | — | — | Reserved (do not reuse) | Interim ID space possibly burned alongside WRK-31 by `e913dab`'s ad hoc numbering; not confirmed in the repo or its history, held back defensively |
|
||||
| WRK-31 | — | — | — | — | Reserved (do not reuse) | Interim ID burned by `e913dab` (session-owned transport disposal), cited in code at `Ipc/WorkerPipeSession.cs:45,177` and `Ipc/WorkerPipeSessionTests.cs` outside the register; not a finding — registered per cycle-3 `20-worker.md` (~line 9) so it is never reassigned. Cycle-3 new findings start at WRK-32. |
|
||||
|
||||
### Contracts & IPC — [30-contracts-ipc.md](30-contracts-ipc.md)
|
||||
|
||||
@@ -92,7 +95,7 @@ Full design + implementation for each row lives in the linked domain doc under i
|
||||
| IPC-28 | Low | — | S | — | Done | docs/Grpc.md missing CommandTooLarge → ResourceExhausted mapping |
|
||||
| IPC-29 | Low | — | S | WRK-26 (discharged by) | Done (discharged by WRK-26) | WorkerFrameProtocol.md missing write-scheduling/sequencing section |
|
||||
| IPC-30 | Low | P0 | M | WRK-21 (same batch) | Done | Oversized event frame: keep session-fatal, make the death structured |
|
||||
| IPC-31 | Info | — | — | — | N/A | Gateway creation-time sequence stamping accepted; diagnostic-only, decision recorded |
|
||||
| IPC-31 | Info | — | — | — | Done (superseded by GWC-28) | Gateway creation-time sequence stamping — resolved when GWC-28 moved gateway stamping to write time; the two sides now implement the same semantics |
|
||||
| IPC-32 | Info | — | S | IPC-25 (folded in) | Done | check-codegen banner relabel 1/4…4/4 |
|
||||
|
||||
### Security & dashboard — [40-security-dashboard.md](40-security-dashboard.md)
|
||||
|
||||
@@ -24,6 +24,7 @@ members, no positional records). The worker builds and tests only on the Windows
|
||||
| WRK-26 | Low | P1 | S | WRK-23 (soft — sequence prose); discharges IPC-29 | Done | Write-priority and overflow doc drift from the WRK-07 change |
|
||||
| WRK-27 | Low | — | S | — | Done | Alarm poll bypasses the watchdog's in-flight suppression (15 s vs 75 s) |
|
||||
| WRK-28 | Low | — | S | WRK-21 (land in the same commit cluster) | Done | 10,000 drain cap is a duplicated magic constant with a comment-only sync contract |
|
||||
| WRK-31 | — | — | — | — | Reserved (do not reuse) | Not a finding. `e913dab` (session-owned transport disposal) coined "WRK-31" ad hoc in code comments (`Ipc/WorkerPipeSession.cs:45,177`) and in `Ipc/WorkerPipeSessionTests.cs`, before the register assigned it. WRK-29/WRK-30 appear nowhere in the repo or its history but may have been burned by the same interim numbering, so they are held back defensively too — see the tracker's WRK-29/30/31 rows. Registered here per cycle-3 `archreview/2026-08-19/20-worker.md` (~line 9, "Finding-ID note"); that review's new findings start at **WRK-32** to guarantee no collision. |
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -231,6 +231,8 @@ The published-side collection (`CollectPublishedSymbols`, `:61-79`) grows matchi
|
||||
|
||||
**Disposition: N/A — no action.** `gateway.md:328-330` already declares sequence a per-sender diagnostic aid, never validated; both validators ignore it; the divergence is harmless and now explicitly documented by the IPC-29 doc section (which states the two sides' stamping points). If sequence ever becomes load-bearing, stamp in the gateway write loop (`WorkerClient.cs:390-397`) — recorded here so the next editor finds the decision.
|
||||
|
||||
**Resolution: Done — superseded by GWC-28 (2026-08-20, per cycle-3 verification).** GWC-28 moved the gateway-side stamp from creation to the point of writing: `WriteLoopAsync` now stamps `Sequence` immediately before `_writer.WriteAsync` (`Server/Workers/WorkerClient.cs:465-471`), and `CreateEnvelope` leaves it unset (`:1229-1236`); `gateway.md:425-431` was rewritten to "both sides stamp at the point of writing", closing the divergence this finding recorded. Regression test `WorkerClientTests.ConcurrentInvokesEmitStrictlyIncreasingSequencesOnTheWire` covers it. Per `archreview/2026-08-19/30-contracts-ipc.md` (IPC-31 row), tracker flipped from `N/A` to `Done (superseded by GWC-28)`. See cycle-3 **IPC-39** for the one interaction the change created with the IPC-03 pre-check.
|
||||
|
||||
---
|
||||
|
||||
## IPC-32 — `check-codegen.ps1` check labels are miscounted `Info` · `—` (rides IPC-25)
|
||||
|
||||
@@ -0,0 +1,89 @@
|
||||
# MxAccessGateway — Remediation Plan & Tracking (2026-08-19 review)
|
||||
|
||||
Source review: [`../00-overall.md`](../00-overall.md) (HEAD `8b3e3b3`, committed as `f4919ba`). Generated 2026-08-20.
|
||||
Tracks the 34 cycle-3 IDs (GWC-31+, WRK-32+, IPC-33+, SEC-37+, CLI-46+, TST-31+) plus the tracker-housekeeping actions the overall asked for. Cycle-1/2 trackers remain the record for their own IDs.
|
||||
|
||||
**Status legend:** `Not started` · `In progress` · `Done` · `Blocked (external)` · `Deferred (P2)` · `N/A`.
|
||||
|
||||
## Execution model
|
||||
|
||||
This plan is executed as **seven parallel batches**, each by one subagent working in an isolated git worktree on its own `fix/c3-<batch>` branch, merged back into `main` by the coordinator in the order listed (lowest conflict surface first). Every batch must: build the affected component(s) on macOS (`NonWindows.slnx` — the x86 worker cannot build here), run only the filtered tests it touches, update docs in the same commit, and cite finding IDs in commit messages (`fix(IPC-34): …`). Worker-side changes (batches E, F) are verified afterwards on windev via `windev-worker-ci.ps1 -Mode test`.
|
||||
|
||||
**Model assignment rationale:** `opus` for batches that change concurrency/authorization logic or require reasoning across several interacting files with regression tests; `sonnet` for contained, well-specified edits with a clear recipe (version bumps, doc reconciliation, a two-line worker fix); `haiku` for none — every batch either touches code invariants or needs judgement about stale prose, where the cost of a wrong mechanical sweep exceeds the saving.
|
||||
|
||||
| Batch | Branch | Findings | Model | Why this model | Status |
|
||||
|---|---|---|---|---|---|
|
||||
| A | `fix/c3-client-versions` | CLI-46 | sonnet | Mechanical five-client bump + README note + one script guard; recipe is fully specified | Done |
|
||||
| B | `fix/c3-glauth-doc` | TST-31 | sonnet | Reconcile a table against one TOML source of truth, re-derive one default, change an assert to a log | Done |
|
||||
| C | `fix/c3-tracker-hygiene` | GWC-31 + housekeeping (close GWC-16/IPC-08/IPC-31, refresh TST-06/TST-17, register WRK-31) | sonnet | Cross-file comment/ID sweep needs care about what each ID *means*, but no logic changes | Done (`3619152`) |
|
||||
| D | `fix/c3-dashboard-acl` | SEC-40, SEC-41, GWC-38 | opus | Authorization semantics, fail-closed decisions, claims re-resolution; each needs a regression test | **Done** (`b177636` GWC-38, `2ef7b93` SEC-41+SEC-40, `8257a03` comment fixup; branch not merged/pushed) |
|
||||
| E | `fix/c3-ipc-boundaries` | IPC-33, IPC-39, IPC-34, IPC-35 | opus | Changes the WorkerClient command lifecycle, the options validator, and the worker drain budget; race-sensitive | **Done** (`577a0db` IPC-33, `a71663a` IPC-39, `b27035b` IPC-34, `ab41362` IPC-35; branch `fix/c3-ipc-boundaries`, not merged/pushed) — worker half pending windev (`WorkerPipeSession.ResolveDrainReplyByteBudget` and the `WorkerPipeSessionTests` floor-test comment cannot be built on macOS) |
|
||||
| F | `fix/c3-worker-residuals` | WRK-32, WRK-33 (+WRK-34 opportunistic) | sonnet | Small, precisely located worker edits + doc prose; cannot be built on macOS so the agent's job is a careful, minimal diff | Done-pending-windev |
|
||||
| G | `fix/c3-session-lifecycle` | GWC-32, GWC-34, GWC-37 | opus | Distributor/dispose lifecycle races and an alarm-monitor re-resolve path; needs lock-discipline reasoning | Done (`058d8d5`, `282a0e7`, `1932217`) |
|
||||
|
||||
**Out of scope for this execution (recorded, not dropped):**
|
||||
|
||||
| ID | Status | Why |
|
||||
|---|---|---|
|
||||
| TST-32 | Blocked (external) | Gitea runner registration-token reset is a web-UI action on the Gitea admin page; no repo change. Operator to-do. |
|
||||
| SEC-37 | Blocked (external) | Family-wide: TLS on the shared GLAuth lives in `scadaproj/infra/glauth/` (separate repo + docker host). Plan separately; after it lands, move the deployed service to `Production`. |
|
||||
| CLI-47/48/49/50, IPC-36, IPC-37, TST-33/34/35/36, GWC-33/35/36, SEC-38/39, IPC-38, WRK-34 | Deferred (P2) | Roadmap tier 10; WRK-34 may be picked up by batch F if trivial. |
|
||||
|
||||
## Batch specifications
|
||||
|
||||
### A — CLI-46 client release-train bump (sonnet)
|
||||
- Bump .NET/Go/Python/Rust clients `0.2.0 → 0.2.1`, Java `0.2.1 → 0.2.2` in every version-bearing file (`pyproject.toml` + `version.py`, `Cargo.toml` workspace + any `version =` consumers, `Directory.Build.props`/csproj `<Version>`, Java `build.gradle`/`gradle.properties`, Go module is tag-driven — update any `Version` const if one exists). Follow `docs/ClientPackaging.md` for the authoritative file list.
|
||||
- `clients/python/README.md` + `docs/ClientPackaging.md`: note that published pypi 0.2.0 has a broken `connect()` (event-loop binding, fixed by `a8f86b5`) and is superseded by 0.2.1.
|
||||
- Add a PR-time check to `scripts/check-codegen.ps1` (or a new `scripts/check-client-versions.ps1` wired where the other checks run): fail if any `clients/<lang>` source differs from its last publish tag while the version constant still equals that tag's version. Document in `docs/ClientPackaging.md`.
|
||||
- Verify: `dotnet build clients/dotnet/...slnx`; `cargo check --workspace`; `python -m pytest` (clients/python); `JAVA_HOME=/opt/homebrew/opt/openjdk@17 gradle test` then **revert** spurious generated-proto churn; `go build ./... && go test ./...`.
|
||||
|
||||
### B — TST-31 glauth.md + live-test credential (sonnet)
|
||||
- Reconcile `glauth.md` against `~/Desktop/scadaproj/infra/glauth/config.toml`: ReadOnly gid, the pre-provisioned users table (drop `readonly/readonly123`, fix `admin` password), the worked example at `:124`. Keep the OPC-UA taxonomy note.
|
||||
- Re-derive `MXGATEWAY_LIVE_MXACCESS_WRITE_SECURED_PASSWORD` default in `docs/GatewayTesting.md` and the `?? "admin123"` literal in `WorkerLiveMxAccessSmokeTests.cs` from the config; make the test **log** (not assert) whether `AuthenticateUser` succeeded.
|
||||
- Verify: `dotnet build src/ZB.MOM.WW.MxGateway.NonWindows.slnx` (IntegrationTests compiles on mac? if excluded, note it).
|
||||
|
||||
### C — GWC-31 + tracker housekeeping (sonnet)
|
||||
- Sweep code comments citing unregistered `GWC-30`/`GWC-31` (perf pass) in `WorkerClient.cs`, `GatewayMetrics.cs`, `WorkerClientTests.cs`, worker `WorkerFrameReader/Writer.cs` to `PERF-nn` keyed to task numbers in `docs/plans/2026-08-15-perf-review-remediation.md`; add a PERF-ID table to that plan. Do not touch the genuine GWC-30 (frame-reader prefix) citations if any exist.
|
||||
- `archreview/REVIEW-PROCESS.md` (or wherever the ID rule lives): state that GWC/WRK/IPC/SEC/CLI/TST prefixes are reserved for archreview registers.
|
||||
- Cycle-2 tracker (`archreview/2026-07-12/remediation/00-tracking.md` + domain docs): GWC-16 → Won't fix (rationale from cycle-3 10-gateway-core.md), IPC-08 → Done (`7c1ea12`, residual IPC-34), IPC-31 → Done (superseded by GWC-28), refresh TST-06/TST-17 rows, register WRK-31 (interim ID burned by `e913dab`) so it is never reused.
|
||||
- Verify: `dotnet build src/ZB.MOM.WW.MxGateway.NonWindows.slnx`; `dotnet test --filter FullyQualifiedName~WorkerClientTests`.
|
||||
|
||||
### D — SEC-40, SEC-41, GWC-38 (opus)
|
||||
- SEC-41: `DashboardSessionAcl` re-resolves tags from `LdapGroupClaimType` claims via `DashboardGroupTagMapping.MapGroupsToTags` against current options (as `HubTokenService` does) instead of trusting login-stamped tag claims; keep stamped claims only as fallback when no group claims exist (document). Test: principal with stale tag claim + group whose grant was removed → denied.
|
||||
- SEC-40: decide **tighten**: `AlarmsHub` gets a dedicated policy requiring a real role claim (anonymous-loopback/Disabled bypass excluded) — and document in `docs/GatewayDashboardDesign.md` ACL section + `docs/GatewayConfiguration.md` GroupToTag row that alarm metadata remains galaxy-wide for authenticated Viewers (central monitor, no per-session provenance). Test for the policy.
|
||||
- GWC-38: `GatewayBrowseScopeProvider` throws `RpcException(PermissionDenied)` (and logs) when `Current` is null **and** API-key auth is enabled; unchanged when auth mode is Disabled. Test both branches.
|
||||
- Verify: build Server; `dotnet test --filter "FullyQualifiedName~DashboardSessionAcl|FullyQualifiedName~AlarmsHub|FullyQualifiedName~BrowseScope"` (adjust to real class names).
|
||||
|
||||
**Outcome (done, branch `fix/c3-dashboard-acl`, commits `b177636`, `2ef7b93`, `8257a03`; not merged, not pushed).** Decisions worth carrying forward:
|
||||
|
||||
- **SEC-41 fallback:** the ACL re-resolves from `mxgateway:ldap_group` claims when the principal has any, and falls back to the stamped `zb:dashboardtag` claims only when it has none. That is exactly the hub-token principal — `HubTokenService.Validate` rehydrates roles and tags but **not** groups, so re-resolution already happened at mint. The fallback cannot re-open the cookie gap, because `DashboardAuthenticator` stamps both claim kinds from one group list.
|
||||
- **SEC-40 requirement type:** new `AuthenticatedDashboardRoleRequirement` + `AuthenticatedDashboardRoleHandler` and a `MxGateway.Dashboard.Alarms` policy. `DashboardAuthorizationHandler` is `AuthorizationHandler<DashboardAuthorizationRequirement>`, so it is never invoked for the new type — the same structural device the Secrets UI's `secrets:manage` already relies on. Both hub schemes are retained on the policy.
|
||||
- **SEC-40 scope answer:** the `/alarms` Blazor page carries the SAME policy (it reads the central monitor in process — gating only the hub left the disclosure one navigation away), and the side-rail Alarms link is wrapped in an `AuthorizeView` on that policy. `AlarmsHubPublisher` is unchanged and stays unfiltered by design: audience is decided at hub admission and the feed carries no per-session provenance.
|
||||
- **GWC-38:** `PermissionDenied` + warning only when `Authentication:Mode=ApiKey`; `Mode=Disabled` still returns unscoped. `GalaxyRepositoryHostWiringTests` phase 1 was changed to push an *unconstrained identity* rather than none — that is production's real unscoped case.
|
||||
- New/changed tests: `AlarmFeedAuthorizationTests` (new, 7), `DashboardSessionAclTests` (+5), `GatewayBrowseScopeProviderTests` (+2), `SecretsNavRenderTests` (+2, and it must register the new policy — an unregistered policy name *throws* inside `AuthorizeView` rather than denying). Filtered run over the touched classes: 92 passed / 0 failed; `NonWindows.slnx` builds zero-warning.
|
||||
|
||||
### E — IPC-33, IPC-39, IPC-34, IPC-35 (opus)
|
||||
- IPC-33 option (a): `GatewayOptionsValidator` adds the upper coupling `Worker:MaxMessageBytes ≤ Protocol:MaxGrpcMessageBytes + 2×reserve`; worker `ResolveDrainReplyByteBudget` becomes `MaxMessageBytes − 2×DrainReplyFrameHeadroomBytes` (floor `MaxMessageBytes/2`); update `docs/GatewayConfiguration.md`, `docs/WorkerFrameProtocol.md`. Test: `GatewayOptionsTests` case for the new bound.
|
||||
- IPC-39: pre-check compares against `MaxMessageBytes − MaxSequenceEncodingBytes (11)`; fix the comment direction.
|
||||
- IPC-34: caller-cancellation arms call `TrySendCancelForAbandonedCommand` (renamed) gated on `removed`, skipped when `_stopCts` is the source; new `WorkerClientErrorCode.CallerCancelled` replacing the `GatewayShutdown` mislabel, mapped to `Cancelled` in the gRPC layer; update `docs/Grpc.md` mapping line, `gateway.md:913-919`, `docs/GatewayProcessDesign.md:596-602`. Test: twin of the timeout test driven by the caller token.
|
||||
- IPC-35: `ReadHandshakeEnvelopeAsync` special-cases `WorkerFault` → `WorkerClientException(WorkerFaulted, CreateWorkerFaultMessage(...))`. Test: fake writes `WorkerFault{MxaccessCreationFailed}` instead of `WorkerReady`, category text surfaces.
|
||||
- Verify: build Server; `dotnet test --filter "FullyQualifiedName~WorkerClientTests|FullyQualifiedName~GatewayOptions"`. Worker half → windev afterwards.
|
||||
|
||||
### F — WRK-32, WRK-33 (sonnet)
|
||||
- WRK-33: `ObserveFaultWhenever(ObserveCommandTaskAsync(commandTask));` at `WorkerPipeSession.cs:1235` (or catch-all-and-log in the observer).
|
||||
- WRK-32: one `pumpStep()` before capturing the `CurrentVersion` baseline in `MxAccessCommandExecutor.cs`; rewrite the executor doc comment (`:996-1003`) to state the recency-based contract; add the same to `docs/MxAccessWorkerInstanceDesign.md` write-completion section and the reply-field comment in `mxaccess_gateway.proto` (**comment-only proto edit → regenerate all generated artifacts in the same commit**, per the repo rule; if regen tooling is unavailable on macOS, skip the proto comment and say so).
|
||||
- WRK-34 if trivial: `CreateFault(StaHung, "Watchdog", …)` with the correlation id moved into the message.
|
||||
- Cannot build on macOS: keep diff minimal, net48-safe (no init-only/positional records). Verified on windev afterwards.
|
||||
|
||||
### G — GWC-32, GWC-34, GWC-37 (opus)
|
||||
- GWC-32: `DisposeAsync` transitions `_state` to `Closed` under `_syncRoot` before completing channels (or `if (_disposed || IsTerminalState()) return;` in `StageWorkerEvent`); update the XML doc to name all three completion paths. Test: dispose a Ready client mid-dispatch → no `ProtocolViolation` fault/metric.
|
||||
- GWC-34: `_eventDistributorDisposed` flag set under `_syncRoot` when nulling; `EnsureDistributorCreated` throws `SessionManagerException(SessionNotReady)` when set. Test: attach after dispose → throws, no silent hang.
|
||||
- GWC-37: on `ApplyProviderModeChangeAsync(toMode == Subtag)` re-resolve the watch list and re-subscribe; additionally surface "zero targets resolved while mode can need them" on the provider status (`Degraded` + reason). Update `docs/` alarm-subtag section. Test with the fake worker: GR discovery fails at start, succeeds at failover → watch list non-empty after failover.
|
||||
- Verify: build Server; `dotnet test --filter "FullyQualifiedName~WorkerClientTests|FullyQualifiedName~GatewaySession|FullyQualifiedName~AlarmMonitor|FullyQualifiedName~AlarmWatchList"`.
|
||||
- **Outcome (Done):** `058d8d5` GWC-32, `282a0e7` GWC-34, `1932217` GWC-37 on `fix/c3-session-lifecycle`. Filtered run 176/176 green; `NonWindows.slnx` clean.
|
||||
- GWC-37 **does not re-send** the watch-list on the failover edge. The worker's `AlarmCommandHandler.Subscribe` rejects a second `SubscribeAlarms` while subscribed, and the only expressible replacement (`UnsubscribeAlarms` + `SubscribeAlarms`) rebuilds `FailoverAlarmConsumer` — restarting it on the primary that just failed and discarding the warm standby — so it would flap the feed rather than repair it. The re-resolve happens and any drift is reported on the same status frame as the failover; a monitor restart is the supported way to apply a new list. Adding a watch-list-update worker command is a possible future item, not done here.
|
||||
- GWC-37 changes `IAlarmWatchListResolver.ResolveAsync` to return `AlarmWatchListResolution`, and gives `Degraded && Mode==ALARMMGR` a real meaning (empty/undiscoverable standby) with a new dashboard label `Alarm Manager (no subtag standby)`.
|
||||
- GWC-32's dispose path now also records `WorkerStopped("dispose")` via the existing once-latch, closing a `workers_running` gauge leak on the dispose-a-Ready-client path. GWC-12's double-dispose interlock got a natural home (the `_disposed` claim moved under `_syncRoot`) but is **not** claimed as fixed.
|
||||
|
||||
## Merge order
|
||||
C is merged **last** (its comment sweep touches the same files E and G edit). Order: A, B, F, D, G, E, C. After merge: full `NonWindows` build + one full gateway test run; then windev worker CI for E+F.
|
||||
@@ -74,7 +74,7 @@ Full design + implementation for each row lives in the linked domain doc under i
|
||||
| GWC-13 | Low | — | M | — | Not started | Worker-ready wait is a 25 ms poll loop |
|
||||
| GWC-14 | Low | P2 | M | — | Done | Replay ring is a `LinkedList` with a node alloc per event |
|
||||
| GWC-15 | Low | P2 | S | — | Done | Per-event gauge reads `ChannelReader.Count` every event |
|
||||
| GWC-16 | Low | — | S | — | Not started | `Invoke` resolves the session twice per command |
|
||||
| GWC-16 | Low | — | S | — | Won't fix | `Invoke` resolves the session twice per command — measured negligible, deliberately not collapsed (see domain doc) |
|
||||
| GWC-17 | Low | — | M | — | Not started | Sessions layer throws `Grpc.Core.RpcException` (layering leak) |
|
||||
| GWC-18 | Low | — | S | — | Not started | `GatewaySession` implements `DisposeAsync` without `IAsyncDisposable` |
|
||||
| GWC-19 | Low | — | S | — | Not started | Dead `GatewaySession.KillWorker(string)` + stale doc |
|
||||
@@ -119,7 +119,7 @@ Full design + implementation for each row lives in the linked domain doc under i
|
||||
| IPC-05 | Medium | P2 | S | — | Done | Redundant deep copies on command and event hot paths |
|
||||
| IPC-06 | Medium | P2 | S | — | Done | `gateway.md` Worker Envelope sketch no longer matches the contract |
|
||||
| IPC-07 | Medium | P2 | S | — | Done | `docs/Grpc.md` says six RPCs; there are seven |
|
||||
| IPC-08 | Medium | — | M | — | Not started | `WorkerCancel` defined and handled but never sent |
|
||||
| IPC-08 | Medium | — | M | — | Done | `WorkerCancel` defined and handled but never sent — resolved by `7c1ea12`/`a1a38b5`; residual (caller-cancellation leg) tracked as cycle-3 IPC-34 |
|
||||
| IPC-09 | Medium | P1 | M | IPC-01 | Done | Known codegen fragilities have no in-repo guards |
|
||||
| IPC-10 | Low | — | S | — | Not started | Envelope `sequence` is write-only; monotonicity never validated |
|
||||
| IPC-11 | Low | — | S | — | Not started | No protocol version negotiation despite `supported_protocol_version` name |
|
||||
@@ -218,7 +218,7 @@ Full design + implementation for each row lives in the linked domain doc under i
|
||||
| TST-03 | High | P1 | M | — | Done | No CI exists |
|
||||
| TST-04 | High | P2 | L | — | Done | Session-resilience epic 16/28 tasks unfinished |
|
||||
| TST-05 | Medium | P1 | S | TST-03 | Partially done | Real-worker control/COM paths verified opt-in only. **Scheduling half closed 2026-08-10** by the TST-25 `nightly-windev` job (`.gitea/workflows/ci.yml`, cron `0 6 * * *` → `scripts/ci/run-windev-ci.sh live`, which sets `MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1`, runs `WorkerLiveMxAccessSmokeTests`, and opens a Gitea issue on failure) — "opt-in, run by memory" is now "runs nightly, reports failures". **Residual: the coverage-audit half.** The live suite's 8 facts cover all six late-added COM commands but none of the five control commands (`Ping`, `GetSessionState`, `GetWorkerInfo`, `DrainEvents`, `ShutdownWorker`), which real workers answer in `Worker/Ipc/WorkerPipeSession.cs` yet are still only exercised through `FakeWorkerHarness` canned replies — precisely the masking the finding named |
|
||||
| TST-06 | Medium | — | M | — | Not started | Dashboard live-data path untested |
|
||||
| TST-06 | Medium | — | M | — | Materially narrowed | Dashboard live-data path untested — `DashboardLiveDataServiceTests`, `EventsHubTests`, `EventsHubViewerRegistryTests`, `AlarmsHubPublisherTests`, `SessionDetailsPageEventAclTests` now exist (landed with SEC-25/TST-16); needs a re-audit-and-close pass, not re-implementation |
|
||||
| TST-07 | Medium | — | S | — | Not started | Real-clock sleeps with negative assertions are latent flakes |
|
||||
| TST-08 | Medium | P1 | M | — | Done | Full-suite orphaned testhost processes (does not reproduce; doc de-stale) |
|
||||
| TST-09 | Medium | — | M | — | Not started | Health checks cover only the auth store |
|
||||
@@ -229,7 +229,7 @@ Full design + implementation for each row lives in the linked domain doc under i
|
||||
| TST-14 | Medium | P2 | S | — | Not started | Repo-root working artifacts need triage |
|
||||
| TST-15 | Medium | P2 | M | TST-04 | Done | Dashboard EventsHub has no per-session ACL |
|
||||
| TST-16 | Medium | P1 | S | — | Done | `Dashboard:ShowTagValues` is a dead flag |
|
||||
| TST-17 | Medium | — | S | — | Not started | Vendor-gated alarm parity residuals silently lossy |
|
||||
| TST-17 | Medium | — | S | — | Materially changed | Vendor-gated alarm parity residuals silently lossy — `WnWrapAlarmConsumer.cs:309-327` now passes `szDomainName`/`szOprFullName`; the vendor 8-arg leg is recorded "unavailable by configuration" with evidence (`docs/AlarmProbeFindings.md`, `d1ae43d`, `540a118`); "silent drop, unchanged" is no longer accurate — needs a worker-domain refresh pass |
|
||||
| TST-18 | Low | — | S | — | Not started | Hosted-service wrappers untested |
|
||||
| TST-19 | Low | — | S | — | Not started | Keep FakeWorkerHarness canned replies in lockstep |
|
||||
| TST-20 | Low | — | S | — | Not started | E2E script papers over a real advise-without-consumer sharp edge |
|
||||
|
||||
@@ -330,6 +330,8 @@ Open question the review cannot settle: whether a future probe (COM-init readine
|
||||
|
||||
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server`; `dotnet test src/ZB.MOM.WW.MxGateway.Tests --filter "FullyQualifiedName~SessionManager"`.
|
||||
|
||||
**Resolution: Won't fix (2026-08-20, closed by cycle-3 verification).** `Grpc/MxAccessGatewayService.cs:111-115` carries a `PERF(followup)` comment documenting the double session lookup as measured-negligible and deliberately not collapsed — the fix would duplicate `InvokeAsync`'s fault mapping at a second entry point, trading a negligible per-command cost for a maintenance liability. Per the cycle-3 gateway-core review (`archreview/2026-08-19/10-gateway-core.md`, GWC-16 row), the rationale is sound; tracker flipped from `Not started` to `Won't fix`.
|
||||
|
||||
---
|
||||
|
||||
## GWC-17 — Sessions layer throws `Grpc.Core.RpcException` `Low` · `—`
|
||||
|
||||
@@ -175,6 +175,8 @@ The frame protocol and proto evolution hygiene are sound; the remediable risk cl
|
||||
|
||||
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server` + `dotnet test ...Tests --filter FullyQualifiedName~WorkerClient`; worker side `-p:Platform=x86`.
|
||||
|
||||
**Resolution: Done (2026-08-20, closed by cycle-3 verification, residual filed).** Wired up substantially as designed: `InvokeAsync` forwards a `WorkerCancel` when the gateway's own command timeout fires and it owned the removal (`Server/Workers/WorkerClient.cs:269-291`), `TrySendCancelForTimedOutCommand` is structurally no-throw (`:835-870`), the worker routes it to `CancelCommand` (`Worker/Ipc/WorkerPipeSession.cs:755-756` → `MxAccess/MxAccessStaSession.cs:529-532` → `Sta/StaCommandDispatcher.CancelQueuedCommand:151`), regression test `WorkerClientTests.InvokeAsync_WhenCommandTimesOut_SendsWorkerCancelForThatCorrelation` exists, and `docs/GatewayProcessDesign.md:596-602` documents it. Landed in `7c1ea12`/`a1a38b5`. **Residual, not covered by this closure:** the caller-cancellation leg (context deadline / client disconnect, as opposed to the gateway's own `CommandTimeout`) sends nothing — filed as cycle-3 **IPC-34**. Per `archreview/2026-08-19/30-contracts-ipc.md` (IPC-08 row), tracker flipped from `Not started` to `Done` with the residual cross-referenced.
|
||||
|
||||
---
|
||||
|
||||
## IPC-09 — Known codegen fragilities have no in-repo guards `Medium` · `P1`
|
||||
|
||||
@@ -191,6 +191,8 @@ This finding is the umbrella; TST-01/02/15 are its actionable slices. The remedi
|
||||
|
||||
**Verification.** `dotnet test ... --filter FullyQualifiedName~DashboardLiveDataServiceTests`.
|
||||
|
||||
**Status refresh (2026-08-20, per cycle-3 verification).** Materially narrowed — the tracker row was stale. `Tests/Gateway/Dashboard/DashboardLiveDataServiceTests.cs`, `EventsHubTests.cs`, `EventsHubViewerRegistryTests.cs`, `AlarmsHubPublisherTests.cs`, and `SessionDetailsPageEventAclTests.cs` all exist now (mostly landed with the SEC-25/TST-16 closeouts and the in-process dashboard-feed work). Per `archreview/2026-08-19/60-testing-docs-gaps.md` (TST-06 row), this needs a re-audit-and-close pass against the current coverage rather than fresh implementation — left open pending that audit, tracker status changed from `Not started` to `Materially narrowed` to reflect the actual state.
|
||||
|
||||
---
|
||||
|
||||
## TST-07 — Real-clock sleeps with negative assertions are latent flakes `Medium` · `—`
|
||||
@@ -371,6 +373,8 @@ If TST-02's interim mitigation (flip retention off) is chosen instead of impleme
|
||||
|
||||
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Worker -p:Platform=x86`; `dotnet test src/ZB.MOM.WW.MxGateway.Worker.Tests/... -p:Platform=x86` (Windows).
|
||||
|
||||
**Status refresh (2026-08-20, per cycle-3 verification).** Materially changed — the tracker row needs a refresh from the worker domain. `WnWrapAlarmConsumer.cs:309-327` now passes `szDomainName`/`szOprFullName` into the COM call (the `_ = ackOperatorDomain` discard is gone), and the 2026-08 probe cycle recorded the vendor 8-arg leg as "unavailable by configuration" with evidence (`docs/AlarmProbeFindings.md`, commits `d1ae43d`, `540a118`). Per `archreview/2026-08-19/60-testing-docs-gaps.md` (TST-17 row), "silent drop, unchanged" is no longer an accurate description of the row; whether a caller-visible degrade signal now exists is the worker reviewer's call to close out. Tracker status changed from `Not started` to `Materially changed`.
|
||||
|
||||
---
|
||||
|
||||
## TST-18 — Hosted-service wrappers untested `Low` · `—`
|
||||
|
||||
@@ -19,7 +19,7 @@
|
||||
<PropertyGroup>
|
||||
<IsPackable>true</IsPackable>
|
||||
<PackageId>ZB.MOM.WW.MxGateway.Client</PackageId>
|
||||
<Version>0.2.0</Version>
|
||||
<Version>0.2.1</Version>
|
||||
<Description>.NET 10 gRPC client for the MxAccessGateway service. Provides typed wrappers, retry, and a lazy-browse walker over the Galaxy Repository hierarchy.</Description>
|
||||
<PackageReadmeFile>README.md</PackageReadmeFile>
|
||||
<!-- Only the shipped library generates XML docs (matching src/Contracts). The Cli and
|
||||
|
||||
@@ -3,7 +3,7 @@ package mxgateway
|
||||
const (
|
||||
// ClientVersion is the released semantic version of this Go client module.
|
||||
// Keep it in sync with the module tag applied by scripts/tag-go-module.ps1.
|
||||
ClientVersion = "0.2.0"
|
||||
ClientVersion = "0.2.1"
|
||||
|
||||
// GatewayProtocolVersion matches GatewayContractInfo.GatewayProtocolVersion
|
||||
// in the shared .NET contracts.
|
||||
|
||||
@@ -483,7 +483,7 @@ repositories {
|
||||
}
|
||||
|
||||
dependencies {
|
||||
implementation 'com.zb.mom.ww.mxgateway:zb-mom-ww-mxgateway-client:0.2.1'
|
||||
implementation 'com.zb.mom.ww.mxgateway:zb-mom-ww-mxgateway-client:0.2.2'
|
||||
}
|
||||
````
|
||||
|
||||
|
||||
@@ -16,11 +16,14 @@ subprojects {
|
||||
// 0.2.0 was already published to the Gitea Maven feed on 2026-06-26,
|
||||
// before the CLI-37/38/40/41 conformance fixes changed the client's
|
||||
// observable behavior (status.category-based validation, hresult < 0,
|
||||
// exact-secret redaction, typed malformed-reply errors). Bump to 0.2.1
|
||||
// so the published coordinate matches the conformant behavior the other
|
||||
// four clients ship at 0.2.0 for the first time. See CLI-39 and the
|
||||
// exact-secret redaction, typed malformed-reply errors). Bumped to 0.2.1
|
||||
// so the published coordinate matched the conformant behavior the other
|
||||
// four clients shipped at 0.2.0 for the first time. See CLI-39 and the
|
||||
// "Versioning" section of docs/ClientPackaging.md.
|
||||
version = '0.2.1'
|
||||
// 0.2.1 was itself published (Maven, 2026-08-07); this is the next
|
||||
// release-train bump (CLI-46), one ahead of the other four clients'
|
||||
// 0.2.1 because the Java coordinate has always trailed by one number.
|
||||
version = '0.2.2'
|
||||
|
||||
pluginManager.withPlugin('java') {
|
||||
java {
|
||||
|
||||
+2
-2
@@ -60,7 +60,7 @@ final class MxGatewayCliTests {
|
||||
|
||||
assertEquals(0, run.exitCode());
|
||||
assertEquals("", run.errors());
|
||||
assertTrue(run.output().contains("mxgateway-java 0.2.1"));
|
||||
assertTrue(run.output().contains("mxgateway-java 0.2.2"));
|
||||
assertTrue(run.output().contains("gatewayProtocolVersion=3"));
|
||||
assertTrue(run.output().contains("workerProtocolVersion=1"));
|
||||
}
|
||||
@@ -90,7 +90,7 @@ final class MxGatewayCliTests {
|
||||
CliRun run = execute(new FakeClientFactory(), "version", "--json");
|
||||
|
||||
assertEquals(0, run.exitCode());
|
||||
assertTrue(run.output().contains("\"clientVersion\":\"0.2.1\""));
|
||||
assertTrue(run.output().contains("\"clientVersion\":\"0.2.2\""));
|
||||
assertTrue(run.output().contains("\"gatewayProtocolVersion\":3"));
|
||||
}
|
||||
|
||||
|
||||
+1
-1
@@ -9,7 +9,7 @@ package com.zb.mom.ww.mxgateway.client;
|
||||
public final class MxGatewayClientVersion {
|
||||
private static final int GATEWAY_PROTOCOL_VERSION = 3;
|
||||
private static final int WORKER_PROTOCOL_VERSION = 1;
|
||||
private static final String CLIENT_VERSION = "0.2.1";
|
||||
private static final String CLIENT_VERSION = "0.2.2";
|
||||
|
||||
private MxGatewayClientVersion() {
|
||||
}
|
||||
|
||||
@@ -484,6 +484,15 @@ mxgw-py smoke --endpoint $env:MXGATEWAY_ENDPOINT --plaintext --api-key-env MXGAT
|
||||
|
||||
## Installing from the Gitea PyPI Feed
|
||||
|
||||
**The published `0.2.0` artifact cannot connect.** Its `connect()` built the
|
||||
`grpc.aio` channel inside `asyncio.to_thread`, and a `grpc.aio` channel binds
|
||||
to the event loop current on the *constructing* thread — so every
|
||||
non-stub connection raised `RuntimeError: There is no current event loop in
|
||||
thread`. Fixed in-tree by commit `a8f86b5` (`resolve_channel_security` now
|
||||
runs off-loop, `create_channel` runs on-loop). The fix is superseded by
|
||||
`0.2.1`; do not install `0.2.0` from the feed below — use `0.2.1` or later,
|
||||
or install from source at a commit at or after `a8f86b5`.
|
||||
|
||||
The client publishes to the internal Gitea PyPI feed:
|
||||
|
||||
````bash
|
||||
|
||||
@@ -6,7 +6,7 @@ build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "zb-mom-ww-mxaccess-gateway-client"
|
||||
version = "0.2.0"
|
||||
version = "0.2.1"
|
||||
description = "Async Python client for MXAccess Gateway."
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.12"
|
||||
|
||||
@@ -1,3 +1,3 @@
|
||||
"""Package version information."""
|
||||
|
||||
__version__ = "0.2.0"
|
||||
__version__ = "0.2.1"
|
||||
|
||||
Generated
+2
-2
@@ -590,7 +590,7 @@ checksum = "1d87ecb2933e8aeadb3e3a02b828fed80a7528047e68b4f424523a0981a3a084"
|
||||
|
||||
[[package]]
|
||||
name = "mxgw-cli"
|
||||
version = "0.2.0"
|
||||
version = "0.2.1"
|
||||
dependencies = [
|
||||
"clap",
|
||||
"futures-util",
|
||||
@@ -1490,7 +1490,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "zb-mom-ww-mxgateway-client"
|
||||
version = "0.2.0"
|
||||
version = "0.2.1"
|
||||
dependencies = [
|
||||
"futures-core",
|
||||
"futures-util",
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
[package]
|
||||
name = "zb-mom-ww-mxgateway-client"
|
||||
version = "0.2.0"
|
||||
version = "0.2.1"
|
||||
edition = "2021"
|
||||
authors = ["Joseph Doherty"]
|
||||
description = "Async Rust client for the MxAccessGateway gRPC service, including a lazy-browse walker over the Galaxy Repository hierarchy."
|
||||
@@ -25,7 +25,7 @@ resolver = "2"
|
||||
|
||||
[workspace.package]
|
||||
edition = "2021"
|
||||
version = "0.2.0"
|
||||
version = "0.2.1"
|
||||
authors = ["Joseph Doherty"]
|
||||
license = "Proprietary"
|
||||
repository = "https://gitea.dohertylan.com/dohertj2/mxaccessgw"
|
||||
|
||||
@@ -835,7 +835,7 @@ carries `degraded = true`.
|
||||
### Watch-list discovery
|
||||
|
||||
`GatewayAlarmMonitor` resolves the subtag watch-list at subscribe time by
|
||||
calling `IAlarmWatchListResolver.GetAlarmAttributesAsync`. The resolver merges:
|
||||
calling `IAlarmWatchListResolver.ResolveAsync`. The resolver merges:
|
||||
|
||||
1. Galaxy Repository SQL (`GetAlarmAttributesAsync`) — objects that have alarm
|
||||
extensions in the configured area.
|
||||
@@ -847,8 +847,42 @@ The resolved list is a set of `AlarmSubtagTarget` messages sent to the worker
|
||||
inside `SubscribeAlarmsCommand.watch_list`. Each target carries the composed
|
||||
MXAccess item addresses for the `InAlarm`, `Acked`, `AckMsg`, and `Priority`
|
||||
subtags (confirmed AVEVA `AlarmExtension` field names, verified against the live
|
||||
ZB Galaxy `attribute_definition` rows). The gateway re-runs discovery on its
|
||||
reconcile cadence and pushes an updated watch-list when the model changes.
|
||||
ZB Galaxy `attribute_definition` rows).
|
||||
|
||||
**The list is fixed for the life of the subscription.** The worker builds its
|
||||
subtag consumer from the targets `SubscribeAlarms` carried and keeps them until
|
||||
the subscription ends; a second `SubscribeAlarms` on a subscribed session is
|
||||
rejected, and the only way to replace the list — `UnsubscribeAlarms` then
|
||||
`SubscribeAlarms` — rebuilds `FailoverAlarmConsumer`, restarting it on the
|
||||
primary and discarding the warm standby state. Re-sending on a failover would
|
||||
therefore flap the feed rather than repair it. A monitor restart (a worker fault,
|
||||
or a gateway restart) re-resolves and re-sends, and that is the supported way to
|
||||
pick up Galaxy objects added since the monitor started.
|
||||
|
||||
Because the list is fixed, the resolution is **graded** rather than merely
|
||||
logged (GWC-37). `ResolveAsync` returns an `AlarmWatchListResolution` carrying
|
||||
whether Galaxy Repository discovery was attempted and whether it failed, and the
|
||||
monitor folds the verdict into the alarm feed's `AlarmProviderStatus`:
|
||||
|
||||
- Zero targets while the effective mode can need them (`Auto` or `ForceSubtag`)
|
||||
sets `degraded = true` with reason `AlarmProviderReasons.SubtagWatchListEmpty`
|
||||
— the subtag provider covers no alarms.
|
||||
- Discovery enabled but failing adds
|
||||
`AlarmProviderReasons.SubtagDiscoveryFailed` and the failure message, so a
|
||||
Galaxy Repository outage at startup is a standing status rather than one log
|
||||
line emitted possibly weeks before the failover that needs the list.
|
||||
- On the failover edge (`ApplyProviderModeChangeAsync` with `toMode == Subtag`)
|
||||
the monitor re-resolves and compares against what it sent. A difference is
|
||||
reported as `AlarmProviderReasons.SubtagWatchListStale` with both counts, on
|
||||
the same status frame as the failover, because it cannot be applied.
|
||||
|
||||
`degraded` on the status is therefore the OR of "the subtag provider is the one
|
||||
serving" and "the subtag watch-list cannot cover anything". In `Auto` the second
|
||||
can be true while the mode is still `ALARMMGR`: the live feed is healthy and the
|
||||
standby behind it is empty. The dashboard renders that combination as the amber
|
||||
`Alarm Manager (no subtag standby)` badge rather than the failover label, so the
|
||||
badge never names the wrong live provider. `ForceAlarmManager` never builds a
|
||||
subtag consumer, so its watch-list is not graded at all.
|
||||
|
||||
Each target's canonical `AlarmFullReference` is composed as
|
||||
`Galaxy!{area}.{reference}` (literal `Galaxy` provider). The `{area}` is the
|
||||
|
||||
@@ -180,6 +180,21 @@ Supported constraints are:
|
||||
| `read_historized_only` | Read/subscription commands must target objects with historized attributes. |
|
||||
| `dashboard_tags` | Dashboard event-visibility tags. **Not a data-access constraint** — see below. |
|
||||
|
||||
`browse_subtrees` is applied by `GatewayBrowseScopeProvider`, the host seam the
|
||||
shared `ZB.MOM.WW.GalaxyRepository` package calls through `IGalaxyBrowseScopeProvider`;
|
||||
it reads the constraint off the ambient per-call identity that
|
||||
`GatewayGrpcAuthorizationInterceptor` pushed earlier in the same call. An empty
|
||||
`browse_subtrees` list is the library's "no scoping" signal, so the provider has to
|
||||
distinguish two states that look alike (GWC-38): an authenticated key with no browse
|
||||
constraint is legitimately unscoped and returns the empty list, whereas **no ambient
|
||||
identity at all** means the interceptor did not run before the Galaxy service and the
|
||||
request's scope cannot be established. In that second case, with
|
||||
`MxGateway:Authentication:Mode = ApiKey`, the provider logs a warning naming the broken
|
||||
ordering invariant and fails the call with `PermissionDenied` rather than serving the full
|
||||
hierarchy — a narrowing control's silent absence is invisible in normal operation, so it is
|
||||
made loud instead. Under `Mode = Disabled` an absent identity is the expected steady state
|
||||
(nothing authenticates, so no key and no constraints exist) and browse stays unscoped.
|
||||
|
||||
`dashboard_tags` is the one member of the blob that constrains nothing on the
|
||||
gRPC data path. No read, write, browse, or subscribe check consults it, and
|
||||
`HasReadConstraints` / `HasWriteConstraints` deliberately ignore it: adding a tag
|
||||
|
||||
@@ -72,6 +72,23 @@ carries the equivalent guard for the Go module: it refuses to create a
|
||||
`ClientVersion` already equals `X.Y.Z` (CLI-21/CLI-39), so a forgotten
|
||||
version bump fails the tag instead of shipping a mismatched module.
|
||||
|
||||
Both of those guards only fire when someone actually runs a publish.
|
||||
CLI-46 found the same hazard recurring silently *between* publishes: a
|
||||
real bug fix (the pypi 0.2.0 `connect()` fix above, commit `a8f86b5`) plus
|
||||
later wire-surface changes shipped into the tree while every version
|
||||
constant still read the already-published number, and nothing recorded
|
||||
that the registry artifact and the in-repo source had diverged.
|
||||
`scripts/check-client-versions.ps1` is the PR-time half of that guard —
|
||||
wired into CI (`.gitea/workflows/ci.yml`, the `portable` job, next to the
|
||||
codegen freshness checks): for each client it finds the latest publish tag
|
||||
(`clients/go/vX.Y.Z` for Go — the only client with a git tag at all; .NET,
|
||||
Python, Rust, and Java publish straight to a package registry via
|
||||
`pack-clients.ps1 -Publish` with no corresponding git tag, so the check
|
||||
reports "no tags found" and passes for those four today) and fails if
|
||||
`clients/<lang>` differs from that tag while the version constant still
|
||||
equals the tag's version. A client with no publish tag is not a
|
||||
violation — there is no "since the last publish" window to check yet.
|
||||
|
||||
As of 2026-08-07 (CLI-39) all five clients — plus `ZB.MOM.WW.MxGateway.Contracts`,
|
||||
which releases in lockstep with the .NET Client — moved to **0.2.0**,
|
||||
converging on one number after four of the five had drifted onto the
|
||||
@@ -111,6 +128,26 @@ confirmed present through the Gitea package API after the push, and
|
||||
`go list -m` resolves the Go tag. These are the numbers a future release
|
||||
bumps off.
|
||||
|
||||
**The published pypi `0.2.0` artifact has a broken `connect()` (CLI-46).**
|
||||
Its `connect()` built the `grpc.aio` channel inside `asyncio.to_thread`, and
|
||||
a `grpc.aio` channel binds to the event loop current on the *constructing*
|
||||
thread, so every non-stub connection raised `RuntimeError: There is no
|
||||
current event loop in thread` — TST-24's first real-wire run caught it.
|
||||
Fixed in-tree by commit `a8f86b5` (2026-08-10): `resolve_channel_security`
|
||||
now runs off-loop, `create_channel` runs on-loop. That fix, plus later
|
||||
wire-surface additions (`d9ea8a8`, `c748361`), shipped after the 2026-08-07
|
||||
publish while the version constants still read the already-published
|
||||
0.2.0/0.2.1 numbers above — see `clients/python/README.md` for the
|
||||
user-facing warning. **0.2.0 is superseded by 0.2.1; do not install it.**
|
||||
|
||||
As of this remediation (CLI-46), the in-source version constants for the
|
||||
.NET, Go, Python, and Rust clients (plus Contracts) read **0.2.1**, and the
|
||||
Java client reads **0.2.2**. These are not yet published — the numbers
|
||||
above (0.2.0 four clients + Contracts, 0.2.1 Java) remain the live registry
|
||||
state until the next `pack-clients.ps1` / `tag-go-module.ps1` run. Once that
|
||||
publish lands, update this section with the new coordinates and commit(s),
|
||||
the same way the 2026-08-07 record above was written after the fact.
|
||||
|
||||
## .NET
|
||||
|
||||
The .NET client uses .NET 10 and references
|
||||
|
||||
@@ -194,7 +194,7 @@ events (a "gap") and must re-snapshot; whatever is still retained is replayed.
|
||||
| Option | Default | Description |
|
||||
|--------|---------|-------------|
|
||||
| `MxGateway:Dashboard:Enabled` | `true` | Enables Blazor Server dashboard route mapping. The dashboard mounts at the host root (`/`); there is no separate path-base prefix. |
|
||||
| `MxGateway:Dashboard:AllowAnonymousLocalhost` | `true` | Allows loopback dashboard requests to bypass the dashboard cookie requirement for local development. Remote requests still require dashboard authentication. |
|
||||
| `MxGateway:Dashboard:AllowAnonymousLocalhost` | `true` | Allows loopback dashboard requests to bypass the dashboard cookie requirement for local development. Remote requests still require dashboard authentication. The bypass is **read-only** — it satisfies a requirement including the `Viewer` role but never `AdminOnly`, so API-key CRUD and session Close/Kill stay closed — and since SEC-40 it does **not** reach the alarm feed: `/alarms` and `/hubs/alarms` carry `MxGateway.Dashboard.Alarms`, a policy built from a requirement type the bypass handler does not handle, so an anonymous loopback caller is denied the page, denied the hub, and not offered the Alarms link. The alarm monitor is gateway-central and its messages carry no session provenance, so alarm metadata cannot be tag-scoped the way session events are — requiring a real role claim is the narrowing that exists. Sign in (or use `DisableLogin` in dev, which mints a genuine principal) to see alarms; no setting re-opens the feed to the bypass. The same exclusion applies to `Authentication:Mode=Disabled`. |
|
||||
| `MxGateway:Dashboard:RequireHttpsCookie` | `true` | Sets the dashboard auth cookie's secure policy. `true` keeps `CookieSecurePolicy.Always` — the cookie is only sent over HTTPS, which matches a production HTTPS deployment. Set to `false` for plain-HTTP dev deployments to use `CookieSecurePolicy.SameAsRequest`; the cookie is still flagged Secure on HTTPS requests, but it can round-trip over HTTP. Browsers drop Secure cookies set over HTTP from non-localhost hosts, so leaving this `true` while serving the dashboard over plain HTTP will break login from any remote browser. |
|
||||
| `MxGateway:Dashboard:CookieName` | _(unset)_ | Dashboard auth cookie name. Leave unset (null/blank) to use the resolved default: `__Host-MxGatewayDashboard` when `RequireHttpsCookie` is `true` (Secure cookie, so the `__Host-` prefix's Secure/no-Domain/`Path=/` guarantees apply), else the plain `MxGatewayDashboard` (a `__Host-` cookie without Secure is silently dropped by browsers). Override it to give a distinct name to a gateway that shares a hostname with another gateway instance: browser cookies are scoped by host+path but **not** by port, so two instances on the same host would otherwise clobber each other's dashboard session under a shared cookie name. An explicit override always wins over the `__Host-` default. Changing it signs out existing dashboard sessions on next deploy. |
|
||||
| `MxGateway:Dashboard:SnapshotIntervalMilliseconds` | `1000` | Dashboard snapshot refresh interval used by the snapshot SignalR hub and the pages that subscribe to it. |
|
||||
@@ -202,7 +202,7 @@ events (a "gap") and must re-snapshot; whatever is still retained is replayed.
|
||||
| `MxGateway:Dashboard:RecentSessionLimit` | `200` | Maximum number of session summaries projected into each dashboard snapshot. |
|
||||
| `MxGateway:Dashboard:ShowTagValues` | `false` | Controls whether tag values reach the dashboard at all. `false` (the default) suppresses them on all three seams that carry one: (1) the **events hub mirror** — `DashboardEventBroadcaster` blanks `MxEvent.value` plus the alarm body's `current_value`/`limit_value` from a deep-cloned copy before it reaches any hub subscriber (see `docs/GatewayDashboardDesign.md`'s `EventsHub` row); (2) the **alarms hub** — `AlarmsHubPublisher` clears `current_value`/`limit_value` from a deep-cloned copy of each `AlarmFeedMessage`, on both value-bearing payload arms (`transition` and `active_alarm`), before broadcasting to `/hubs/alarms`; (3) the **`/browse` live-value panel** — `DashboardLiveDataService` substitutes the literal `[redacted]` for the value text of each **successfully read** tag; a failed read keeps its `-` placeholder (there was no value to suppress, and the row's error is left untouched, so the two cannot contradict each other). Both hub redactions clone: the source `MxEvent` is shared with the gRPC event stream and the replay ring, and the source `AlarmFeedMessage` fans out to gRPC `StreamAlarms` subscribers, so neither is mutated in place and **no gRPC client is affected by this flag** — it is a dashboard-display control only. Everything that is not the value survives on every seam: tag reference, alarm reference/severity/state/operator, data type, quality, status, and timestamps still render, so the dashboard stays diagnostic without disclosing process values. This is one of two independent layers: `IDashboardSessionAcl` decides *which* sessions a caller may subscribe to at all (see `GroupToTag` / `UntaggedSessionVisibility` below), while this flag decides what a permitted subscriber sees. Setting it `true` exposes tag values to everyone the ACL admits — every Administrator, plus each Viewer holding a matching tag — and, because the alarms hub is session-less, exposes alarm values to every dashboard client that can reach `/hubs/alarms`. |
|
||||
| `MxGateway:Dashboard:GroupToRole` | _(empty)_ | LDAP group → dashboard role mapping. Keys must be **short** LDAP group names — the CN / first-RDN value, e.g. `GwAdmin`, not `ou=GwAdmin,ou=groups,dc=zb,dc=local` — matched case-insensitively. The shared `ZB.MOM.WW.Auth.Ldap` provider delivers a user's groups already stripped to short names, so a full-DN key can never match and the group silently maps to nothing. (The mapper does try the full literal string before falling back to the leading-RDN value, but on the live login path both branches see the same short name; the fallback only matters to non-library callers of the `IGroupRoleMapper<string>` seam.) Values must be `Administrator` (read/write, API-key CRUD) or `Viewer` (read-only) — matched ordinally by the startup validator, so the spelling is exact and `Admin` is rejected. A user whose LDAP groups don't intersect this map cannot sign in; with no mapping at all, only the loopback bypass admits anyone. |
|
||||
| `MxGateway:Dashboard:GroupToTag` | _(empty)_ | LDAP group → dashboard visibility tags. Keys follow the same convention as `GroupToRole` — short LDAP group names (CN / first-RDN value), matched case-insensitively; a full-DN key never matches the pre-stripped groups the LDAP provider returns. Values are tag lists. A dashboard user's granted tag set is the union over the groups they belong to; an unmapped group contributes nothing. **Visibility only:** tags scope which sessions' event streams a Viewer may observe on the dashboard — they never grant or deny data access, which stays with the API key's scopes and constraints. Independent of `GroupToRole`: a group may appear in either map, both, or neither. Empty (the default) means Viewers hold no tags, so under the default `UntaggedSessionVisibility` they observe no session's events. |
|
||||
| `MxGateway:Dashboard:GroupToTag` | _(empty)_ | LDAP group → dashboard visibility tags. Keys follow the same convention as `GroupToRole` — short LDAP group names (CN / first-RDN value), matched case-insensitively; a full-DN key never matches the pre-stripped groups the LDAP provider returns. Values are tag lists. A dashboard user's granted tag set is the union over the groups they belong to; an unmapped group contributes nothing. **Visibility only:** tags scope which sessions' event streams a Viewer may observe on the dashboard — they never grant or deny data access, which stays with the API key's scopes and constraints. Independent of `GroupToRole`: a group may appear in either map, both, or neither. Empty (the default) means Viewers hold no tags, so under the default `UntaggedSessionVisibility` they observe no session's events. **Applies to session event streams only:** the alarm feed (`/alarms`, `/hubs/alarms`) is deliberately *not* tag-scoped — the alarm monitor is gateway-central and session-less, so an `AlarmFeedMessage` carries no session or owning key to intersect a tag grant against. Alarm metadata (reference, state, severity, category, area, acknowledging operator) is therefore galaxy-wide for every authenticated dashboard role, regardless of tag grant; alarm *values* still follow `ShowTagValues`. What SEC-40 narrowed there is admission rather than scope: the environment bypasses (`AllowAnonymousLocalhost`, `Authentication:Mode=Disabled`) no longer receive the feed at all. **Revocation:** the ACL re-resolves a caller's grant from their LDAP group claims against this map on every subscribe, so removing an entry takes effect at the next subscribe after the gateway restart that rebinds options (options bind once at startup); a change to the user's LDAP group membership takes effect when their principal is rebuilt — at re-login for cookie sessions, within one 5-minute hub-token lifetime for token clients. An already-open subscription is never re-checked and lives until it drops. |
|
||||
| `MxGateway:Dashboard:UntaggedSessionVisibility` | `AdminOnly` | Who may observe a session that carries no tags (its owning API key declared none). `AdminOnly` (default, fail-closed) restricts untagged sessions to dashboard Administrators. `AllViewers` shows them to every Viewer — opt-in for a single-tenant deployment that wants the pre-tag behaviour. Administrators always see every session regardless of tags. |
|
||||
| `MxGateway:Dashboard:DisableLogin` | `false` | Dev/test only. When `true`, replaces the cookie authentication handler with `DashboardAutoLoginAuthenticationHandler`, which auto-authenticates every dashboard request — including requests from remote browsers, not just loopback — as `AutoLoginUser` holding both `Administrator` and `Viewer` roles. No login form, LDAP bind, or cookie is involved. A loud one-time startup warning is logged. Differs from `AllowAnonymousLocalhost`: `DisableLogin` mints a real authenticated principal (so role-gated write affordances appear), whereas `AllowAnonymousLocalhost` satisfies the authorization requirement on loopback only without minting a principal (write affordances stay hidden). Never enable in production. **Production hard-stop (SEC-04):** when the host runs in the `Production` environment and `DisableLogin` is `true`, startup validation fails and the process aborts — the flag is only accepted outside Production, where the one-time startup warning still fires. |
|
||||
| `MxGateway:Dashboard:AutoLoginUser` | `(null)` | Username stamped on the synthetic principal when `DisableLogin` is `true`. Default `(null)` — a null or blank value falls back to `multi-role`. Has no effect when `DisableLogin` is `false`. |
|
||||
@@ -453,11 +453,11 @@ native wnwrap alarm-manager provider and the subtag-monitoring fallback.
|
||||
|
||||
| Option | Default | Description |
|
||||
|--------|---------|-------------|
|
||||
| `MxGateway:Alarms:Fallback:Mode` | `Auto` | Provider selection mode. `Auto` uses the alarm manager as primary and fails over to subtag monitoring after consecutive COM failures, then fails back automatically. `ForceAlarmManager` disables failover. `ForceSubtag` forces subtag monitoring on from startup. Values are case-insensitive. |
|
||||
| `MxGateway:Alarms:Fallback:Mode` | `Auto` | Provider selection mode. `Auto` uses the alarm manager as primary and fails over to subtag monitoring after consecutive COM failures, then fails back automatically. `ForceAlarmManager` disables failover (and is the one mode whose subtag watch-list is never graded, because no subtag consumer is built). `ForceSubtag` forces subtag monitoring on from startup. Values are case-insensitive. |
|
||||
| `MxGateway:Alarms:Fallback:ConsecutiveFailureThreshold` | `3` | Number of consecutive wnwrap COM failures (`COMException` or failure HRESULT from `Subscribe` / `GetXmlCurrentAlarms2`) before the monitor switches to subtag mode. Floored at 1. |
|
||||
| `MxGateway:Alarms:Fallback:FailbackProbeIntervalSeconds` | `30` | While in subtag mode, how often (in seconds) the monitor probes the wnwrap provider to detect recovery. Floored at 1. |
|
||||
| `MxGateway:Alarms:Fallback:FailbackStableProbes` | `3` | Number of consecutive clean wnwrap probes required before the monitor switches back to the alarm manager. Floored at 1. |
|
||||
| `MxGateway:Alarms:Fallback:Discovery:UseGalaxyRepository` | `true` | When `true`, the monitor queries the Galaxy Repository SQL database to build the subtag watch-list for the configured area. |
|
||||
| `MxGateway:Alarms:Fallback:Discovery:UseGalaxyRepository` | `true` | When `true`, the monitor queries the Galaxy Repository SQL database to build the subtag watch-list for the configured area. A failed query is not fatal — the monitor continues with whatever configuration contributed — but with `Mode` `Auto` or `ForceSubtag` the resulting coverage gap is reported on the alarm feed's `AlarmProviderStatus` (`degraded = true` with a `Subtag watch-list is empty` / `Galaxy Repository alarm discovery failed` reason) and shown on the dashboard as the amber `Alarm Manager (no subtag standby)` badge, instead of only a startup log line. |
|
||||
| `MxGateway:Alarms:Fallback:Discovery:Area` | _(empty)_ | Galaxy area to scope the Repository query to. Falls back to `MxGateway:Alarms:DefaultArea` when empty. Ignored when `UseGalaxyRepository` is `false`. This area is **not** used to compose a Repository-derived alarm's canonical `Galaxy!{area}.{reference}`: each discovered alarm uses its object's real Galaxy area (discovered via `gobject.area_gobject_id`), so the reference's group matches what the native alarmmgr emits. `Discovery:Area` / `DefaultArea` is used as the composition area only for explicit `IncludeAttributes` entries, which carry no discovered area. |
|
||||
| `MxGateway:Alarms:Fallback:Discovery:IncludeAttributes` | _(empty)_ | Explicit MXAccess attribute paths to add to the subtag watch-list, supplementing (or replacing, when `UseGalaxyRepository` is `false`) the Repository-derived list. |
|
||||
| `MxGateway:Alarms:Fallback:Discovery:ExcludeAttributes` | _(empty)_ | Attribute paths to remove from the merged watch-list (case-insensitive). The exclude is applied after the Repository-derived rows and the explicit `IncludeAttributes` entries are combined, so an exclude that matches an explicit include suppresses it too — excludes win. Ignored when `UseGalaxyRepository` is `false`. |
|
||||
|
||||
@@ -274,7 +274,7 @@ bearer). Each hub class is `[Authorize(Policy = HubClientsPolicy)]`.
|
||||
| Hub | Path | Producer | Payload | Routing |
|
||||
|---|---|---|---|---|
|
||||
| `DashboardSnapshotHub` | `/hubs/snapshot` | `DashboardSnapshotPublisher` (BackgroundService consuming `IDashboardSnapshotService.WatchSnapshotsAsync`) | `DashboardSnapshot` | Sent to all connected clients on every snapshot tick, but only while at least one client is connected (see "Idle gating" below); new connections receive the current snapshot synchronously in `OnConnectedAsync`. |
|
||||
| `AlarmsHub` | `/hubs/alarms` | `AlarmsHubPublisher` (BackgroundService consuming `IGatewayAlarmService.StreamAsync(filter: null)`) | `AlarmFeedMessage` (`active_alarm` / `snapshot_complete` / `transition` / `provider_status` / `snapshot_status`) | Connected clients auto-join `__alarms__`; all clients receive every message. Publisher auto-reconnects every 5s on stream faults. The alarm value fields (`current_value` / `limit_value`) are stripped from a deep-cloned copy of the message when `Dashboard:ShowTagValues` is false (the default) — on both value-bearing payload arms, so neither a live transition nor a snapshot record leaks a process value; every other field still renders. The source message is never mutated: it fans out from the same feed to gRPC `StreamAlarms` subscribers, which this dashboard-display flag does not govern. Arms carrying no value are forwarded as-is, uncloned. |
|
||||
| `AlarmsHub` | `/hubs/alarms` | `AlarmsHubPublisher` (BackgroundService consuming `IGatewayAlarmService.StreamAsync(filter: null)`) | `AlarmFeedMessage` (`active_alarm` / `snapshot_complete` / `transition` / `provider_status` / `snapshot_status`) | Guarded by `MxGateway.Dashboard.Alarms`, not `HubClients`: a real authenticated dashboard role is required, so the `Authentication:Mode=Disabled` and anonymous-localhost bypasses are excluded (SEC-40). Connected clients auto-join `__alarms__`; all admitted clients receive every message — the feed is galaxy-wide and carries no session provenance, so no per-caller scoping applies (see "Alarm-feed visibility"). Publisher auto-reconnects every 5s on stream faults. The alarm value fields (`current_value` / `limit_value`) are stripped from a deep-cloned copy of the message when `Dashboard:ShowTagValues` is false (the default) — on both value-bearing payload arms, so neither a live transition nor a snapshot record leaks a process value; every other field still renders. The source message is never mutated: it fans out from the same feed to gRPC `StreamAlarms` subscribers, which this dashboard-display flag does not govern. Arms carrying no value are forwarded as-is, uncloned. |
|
||||
| `EventsHub` | `/hubs/events` | `DashboardEventBroadcaster` invoked by each session's internal dashboard-mirror subscriber on its `SessionEventDistributor` (registered when the session becomes Ready) | `MxEvent` | Clients call `SubscribeSession(sessionId)` to join `session:{id}`, which also registers them in `EventsHubViewerRegistry` — the mirror is gated on that registry, which counts hub and in-process viewers alike (see "Mirror gating" below). The dashboard is a first-class distributor subscriber, so it receives the session's events whether or not a gRPC client is streaming. It sees RAW session events — not the per-gRPC-subscriber `AfterWorkerSequence` filtering that `EventStreamService` applies at its own boundary — because the dashboard is a separate LDAP-authenticated monitoring view meant to show the session's full event activity. Tag values are stripped from the mirrored `MxEvent` copy by `DashboardEventBroadcaster` when `Dashboard:ShowTagValues` is false (the default) — event metadata (tag reference, quality, status, timestamps) still renders, but the value fields are blanked, so no value leaks through this seam. `SubscribeSession` is gated by `IDashboardSessionAcl` (SEC-25 / TST-15): a denied caller gets a `HubException`, is not joined to the group, and is not registered as a viewer, so the mirror stays off for a session nobody is legitimately watching. The same ACL gates the in-process seam the session-details page uses, so neither path is the weaker one. Value redaction remains an independent layer — it bounds what a *permitted* subscriber sees. |
|
||||
|
||||
### Default cadences
|
||||
@@ -556,6 +556,14 @@ re-asserts the truncation verdict as the reconcile baseline. If `MxGateway:Alarm
|
||||
false the central monitor never starts, and the page says so instead of showing
|
||||
an empty list with no explanation.
|
||||
|
||||
The page carries `[Authorize(Policy = MxGateway.Dashboard.Alarms)]` rather than relying on the
|
||||
`Viewer` policy `MapRazorComponents<App>()` applies to every component route (SEC-40), so it
|
||||
needs a real authenticated dashboard role: the `Authentication:Mode=Disabled` and
|
||||
anonymous-localhost bypasses reach every other read-only page but not this one. It is the same
|
||||
policy `AlarmsHub` carries — the two read one central monitor and must not disagree about who
|
||||
may see it — and the side rail hides the Alarms link for callers the policy denies. What a
|
||||
permitted caller sees is galaxy-wide and not tag-scoped; see "Alarm-feed visibility".
|
||||
|
||||
### Live data source
|
||||
|
||||
Both the Browse subscription panel and the Alarms page read live MXAccess data
|
||||
@@ -706,14 +714,25 @@ Implemented behavior:
|
||||
generic credential-rejected message;
|
||||
- antiforgery tokens guard the login and logout POSTs.
|
||||
|
||||
Three authorization policies are registered:
|
||||
Four authorization policies are registered:
|
||||
|
||||
- `MxGateway.Dashboard.Viewer` — Razor component routes. Satisfied by Admin or
|
||||
Viewer.
|
||||
- `MxGateway.Dashboard.Admin` — Admin-only write surfaces (API-key CRUD).
|
||||
- `MxGateway.Dashboard.HubClients` — SignalR hubs. Accepts the dashboard
|
||||
cookie OR a `MxGateway.Dashboard.HubToken` bearer (used by WebSocket upgrades
|
||||
where the cookie can't be forwarded).
|
||||
- `MxGateway.Dashboard.HubClients` — the snapshot and events SignalR hubs. Accepts
|
||||
the dashboard cookie OR a `MxGateway.Dashboard.HubToken` bearer (used by WebSocket
|
||||
upgrades where the cookie can't be forwarded).
|
||||
- `MxGateway.Dashboard.Alarms` — the alarm feed: `AlarmsHub` and the `/alarms`
|
||||
page. Same two schemes as `HubClients`, but built from
|
||||
`AuthenticatedDashboardRoleRequirement` instead of
|
||||
`DashboardAuthorizationRequirement` (SEC-40). Because `DashboardAuthorizationHandler`
|
||||
handles only the latter type, the read-only environment bypasses below cannot
|
||||
satisfy this policy — see "Alarm-feed visibility".
|
||||
|
||||
The first three requirement types are satisfiable by the environment bypasses;
|
||||
`MxGateway.Dashboard.Alarms` and the mounted Secrets UI's `secrets:manage` are not,
|
||||
both for the same structural reason — the bypass handler is typed to a requirement
|
||||
they do not use.
|
||||
|
||||
The in-process page feeds carry no authentication of their own, and need none:
|
||||
`MapRazorComponents<App>()` applies `RequireAuthorization(ViewerPolicy)` to the
|
||||
@@ -733,12 +752,30 @@ caller's granted tags, ordinal-ignore-case. A session's tags are inherited from
|
||||
owning API key's `--dashboard-tags` constraint and are immutable for the session's
|
||||
life, so a subscribe-time decision cannot go stale while the subscription lives and
|
||||
no per-event re-check is needed. A Viewer's grant comes from
|
||||
`Dashboard:GroupToTag` applied to their LDAP groups, stamped as
|
||||
`zb:dashboardtag` claims at cookie login and re-resolved (not copied) at hub-token
|
||||
mint, so the token's 5-minute lifetime bounds how long a revoked grant survives. A
|
||||
principal carrying no tag claims — anonymous localhost included — is an
|
||||
empty-grant Viewer. The mirror's value redaction is an independent layer: it bounds
|
||||
what a permitted subscriber sees, not who may subscribe.
|
||||
`Dashboard:GroupToTag` applied to their LDAP groups. Since SEC-41 the ACL **re-resolves**
|
||||
that grant on every subscribe, from the caller's `mxgateway:ldap_group` claims against the
|
||||
currently bound `GroupToTag` map — the same one `DashboardGroupTagMapping.MapGroupsToTags`
|
||||
call the hub-token mint makes — rather than trusting the `zb:dashboardtag` claims stamped
|
||||
into the cookie at login. Those stamped claims were resolved under whatever map was bound
|
||||
when the user signed in, and the auth cookie outlives the gateway restart that applies a
|
||||
`GroupToTag` edit, so trusting them let a revoked grant survive the restart meant to remove
|
||||
it. The stamped claims remain the **fallback** for a principal carrying no group claims,
|
||||
which in practice means the hub-token principal: the token payload carries roles and tags
|
||||
but no LDAP groups, and the mint already re-resolved them. That fallback cannot re-open the
|
||||
cookie hole, because `DashboardAuthenticator` stamps group and tag claims from one group
|
||||
list — a cookie principal holding a tag claim always holds the groups it came from, and a
|
||||
login yielding no groups yields no tags. A principal carrying neither kind of claim —
|
||||
anonymous localhost included — is an empty-grant Viewer. The mirror's value redaction is an
|
||||
independent layer: it bounds what a permitted subscriber sees, not who may subscribe.
|
||||
|
||||
**Revocation bounds.** Three different revocations propagate on three different clocks, and
|
||||
the difference is intrinsic to where each fact lives:
|
||||
|
||||
| Revocation | Takes effect |
|
||||
|---|---|
|
||||
| A `GroupToTag` entry removed from config | At the next subscribe after the gateway restart that rebinds options — identically on both seams, since both re-resolve against current options. `GatewayOptions` binds once at startup, so the restart is what applies the edit; nothing further is needed. |
|
||||
| An LDAP group-membership revocation | When the principal is next rebuilt: at re-login for the cookie seam, bounded by the cookie's sliding 8-hour idle life (renewable by activity); within one 5-minute token lifetime for the hub-token seam. Neither seam re-queries LDAP per subscribe. |
|
||||
| Either, for an **already-open** subscription | Never — the subscription lives until it drops. The ACL is a subscribe-time gate by design (session tags are immutable, so no per-event re-check is warranted); closing an in-flight stream on a grant change would need a revocation channel the dashboard does not have. Force it by closing the session, or by restarting the gateway, which tears down every circuit and hub connection. |
|
||||
|
||||
Two environmental bypasses still apply, both scoped to **read-only** access:
|
||||
`MxGateway:Authentication:Mode = Disabled` and `MxGateway:Dashboard:AllowAnonymousLocalhost`
|
||||
@@ -746,11 +783,50 @@ Two environmental bypasses still apply, both scoped to **read-only** access:
|
||||
role (`AnyDashboardRole`) but **never** the `AdminOnly` requirement. Destructive/admin
|
||||
surfaces (API-key CRUD, session Close, worker Kill) still demand a real `Administrator`
|
||||
role claim, so the "anonymous localhost is read-only" contract holds at the policy layer
|
||||
rather than only in downstream service re-checks. Remote requests (outside the `Disabled`
|
||||
rather than only in downstream service re-checks. Since SEC-40 the bypasses also do not
|
||||
reach the **alarm feed** (`MxGateway.Dashboard.Alarms`) — read-only is not the same as
|
||||
"may see the whole plant's alarm metadata"; see "Alarm-feed visibility" below. Remote requests (outside the `Disabled`
|
||||
bypass) always require an authenticated principal carrying at least the Viewer role. The
|
||||
loopback test trusts `Connection.RemoteIpAddress`; if forwarded-headers middleware is ever
|
||||
added upstream, `DashboardAuthorizationHandler.IsLoopbackRequest()` must be revisited.
|
||||
|
||||
### Alarm-feed visibility
|
||||
|
||||
The alarm feed is **not** tag-scoped, and that is a deliberate, bounded decision rather than
|
||||
an oversight in the SEC-25 ACL.
|
||||
|
||||
The gateway's alarm monitor is central and session-less: one subscription expression, one
|
||||
`IGatewayAlarmService` feed, consumed by `AlarmsHubPublisher` (which re-broadcasts every
|
||||
message to the `__alarms__` group) and by the `/alarms` page in process. An `AlarmFeedMessage`
|
||||
carries no session id and no owning API key, so there is nothing for the per-session tag
|
||||
intersection to intersect — a tag ACL cannot be applied to it without inventing provenance the
|
||||
feed does not have. Deriving a filter from the caller's tags would also mean equating a
|
||||
visibility tag with an alarm-reference prefix, which `GroupToTag` deliberately does not do
|
||||
(tags are visibility labels, never data-access constraints).
|
||||
|
||||
So **alarm metadata stays galaxy-wide for every authenticated dashboard Viewer**: alarm
|
||||
reference (a full tag path), state, severity, priority, category, area, and acknowledging
|
||||
operator are all visible to any real dashboard role, regardless of which sessions their tag
|
||||
grant covers. Alarm *values* (`current_value` / `limit_value`) are still stripped when
|
||||
`Dashboard:ShowTagValues` is false (the default), on the hub seam as on the events seam.
|
||||
|
||||
What SEC-40 tightened is the entry gate rather than the scope. Both readers of the monitor —
|
||||
`AlarmsHub` and the `/alarms` page — carry `MxGateway.Dashboard.Alarms`, whose requirement type
|
||||
the environment-bypass handler does not handle. Consequences:
|
||||
|
||||
- `MxGateway:Authentication:Mode = Disabled` and anonymous localhost (`AllowAnonymousLocalhost`,
|
||||
default `true`) **no longer receive the alarm feed at all**. They keep read-only access to every
|
||||
other dashboard page; `/alarms` and `/hubs/alarms` deny, and the side rail hides the Alarms link
|
||||
rather than offering a dead end.
|
||||
- `Dashboard:DisableLogin` auto-login is unaffected: it mints a genuinely authenticated principal
|
||||
holding both roles, so it satisfies the policy like any signed-in user.
|
||||
- Gating only the hub was rejected: the page reads the same monitor in process, so the disclosure
|
||||
would have been one navigation away.
|
||||
|
||||
Operators who *want* the alarm feed on an unauthenticated local console must sign in (or, in a dev
|
||||
environment, set `Dashboard:DisableLogin=true`); there is deliberately no config flag that re-opens
|
||||
the feed to the bypasses.
|
||||
|
||||
### DisableLogin dev bypass
|
||||
|
||||
`MxGateway:Dashboard:DisableLogin` (default `false`) is a third bypass for
|
||||
@@ -824,13 +900,16 @@ the deliberate mitigation: it bounds that exposure window without the cost of a
|
||||
revocation store. It now bounds a stale *tag* grant the same way: the token carries
|
||||
the tags resolved from the caller's LDAP groups at mint time, so removing a
|
||||
`GroupToTag` entry takes effect for token-authenticated hub connections within one
|
||||
lifetime. That 5-minute staleness bound covers token-authenticated connections only:
|
||||
a cookie principal carries the `zb:dashboardtag` claims stamped at login for the
|
||||
cookie's whole life, so for cookie-authenticated (in-process page) subscriptions a
|
||||
revoked `GroupToTag` grant takes effect at the user's next login, not within five
|
||||
minutes. That is where the per-session ACL's revocation need landed — a jti
|
||||
denylist stays deferred, since the short lifetime already bounds every grant the
|
||||
token carries.
|
||||
lifetime. Since SEC-41 the cookie seam is no longer the weaker one: the ACL
|
||||
re-resolves a cookie principal's grant from its group claims against current
|
||||
options on every subscribe, so a `GroupToTag` edit takes effect on both seams at
|
||||
the next subscribe after the restart that rebinds options — the stamped
|
||||
`zb:dashboardtag` claims are consulted only for a principal with no group claims
|
||||
to re-resolve from, i.e. the hub-token principal itself. What the token's lifetime
|
||||
still bounds uniquely is a stale **role** set and a stale LDAP **group** list, since
|
||||
both are copied into the payload at mint. See the revocation-bounds table under
|
||||
"Authentication And Authorization". A jti denylist stays deferred — the short
|
||||
lifetime already bounds every claim the token carries.
|
||||
|
||||
## Configuration
|
||||
|
||||
|
||||
@@ -593,9 +593,19 @@ Pending command handling:
|
||||
Timeouts should not assume the COM call stopped. A timed-out command may still
|
||||
finish inside the worker.
|
||||
|
||||
On timeout the client also forwards a `WorkerCancel` carrying the abandoned
|
||||
correlation id, best-effort: the gateway has stopped waiting, but the worker has
|
||||
not stopped working, and the worker owns a single STA. `WorkerPipeSession` routes
|
||||
On timeout **and on caller cancellation** the client also forwards a
|
||||
`WorkerCancel` carrying the abandoned correlation id, best-effort: the gateway
|
||||
has stopped waiting, but the worker has not stopped working, and the worker owns
|
||||
a single STA. Caller cancellation — a per-call deadline shorter than the
|
||||
gateway's command timeout, or a client disconnect cancelling the RPC's token — is
|
||||
the more common of the two paths, and under a retry storm it is exactly the
|
||||
backlog the cancel exists to shorten, so both arms send it; the pending command
|
||||
is failed as `CallerCancelled` rather than `GatewayShutdown` so metrics and logs
|
||||
name the real cause (both still map to gRPC `Cancelled`). The send is gated on
|
||||
the same ownership check as the timeout arm — only the arm that removed the
|
||||
pending entry sends, so a reply that won the race is never chased — and is
|
||||
skipped when the client's own stop token has fired, since the outbound channel is
|
||||
being completed and every pending command is failed anyway. `WorkerPipeSession` routes
|
||||
the cancel to `CancelCommand`, which drops the correlation from the STA queue if
|
||||
it has not started and replies `Canceled` for it. A cancel that arrives after the
|
||||
command reached MXAccess is a no-op — there is no way to abort an in-flight COM
|
||||
@@ -606,9 +616,10 @@ the worker reads the command and then its cancel and drops it before execution.
|
||||
|
||||
Cancels ride the same outbound channel as commands, whose capacity is
|
||||
`MaxPendingCommands + 4`: the reserve above the pending-command limit is what
|
||||
absorbs them, so a burst of timeouts stays bounded and cannot deadlock the
|
||||
enqueue path. Failing to send the cancel is logged at debug and never replaces
|
||||
the `CommandTimeout` the caller is owed.
|
||||
absorbs them, so a burst of timeouts or cancellations stays bounded and cannot
|
||||
deadlock the enqueue path. Failing to send the cancel is logged at debug and
|
||||
never replaces the `CommandTimeout` — or the `OperationCanceledException` — the
|
||||
caller is owed.
|
||||
|
||||
Cancellation outranks the deadline. When a caller's token is canceled around the
|
||||
same time the timeout fires, the command is reported as canceled
|
||||
|
||||
@@ -116,7 +116,7 @@ Optional live smoke variables:
|
||||
| `MXGATEWAY_LIVE_MXACCESS_CLIENT_NAME` | `ZB.MOM.WW.MxGateway.IntegrationTests` | Client name passed to `Register`. |
|
||||
| `MXGATEWAY_LIVE_MXACCESS_EVENT_TIMEOUT_SECONDS` | `15` | Maximum wait for the first `OnDataChange` (also used for the `OnWriteComplete` round-trip and the abnormal-exit fault transition). |
|
||||
| `MXGATEWAY_LIVE_MXACCESS_WRITE_SECURED_USER` | `admin` | ArchestrA user name passed to `AuthenticateUser` before the `WriteSecured` parity step. |
|
||||
| `MXGATEWAY_LIVE_MXACCESS_WRITE_SECURED_PASSWORD` | `admin123` | Password paired with the user above. Never logged; the test asserts the value does not appear in the WriteSecured diagnostic message. |
|
||||
| `MXGATEWAY_LIVE_MXACCESS_WRITE_SECURED_PASSWORD` | `password` | Password paired with the user above (matches the shared GLAuth `admin` user's dev password — see `glauth.md`). Never logged; the test asserts the value does not appear in the WriteSecured diagnostic message. |
|
||||
|
||||
The test output includes session id, worker process id, command status,
|
||||
HRESULT/status diagnostics, event sequence and handles, close status, and worker
|
||||
|
||||
+1
-1
@@ -282,7 +282,7 @@ StatusCode statusCode = exception.ErrorCode switch
|
||||
};
|
||||
```
|
||||
|
||||
`WorkerClientException` follows the same pattern: `CommandTimeout` becomes `DeadlineExceeded`, `GatewayShutdown` becomes `Cancelled`, `InvalidState` becomes `FailedPrecondition`, `ProtocolViolation` becomes `Internal`, `CommandTooLarge` becomes `ResourceExhausted`, and unmapped codes fall through to `Unavailable`.
|
||||
`WorkerClientException` follows the same pattern: `CommandTimeout` becomes `DeadlineExceeded`, `GatewayShutdown` and `CallerCancelled` both become `Cancelled`, `InvalidState` becomes `FailedPrecondition`, `ProtocolViolation` becomes `Internal`, `CommandTooLarge` becomes `ResourceExhausted`, and unmapped codes fall through to `Unavailable`. `CallerCancelled` is the caller's own token firing — a per-call deadline shorter than the gateway command timeout, or a disconnect — and is separate from `GatewayShutdown` only so metrics and logs stop attributing those calls to a shutdown that did not happen; the status the client observes is the same.
|
||||
|
||||
## Event Streaming Model
|
||||
|
||||
|
||||
+2
-2
@@ -72,7 +72,7 @@ Observable gauges are pull-based; the `Meter` invokes the supplied callback when
|
||||
|------------|--------------|-------------|
|
||||
| `mxgateway.sessions.open` | `_openSessions` | Currently open sessions tracked by `SessionManager`. |
|
||||
| `mxgateway.workers.running` | `_workersRunning` | Worker clients in a running state. |
|
||||
| `mxgateway.events.worker_queue.depth` | `_workerEventQueueDepthSources` (summed on demand) | Undelivered worker events held by `WorkerClient` — staged *and* queued (GWC-24) — summed across every live client at collection time (GWC-30). Each client owns an interlocked counter incremented when the read loop stages an event and decremented when the consumer reads it, and registers it as a gauge source for its lifetime, so a backlog stuck in a staging channel is visible and concurrent sessions add up instead of overwriting one another. |
|
||||
| `mxgateway.events.worker_queue.depth` | `_workerEventQueueDepthSources` (summed on demand) | Undelivered worker events held by `WorkerClient` — staged *and* queued (GWC-24) — summed across every live client at collection time (PERF-02). Each client owns an interlocked counter incremented when the read loop stages an event and decremented when the consumer reads it, and registers it as a gauge source for its lifetime, so a backlog stuck in a staging channel is visible and concurrent sessions add up instead of overwriting one another. |
|
||||
| `mxgateway.events.grpc_stream_queue.depth` | `_eventStreamBacklogSources` (summed on demand) | Live backlog buffered across every active `EventStreamService` subscriber, summed from the subscribers' channel `Count` at collection time. |
|
||||
|
||||
## Snapshot Shape
|
||||
@@ -111,7 +111,7 @@ The scalar fields mirror the counters and gauges. The four dictionaries provide
|
||||
- `EventsBySession` keys by `sessionId`; entries are removed via `RemoveSessionEvents` when a session closes so the map does not grow without bound.
|
||||
- `RetryAttemptsByArea` keys by the resilience `area` tag, e.g. `worker_startup`.
|
||||
|
||||
`EventsReceived` is read with `Interlocked.Read(ref _eventsReceived)` because `EventReceived` increments it via `Interlocked.Increment` outside the lock to keep the event-ingestion path non-blocking. `CommandsStarted`, `CommandsSucceeded`, `CommandsFailed`, and `CommandFailuresByMethod` are read the same way: the command counters run two-to-three times per gRPC call, so they are recorded with `Interlocked` and a `ConcurrentDictionary` rather than under `_syncRoot` (GWC-30). The two queue depths are pulled from their registered sources before the lock is taken, since those delegates reach into subscriber channels and worker clients.
|
||||
`EventsReceived` is read with `Interlocked.Read(ref _eventsReceived)` because `EventReceived` increments it via `Interlocked.Increment` outside the lock to keep the event-ingestion path non-blocking. `CommandsStarted`, `CommandsSucceeded`, `CommandsFailed`, and `CommandFailuresByMethod` are read the same way: the command counters run two-to-three times per gRPC call, so they are recorded with `Interlocked` and a `ConcurrentDictionary` rather than under `_syncRoot` (PERF-02). The two queue depths are pulled from their registered sources before the lock is taken, since those delegates reach into subscriber channels and worker clients.
|
||||
|
||||
## Recording Sites
|
||||
|
||||
|
||||
@@ -317,6 +317,30 @@ whose write workload is effectively fire-and-forget — no consumer reads the
|
||||
reply's `statuses` — can lower it, or set `0` to skip the wait entirely and
|
||||
reply on acceptance alone.
|
||||
|
||||
**Write-completion correlation is recency-based, not per-call.** MXAccess's
|
||||
`OnWriteComplete` callback carries no correlation token — only the server/item
|
||||
handle pair — so `MxAccessCommandExecutor` correlates a completion to a write
|
||||
by *version*: it snapshots `MxAccessWriteCompletionCache.CurrentVersion` for
|
||||
the handle as a baseline before the COM call, then reports the first
|
||||
completion whose version exceeds that baseline as the write's outcome
|
||||
(`AwaitWriteCompletion`). This is exact for the common case, but if an earlier
|
||||
write to the same handle times out unconfirmed — its `OnWriteComplete` still
|
||||
undelivered when the wait gives up — and the caller immediately issues another
|
||||
write to the same tag, the stale callback can dispatch during the second
|
||||
write's pump-wait and be reported as *its* outcome. The executor mitigates the
|
||||
most common instance of this (`CaptureCompletionBaseline`) by running one
|
||||
`pumpStep()` immediately before capturing the baseline, so a completion already
|
||||
queued for the handle folds into the baseline instead of being available to
|
||||
ride the next write's reply; this closes the gap for a callback that arrived
|
||||
before the new write started, but a callback that MXAccess queues after that
|
||||
pump step and before the COM call returns (or during the wait itself) is still
|
||||
indistinguishable from the new write's own completion. Full closure is not
|
||||
possible without an MXAccess-side correlation token, which does not exist —
|
||||
this is a property of the COM surface, not a gap this worker introduced (a
|
||||
native client correlating by handle has the identical ambiguity). The exposure
|
||||
is bounded: `statuses` is an enrichment field, and `Hresult`/`ProtocolStatus`
|
||||
stay acceptance-only regardless of which completion (if any) was attributed.
|
||||
|
||||
## COM Creation
|
||||
|
||||
The MXAccess analysis source at `C:\Users\dohertj2\Desktop\mxaccess` identifies
|
||||
|
||||
@@ -213,6 +213,8 @@ The repair transitions the monitor's reconcile broadcasts on the alarm feed (Rai
|
||||
|
||||
`AttachInternalEventSubscriber` enforces the same readiness gate as `AttachEventSubscriber` — a session (or worker) that is not `Ready` throws `SessionNotReady` *before* the distributor is constructed. A premature attach would otherwise start the pump against a source that throws, completing every subscriber with that error and latching the distributor for the session's whole lifetime.
|
||||
|
||||
The same one-distributor-per-session rule is enforced at the other end of the lifecycle. Session disposal disposes the distributor and latches the fact, so an attach that passed its readiness gate and then lost the CPU until the session was torn down throws `SessionNotReady` rather than constructing a replacement distributor. A replacement can never be pumped — the pump-start flag is a one-way latch — so its subscriber would block on an empty channel that is never completed until the client's own deadline expired, with no error, log or metric to explain it. Every attach entry point (`AttachEventSubscriber`, `AttachEventSubscriberWithReplay`, `AttachInternalEventSubscriber`) fails this way, and each rolls back the subscriber-count slot it took, so a caller that loses the race to teardown sees the same failure it would have seen a moment later from the readiness gate.
|
||||
|
||||
Sessions open with `MxGateway:Sessions:DefaultLeaseSeconds` (default 1800) added to the open timestamp. Unary client activity refreshes the lease by the same duration. `ExtendLease` and `IsLeaseExpired` cooperate with `SessionManager.CloseExpiredLeasesAsync`, which iterates a registry snapshot and closes any session whose lease has expired with `LeaseExpiredReason`. `SessionLeaseMonitorHostedService` runs that sweep every `MxGateway:Sessions:LeaseSweepIntervalSeconds` seconds (default 30).
|
||||
|
||||
#### Teardown parallelism
|
||||
|
||||
@@ -241,7 +241,7 @@ Run `dotnet build src/ZB.MOM.WW.MxGateway.NonWindows.slnx` (expect 0 warnings) a
|
||||
- Modify: `src/ZB.MOM.WW.MxGateway.Worker.Tests/Ipc/WorkerPipeSessionTests.cs`
|
||||
- Modify: `src/ZB.MOM.WW.MxGateway.Worker/Ipc/WorkerFrameReader.cs` (comment only)
|
||||
|
||||
**Why:** On net48, `NamedPipeClientStream.ReadAsync` ignores its `CancellationToken` (`WorkerFrameReader.cs:109-111`). Fault-path exits (event-drain fault, oversized event, watchdog, heartbeat write failure) leave `readTask` pending; it is unblocked only when `WorkerPipeClient`'s `using` disposes the pipe, at which point it faults with `ObjectDisposedException`/`IOException` on a Task nobody observes (the finally at `:303-309` awaits only heartbeat and drain). The frame-pooling change (GWC-30) makes this sharper: the abandoned read owns the per-instance `_lengthPrefix` buffer and possibly a rented ArrayPool payload — the reader's single-consumer invariant holds today only because nothing ever reads again after abandonment.
|
||||
**Why:** On net48, `NamedPipeClientStream.ReadAsync` ignores its `CancellationToken` (`WorkerFrameReader.cs:109-111`). Fault-path exits (event-drain fault, oversized event, watchdog, heartbeat write failure) leave `readTask` pending; it is unblocked only when `WorkerPipeClient`'s `using` disposes the pipe, at which point it faults with `ObjectDisposedException`/`IOException` on a Task nobody observes (the finally at `:303-309` awaits only heartbeat and drain). The frame-pooling change (PERF-22) makes this sharper: the abandoned read owns the per-instance `_lengthPrefix` buffer and possibly a rented ArrayPool payload — the reader's single-consumer invariant holds today only because nothing ever reads again after abandonment.
|
||||
|
||||
**Spec — constraints, implementer designs within them:**
|
||||
1. **No unobserved faulted Task.** After the stream is disposed, `readTask`'s fault must be awaited/observed (reuse `ObserveBackgroundTaskStopAsync`'s timeout-and-log shape, `:312-348`) before `WorkerPipeClient.RunAsync` returns.
|
||||
|
||||
@@ -608,6 +608,24 @@ Commit anything found: `docs: remediation plan doc sweep`
|
||||
|
||||
---
|
||||
|
||||
## PERF-ID register
|
||||
|
||||
This plan's tasks were cited in code comments and test names as `GWC-30`/`GWC-31` before
|
||||
those labels were recognized as colliding with the `archreview` finding register — see
|
||||
cycle-3 finding `GWC-31` (`archreview/2026-08-19/10-gateway-core.md`). The comments were
|
||||
swept to the `PERF-nn` prefix below (`nn` = this doc's task number); `GWC-30`/`GWC-31` no
|
||||
longer appear in perf-pass comments. Citations of the *genuine* archreview `GWC-30`
|
||||
("Frame reader still allocates a fresh 4-byte prefix array per frame",
|
||||
`archreview/2026-07-12/10-gateway-core.md`) — the gateway-side
|
||||
`Server/Workers/WorkerFrameReader.cs` fix and its test — were left untouched; they refer
|
||||
to a real register entry, not this plan.
|
||||
|
||||
| PERF-ID | Task | Files |
|
||||
|---|---|---|
|
||||
| PERF-02 | Task 2 — Metrics: pull-gauge for worker queue depth, lock-free command counters | `Server/Workers/WorkerClient.cs` (`:44`, `:355`), `Server/Metrics/GatewayMetrics.cs` (`:46`, `:213`, `:284`, `:463`, `:524`), `Tests/Metrics/GatewayMetricsTests.cs` (`:22`, `:61`), `docs/Metrics.md` |
|
||||
| PERF-07 | Task 7 — WorkerClient: pooled-timer timeout, single sizing pass, `WorkerCancel` on timeout | `Server/Workers/WorkerClient.cs` (`:236`, `:253`), `Tests/Gateway/Workers/WorkerClientTests.cs` (`:163`, `:179`) |
|
||||
| PERF-22 | Task 22 — Worker frame writer/reader: pooled buffers | `Worker/Ipc/WorkerFrameReader.cs` (`:23`), `Worker/Ipc/WorkerFrameWriter.cs` (`:671`), `docs/plans/2026-08-15-deferred-remediation.md` (Task cross-reference) |
|
||||
|
||||
## Explicitly deferred (decided, not forgotten)
|
||||
|
||||
| Finding | Why deferred |
|
||||
|
||||
+26
-1
@@ -249,6 +249,22 @@ monitoring (forced)") when subtag mode is the configured `Fallback:Mode=ForceSub
|
||||
as a fault. Metrics: `mxgateway.alarms.provider_mode` gauge (1 = alarmmgr,
|
||||
2 = subtag) and `mxgateway.alarms.provider_switches` counter.
|
||||
|
||||
**Standby-coverage visibility:** `degraded` also covers the case where the alarm
|
||||
manager is serving normally but the subtag standby behind it could not cover
|
||||
anything — a watch-list that resolved to zero targets, or Galaxy Repository
|
||||
discovery that was enabled and failed. That condition used to be a single startup
|
||||
log line, invisible until the failover that needed the standby found it empty; it
|
||||
is now a standing `AlarmProviderStatus` reason
|
||||
(`AlarmProviderReasons.SubtagWatchListEmpty` / `SubtagDiscoveryFailed`) carried on
|
||||
the feed and rendered as the amber `Alarm Manager (no subtag standby)` badge, which
|
||||
keeps the failover label from naming the wrong live provider. The watch-list the
|
||||
worker holds is fixed for the life of the subscription, so on the failover edge the
|
||||
monitor re-resolves and reports a difference
|
||||
(`AlarmProviderReasons.SubtagWatchListStale`, with both counts) rather than
|
||||
re-sending it — re-sending would rebuild the failover consumer on the primary that
|
||||
just failed. A monitor restart re-resolves and re-sends. See
|
||||
`docs/AlarmClientDiscovery.md`.
|
||||
|
||||
**Truncated-snapshot visibility:** `GetXmlCurrentAlarms2` caps its reply at
|
||||
`MxGateway:Alarms:MaxAlarmsPerFetch` and offers no confirmed "more available"
|
||||
flag, so a reply holding exactly the cap is treated as truncated. On such a
|
||||
@@ -917,10 +933,19 @@ Timeouts:
|
||||
|
||||
Cancellation:
|
||||
|
||||
- canceling the gRPC call should stop waiting in the gateway,
|
||||
- canceling the gRPC call stops the gateway waiting **and** forwards a
|
||||
`WorkerCancel` for the abandoned correlation, the same best-effort frame the
|
||||
command timeout sends — a client deadline or a disconnect is the common
|
||||
abandonment path, and leaving the correlation on the worker's single STA queue
|
||||
makes live callers queue behind work nobody is waiting for,
|
||||
- the cancel is sent only by the caller that owned the pending-entry removal (so
|
||||
a reply that won the race is never chased) and is skipped once the client is
|
||||
stopping, where every pending command fails anyway,
|
||||
- it cannot safely abort an in-flight COM call on the STA,
|
||||
- the worker should finish the COM call and discard or log the late reply if
|
||||
the correlation was canceled,
|
||||
- caller cancellation is reported as `CallerCancelled` (mapped to gRPC
|
||||
`Cancelled`), distinct from a gateway shutdown, so metrics name the real cause,
|
||||
- hard cancellation means killing the worker process.
|
||||
|
||||
## Event Delivery And Backpressure
|
||||
|
||||
@@ -52,31 +52,46 @@ LmxOpcUa write rights doesn't need a second account for the gw.
|
||||
|
||||
| Group | GID | DN | LmxOpcUa meaning | Suggested mxgw mapping |
|
||||
|---|---|---|---|---|
|
||||
| ReadOnly | 5501 | `ou=ReadOnly,ou=groups,dc=zb,dc=local` | Browse + read OPC UA nodes | `Browse` + `Subscribe` (read paths only) |
|
||||
| WriteOperate | 5502 | `ou=WriteOperate,ou=groups,dc=zb,dc=local` | Write FreeAccess / Operate attrs | `Write` (plain) |
|
||||
| WriteTune | 5504 | `ou=WriteTune,ou=groups,dc=zb,dc=local` | Write Tune attrs | `WriteSecured` (Tune only) |
|
||||
| WriteConfigure | 5505 | `ou=WriteConfigure,ou=groups,dc=zb,dc=local` | Write Configure attrs | `WriteSecured` (Configure) |
|
||||
| AlarmAck | 5503 | `ou=AlarmAck,ou=groups,dc=zb,dc=local` | Acknowledge alarms | gw alarm-ack RPC, when added |
|
||||
| ReadOnly | 5601 | `ou=ReadOnly,ou=groups,dc=zb,dc=local` | Browse + read OPC UA nodes | `Browse` + `Subscribe` (read paths only) |
|
||||
| WriteOperate | 5602 | `ou=WriteOperate,ou=groups,dc=zb,dc=local` | Write FreeAccess / Operate attrs | `Write` (plain) |
|
||||
| WriteTune | 5603 | `ou=WriteTune,ou=groups,dc=zb,dc=local` | Write Tune attrs | `WriteSecured` (Tune only) |
|
||||
| WriteConfigure | 5604 | `ou=WriteConfigure,ou=groups,dc=zb,dc=local` | Write Configure attrs | `WriteSecured` (Configure) |
|
||||
| AlarmAck | 5605 | `ou=AlarmAck,ou=groups,dc=zb,dc=local` | Acknowledge alarms | gw alarm-ack RPC, when added |
|
||||
|
||||
These are the shared config's OPC-perm (`560x`) family — see
|
||||
`scadaproj/infra/glauth/README.md`'s directory-layout table, which
|
||||
partitions groups into non-overlapping gid ranges per sister app
|
||||
(`SCADA-*` 55xx, OPC-perm 560x, `Gw*` 561x, `OtOpcUa-*` 57xx) so the
|
||||
three apps coexist in one directory.
|
||||
|
||||
**A user can be in multiple groups** — `othergroups = [...]` in the
|
||||
config is a list. `admin` is the canonical example (in every role
|
||||
group below).
|
||||
config is a list. `multi-role` is the canonical example (in every
|
||||
group in the file, not just the ones below).
|
||||
|
||||
## Pre-provisioned users
|
||||
|
||||
Per-role OPC-perm testers are named `opc-*` rather than the bare role name
|
||||
(`readonly`, `writeop`, …) because a tester username must not
|
||||
case-collide with a group name — see the naming rule in
|
||||
`scadaproj/infra/glauth/README.md`.
|
||||
|
||||
| Username | Password | UID | Primary group | Other groups | Capabilities |
|
||||
|---|---|---|---|---|---|
|
||||
| `readonly` | `readonly123` | 5001 | ReadOnly | — | Browse, read |
|
||||
| `writeop` | `writeop123` | 5002 | WriteOperate | — | + plain Write |
|
||||
| `writetune` | `writetune123` | 5005 | WriteTune | — | + WriteSecured (Tune) |
|
||||
| `writeconfig` | `writeconfig123` | 5006 | WriteConfigure | — | + WriteSecured (Configure) |
|
||||
| `alarmack` | `alarmack123` | 5003 | AlarmAck | — | Alarm acknowledgment |
|
||||
| `admin` | `admin123` | 5004 | ReadOnly | WriteOperate, AlarmAck, WriteTune, WriteConfigure | All roles |
|
||||
| `opc-readonly` | `password` | 5101 | ReadOnly | — | Browse, read |
|
||||
| `opc-writeop` | `password` | 5102 | WriteOperate | — | + plain Write |
|
||||
| `opc-writetune` | `password` | 5103 | WriteTune | — | + WriteSecured (Tune) |
|
||||
| `opc-writeconfig` | `password` | 5104 | WriteConfigure | — | + WriteSecured (Configure) |
|
||||
| `opc-alarmack` | `password` | 5105 | AlarmAck | — | Alarm acknowledgment |
|
||||
| `multi-role` | `password` | 5005 | SCADA-Admins | every other group in the file, incl. all five OPC-perm groups + `GwAdmin`/`GwReader` | All roles, all apps (cross-app canonical login) |
|
||||
| `admin` | `password` | 5001 | SCADA-Admins | `GwAdmin`, `OtOpcUa-Admins` | Dashboard Administrator + ScadaBridge/OtOpcUa admin — **not** a member of any OPC-perm group, so it has no gw-side Write/WriteSecured/AlarmAck capability |
|
||||
| `serviceaccount` | `<service-account-password>` | 5999 | ReadOnly | — | LDAP search capability (for bind-then-search) |
|
||||
|
||||
For mxaccessgw dev, `admin` covers every gw-side capability test;
|
||||
`readonly` is the right "negative" case for proving Browse-OK /
|
||||
Write-denied.
|
||||
For mxaccessgw dev, `multi-role` covers every gw-side capability test
|
||||
(it is a member of all five OPC-perm groups as well as `GwAdmin`);
|
||||
`opc-readonly` is the right "negative" case for proving Browse-OK /
|
||||
Write-denied. `admin` is useful for dashboard-Administrator and
|
||||
ScadaBridge/OtOpcUa-admin testing but, per the table above, carries no
|
||||
OPC-perm group membership of its own.
|
||||
|
||||
The gateway dashboard uses two gateway-specific groups beyond the LmxOpcUa taxonomy:
|
||||
`GwAdmin` (gid 5610 → role `Administrator`) and `GwReader` (gid 5611 → role `Viewer`).
|
||||
@@ -121,7 +136,7 @@ principal cannot show that.
|
||||
|
||||
```
|
||||
DN: cn=admin,dc=zb,dc=local
|
||||
Password: admin123
|
||||
Password: password
|
||||
```
|
||||
|
||||
Construct the DN from the username; bind. Works on GLAuth because
|
||||
|
||||
@@ -0,0 +1,210 @@
|
||||
#!/usr/bin/env pwsh
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Fails a PR that changes a client's source without bumping its version
|
||||
past the last thing actually published (CLI-46).
|
||||
|
||||
.DESCRIPTION
|
||||
CLI-39 (2026-07-12 cycle) added publish-time guards that refuse to
|
||||
re-publish an already-published name+version
|
||||
(scripts/pack-clients.ps1's Assert-GiteaPackageNotPublished,
|
||||
scripts/tag-go-module.ps1's ClientVersion-vs-tag check). Those guards
|
||||
only fire when someone actually runs a publish. CLI-46 (2026-08-19
|
||||
cycle) found the same hazard recurring silently in between publishes:
|
||||
library source (including a real bug fix, commit a8f86b5) shipped
|
||||
while the version constant still equalled the already-published
|
||||
number, so nothing in the tree recorded that the published artifact
|
||||
and the in-repo source had diverged. This script is the PR-time half
|
||||
of that guard: catch the drift as soon as it happens, not at the next
|
||||
publish.
|
||||
|
||||
For each client this repo ships, the script:
|
||||
1. Finds the latest "publish tag" for that client, per the tag
|
||||
naming convention its own publish tooling actually uses.
|
||||
2. If no such tag exists, says so and passes (nothing to compare
|
||||
against yet — this is expected for every client except Go; see
|
||||
below).
|
||||
3. If a tag exists, reads the version recorded in-source at that tag
|
||||
and the version recorded in-source now. If `clients/<lang>` has
|
||||
changed since the tag AND the version constant is unchanged, that
|
||||
is exactly the CLI-46 pattern: source moved, the published
|
||||
coordinate did not, and nothing says so. Fail.
|
||||
|
||||
Tag conventions, as implemented today (verified against
|
||||
scripts/pack-clients.ps1, scripts/tag-go-module.ps1, and `git tag -l`):
|
||||
- Go is the only client with a git tag at all: `tag-go-module.ps1`
|
||||
creates `clients/go/vX.Y.Z` (CLI-21/CLI-39 already keep this tag's
|
||||
version in lockstep with clients/go/mxgateway/version.go's
|
||||
ClientVersion at *tag time*; this script is what keeps it in
|
||||
lockstep *between* tags).
|
||||
- .NET, Python, Rust, and Java publish straight to a package
|
||||
registry via `pack-clients.ps1 -Publish` (NuGet/PyPI/cargo/Maven
|
||||
API pushes) with no corresponding git tag of any kind — there is
|
||||
nothing in this repo's history to diff against for those four.
|
||||
The script looks for a same-shaped tag anyway
|
||||
(`clients/<lang>/v*`) so it stays correct if that ever changes,
|
||||
but today it will report "no tags found" and pass for all four.
|
||||
|
||||
A client with no tag is not a violation of anything — it means no
|
||||
release train has ever tagged that client, so there is no "since the
|
||||
last publish" window to check. The check exists to catch drift once a
|
||||
baseline exists, not to demand one.
|
||||
|
||||
.EXAMPLE
|
||||
pwsh scripts/check-client-versions.ps1
|
||||
#>
|
||||
|
||||
[CmdletBinding()]
|
||||
param()
|
||||
|
||||
Set-StrictMode -Version Latest
|
||||
$ErrorActionPreference = 'Stop'
|
||||
|
||||
$repoRoot = (Resolve-Path (Join-Path $PSScriptRoot '..')).Path
|
||||
Push-Location $repoRoot
|
||||
try {
|
||||
# One entry per official client. `TagGlob` is the tag naming pattern that
|
||||
# client's own publish tooling uses (see doc comment above); `VersionFile`
|
||||
# + `VersionPattern` locate the single version-of-record constant
|
||||
# docs/ClientPackaging.md's "Versioning" section names as authoritative
|
||||
# for that client; `SourceDir` is the diff scope (matches the literal
|
||||
# `clients/<lang>` scope described in the CLI-46 remediation task).
|
||||
$clients = @(
|
||||
[ordered]@{
|
||||
Name = 'dotnet'
|
||||
SourceDir = 'clients/dotnet'
|
||||
TagGlob = 'clients/dotnet/v*'
|
||||
VersionFile = 'clients/dotnet/ZB.MOM.WW.MxGateway.Client/ZB.MOM.WW.MxGateway.Client.csproj'
|
||||
VersionPattern = '<Version>([^<]+)</Version>'
|
||||
},
|
||||
[ordered]@{
|
||||
Name = 'go'
|
||||
SourceDir = 'clients/go'
|
||||
TagGlob = 'clients/go/v*'
|
||||
VersionFile = 'clients/go/mxgateway/version.go'
|
||||
VersionPattern = 'ClientVersion\s*=\s*"([^"]+)"'
|
||||
},
|
||||
[ordered]@{
|
||||
Name = 'python'
|
||||
SourceDir = 'clients/python'
|
||||
TagGlob = 'clients/python/v*'
|
||||
VersionFile = 'clients/python/src/zb_mom_ww_mxgateway/version.py'
|
||||
VersionPattern = '__version__\s*=\s*"([^"]+)"'
|
||||
},
|
||||
[ordered]@{
|
||||
Name = 'rust'
|
||||
SourceDir = 'clients/rust'
|
||||
TagGlob = 'clients/rust/v*'
|
||||
VersionFile = 'clients/rust/Cargo.toml'
|
||||
VersionPattern = 'version\s*=\s*"([^"]+)"'
|
||||
},
|
||||
[ordered]@{
|
||||
Name = 'java'
|
||||
SourceDir = 'clients/java'
|
||||
TagGlob = 'clients/java/v*'
|
||||
VersionFile = 'clients/java/build.gradle'
|
||||
VersionPattern = "version\s*=\s*'([^']+)'"
|
||||
}
|
||||
)
|
||||
|
||||
function Get-LatestTag {
|
||||
param([string]$Glob)
|
||||
|
||||
# --sort=-v:refname orders by version-aware descending semver, so the
|
||||
# first line is the newest tag matching the glob. Empty result means
|
||||
# no tag of this shape has ever been created for this client.
|
||||
$tags = git tag -l --sort='-v:refname' $Glob
|
||||
if (-not $tags) {
|
||||
return $null
|
||||
}
|
||||
return @($tags -split "`n")[0].Trim()
|
||||
}
|
||||
|
||||
function Get-VersionFromFile {
|
||||
param([string]$Path, [string]$Pattern)
|
||||
|
||||
if (-not (Test-Path $Path)) {
|
||||
throw "Version file not found: $Path"
|
||||
}
|
||||
$content = Get-Content -Path $Path -Raw
|
||||
$match = [regex]::Match($content, $Pattern)
|
||||
if (-not $match.Success) {
|
||||
throw "Could not find a version matching /$Pattern/ in $Path"
|
||||
}
|
||||
return $match.Groups[1].Value
|
||||
}
|
||||
|
||||
function Get-VersionFromFileAtRevision {
|
||||
param([string]$Revision, [string]$Path, [string]$Pattern)
|
||||
|
||||
$content = git show "${Revision}:${Path}" 2>$null
|
||||
if ($LASTEXITCODE -ne 0 -or -not $content) {
|
||||
# The version file did not exist (or was at a different path) at
|
||||
# that tag. Treat as "unknown" rather than failing the whole
|
||||
# check — the drift comparison below only needs "does the
|
||||
# current version still equal the tag's version", and an
|
||||
# unresolvable historical version can never equal the current
|
||||
# one, so this degrades to "assume it changed" (safe: it will
|
||||
# not mask a real drift).
|
||||
return $null
|
||||
}
|
||||
$match = [regex]::Match(($content -join "`n"), $Pattern)
|
||||
if (-not $match.Success) {
|
||||
return $null
|
||||
}
|
||||
return $match.Groups[1].Value
|
||||
}
|
||||
|
||||
$failures = New-Object System.Collections.Generic.List[string]
|
||||
|
||||
foreach ($client in $clients) {
|
||||
Write-Host ''
|
||||
Write-Host "== $($client.Name) ==" -ForegroundColor Cyan
|
||||
|
||||
$tag = Get-LatestTag -Glob $client.TagGlob
|
||||
if (-not $tag) {
|
||||
Write-Host " No publish tags matching '$($client.TagGlob)' — nothing to compare against yet. Pass." -ForegroundColor Yellow
|
||||
continue
|
||||
}
|
||||
|
||||
Write-Host " Latest publish tag: $tag"
|
||||
|
||||
$currentVersion = Get-VersionFromFile -Path $client.VersionFile -Pattern $client.VersionPattern
|
||||
Write-Host " Current version constant ($($client.VersionFile)): $currentVersion"
|
||||
|
||||
# Any diff at all in the client's source tree since its last publish
|
||||
# tag — matches the CLI-46 remediation task's literal check.
|
||||
git diff --quiet $tag -- $client.SourceDir
|
||||
$sourceChanged = ($LASTEXITCODE -ne 0)
|
||||
|
||||
if (-not $sourceChanged) {
|
||||
Write-Host " No changes under $($client.SourceDir) since $tag. Pass." -ForegroundColor Green
|
||||
continue
|
||||
}
|
||||
|
||||
$tagVersion = Get-VersionFromFileAtRevision -Revision $tag -Path $client.VersionFile -Pattern $client.VersionPattern
|
||||
Write-Host " Source under $($client.SourceDir) changed since $tag (tag's version constant: $(if ($tagVersion) { $tagVersion } else { '<unresolvable>' }))."
|
||||
|
||||
if ($currentVersion -eq $tagVersion) {
|
||||
$failures.Add("$($client.Name): source changed since publish tag '$tag' but $($client.VersionFile) still reads '$currentVersion' (the version at '$tag'). Bump the version before another publish (CLI-46).")
|
||||
Write-Host " FAIL: version constant unchanged since the publish tag." -ForegroundColor Red
|
||||
}
|
||||
else {
|
||||
Write-Host " Version constant already bumped ('$tagVersion' -> '$currentVersion') since $tag. Pass." -ForegroundColor Green
|
||||
}
|
||||
}
|
||||
|
||||
Write-Host ''
|
||||
if ($failures.Count -gt 0) {
|
||||
Write-Host 'Client version freshness check FAILED:' -ForegroundColor Red
|
||||
foreach ($failure in $failures) {
|
||||
Write-Host " - $failure" -ForegroundColor Red
|
||||
}
|
||||
exit 1
|
||||
}
|
||||
|
||||
Write-Host 'Client version freshness check passed.' -ForegroundColor Green
|
||||
}
|
||||
finally {
|
||||
Pop-Location
|
||||
}
|
||||
@@ -7,7 +7,7 @@
|
||||
<PropertyGroup>
|
||||
<IsPackable>true</IsPackable>
|
||||
<PackageId>ZB.MOM.WW.MxGateway.Contracts</PackageId>
|
||||
<Version>0.2.0</Version>
|
||||
<Version>0.2.1</Version>
|
||||
<Authors>Joseph Doherty</Authors>
|
||||
<Company>ZB MOM WW</Company>
|
||||
<Copyright>Copyright (c) ZB MOM WW. All rights reserved.</Copyright>
|
||||
|
||||
@@ -1,4 +1,3 @@
|
||||
using ZB.MOM.WW.MxGateway.Contracts.Proto;
|
||||
using ZB.MOM.WW.MxGateway.Server.Alarms;
|
||||
using ZB.MOM.WW.MxGateway.Server.Configuration;
|
||||
|
||||
@@ -19,14 +18,18 @@ public sealed class EmptyAlarmWatchListResolver : IAlarmWatchListResolver
|
||||
/// <summary>Shared no-op instance.</summary>
|
||||
public static readonly EmptyAlarmWatchListResolver Instance = new();
|
||||
|
||||
private static readonly IReadOnlyList<AlarmSubtagTarget> Empty = [];
|
||||
private static readonly AlarmWatchListResolution Empty = new(
|
||||
[],
|
||||
DiscoveryAttempted: false,
|
||||
DiscoveryFailed: false,
|
||||
DiscoveryFailureMessage: null);
|
||||
|
||||
private EmptyAlarmWatchListResolver()
|
||||
{
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public Task<IReadOnlyList<AlarmSubtagTarget>> ResolveAsync(
|
||||
public Task<AlarmWatchListResolution> ResolveAsync(
|
||||
AlarmsOptions options,
|
||||
CancellationToken cancellationToken = default)
|
||||
{
|
||||
|
||||
@@ -514,21 +514,37 @@ public sealed class WorkerLiveMxAccessSmokeTests(ITestOutputHelper output)
|
||||
// AuthenticateUser resolves an ArchestrA user id for the WriteSecured call.
|
||||
// Credentials are env-overridable so the test honors the gateway's "do not
|
||||
// log secrets" rule and works against either MXAccess's own user store or
|
||||
// the LmxOpcUa-baseline GLAuth-bridged ArchestrA identity (admin/admin123).
|
||||
// the LmxOpcUa-baseline GLAuth-bridged ArchestrA identity (admin/password —
|
||||
// see glauth.md; the shared GLAuth's dev-user password is "password", not
|
||||
// the old per-box "admin123").
|
||||
MxCommandReply authReply = await fixture.Service.Invoke(
|
||||
CreateAuthenticateUserRequest(sessionId, serverHandle, verifyUser, verifyPassword),
|
||||
new TestServerCallContext()).ConfigureAwait(false);
|
||||
recordedOutput.WriteLine(
|
||||
$"AuthenticateUser status={authReply.ProtocolStatus.Code} hresult={authReply.Hresult} user_id={authReply.AuthenticateUser?.UserId}");
|
||||
|
||||
// AuthenticateUser is allowed to fail (the underlying provider may reject
|
||||
// the credential pair); we use the returned user id if non-zero and fall
|
||||
// back to 0 ("operator only" / no verifier) so the parity assertion holds.
|
||||
int currentUserId = authReply.ProtocolStatus.Code == ProtocolStatusCode.Ok
|
||||
// AuthenticateUser is allowed to fail — MXAccess parity means a rejected
|
||||
// credential is a legitimate reply, not a bug, so this must never be an
|
||||
// Assert. But a silent fall-through to "operator only" (currentUserId 0)
|
||||
// quietly demotes the rest of this test from an authenticated round-trip
|
||||
// to a bare failure-reply-shape check, which a stale rig credential (e.g.
|
||||
// TST-31) would otherwise hide indefinitely in nightly runs. Log the
|
||||
// outcome loudly — never the password (verifyUser only) — so that
|
||||
// degradation is visible in the test output even though it never fails
|
||||
// the test by itself.
|
||||
bool authenticateUserSucceeded = authReply.ProtocolStatus.Code == ProtocolStatusCode.Ok
|
||||
&& authReply.AuthenticateUser is not null
|
||||
&& authReply.AuthenticateUser.UserId != 0
|
||||
? authReply.AuthenticateUser.UserId
|
||||
: 0;
|
||||
&& authReply.AuthenticateUser.UserId != 0;
|
||||
recordedOutput.WriteLine(
|
||||
authenticateUserSucceeded
|
||||
? $"AuthenticateUser SUCCEEDED for user={verifyUser} user_id={authReply.AuthenticateUser!.UserId}"
|
||||
: $"AuthenticateUser FAILED (credential rejected or not recognized by the provider) "
|
||||
+ $"for user={verifyUser} status={authReply.ProtocolStatus.Code} hresult={authReply.Hresult} "
|
||||
+ $"diagnostic={authReply.DiagnosticMessage} -- WriteSecured below will run as an "
|
||||
+ $"unauthenticated (currentUserId=0) failure-reply-shape check only, not an "
|
||||
+ $"authenticated round-trip. If this is unexpected, verify "
|
||||
+ $"MXGATEWAY_LIVE_MXACCESS_WRITE_SECURED_USER/_PASSWORD against the rig's ArchestrA "
|
||||
+ $"identity (see glauth.md).");
|
||||
|
||||
int currentUserId = authenticateUserSucceeded ? authReply.AuthenticateUser!.UserId : 0;
|
||||
|
||||
MxCommandReply writeSecuredReply = await fixture.Service.Invoke(
|
||||
CreateWriteSecuredRequest(
|
||||
@@ -1429,7 +1445,7 @@ public sealed class WorkerLiveMxAccessSmokeTests(ITestOutputHelper output)
|
||||
string verifyUser = Environment.GetEnvironmentVariable("MXGATEWAY_LIVE_MXACCESS_WRITE_SECURED_USER")
|
||||
?? "admin";
|
||||
string verifyPassword = Environment.GetEnvironmentVariable("MXGATEWAY_LIVE_MXACCESS_WRITE_SECURED_PASSWORD")
|
||||
?? "admin123";
|
||||
?? "password";
|
||||
return (verifyUser, verifyPassword);
|
||||
}
|
||||
|
||||
|
||||
@@ -15,4 +15,32 @@ public static class AlarmProviderReasons
|
||||
/// distinguish a forced subtag mode from an unexpected degraded failover.
|
||||
/// </summary>
|
||||
public const string ForcedSubtag = "Forced subtag mode (configuration)";
|
||||
|
||||
/// <summary>
|
||||
/// Reason prefix set when the effective mode can need the subtag watch-list
|
||||
/// (<c>Auto</c> or <c>ForceSubtag</c>) but the list resolved to zero targets, so the
|
||||
/// subtag provider covers no alarms (GWC-37). In <c>Auto</c> this is a warning about the
|
||||
/// standby, not about the currently-serving alarm manager, which is why the dashboard
|
||||
/// renders it against the <c>ALARMMGR</c> mode as a coverage badge rather than a failover.
|
||||
/// </summary>
|
||||
public const string SubtagWatchListEmpty =
|
||||
"Subtag watch-list is empty: the subtag provider covers no alarms";
|
||||
|
||||
/// <summary>
|
||||
/// Reason prefix set when Galaxy Repository alarm discovery was enabled but failed, so
|
||||
/// the watch-list carries only what configuration contributed and may be incomplete
|
||||
/// (GWC-37). Distinct from <see cref="SubtagWatchListEmpty"/>: discovery failing with a
|
||||
/// usable config-only list is a completeness warning, not a total loss of coverage.
|
||||
/// </summary>
|
||||
public const string SubtagDiscoveryFailed =
|
||||
"Galaxy Repository alarm discovery failed: the subtag watch-list may be incomplete";
|
||||
|
||||
/// <summary>
|
||||
/// Reason prefix set when a re-resolution at the failover edge produced a different
|
||||
/// watch-list than the one sent at <c>SubscribeAlarms</c> (GWC-37). The worker keeps the
|
||||
/// list it was given for the life of the subscription, so the difference is reported
|
||||
/// rather than applied; a monitor restart re-resolves and re-sends.
|
||||
/// </summary>
|
||||
public const string SubtagWatchListStale =
|
||||
"Subtag watch-list differs from the one the worker is using";
|
||||
}
|
||||
|
||||
@@ -0,0 +1,29 @@
|
||||
using ZB.MOM.WW.MxGateway.Contracts.Proto;
|
||||
|
||||
namespace ZB.MOM.WW.MxGateway.Server.Alarms;
|
||||
|
||||
/// <summary>
|
||||
/// The outcome of one subtag watch-list resolution: the composed targets plus enough
|
||||
/// provenance for the caller to tell an empty list that means "nothing was configured"
|
||||
/// from one that means "discovery was configured and silently produced nothing"
|
||||
/// (GWC-37). <see cref="IAlarmWatchListResolver.ResolveAsync"/> deliberately never throws
|
||||
/// on a discovery failure, so without this the two are indistinguishable at the call site
|
||||
/// and a Galaxy Repository outage at startup can only be found in a log line.
|
||||
/// </summary>
|
||||
/// <param name="Targets">The resolved watch-list, in order; possibly empty.</param>
|
||||
/// <param name="DiscoveryAttempted">
|
||||
/// Galaxy Repository discovery was enabled for this resolution
|
||||
/// (<c>Fallback:Discovery:UseGalaxyRepository</c>).
|
||||
/// </param>
|
||||
/// <param name="DiscoveryFailed">
|
||||
/// Discovery was attempted and failed, so <paramref name="Targets"/> holds only what
|
||||
/// configuration contributed.
|
||||
/// </param>
|
||||
/// <param name="DiscoveryFailureMessage">
|
||||
/// The discovery failure's message, or <c>null</c> when discovery did not fail.
|
||||
/// </param>
|
||||
public sealed record AlarmWatchListResolution(
|
||||
IReadOnlyList<AlarmSubtagTarget> Targets,
|
||||
bool DiscoveryAttempted,
|
||||
bool DiscoveryFailed,
|
||||
string? DiscoveryFailureMessage);
|
||||
@@ -31,7 +31,7 @@ public sealed class AlarmWatchListResolver : IAlarmWatchListResolver
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public async Task<IReadOnlyList<AlarmSubtagTarget>> ResolveAsync(
|
||||
public async Task<AlarmWatchListResolution> ResolveAsync(
|
||||
AlarmsOptions options,
|
||||
CancellationToken cancellationToken = default)
|
||||
{
|
||||
@@ -39,6 +39,11 @@ public sealed class AlarmWatchListResolver : IAlarmWatchListResolver
|
||||
|
||||
AlarmDiscoveryOptions discovery = options.Fallback.Discovery;
|
||||
|
||||
// GWC-37: the caller cannot grade an empty watch-list without knowing whether discovery
|
||||
// was asked to contribute and whether it failed while doing so, because the failure is
|
||||
// deliberately swallowed below.
|
||||
string? discoveryFailureMessage = null;
|
||||
|
||||
// Config fallback area used only for explicit IncludeAttributes entries (which
|
||||
// carry no discovered area): discovery area, else the default area (may be empty).
|
||||
string configFallbackArea = string.IsNullOrEmpty(discovery.Area) ? options.DefaultArea : discovery.Area;
|
||||
@@ -68,6 +73,7 @@ public sealed class AlarmWatchListResolver : IAlarmWatchListResolver
|
||||
_logger.LogWarning(
|
||||
ex,
|
||||
"Galaxy Repository alarm-attribute discovery failed; continuing with configuration-only watch-list.");
|
||||
discoveryFailureMessage = ex.Message;
|
||||
rows = [];
|
||||
}
|
||||
|
||||
@@ -148,7 +154,11 @@ public sealed class AlarmWatchListResolver : IAlarmWatchListResolver
|
||||
_logger.LogDebug("Resolved alarm subtag watch-list with {TargetCount} target(s).", targets.Count);
|
||||
}
|
||||
|
||||
return targets;
|
||||
return new AlarmWatchListResolution(
|
||||
targets,
|
||||
DiscoveryAttempted: discovery.UseGalaxyRepository,
|
||||
DiscoveryFailed: discoveryFailureMessage is not null,
|
||||
discoveryFailureMessage);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
|
||||
@@ -57,6 +57,20 @@ public sealed class GatewayAlarmMonitor : BackgroundService, IGatewayAlarmServic
|
||||
private string _providerReason = string.Empty;
|
||||
private DateTimeOffset _providerSince = DateTimeOffset.UtcNow;
|
||||
|
||||
// GWC-37 subtag-coverage verdict, guarded by _sync: whether the watch-list the worker is
|
||||
// using can actually cover anything, and why not. Independent of _providerDegraded (which
|
||||
// means "the subtag provider is the one serving"): in Auto mode the alarm manager can be
|
||||
// serving perfectly while the standby behind it covers zero alarms, and that is invisible
|
||||
// until the failover that needs it. Folded into BuildProviderStatus so it reaches the feed,
|
||||
// the dashboard badge and every late-joining subscriber rather than only a startup log line.
|
||||
private bool _subtagCoverageMissing;
|
||||
private string _subtagCoverageReason = string.Empty;
|
||||
|
||||
// The watch-list references actually sent to the worker in SubscribeAlarms, guarded by _sync.
|
||||
// The worker holds that list for the life of the subscription, so a later re-resolution is
|
||||
// compared against this to report drift rather than silently diverging.
|
||||
private IReadOnlyCollection<string> _sentWatchListReferences = [];
|
||||
|
||||
// Whether the worker's most recent reconcile fetch was capped, guarded by _sync.
|
||||
// Written only by ApplyReconcile (and cleared with the cache), so it describes the last full
|
||||
// reconcile — not necessarily the current _alarms contents, which live transitions keep moving
|
||||
@@ -329,12 +343,25 @@ public sealed class GatewayAlarmMonitor : BackgroundService, IGatewayAlarmServic
|
||||
|
||||
private async Task SubscribeAlarmsAsync(string sessionId, string subscription, CancellationToken cancellationToken)
|
||||
{
|
||||
IReadOnlyList<AlarmSubtagTarget> watchList = await _watchListResolver
|
||||
AlarmWatchListResolution resolution = await _watchListResolver
|
||||
.ResolveAsync(_options, cancellationToken)
|
||||
.ConfigureAwait(false);
|
||||
IReadOnlyList<AlarmSubtagTarget> watchList = resolution.Targets;
|
||||
|
||||
AlarmProviderMode forcedMode = MapForcedMode(_options.Fallback.Mode);
|
||||
|
||||
// GWC-37: grade the coverage before the command goes out, so a Galaxy Repository outage at
|
||||
// startup is a standing status on the feed instead of one log line an operator reads weeks
|
||||
// later — when the failover that needed the watch-list finds it empty. Broadcast because
|
||||
// feed subscribers outlive the monitor's worker session and may already be attached.
|
||||
lock (_sync)
|
||||
{
|
||||
_sentWatchListReferences = watchList
|
||||
.Select(target => target.AlarmFullReference)
|
||||
.ToHashSet(StringComparer.OrdinalIgnoreCase);
|
||||
SetSubtagCoverage(GradeSubtagCoverage(forcedMode, resolution, staleAgainstSent: false), broadcast: true);
|
||||
}
|
||||
|
||||
_logger.LogInformation(
|
||||
"Alarm subscribe: forcedMode={ForcedMode} configMode={ConfigMode} watchList={WatchListCount}.",
|
||||
forcedMode, _options.Fallback.Mode, watchList.Count);
|
||||
@@ -518,6 +545,20 @@ public sealed class GatewayAlarmMonitor : BackgroundService, IGatewayAlarmServic
|
||||
AlarmProviderMode toMode = change.Mode;
|
||||
string reason = change.Reason ?? string.Empty;
|
||||
|
||||
// GWC-37: this is the moment the subtag watch-list stops being theoretical — the provider
|
||||
// that is about to serve is the one built from it. Re-resolve BEFORE the status is
|
||||
// published so the frame the operator sees on the failover carries the coverage verdict
|
||||
// rather than a second frame later, and so a watch-list that has drifted since
|
||||
// SubscribeAlarms is reported instead of silently diverging. The re-resolution is NOT
|
||||
// re-sent: the worker rejects a second SubscribeAlarms while subscribed, and the only way
|
||||
// to replace the list — Unsubscribe then Subscribe — rebuilds the FailoverAlarmConsumer,
|
||||
// which restarts on the primary that just failed and discards the warm standby state, so
|
||||
// it would flap the feed rather than repair it. A monitor restart re-resolves and re-sends.
|
||||
if (toMode == AlarmProviderMode.Subtag)
|
||||
{
|
||||
await RefreshSubtagCoverageAsync(cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
AlarmProviderStatus status;
|
||||
int fromModeInt;
|
||||
lock (_sync)
|
||||
@@ -565,14 +606,141 @@ public sealed class GatewayAlarmMonitor : BackgroundService, IGatewayAlarmServic
|
||||
}
|
||||
}
|
||||
|
||||
// GWC-37: re-resolves the subtag watch-list and re-grades the coverage verdict without
|
||||
// touching the worker's subscription. Best-effort: a resolver that throws (only cancellation
|
||||
// and programming errors do — discovery failure is reported in the resolution) leaves the
|
||||
// verdict from the last subscribe in place, because a stale verdict is better than none.
|
||||
// Does not broadcast: the callers publish a status frame of their own immediately after.
|
||||
private async Task RefreshSubtagCoverageAsync(CancellationToken cancellationToken)
|
||||
{
|
||||
AlarmProviderMode forcedMode = MapForcedMode(_options.Fallback.Mode);
|
||||
AlarmWatchListResolution resolution;
|
||||
try
|
||||
{
|
||||
resolution = await _watchListResolver.ResolveAsync(_options, cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
catch (OperationCanceledException)
|
||||
{
|
||||
throw;
|
||||
}
|
||||
catch (Exception exception)
|
||||
{
|
||||
_logger.LogWarning(
|
||||
exception,
|
||||
"Re-resolving the alarm subtag watch-list failed; keeping the coverage verdict from the last subscribe.");
|
||||
return;
|
||||
}
|
||||
|
||||
lock (_sync)
|
||||
{
|
||||
bool stale = !resolution.Targets
|
||||
.Select(target => target.AlarmFullReference)
|
||||
.ToHashSet(StringComparer.OrdinalIgnoreCase)
|
||||
.SetEquals(_sentWatchListReferences);
|
||||
SetSubtagCoverage(GradeSubtagCoverage(forcedMode, resolution, stale), broadcast: false);
|
||||
}
|
||||
}
|
||||
|
||||
// Grades one resolution against the effective mode. Returns the reason the subtag coverage is
|
||||
// unsound, or an empty string when it is sound (or irrelevant, as under ForceAlarmManager).
|
||||
// Empty targets are the total-loss case; a failed discovery with a usable config-only list is
|
||||
// a completeness warning; drift is reported separately because the worker cannot be updated.
|
||||
private string GradeSubtagCoverage(
|
||||
AlarmProviderMode forcedMode,
|
||||
AlarmWatchListResolution resolution,
|
||||
bool staleAgainstSent)
|
||||
{
|
||||
// ForceAlarmManager never builds a subtag consumer, so its watch-list is genuinely unused.
|
||||
if (forcedMode == AlarmProviderMode.Alarmmgr)
|
||||
{
|
||||
return string.Empty;
|
||||
}
|
||||
|
||||
// Nothing was asked of discovery and nothing was configured: subtag fallback is simply not
|
||||
// in use in this deployment (Auto with an empty list subscribes alarmmgr-only), so there is
|
||||
// no coverage to warn about.
|
||||
if (resolution.Targets.Count == 0
|
||||
&& !resolution.DiscoveryAttempted
|
||||
&& _options.Fallback.Discovery.IncludeAttributes.Length == 0
|
||||
&& forcedMode != AlarmProviderMode.Subtag)
|
||||
{
|
||||
return string.Empty;
|
||||
}
|
||||
|
||||
List<string> reasons = [];
|
||||
if (resolution.Targets.Count == 0)
|
||||
{
|
||||
reasons.Add(AlarmProviderReasons.SubtagWatchListEmpty);
|
||||
}
|
||||
|
||||
if (resolution.DiscoveryFailed)
|
||||
{
|
||||
reasons.Add(resolution.DiscoveryFailureMessage is { Length: > 0 } message
|
||||
? $"{AlarmProviderReasons.SubtagDiscoveryFailed} ({message})"
|
||||
: AlarmProviderReasons.SubtagDiscoveryFailed);
|
||||
}
|
||||
|
||||
if (staleAgainstSent)
|
||||
{
|
||||
reasons.Add(
|
||||
$"{AlarmProviderReasons.SubtagWatchListStale} ({resolution.Targets.Count} target(s) now "
|
||||
+ $"resolve against {_sentWatchListReferences.Count} sent at subscribe); restart the alarm "
|
||||
+ "monitor to re-send it");
|
||||
}
|
||||
|
||||
return string.Join("; ", reasons);
|
||||
}
|
||||
|
||||
// Caller holds _sync. Applies a coverage verdict, logging and (optionally) broadcasting only
|
||||
// when it actually changed so a stable degraded state does not repeat a frame every pass.
|
||||
private void SetSubtagCoverage(string reason, bool broadcast)
|
||||
{
|
||||
bool missing = reason.Length > 0;
|
||||
if (missing == _subtagCoverageMissing && string.Equals(reason, _subtagCoverageReason, StringComparison.Ordinal))
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
_subtagCoverageMissing = missing;
|
||||
_subtagCoverageReason = reason;
|
||||
|
||||
if (missing)
|
||||
{
|
||||
_logger.LogWarning("Alarm subtag coverage is unsound: {Reason}", reason);
|
||||
}
|
||||
else
|
||||
{
|
||||
_logger.LogInformation("Alarm subtag coverage is sound.");
|
||||
}
|
||||
|
||||
if (broadcast)
|
||||
{
|
||||
BroadcastToAll(new AlarmFeedMessage { ProviderStatus = BuildProviderStatus() });
|
||||
}
|
||||
}
|
||||
|
||||
// Caller holds _sync. Builds an AlarmProviderStatus snapshot of the current state.
|
||||
//
|
||||
// GWC-37: degraded is the OR of "the subtag provider is the one serving" and "the subtag
|
||||
// watch-list cannot cover anything". The second is what makes an empty watch-list visible in
|
||||
// Auto mode, where the alarm manager is serving happily and the standby behind it is empty.
|
||||
// The reasons are concatenated when both apply, which is the worst case worth naming loudly:
|
||||
// failed over to a provider that covers nothing.
|
||||
private AlarmProviderStatus BuildProviderStatus()
|
||||
{
|
||||
bool degraded = _providerDegraded || _subtagCoverageMissing;
|
||||
string reason = (_providerReason, _subtagCoverageReason) switch
|
||||
{
|
||||
("", var coverage) => coverage,
|
||||
(var mode, "") => mode,
|
||||
(var mode, var coverage) => $"{mode} — {coverage}",
|
||||
};
|
||||
|
||||
return new AlarmProviderStatus
|
||||
{
|
||||
Mode = _providerMode,
|
||||
Degraded = _providerDegraded,
|
||||
Reason = _providerReason,
|
||||
Degraded = degraded,
|
||||
Reason = reason,
|
||||
Since = Timestamp.FromDateTimeOffset(_providerSince),
|
||||
};
|
||||
}
|
||||
|
||||
@@ -18,13 +18,15 @@ public interface IAlarmWatchListResolver
|
||||
/// <param name="options">Alarm configuration carrying discovery and subtag-name settings.</param>
|
||||
/// <param name="cancellationToken">Token to cancel the asynchronous operation.</param>
|
||||
/// <returns>
|
||||
/// The resolved <see cref="AlarmSubtagTarget"/> watch-list, possibly empty.
|
||||
/// The resolved <see cref="AlarmSubtagTarget"/> watch-list, possibly empty, together
|
||||
/// with the discovery provenance the caller needs to grade it.
|
||||
/// Discovery being unavailable never throws — it yields an empty (or
|
||||
/// config-only) list and the caller decides what to do with it. Cancellation
|
||||
/// config-only) list with <see cref="AlarmWatchListResolution.DiscoveryFailed"/> set,
|
||||
/// and the caller decides what to do with it. Cancellation
|
||||
/// is the one exception: a triggered <paramref name="cancellationToken"/>
|
||||
/// still propagates an <see cref="OperationCanceledException"/>.
|
||||
/// </returns>
|
||||
Task<IReadOnlyList<AlarmSubtagTarget>> ResolveAsync(
|
||||
Task<AlarmWatchListResolution> ResolveAsync(
|
||||
AlarmsOptions options,
|
||||
CancellationToken cancellationToken = default);
|
||||
}
|
||||
|
||||
@@ -0,0 +1,34 @@
|
||||
using Microsoft.AspNetCore.Authorization;
|
||||
|
||||
namespace ZB.MOM.WW.MxGateway.Server.Dashboard;
|
||||
|
||||
/// <summary>
|
||||
/// Handler for <see cref="AuthenticatedDashboardRoleRequirement"/>: succeeds only when the caller
|
||||
/// is authenticated AND holds one of the requirement's roles. It reads no options at all, which
|
||||
/// is the point — there is no configuration value that can turn this into a bypass (SEC-40).
|
||||
/// </summary>
|
||||
public sealed class AuthenticatedDashboardRoleHandler
|
||||
: AuthorizationHandler<AuthenticatedDashboardRoleRequirement>
|
||||
{
|
||||
/// <inheritdoc />
|
||||
protected override Task HandleRequirementAsync(
|
||||
AuthorizationHandlerContext context,
|
||||
AuthenticatedDashboardRoleRequirement requirement)
|
||||
{
|
||||
if (context.User.Identity?.IsAuthenticated != true)
|
||||
{
|
||||
return Task.CompletedTask;
|
||||
}
|
||||
|
||||
foreach (string role in requirement.RequiredRoles)
|
||||
{
|
||||
if (context.User.IsInRole(role))
|
||||
{
|
||||
context.Succeed(requirement);
|
||||
return Task.CompletedTask;
|
||||
}
|
||||
}
|
||||
|
||||
return Task.CompletedTask;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,39 @@
|
||||
using Microsoft.AspNetCore.Authorization;
|
||||
|
||||
namespace ZB.MOM.WW.MxGateway.Server.Dashboard;
|
||||
|
||||
/// <summary>
|
||||
/// Requirement satisfied ONLY by an authenticated principal holding one of
|
||||
/// <see cref="RequiredRoles"/> as a real role claim. Unlike
|
||||
/// <see cref="DashboardAuthorizationRequirement"/> there is no environment bypass: it is a
|
||||
/// distinct requirement type, so <see cref="DashboardAuthorizationHandler"/> —
|
||||
/// an <c>AuthorizationHandler<DashboardAuthorizationRequirement></c> — is never invoked for
|
||||
/// it and its <c>Authentication:Mode=Disabled</c> / anonymous-localhost arms cannot reach it.
|
||||
/// The type distinction is the enforcement; adding a bypass arm to the other handler cannot leak
|
||||
/// into this one by accident.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Introduced for SEC-40 to gate the alarm feed (<c>AlarmsHub</c> and the <c>/alarms</c> page),
|
||||
/// which is gateway-central and session-less and therefore carries no per-session provenance the
|
||||
/// SEC-25 tag ACL could scope it by. Requiring a real dashboard role claim is the narrowing that
|
||||
/// is actually available: it keeps galaxy-wide alarm metadata — alarm reference (a full tag path),
|
||||
/// state, severity, category, area, acknowledging operator — away from the read-only environment
|
||||
/// bypasses, which exist for local development and unauthenticated deployments rather than as a
|
||||
/// grant of plant-wide visibility. The same pattern the shared Secrets UI relies on: a policy the
|
||||
/// dashboard bypass handler cannot satisfy because the requirement type differs.
|
||||
/// </remarks>
|
||||
/// <param name="requiredRoles">Roles any one of which satisfies the requirement.</param>
|
||||
public sealed class AuthenticatedDashboardRoleRequirement(IReadOnlyList<string> requiredRoles)
|
||||
: IAuthorizationRequirement
|
||||
{
|
||||
/// <summary>Gets the roles, any one of which satisfies this requirement.</summary>
|
||||
public IReadOnlyList<string> RequiredRoles { get; } = requiredRoles;
|
||||
|
||||
/// <summary>
|
||||
/// Gets the requirement satisfied by an authenticated caller holding either dashboard role.
|
||||
/// Same role set as <see cref="DashboardAuthorizationRequirement.AnyDashboardRole"/>; the
|
||||
/// difference is entirely that no environment bypass applies.
|
||||
/// </summary>
|
||||
public static AuthenticatedDashboardRoleRequirement AnyDashboardRole { get; } =
|
||||
new([DashboardRoles.Admin, DashboardRoles.Viewer]);
|
||||
}
|
||||
@@ -12,7 +12,14 @@
|
||||
<NavRailItem Href="/sessions" Text="Sessions" />
|
||||
<NavRailItem Href="/workers" Text="Workers" />
|
||||
<NavRailItem Href="/events" Text="Events" />
|
||||
<NavRailItem Href="/alarms" Text="Alarms" />
|
||||
@* Gated on the SAME policy the /alarms page and AlarmsHub enforce (SEC-40), for the
|
||||
same reason the Secrets link below is: the page denies an environment-bypass
|
||||
caller outright, so an ungated link would be a dead end. *@
|
||||
<AuthorizeView Policy="@DashboardAuthenticationDefaults.AlarmsPolicy">
|
||||
<Authorized>
|
||||
<NavRailItem Href="/alarms" Text="Alarms" />
|
||||
</Authorized>
|
||||
</AuthorizeView>
|
||||
</NavRailSection>
|
||||
<NavRailSection Title="Galaxy" Key="galaxy">
|
||||
<NavRailItem Href="/galaxy" Text="Repository" />
|
||||
|
||||
@@ -1,6 +1,15 @@
|
||||
@page "/alarms"
|
||||
@implements IAsyncDisposable
|
||||
@using Microsoft.AspNetCore.Authorization
|
||||
@using ZB.MOM.WW.MxGateway.Server.Alarms
|
||||
@* SEC-40: the alarm feed needs a real authenticated dashboard role, not the read-only
|
||||
environment bypasses (Authentication:Mode=Disabled, anonymous localhost) that satisfy the
|
||||
ViewerPolicy MapRazorComponents<App>() applies. This page reads the gateway-central alarm
|
||||
monitor in process, exactly the surface AlarmsHub serves to external clients, so it carries the
|
||||
SAME policy as the hub — gating only the hub would leave the disclosure one navigation away.
|
||||
Alarm metadata stays galaxy-wide for an authenticated Viewer: the monitor has no per-session
|
||||
provenance, so the SEC-25 tag ACL has nothing to intersect. *@
|
||||
@attribute [Authorize(Policy = DashboardAuthenticationDefaults.AlarmsPolicy)]
|
||||
@inject IDashboardLiveDataService LiveData
|
||||
@inject IOptions<GatewayOptions> GatewayOptions
|
||||
@inject IGatewayAlarmService AlarmService
|
||||
|
||||
@@ -30,6 +30,14 @@ public sealed record DashboardAlarmProviderStatus(
|
||||
/// </summary>
|
||||
public const string ForcedSubtagLabel = "Subtag monitoring (forced)";
|
||||
|
||||
/// <summary>
|
||||
/// Badge label shown when the alarm manager is serving normally but the subtag standby
|
||||
/// behind it covers nothing — an empty or undiscoverable watch-list (GWC-37). The feed is
|
||||
/// healthy today; what is degraded is the fallback that would take over tomorrow, so the
|
||||
/// badge names the standby rather than claiming the feed has failed over.
|
||||
/// </summary>
|
||||
public const string NoSubtagStandbyLabel = "Alarm Manager (no subtag standby)";
|
||||
|
||||
private const string HealthyBadge = "bg-success";
|
||||
private const string DegradedBadge = "bg-warning text-dark";
|
||||
|
||||
@@ -69,8 +77,18 @@ public sealed record DashboardAlarmProviderStatus(
|
||||
&& status.Mode == AlarmProviderMode.Subtag
|
||||
&& string.Equals(reason, AlarmProviderReasons.ForcedSubtag, StringComparison.Ordinal);
|
||||
|
||||
string label = !degraded ? AlarmManagerLabel : forced ? ForcedSubtagLabel : DegradedLabel;
|
||||
string badge = !degraded ? HealthyBadge : forced ? ForcedSubtagBadge : DegradedBadge;
|
||||
// GWC-37: degraded WITHOUT subtag mode means the alarm manager is still serving and the
|
||||
// standby is the unsound half. "Subtag monitoring (degraded)" would misreport which
|
||||
// provider is live, so this state gets its own label; the amber badge is shared because
|
||||
// the operator action — fix discovery — is equally overdue in both.
|
||||
bool standbyOnlyWarning = degraded && status.Mode != AlarmProviderMode.Subtag;
|
||||
|
||||
string label = !degraded
|
||||
? AlarmManagerLabel
|
||||
: standbyOnlyWarning ? NoSubtagStandbyLabel : forced ? ForcedSubtagLabel : DegradedLabel;
|
||||
string badge = !degraded
|
||||
? HealthyBadge
|
||||
: standbyOnlyWarning ? DegradedBadge : forced ? ForcedSubtagBadge : DegradedBadge;
|
||||
|
||||
return new DashboardAlarmProviderStatus(
|
||||
Mode: status.Mode,
|
||||
|
||||
@@ -33,6 +33,18 @@ public static class DashboardAuthenticationDefaults
|
||||
/// </summary>
|
||||
public const string HubClientsPolicy = "MxGateway.Dashboard.HubClients";
|
||||
|
||||
/// <summary>
|
||||
/// Policy guarding the alarm feed — the <c>AlarmsHub</c> SignalR surface and the
|
||||
/// <c>/alarms</c> page that reads the same central monitor in process (SEC-40). Accepts the
|
||||
/// same two schemes as <see cref="HubClientsPolicy"/> but carries
|
||||
/// <see cref="AuthenticatedDashboardRoleRequirement"/>, a requirement type
|
||||
/// <see cref="DashboardAuthorizationHandler"/> does not handle, so the environment bypasses
|
||||
/// (<c>Authentication:Mode=Disabled</c> and anonymous localhost) cannot satisfy it. The alarm
|
||||
/// feed is galaxy-wide and session-less, so unlike a session event stream there is no tag
|
||||
/// intersection to narrow it with — requiring a real role claim is the available narrowing.
|
||||
/// </summary>
|
||||
public const string AlarmsPolicy = "MxGateway.Dashboard.Alarms";
|
||||
|
||||
public const string LdapGroupClaimType = "mxgateway:ldap_group";
|
||||
public const string KeyPrefixClaimType = "mxgateway:key_prefix";
|
||||
|
||||
@@ -40,9 +52,13 @@ public static class DashboardAuthenticationDefaults
|
||||
/// Claim carrying one dashboard event-visibility tag the caller is granted (SEC-25). Stamped
|
||||
/// at cookie login by <see cref="DashboardAuthenticator"/> and at hub-token mint by
|
||||
/// <see cref="HubTokenService"/>, both resolving the caller's LDAP groups through
|
||||
/// <c>MxGateway:Dashboard:GroupToTag</c>; read by <see cref="IDashboardSessionAcl"/>. A
|
||||
/// principal carrying none of these claims is an empty-grant Viewer, which is the fail-closed
|
||||
/// default. Visibility only — it never grants data access.
|
||||
/// <c>MxGateway:Dashboard:GroupToTag</c>. Since SEC-41 <see cref="IDashboardSessionAcl"/> reads
|
||||
/// this claim only as a FALLBACK, for a principal that carries no
|
||||
/// <see cref="LdapGroupClaimType"/> claims to re-resolve from (the hub-token principal, whose
|
||||
/// payload carries tags but not groups); a cookie principal is re-resolved against current
|
||||
/// options instead, so a stamped tag cannot outlive a config-revoked grant. A principal
|
||||
/// carrying neither kind of claim is an empty-grant Viewer, which is the fail-closed default.
|
||||
/// Visibility only — it never grants data access.
|
||||
/// </summary>
|
||||
public const string DashboardTagClaimType = "zb:dashboardtag";
|
||||
|
||||
|
||||
@@ -104,9 +104,11 @@ public static class DashboardEndpointRouteBuilderExtensions
|
||||
.AllowAnonymous()
|
||||
.WithName("DashboardAccessDenied");
|
||||
|
||||
// SignalR hubs. Authorization is enforced on the hub class via
|
||||
// [Authorize(Policy = HubClientsPolicy)] so the policy accepts either
|
||||
// the dashboard cookie or a HubToken bearer.
|
||||
// SignalR hubs. Authorization is enforced on the hub class via [Authorize], so both
|
||||
// policies below accept either the dashboard cookie or a HubToken bearer. Snapshot and
|
||||
// events use HubClientsPolicy; AlarmsHub uses AlarmsPolicy, which additionally requires a
|
||||
// real authenticated dashboard role (SEC-40) — see AlarmsHub for why the alarm feed is
|
||||
// gated harder than the session-scoped ones.
|
||||
endpoints.MapHub<DashboardSnapshotHub>("/hubs/snapshot");
|
||||
endpoints.MapHub<AlarmsHub>("/hubs/alarms");
|
||||
endpoints.MapHub<EventsHub>("/hubs/events");
|
||||
|
||||
@@ -177,8 +177,20 @@ public static class DashboardServiceCollectionExtensions
|
||||
DashboardAuthenticationDefaults.AuthenticationScheme,
|
||||
DashboardAuthenticationDefaults.HubAuthenticationScheme)
|
||||
.AddRequirements(DashboardAuthorizationRequirement.AnyDashboardRole));
|
||||
// SEC-40. Same two schemes as HubClientsPolicy (the alarm feed has an in-page reader
|
||||
// and an external hub reader), but a requirement type DashboardAuthorizationHandler
|
||||
// does not handle, so neither Authentication:Mode=Disabled nor anonymous localhost
|
||||
// can satisfy it. Galaxy-wide alarm metadata therefore needs a real dashboard role.
|
||||
authorization.AddPolicy(
|
||||
DashboardAuthenticationDefaults.AlarmsPolicy,
|
||||
policy => policy
|
||||
.AddAuthenticationSchemes(
|
||||
DashboardAuthenticationDefaults.AuthenticationScheme,
|
||||
DashboardAuthenticationDefaults.HubAuthenticationScheme)
|
||||
.AddRequirements(AuthenticatedDashboardRoleRequirement.AnyDashboardRole));
|
||||
});
|
||||
services.AddSingleton<IAuthorizationHandler, DashboardAuthorizationHandler>();
|
||||
services.AddSingleton<IAuthorizationHandler, AuthenticatedDashboardRoleHandler>();
|
||||
|
||||
return services;
|
||||
}
|
||||
|
||||
@@ -26,22 +26,53 @@ namespace ZB.MOM.WW.MxGateway.Server.Dashboard;
|
||||
/// (ordinal-ignore-case).</description></item>
|
||||
/// </list>
|
||||
/// <para>
|
||||
/// Granted tags are read from the caller's <see cref="DashboardAuthenticationDefaults.DashboardTagClaimType"/>
|
||||
/// claims, stamped at login (<see cref="DashboardAuthenticator"/>) or at hub-token mint
|
||||
/// (<see cref="HubTokenService"/>). A principal with no such claims — anonymous localhost included —
|
||||
/// is an empty-grant Viewer.
|
||||
/// <strong>Where the grant comes from (SEC-41).</strong> The grant is re-resolved on every call
|
||||
/// from the caller's <see cref="DashboardAuthenticationDefaults.LdapGroupClaimType"/> claims
|
||||
/// through <c>MxGateway:Dashboard:GroupToTag</c> as it stands in the currently bound options —
|
||||
/// the same single <see cref="DashboardGroupTagMapping.MapGroupsToTags"/> call
|
||||
/// <see cref="HubTokenService.Issue(ClaimsPrincipal)"/> makes at token mint. It deliberately does
|
||||
/// NOT trust the <see cref="DashboardAuthenticationDefaults.DashboardTagClaimType"/> claims
|
||||
/// stamped into the cookie at login: those were resolved under whatever map was bound when the
|
||||
/// user signed in, and the auth cookie survives the gateway restart that applies a
|
||||
/// <c>GroupToTag</c> edit (options bind once at startup; data-protection keys persist). Trusting
|
||||
/// them let a revoked grant live on for the sliding 8-hour cookie life, renewable indefinitely by
|
||||
/// activity, on the one seam — cookie-authenticated — that the 5-minute hub-token bound did not
|
||||
/// cover.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <strong>Fallback.</strong> A principal carrying no group claims at all falls back to its
|
||||
/// stamped tag claims. That arm exists for the hub-token principal: <see cref="HubTokenService"/>
|
||||
/// re-resolves the grant at mint and rehydrates it as tag claims only — the token payload carries
|
||||
/// no LDAP groups — so re-resolution has already happened for that caller, one token lifetime ago
|
||||
/// at worst. The fallback cannot re-open the cookie hole it replaces, because a cookie principal
|
||||
/// with a tag claim always also carries the group claims it was derived from
|
||||
/// (<see cref="DashboardAuthenticator"/> stamps both from one group list), and a login that
|
||||
/// yielded no groups yields no tags either. A principal with neither kind of claim — anonymous
|
||||
/// localhost included — is an empty-grant Viewer.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <strong>Revocation bounds this produces</strong> (mirrored in
|
||||
/// <c>docs/GatewayDashboardDesign.md</c>): a <c>GroupToTag</c> entry removed from config takes
|
||||
/// effect on the next subscribe after the restart that rebinds options, on both seams; an LDAP
|
||||
/// group-membership revocation takes effect when the principal is rebuilt — at re-login (bounded
|
||||
/// by the cookie's sliding 8-hour idle life) for the cookie seam, and within one 5-minute token
|
||||
/// lifetime for the hub-token seam; an already-open subscription is never re-checked and lives
|
||||
/// until it drops.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// This sits on the <em>subscribe</em> path, not the per-event path, and must stay cheap enough to
|
||||
/// keep it there: the only per-call work is the claim scan plus a session-registry lookup, with no
|
||||
/// intermediate collection built. A per-event re-check is deliberately not needed — a joined SignalR
|
||||
/// keep it there: the only per-call work is a claim scan, the group→tag map lookup, and a
|
||||
/// session-registry lookup. A per-event re-check is deliberately not needed — a joined SignalR
|
||||
/// group and an in-process subscription are both per-session, and <see cref="GatewaySession.Tags"/>
|
||||
/// is immutable for the session's life, so the decision taken at subscribe time cannot go stale
|
||||
/// while the subscription lives.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
/// <param name="sessionManager">Registry the session id is resolved against.</param>
|
||||
/// <param name="options">Gateway options supplying <c>Dashboard:UntaggedSessionVisibility</c>.</param>
|
||||
/// <param name="options">
|
||||
/// Gateway options supplying <c>Dashboard:UntaggedSessionVisibility</c> and the
|
||||
/// <c>Dashboard:GroupToTag</c> map the caller's grant is re-resolved through.
|
||||
/// </param>
|
||||
public sealed class DashboardSessionAcl(
|
||||
ISessionManager sessionManager,
|
||||
IOptions<GatewayOptions> options) : IDashboardSessionAcl
|
||||
@@ -64,14 +95,53 @@ public sealed class DashboardSessionAcl(
|
||||
return false;
|
||||
}
|
||||
|
||||
DashboardOptions dashboard = options.Value.Dashboard;
|
||||
|
||||
if (session.Tags.Count == 0)
|
||||
{
|
||||
return options.Value.Dashboard.UntaggedSessionVisibility == UntaggedSessionVisibility.AllViewers;
|
||||
return dashboard.UntaggedSessionVisibility == UntaggedSessionVisibility.AllViewers;
|
||||
}
|
||||
|
||||
// Session.Tags is an ordinal-ignore-case set, so the containment test carries the
|
||||
// comparison; scanning the claims (rather than materializing the grant) keeps this
|
||||
// allocation-free beyond the claim enumerator.
|
||||
return GrantIntersectsSessionTags(principal, session, dashboard);
|
||||
}
|
||||
|
||||
private static bool GrantIntersectsSessionTags(
|
||||
ClaimsPrincipal principal,
|
||||
GatewaySession session,
|
||||
DashboardOptions dashboard)
|
||||
{
|
||||
// Collected rather than streamed because MapGroupsToTags takes the whole group list;
|
||||
// the list stays null (no allocation) for the hub-token principal, which carries none.
|
||||
List<string>? groups = null;
|
||||
foreach (Claim groupClaim in principal.FindAll(DashboardAuthenticationDefaults.LdapGroupClaimType))
|
||||
{
|
||||
(groups ??= []).Add(groupClaim.Value);
|
||||
}
|
||||
|
||||
if (groups is not null)
|
||||
{
|
||||
// SEC-41: current options, not the login-time snapshot. MapGroupsToTags returns an
|
||||
// OrdinalIgnoreCase set, so probing it with the session's tags carries the
|
||||
// case-insensitive comparison in the same direction the claim scan below does.
|
||||
IReadOnlySet<string> granted = DashboardGroupTagMapping.MapGroupsToTags(
|
||||
groups,
|
||||
dashboard.GroupToTag);
|
||||
|
||||
foreach (string sessionTag in session.Tags)
|
||||
{
|
||||
if (granted.Contains(sessionTag))
|
||||
{
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
return false;
|
||||
}
|
||||
|
||||
// No group claims: the hub-token arm documented on the class. Session.Tags is an
|
||||
// ordinal-ignore-case set, so the containment test carries the comparison; scanning the
|
||||
// claims (rather than materializing the grant) keeps this allocation-free beyond the
|
||||
// claim enumerator.
|
||||
foreach (Claim tagClaim in principal.FindAll(DashboardAuthenticationDefaults.DashboardTagClaimType))
|
||||
{
|
||||
if (session.Tags.Contains(tagClaim.Value))
|
||||
|
||||
@@ -9,7 +9,18 @@ namespace ZB.MOM.WW.MxGateway.Server.Dashboard.Hubs;
|
||||
/// <see cref="AllAlarmsGroup"/> on connect and receive every
|
||||
/// <c>AlarmFeedMessage</c> the monitor emits.
|
||||
/// </summary>
|
||||
[Authorize(Policy = DashboardAuthenticationDefaults.HubClientsPolicy)]
|
||||
/// <remarks>
|
||||
/// The feed is galaxy-wide: the alarm monitor is gateway-central and its messages carry no
|
||||
/// session provenance, so the SEC-25 per-session tag ACL that scopes <c>EventsHub</c> has nothing
|
||||
/// to intersect here and is deliberately not applied — every authenticated Viewer sees the whole
|
||||
/// subscription expression's alarm metadata. What SEC-40 tightened instead is the entry gate:
|
||||
/// <see cref="DashboardAuthenticationDefaults.AlarmsPolicy"/> rather than the
|
||||
/// <see cref="DashboardAuthenticationDefaults.HubClientsPolicy"/> the other hubs use, so the
|
||||
/// read-only environment bypasses (<c>Authentication:Mode=Disabled</c>, anonymous localhost) no
|
||||
/// longer receive the feed. The <c>/alarms</c> page carries the same policy — the two readers of
|
||||
/// one monitor must not disagree about who may see it.
|
||||
/// </remarks>
|
||||
[Authorize(Policy = DashboardAuthenticationDefaults.AlarmsPolicy)]
|
||||
public sealed class AlarmsHub : Hub
|
||||
{
|
||||
public const string AllAlarmsGroup = "__alarms__";
|
||||
|
||||
@@ -22,6 +22,15 @@ namespace ZB.MOM.WW.MxGateway.Server.Dashboard.Hubs;
|
||||
/// The source message is never mutated: it fans out from one feed to the gRPC
|
||||
/// <c>StreamAlarms</c> subscribers and the alarms page as well, and none of
|
||||
/// those audiences is subject to this dashboard-display flag.
|
||||
/// <para>
|
||||
/// The broadcast stays deliberately unfiltered (SEC-40). Audience is decided once, at hub
|
||||
/// admission — <see cref="AlarmsHub"/> carries
|
||||
/// <see cref="DashboardAuthenticationDefaults.AlarmsPolicy"/>, so only an authenticated caller
|
||||
/// with a real dashboard role is ever in <see cref="AlarmsHub.AllAlarmsGroup"/> — and the alarm
|
||||
/// monitor is gateway-central with no per-session provenance, so there is no per-connection
|
||||
/// narrowing this loop could apply even in principle. Adding a filter here would imply a
|
||||
/// per-caller scope the feed does not carry.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
/// <param name="alarmService">The gateway's central alarm feed.</param>
|
||||
/// <param name="hubContext">Hub context used to broadcast to the alarms group.</param>
|
||||
|
||||
@@ -150,6 +150,12 @@ public static class GatewayApplication
|
||||
builder.Services.Configure<AuthorizationOptions>(o => o.AddSecretsAuthorization());
|
||||
// Register the gateway's browse-scope provider before AddZbGalaxyRepository so the
|
||||
// library's TryAddSingleton default (NullGalaxyBrowseScopeProvider) does not win.
|
||||
// The provider depends on GatewayGrpcAuthorizationInterceptor (registered globally above)
|
||||
// having pushed the ambient identity before the library's galaxy service runs. That
|
||||
// ordering is still required, but it is no longer trusted silently: with
|
||||
// Authentication:Mode=ApiKey the provider denies with PermissionDenied and logs the broken
|
||||
// invariant when the identity is absent, instead of resolving to the full hierarchy
|
||||
// (GWC-38). Under Mode=Disabled identity-absent is expected and stays unscoped.
|
||||
builder.Services.AddSingleton<ZB.MOM.WW.GalaxyRepository.Grpc.IGalaxyBrowseScopeProvider,
|
||||
Security.Authorization.GatewayBrowseScopeProvider>();
|
||||
|
||||
|
||||
@@ -1051,6 +1051,7 @@ public sealed class MxAccessGatewayService(
|
||||
{
|
||||
WorkerClientErrorCode.CommandTimeout => StatusCode.DeadlineExceeded,
|
||||
WorkerClientErrorCode.GatewayShutdown => StatusCode.Cancelled,
|
||||
WorkerClientErrorCode.CallerCancelled => StatusCode.Cancelled,
|
||||
WorkerClientErrorCode.InvalidState => StatusCode.FailedPrecondition,
|
||||
WorkerClientErrorCode.ProtocolViolation => StatusCode.Internal,
|
||||
WorkerClientErrorCode.CommandTooLarge => StatusCode.ResourceExhausted,
|
||||
|
||||
@@ -43,7 +43,7 @@ public sealed class GatewayMetrics : IDisposable
|
||||
private readonly ConcurrentDictionary<long, Func<int>> _eventStreamBacklogSources = new();
|
||||
private long _nextEventStreamBacklogSourceId;
|
||||
|
||||
// GWC-30: the same pull model for the worker event queue depth. It replaces a pushed scalar that
|
||||
// PERF-02: the same pull model for the worker event queue depth. It replaces a pushed scalar that
|
||||
// every WorkerClient wrote twice per event (staged, consumed) under _syncRoot — a process-wide
|
||||
// lock on the hottest path, and last-writer-wins across sessions, so the gauge reported one
|
||||
// arbitrary session's backlog instead of the gateway's. Each client registers a source returning
|
||||
@@ -210,7 +210,7 @@ public sealed class GatewayMetrics : IDisposable
|
||||
/// <param name="method">Name of the command method.</param>
|
||||
public void CommandStarted(string method)
|
||||
{
|
||||
// GWC-30: the three command counters run two-to-three times per gRPC call, so they use
|
||||
// PERF-02: the three command counters run two-to-three times per gRPC call, so they use
|
||||
// Interlocked rather than _syncRoot — the same idiom as EventReceived. Nothing here needs a
|
||||
// consistent multi-field view; GetSnapshot reads each with Interlocked.Read.
|
||||
Interlocked.Increment(ref _commandsStarted);
|
||||
@@ -281,7 +281,7 @@ public sealed class GatewayMetrics : IDisposable
|
||||
/// Registers a live depth source for the worker event queue-depth gauge and returns a handle
|
||||
/// that removes it when disposed. Each <c>WorkerClient</c> registers once and reports its own
|
||||
/// undelivered (staged + queued) event count, so the gauge is the gateway-wide sum instead of
|
||||
/// the last value any one session happened to push (GWC-30).
|
||||
/// the last value any one session happened to push (PERF-02).
|
||||
/// </summary>
|
||||
/// <param name="depth">
|
||||
/// Returns this worker client's current undelivered event count. Invoked only at collection
|
||||
@@ -460,7 +460,7 @@ public sealed class GatewayMetrics : IDisposable
|
||||
{
|
||||
// Compute the live queue depths outside _syncRoot: the sources are the subscriber channels'
|
||||
// Count (their own locks) and the worker clients' interlocked counters, neither of which may
|
||||
// run under this lock. GWC-15, GWC-30.
|
||||
// run under this lock. GWC-15, PERF-02.
|
||||
int workerEventQueueDepth = GetWorkerEventQueueDepth();
|
||||
int grpcEventStreamQueueDepth = GetGrpcEventStreamQueueDepth();
|
||||
lock (_syncRoot)
|
||||
@@ -521,7 +521,7 @@ public sealed class GatewayMetrics : IDisposable
|
||||
}
|
||||
}
|
||||
|
||||
// Sums the undelivered event backlog across every live worker client (GWC-30).
|
||||
// Sums the undelivered event backlog across every live worker client (PERF-02).
|
||||
private int GetWorkerEventQueueDepth() => SumSources(_workerEventQueueDepthSources);
|
||||
|
||||
// Sums the live backlog across every registered event-stream subscriber.
|
||||
|
||||
+53
-7
@@ -1,26 +1,72 @@
|
||||
using Grpc.Core;
|
||||
using Microsoft.Extensions.Options;
|
||||
using ZB.MOM.WW.GalaxyRepository.Grpc;
|
||||
using ZB.MOM.WW.MxGateway.Server.Configuration;
|
||||
using ZB.MOM.WW.MxGateway.Server.Security.Authentication;
|
||||
|
||||
namespace ZB.MOM.WW.MxGateway.Server.Security.Authorization;
|
||||
|
||||
/// <summary>Scopes Galaxy browse results to the calling API key's BrowseSubtrees constraint.</summary>
|
||||
public sealed class GatewayBrowseScopeProvider(IGatewayRequestIdentityAccessor identityAccessor)
|
||||
/// <remarks>
|
||||
/// Distinguishes two states the pre-GWC-38 code collapsed into one: "authenticated, no
|
||||
/// BrowseSubtrees constraint" (legitimately unscoped — the common case) and "no ambient identity
|
||||
/// at all" (the interceptor-ordering invariant below is broken). The second used to resolve to the
|
||||
/// full hierarchy silently, which made a scoping bypass invisible; it now denies loudly.
|
||||
/// </remarks>
|
||||
/// <param name="identityAccessor">Accessor for the ambient per-call API-key identity.</param>
|
||||
/// <param name="options">Gateway options supplying <c>MxGateway:Authentication:Mode</c>.</param>
|
||||
/// <param name="logger">Logger used to name the broken invariant when identity is absent.</param>
|
||||
public sealed class GatewayBrowseScopeProvider(
|
||||
IGatewayRequestIdentityAccessor identityAccessor,
|
||||
IOptions<GatewayOptions> options,
|
||||
ILogger<GatewayBrowseScopeProvider> logger)
|
||||
: IGalaxyBrowseScopeProvider
|
||||
{
|
||||
/// <summary>Resolves the Galaxy browse subtrees the calling API key is scoped to, from the ambient request identity.</summary>
|
||||
/// <param name="context">The gRPC server call context for the in-flight browse request.</param>
|
||||
/// <returns>The caller's configured browse subtrees, or <see langword="null"/>/empty when the caller is unscoped (full hierarchy).</returns>
|
||||
/// <exception cref="RpcException">
|
||||
/// <see cref="StatusCode.PermissionDenied"/> when API-key authentication is enabled but no
|
||||
/// ambient identity is present — the invariant described below has been broken and the
|
||||
/// request's scope cannot be established.
|
||||
/// </exception>
|
||||
public IReadOnlyList<string>? ResolveBrowseSubtrees(ServerCallContext context)
|
||||
{
|
||||
// Invariant: the caller identity is the ambient one pushed by
|
||||
// GatewayGrpcAuthorizationInterceptor earlier in this same call. That interceptor is
|
||||
// registered globally and therefore runs before the library's galaxy service, so
|
||||
// Current is populated for any authenticated galaxy RPC. If it is somehow absent we
|
||||
// return empty BrowseSubtrees, which the library treats as "no scoping" (full
|
||||
// hierarchy) — safe only because the global interceptor has already authenticated and
|
||||
// authorized the request first. Do not reorder the interceptor without revisiting this.
|
||||
ApiKeyConstraints constraints = identityAccessor.Current?.EffectiveConstraints ?? ApiKeyConstraints.Empty;
|
||||
return constraints.BrowseSubtrees;
|
||||
// Current is populated for any authenticated galaxy RPC. Do not reorder the interceptor
|
||||
// without revisiting this.
|
||||
ApiKeyIdentity? identity = identityAccessor.Current;
|
||||
if (identity is not null)
|
||||
{
|
||||
return identity.EffectiveConstraints.BrowseSubtrees;
|
||||
}
|
||||
|
||||
// Mode=Disabled is the one configuration in which identity-absent is expected rather than
|
||||
// broken: nothing authenticates, so no key and no constraints exist to scope by, and the
|
||||
// deployment has already accepted an unauthenticated gRPC surface. Empty BrowseSubtrees is
|
||||
// what the library reads as "no scoping".
|
||||
if (options.Value.Authentication.Mode == AuthenticationMode.Disabled)
|
||||
{
|
||||
return ApiKeyConstraints.Empty.BrowseSubtrees;
|
||||
}
|
||||
|
||||
// Authentication is on, so an identity MUST have been pushed before the galaxy service ran.
|
||||
// Failing open here would hand a browse-scoped key the full hierarchy with nothing logged
|
||||
// and nothing failing — a constraint-narrowing control's absence is invisible in normal
|
||||
// operation. Deny instead, and name the invariant so the misconfiguration is diagnosable
|
||||
// from one log line rather than from a scoping audit nobody thought to run.
|
||||
logger.LogWarning(
|
||||
"Galaxy browse scope could not be resolved for {Method}: no ambient API-key identity "
|
||||
+ "while Authentication:Mode is {Mode}. GatewayGrpcAuthorizationInterceptor must run "
|
||||
+ "before the Galaxy service on every call; the request is denied rather than served "
|
||||
+ "unscoped.",
|
||||
context.Method,
|
||||
options.Value.Authentication.Mode);
|
||||
|
||||
throw new RpcException(new Status(
|
||||
StatusCode.PermissionDenied,
|
||||
"Browse scope could not be established for this request."));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -39,6 +39,14 @@ public sealed class GatewaySession
|
||||
private bool _everHadEventSubscriber;
|
||||
private SessionEventDistributor? _eventDistributor;
|
||||
private bool _eventDistributorStarted;
|
||||
// GWC-34: set under _syncRoot when DisposeAsync nulls _eventDistributor, making the
|
||||
// "one distributor per session, never reset" lifecycle symmetric at both ends. Without it
|
||||
// EnsureDistributorCreated reads the null as "not created yet" and builds a REPLACEMENT
|
||||
// distributor after teardown — one whose pump can never start (_eventDistributorStarted is a
|
||||
// one-way latch) and whose channel is never completed, so an attach that stalled across the
|
||||
// close+dispose registers on it and blocks in WaitToReadAsync until the client's own deadline,
|
||||
// silently and invisibly to metrics.
|
||||
private bool _eventDistributorDisposed;
|
||||
private bool _dashboardMirrorStarted;
|
||||
private IEventSubscriberLease? _dashboardMirrorLease;
|
||||
private Task? _dashboardMirrorTask;
|
||||
@@ -532,10 +540,23 @@ public sealed class GatewaySession
|
||||
// start). Both the construction and the started-flag flip happen under _syncRoot so two
|
||||
// concurrent callers (e.g. MarkReady's dashboard mirror and a racing first
|
||||
// AttachEventSubscriber) agree on a single distributor and a single start.
|
||||
//
|
||||
// GWC-34: after DisposeAsync the null distributor means "gone", not "not built yet", so this
|
||||
// throws instead of constructing a replacement nothing will ever pump. The attach paths already
|
||||
// roll back and surface the exception, which is the same SessionNotReady the readiness gates
|
||||
// raise — a caller that loses the race to teardown now fails immediately instead of hanging.
|
||||
private SessionEventDistributor EnsureDistributorCreated(out bool startNow)
|
||||
{
|
||||
lock (_syncRoot)
|
||||
{
|
||||
if (_eventDistributorDisposed)
|
||||
{
|
||||
throw new SessionManagerException(
|
||||
SessionManagerErrorCode.SessionNotReady,
|
||||
$"Session {SessionId} is not ready for event streaming. Its event distributor was "
|
||||
+ "disposed with the session.");
|
||||
}
|
||||
|
||||
if (_eventDistributor is null)
|
||||
{
|
||||
EventOptions eventOptions = _eventStreaming.EventOptions;
|
||||
@@ -672,7 +693,37 @@ public sealed class GatewaySession
|
||||
// dashboard mirror only disconnects itself and never faults the session.
|
||||
// These three calls are OUTSIDE _syncRoot to avoid holding it across
|
||||
// EnsureDistributorCreated's own lock and StartAsync's Task.Run.
|
||||
SessionEventDistributor distributor = EnsureDistributorCreated(out bool startNow);
|
||||
//
|
||||
// GWC-34: the state check above can be overtaken by a close+dispose before we get here, and
|
||||
// EnsureDistributorCreated now says so instead of building an orphan distributor. The
|
||||
// dashboard mirror is a best-effort observability attachment on a session that is already
|
||||
// gone, so it is abandoned here rather than propagated out of MarkReady — the mirror-started
|
||||
// latch stays set (nothing may start one on a disposed session) and the loop's token source
|
||||
// is released so it is not orphaned.
|
||||
SessionEventDistributor distributor;
|
||||
bool startNow;
|
||||
try
|
||||
{
|
||||
distributor = EnsureDistributorCreated(out startNow);
|
||||
}
|
||||
catch (SessionManagerException exception)
|
||||
{
|
||||
CancellationTokenSource? abandonedCts;
|
||||
lock (_syncRoot)
|
||||
{
|
||||
abandonedCts = _dashboardMirrorCts;
|
||||
_dashboardMirrorCts = null;
|
||||
}
|
||||
|
||||
abandonedCts?.Dispose();
|
||||
_eventStreaming.DistributorLogger.LogDebug(
|
||||
exception,
|
||||
"Dashboard event mirror for session {SessionId} was abandoned: the session was disposed "
|
||||
+ "before the mirror could attach.",
|
||||
SessionId);
|
||||
return;
|
||||
}
|
||||
|
||||
IEventSubscriberLease lease = distributor.Register(isInternal: true);
|
||||
StartPumpIfRequested(distributor, startNow);
|
||||
|
||||
@@ -1867,6 +1918,9 @@ public sealed class GatewaySession
|
||||
{
|
||||
distributor = _eventDistributor;
|
||||
_eventDistributor = null;
|
||||
// GWC-34: latch the null-out so EnsureDistributorCreated cannot read it as
|
||||
// "not created yet" and hand a racing attach a distributor that will never pump.
|
||||
_eventDistributorDisposed = true;
|
||||
}
|
||||
|
||||
if (distributor is not null)
|
||||
|
||||
@@ -41,7 +41,7 @@ public sealed class WorkerClient : IWorkerClient
|
||||
private readonly SemaphoreSlim _pendingCommandSlots;
|
||||
private readonly CancellationTokenSource _stopCts = new();
|
||||
|
||||
// GWC-30: this client's contribution to the gateway-wide worker queue-depth gauge. Registered
|
||||
// PERF-02: this client's contribution to the gateway-wide worker queue-depth gauge. Registered
|
||||
// once here and read only when the gauge is scraped, so staging and consuming an event cost an
|
||||
// Interlocked on _eventQueueDepth and nothing else — previously each of those two hot-path steps
|
||||
// called into GatewayMetrics and took its process-wide lock. Null when metrics are disabled.
|
||||
@@ -233,24 +233,31 @@ public sealed class WorkerClient : IWorkerClient
|
||||
// size the caller controls; checking here keeps a MessageTooLarge in the write loop a
|
||||
// genuine desync signal.
|
||||
//
|
||||
// PERF(GWC-31): this size cannot be handed to WorkerFrameWriter to spare its own
|
||||
// PERF-07: this size cannot be handed to WorkerFrameWriter to spare its own
|
||||
// CalculateSize. WriteLoopAsync stamps envelope.Sequence immediately before the write
|
||||
// (GWC-28), and a non-zero varint field grows the encoding — so the number computed here
|
||||
// is a lower bound on the frame the writer actually emits, never the frame length. Passing
|
||||
// it as a knownSize would under-length the prefix and desync the worker's framing. The
|
||||
// pre-check stays a pre-check: it is conservative in the right direction.
|
||||
// it as a knownSize would under-length the prefix and desync the worker's framing.
|
||||
//
|
||||
// IPC-39: being a lower bound is conservative for the knownSize question but the wrong
|
||||
// direction for this pre-check, which must not admit an envelope the writer will then
|
||||
// reject with a session-fatal MessageTooLarge. Compare against the frame maximum less the
|
||||
// MaxSequenceEncodingBytes the stamp can add, so every envelope this check accepts still
|
||||
// fits once the sequence is written.
|
||||
int envelopeSize = commandEnvelope.CalculateSize();
|
||||
if (envelopeSize > _connection.FrameOptions.MaxMessageBytes)
|
||||
int preCheckLimit = _connection.FrameOptions.MaxMessageBytes - MaxSequenceEncodingBytes;
|
||||
if (envelopeSize > preCheckLimit)
|
||||
{
|
||||
throw new WorkerClientException(
|
||||
WorkerClientErrorCode.CommandTooLarge,
|
||||
$"Worker command {method} serializes to {envelopeSize} bytes, exceeding the negotiated "
|
||||
+ $"worker-frame maximum of {_connection.FrameOptions.MaxMessageBytes} bytes.");
|
||||
+ $"worker-frame maximum of {_connection.FrameOptions.MaxMessageBytes} bytes less the "
|
||||
+ $"{MaxSequenceEncodingBytes}-byte sequence-stamp allowance.");
|
||||
}
|
||||
|
||||
await EnqueueAsync(commandEnvelope, cancellationToken).ConfigureAwait(false);
|
||||
|
||||
// GWC-31: one pooled timer instead of a linked CTS + Task.Delay + WhenAny per command.
|
||||
// PERF-07: one pooled timer instead of a linked CTS + Task.Delay + WhenAny per command.
|
||||
// Task.WaitAsync arms a TimerQueueTimer on the shared timer queue and cancels it when the
|
||||
// reply lands, so the steady-state cost of a command that replies in time is a single
|
||||
// continuation — the old shape allocated a linked CancellationTokenSource, its
|
||||
@@ -284,7 +291,10 @@ public sealed class WorkerClient : IWorkerClient
|
||||
// throw, so it can never replace the timeout the caller is owed.
|
||||
if (removed)
|
||||
{
|
||||
TrySendCancelForTimedOutCommand(correlationId, method, timeout);
|
||||
TrySendCancelForAbandonedCommand(
|
||||
correlationId,
|
||||
method,
|
||||
$"gateway command timeout after {timeout}");
|
||||
}
|
||||
|
||||
throw new WorkerClientException(
|
||||
@@ -293,11 +303,7 @@ public sealed class WorkerClient : IWorkerClient
|
||||
}
|
||||
catch (OperationCanceledException)
|
||||
{
|
||||
RemovePendingCommandAsFailed(
|
||||
correlationId,
|
||||
pendingCommand,
|
||||
WorkerClientErrorCode.GatewayShutdown,
|
||||
"Command wait was canceled.");
|
||||
AbandonCommandOnCallerCancellation(correlationId, pendingCommand, method);
|
||||
|
||||
// WaitAsync surfaces TaskCanceledException; throwing through the token keeps the
|
||||
// exception the caller observes exactly what the hand-rolled wait produced.
|
||||
@@ -309,11 +315,7 @@ public sealed class WorkerClient : IWorkerClient
|
||||
// The deadline and the caller's cancellation raced and WaitAsync picked the timer.
|
||||
// The old wait classified this as cancellation, so this clause — reached only when
|
||||
// the filter above saw a canceled token — reproduces that treatment exactly.
|
||||
RemovePendingCommandAsFailed(
|
||||
correlationId,
|
||||
pendingCommand,
|
||||
WorkerClientErrorCode.GatewayShutdown,
|
||||
"Command wait was canceled.");
|
||||
AbandonCommandOnCallerCancellation(correlationId, pendingCommand, method);
|
||||
cancellationToken.ThrowIfCancellationRequested();
|
||||
throw;
|
||||
}
|
||||
@@ -352,7 +354,7 @@ public sealed class WorkerClient : IWorkerClient
|
||||
{
|
||||
await foreach (WorkerEvent workerEvent in _events.Reader.ReadAllAsync(cancellationToken).ConfigureAwait(false))
|
||||
{
|
||||
// No metrics call on the hot path: the gauge pulls _eventQueueDepth when scraped (GWC-30).
|
||||
// No metrics call on the hot path: the gauge pulls _eventQueueDepth when scraped (PERF-02).
|
||||
Interlocked.Decrement(ref _eventQueueDepth);
|
||||
yield return workerEvent;
|
||||
}
|
||||
@@ -418,12 +420,32 @@ public sealed class WorkerClient : IWorkerClient
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
public async ValueTask DisposeAsync()
|
||||
{
|
||||
if (_disposed)
|
||||
// GWC-32: disposal is the third channel-completing path, and it must enter a terminal state
|
||||
// like the other two (MarkClosed, SetFaulted) BEFORE the channels are completed. Otherwise a
|
||||
// client disposed while still Ready keeps reporting IsTerminalState() == false, and a read
|
||||
// loop already inside DispatchEnvelope (the _stopCts.Cancel below only interrupts the NEXT
|
||||
// ReadAsync) takes StageWorkerEvent's overflow arm on the completed staging channel: a
|
||||
// spurious ProtocolViolation fault, a worker-event-staging overflow metric and a redundant
|
||||
// kill, all during an intentional teardown. MarkClosed is deliberately not reused here: it
|
||||
// would overwrite Faulted with Closed and it re-runs the whole teardown sequence this method
|
||||
// performs itself. The transition is skipped when the state is already terminal, so a
|
||||
// dispose-after-fault keeps Faulted and a double dispose changes nothing.
|
||||
bool becameTerminalHere;
|
||||
lock (_syncRoot)
|
||||
{
|
||||
return;
|
||||
if (_disposed)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
_disposed = true;
|
||||
becameTerminalHere = _state is not (WorkerClientState.Closed or WorkerClientState.Faulted);
|
||||
if (becameTerminalHere)
|
||||
{
|
||||
_state = WorkerClientState.Closed;
|
||||
}
|
||||
}
|
||||
|
||||
_disposed = true;
|
||||
// Drop out of the worker queue-depth gauge before teardown: whatever this client still holds
|
||||
// is about to be discarded, and the sum must not keep counting a client that is going away.
|
||||
_eventQueueDepthRegistration?.Dispose();
|
||||
@@ -432,6 +454,16 @@ public sealed class WorkerClient : IWorkerClient
|
||||
_outboundEnvelopes.Writer.TryComplete();
|
||||
_eventStaging.Writer.TryComplete();
|
||||
_events.Writer.TryComplete();
|
||||
if (becameTerminalHere)
|
||||
{
|
||||
// A worker that reached Ready and is now being disposed without a prior Close or fault
|
||||
// (SessionManager.OpenSessionAsync's failure arm does exactly this) stopped, so the
|
||||
// running-worker gauge must come down. RecordWorkerStoppedOnce is latched on
|
||||
// _workerStartRecorded, so a client that already recorded its stop through MarkClosed or
|
||||
// SetFaulted cannot record a second one here.
|
||||
RecordWorkerStoppedOnce("dispose");
|
||||
}
|
||||
|
||||
CompletePendingCommands(
|
||||
new WorkerClientException(
|
||||
WorkerClientErrorCode.GatewayShutdown,
|
||||
@@ -455,6 +487,18 @@ public sealed class WorkerClient : IWorkerClient
|
||||
_stopCts.Dispose();
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Worst-case number of bytes <see cref="WriteLoopAsync"/>'s sequence stamp adds to an
|
||||
/// envelope that was sized before it: one tag byte plus a ten-byte varint, the maximum
|
||||
/// encoding of a 64-bit protobuf field. The <c>CommandTooLarge</c> pre-check subtracts this
|
||||
/// from the negotiated frame maximum (IPC-39) because it measures the envelope while
|
||||
/// <c>Sequence</c> is still 0 — an unset field costs nothing to encode, so the measured size
|
||||
/// is a lower bound on the frame the writer emits, and without the allowance an envelope in
|
||||
/// the last eleven bytes of the range would pass the pre-check and then fault the session on
|
||||
/// the write.
|
||||
/// </summary>
|
||||
private const int MaxSequenceEncodingBytes = 11;
|
||||
|
||||
/// <summary>Manages writing envelopes to the worker pipe.</summary>
|
||||
private async Task WriteLoopAsync()
|
||||
{
|
||||
@@ -633,8 +677,10 @@ public sealed class WorkerClient : IWorkerClient
|
||||
/// Hands a received worker event to the dedicated event writer without blocking the read loop.
|
||||
/// <c>TryWrite</c> is non-blocking, so the read loop keeps dispatching replies, heartbeats and
|
||||
/// faults regardless of how backed up the event path is. It returns <c>false</c> in two cases:
|
||||
/// the staging channel has been completed during shutdown (the event is safely dropped because
|
||||
/// the client is already terminal), or the channel is full at its
|
||||
/// the staging channel has been completed by one of the three completing paths —
|
||||
/// <see cref="MarkClosed"/>, <see cref="SetFaulted"/> or <see cref="DisposeAsync"/>, each of
|
||||
/// which enters a terminal state before completing the channels (GWC-32), so the event is
|
||||
/// safely dropped because the client is already terminal — or the channel is full at its
|
||||
/// 2 × <see cref="WorkerClientOptions.EventChannelCapacity"/> bound. The latter means the
|
||||
/// consumer has been draining slower than the worker produces for the whole time it took the
|
||||
/// worker to emit that many further events, so the client is faulted immediately (GWC-24) —
|
||||
@@ -803,6 +849,45 @@ public sealed class WorkerClient : IWorkerClient
|
||||
return true;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Fails the pending command for a caller whose own token fired — a client deadline shorter
|
||||
/// than the gateway's command timeout, or a disconnect that cancelled the RPC's token — and
|
||||
/// forwards a <c>WorkerCancel</c> for the correlation the caller has abandoned (IPC-34).
|
||||
/// <para>
|
||||
/// Caller cancellation is the common abandonment path, not the rare one, and it leaves the
|
||||
/// worker in exactly the state the timeout arm's cancel exists for: the correlation is still
|
||||
/// on the worker's single STA queue and would execute in its turn, behind live callers, for
|
||||
/// a caller that is already gone. Under a burst of deadline-cancelled calls that backlog is
|
||||
/// precisely what the cancel was added to shorten, so both arms send it.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Gated the same way as the timeout arm: only the caller that owned the removal sends, so a
|
||||
/// reply that won the race is never chased. Additionally skipped once <c>_stopCts</c> has
|
||||
/// fired — the client is closing, the outbound channel is being completed and every pending
|
||||
/// command failed anyway, so a cancel would be at best pointless and at worst a write racing
|
||||
/// teardown.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
/// <param name="correlationId">Correlation id of the abandoned command.</param>
|
||||
/// <param name="pendingCommand">The pending command entry to fail.</param>
|
||||
/// <param name="method">Command method name, for the cancel reason and diagnostics.</param>
|
||||
private void AbandonCommandOnCallerCancellation(
|
||||
string correlationId,
|
||||
PendingCommand pendingCommand,
|
||||
string method)
|
||||
{
|
||||
bool removed = RemovePendingCommandAsFailed(
|
||||
correlationId,
|
||||
pendingCommand,
|
||||
WorkerClientErrorCode.CallerCancelled,
|
||||
"Command wait was canceled.");
|
||||
|
||||
if (removed && !_stopCts.IsCancellationRequested)
|
||||
{
|
||||
TrySendCancelForAbandonedCommand(correlationId, method, "caller cancelled the command");
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Forwards a <c>WorkerCancel</c> for a correlation the gateway has given up waiting for, so the
|
||||
/// worker can drop it from its STA queue (<c>WorkerPipeSession</c> routes the envelope to
|
||||
@@ -819,9 +904,10 @@ public sealed class WorkerClient : IWorkerClient
|
||||
/// <remarks>
|
||||
/// The whole body sits under one catch-all that debug-logs, because the guarantee this method
|
||||
/// owes its caller is structural, not incidental: the caller is on the throw path for the
|
||||
/// timeout, so anything escaping here — an envelope that failed to build, a <c>TryWrite</c>
|
||||
/// against a disposed channel, a scheduler refusing the detached task — would replace the
|
||||
/// <see cref="WorkerClientErrorCode.CommandTimeout"/> the caller is owed with an unrelated
|
||||
/// timeout or the cancellation, so anything escaping here — an envelope that failed to build,
|
||||
/// a <c>TryWrite</c> against a disposed channel, a scheduler refusing the detached task —
|
||||
/// would replace the <see cref="WorkerClientErrorCode.CommandTimeout"/> (or the
|
||||
/// <see cref="OperationCanceledException"/>) the caller is owed with an unrelated
|
||||
/// exception. Losing the cancel costs the worker one wasted command; losing the timeout
|
||||
/// misreports why the call failed. The detached task carries its own handler for the same
|
||||
/// reason: its failures (including the <see cref="ObjectDisposedException"/> from
|
||||
@@ -829,13 +915,13 @@ public sealed class WorkerClient : IWorkerClient
|
||||
/// and would otherwise be unobserved. It is deliberately not tracked or awaited — it holds no
|
||||
/// resource the shutdown path needs back, and the outbound channel is completed on close.
|
||||
/// </remarks>
|
||||
/// <param name="correlationId">Correlation id of the command that timed out.</param>
|
||||
/// <param name="method">Command method name, for the cancel reason and diagnostics.</param>
|
||||
/// <param name="timeout">The elapsed command timeout, for the cancel reason.</param>
|
||||
private void TrySendCancelForTimedOutCommand(
|
||||
/// <param name="correlationId">Correlation id of the abandoned command.</param>
|
||||
/// <param name="method">Command method name, for diagnostics.</param>
|
||||
/// <param name="reason">Why the gateway abandoned the command, carried on the <c>WorkerCancel</c>.</param>
|
||||
private void TrySendCancelForAbandonedCommand(
|
||||
string correlationId,
|
||||
string method,
|
||||
TimeSpan timeout)
|
||||
string reason)
|
||||
{
|
||||
try
|
||||
{
|
||||
@@ -843,7 +929,7 @@ public sealed class WorkerClient : IWorkerClient
|
||||
correlationId,
|
||||
envelope => envelope.WorkerCancel = new WorkerCancel
|
||||
{
|
||||
Reason = $"gateway command timeout after {timeout}",
|
||||
Reason = reason,
|
||||
});
|
||||
|
||||
if (_outboundEnvelopes.Writer.TryWrite(cancelEnvelope))
|
||||
@@ -869,10 +955,10 @@ public sealed class WorkerClient : IWorkerClient
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Records a cancel that could not be forwarded for a timed-out command.</summary>
|
||||
/// <summary>Records a cancel that could not be forwarded for an abandoned command.</summary>
|
||||
/// <param name="exception">The failure that stopped the cancel from being sent.</param>
|
||||
/// <param name="method">Command method name of the timed-out command.</param>
|
||||
/// <param name="correlationId">Correlation id of the timed-out command.</param>
|
||||
/// <param name="method">Command method name of the abandoned command.</param>
|
||||
/// <param name="correlationId">Correlation id of the abandoned command.</param>
|
||||
private void LogCancelNotForwarded(
|
||||
Exception exception,
|
||||
string method,
|
||||
@@ -880,7 +966,7 @@ public sealed class WorkerClient : IWorkerClient
|
||||
{
|
||||
_logger.LogDebug(
|
||||
exception,
|
||||
"Could not forward a cancel for timed-out worker command {Method} on session {SessionId} "
|
||||
"Could not forward a cancel for abandoned worker command {Method} on session {SessionId} "
|
||||
+ "and correlation {CorrelationId}.",
|
||||
method,
|
||||
SessionId,
|
||||
@@ -896,6 +982,21 @@ public sealed class WorkerClient : IWorkerClient
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
WorkerEnvelope envelope = await _reader.ReadAsync(cancellationToken).ConfigureAwait(false);
|
||||
|
||||
// IPC-35: a fault frame during the handshake is the worker reporting a structured startup
|
||||
// failure — most often MXAccess COM creation (not installed, not registered, access denied)
|
||||
// — not a protocol problem. Flattening it to ProtocolViolation discarded the category and
|
||||
// diagnostic exactly where they matter most: first session open on a freshly deployed host,
|
||||
// where the message reaches the operator log and the client's OpenFailed detail. Use the
|
||||
// same formatter the steady-state fault path uses so both windows read alike.
|
||||
if (envelope.BodyCase == WorkerEnvelope.BodyOneofCase.WorkerFault
|
||||
&& expectedBody != WorkerEnvelope.BodyOneofCase.WorkerFault)
|
||||
{
|
||||
throw new WorkerClientException(
|
||||
WorkerClientErrorCode.WorkerFaulted,
|
||||
CreateWorkerFaultMessage(envelope.WorkerFault));
|
||||
}
|
||||
|
||||
if (envelope.BodyCase != expectedBody)
|
||||
{
|
||||
throw new WorkerClientException(
|
||||
|
||||
@@ -10,6 +10,14 @@ public enum WorkerClientErrorCode
|
||||
HeartbeatExpired,
|
||||
ShutdownTimeout,
|
||||
GatewayShutdown,
|
||||
|
||||
// The caller's own cancellation token fired while it waited for the reply — a client deadline
|
||||
// shorter than the gateway's command timeout, or a disconnect that cancelled the RPC's token.
|
||||
// Distinct from GatewayShutdown (which these arms used to report) so metrics and logs stop
|
||||
// recording a shutdown that did not happen; both map to Cancelled at the gRPC boundary, so the
|
||||
// status the client observes is unchanged.
|
||||
CallerCancelled,
|
||||
|
||||
WriteFailed,
|
||||
PendingCommandLimitExceeded,
|
||||
|
||||
|
||||
@@ -384,9 +384,14 @@ public sealed class AlarmFailoverEndToEndTests
|
||||
private sealed class StubWatchListResolver(IReadOnlyList<AlarmSubtagTarget> targets) : IAlarmWatchListResolver
|
||||
{
|
||||
/// <inheritdoc />
|
||||
public Task<IReadOnlyList<AlarmSubtagTarget>> ResolveAsync(
|
||||
public Task<AlarmWatchListResolution> ResolveAsync(
|
||||
AlarmsOptions options,
|
||||
CancellationToken cancellationToken = default) => Task.FromResult(targets);
|
||||
CancellationToken cancellationToken = default) =>
|
||||
Task.FromResult(new AlarmWatchListResolution(
|
||||
targets,
|
||||
DiscoveryAttempted: false,
|
||||
DiscoveryFailed: false,
|
||||
DiscoveryFailureMessage: null));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
|
||||
@@ -507,10 +507,14 @@ public sealed class AlarmTruncationSignalTests
|
||||
private sealed class StubWatchListResolver : IAlarmWatchListResolver
|
||||
{
|
||||
/// <inheritdoc />
|
||||
public Task<IReadOnlyList<AlarmSubtagTarget>> ResolveAsync(
|
||||
public Task<AlarmWatchListResolution> ResolveAsync(
|
||||
AlarmsOptions options,
|
||||
CancellationToken cancellationToken = default) =>
|
||||
Task.FromResult<IReadOnlyList<AlarmSubtagTarget>>([]);
|
||||
Task.FromResult(new AlarmWatchListResolution(
|
||||
[],
|
||||
DiscoveryAttempted: false,
|
||||
DiscoveryFailed: false,
|
||||
DiscoveryFailureMessage: null));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
|
||||
@@ -16,6 +16,21 @@ public sealed class AlarmWatchListResolverTests
|
||||
private static AlarmWatchListResolver CreateResolver(IGalaxyRepository repository) =>
|
||||
new(repository, NullLogger<AlarmWatchListResolver>.Instance);
|
||||
|
||||
/// <summary>
|
||||
/// Resolves and projects to the target list. These cases assert composition rules, which
|
||||
/// live in <see cref="AlarmWatchListResolution.Targets"/>; the discovery-provenance fields
|
||||
/// the resolution also carries (GWC-37) are asserted by their own cases below.
|
||||
/// </summary>
|
||||
/// <param name="resolver">The resolver under test.</param>
|
||||
/// <param name="options">Alarm options to resolve against.</param>
|
||||
/// <param name="cancellationToken">Token to cancel the resolution.</param>
|
||||
/// <returns>The resolved targets.</returns>
|
||||
private static async Task<IReadOnlyList<AlarmSubtagTarget>> ResolveTargetsAsync(
|
||||
AlarmWatchListResolver resolver,
|
||||
AlarmsOptions options,
|
||||
CancellationToken cancellationToken = default) =>
|
||||
(await resolver.ResolveAsync(options, cancellationToken)).Targets;
|
||||
|
||||
private static AlarmsOptions Options(
|
||||
bool useGalaxyRepository = true,
|
||||
string area = "",
|
||||
@@ -58,7 +73,7 @@ public sealed class AlarmWatchListResolverTests
|
||||
|
||||
AlarmWatchListResolver resolver = CreateResolver(repo);
|
||||
|
||||
IReadOnlyList<AlarmSubtagTarget> result = await resolver.ResolveAsync(Options(
|
||||
IReadOnlyList<AlarmSubtagTarget> result = await ResolveTargetsAsync(resolver, Options(
|
||||
include: ["pump01.fault", "Valve03.Position.Lo"],
|
||||
exclude: ["Tank02.Level.HiHi"]));
|
||||
|
||||
@@ -81,7 +96,7 @@ public sealed class AlarmWatchListResolverTests
|
||||
|
||||
AlarmWatchListResolver resolver = CreateResolver(repo);
|
||||
|
||||
IReadOnlyList<AlarmSubtagTarget> result = await resolver.ResolveAsync(Options(
|
||||
IReadOnlyList<AlarmSubtagTarget> result = await ResolveTargetsAsync(resolver, Options(
|
||||
subtags: new AlarmSubtagNameOptions
|
||||
{
|
||||
Active = "InAlarm",
|
||||
@@ -110,7 +125,7 @@ public sealed class AlarmWatchListResolverTests
|
||||
AlarmWatchListResolver resolver = CreateResolver(repo);
|
||||
|
||||
// Default Priority is "Priority"; force it empty alongside empty AckComment.
|
||||
IReadOnlyList<AlarmSubtagTarget> result = await resolver.ResolveAsync(Options(
|
||||
IReadOnlyList<AlarmSubtagTarget> result = await ResolveTargetsAsync(resolver, Options(
|
||||
subtags: new AlarmSubtagNameOptions { Priority = string.Empty, AckComment = string.Empty }));
|
||||
|
||||
AlarmSubtagTarget target = Assert.Single(result);
|
||||
@@ -140,7 +155,7 @@ public sealed class AlarmWatchListResolverTests
|
||||
AlarmWatchListResolver resolver = CreateResolver(repo);
|
||||
|
||||
// Config area "DEV" must be ignored for a GR row that has a discovered area.
|
||||
IReadOnlyList<AlarmSubtagTarget> result = await resolver.ResolveAsync(
|
||||
IReadOnlyList<AlarmSubtagTarget> result = await ResolveTargetsAsync(resolver,
|
||||
Options(area: "DEV", defaultArea: "DEV"));
|
||||
|
||||
AlarmSubtagTarget target = Assert.Single(result);
|
||||
@@ -161,7 +176,7 @@ public sealed class AlarmWatchListResolverTests
|
||||
AlarmWatchListResolver resolver = CreateResolver(repo);
|
||||
|
||||
// No discovery area and no default area.
|
||||
IReadOnlyList<AlarmSubtagTarget> result = await resolver.ResolveAsync(Options());
|
||||
IReadOnlyList<AlarmSubtagTarget> result = await ResolveTargetsAsync(resolver, Options());
|
||||
|
||||
AlarmSubtagTarget target = Assert.Single(result);
|
||||
Assert.Equal("Galaxy!Tank01.Level.HiHi", target.AlarmFullReference);
|
||||
@@ -178,7 +193,7 @@ public sealed class AlarmWatchListResolverTests
|
||||
|
||||
AlarmWatchListResolver resolver = CreateResolver(repo);
|
||||
|
||||
IReadOnlyList<AlarmSubtagTarget> result = await resolver.ResolveAsync(Options(
|
||||
IReadOnlyList<AlarmSubtagTarget> result = await ResolveTargetsAsync(resolver, Options(
|
||||
area: "Site_A",
|
||||
include: ["Tank01.Level.HiHi"]));
|
||||
|
||||
@@ -196,7 +211,7 @@ public sealed class AlarmWatchListResolverTests
|
||||
|
||||
AlarmWatchListResolver resolver = CreateResolver(repo);
|
||||
|
||||
IReadOnlyList<AlarmSubtagTarget> result = await resolver.ResolveAsync(Options(
|
||||
IReadOnlyList<AlarmSubtagTarget> result = await ResolveTargetsAsync(resolver, Options(
|
||||
area: "",
|
||||
defaultArea: "Plant",
|
||||
include: ["Tank01.Level.HiHi"]));
|
||||
@@ -217,7 +232,7 @@ public sealed class AlarmWatchListResolverTests
|
||||
|
||||
AlarmWatchListResolver resolver = CreateResolver(repo);
|
||||
|
||||
IReadOnlyList<AlarmSubtagTarget> result = await resolver.ResolveAsync(Options(
|
||||
IReadOnlyList<AlarmSubtagTarget> result = await ResolveTargetsAsync(resolver, Options(
|
||||
useGalaxyRepository: false,
|
||||
include: ["Tank01.Level.HiHi"]));
|
||||
|
||||
@@ -235,7 +250,7 @@ public sealed class AlarmWatchListResolverTests
|
||||
|
||||
AlarmWatchListResolver resolver = CreateResolver(repo);
|
||||
|
||||
IReadOnlyList<AlarmSubtagTarget> result = await resolver.ResolveAsync(Options(
|
||||
IReadOnlyList<AlarmSubtagTarget> result = await ResolveTargetsAsync(resolver, Options(
|
||||
include: ["Tank01.Level.HiHi"]));
|
||||
|
||||
// Did not throw; discovery set empty, include retained.
|
||||
@@ -243,6 +258,51 @@ public sealed class AlarmWatchListResolverTests
|
||||
Assert.Equal("Galaxy!Tank01.Level.HiHi", target.AlarmFullReference);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// GWC-37: swallowing the discovery failure is right for the resolver but leaves the
|
||||
/// caller unable to tell "discovery produced nothing" from "nothing was configured", which
|
||||
/// is why the empty watch-list could sit unnoticed for a monitor's whole life. The
|
||||
/// resolution therefore reports whether discovery was attempted, whether it failed, and
|
||||
/// the failure message the provider status quotes.
|
||||
/// </summary>
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
[Fact]
|
||||
public async Task ResolveAsync_RepositoryThrows_ReportsDiscoveryFailureOnTheResolution()
|
||||
{
|
||||
ThrowingGalaxyRepository repo = new(new InvalidOperationException("SQL unavailable"));
|
||||
|
||||
AlarmWatchListResolver resolver = CreateResolver(repo);
|
||||
|
||||
AlarmWatchListResolution resolution = await resolver.ResolveAsync(Options());
|
||||
|
||||
Assert.Empty(resolution.Targets);
|
||||
Assert.True(resolution.DiscoveryAttempted);
|
||||
Assert.True(resolution.DiscoveryFailed);
|
||||
Assert.Equal("SQL unavailable", resolution.DiscoveryFailureMessage);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The complement of the case above: discovery disabled is not discovery failed. An empty
|
||||
/// watch-list here means the deployment did not ask for subtag coverage, which the monitor
|
||||
/// must not report as a degraded standby (GWC-37).
|
||||
/// </summary>
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
[Fact]
|
||||
public async Task ResolveAsync_RepositoryDisabled_ReportsDiscoveryNotAttempted()
|
||||
{
|
||||
StubGalaxyRepository repo = new([]);
|
||||
|
||||
AlarmWatchListResolver resolver = CreateResolver(repo);
|
||||
|
||||
AlarmWatchListResolution resolution = await resolver.ResolveAsync(
|
||||
Options(useGalaxyRepository: false));
|
||||
|
||||
Assert.Empty(resolution.Targets);
|
||||
Assert.False(resolution.DiscoveryAttempted);
|
||||
Assert.False(resolution.DiscoveryFailed);
|
||||
Assert.Null(resolution.DiscoveryFailureMessage);
|
||||
}
|
||||
|
||||
/// <summary>Verifies the source object reference is derived from a config include entry, using the portion before the first dot when present.</summary>
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
[Fact]
|
||||
@@ -252,7 +312,7 @@ public sealed class AlarmWatchListResolverTests
|
||||
|
||||
AlarmWatchListResolver resolver = CreateResolver(repo);
|
||||
|
||||
IReadOnlyList<AlarmSubtagTarget> result = await resolver.ResolveAsync(Options(
|
||||
IReadOnlyList<AlarmSubtagTarget> result = await ResolveTargetsAsync(resolver, Options(
|
||||
include: ["Tank01.Level.HiHi", "StandaloneTag"]));
|
||||
|
||||
Assert.Equal("Tank01", result[0].SourceObjectReference);
|
||||
@@ -273,7 +333,7 @@ public sealed class AlarmWatchListResolverTests
|
||||
|
||||
AlarmWatchListResolver resolver = CreateResolver(repo);
|
||||
|
||||
IReadOnlyList<AlarmSubtagTarget> result = await resolver.ResolveAsync(Options(
|
||||
IReadOnlyList<AlarmSubtagTarget> result = await ResolveTargetsAsync(resolver, Options(
|
||||
useGalaxyRepository: false,
|
||||
include: ["Tank01.Level.HiHi"],
|
||||
exclude: ["Tank01.Level.HiHi"]));
|
||||
@@ -299,7 +359,7 @@ public sealed class AlarmWatchListResolverTests
|
||||
|
||||
AlarmWatchListResolver resolver = CreateResolver(repo);
|
||||
|
||||
IReadOnlyList<AlarmSubtagTarget> result = await resolver.ResolveAsync(Options(
|
||||
IReadOnlyList<AlarmSubtagTarget> result = await ResolveTargetsAsync(resolver, Options(
|
||||
useGalaxyRepository: true,
|
||||
exclude: ["Tank02.Level.HiHi"]));
|
||||
|
||||
@@ -324,7 +384,7 @@ public sealed class AlarmWatchListResolverTests
|
||||
AlarmWatchListResolver resolver = CreateResolver(repo);
|
||||
|
||||
// The exclude array contains a whitespace-only string — should be a no-op.
|
||||
IReadOnlyList<AlarmSubtagTarget> result = await resolver.ResolveAsync(Options(
|
||||
IReadOnlyList<AlarmSubtagTarget> result = await ResolveTargetsAsync(resolver, Options(
|
||||
useGalaxyRepository: true,
|
||||
exclude: [" "]));
|
||||
|
||||
@@ -351,7 +411,7 @@ public sealed class AlarmWatchListResolverTests
|
||||
AlarmWatchListResolver resolver = CreateResolver(repo);
|
||||
|
||||
await Assert.ThrowsAnyAsync<OperationCanceledException>(() =>
|
||||
resolver.ResolveAsync(
|
||||
ResolveTargetsAsync(resolver,
|
||||
Options(include: ["Tank01.Level.HiHi"]),
|
||||
cts.Token));
|
||||
}
|
||||
@@ -375,7 +435,7 @@ public sealed class AlarmWatchListResolverTests
|
||||
|
||||
// Pump01.Fault is an explicit include AND an exclude (case-insensitive). It must
|
||||
// be dropped; the GR row Tank01 survives.
|
||||
IReadOnlyList<AlarmSubtagTarget> result = await resolver.ResolveAsync(Options(
|
||||
IReadOnlyList<AlarmSubtagTarget> result = await ResolveTargetsAsync(resolver, Options(
|
||||
useGalaxyRepository: true,
|
||||
include: ["Pump01.Fault"],
|
||||
exclude: ["pump01.fault"]));
|
||||
|
||||
@@ -498,10 +498,14 @@ public sealed class GatewayAlarmMonitorAttachOrderTests
|
||||
private sealed class StubWatchListResolver : IAlarmWatchListResolver
|
||||
{
|
||||
/// <inheritdoc />
|
||||
public Task<IReadOnlyList<AlarmSubtagTarget>> ResolveAsync(
|
||||
public Task<AlarmWatchListResolution> ResolveAsync(
|
||||
AlarmsOptions options,
|
||||
CancellationToken cancellationToken = default) =>
|
||||
Task.FromResult<IReadOnlyList<AlarmSubtagTarget>>([]);
|
||||
Task.FromResult(new AlarmWatchListResolution(
|
||||
[],
|
||||
DiscoveryAttempted: false,
|
||||
DiscoveryFailed: false,
|
||||
DiscoveryFailureMessage: null));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
|
||||
@@ -623,6 +623,184 @@ public sealed class GatewayAlarmMonitorProviderModeTests
|
||||
await monitor.StopAsync(CancellationToken.None);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// GWC-37: a Galaxy Repository outage at startup used to leave one <c>LogWarning</c> and an
|
||||
/// empty subtag watch-list that stayed empty for the monitor's whole life — invisible until
|
||||
/// a failover found the standby covering nothing. The condition is now a standing verdict
|
||||
/// on the provider status, so it reaches the alarm feed, the dashboard badge and every
|
||||
/// late-joining subscriber. The mode stays <c>ALARMMGR</c>: the live provider is healthy,
|
||||
/// it is the standby behind it that is unsound.
|
||||
/// </summary>
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
[Fact]
|
||||
public async Task SubtagCoverage_WhenDiscoveryFailsAtSubscribe_ProviderStatusIsDegradedWithReason()
|
||||
{
|
||||
using GatewayMetrics metrics = new();
|
||||
FakeSessionManager sessions = new();
|
||||
ScriptedWatchListResolver resolver = new(DiscoveryFailedResolution());
|
||||
using GatewayAlarmMonitor monitor = CreateMonitor(sessions, metrics, resolver);
|
||||
|
||||
using CancellationTokenSource cts = new();
|
||||
await monitor.StartAsync(cts.Token);
|
||||
await sessions.WaitForSubscribeAsync(WaitTimeout);
|
||||
|
||||
// The empty list still went to the worker — the verdict is a report, not a behaviour change.
|
||||
Assert.Empty(sessions.LastSubscribeCommand!.WatchList);
|
||||
|
||||
AlarmProviderStatus status = await ReadFirstProviderStatusAsync(monitor);
|
||||
|
||||
Assert.Equal(AlarmProviderMode.Alarmmgr, status.Mode);
|
||||
Assert.True(status.Degraded);
|
||||
Assert.Contains(AlarmProviderReasons.SubtagWatchListEmpty, status.Reason, StringComparison.Ordinal);
|
||||
Assert.Contains(AlarmProviderReasons.SubtagDiscoveryFailed, status.Reason, StringComparison.Ordinal);
|
||||
Assert.Contains("gr offline", status.Reason, StringComparison.Ordinal);
|
||||
|
||||
await cts.CancelAsync();
|
||||
await monitor.StopAsync(CancellationToken.None);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// GWC-37: the failover edge is the moment the watch-list stops being theoretical, so the
|
||||
/// monitor re-resolves it there and publishes the fresh verdict on the same status frame
|
||||
/// the failover produces. Discovery that has recovered since <c>SubscribeAlarms</c> is
|
||||
/// reported as drift rather than applied — the worker rejects a second <c>SubscribeAlarms</c>
|
||||
/// while subscribed, and the only way to replace the list rebuilds the failover consumer on
|
||||
/// the primary that just failed. The status names both the fresh count and the sent count so
|
||||
/// an operator knows a monitor restart will pick the new list up.
|
||||
/// </summary>
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
[Fact]
|
||||
public async Task SubtagCoverage_WhenDiscoveryRecoversByFailover_ReResolvesAndReportsDrift()
|
||||
{
|
||||
using GatewayMetrics metrics = new();
|
||||
FakeSessionManager sessions = new();
|
||||
ScriptedWatchListResolver resolver = new(
|
||||
DiscoveryFailedResolution(),
|
||||
new AlarmWatchListResolution(
|
||||
[Target("Galaxy!Area.Tank01.Level.HiHi"), Target("Galaxy!Area.Tank02.Level.HiHi")],
|
||||
DiscoveryAttempted: true,
|
||||
DiscoveryFailed: false,
|
||||
DiscoveryFailureMessage: null));
|
||||
using GatewayAlarmMonitor monitor = CreateMonitor(sessions, metrics, resolver);
|
||||
|
||||
using CancellationTokenSource cts = new();
|
||||
await monitor.StartAsync(cts.Token);
|
||||
await sessions.WaitForSubscribeAsync(WaitTimeout);
|
||||
Assert.Equal(1, resolver.Calls);
|
||||
|
||||
List<AlarmFeedMessage> received = [];
|
||||
TaskCompletionSource baselineReceived = new(TaskCreationOptions.RunContinuationsAsynchronously);
|
||||
using CancellationTokenSource streamCts = new();
|
||||
Task reader = Task.Run(async () =>
|
||||
{
|
||||
try
|
||||
{
|
||||
await foreach (AlarmFeedMessage message in monitor.StreamAsync(null, streamCts.Token))
|
||||
{
|
||||
lock (received)
|
||||
{
|
||||
received.Add(message);
|
||||
if (received.Count == 1)
|
||||
{
|
||||
baselineReceived.TrySetResult();
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
catch (OperationCanceledException)
|
||||
{
|
||||
// Expected when the test cancels the stream.
|
||||
}
|
||||
});
|
||||
|
||||
await baselineReceived.Task.WaitAsync(WaitTimeout);
|
||||
|
||||
sessions.EmitEvent(new MxEvent
|
||||
{
|
||||
OnAlarmProviderModeChanged = new OnAlarmProviderModeChangedEvent
|
||||
{
|
||||
Mode = AlarmProviderMode.Subtag,
|
||||
Reason = "alarmmgr failed",
|
||||
At = Timestamp.FromDateTimeOffset(DateTimeOffset.UtcNow),
|
||||
},
|
||||
});
|
||||
|
||||
AlarmFeedMessage degraded = await WaitForAsync(
|
||||
received,
|
||||
m => m.PayloadCase == AlarmFeedMessage.PayloadOneofCase.ProviderStatus
|
||||
&& m.ProviderStatus.Mode == AlarmProviderMode.Subtag,
|
||||
WaitTimeout);
|
||||
|
||||
// The re-resolve happened on the failover edge, before the status was published.
|
||||
Assert.Equal(2, resolver.Calls);
|
||||
|
||||
string reason = degraded.ProviderStatus.Reason;
|
||||
Assert.Contains("alarmmgr failed", reason, StringComparison.Ordinal);
|
||||
Assert.Contains(AlarmProviderReasons.SubtagWatchListStale, reason, StringComparison.Ordinal);
|
||||
Assert.Contains("2 target(s) now resolve against 0 sent", reason, StringComparison.Ordinal);
|
||||
|
||||
// The stale-list warning replaced the empty/failed pair: discovery works again, so the only
|
||||
// remaining problem is that the worker is still using the list it was handed.
|
||||
Assert.DoesNotContain(AlarmProviderReasons.SubtagWatchListEmpty, reason, StringComparison.Ordinal);
|
||||
Assert.DoesNotContain(AlarmProviderReasons.SubtagDiscoveryFailed, reason, StringComparison.Ordinal);
|
||||
|
||||
await streamCts.CancelAsync();
|
||||
await reader;
|
||||
await cts.CancelAsync();
|
||||
await monitor.StopAsync(CancellationToken.None);
|
||||
}
|
||||
|
||||
/// <summary>Reads the provider-status frame every stream opens with, then closes the stream.</summary>
|
||||
/// <param name="monitor">The monitor to read from.</param>
|
||||
/// <returns>The provider status carried by the first frame.</returns>
|
||||
private static async Task<AlarmProviderStatus> ReadFirstProviderStatusAsync(GatewayAlarmMonitor monitor)
|
||||
{
|
||||
using CancellationTokenSource streamCts = new(WaitTimeout);
|
||||
await foreach (AlarmFeedMessage message in monitor.StreamAsync(null, streamCts.Token))
|
||||
{
|
||||
if (message.PayloadCase == AlarmFeedMessage.PayloadOneofCase.ProviderStatus)
|
||||
{
|
||||
return message.ProviderStatus;
|
||||
}
|
||||
}
|
||||
|
||||
throw new TimeoutException("No provider-status message was received in time.");
|
||||
}
|
||||
|
||||
/// <summary>A resolution standing for "Galaxy Repository discovery was on and it failed".</summary>
|
||||
/// <returns>The resolution.</returns>
|
||||
private static AlarmWatchListResolution DiscoveryFailedResolution() => new(
|
||||
[],
|
||||
DiscoveryAttempted: true,
|
||||
DiscoveryFailed: true,
|
||||
DiscoveryFailureMessage: "gr offline");
|
||||
|
||||
private static AlarmSubtagTarget Target(string fullReference) => new()
|
||||
{
|
||||
AlarmFullReference = fullReference,
|
||||
SourceObjectReference = fullReference,
|
||||
ActiveSubtag = $"{fullReference}.InAlarm",
|
||||
AckedSubtag = $"{fullReference}.Acked",
|
||||
};
|
||||
|
||||
private static GatewayAlarmMonitor CreateMonitor(
|
||||
FakeSessionManager sessions,
|
||||
GatewayMetrics metrics,
|
||||
IAlarmWatchListResolver resolver)
|
||||
{
|
||||
AlarmsOptions options = new()
|
||||
{
|
||||
Enabled = true,
|
||||
SubscriptionExpression = @"\\NODE\Galaxy!Area",
|
||||
};
|
||||
return new GatewayAlarmMonitor(
|
||||
sessions,
|
||||
resolver,
|
||||
metrics,
|
||||
Microsoft.Extensions.Options.Options.Create(new GatewayOptions { Alarms = options }),
|
||||
NullLogger<GatewayAlarmMonitor>.Instance);
|
||||
}
|
||||
|
||||
private static GatewayAlarmMonitor CreateMonitor(FakeSessionManager sessions, GatewayMetrics metrics)
|
||||
{
|
||||
AlarmsOptions options = new()
|
||||
@@ -693,13 +871,42 @@ public sealed class GatewayAlarmMonitorProviderModeTests
|
||||
throw new TimeoutException("Condition was not met in time.");
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// <see cref="IAlarmWatchListResolver"/> that hands out a scripted sequence of
|
||||
/// resolutions — one per call, the last repeating — and counts the calls, so a test can
|
||||
/// drive "discovery was down at subscribe and back up by the failover" and assert that the
|
||||
/// second resolution happened at all (GWC-37).
|
||||
/// </summary>
|
||||
/// <param name="script">Resolutions to return, in call order.</param>
|
||||
private sealed class ScriptedWatchListResolver(params AlarmWatchListResolution[] script) : IAlarmWatchListResolver
|
||||
{
|
||||
private int _calls;
|
||||
|
||||
/// <summary>Gets the number of times <see cref="ResolveAsync"/> has been called.</summary>
|
||||
public int Calls => Volatile.Read(ref _calls);
|
||||
|
||||
/// <inheritdoc />
|
||||
public Task<AlarmWatchListResolution> ResolveAsync(
|
||||
AlarmsOptions options,
|
||||
CancellationToken cancellationToken = default)
|
||||
{
|
||||
int index = Interlocked.Increment(ref _calls) - 1;
|
||||
return Task.FromResult(script[Math.Min(index, script.Length - 1)]);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary><see cref="IAlarmWatchListResolver"/> that returns a fixed watch-list.</summary>
|
||||
private sealed class StubWatchListResolver(IReadOnlyList<AlarmSubtagTarget> targets) : IAlarmWatchListResolver
|
||||
{
|
||||
/// <inheritdoc />
|
||||
public Task<IReadOnlyList<AlarmSubtagTarget>> ResolveAsync(
|
||||
public Task<AlarmWatchListResolution> ResolveAsync(
|
||||
AlarmsOptions options,
|
||||
CancellationToken cancellationToken = default) => Task.FromResult(targets);
|
||||
CancellationToken cancellationToken = default) =>
|
||||
Task.FromResult(new AlarmWatchListResolution(
|
||||
targets,
|
||||
DiscoveryAttempted: false,
|
||||
DiscoveryFailed: false,
|
||||
DiscoveryFailureMessage: null));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
using System.Security.Claims;
|
||||
using Microsoft.AspNetCore.Authorization;
|
||||
using Microsoft.AspNetCore.Components;
|
||||
using Microsoft.AspNetCore.Components.Authorization;
|
||||
using Microsoft.AspNetCore.Components.Web;
|
||||
@@ -13,7 +14,9 @@ using ZB.MOM.WW.Secrets.Ui;
|
||||
namespace ZB.MOM.WW.MxGateway.Tests.Dashboard;
|
||||
|
||||
/// <summary>
|
||||
/// Renders <see cref="MainLayout"/> and asserts whether the side rail emits the Secrets link.
|
||||
/// Renders <see cref="MainLayout"/> and asserts which policy-gated links the side rail emits —
|
||||
/// Secrets (<c>secrets:manage</c>) and, since SEC-40, Alarms
|
||||
/// (<see cref="DashboardAuthenticationDefaults.AlarmsPolicy"/>).
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
@@ -39,6 +42,7 @@ namespace ZB.MOM.WW.MxGateway.Tests.Dashboard;
|
||||
public sealed class SecretsNavRenderTests
|
||||
{
|
||||
private const string SecretsHref = "/admin/secrets";
|
||||
private const string AlarmsHref = "/alarms";
|
||||
|
||||
/// <summary>
|
||||
/// The gate's proof. A Viewer holds a real, authenticated identity and still must not be offered
|
||||
@@ -95,11 +99,48 @@ public sealed class SecretsNavRenderTests
|
||||
Assert.Contains("/apikeys", html, StringComparison.Ordinal);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// SEC-40's nav half: an anonymous caller — the read-only localhost path — is not offered the
|
||||
/// Alarms link, because the page and hub now refuse them. Same reasoning as the Secrets
|
||||
/// negative above: this is the assertion that distinguishes a gate from an inert one.
|
||||
/// </summary>
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
[Fact]
|
||||
public async Task Rail_OmitsAlarmsLink_ForAnonymous()
|
||||
{
|
||||
string html = await RenderRailAsync();
|
||||
|
||||
Assert.DoesNotContain(AlarmsHref, html, StringComparison.Ordinal);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The control for that negative: an authenticated Viewer still gets the Alarms link. SEC-40
|
||||
/// excludes the environment bypasses, not Viewers — alarm metadata stays galaxy-wide for a
|
||||
/// real dashboard role because the central monitor carries no per-session provenance.
|
||||
/// </summary>
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
[Fact]
|
||||
public async Task Rail_EmitsAlarmsLink_ForViewer()
|
||||
{
|
||||
string html = await RenderRailAsync(DashboardRoles.Viewer);
|
||||
|
||||
Assert.Contains(AlarmsHref, html, StringComparison.Ordinal);
|
||||
}
|
||||
|
||||
private static async Task<string> RenderRailAsync(params string[] roles)
|
||||
{
|
||||
var services = new ServiceCollection();
|
||||
services.AddLogging();
|
||||
services.AddAuthorization(options => options.AddSecretsAuthorization());
|
||||
services.AddAuthorization(options =>
|
||||
{
|
||||
options.AddSecretsAuthorization();
|
||||
// The rail's Alarms AuthorizeView resolves this by name; an unregistered policy throws
|
||||
// rather than denying, so it must be present for any rail render (SEC-40).
|
||||
options.AddPolicy(
|
||||
DashboardAuthenticationDefaults.AlarmsPolicy,
|
||||
policy => policy.AddRequirements(AuthenticatedDashboardRoleRequirement.AnyDashboardRole));
|
||||
});
|
||||
services.AddSingleton<IAuthorizationHandler, AuthenticatedDashboardRoleHandler>();
|
||||
services.AddCascadingAuthenticationState();
|
||||
services.AddSingleton<AuthenticationStateProvider>(new StubAuthenticationStateProvider(roles));
|
||||
services.AddSingleton<NavigationManager, StubNavigationManager>();
|
||||
|
||||
@@ -0,0 +1,198 @@
|
||||
using System.Net;
|
||||
using System.Reflection;
|
||||
using System.Security.Claims;
|
||||
using Microsoft.AspNetCore.Authorization;
|
||||
using Microsoft.AspNetCore.Builder;
|
||||
using Microsoft.AspNetCore.Http;
|
||||
using Microsoft.AspNetCore.Routing;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.Options;
|
||||
using ZB.MOM.WW.MxGateway.Server;
|
||||
using ZB.MOM.WW.MxGateway.Server.Configuration;
|
||||
using ZB.MOM.WW.MxGateway.Server.Dashboard;
|
||||
using ZB.MOM.WW.MxGateway.Server.Dashboard.Hubs;
|
||||
using AuthenticationOptions = ZB.MOM.WW.MxGateway.Server.Configuration.AuthenticationOptions;
|
||||
|
||||
namespace ZB.MOM.WW.MxGateway.Tests.Gateway.Dashboard;
|
||||
|
||||
/// <summary>
|
||||
/// Covers SEC-40: the alarm feed requires a real authenticated dashboard role, so the read-only
|
||||
/// environment bypasses (<c>Authentication:Mode=Disabled</c>, anonymous localhost) no longer
|
||||
/// receive galaxy-wide alarm metadata.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// The enforcement is a requirement TYPE distinction — <see cref="DashboardAuthorizationHandler"/>
|
||||
/// is an <c>AuthorizationHandler<DashboardAuthorizationRequirement></c> and therefore never
|
||||
/// runs for <see cref="AuthenticatedDashboardRoleRequirement"/> — so the tests below assert both
|
||||
/// halves: that both handlers together still deny a bypassed caller, and that the policy the two
|
||||
/// alarm surfaces actually carry is built from that requirement rather than the bypassable one.
|
||||
/// Asserting only the handler would pass even if the hub were left on the old policy.
|
||||
/// </remarks>
|
||||
public sealed class AlarmFeedAuthorizationTests
|
||||
{
|
||||
/// <summary>
|
||||
/// The bypass that satisfies <see cref="DashboardAuthorizationRequirement.AnyDashboardRole"/>
|
||||
/// — auth disabled AND a loopback request with <c>AllowAnonymousLocalhost</c> on, i.e. both
|
||||
/// arms at once — does not satisfy the alarm requirement, with the real bypass handler present
|
||||
/// in the same evaluation.
|
||||
/// </summary>
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
[Fact]
|
||||
public async Task AlarmRequirement_AnonymousLocalhostAndAuthDisabled_DoesNotSucceed()
|
||||
{
|
||||
AuthorizationHandlerContext context = await EvaluateAsync(
|
||||
new ClaimsPrincipal(new ClaimsIdentity()),
|
||||
AuthenticatedDashboardRoleRequirement.AnyDashboardRole);
|
||||
|
||||
Assert.False(context.HasSucceeded);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Control: the SAME evaluation environment satisfies the bypassable requirement. Without this
|
||||
/// pairing the test above could pass because the bypass was misconfigured rather than because
|
||||
/// the requirement type excludes it.
|
||||
/// </summary>
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
[Fact]
|
||||
public async Task ViewerRequirement_AnonymousLocalhostAndAuthDisabled_StillSucceeds()
|
||||
{
|
||||
AuthorizationHandlerContext context = await EvaluateAsync(
|
||||
new ClaimsPrincipal(new ClaimsIdentity()),
|
||||
DashboardAuthorizationRequirement.AnyDashboardRole);
|
||||
|
||||
Assert.True(context.HasSucceeded);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// An authenticated Viewer — the ordinary LDAP-logged-in dashboard user — does satisfy the
|
||||
/// alarm requirement: SEC-40 excludes the environment bypasses, not Viewers. Alarm metadata
|
||||
/// stays galaxy-wide for them because the monitor is gateway-central and carries no
|
||||
/// per-session provenance to tag-scope by.
|
||||
/// </summary>
|
||||
/// <param name="role">The dashboard role the authenticated caller holds.</param>
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
[Theory]
|
||||
[InlineData(DashboardRoles.Viewer)]
|
||||
[InlineData(DashboardRoles.Admin)]
|
||||
public async Task AlarmRequirement_AuthenticatedDashboardRole_Succeeds(string role)
|
||||
{
|
||||
AuthorizationHandlerContext context = await EvaluateAsync(
|
||||
CreatePrincipal(role),
|
||||
AuthenticatedDashboardRoleRequirement.AnyDashboardRole);
|
||||
|
||||
Assert.True(context.HasSucceeded);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// A principal carrying a dashboard role claim but no authentication type does not satisfy the
|
||||
/// requirement: "real role claim" means an authenticated identity, matching how the session
|
||||
/// ACL and <c>DashboardSessionAdminService</c> read a role.
|
||||
/// </summary>
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
[Fact]
|
||||
public async Task AlarmRequirement_UnauthenticatedRoleClaim_DoesNotSucceed()
|
||||
{
|
||||
ClaimsPrincipal principal = new(new ClaimsIdentity(
|
||||
[new Claim(ClaimTypes.Role, DashboardRoles.Admin)],
|
||||
authenticationType: null,
|
||||
nameType: ClaimTypes.Name,
|
||||
roleType: ClaimTypes.Role));
|
||||
|
||||
AuthorizationHandlerContext context = await EvaluateAsync(
|
||||
principal,
|
||||
AuthenticatedDashboardRoleRequirement.AnyDashboardRole);
|
||||
|
||||
Assert.False(context.HasSucceeded);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The registered policy is built from the non-bypassable requirement and from neither
|
||||
/// <see cref="DashboardAuthorizationRequirement"/> nor a role list the bypass handler could
|
||||
/// reach, and it still accepts both hub-capable schemes so an external hub client can
|
||||
/// authenticate with a bearer token.
|
||||
/// </summary>
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
[Fact]
|
||||
public async Task AlarmsPolicy_IsRegisteredWithNonBypassableRequirement()
|
||||
{
|
||||
await using WebApplication app = GatewayApplication.Build([]);
|
||||
IAuthorizationPolicyProvider provider = app.Services
|
||||
.GetRequiredService<IAuthorizationPolicyProvider>();
|
||||
|
||||
AuthorizationPolicy? policy = await provider
|
||||
.GetPolicyAsync(DashboardAuthenticationDefaults.AlarmsPolicy);
|
||||
|
||||
Assert.NotNull(policy);
|
||||
Assert.Contains(policy.Requirements, r => r is AuthenticatedDashboardRoleRequirement);
|
||||
Assert.DoesNotContain(policy.Requirements, r => r is DashboardAuthorizationRequirement);
|
||||
Assert.Equal(
|
||||
[
|
||||
DashboardAuthenticationDefaults.AuthenticationScheme,
|
||||
DashboardAuthenticationDefaults.HubAuthenticationScheme,
|
||||
],
|
||||
policy.AuthenticationSchemes);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Both readers of the one central monitor carry the alarm policy: the hub (external clients)
|
||||
/// and the <c>/alarms</c> Razor endpoint (the in-page reader). Gating only the hub would leave
|
||||
/// the same metadata one navigation away for a bypassed caller.
|
||||
/// </summary>
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
[Fact]
|
||||
public async Task AlarmsHubAndAlarmsPage_BothCarryTheAlarmsPolicy()
|
||||
{
|
||||
AuthorizeAttribute? hubAttribute = typeof(AlarmsHub)
|
||||
.GetCustomAttribute<AuthorizeAttribute>(inherit: false);
|
||||
|
||||
Assert.Equal(DashboardAuthenticationDefaults.AlarmsPolicy, hubAttribute?.Policy);
|
||||
|
||||
await using WebApplication app = GatewayApplication.Build([]);
|
||||
RouteEndpoint alarmsPage = ((IEndpointRouteBuilder)app).DataSources
|
||||
.SelectMany(source => source.Endpoints)
|
||||
.OfType<RouteEndpoint>()
|
||||
.Single(endpoint => endpoint.RoutePattern.RawText == "/alarms");
|
||||
|
||||
Assert.Contains(
|
||||
alarmsPage.Metadata.OfType<IAuthorizeData>(),
|
||||
data => data.Policy == DashboardAuthenticationDefaults.AlarmsPolicy);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Runs BOTH dashboard authorization handlers over one requirement in the most permissive
|
||||
/// environment the gateway supports (auth disabled, loopback, anonymous localhost allowed),
|
||||
/// mirroring how the authorization service invokes every registered handler.
|
||||
/// </summary>
|
||||
/// <param name="principal">The caller.</param>
|
||||
/// <param name="requirement">The single requirement under evaluation.</param>
|
||||
/// <returns>The context after both handlers have run.</returns>
|
||||
private static async Task<AuthorizationHandlerContext> EvaluateAsync(
|
||||
ClaimsPrincipal principal,
|
||||
IAuthorizationRequirement requirement)
|
||||
{
|
||||
DefaultHttpContext httpContext = new();
|
||||
httpContext.Connection.RemoteIpAddress = IPAddress.Loopback;
|
||||
|
||||
IOptions<GatewayOptions> options = Options.Create(new GatewayOptions
|
||||
{
|
||||
Authentication = new AuthenticationOptions { Mode = AuthenticationMode.Disabled },
|
||||
Dashboard = new DashboardOptions { AllowAnonymousLocalhost = true },
|
||||
});
|
||||
|
||||
AuthorizationHandlerContext context = new([requirement], principal, httpContext);
|
||||
|
||||
await new DashboardAuthorizationHandler(
|
||||
new HttpContextAccessor { HttpContext = httpContext },
|
||||
options).HandleAsync(context);
|
||||
await new AuthenticatedDashboardRoleHandler().HandleAsync(context);
|
||||
|
||||
return context;
|
||||
}
|
||||
|
||||
private static ClaimsPrincipal CreatePrincipal(string role) =>
|
||||
new(new ClaimsIdentity(
|
||||
[new Claim(ClaimTypes.Role, role)],
|
||||
DashboardAuthenticationDefaults.AuthenticationScheme,
|
||||
ClaimTypes.Name,
|
||||
ClaimTypes.Role));
|
||||
}
|
||||
+6
-3
@@ -179,8 +179,10 @@ public sealed class DashboardBrowseAndAlarmModelTests
|
||||
|
||||
/// <summary>
|
||||
/// An explicitly-degraded status whose mode is still Alarmmgr (the
|
||||
/// <c>Degraded || Mode==Subtag</c> guard's second, independent branch) must still
|
||||
/// map to the degraded amber badge.
|
||||
/// <c>Degraded || Mode==Subtag</c> guard's second, independent branch) must map to the
|
||||
/// amber badge, but not to the failover label: GWC-37 gives that combination a meaning —
|
||||
/// the alarm manager is serving and the subtag standby behind it covers nothing — so
|
||||
/// reporting "Subtag monitoring (degraded)" would name the wrong live provider.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void FromProviderStatus_Alarmmgr_DegradedFlagSet_WarningBadge()
|
||||
@@ -195,8 +197,9 @@ public sealed class DashboardBrowseAndAlarmModelTests
|
||||
DashboardAlarmProviderStatus model = DashboardAlarmProviderStatus.FromProviderStatus(status);
|
||||
|
||||
Assert.True(model.IsDegraded);
|
||||
Assert.Equal(DashboardAlarmProviderStatus.DegradedLabel, model.Label);
|
||||
Assert.Equal(DashboardAlarmProviderStatus.NoSubtagStandbyLabel, model.Label);
|
||||
Assert.Contains("bg-warning", model.BadgeCssClass, StringComparison.Ordinal);
|
||||
Assert.Equal(AlarmProviderMode.Alarmmgr, model.Mode);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
|
||||
@@ -178,12 +178,112 @@ public sealed class DashboardSessionAclTests
|
||||
Assert.False(CreateAcl().CanViewSession(principal, TaggedSessionId));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// SEC-41, the load-bearing regression: a cookie-style principal whose tag claim was stamped
|
||||
/// at login under an older <c>GroupToTag</c> map is DENIED once that map no longer grants the
|
||||
/// tag to any group the principal belongs to. The auth cookie survives the restart that
|
||||
/// rebinds options, so trusting the stamped claim let a config-revoked grant live on for the
|
||||
/// sliding 8-hour cookie life; the ACL now re-resolves from the group claims the same
|
||||
/// principal carries, against the currently bound map.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void CanViewSession_StampedTagClaimRevokedFromCurrentGroupToTag_Denied()
|
||||
{
|
||||
// Current config grants team-a to nobody the caller belongs to: "plant-ops" was the group
|
||||
// whose entry was removed, and the surviving entry is for an unrelated group.
|
||||
DashboardSessionAcl acl = CreateAcl(groupToTag: new() { ["other-team"] = ["team-b"] });
|
||||
|
||||
ClaimsPrincipal stale = Principal(
|
||||
roles: [DashboardRoles.Viewer],
|
||||
tags: ["team-a"],
|
||||
groups: ["plant-ops"]);
|
||||
|
||||
Assert.False(acl.CanViewSession(stale, TaggedSessionId));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The re-resolution's allowing direction: the same principal is allowed when the current map
|
||||
/// still grants the session's tag to one of its groups — and is allowed even when it carries
|
||||
/// NO stamped tag claim at all, which is what proves the decision comes from the group claims
|
||||
/// rather than from the stamped ones.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void CanViewSession_GroupClaimGrantsTagUnderCurrentMap_Allowed()
|
||||
{
|
||||
DashboardSessionAcl acl = CreateAcl(groupToTag: new() { ["plant-ops"] = ["team-a"] });
|
||||
|
||||
Assert.True(acl.CanViewSession(
|
||||
Principal(roles: [DashboardRoles.Viewer], tags: ["team-a"], groups: ["plant-ops"]),
|
||||
TaggedSessionId));
|
||||
|
||||
Assert.True(acl.CanViewSession(
|
||||
Principal(roles: [DashboardRoles.Viewer], tags: null, groups: ["plant-ops"]),
|
||||
TaggedSessionId));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Re-resolution is ordinal-ignore-case on both sides — the group key against the map (which
|
||||
/// <see cref="DashboardGroupTagMapping"/> owns) and the resolved tag against the session's tag
|
||||
/// set. Pinned because the pre-SEC-41 claim scan got the second comparison from
|
||||
/// <c>Session.Tags</c>'s own comparer, and the re-resolution arm probes the grant set in the
|
||||
/// opposite direction.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void CanViewSession_GroupAndTagCasingDiffer_Allowed()
|
||||
{
|
||||
DashboardSessionAcl acl = CreateAcl(groupToTag: new() { ["PLANT-OPS"] = ["TEAM-A"] });
|
||||
|
||||
Assert.True(acl.CanViewSession(
|
||||
Principal(roles: [DashboardRoles.Viewer], tags: null, groups: ["plant-ops"]),
|
||||
TaggedSessionId));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// A principal carrying group claims that map to nothing is an empty-grant Viewer, even though
|
||||
/// it holds a stamped tag claim: once there is a group list to re-resolve from, the stamped
|
||||
/// claims are not consulted at all. This is the arm that closes the revocation gap for a user
|
||||
/// whose LDAP groups changed shape rather than whose config entry was removed.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void CanViewSession_GroupClaimsPresentButUnmapped_IgnoresStampedTagClaim()
|
||||
{
|
||||
DashboardSessionAcl acl = CreateAcl(groupToTag: []);
|
||||
|
||||
Assert.False(acl.CanViewSession(
|
||||
Principal(roles: [DashboardRoles.Viewer], tags: ["team-a"], groups: ["plant-ops"]),
|
||||
TaggedSessionId));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The documented fallback: a hub-token principal carries the tags <see cref="HubTokenService"/>
|
||||
/// re-resolved at mint but no LDAP group claims to re-resolve from, so its stamped tag claims
|
||||
/// are honoured. Asserted with an EMPTY current map, which is the state that would deny if the
|
||||
/// fallback were ever dropped — and would silently break every token-authenticated hub join.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void CanViewSession_HubTokenStylePrincipalWithNoGroupClaims_UsesStampedTags()
|
||||
{
|
||||
DashboardSessionAcl acl = CreateAcl(groupToTag: []);
|
||||
|
||||
Assert.True(acl.CanViewSession(
|
||||
Principal(roles: [DashboardRoles.Viewer], tags: ["team-a"], groups: null),
|
||||
TaggedSessionId));
|
||||
}
|
||||
|
||||
private static DashboardSessionAcl CreateAcl(
|
||||
UntaggedSessionVisibility visibility = UntaggedSessionVisibility.AdminOnly)
|
||||
UntaggedSessionVisibility visibility = UntaggedSessionVisibility.AdminOnly,
|
||||
Dictionary<string, string[]>? groupToTag = null)
|
||||
{
|
||||
GatewayOptions options = new()
|
||||
{
|
||||
Dashboard = new DashboardOptions { UntaggedSessionVisibility = visibility },
|
||||
Dashboard = new DashboardOptions
|
||||
{
|
||||
UntaggedSessionVisibility = visibility,
|
||||
// Mirrors the binder's comparer so a test's group key behaves like a config one.
|
||||
GroupToTag = groupToTag is null
|
||||
? new Dictionary<string, string[]>(StringComparer.OrdinalIgnoreCase)
|
||||
: new Dictionary<string, string[]>(groupToTag, StringComparer.OrdinalIgnoreCase),
|
||||
},
|
||||
};
|
||||
|
||||
return new DashboardSessionAcl(
|
||||
@@ -193,13 +293,26 @@ public sealed class DashboardSessionAclTests
|
||||
Options.Create(options));
|
||||
}
|
||||
|
||||
private static ClaimsPrincipal Principal(string[] roles, string[]? tags = null)
|
||||
/// <summary>
|
||||
/// Builds a dashboard principal. <paramref name="groups"/> is the SEC-41 discriminator: a
|
||||
/// principal WITH group claims is re-resolved against the current map (the cookie seam), one
|
||||
/// WITHOUT falls back to its stamped tag claims (the hub-token seam). Existing cases pass
|
||||
/// tags only, so they continue to exercise the fallback arm exactly as before.
|
||||
/// </summary>
|
||||
/// <param name="roles">Role claims to stamp.</param>
|
||||
/// <param name="tags">Login-stamped visibility tag claims, if any.</param>
|
||||
/// <param name="groups">LDAP group claims, if any.</param>
|
||||
/// <returns>An authenticated principal carrying the requested claims.</returns>
|
||||
private static ClaimsPrincipal Principal(string[] roles, string[]? tags = null, string[]? groups = null)
|
||||
{
|
||||
List<Claim> claims = [new Claim(ClaimTypes.Name, "viewer-user")];
|
||||
claims.AddRange(roles.Select(role => new Claim(ClaimTypes.Role, role)));
|
||||
claims.AddRange((tags ?? []).Select(tag => new Claim(
|
||||
DashboardAuthenticationDefaults.DashboardTagClaimType,
|
||||
tag)));
|
||||
claims.AddRange((groups ?? []).Select(group => new Claim(
|
||||
DashboardAuthenticationDefaults.LdapGroupClaimType,
|
||||
group)));
|
||||
|
||||
return new ClaimsPrincipal(new ClaimsIdentity(
|
||||
claims,
|
||||
|
||||
@@ -1,5 +1,8 @@
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using Microsoft.Extensions.Options;
|
||||
using ZB.MOM.WW.GalaxyRepository;
|
||||
using ZB.MOM.WW.GalaxyRepository.Grpc;
|
||||
using ZB.MOM.WW.MxGateway.Server.Configuration;
|
||||
using ZB.MOM.WW.MxGateway.Server.Security.Authentication;
|
||||
using ZB.MOM.WW.MxGateway.Server.Security.Authorization;
|
||||
using ZB.MOM.WW.MxGateway.Tests.TestSupport;
|
||||
@@ -23,8 +26,11 @@ public sealed class GalaxyRepositoryHostWiringTests
|
||||
{
|
||||
/// <summary>
|
||||
/// Verifies the full production authz-filtering chain end-to-end.
|
||||
/// Phase 1: no identity pushed → <c>BrowseChildren</c> returns both children
|
||||
/// (unconstrained). Phase 2: identity with <c>BrowseSubtrees = ["NonExistent"]</c>
|
||||
/// Phase 1: an identity carrying no <c>BrowseSubtrees</c> pushed → <c>BrowseChildren</c>
|
||||
/// returns both children (unconstrained). That, not an absent identity, is production's
|
||||
/// unscoped case since GWC-38 — with <c>Authentication:Mode=ApiKey</c> an absent identity
|
||||
/// now denies, which the isolated provider test pins.
|
||||
/// Phase 2: identity with <c>BrowseSubtrees = ["NonExistent"]</c>
|
||||
/// pushed via <see cref="GatewayRequestIdentityAccessor"/> → the real
|
||||
/// <see cref="GatewayBrowseScopeProvider"/> threads the constraint into the lib
|
||||
/// service → <c>BrowseChildren</c> returns empty children.
|
||||
@@ -35,7 +41,16 @@ public sealed class GalaxyRepositoryHostWiringTests
|
||||
{
|
||||
// Arrange: construct the REAL host seam — not stubs.
|
||||
GatewayRequestIdentityAccessor identityAccessor = new();
|
||||
GatewayBrowseScopeProvider scopeProvider = new(identityAccessor);
|
||||
GatewayBrowseScopeProvider scopeProvider = new(
|
||||
identityAccessor,
|
||||
Options.Create(new GatewayOptions
|
||||
{
|
||||
Authentication = new Server.Configuration.AuthenticationOptions
|
||||
{
|
||||
Mode = AuthenticationMode.ApiKey,
|
||||
},
|
||||
}),
|
||||
NullLogger<GatewayBrowseScopeProvider>.Instance);
|
||||
|
||||
GalaxyObject[] objects = CreateFilterObjects();
|
||||
GalaxyHierarchyCacheEntry entry = GalaxyHierarchyCacheEntry.Empty with
|
||||
@@ -55,13 +70,23 @@ public sealed class GalaxyRepositoryHostWiringTests
|
||||
new GalaxyDeployNotifier(),
|
||||
scopeProvider);
|
||||
|
||||
// Phase 1: no identity on the accessor → browse is unconstrained.
|
||||
// Pump_001 and Valve_001 are both children of Line3 (GobjectId = 2).
|
||||
BrowseChildrenReply unconstrained = await service.BrowseChildren(
|
||||
new BrowseChildrenRequest { ParentGobjectId = 2 },
|
||||
new TestServerCallContext());
|
||||
// Phase 1: an authenticated identity with no BrowseSubtrees constraint → browse is
|
||||
// unconstrained. Pump_001 and Valve_001 are both children of Line3 (GobjectId = 2).
|
||||
ApiKeyIdentity unscopedIdentity = new(
|
||||
KeyId: "test-key",
|
||||
KeyPrefix: "mxgw_test",
|
||||
DisplayName: "unscoped",
|
||||
Scopes: new HashSet<string>(StringComparer.Ordinal) { GatewayScopes.MetadataRead },
|
||||
Constraints: ApiKeyConstraints.Empty);
|
||||
|
||||
Assert.Equal(2, unconstrained.Children.Count);
|
||||
using (identityAccessor.Push(unscopedIdentity))
|
||||
{
|
||||
BrowseChildrenReply unconstrained = await service.BrowseChildren(
|
||||
new BrowseChildrenRequest { ParentGobjectId = 2 },
|
||||
new TestServerCallContext());
|
||||
|
||||
Assert.Equal(2, unconstrained.Children.Count);
|
||||
}
|
||||
|
||||
// Phase 2: push a constraint whose BrowseSubtrees do not match any seeded object.
|
||||
// GatewayBrowseScopeProvider.ResolveBrowseSubtrees must return ["NonExistent"],
|
||||
|
||||
@@ -719,6 +719,47 @@ public sealed class GatewaySessionTests
|
||||
Assert.Equal([1UL, 2UL], received.Select(mxEvent => mxEvent.WorkerSequence).ToArray());
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// GWC-34, the disposal-side twin of the GWC-27 premature-attach poison: an attach that
|
||||
/// passed its readiness gate and then lost the CPU until the session was torn down used to
|
||||
/// find <c>_eventDistributor</c> null, read that as "not created yet" and build a
|
||||
/// REPLACEMENT distributor. Its pump could never start (the started flag is a one-way latch)
|
||||
/// and its channel was never completed, so the subscriber registered on it and blocked in
|
||||
/// <c>WaitToReadAsync</c> until its own deadline — no error, no log, no metric. Disposal now
|
||||
/// latches the null-out, so every attach entry point fails immediately and loudly instead.
|
||||
/// Attaching twice pins the rollback: the second call must still report
|
||||
/// <c>SessionNotReady</c>, not <c>EventSubscriberAlreadyActive</c>, which it only can if the
|
||||
/// first failure released the subscriber slot it took.
|
||||
/// </summary>
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
[Fact]
|
||||
public async Task AttachAfterDisposeThrowsSessionNotReadyInsteadOfRegisteringOnANeverPumpedDistributor()
|
||||
{
|
||||
FakeWorkerClient workerClient = new();
|
||||
GatewaySession session = CreateReadySession(workerClient);
|
||||
|
||||
// Disposal leaves the session state Ready (only CloseAsync transitions it), so the attach
|
||||
// paths' readiness gates pass and the calls below genuinely reach EnsureDistributorCreated —
|
||||
// the exact position the racing attach in the finding occupies.
|
||||
await session.DisposeAsync();
|
||||
|
||||
SessionManagerException attachException = Assert.Throws<SessionManagerException>(
|
||||
() => session.AttachEventSubscriber(maxSubscribers: 1));
|
||||
Assert.Equal(SessionManagerErrorCode.SessionNotReady, attachException.ErrorCode);
|
||||
|
||||
SessionManagerException secondAttachException = Assert.Throws<SessionManagerException>(
|
||||
() => session.AttachEventSubscriber(maxSubscribers: 1));
|
||||
Assert.Equal(SessionManagerErrorCode.SessionNotReady, secondAttachException.ErrorCode);
|
||||
|
||||
SessionManagerException replayException = Assert.Throws<SessionManagerException>(
|
||||
() => session.AttachEventSubscriberWithReplay(maxSubscribers: 1, afterSequence: 0));
|
||||
Assert.Equal(SessionManagerErrorCode.SessionNotReady, replayException.ErrorCode);
|
||||
|
||||
SessionManagerException internalException = Assert.Throws<SessionManagerException>(
|
||||
() => session.AttachInternalEventSubscriber());
|
||||
Assert.Equal(SessionManagerErrorCode.SessionNotReady, internalException.ErrorCode);
|
||||
}
|
||||
|
||||
private static GatewaySession CreateSession()
|
||||
{
|
||||
return new GatewaySession(
|
||||
|
||||
@@ -14,6 +14,10 @@ public sealed class WorkerClientTests
|
||||
private const string SessionId = "session-worker-client";
|
||||
private const string Nonce = "nonce-worker-client";
|
||||
private const int WorkerProcessId = 4321;
|
||||
|
||||
/// <summary>Upper bound on the events <see cref="EmitEventsUntilStoppedAsync"/> writes.</summary>
|
||||
private const ulong EmittedEventCeiling = 2000;
|
||||
|
||||
private static readonly TimeSpan TestTimeout = TimeSpan.FromSeconds(5);
|
||||
|
||||
/// <summary>Verifies that StartAsync enters ready state after receiving worker hello and ready messages.</summary>
|
||||
@@ -160,7 +164,7 @@ public sealed class WorkerClientTests
|
||||
TestTimeout,
|
||||
CancellationToken.None);
|
||||
|
||||
// The timeout also emits a WorkerCancel for the abandoned correlation (GWC-31), which sits
|
||||
// The timeout also emits a WorkerCancel for the abandoned correlation (PERF-07), which sits
|
||||
// ahead of the second command on the FIFO pipe; skip it rather than mistaking it for the
|
||||
// command this assertion is about.
|
||||
WorkerEnvelope secondCommand = await ReadNextCommandAsync(pipePair, timedOutCommand.CorrelationId);
|
||||
@@ -176,7 +180,7 @@ public sealed class WorkerClientTests
|
||||
/// <summary>
|
||||
/// A command timeout abandons the gateway-side wait, but the worker keeps the correlation on
|
||||
/// its single STA queue and would still run it — so the gateway forwards a <c>WorkerCancel</c>
|
||||
/// for the abandoned correlation id (GWC-31). Without it, a client that retries after a
|
||||
/// for the abandoned correlation id (PERF-07). Without it, a client that retries after a
|
||||
/// timeout stacks work the worker still intends to execute. Asserted on the wire because the
|
||||
/// cancel is protocol behavior the worker depends on, not an internal detail.
|
||||
/// </summary>
|
||||
@@ -216,6 +220,151 @@ public sealed class WorkerClientTests
|
||||
Assert.Equal(WorkerClientState.Ready, client.State);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Caller cancellation abandons the correlation exactly as a gateway timeout does — a client
|
||||
/// deadline shorter than the gateway's command timeout, or a disconnect, is the common
|
||||
/// abandonment path — so it forwards the same <c>WorkerCancel</c> (IPC-34). Without it a
|
||||
/// retry storm of deadline-cancelled calls leaves the worker's single STA working through
|
||||
/// commands nobody is waiting for while live callers queue behind them. Twin of
|
||||
/// <see cref="InvokeAsync_WhenCommandTimesOut_SendsWorkerCancelForThatCorrelation"/>, driven
|
||||
/// by the caller's token instead of the timer.
|
||||
/// </summary>
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
[Fact]
|
||||
public async Task InvokeAsync_WhenCallerCancels_SendsWorkerCancelForThatCorrelation()
|
||||
{
|
||||
await using PipePair pipePair = await PipePair.CreateAsync();
|
||||
await using WorkerClient client = CreateClient(pipePair);
|
||||
await CompleteHandshakeAsync(client, pipePair);
|
||||
|
||||
using CancellationTokenSource callerCts = new();
|
||||
|
||||
// Advise rather than a control command: control commands bypass the worker's STA queue, so a
|
||||
// data command is the case the cancel actually exists for. The timeout is the full test
|
||||
// timeout so the cancellation, not the timer, is unambiguously what abandons the command.
|
||||
Task<WorkerCommandReply> invokeTask = client.InvokeAsync(
|
||||
CreateCommand(MxCommandKind.Advise),
|
||||
TestTimeout,
|
||||
callerCts.Token);
|
||||
|
||||
WorkerEnvelope commandEnvelope = await pipePair.WorkerReader.ReadAsync().AsTask().WaitAsync(TestTimeout);
|
||||
Assert.Equal(WorkerEnvelope.BodyOneofCase.WorkerCommand, commandEnvelope.BodyCase);
|
||||
|
||||
await callerCts.CancelAsync();
|
||||
|
||||
await Assert.ThrowsAnyAsync<OperationCanceledException>(
|
||||
async () => await invokeTask.WaitAsync(TestTimeout));
|
||||
|
||||
WorkerEnvelope cancelEnvelope = await pipePair.WorkerReader.ReadAsync().AsTask().WaitAsync(TestTimeout);
|
||||
|
||||
Assert.Equal(WorkerEnvelope.BodyOneofCase.WorkerCancel, cancelEnvelope.BodyCase);
|
||||
Assert.Equal(commandEnvelope.CorrelationId, cancelEnvelope.CorrelationId);
|
||||
Assert.False(string.IsNullOrWhiteSpace(cancelEnvelope.WorkerCancel.Reason));
|
||||
Assert.True(
|
||||
cancelEnvelope.Sequence > commandEnvelope.Sequence,
|
||||
$"The cancel arrived with sequence {cancelEnvelope.Sequence} after {commandEnvelope.Sequence}; "
|
||||
+ "envelope sequences must be strictly increasing in wire order.");
|
||||
|
||||
// One caller gave up; the session did not fault.
|
||||
Assert.Equal(WorkerClientState.Ready, client.State);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The cancel is forwarded for a caller that abandoned its command, never for the client's own
|
||||
/// teardown (IPC-34): once the stop token has fired the outbound channel is being completed and
|
||||
/// every pending command is failed anyway, so a cancel would at best be pointless and at worst
|
||||
/// a write racing teardown. Asserted on the wire — the guard in the cancellation arm is the
|
||||
/// code-level expression of it, and the completed outbound channel is the second line of
|
||||
/// defense, so this pins the observable contract rather than either mechanism.
|
||||
/// </summary>
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
[Fact]
|
||||
public async Task InvokeAsync_WhenClientStopsUnderInFlightCommand_DoesNotSendWorkerCancel()
|
||||
{
|
||||
await using PipePair pipePair = await PipePair.CreateAsync();
|
||||
await using WorkerClient client = CreateClient(pipePair);
|
||||
await CompleteHandshakeAsync(client, pipePair);
|
||||
|
||||
using CancellationTokenSource callerCts = new();
|
||||
Task<WorkerCommandReply> invokeTask = client.InvokeAsync(
|
||||
CreateCommand(MxCommandKind.Advise),
|
||||
TestTimeout,
|
||||
callerCts.Token);
|
||||
|
||||
WorkerEnvelope commandEnvelope = await pipePair.WorkerReader.ReadAsync().AsTask().WaitAsync(TestTimeout);
|
||||
Assert.Equal(WorkerEnvelope.BodyOneofCase.WorkerCommand, commandEnvelope.BodyCase);
|
||||
|
||||
// Read ahead before tearing down so anything the teardown path puts on the wire is caught.
|
||||
// Exceptions are swallowed: the read is expected to be still pending at the assertion, and
|
||||
// the stream is disposed underneath it when the test ends.
|
||||
Task<WorkerEnvelope?> pendingRead = Task.Run(async () =>
|
||||
{
|
||||
try
|
||||
{
|
||||
return await pipePair.WorkerReader.ReadAsync();
|
||||
}
|
||||
catch (Exception)
|
||||
{
|
||||
return null;
|
||||
}
|
||||
});
|
||||
|
||||
// Kill fires the client's stop token and then fails the pending command; a caller token that
|
||||
// is wired to gateway shutdown (as the session's is) cancels in the same window.
|
||||
client.Kill("test teardown");
|
||||
await callerCts.CancelAsync();
|
||||
|
||||
await Assert.ThrowsAnyAsync<Exception>(async () => await invokeTask.WaitAsync(TestTimeout));
|
||||
|
||||
await Task.Delay(TimeSpan.FromMilliseconds(250));
|
||||
if (pendingRead.IsCompleted)
|
||||
{
|
||||
WorkerEnvelope? unexpected = await pendingRead;
|
||||
Assert.True(
|
||||
unexpected is null || unexpected.BodyCase != WorkerEnvelope.BodyOneofCase.WorkerCancel,
|
||||
"A stopping worker client must not put a WorkerCancel on the wire.");
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// A <c>WorkerFault</c> arriving in place of a handshake frame is the worker reporting a
|
||||
/// structured startup failure — most often MXAccess COM creation — not a protocol problem, so
|
||||
/// the gateway surfaces <see cref="WorkerClientErrorCode.WorkerFaulted"/> with the fault's
|
||||
/// category and diagnostic rather than a generic protocol violation naming only the body type
|
||||
/// (IPC-35). This is the window where the information matters most: first session open on a
|
||||
/// freshly deployed host, where the message reaches the operator log and the client's
|
||||
/// <c>OpenFailed</c> detail.
|
||||
/// </summary>
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
[Fact]
|
||||
public async Task StartAsync_WhenWorkerFaultsDuringHandshake_SurfacesFaultCategoryAndDiagnostic()
|
||||
{
|
||||
const string diagnostic = "CoCreateInstance for LMXProxy.LMXProxyServer.1 failed (0x80040154).";
|
||||
|
||||
await using FakeWorkerHarness harness = await FakeWorkerHarness.CreateConnectedPairAsync();
|
||||
await using WorkerClient client = harness.CreateClient();
|
||||
|
||||
Task startTask = client.StartAsync(CancellationToken.None);
|
||||
|
||||
WorkerEnvelope gatewayHello = await harness.ReadGatewayEnvelopeAsync().WaitAsync(TestTimeout);
|
||||
Assert.Equal(WorkerEnvelope.BodyOneofCase.GatewayHello, gatewayHello.BodyCase);
|
||||
|
||||
// The hello succeeds; the fault replaces the WorkerReady, which is where a COM creation
|
||||
// failure actually lands (the worker creates MXAccess after answering the hello).
|
||||
await harness.SendWorkerHelloAsync();
|
||||
await harness.EmitFaultAsync(WorkerFaultCategory.MxaccessCreationFailed, diagnostic);
|
||||
|
||||
WorkerClientException exception = await Assert.ThrowsAsync<WorkerClientException>(
|
||||
async () => await startTask.WaitAsync(TestTimeout));
|
||||
|
||||
Assert.Equal(WorkerClientErrorCode.WorkerFaulted, exception.ErrorCode);
|
||||
Assert.Contains(
|
||||
WorkerFaultCategory.MxaccessCreationFailed.ToString(),
|
||||
exception.Message,
|
||||
StringComparison.Ordinal);
|
||||
Assert.Contains(diagnostic, exception.Message, StringComparison.Ordinal);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The envelope <c>sequence</c> is a monotonic per-sender counter (gateway.md), so the values
|
||||
/// observed on the pipe must be strictly increasing in wire order. Stamping the sequence when
|
||||
@@ -560,6 +709,96 @@ public sealed class WorkerClientTests
|
||||
Assert.True(process.Disposed);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// GWC-32: disposing a still-<c>Ready</c> client while the worker is emitting events must be
|
||||
/// an ordinary teardown, not a fault. <c>DisposeAsync</c> completes the same three channels
|
||||
/// <c>MarkClosed</c> and <c>SetFaulted</c> do, so it must enter a terminal state first;
|
||||
/// otherwise the read loop — already inside <c>DispatchEnvelope</c> when the staging channel
|
||||
/// is completed, or reading a stream that is torn down underneath it — misdiagnoses the
|
||||
/// completed channel as the GWC-24 sustained-backpressure overflow and raises a spurious
|
||||
/// <c>ProtocolViolation</c> fault, a <c>worker-event-staging</c> overflow metric and a
|
||||
/// redundant kill during an intentional dispose. The event channel is sized far above the
|
||||
/// burst so a genuine overflow is impossible, and <c>DisposeAsync</c> awaits its background
|
||||
/// tasks, so the metrics read afterwards is not racing a late fault.
|
||||
/// </summary>
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
[Fact]
|
||||
public async Task DisposeAsync_WhileWorkerIsEmittingEvents_ClosesWithoutFaultingOrOverflowing()
|
||||
{
|
||||
// The burst is capped at EmittedEventCeiling and the staging channel holds 2 x capacity, so
|
||||
// even if every emitted event were staged and none consumed the GWC-24 bound is unreachable:
|
||||
// an overflow metric in this test can only be the dispose-race misdiagnosis.
|
||||
const int capacity = 2048;
|
||||
using GatewayMetrics metrics = new();
|
||||
await using PipePair pipePair = await PipePair.CreateAsync();
|
||||
WorkerClient client = CreateClient(
|
||||
pipePair,
|
||||
new WorkerClientOptions
|
||||
{
|
||||
EventChannelCapacity = capacity,
|
||||
EventChannelFullModeTimeout = TimeSpan.FromMinutes(5),
|
||||
HeartbeatGrace = TimeSpan.FromSeconds(30),
|
||||
HeartbeatCheckInterval = TimeSpan.FromSeconds(30),
|
||||
},
|
||||
metrics: metrics);
|
||||
await CompleteHandshakeAsync(client, pipePair);
|
||||
|
||||
using CancellationTokenSource emitCts = new(TestTimeout);
|
||||
Task emitTask = EmitEventsUntilStoppedAsync(pipePair, emitCts.Token);
|
||||
|
||||
// Proves the read loop is mid-stream — not parked before its first event — when dispose runs.
|
||||
await WaitUntilAsync(() => metrics.GetSnapshot().WorkerEventQueueDepth > 0, TestTimeout);
|
||||
|
||||
await client.DisposeAsync().AsTask().WaitAsync(TestTimeout);
|
||||
await emitCts.CancelAsync();
|
||||
await emitTask.WaitAsync(TestTimeout);
|
||||
|
||||
Assert.Equal(WorkerClientState.Closed, client.State);
|
||||
|
||||
GatewayMetricsSnapshot snapshot = metrics.GetSnapshot();
|
||||
Assert.Equal(0L, snapshot.Faults);
|
||||
Assert.Equal(0L, snapshot.QueueOverflows);
|
||||
Assert.Equal(0L, snapshot.WorkerKills);
|
||||
|
||||
// Disposal is now a terminal transition, so the worker that reached Ready is recorded as
|
||||
// stopped exactly once — the running-worker gauge no longer leaks on this path.
|
||||
Assert.Equal(0, snapshot.WorkersRunning);
|
||||
Assert.Equal(1L, snapshot.WorkerExits);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Writes worker events from the fake worker side until cancelled or until
|
||||
/// <see cref="EmittedEventCeiling"/> events have been written, so the gateway read loop has a
|
||||
/// continuous stream to dispatch while a test tears the client down. The ceiling keeps the
|
||||
/// burst structurally below the staging bound the caller configures. Every failure is
|
||||
/// swallowed: disposal closes the gateway end of the pipe underneath this writer, which is
|
||||
/// the situation under test rather than a test failure.
|
||||
/// </summary>
|
||||
/// <param name="pipePair">The connected pipe pair whose worker side emits the events.</param>
|
||||
/// <param name="cancellationToken">Stops the emission.</param>
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
private static async Task EmitEventsUntilStoppedAsync(
|
||||
PipePair pipePair,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
await Task.Yield();
|
||||
try
|
||||
{
|
||||
for (ulong sequence = 1;
|
||||
sequence <= EmittedEventCeiling && !cancellationToken.IsCancellationRequested;
|
||||
sequence++)
|
||||
{
|
||||
await pipePair.WorkerWriter.WriteAsync(
|
||||
CreateEventEnvelope(sequence, MxEventFamily.OnDataChange),
|
||||
cancellationToken);
|
||||
}
|
||||
}
|
||||
catch (Exception)
|
||||
{
|
||||
// Broken pipe, disposed stream or cancellation — all expected once dispose starts.
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Verifies that a heartbeat envelope updates the last-heartbeat timestamp and worker
|
||||
/// process id. Uses a <see cref="ManualTimeProvider"/> so the timestamp advance is
|
||||
|
||||
@@ -19,7 +19,7 @@ public sealed class GatewayMetricsTests
|
||||
metrics.CommandFailed("WriteSecured", "AuthorizationFailed", TimeSpan.FromMilliseconds(12));
|
||||
metrics.EventReceived("session-1", "OnDataChange");
|
||||
metrics.EventReceived("session-1", "OnDataChange");
|
||||
// GWC-30: the worker queue-depth gauge sums one live source per worker client, so the two
|
||||
// PERF-02: the worker queue-depth gauge sums one live source per worker client, so the two
|
||||
// registrations below stand in for two concurrent sessions holding 3 and 4 events.
|
||||
using IDisposable workerDepthSourceA = metrics.RegisterWorkerEventQueueDepthSource(static () => 3);
|
||||
using IDisposable workerDepthSourceB = metrics.RegisterWorkerEventQueueDepthSource(static () => 4);
|
||||
@@ -58,7 +58,7 @@ public sealed class GatewayMetricsTests
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// GWC-30: the worker queue-depth gauge sums every registered source rather than holding a
|
||||
/// PERF-02: the worker queue-depth gauge sums every registered source rather than holding a
|
||||
/// single pushed scalar, so concurrent sessions add up instead of overwriting one another,
|
||||
/// and a disposed registration (a worker client going away) drops out of the sum. Disposal
|
||||
/// is idempotent because a client's dispose path can run twice.
|
||||
|
||||
+61
-6
@@ -1,3 +1,7 @@
|
||||
using Grpc.Core;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using Microsoft.Extensions.Options;
|
||||
using ZB.MOM.WW.MxGateway.Server.Configuration;
|
||||
using ZB.MOM.WW.MxGateway.Server.Security.Authentication;
|
||||
using ZB.MOM.WW.MxGateway.Server.Security.Authorization;
|
||||
using ZB.MOM.WW.MxGateway.Tests.TestSupport;
|
||||
@@ -19,7 +23,7 @@ public sealed class GatewayBrowseScopeProviderTests
|
||||
public void ResolveBrowseSubtrees_WithCurrentIdentity_ReturnsConstraintBrowseSubtrees()
|
||||
{
|
||||
StubIdentityAccessor accessor = new(CreateIdentity(["AreaA"]));
|
||||
GatewayBrowseScopeProvider provider = new(accessor);
|
||||
GatewayBrowseScopeProvider provider = CreateProvider(accessor, AuthenticationMode.ApiKey);
|
||||
|
||||
IReadOnlyList<string>? result = provider.ResolveBrowseSubtrees(new TestServerCallContext());
|
||||
|
||||
@@ -27,14 +31,16 @@ public sealed class GatewayBrowseScopeProviderTests
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// With no ambient identity, the provider falls back to the empty browse-subtree
|
||||
/// list, which the library treats as "no scoping" rather than denying everything.
|
||||
/// An authenticated key with no BrowseSubtrees constraint is legitimately unscoped: the
|
||||
/// empty list is what the library reads as "no scoping". This is the state GWC-38 required
|
||||
/// to stay distinguishable from an absent identity, so it is asserted alongside its
|
||||
/// denying twin below rather than left implicit.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void ResolveBrowseSubtrees_WithNoCurrentIdentity_ReturnsEmptyList()
|
||||
public void ResolveBrowseSubtrees_WithUnconstrainedIdentity_ReturnsEmptyList()
|
||||
{
|
||||
StubIdentityAccessor accessor = new(current: null);
|
||||
GatewayBrowseScopeProvider provider = new(accessor);
|
||||
StubIdentityAccessor accessor = new(CreateIdentity([]));
|
||||
GatewayBrowseScopeProvider provider = CreateProvider(accessor, AuthenticationMode.ApiKey);
|
||||
|
||||
IReadOnlyList<string>? result = provider.ResolveBrowseSubtrees(new TestServerCallContext());
|
||||
|
||||
@@ -42,6 +48,55 @@ public sealed class GatewayBrowseScopeProviderTests
|
||||
Assert.Empty(result);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// GWC-38: with API-key authentication enabled, an absent ambient identity means the
|
||||
/// interceptor-ordering invariant is broken — the request's browse scope cannot be
|
||||
/// established. The provider must DENY rather than resolve to the full hierarchy, because
|
||||
/// a silently-unscoped browse is an invisible authorization bypass. The status code is
|
||||
/// pinned: PermissionDenied is what the gRPC surface returns for an authorization failure
|
||||
/// the caller cannot fix by re-authenticating.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void ResolveBrowseSubtrees_WithNoCurrentIdentityAndApiKeyAuth_ThrowsPermissionDenied()
|
||||
{
|
||||
StubIdentityAccessor accessor = new(current: null);
|
||||
GatewayBrowseScopeProvider provider = CreateProvider(accessor, AuthenticationMode.ApiKey);
|
||||
|
||||
RpcException exception = Assert.Throws<RpcException>(
|
||||
() => provider.ResolveBrowseSubtrees(new TestServerCallContext()));
|
||||
|
||||
Assert.Equal(StatusCode.PermissionDenied, exception.StatusCode);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Under <c>Authentication:Mode=Disabled</c> nothing authenticates, so an absent identity
|
||||
/// is the expected steady state rather than a broken invariant: there is no key and no
|
||||
/// constraint to scope by. The provider keeps returning unscoped, so disabling
|
||||
/// authentication does not turn every Galaxy browse into a PermissionDenied.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void ResolveBrowseSubtrees_WithNoCurrentIdentityAndAuthDisabled_ReturnsEmptyList()
|
||||
{
|
||||
StubIdentityAccessor accessor = new(current: null);
|
||||
GatewayBrowseScopeProvider provider = CreateProvider(accessor, AuthenticationMode.Disabled);
|
||||
|
||||
IReadOnlyList<string>? result = provider.ResolveBrowseSubtrees(new TestServerCallContext());
|
||||
|
||||
Assert.NotNull(result);
|
||||
Assert.Empty(result);
|
||||
}
|
||||
|
||||
private static GatewayBrowseScopeProvider CreateProvider(
|
||||
StubIdentityAccessor accessor,
|
||||
AuthenticationMode mode) =>
|
||||
new(
|
||||
accessor,
|
||||
Options.Create(new GatewayOptions
|
||||
{
|
||||
Authentication = new Server.Configuration.AuthenticationOptions { Mode = mode },
|
||||
}),
|
||||
NullLogger<GatewayBrowseScopeProvider>.Instance);
|
||||
|
||||
private static ApiKeyIdentity CreateIdentity(IReadOnlyList<string> browseSubtrees) =>
|
||||
new(
|
||||
KeyId: "operator01",
|
||||
|
||||
@@ -20,7 +20,7 @@ public sealed class WorkerFrameReader
|
||||
private readonly WorkerFrameProtocolOptions _options;
|
||||
private readonly Stream _stream;
|
||||
|
||||
// Reused across frames rather than allocated per read (GWC-30). Safe because ReadAsync is
|
||||
// Reused across frames rather than allocated per read (PERF-22). Safe because ReadAsync is
|
||||
// single-consumer by construction; the prefix is fully overwritten by every read.
|
||||
//
|
||||
// The single-consumer invariant has to survive teardown as well as steady state, because a read
|
||||
|
||||
@@ -668,10 +668,10 @@ public sealed class WorkerFrameWriter
|
||||
// Serialize once into a single pooled buffer that carries the 4-byte length prefix followed
|
||||
// by the payload, then issue one stream write. This avoids a second serialization pass, a
|
||||
// separate prefix array, a separate prefix write, and any per-frame heap allocation — the
|
||||
// gateway-side writer's GWC-30 shape, now matched on the net48 side. The rented buffer may
|
||||
// be larger than requested, so only the first frameLength bytes are ever written. The buffer
|
||||
// is returned only after this frame's write has completed; the batch flush deferred to the
|
||||
// end of the drain (see DrainQueuedFramesAsync) does not read from it.
|
||||
// gateway-side writer's pooled-buffer shape (PERF-22), now matched on the net48 side. The
|
||||
// rented buffer may be larger than requested, so only the first frameLength bytes are ever
|
||||
// written. The buffer is returned only after this frame's write has completed; the batch
|
||||
// flush deferred to the end of the drain (see DrainQueuedFramesAsync) does not read from it.
|
||||
int frameLength = sizeof(uint) + payloadLength;
|
||||
byte[] frame = ArrayPool<byte>.Shared.Rent(frameLength);
|
||||
try
|
||||
|
||||
@@ -1246,7 +1246,14 @@ public sealed class WorkerPipeSession
|
||||
_activeCommandTasks.Add(commandTask);
|
||||
}
|
||||
|
||||
_ = ObserveCommandTaskAsync(commandTask);
|
||||
// WRK-33: every other discarded task in this file is wrapped with ObserveFaultWhenever
|
||||
// (NEXT-04 discipline) so a fault reaching the finalizer unobserved is impossible. This
|
||||
// wrapper task was the one exception — ObserveCommandTaskAsync only swallows
|
||||
// OperationCanceledException, so any other exception escaping ProcessCommandAsync's own
|
||||
// catch path (e.g. a future edit, or a failure past TryWriteFaultAsync's narrow filter)
|
||||
// would fault the discarded task with no observer. Close the gap the same way the rest of
|
||||
// the file does.
|
||||
ObserveFaultWhenever(ObserveCommandTaskAsync(commandTask));
|
||||
}
|
||||
|
||||
private async Task ObserveCommandTaskAsync(Task commandTask)
|
||||
@@ -1395,11 +1402,18 @@ public sealed class WorkerPipeSession
|
||||
// the StaHung fault just sent. Without this the heartbeat loop keeps
|
||||
// advertising a non-faulted state that contradicts the fault.
|
||||
_state = WorkerState.Faulted;
|
||||
// WRK-34: CommandMethod is a method-name field, not a correlation-id field, so a
|
||||
// consumer reading it (dashboard, gateway logs) shouldn't see a bare correlation id
|
||||
// there. Use a stable label and carry the stuck command's correlation id (if any) in
|
||||
// the diagnostic message instead.
|
||||
string correlationSuffix = string.IsNullOrEmpty(snapshot.CurrentCommandCorrelationId)
|
||||
? string.Empty
|
||||
: $" Stuck command correlation id: {snapshot.CurrentCommandCorrelationId}.";
|
||||
await TryWriteFaultAsync(
|
||||
CreateFault(
|
||||
WorkerFaultCategory.StaHung,
|
||||
snapshot.CurrentCommandCorrelationId,
|
||||
$"STA activity is stale by {staleFor}."),
|
||||
"Watchdog",
|
||||
$"STA activity is stale by {staleFor}.{correlationSuffix}"),
|
||||
cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
|
||||
@@ -445,7 +445,8 @@ public sealed class MxAccessCommandExecutor : IStaCommandExecutor
|
||||
// also fire-and-forget in MXAccess, and its commit outcome only exists
|
||||
// in the later OnWriteComplete callback.
|
||||
MxAccessWriteCompletionCache completionCache = session.WriteCompletionCache;
|
||||
ulong completionBaseline = completionCache.CurrentVersion(
|
||||
ulong completionBaseline = CaptureCompletionBaseline(
|
||||
completionCache,
|
||||
writeCommand.ServerHandle,
|
||||
writeCommand.ItemHandle);
|
||||
|
||||
@@ -485,7 +486,8 @@ public sealed class MxAccessCommandExecutor : IStaCommandExecutor
|
||||
|
||||
// Same pre-call baseline rule as ExecuteWriteSecured.
|
||||
MxAccessWriteCompletionCache completionCache = session.WriteCompletionCache;
|
||||
ulong completionBaseline = completionCache.CurrentVersion(
|
||||
ulong completionBaseline = CaptureCompletionBaseline(
|
||||
completionCache,
|
||||
write2Command.ServerHandle,
|
||||
write2Command.ItemHandle);
|
||||
|
||||
@@ -523,7 +525,8 @@ public sealed class MxAccessCommandExecutor : IStaCommandExecutor
|
||||
// immediately after WriteSecured bumps the version past this snapshot,
|
||||
// so a fast commit still correlates (no missed-callback window).
|
||||
MxAccessWriteCompletionCache completionCache = session.WriteCompletionCache;
|
||||
ulong completionBaseline = completionCache.CurrentVersion(
|
||||
ulong completionBaseline = CaptureCompletionBaseline(
|
||||
completionCache,
|
||||
writeSecuredCommand.ServerHandle,
|
||||
writeSecuredCommand.ItemHandle);
|
||||
|
||||
@@ -564,7 +567,8 @@ public sealed class MxAccessCommandExecutor : IStaCommandExecutor
|
||||
|
||||
// Same pre-call baseline rule as ExecuteWriteSecured.
|
||||
MxAccessWriteCompletionCache completionCache = session.WriteCompletionCache;
|
||||
ulong completionBaseline = completionCache.CurrentVersion(
|
||||
ulong completionBaseline = CaptureCompletionBaseline(
|
||||
completionCache,
|
||||
writeSecured2Command.ServerHandle,
|
||||
writeSecured2Command.ItemHandle);
|
||||
|
||||
@@ -994,7 +998,31 @@ public sealed class MxAccessCommandExecutor : IStaCommandExecutor
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Bounded pump-wait for the OnWriteComplete row matching a
|
||||
/// Captures the write-completion baseline for <paramref name="serverHandle"/> /
|
||||
/// <paramref name="itemHandle"/> immediately before a Write/Write2/WriteSecured/
|
||||
/// WriteSecured2 COM call. Pumps once first (WRK-32) so an <c>OnWriteComplete</c>
|
||||
/// already queued for this handle — most commonly the still-undelivered callback of an
|
||||
/// earlier write to the same tag that this worker gave up waiting on — folds into the
|
||||
/// baseline instead of being available to ride this write's reply.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// This narrows, but does not close, the correlation gap documented on
|
||||
/// <see cref="AwaitWriteCompletion"/>: a completion that MXAccess queues after this pump
|
||||
/// step but before the COM call returns is still indistinguishable from this write's own
|
||||
/// completion, because <c>OnWriteComplete</c> carries no per-call token — only the
|
||||
/// server/item handle pair.
|
||||
/// </remarks>
|
||||
private ulong CaptureCompletionBaseline(
|
||||
MxAccessWriteCompletionCache completionCache,
|
||||
int serverHandle,
|
||||
int itemHandle)
|
||||
{
|
||||
pumpStep();
|
||||
return completionCache.CurrentVersion(serverHandle, itemHandle);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Bounded pump-wait for the OnWriteComplete row matching a Write/Write2/
|
||||
/// WriteSecured/WriteSecured2 call, copied onto the reply when it
|
||||
/// arrives in time. The executor holds the STA thread but pumps
|
||||
/// Windows messages each poll (ReadBulk precedent) so the COM callback
|
||||
@@ -1002,6 +1030,21 @@ public sealed class MxAccessCommandExecutor : IStaCommandExecutor
|
||||
/// statuses — the consumer's unconfirmed path, never a synthesized
|
||||
/// failure. Protocol status/hresult stay acceptance-only either way.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Correlation is recency-based per handle, not per call: MXAccess's
|
||||
/// <c>OnWriteComplete</c> callback carries no correlation token, only the server/item
|
||||
/// handle pair, so this method reports the first completion whose version exceeds
|
||||
/// <paramref name="completionBaseline"/> as this write's outcome. If an earlier write to
|
||||
/// the same handle times out unconfirmed and its callback is still undelivered, that
|
||||
/// stale completion can dispatch during this write's pump-wait and ride this reply
|
||||
/// instead — <see cref="CaptureCompletionBaseline"/>'s pre-call pump only folds in a
|
||||
/// completion already queued before this write started; it cannot fold in one queued
|
||||
/// during the COM call or the wait itself. The <c>Hresult</c>/<c>ProtocolStatus</c>
|
||||
/// fields stay acceptance-only regardless, so the exposure is limited to the enrichment
|
||||
/// <c>statuses</c> field, and it is inherent to the COM surface — a native client
|
||||
/// correlating by handle has the identical ambiguity. Full closure needs an
|
||||
/// MXAccess-side correlation token, which does not exist.
|
||||
/// </remarks>
|
||||
private void AwaitWriteCompletion(
|
||||
MxCommandReply reply,
|
||||
MxAccessWriteCompletionCache completionCache,
|
||||
|
||||
Reference in New Issue
Block a user