refactor(comm): retire ClusterClient naming after the gRPC cutover
Phase 4 of the ClusterClient -> gRPC migration deleted the Akka transports but left the naming behind: `ClusterClientSiteAuditClient` was transport- agnostic and worked unchanged, so it survived the deletion under a name that now describes a transport the repo no longer has. Same for a scatter of doc-comments still framing gRPC as "the new transport" beside an Akka one that is gone. Renames it to `SiteCommunicationAuditClient` (and its test file) and rewrites the stale comments to describe the single transport that exists. Also tightens CLAUDE.md: drops the self-describing directory listing and the 27-component enumeration in favour of the non-obvious parts only. Behaviour-neutral: names and prose only. Recorded as the Phase 4 follow-up in docs/plans/2026-07-22-clusterclient-to-grpc-plan.md.
This commit is contained in:
Joseph Doherty
@@ -8,29 +8,12 @@ When a change is requested, the default assumption is: update the design doc *an
|
||||
|
||||
### Top-level directories
|
||||
|
||||
- `src/` — C#/.NET implementation, one project per component (e.g. `ZB.MOM.WW.ScadaBridge.AuditLog`, `ZB.MOM.WW.ScadaBridge.NotificationOutbox`, `ZB.MOM.WW.ScadaBridge.SiteCallAudit`, `ZB.MOM.WW.ScadaBridge.CentralUI`, `ZB.MOM.WW.ScadaBridge.Host`, …). Solution file: `ZB.MOM.WW.ScadaBridge.slnx`.
|
||||
- `tests/` — Test projects (unit + integration).
|
||||
- `docs/` — Design documentation: `docs/requirements/` (high-level + per-component specs), `docs/test_infra/` (test infrastructure), `docs/plans/` (design-decision and implementation-plan docs). The spec the code implements.
|
||||
- `docker/` — 8-node cluster topology (2 central + 3 sites), `deploy.sh`, per-node `appsettings.*.json`. See [`docker/README.md`](docker/README.md) for setup, ports, and management commands. Rebuild + redeploy with `bash docker/deploy.sh`.
|
||||
- `docker-env2/` — Minimal second cluster topology (2 central + 1 site × 2 nodes), runs concurrently with `docker/` on host ports 91XX. Built specifically for testing the Transport (#24) feature with two real environments. See [`docker-env2/README.md`](docker-env2/README.md). Rebuild + redeploy with `bash docker-env2/deploy.sh`.
|
||||
- `infra/` — Docker Compose for local test services (MS SQL, OPC UA, SMTP, REST API, Traefik). **LDAP is no longer started here** — dev/test LDAP is the shared GLAuth on `10.100.0.35:3893` (source of truth: `scadaproj/infra/glauth/`).
|
||||
- `deploy/` — Production/on-host deployment artifacts (e.g. `deploy/wonder-app-vd03/`: `appsettings.Central.json`, `appsettings.Site.json`, `install.ps1`/`uninstall.ps1`, `RUNBOOK.md`).
|
||||
- `deployments/` — Deployment topology notes (`docker-cluster.md`, `docker-cluster-env2.md`, `README.md`).
|
||||
- `code-reviews/` — Per-component code-review notes (one folder per component, plus `_template`).
|
||||
- `tools/` — Repo maintenance/utility scripts (e.g. `rename-to-scadabridge.sh`).
|
||||
- `AkkaDotNet/` — Akka.NET reference documentation and best-practices notes.
|
||||
- `deprecated/` — Retired docs/notes kept for reference.
|
||||
- `logs/` — Local runtime log output.
|
||||
- `vendor/` — Vendored third-party assets (currently an empty placeholder).
|
||||
- `.claude/` — Claude Code project config (settings, skills, agents).
|
||||
Layout is self-describing (`ls`, plus each directory's own README). Only the non-obvious parts are recorded here:
|
||||
|
||||
### Key documents
|
||||
|
||||
- `README.md` — Master index with component table and architecture diagrams.
|
||||
- `docs/requirements/HighLevelReqs.md` — Complete high-level requirements covering all functional areas.
|
||||
- `docs/requirements/Component-*.md` — Individual component design documents (one per component) — the spec the code implements.
|
||||
- `docs/test_infra/test_infra.md` — Master test infrastructure doc (OPC UA, MS SQL, SMTP, REST API, Traefik). LDAP is the shared GLAuth on `10.100.0.35:3893` (not a local infra container; see `scadaproj/infra/glauth/`).
|
||||
- `docs/plans/` — Design decision and implementation-plan documents from refinement sessions.
|
||||
- `docker/` — the primary 8-node cluster topology (2 central + 3 sites). Rebuild + redeploy with `bash docker/deploy.sh`.
|
||||
- `docker-env2/` — minimal second cluster (2 central + 1 site × 2 nodes) that runs **concurrently** with `docker/` on host ports 91XX; built for testing Transport (#24) against two real environments. Rebuild with `bash docker-env2/deploy.sh`.
|
||||
- `infra/` — Docker Compose for local test services (MS SQL, OPC UA, SMTP, REST API, Traefik). **LDAP is no longer started here** — dev/test LDAP is the shared GLAuth on `10.100.0.35:3893` (source of truth: `scadaproj/infra/glauth/`), and the same applies to `docs/test_infra/test_infra.md`, which still reads as if it were a local container.
|
||||
- `deprecated/` — retired docs/notes kept only for reference; not current.
|
||||
|
||||
## Sister Projects
|
||||
|
||||
@@ -70,35 +53,11 @@ Other peers in the `scadaproj` family (see `scadaproj/CLAUDE.md` for details): `
|
||||
- Run tests with `dotnet test ZB.MOM.WW.ScadaBridge.slnx`. `ZB.MOM.WW.ScadaBridge.CLI.Tests` is now a member of the slnx (added 2026-07; arch-review 08 §2.4) — the old "silently skipped" gotcha no longer applies, so a solution-level test run exercises the CLI suite too.
|
||||
- **Propagate cross-repo changes to the umbrella index.** This repo is indexed by the parent workspace `~/Desktop/scadaproj`. When a fact its index records changes here — remote/push status, stack/component summary, cross-project wire relationships (OPC UA → OtOpcUa, gRPC → mxaccessgw), or the solution/namespace shape — update the **ScadaBridge entry in [`../scadaproj/CLAUDE.md`](../scadaproj/CLAUDE.md)** in the same change so the umbrella index never drifts from this repo. (Mirrors the same rule in the peer repos, e.g. `MxAccessGateway`/`HistorianGateway`.)
|
||||
|
||||
## Current Component List (27 components)
|
||||
## Component List
|
||||
|
||||
1. Template Engine — Template modeling, inheritance, composition, validation, flattening, diffs.
|
||||
2. Deployment Manager — Central-side deployment pipeline, system-wide artifact deployment, instance lifecycle.
|
||||
3. Site Runtime — Site-side actor hierarchy (Deployment Manager singleton, Instance/Script/Alarm Actors), script compilation, Akka stream.
|
||||
4. Data Connection Layer — Protocol abstraction (OPC UA, custom), subscription management, clean data pipe.
|
||||
5. Central–Site Communication — Akka.NET ClusterClient (command/control) + gRPC server-streaming (real-time data), message patterns, debug streaming.
|
||||
6. Store-and-Forward Engine — Buffering, fixed-interval retry, parking, SQLite persistence, replication.
|
||||
7. External System Gateway — External system definitions, API method invocation, database connections.
|
||||
8. Notification Service — Central-only notification-list and SMTP definitions, per-type delivery adapters (sites no longer deliver notifications).
|
||||
9. Central UI — Web-based management interface, all workflows.
|
||||
10. Security & Auth — LDAP/AD authentication, role-based authorization, site-scoped permissions.
|
||||
11. Health Monitoring — Site health metrics collection and central reporting.
|
||||
12. Site Event Logging — Local operational event logs at sites with central query access.
|
||||
13. Cluster Infrastructure — Akka.NET cluster setup, active/standby failover, singleton support.
|
||||
14. Inbound API — Web API for external systems, API key auth, script-based implementations.
|
||||
15. Host — Single deployable binary, role-based component registration, Akka.NET bootstrap.
|
||||
16. Commons — Shared types, POCO entity classes, repository interfaces, message contracts.
|
||||
17. Configuration Database — EF Core data access layer, repositories, unit-of-work, audit logging (IAuditService), migrations.
|
||||
18. Management Service — Akka.NET actor providing programmatic access to all admin operations, ClusterClientReceptionist registration.
|
||||
19. CLI — Command-line tool using HTTP Management API, System.CommandLine, JSON/table output.
|
||||
20. Traefik Proxy — Reverse proxy/load balancer fronting central cluster, active node routing via `/health/active`, automatic failover.
|
||||
21. Notification Outbox — Central component ingesting store-and-forwarded notifications, `Notifications` audit table, dispatcher loop, retry/parking, delivery KPIs.
|
||||
22. Site Call Audit — Central component auditing site cached calls (`CachedCall`/`CachedWrite`); `SiteCalls` audit table, telemetry ingest, reconciliation, KPIs, central→site Retry/Discard relay; sites remain the source of truth.
|
||||
23. Audit Log — Central append-only AuditLog table spanning every script-trust-boundary action (outbound API sync+cached, outbound DB sync+cached, notifications, inbound API). Site SQLite hot-path + gRPC telemetry + reconciliation; combined telemetry with Site Call Audit; central direct-write for Notification Outbox dispatch + Inbound API; monthly partitioning, 365-day retention.
|
||||
24. Transport — File-based, encrypted bundle export/import via Central UI. Templates (with **all** child collections — attributes/alarms/scripts/compositions/**native-alarm-sources**, field-complete incl. `LockedInDerived` + script cadence/timeout), system artifacts, central-only configuration, plus site/instance-scoped config (`Site`s, site `DataConnection`s, `Instance`s + real `AreaName` by name) reconciled across environments by a `BundleNameMap` name-mapping subsystem. Per-conflict resolution with a per-line Myers diff. Runs the **script trust gate** at import review (5th `ScriptTrustValidator` call site — forbidden-API scripts rejected pre-runtime; covers template/shared/ApiMethod bodies, template script + alarm Expression-trigger bodies, **and instance alarm-override trigger expressions** — all hard-block; template-script name-resolution findings are advisory warnings, ApiMethod findings hard-block). Correlated audit via `BundleImportId`. Never touches site runtime nodes (imported instances land `NotDeployed`).
|
||||
25. Script Analysis — Shared authoritative script-trust analyzer: unified forbidden-API deny-list (`ScriptTrustPolicy`), fused semantic+syntactic validator (`ScriptTrustValidator`), Roslyn compile wrapper (`RoslynScriptCompiler`), and compile-only globals stubs (`ScriptCompileSurface`/`TriggerCompileSurface`); consumed by Template Engine, Site Runtime, Inbound API, and Central UI.
|
||||
26. KPI History — Reusable central KPI-history backbone: tall/EAV `KpiSample` store in central MS SQL, `KpiHistoryRecorderActor` cluster singleton (`kpi-history-recorder`, not readiness-gated) sampling DI-registered `IKpiSampleSource`s every minute, bucketed query (`GetRawSeriesAsync` + `KpiSeriesBucketer`) + scoped `KpiHistoryQueryService`, and a reusable custom-SVG `KpiTrendChart`; ships trends for Notification Outbox, Site Call Audit, Audit Log, and Site Health.
|
||||
27. DelmiaNotifier — Standalone external client tool (NOT a cluster component, NOT in the Host): a compact Native-AOT (`win-x64`) console app (`WWNotifier.exe`) that DELMIA Apriso shells out to per recipe download. POSTs to the Inbound API `DelmiaRecipeDownload` method (`X-API-Key`, key from `SCADABRIDGE_API_KEY`), with connect-failure-only failover across a comma-list of base URLs, and reports the legacy `YES`/`NO` + exit-code stdout contract — a drop-in replacement for the legacy `WWNotifier` (see `docs/former-api-specs/dnc/`). Zero-dependency BCL-only, `System.Text.Json` source-gen. Project README: `src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier/README.md`; design: `docs/plans/2026-06-26-delmia-recipe-notifier-design.md`.
|
||||
27 components. The catalog — what each one is and its key design commitments — lives in the
|
||||
`scadabridge-components` skill (`.claude/skills/scadabridge-components/SKILL.md`); the authoritative
|
||||
spec for each is `docs/requirements/Component-<Name>.md`, and `README.md` carries the component table.
|
||||
|
||||
## Key Design Decisions (for context across sessions)
|
||||
|
||||
@@ -120,29 +79,26 @@ Other peers in the `scadaproj` family (see `scadaproj/CLAUDE.md` for details): `
|
||||
- DCL write failures returned synchronously to calling script.
|
||||
- Tag path resolution retried periodically for devices still booting.
|
||||
- Static attribute writes persisted to local SQLite (survive restart/failover, reset on redeployment).
|
||||
- **Consolidated site database (LocalDb Phase 1 + 2, complete 2026-07-20).** **Ten** tables now live in ONE `ZB.MOM.WW.LocalDb`-managed SQLite file — the Phase 1 pair (`OperationTracking`, `site_events`) plus Phase 2's `sf_messages` and the seven site config tables (`deployed_configurations`, `static_attribute_overrides`, `shared_scripts`, `external_systems`, `database_connections`, `data_connection_definitions`, `native_alarm_state`), configured by the **required** `LocalDb:Path` (`/app/data/site-localdb.db` on the rig; validated with `ValidateOnStart`, so a site config missing it fails to boot). Both are `RegisterReplicated` tables. Consequences worth knowing:
|
||||
- **Consolidated site database (LocalDb Phase 1 + 2, complete 2026-07-20).** Ten tables now live in ONE `ZB.MOM.WW.LocalDb`-managed SQLite file (enumerate them with `grep -r RegisterReplicated src/`), configured by the **required** `LocalDb:Path` (`/app/data/site-localdb.db` on the rig; validated with `ValidateOnStart`, so a site config missing it fails to boot). Consequences worth knowing:
|
||||
- `site_events.id` changed from autoincrement INTEGER to an application-minted **GUID**. Last-writer-wins keys on the primary key, so two nodes independently minting `id=1,2,3…` would destroy each other's events rather than merge them. The event-log read path uses a composite `(timestamp, id)` keyset cursor with an **opaque string** continuation token; `EventLogEntry.Id` and both `ContinuationToken`s are `string`/`string?` on the site↔central Akka DTOs.
|
||||
- `ScadaBridge:OperationTracking:ConnectionString`, `ScadaBridge:SiteEventLog:DatabasePath` and — as of Phase 2 — `ScadaBridge:StoreAndForward:SqliteDbPath` + `ScadaBridge:Database:SiteDbPath` are all **migration-only** — nothing reads them but `SiteLocalDbLegacyMigrator`, which copies a pre-Phase-1 file in once (deterministic `mig-{NodeName}-{legacyId}` event ids, `INSERT OR IGNORE`, runs AFTER `RegisterReplicated` so migrated rows replicate) and renames it `.migrated`. Delete the keys once a node has migrated.
|
||||
- This incidentally fixes a data-loss bug: both legacy databases defaulted to CWD-relative paths **outside** the mounted volume and were discarded on every container recreate.
|
||||
- **Replication is default-OFF and opt-in** via `LocalDb:Replication:PeerAddress` + a matching `ApiKey` on both nodes. `LocalDbSyncAuthInterceptor` is **fail-closed**: no configured key means no sync stream is accepted at all, so a key typo does not degrade to unauthenticated replication — the pair simply stops converging. The sync endpoint shares the existing site gRPC h2c listener (8083); no new port. Rig posture: **site-a replicated, site-b/site-c deliberately not**, so both states are proven side-by-side. Status surfaces on the site health report as `LocalDbReplicationConnected` / `LocalDbOplogBacklog` (both nullable — null means "no data", NOT "disconnected with an empty backlog") and as `localdb_*` Prometheus series. Note `ZbTelemetryOptions.Meters` is an **allowlist** (`SiteServiceRegistration.ObservedMeters`); an unlisted meter exports nothing, silently.
|
||||
- Design: **scadaproj** `docs/plans/2026-07-19-scadabridge-localdb-design.md` (that doc lives in the umbrella repo, not here); Phase 1 plan + Phase 2 gate are here under `docs/plans/`. **Phase 2 is COMPLETE** (branch `feat/localdb-phase2`, live gate PASS 2026-07-20 — all 10 checks, evidence in `docs/plans/2026-07-19-localdb-phase2-live-gate.md`). It moved the config tables + `sf_messages` in and **deleted** `SiteReplicationActor`, its `ReplicationMessages`, StoreAndForward's `ReplicationService`, and `StoreAndForwardStorage.ReplaceAllAsync`. What those did, and why nothing replaced them:
|
||||
- Design: **scadaproj** `docs/plans/2026-07-19-scadabridge-localdb-design.md` (that doc lives in the umbrella repo, not here); Phase 1 plan + Phase 2 gate are here under `docs/plans/`. Phase 2 **deleted** `SiteReplicationActor`, its `ReplicationMessages`, StoreAndForward's `ReplicationService`, and `StoreAndForwardStorage.ReplaceAllAsync` — do not reintroduce them:
|
||||
|
||||
- `SiteReplicationActor` pushed config deploys to the peer and ran a **notify-and-fetch** exchange (tell the standby a deploy happened; it HTTP-fetches the config itself, with retries and a superseded-404 path). Config rows now simply replicate — **the standby makes no fetch at all** during a deploy (verified live). `SiteReconciliationActor` still fetches at node STARTUP when central reports gaps; that path survives and is a different thing.
|
||||
- `ReplicationService` fanned each buffer mutation (add/remove/park/requeue) to the standby by hand. CDC triggers on `sf_messages` do it now.
|
||||
- `ReplaceAllAsync` was a destructive delete-all-then-insert-all resync. It was not merely unused after the cutover but **unsafe to keep**: a mass DELETE on a now-replicated table would be captured and shipped to the peer. LocalDb's snapshot resync merges per row under LWW and never deletes, which is also why the old N1 directional-authority guard is gone — there is no wipe left to gate.
|
||||
- CDC replication does all three jobs now: config deploys reach the standby as ordinary row changes — **the standby makes no fetch at all** during a deploy (`SiteReconciliationActor`'s node-STARTUP fetch when central reports gaps is a different, surviving path) — and buffer mutations replicate via triggers on `sf_messages`. `ReplaceAllAsync` was a destructive delete-all-then-insert-all resync and is **unsafe to reintroduce**: a mass DELETE on a replicated table would be captured and shipped to the peer. LocalDb's snapshot resync merges per row under LWW and never deletes, which is also why the old N1 directional-authority guard is gone — there is no wipe left to gate.
|
||||
- **`notification_lists` and `smtp_configurations` are created but deliberately NOT registered.** They are permanently empty on a site (no writer since 2026-07-10, the migrator skips them, the active-node purge keeps them empty), and registering them would open a standing replication channel whose only historical payload was plaintext SMTP passwords. Pinned by a security-named test, and verified live: those two tables have **no CDC triggers** on either rig node.
|
||||
- **Operational constraints (read before upgrading a site pair):** stop and start both nodes TOGETHER — rolling one at a time is no longer supported, since the legacy `SfBufferSnapshot` compatibility handler went with the replicator. And a node offline longer than `LocalDb:Replication:TombstoneRetention` (default 7 days) can resurrect deleted rows on rejoin. See `docs/deployment/topology-guide.md`.
|
||||
- `LocalDb:Replication:MaxBatchSize` batches by ROW COUNT, not bytes, against a 4 MB gRPC cap — the rig pins it to **16** (~70 KB worst-case `config_json` x 16 ~= 1.1 MB). The 500 default would allow ~35 MB.
|
||||
- All timestamps are UTC throughout the system.
|
||||
- Inter-cluster communication uses **three** transports, not two — **all cross-cluster command/control and data now rides gRPC** after the ClusterClient→gRPC migration's Phase 4 (`docs/plans/2026-07-22-clusterclient-to-grpc-plan.md`) deleted Akka `ClusterClient`/`ClusterClientReceptionist`: (1) **gRPC command/control** — site→central over the central-hosted `CentralControlService` (`GrpcCentralTransport`, sticky central-a→central-b channel pair; deployments/notifications/health/heartbeat/audit-ingest/reconcile), and central→site over the site-hosted `SiteCommandService` (`GrpcSiteTransport`, per-site NodeA→NodeB channel pair; the 28 lifecycle/OPC-UA/query/parked/route/failover commands); (2) **gRPC** server-streaming for real-time data (attribute values, alarm states, `SiteStreamService`); and (3) **plain token-gated HTTP** for the deployment config itself — notify-and-fetch, the site pulls the config from `DeploymentConfigEndpoints` (`ManagementService/DeploymentConfigEndpoints.cs`) with an `X-Deployment-Token` header, `AllowAnonymous` with the per-deployment token as the entire security boundary. The gRPC boundary is per-site PSK-authenticated (`ControlPlaneAuthInterceptor`, unchanged). There is **no receptionist registration** — discovery is by dialling configured endpoints; central builds one `SitePairChannelProvider` per site (addresses from `Site.GrpcNodeAAddress`/`GrpcNodeBAddress`, refreshed from the DB every 60s and on admin changes), sites dial `ScadaBridge:Communication:CentralGrpcEndpoints` (both central nodes, h2c on `CentralGrpcPort` 8083, **NOT** via Traefik). **Discovery is asymmetric by design:** central discovers site gRPC addresses from the *database* (refreshable at runtime), sites discover central from *appsettings* (`CentralGrpcEndpoints`, static — restart required; `StartupValidator` requires a Site node to list at least one). `Akka.Cluster.Tools` stays for ClusterSingleton; only the ClusterClient part is gone. **Central never buffers for an unreachable site** — the send fails with the caller's Ask/deadline timing out; a `ConnectionStateChanged` mechanism built for this was deleted as dead code.
|
||||
- **All clusters share ONE ActorSystem name**, `"scadabridge"` — hardcoded at `AkkaHostedService.cs:191`. Central and each site are separate clusters *only* by seed-node partitioning. This is required, not incidental: Akka.Remote address matching means a ClusterClient could not reach a differently-named system.
|
||||
- **All clusters share ONE ActorSystem name**, `"scadabridge"` — hardcoded in `AkkaHostedService` at the `ActorSystem.Create` call. Central and each site are separate clusters *only* by seed-node partitioning. The constraint originated with ClusterClient (Akka.Remote address matching meant it could not reach a differently-named system); whether it is still load-bearing after the gRPC migration has **not** been re-verified, so treat the name as fixed until someone checks.
|
||||
- **`ActiveNodeEvaluator.SelfIsOldestUp` is THE single definition of "active node"** (`Communication/ClusterState/ActiveNodeEvaluator.cs`) — the **oldest Up member** in a role scope, and explicitly **never `cluster.State.Leader`**: leadership (lowest address) is an Akka-internal concept that diverges from singleton placement permanently once the original first node restarts and rejoins, and both sides claim it during a partition. The equivalence *oldest-Up == where `ClusterSingletonManager` places singletons* **is** the design. `ClusterActivityEvaluator.SelfIsOldest`, the S&F delivery gate, `/health/active` and the heartbeat `IsActive` stamp all delegate here.
|
||||
- Site nodes carry **two Akka roles**: the base `Site` plus a site-specific `site-{SiteId}` (`BuildRoles`, `AkkaHostedService.cs:386`). Singletons scope to the **site-specific** role.
|
||||
- **The gRPC boundary is authenticated (PSK) as of 2026-07-22; Akka remoting still is not, and nothing is encrypted.** Akka remoting sets no `enable-ssl`, no secure cookie, no `trusted-selection-paths` — so the ClusterClient command/control path remains open to anyone who can reach the remoting port, and the boundary still assumes a trusted network. The gRPC listener stays **h2c**, but `SiteStreamService` is no longer open: `ControlPlaneAuthInterceptor` (`Host/ControlPlaneAuthInterceptor.cs`) gates `/sitestream.SiteStreamService/` — including the `PullAuditEvents`/`PullSiteCalls` RPCs that return audit rows — against a **per-site preshared key**, fail-closed, constant-time compared, alongside the separate `LocalDbSyncAuthInterceptor` on `/localdb_sync.v1.LocalDbSync/` with its own separate key. **Site side:** `ScadaBridge:Communication:GrpcPsk`, in production `${secret:SB-GRPC-PSK-<siteId>}`, and **`StartupValidator` refuses to boot a site node without it** (an unset key would leave the node healthy-looking but serving nothing). **Central side:** `SitePskProvider` resolves `SB-GRPC-PSK-{siteId}` from the secrets store at channel-build time (sites are added at runtime, so no boot-time expansion is possible), with `ScadaBridge:Communication:SitePsks:{siteId}` as an override for hosts running without a master key — the docker rig uses the latter. One key per site, never fleet-wide. A bearer token over h2c is readable and replayable on-path; TLS is the follow-on hardening and needs no change to this design. Introduced by Phase 0 of the ClusterClient→gRPC migration (`docs/plans/2026-07-22-clusterclient-to-grpc-plan.md`).
|
||||
- **Akka frame size is the default 128 KB with `log-frame-size-exceeding` off**, and no custom serializer is configured — so payload carried over ClusterClient is JSON-escaped a second time by the default Newtonsoft serializer, roughly doubling it. Over the limit the transport drops **that one message** without tearing down the association (heartbeats keep flowing, the site still reports healthy) and the central Ask simply times out. See `docs/known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md`; `DeployArtifactsCommand` still carries payload and remains exposed.
|
||||
- gRPC streaming channel — **note the direction is inverted from the data flow**: data moves site→central, but each **site node hosts the gRPC server** (`SiteStreamGrpcServer`, Kestrel h2c, port 8083, mapped **only in the Site branch** of `Program.cs`) and **central is the client**, dialling in. There is **no gRPC server on central at all**, which is why the two `Ingest*` unary RPCs — documented as a "central-side ingest surface" — are dead in practice (acknowledged in `AkkaHostedService.cs:510-519`); sites reach central over ClusterClient instead. Central creates per-site `SiteStreamGrpcClient` via `SiteStreamGrpcClientFactory`, keyed **`(siteId, endpoint)`** — the key was widened from site-only to fix an arch-review High where one session's NodeA→NodeB flip disposed a channel another session was still using. Site entity has GrpcNodeAAddress/GrpcNodeBAddress fields. Proto: `sitestream.proto`, **6 RPCs** (2 server-streaming: `SubscribeInstance`, `SubscribeSite` (site-wide, alarm-only); 4 unary: `IngestAuditEvents`, `IngestCachedTelemetry`, `PullAuditEvents`, `PullSiteCalls`), `SiteStreamEvent` (oneof: AttributeValueUpdate, AlarmStateUpdate). Field numbers are never reused; evolution is additive only (`AlarmStateUpdate` grew 7→23 fields for the native-alarm mirror). Generated C# is **vendored** under `Communication/SiteStreamGrpc/` with the `<Protobuf>` include commented out — regeneration is a manual toggle-build-copy-untoggle. DebugStreamEvent message removed (no longer flows through ClusterClient).
|
||||
- Native alarms: a read-only mirror of native alarms from OPC UA Alarms & Conditions servers and the MxAccess Gateway, unified onto an A&C-style condition model (`AlarmConditionState`: orthogonal Active/Acked/Confirmed/Shelved/Suppressed + 0–1000 severity) plus an `AlarmKind` discriminator (Computed/NativeOpcUa/NativeMxAccess). New DCL capability seam `IAlarmSubscribableConnection` (implemented by the OPC UA and MxGateway adapters); the `DataConnectionActor` opens ONE alarm feed per connection and routes transitions to instances by source-object reference. A `NativeAlarmActor` (peer to the computed `AlarmActor` under `InstanceActor`) mirrors one source binding: snapshot atomic-swap on (re)subscribe, retention (drops once inactive+acked), per-source cap, and site SQLite persistence (`native_alarm_state`, survives failover, cleared on redeploy/undeploy — mirrors static overrides). State streams to central over the additively-enriched gRPC `AlarmStateUpdate` (the existing computed `AlarmStateChanged` was enriched additively) and seeds via the DebugView snapshot. Authoring: `TemplateNativeAlarmSource` / `InstanceNativeAlarmSourceOverride` entities flatten to `ResolvedNativeAlarmSource` (inherit/compose/override); management commands + ManagementActor handlers + CLI (`template/instance native-alarm-source`) + Central UI (template editor tab + instance override panel) + enriched DebugView alarm table. Read-only — no ack-back; no central tables.
|
||||
- OPC UA / MxGateway UX (M7): operator **Alarm Summary** page (`/monitoring/alarms`, RequireDeployment, read-only) fans out the existing per-instance `DebugViewSnapshot` Ask (SemaphoreSlim-capped, partial-results tolerant) and aggregates client-side — no central alarm store; shared `AlarmStateBadges` component. **Aggregated live stream shipped 2026-07-10** (`docs/plans/2026-07-10-aggregated-live-alarm-stream-plan.md`): a transient in-memory per-site central live cache (`ISiteAlarmLiveCache`) fed by a site-wide, alarm-only `SubscribeSite` gRPC stream (seed-then-stream), pushing near-real-time deltas to the page over the Blazor circuit with the 15s poll kept as fallback + NotReporting authority — still no persisted central alarm store. OPC UA node browser gains `BrowseNext` continuation paging ("Load more"), a bounded recursive address-space **search** (`IAddressSpaceSearchable` seam; depth + result caps; substring on DisplayName/path), and **type-info** (DataType/ValueRank/Writable on `BrowseNode` for Variables). Attribute-override **CSV bulk import** (`OverrideCsvParser`, all-or-nothing) via InstanceConfigure `InputFile` + CLI `instance import-overrides --file` (native-alarm-source-override CSV deferred). **Verify-endpoint** probe (temporary `RealOpcUaClient`, short timeout, captures an untrusted server cert but NEVER trusts it) + **site-local cert trust**: per-node `CertStoreActor` (runs on every site node, not a singleton) writing the `.der` into the node's OPC UA trusted-peer PKI store; DeploymentManager broadcasts `TrustServerCertCommand`/`RemoveServerCertCommand` to BOTH site nodes so PKI stores stay consistent across failover; Admin-gated cert-management UI (`/design/connections/{id}/certificates`). No central persistence of cert trust (follow-up).
|
||||
- Site nodes carry **two Akka roles**: the base `Site` plus a site-specific `site-{SiteId}` (`AkkaHostedService.BuildRoles`). Singletons scope to the **site-specific** role.
|
||||
- **The gRPC boundary is authenticated (PSK) as of 2026-07-22; Akka remoting still is not, and nothing is encrypted.** Akka remoting sets no `enable-ssl`, no secure cookie, no `trusted-selection-paths` — so intra-cluster Akka remoting remains open to anyone who can reach the remoting port, and that boundary still assumes a trusted network. The gRPC listener stays **h2c**, but `SiteStreamService` is no longer open: `ControlPlaneAuthInterceptor` (`Host/ControlPlaneAuthInterceptor.cs`) gates `/sitestream.SiteStreamService/` — including the `PullAuditEvents`/`PullSiteCalls` RPCs that return audit rows — against a **per-site preshared key**, fail-closed, constant-time compared, alongside the separate `LocalDbSyncAuthInterceptor` on `/localdb_sync.v1.LocalDbSync/` with its own separate key. **Site side:** `ScadaBridge:Communication:GrpcPsk`, in production `${secret:SB-GRPC-PSK-<siteId>}`, and **`StartupValidator` refuses to boot a site node without it** (an unset key would leave the node healthy-looking but serving nothing). **Central side:** `SitePskProvider` resolves `SB-GRPC-PSK-{siteId}` from the secrets store at channel-build time (sites are added at runtime, so no boot-time expansion is possible), with `ScadaBridge:Communication:SitePsks:{siteId}` as an override for hosts running without a master key — the docker rig uses the latter. One key per site, never fleet-wide. A bearer token over h2c is readable and replayable on-path; TLS is the follow-on hardening and needs no change to this design. Introduced by Phase 0 of the ClusterClient→gRPC migration (`docs/plans/2026-07-22-clusterclient-to-grpc-plan.md`).
|
||||
- gRPC streaming channel — **note the direction is inverted from the data flow**: data moves site→central, but each **site node hosts the gRPC server** (`SiteStreamGrpcServer`, Kestrel h2c, port 8083, mapped **only in the Site branch** of `Program.cs`) and **central is the client**, dialling in. Central creates per-site `SiteStreamGrpcClient` via `SiteStreamGrpcClientFactory`, keyed **`(siteId, endpoint)`** — the key was widened from site-only to fix an arch-review High where one session's NodeA→NodeB flip disposed a channel another session was still using. Proto evolution is **additive only** and field numbers are never reused (`AlarmStateUpdate` grew 7→23 fields for the native-alarm mirror). Generated C# is **vendored** under `Communication/SiteStreamGrpc/` with the `<Protobuf>` include commented out — regeneration is a manual toggle-build-copy-untoggle.
|
||||
- Native alarms are a **read-only** mirror of OPC UA Alarms & Conditions and MxAccess Gateway alarms — **no ack-back, no central tables**; state lives in the site's `native_alarm_state`, survives failover, and is cleared on redeploy/undeploy (mirrors static overrides). Central's per-site live alarm cache (`ISiteAlarmLiveCache`) is **transient in-memory only** — there is deliberately no persisted central alarm store, so the 15s poll remains the NotReporting authority behind the live stream. See `Component-DataConnectionLayer.md` / `Component-CentralUI.md` for the model and the authoring surface.
|
||||
- OPC UA cert trust is **site-local and not persisted centrally** (follow-up): the verify-endpoint probe captures an untrusted server cert but **NEVER trusts it**, and DeploymentManager broadcasts `TrustServerCertCommand`/`RemoveServerCertCommand` to **BOTH** site nodes — `CertStoreActor` runs on every site node, not as a singleton, so PKI stores stay consistent across failover.
|
||||
|
||||
### External Integrations
|
||||
- External System Gateway: HTTP/REST only, JSON serialization, API key + Basic Auth.
|
||||
@@ -218,14 +174,14 @@ Other peers in the `scadaproj` family (see `scadaproj/CLAUDE.md` for details): `
|
||||
- Failure detection: 2s heartbeat, 10s threshold. Total failover ~25s (drill-measured 2026-07-21 under auto-down: active-crash TAKEOVER in 28s, standby-crash removal in 27s with 0 routing blips — `docker/failover-drill.sh`).
|
||||
- CoordinatedShutdown for graceful singleton handover.
|
||||
- Automatic dual-node recovery from persistent storage.
|
||||
- **Active/standby is decided by `ActiveNodeEvaluator.SelfIsOldestUp`, never by cluster leadership** — see the Architecture note above. `/health/active` is **central-only** (site nodes map no `/health/*` at all) and backs both Traefik's active-node routing and `IActiveNodeGate`, so the proxy and the Inbound API always agree on which node is active. Central never needs to know which *site* node is active: ClusterClient contact rotation reaches either receptionist and the site-internal `ClusterSingletonProxy` lands the work on the active node for free. The **exception is gRPC**, which picks `GrpcNodeAAddress`/`GrpcNodeBAddress` explicitly and flips on error.
|
||||
- **Active/standby is decided by `ActiveNodeEvaluator.SelfIsOldestUp`, never by cluster leadership** — see the Architecture note above. `/health/active` is **central-only** (site nodes map no `/health/*` at all) and backs both Traefik's active-node routing and `IActiveNodeGate`, so the proxy and the Inbound API always agree on which node is active. Central reaches a site by dialling `GrpcNodeAAddress`/`GrpcNodeBAddress` explicitly and flipping on error; whichever node answers, the site-internal `ClusterSingletonProxy` lands the work on the active node for free, so central still never needs to track which *site* node is active.
|
||||
- **Seed-node ordering: every node lists ITSELF first (decision 2026-07-22) — the boot-alone gap is CLOSED.** Only `seed-nodes[0]` may self-join to form a new cluster (Akka runs `FirstSeedNodeProcess` for it, `JoinSeedNodeProcess` — which can never form one — for everyone else). All 14 shipped node appsettings now lead with the node's own address, so any node can cold-start alone and become operational unattended (~5s, `seed-node-timeout`); `StartupValidator` fails the boot if the ordering is broken (compares host AND port; Akka does no DNS canonicalisation). Two nodes cold-starting together while mutually reachable converge on ONE cluster via the `InitJoin` handshake — they split only under a genuine boot-time partition, the same class auto-down accepts. **An external self-form timer (`Cluster.Join(SelfAddress)` after a window) was implemented and REJECTED:** it sits outside the join handshake, so on a routine standby restart — where the peer is alive but the join is stalled behind removal of the node's own stale incarnation — it fires mid-join and permanently splits the pair (measured: still split after 90s). Regression tests: `SelfFirstSeedBootstrapTests`. The keep-oldest active-crash total outage was separately closed by the auto-down decision. See `docs/requirements/Component-ClusterInfrastructure.md` → Seed Node Ordering.
|
||||
- **Simultaneous-cold-start split-brain guard (opt-in, Gitea #33) — the residual self-first cost, closed.** Self-first-on-both means a *truly simultaneous* cold start (shared power/hypervisor event) races `FirstSeedNodeProcess` on BOTH nodes → two 1-node clusters that never merge (in-process loopback tests converge and hide it; real parallel VM starts don't — OtOpcUa reproduced it live). Ported from OtOpcUa (`lmxopcua` `d1dac87f`): a **dark switch `ScadaBridge:Cluster:BootstrapGuard:Enabled` (default OFF, guard-off byte-identical)**. On: `BuildHocon` emits an EMPTY seed list (Akka does not auto-join) and `ClusterBootstrapCoordinator` (`IHostedService`, registered in BOTH the Central and Site composition roots) issues ONE reachability-gated `JoinSeedNodes` — the pure `ClusterBootstrapGuard` core makes the lower canonical `host:port` the **founder** (self-first, forms immediately), the higher node TCP-probes the founder up to `PartnerProbeSeconds` (25s) and joins **peer-first** if reachable else **self-first** (cold-start-alone preserved). Decided BEFORE a single join, **never re-forms mid-handshake** (that was the rejected self-form-timer's flaw); case-insensitive founder tie-break; probe timings validated `>0` when enabled. Residual accepted trade: founder dies in the probe→join window → higher node hangs unjoined, coordinator WARNs, a restart recovers it. Tests: `ClusterBootstrapGuardTests` (pure) + `ClusterBootstrapCoordinatorTests` (real-ActorSystem, incl. both-cold-start-together-form-one-cluster). See `docs/requirements/Component-ClusterInfrastructure.md` → Simultaneous cold start.
|
||||
|
||||
### UI & Monitoring
|
||||
- Central UI: Blazor Server (ASP.NET Core + SignalR) with Bootstrap CSS. No third-party component frameworks (no Blazorise, MudBlazor, Radzen, etc.). Build custom Blazor components for tables, grids, forms, etc.
|
||||
- UI design: Clean, corporate, internal-use aesthetic. Not flashy. Use the `frontend-design` skill when designing UI pages/components.
|
||||
- Debug view: real-time streaming via DebugStreamBridgeActor + gRPC (events via SiteStreamGrpcClient, snapshot via ClusterClient). Health dashboard: 10s polling timer. Deployment status: real-time push via SignalR.
|
||||
- Debug view: real-time streaming via DebugStreamBridgeActor + gRPC (events via `SiteStreamGrpcClient`, snapshot via the `SiteCommandService` query surface — `QueryReply.DebugViewSnapshot`). Health dashboard: 10s polling timer. Deployment status: real-time push via SignalR.
|
||||
- Health reports: 30s interval, 60s offline threshold, monotonic sequence numbers, raw error counts per interval.
|
||||
- Dead letter monitoring as a health metric.
|
||||
- Site Event Logging: 30-day retention, 1GB storage cap, daily purge, paginated queries with keyword search.
|
||||
@@ -254,14 +210,9 @@ Other peers in the `scadaproj` family (see `scadaproj/CLAUDE.md` for details): `
|
||||
|
||||
## Tool Usage
|
||||
|
||||
- When consulting with the Codex MCP tool, use model `gpt-5.4`.
|
||||
- When a task requires setting up or controlling system state (sites, templates, instances, data connections, deployments, security, etc.) and the Central UI is not needed, prefer the ScadaBridge CLI over manual DB edits or UI navigation. See [`src/ZB.MOM.WW.ScadaBridge.CLI/README.md`](src/ZB.MOM.WW.ScadaBridge.CLI/README.md) for the full command reference.
|
||||
|
||||
### CLI Quick Reference (Docker / OrbStack)
|
||||
|
||||
- **Management URL**: `http://localhost:9000` — the CLI connects via the Traefik load balancer, which routes to the active central node. Direct access: central-a on port 9001, central-b on port 9002.
|
||||
- **Test user**: `--username multi-role --password password` — has Admin, Design, and Deployment roles. The `admin` user only has the Admin role and cannot create templates, data connections, or deploy.
|
||||
- **Config file**: `~/.scadabridge/config.json` — stores `managementUrl` and default format. See `docker/README.md` for a ready-to-use test config.
|
||||
- **Rebuild cluster**: `bash docker/deploy.sh` — builds the `scadabridge:latest` image and recreates all containers. Run this after code changes to ManagementActor, Host, or any server-side component.
|
||||
- **Infrastructure services**: `cd infra && docker compose up -d` — starts MS SQL, OPC UA, SMTP, and REST API. These are separate from the cluster containers in `docker/`. **LDAP is NOT started here** — it is the shared GLAuth on `10.100.0.35:3893` (dc=zb,dc=local); source of truth and config: `scadaproj/infra/glauth/`.
|
||||
- **All test LDAP passwords**: `password` (see `scadaproj/infra/glauth/config.toml` for users and groups; canonical cross-app login: `multi-role`).
|
||||
Management URL, test credentials, rebuild/redeploy commands and the infra services are in the
|
||||
`scadabridge-cluster-ops` skill (`.claude/skills/scadabridge-cluster-ops/SKILL.md`).
|
||||
|
||||
@@ -144,8 +144,8 @@ public static class ServiceCollectionExtensions
|
||||
|
||||
// ISiteStreamAuditClient: NoOp default. This binding remains correct for
|
||||
// central/test composition roots that have no SiteCommunicationActor.
|
||||
// The real implementation is ClusterClientSiteAuditClient, which pushes
|
||||
// audit telemetry to central over Akka ClusterClient via the site's
|
||||
// The real implementation is SiteCommunicationAuditClient, which pushes
|
||||
// audit telemetry to central via the site's
|
||||
// SiteCommunicationActor — the Host wires it directly into the
|
||||
// SiteAuditTelemetryActor's Props.Create call for site roles (it cannot
|
||||
// be a DI singleton because it needs the SiteCommunicationActor IActorRef,
|
||||
|
||||
@@ -37,7 +37,7 @@ namespace ZB.MOM.WW.ScadaBridge.AuditLog.Site.Telemetry;
|
||||
/// <b>Local-write only — the wire push is the drain actor's job.</b> This
|
||||
/// forwarder is deliberately synchronous against the two site-local SQLite
|
||||
/// stores and never pushes to central itself. The site→central transport is
|
||||
/// now live: <c>ClusterClientSiteAuditClient</c> is the production binding of
|
||||
/// now live: <c>SiteCommunicationAuditClient</c> is the production binding of
|
||||
/// <see cref="ISiteStreamAuditClient"/> on site roles (with
|
||||
/// <c>NoOpSiteStreamAuditClient</c> retained only for central/test composition
|
||||
/// roots). The push happens out-of-band: <see cref="SiteAuditTelemetryActor"/>
|
||||
|
||||
@@ -6,9 +6,9 @@ namespace ZB.MOM.WW.ScadaBridge.AuditLog.Site.Telemetry;
|
||||
/// Mockable abstraction over the central site-audit push surface that
|
||||
/// <see cref="SiteAuditTelemetryActor"/> uses to forward <see cref="AuditEventBatch"/>
|
||||
/// payloads. The production implementation is
|
||||
/// <see cref="ClusterClientSiteAuditClient"/> — a ClusterClient-based client,
|
||||
/// wired in the Host for site roles, that forwards batches to central via the
|
||||
/// site's <c>SiteCommunicationActor</c>. Unit tests substitute via NSubstitute
|
||||
/// <see cref="SiteCommunicationAuditClient"/> — wired in the Host for site
|
||||
/// roles, it forwards batches to central via the site's
|
||||
/// <c>SiteCommunicationActor</c>. Unit tests substitute via NSubstitute
|
||||
/// against this interface so the actor never needs a live transport.
|
||||
/// </summary>
|
||||
public interface ISiteStreamAuditClient
|
||||
@@ -36,8 +36,8 @@ public interface ISiteStreamAuditClient
|
||||
/// once central has acknowledged them.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// The production <see cref="ClusterClientSiteAuditClient"/> forwards over
|
||||
/// the ClusterClient transport; the <see cref="NoOpSiteStreamAuditClient"/>
|
||||
/// The production <see cref="SiteCommunicationAuditClient"/> forwards over
|
||||
/// the site's central transport; the <see cref="NoOpSiteStreamAuditClient"/>
|
||||
/// DI default (used by central and test composition roots) returns an empty
|
||||
/// ack so no rows are flipped.
|
||||
/// </remarks>
|
||||
|
||||
@@ -7,9 +7,8 @@ namespace ZB.MOM.WW.ScadaBridge.AuditLog.Site.Telemetry;
|
||||
/// <see cref="ZB.MOM.WW.ScadaBridge.AuditLog.ServiceCollectionExtensions.AddAuditLog"/>.
|
||||
/// It is a no-op binding for composition roots that have no
|
||||
/// <c>SiteCommunicationActor</c> — central and test roots. Site roles override
|
||||
/// it in the Host with the ClusterClient-based
|
||||
/// <see cref="ClusterClientSiteAuditClient"/>, which actually forwards audit
|
||||
/// telemetry to central.
|
||||
/// it in the Host with <see cref="SiteCommunicationAuditClient"/>, which
|
||||
/// actually forwards audit telemetry to central.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
@@ -33,7 +32,7 @@ public sealed class NoOpSiteStreamAuditClient : ISiteStreamAuditClient
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(batch);
|
||||
// Empty ack — no EventIds will be flipped to Forwarded, so rows stay
|
||||
// Pending until the real ClusterClientSiteAuditClient (or a test stub)
|
||||
// Pending until the real SiteCommunicationAuditClient (or a test stub)
|
||||
// takes over.
|
||||
return Task.FromResult(EmptyAck);
|
||||
}
|
||||
@@ -45,7 +44,7 @@ public sealed class NoOpSiteStreamAuditClient : ISiteStreamAuditClient
|
||||
// Empty ack — same rationale as IngestAuditEventsAsync. The site still
|
||||
// writes the audit + tracking rows to its SQLite stores authoritatively;
|
||||
// central-side state only materialises once the real
|
||||
// ClusterClientSiteAuditClient (or a test stub) is wired in.
|
||||
// SiteCommunicationAuditClient (or a test stub) is wired in.
|
||||
return Task.FromResult(EmptyAck);
|
||||
}
|
||||
}
|
||||
|
||||
+11
-11
@@ -7,11 +7,11 @@ namespace ZB.MOM.WW.ScadaBridge.AuditLog.Site.Telemetry;
|
||||
|
||||
/// <summary>
|
||||
/// Production <see cref="ISiteStreamAuditClient"/> binding for site composition
|
||||
/// roots: pushes audit telemetry to central over Akka <c>ClusterClient</c> via
|
||||
/// the site's <c>SiteCommunicationActor</c>. The actor forwards the command to
|
||||
/// <c>/user/central-communication</c> and the central
|
||||
/// <c>CentralCommunicationActor</c> Asks the <c>AuditLogIngestActor</c> proxy —
|
||||
/// the same command/control transport notifications already use. Wired by the
|
||||
/// roots: pushes audit telemetry to central via the site's
|
||||
/// <c>SiteCommunicationActor</c>, which owns the transport choice (today
|
||||
/// <c>GrpcCentralTransport</c>, over the central-hosted <c>CentralControlService</c>).
|
||||
/// The central <c>CentralCommunicationActor</c> Asks the <c>AuditLogIngestActor</c>
|
||||
/// proxy — the same command/control path notifications already use. Wired by the
|
||||
/// Host for site roles; central and test composition roots keep the
|
||||
/// <see cref="NoOpSiteStreamAuditClient"/> DI default (they have no
|
||||
/// <c>SiteCommunicationActor</c>).
|
||||
@@ -34,12 +34,12 @@ namespace ZB.MOM.WW.ScadaBridge.AuditLog.Site.Telemetry;
|
||||
/// <see cref="CachedTelemetryBatch"/>) because the
|
||||
/// <see cref="SiteAuditTelemetryActor"/> builds them with
|
||||
/// <see cref="AuditEventDtoMapper.ToDto"/>. This client converts them back into
|
||||
/// the <see cref="AuditEvent"/> / <see cref="SiteCall"/> entities the Akka
|
||||
/// command messages carry — the same DTO→entity translation the
|
||||
/// the <see cref="AuditEvent"/> / <see cref="SiteCall"/> entities the command
|
||||
/// messages carry — the same DTO→entity translation the
|
||||
/// <c>SiteStreamGrpcServer</c> performs for the gRPC reconciliation path.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public sealed class ClusterClientSiteAuditClient : ISiteStreamAuditClient
|
||||
public sealed class SiteCommunicationAuditClient : ISiteStreamAuditClient
|
||||
{
|
||||
private readonly IActorRef _siteCommunicationActor;
|
||||
private readonly TimeSpan _askTimeout;
|
||||
@@ -49,15 +49,15 @@ public sealed class ClusterClientSiteAuditClient : ISiteStreamAuditClient
|
||||
/// </summary>
|
||||
/// <param name="siteCommunicationActor">
|
||||
/// The site's <c>SiteCommunicationActor</c> — it forwards the ingest command
|
||||
/// over the registered central ClusterClient and routes the reply back to
|
||||
/// this client's Ask.
|
||||
/// to central over its configured <c>ICentralTransport</c> and routes the
|
||||
/// reply back to this client's Ask.
|
||||
/// </param>
|
||||
/// <param name="askTimeout">
|
||||
/// Ask timeout for the round-trip to central. On expiry the Ask throws
|
||||
/// <see cref="Akka.Actor.AskTimeoutException"/>, which the drain loop treats
|
||||
/// as transient (rows stay <c>Pending</c>).
|
||||
/// </param>
|
||||
public ClusterClientSiteAuditClient(IActorRef siteCommunicationActor, TimeSpan askTimeout)
|
||||
public SiteCommunicationAuditClient(IActorRef siteCommunicationActor, TimeSpan askTimeout)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(siteCommunicationActor);
|
||||
_siteCommunicationActor = siteCommunicationActor;
|
||||
@@ -10,9 +10,9 @@ public class Site
|
||||
public string SiteIdentifier { get; set; }
|
||||
/// <summary>Optional description of the site.</summary>
|
||||
public string? Description { get; set; }
|
||||
/// <summary>Akka remote address for site node A (ClusterClient contact point).</summary>
|
||||
/// <summary>Akka remote address for site node A.</summary>
|
||||
public string? NodeAAddress { get; set; }
|
||||
/// <summary>Akka remote address for site node B (ClusterClient contact point).</summary>
|
||||
/// <summary>Akka remote address for site node B.</summary>
|
||||
public string? NodeBAddress { get; set; }
|
||||
/// <summary>gRPC endpoint for site node A used by the central SiteStreamGrpcClient.</summary>
|
||||
public string? GrpcNodeAAddress { get; set; }
|
||||
|
||||
@@ -28,7 +28,7 @@ public enum SiteCallRelayOutcome
|
||||
NotParked,
|
||||
|
||||
/// <summary>
|
||||
/// The owning site could not be reached (offline / no ClusterClient route /
|
||||
/// The owning site could not be reached (offline / no route to the site /
|
||||
/// relay timed out). The action was NOT applied; the operator may retry once
|
||||
/// the site is back online.
|
||||
/// </summary>
|
||||
|
||||
@@ -19,8 +19,8 @@ namespace ZB.MOM.WW.ScadaBridge.Commons.Messages.DebugView;
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// A new dedicated message type (<c>DebugViewInstanceNotFound</c>) was
|
||||
/// considered but rejected: the ClusterClient / ClusterClientReceptionist
|
||||
/// channel is typed on the request side and the bridge actor is already
|
||||
/// considered but rejected: the site↔central channel is typed on the request
|
||||
/// side and the bridge actor is already
|
||||
/// pattern-matching on <c>DebugViewSnapshot</c> for the initial-snapshot TCS
|
||||
/// in <c>DebugStreamService</c>. Introducing a second reply type would require
|
||||
/// every consumer to handle an additional <c>Ask</c> result union — more change
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
namespace ZB.MOM.WW.ScadaBridge.Commons.Messages.Deployment;
|
||||
|
||||
/// <summary>
|
||||
/// Site→central (over ClusterClient) on node startup: the node's local deployed inventory,
|
||||
/// Site→central on node startup: the node's local deployed inventory,
|
||||
/// so central can reply with fetch tokens for whatever the node is missing or stale
|
||||
/// (self-heal a node that was down during a deploy).
|
||||
/// </summary>
|
||||
|
||||
@@ -108,8 +108,8 @@ public record SiteHealthReport(
|
||||
|
||||
/// <summary>
|
||||
/// Broadcast wrapper used between central nodes to keep per-node
|
||||
/// CentralHealthAggregator state in sync. ClusterClient load-balances each
|
||||
/// incoming SiteHealthReport to one central node; that node re-publishes
|
||||
/// CentralHealthAggregator state in sync. A site delivers each
|
||||
/// SiteHealthReport to ONE central node; that node re-publishes
|
||||
/// this wrapper on a DistributedPubSub topic so the peer node's aggregator
|
||||
/// also processes the report (idempotently — sequence numbers guard against
|
||||
/// double-counting).
|
||||
|
||||
@@ -9,7 +9,7 @@ namespace ZB.MOM.WW.ScadaBridge.Commons.Messages.Instance;
|
||||
/// <b>Site-local only.</b> The optional <see cref="Predicate"/> is a non-serializable
|
||||
/// in-process delegate, so this message MUST flow only within a single site node's
|
||||
/// actor system (script execution → Instance Actor). It is never sent across the
|
||||
/// ClusterClient / gRPC boundary. The value-equality form (<see cref="TargetValueEncoded"/>)
|
||||
/// site↔central gRPC boundary. The value-equality form (<see cref="TargetValueEncoded"/>)
|
||||
/// would serialize, but the routed/inbound variant is deliberately out of scope here.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
|
||||
@@ -6,7 +6,7 @@ namespace ZB.MOM.WW.ScadaBridge.Commons.Messages.Management;
|
||||
/// Bidirectional name registry for management command records. The registry contains
|
||||
/// exactly the non-abstract <c>*Command</c> types declared in the
|
||||
/// <c>ZB.MOM.WW.ScadaBridge.Commons.Messages.Management</c> namespace; these are the commands that
|
||||
/// travel over the HTTP / ClusterClient management boundary.
|
||||
/// travel over the HTTP management boundary.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <see cref="Resolve"/> and <see cref="GetCommandName"/> are symmetric:
|
||||
|
||||
@@ -2,7 +2,7 @@ namespace ZB.MOM.WW.ScadaBridge.Commons.Messages.Management;
|
||||
|
||||
// Schema-library authoring commands. The reusable named JSON-Schema
|
||||
// library (the SharedSchema entity + ISharedSchemaRepository) gains its CRUD
|
||||
// surface here. These records travel the same HTTP / ClusterClient management
|
||||
// surface here. These records travel the same HTTP management
|
||||
// boundary as every other *Command and are auto-discovered by reflection in
|
||||
// ManagementCommandRegistry (no manual registry entry needed).
|
||||
//
|
||||
|
||||
@@ -14,7 +14,7 @@ namespace ZB.MOM.WW.ScadaBridge.Commons.Messages.RemoteQuery;
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Either site node may receive this: <c>SiteCommunicationActor</c> is registered per node
|
||||
/// (not as a singleton), and ClusterClient contact rotation reaches whichever answers. That is
|
||||
/// (not as a singleton), and central's gRPC dial reaches whichever node answers. That is
|
||||
/// fine — <c>Cluster.Leave(address)</c> is valid from any member, and the target is resolved
|
||||
/// from cluster state rather than from who received the message.
|
||||
/// </para>
|
||||
|
||||
@@ -89,7 +89,7 @@ public class CentralCommunicationActor : ReceiveActor
|
||||
/// Default Ask timeout for routing audit ingest commands to the
|
||||
/// Effective Ask timeout for audit ingest routing. Defaults to
|
||||
/// <see cref="Grpc.SiteStreamGrpcServer.AuditIngestAskTimeout"/> (30 s) — the two
|
||||
/// audit-ingest transports (gRPC vs ClusterClient) now share one source of truth
|
||||
/// audit-ingest entry points (the site stream server and the control plane) share one source of truth
|
||||
/// for the timeout. Overridable via the constructor so tests can exercise the
|
||||
/// timeout/fault path without waiting 30 s. When the window is exceeded the Ask
|
||||
/// faults and that fault is piped back to the caller as a
|
||||
@@ -167,8 +167,8 @@ public class CentralCommunicationActor : ReceiveActor
|
||||
});
|
||||
|
||||
// Notification Outbox ingest: a site forwards a buffered NotificationSubmit to the
|
||||
// central cluster via ClusterClient. Forward to the outbox proxy so the original
|
||||
// Sender (the site's ClusterClient path) is preserved and the NotificationSubmitAck
|
||||
// central cluster. Forward to the outbox proxy so the original
|
||||
// Sender (the site's transport path) is preserved and the NotificationSubmitAck
|
||||
// routes straight back to the site.
|
||||
Receive<NotificationSubmit>(HandleNotificationSubmit);
|
||||
|
||||
@@ -186,9 +186,9 @@ public class CentralCommunicationActor : ReceiveActor
|
||||
});
|
||||
|
||||
// Audit Log site→central ingest: a site forwards a batch of audit
|
||||
// events to the central cluster via ClusterClient. Ask the ingest proxy
|
||||
// events to the central cluster. Ask the ingest proxy
|
||||
// and pipe the IngestAuditEventsReply back to the original Sender (the
|
||||
// site's ClusterClient path) so the site can flip its rows to Forwarded.
|
||||
// site's transport path) so the site can flip its rows to Forwarded.
|
||||
Receive<IngestAuditEventsCommand>(HandleIngestAuditEvents);
|
||||
|
||||
// Audit Log combined-telemetry ingest: routes to the same proxy
|
||||
@@ -196,9 +196,9 @@ public class CentralCommunicationActor : ReceiveActor
|
||||
Receive<IngestCachedTelemetryCommand>(HandleIngestCachedTelemetry);
|
||||
|
||||
// Startup reconciliation: a site node forwards its local deployed inventory on
|
||||
// startup via ClusterClient. Resolve the scoped ReconcileService, diff the
|
||||
// startup. Resolve the scoped ReconcileService, diff the
|
||||
// inventory against central's expected set, and pipe the ReconcileSiteResponse
|
||||
// (gap fetch tokens + orphans) straight back to the site node's ClusterClient.
|
||||
// (gap fetch tokens + orphans) straight back to the site node.
|
||||
Receive<ReconcileSiteRequest>(HandleReconcileSiteRequest);
|
||||
}
|
||||
|
||||
@@ -254,7 +254,7 @@ public class CentralCommunicationActor : ReceiveActor
|
||||
}
|
||||
|
||||
// Capture Sender before the async/PipeTo — Akka resets Sender between
|
||||
// dispatches. The reply is piped straight back to the site's ClusterClient.
|
||||
// dispatches. The reply is piped straight back to the calling site node.
|
||||
// On an Ask timeout or a faulted reply, PipeTo delivers a Status.Failure to
|
||||
// replyTo: the fault propagates to the caller rather than being swallowed.
|
||||
// The site's own Ask through this path then faults, and the site drain loop
|
||||
@@ -285,10 +285,10 @@ public class CentralCommunicationActor : ReceiveActor
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Startup reconciliation (site→central over ClusterClient): resolve the scoped
|
||||
/// Startup reconciliation (site→central): resolve the scoped
|
||||
/// <see cref="ReconcileService"/> in a DI scope, diff the node's reported inventory
|
||||
/// against central's expected set, and pipe the <see cref="ReconcileSiteResponse"/>
|
||||
/// back to the site node's ClusterClient path. The actor stays thin — all the diff
|
||||
/// back to the site node's transport path. The actor stays thin — all the diff
|
||||
/// and staging logic lives in the service. Mirrors the DB-access pattern used by
|
||||
/// <see cref="LoadSiteAddressesFromDb"/> (Task.Run + CreateScope + PipeTo) and the
|
||||
/// Sender-preservation pattern of <see cref="HandleIngestAuditEvents"/>.
|
||||
@@ -331,7 +331,7 @@ public class CentralCommunicationActor : ReceiveActor
|
||||
MarkHeartbeatLocally(heartbeat);
|
||||
|
||||
// Fan the heartbeat out to the peer central node so BOTH aggregators mark
|
||||
// it, regardless of which central node the site's ClusterClient delivered
|
||||
// it, regardless of which central node the site delivered
|
||||
// to. Without this, a heartbeat that only ever reaches one node leaves the
|
||||
// other node's aggregator blind to that site's liveness after a failover
|
||||
// (arch review 02, Low). MarkHeartbeat is idempotent (timestamp overwrite),
|
||||
@@ -358,7 +358,7 @@ public class CentralCommunicationActor : ReceiveActor
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Handles a report delivered directly from a site (via ClusterClient):
|
||||
/// Handles a report delivered directly from a site:
|
||||
/// process locally, then fan out to the peer central node so its
|
||||
/// aggregator stays in sync.
|
||||
/// </summary>
|
||||
@@ -407,7 +407,7 @@ public class CentralCommunicationActor : ReceiveActor
|
||||
// HandleConnectionStateChanged removed — no production
|
||||
// caller emitted ConnectionStateChanged, so the workflow ran only in tests.
|
||||
// Disconnect detection is owned by the transport layers (gRPC keepalive +
|
||||
// ClusterClient/Ask timeout).
|
||||
// Ask timeout).
|
||||
|
||||
private void HandleSiteEnvelope(SiteEnvelope envelope)
|
||||
{
|
||||
@@ -491,8 +491,8 @@ public class CentralCommunicationActor : ReceiveActor
|
||||
|
||||
private void HandleSiteAddressCacheLoaded(SiteAddressCacheLoaded msg)
|
||||
{
|
||||
// Per-transport per-site resource reconciliation (create/stop ClusterClients, or
|
||||
// build/drop gRPC channel pairs). Runs on the actor thread each refresh tick.
|
||||
// Per-site transport resource reconciliation (build/drop gRPC channel
|
||||
// pairs). Runs on the actor thread each refresh tick.
|
||||
_transport.ReconcileSites(msg);
|
||||
|
||||
// Self-healing eviction: a site deleted from configuration would otherwise
|
||||
@@ -525,7 +525,7 @@ public class CentralCommunicationActor : ReceiveActor
|
||||
|
||||
// Subscribe to the peer-replication topic so we receive health reports
|
||||
// delivered to the other central node and keep our local aggregator
|
||||
// in sync (ClusterClient load-balances reports across nodes).
|
||||
// in sync (a site may deliver its report to either central node).
|
||||
// Tolerant of non-clustered hosts (TestKit) where the extension is absent.
|
||||
try
|
||||
{
|
||||
@@ -581,7 +581,7 @@ public record RefreshSiteAddresses;
|
||||
/// discipline. The producer wraps the constructed buckets with
|
||||
/// <c>List<T>.AsReadOnly()</c> before piping to Self.
|
||||
/// </summary>
|
||||
/// <param name="SiteContacts">Akka ClusterClient contact addresses per site (from NodeA/NodeBAddress).</param>
|
||||
/// <param name="SiteContacts">Akka remote addresses per site (from NodeA/NodeBAddress).</param>
|
||||
/// <param name="KnownSiteIds">Every configured site id, address-bearing or not, for aggregator pruning.</param>
|
||||
/// <param name="GrpcContacts">
|
||||
/// gRPC endpoint pairs per site (from GrpcNodeA/GrpcNodeBAddress) — the streaming path's columns,
|
||||
@@ -603,8 +603,8 @@ public readonly record struct SiteGrpcEndpoints(string? NodeA, string? NodeB);
|
||||
/// <summary>
|
||||
/// Peer-replication envelope for a site heartbeat, fanned out over the same
|
||||
/// DistributedPubSub topic as <see cref="SiteHealthReportReplica"/> so both
|
||||
/// central aggregators mark heartbeats regardless of which node the site's
|
||||
/// ClusterClient delivered to. Only ever travels central↔central (same assembly
|
||||
/// central aggregators mark heartbeats regardless of which node the site
|
||||
/// delivered to. Only ever travels central↔central (same assembly
|
||||
/// version, rolled together), so the default reflective serializer is safe.
|
||||
/// </summary>
|
||||
public sealed record SiteHeartbeatReplica(HeartbeatMessage Heartbeat);
|
||||
|
||||
@@ -94,10 +94,10 @@ public class DebugStreamBridgeActor : ReceiveActor, IWithTimers
|
||||
/// <summary>
|
||||
/// Initializes the debug stream bridge actor and registers message handlers.
|
||||
/// </summary>
|
||||
/// <param name="siteIdentifier">Site identifier for targeting ClusterClient messages and logging.</param>
|
||||
/// <param name="siteIdentifier">Site identifier for targeting site-addressed messages and logging.</param>
|
||||
/// <param name="instanceUniqueName">Unique name of the instance whose debug stream is being bridged.</param>
|
||||
/// <param name="correlationId">Correlation id for the debug session.</param>
|
||||
/// <param name="centralCommunicationActor">Actor used to forward ClusterClient messages to the site.</param>
|
||||
/// <param name="centralCommunicationActor">Actor used to forward site-addressed messages to the site.</param>
|
||||
/// <param name="onEvent">Callback invoked on each received debug event.</param>
|
||||
/// <param name="onTerminated">Callback invoked when the stream terminates.</param>
|
||||
/// <param name="grpcFactory">Factory for creating gRPC streaming clients.</param>
|
||||
@@ -124,7 +124,7 @@ public class DebugStreamBridgeActor : ReceiveActor, IWithTimers
|
||||
_grpcNodeAAddress = grpcNodeAAddress;
|
||||
_grpcNodeBAddress = grpcNodeBAddress;
|
||||
|
||||
// Initial snapshot response from the site (via ClusterClient).
|
||||
// Initial snapshot response from the site.
|
||||
// If the site reports InstanceNotFound=true the instance is not
|
||||
// deployed there. Under the stream-first lifecycle the gRPC stream
|
||||
// was already opened in PreStart, so the not-found path must tear it down
|
||||
|
||||
@@ -4,13 +4,11 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Actors;
|
||||
|
||||
/// <summary>
|
||||
/// The central→site command-send seam, injected into <see cref="CentralCommunicationActor"/>
|
||||
/// below the <see cref="SiteEnvelope"/> handler. Exactly one implementation is active per node,
|
||||
/// chosen by <c>ScadaBridge:Communication:SiteTransport</c>:
|
||||
/// <list type="bullet">
|
||||
/// <item><see cref="AkkaSiteTransport"/> — today's per-site <c>ClusterClient</c> path (default).</item>
|
||||
/// <item><see cref="ZB.MOM.WW.ScadaBridge.Communication.Grpc.GrpcSiteTransport"/> — the site
|
||||
/// <c>SiteCommandService</c> gRPC plane.</item>
|
||||
/// </list>
|
||||
/// below the <see cref="SiteEnvelope"/> handler. The sole implementation is
|
||||
/// <see cref="ZB.MOM.WW.ScadaBridge.Communication.Grpc.GrpcSiteTransport"/>, over the site
|
||||
/// <c>SiteCommandService</c> gRPC plane. The seam predates that: it was introduced so the Akka
|
||||
/// <c>ClusterClient</c> path and the gRPC plane could run side by side during the migration, and
|
||||
/// is kept because it is the natural substitution point for tests.
|
||||
/// The producers above the seam (<c>CommunicationService</c>'s 27 commands, <c>SiteCallAuditActor</c>'s
|
||||
/// 2 parked relays, and <c>DebugStreamBridgeActor</c>'s subscribe/unsubscribe) are unchanged — they
|
||||
/// still <c>Ask</c>/<c>Tell</c> a <see cref="SiteEnvelope"/> to the actor, which delegates here.
|
||||
@@ -27,14 +25,14 @@ public interface ISiteCommandTransport
|
||||
/// delivered to <paramref name="replyTo"/> — for an <c>Ask</c> that is the temporary ask actor
|
||||
/// (completing the caller's task); for a <c>Tell</c>-with-sender (the debug bridge) that is the
|
||||
/// originating actor. A message with no route (an unknown site) is warned and dropped so the
|
||||
/// caller's <c>Ask</c> times out, exactly as today's ClusterClient path behaves.
|
||||
/// caller's <c>Ask</c> times out — central never buffers for an unreachable site.
|
||||
/// </summary>
|
||||
/// <param name="envelope">The site-addressed command envelope.</param>
|
||||
/// <param name="replyTo">Where a reply (or a <see cref="Status.Failure"/>) is delivered.</param>
|
||||
void Send(SiteEnvelope envelope, IActorRef replyTo);
|
||||
|
||||
/// <summary>
|
||||
/// Reconciles per-site transport resources (ClusterClients for Akka, channel pairs for gRPC)
|
||||
/// Reconciles per-site transport resources (gRPC channel pairs)
|
||||
/// against the freshly loaded site set. Called once per DB refresh tick with the same cache
|
||||
/// message the actor already receives — the ONE DB-poll loop feeds both transports.
|
||||
/// </summary>
|
||||
|
||||
@@ -536,7 +536,7 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
|
||||
if (_retryCount > MaxRetries)
|
||||
{
|
||||
// Give up the stream, but do NOT stop the aggregator: the periodic reconcile
|
||||
// still refreshes the cache from ClusterClient snapshots, so the page keeps a
|
||||
// still refreshes the cache from site snapshots, so the page keeps a
|
||||
// (slower) live-ish view rather than going dark. A later reconcile-triggered
|
||||
// reconnect is not attempted here; the stream is simply left down.
|
||||
_log.Error("Site-alarm gRPC stream for {0} exceeded max retries ({1}); leaving stream down, " +
|
||||
|
||||
@@ -21,8 +21,8 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Actors;
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// Two transports call this one unit: the Akka <see cref="SiteCommunicationActor"/>
|
||||
/// (ClusterClient) and the new <c>SiteCommandGrpcService</c> (gRPC). Centralising the
|
||||
/// Two entry points call this one unit: the site-side <see cref="SiteCommunicationActor"/>
|
||||
/// and <c>SiteCommandGrpcService</c>. Centralising the
|
||||
/// table means the two can never drift on where a command goes. Each transport keeps
|
||||
/// its own send mechanics — the actor <c>Forward</c>s (preserving the Ask sender), the
|
||||
/// gRPC service <c>Ask</c>s and encodes the reply — but both read the same
|
||||
@@ -219,9 +219,9 @@ public sealed class SiteCommandDispatcher
|
||||
private static Route Immediate(object reply) => new(RouteDisposition.ImmediateReply, null, reply);
|
||||
|
||||
/// <summary>
|
||||
/// The actor's failover path: resolve the standby AND issue the leave in one step (today's
|
||||
/// coupled behaviour over ClusterClient, where <c>Tell</c> merely enqueues the ack so leave
|
||||
/// order is immaterial), then return the ack.
|
||||
/// The actor's failover path: resolve the standby AND issue the leave in one step (coupled,
|
||||
/// because <c>Tell</c> merely enqueues the ack so leave order is immaterial here), then
|
||||
/// return the ack.
|
||||
/// </summary>
|
||||
/// <param name="msg">The failover command.</param>
|
||||
/// <returns>The ack to send back.</returns>
|
||||
@@ -283,8 +283,8 @@ public sealed class SiteCommandDispatcher
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
// A fault must be reported to the operator, never thrown into supervision (over
|
||||
// ClusterClient) or surfaced as a broken stream (over gRPC).
|
||||
// A fault must be reported to the operator, never thrown into supervision or
|
||||
// surfaced as a broken stream (over gRPC).
|
||||
return new FailoverOutcome(
|
||||
new SiteFailoverAck(
|
||||
msg.CorrelationId, Accepted: false, TargetAddress: null, ErrorMessage: ex.Message),
|
||||
|
||||
@@ -16,9 +16,10 @@ using ZB.MOM.WW.ScadaBridge.Commons.Messages.RemoteQuery;
|
||||
namespace ZB.MOM.WW.ScadaBridge.Communication.Actors;
|
||||
|
||||
/// <summary>
|
||||
/// Site-side actor that receives messages from central via ClusterClient and routes
|
||||
/// them to the appropriate local actors. Also sends heartbeats and health reports
|
||||
/// to central via the registered ClusterClient.
|
||||
/// Site-side actor that receives messages from central over the site-hosted
|
||||
/// <c>SiteCommandService</c> gRPC plane and routes them to the appropriate local actors.
|
||||
/// Also sends heartbeats and health reports to central over its
|
||||
/// <see cref="ICentralTransport"/> (<see cref="Grpc.GrpcCentralTransport"/> in production).
|
||||
///
|
||||
/// Routes all 8 message patterns to local handlers.
|
||||
/// </summary>
|
||||
@@ -157,15 +158,15 @@ public class SiteCommunicationActor : ReceiveActor, IWithTimers
|
||||
// the SITE-SPECIFIC role, because that is what site singletons (the Deployment
|
||||
// Manager) are placed on. Either node may receive this (the actor is per-node, not a
|
||||
// singleton, and contact rotation picks whichever answers); the target is resolved
|
||||
// from cluster state, not from who received the message. Over ClusterClient the ack
|
||||
// Tell merely enqueues, so the dispatcher resolves-and-leaves in one step (the gRPC
|
||||
// transport defers the leave to keep ack-before-Leave — see PrepareFailover).
|
||||
// from cluster state, not from who received the message. The gRPC transport defers the
|
||||
// leave to keep ack-before-Leave — see PrepareFailover. (Under the removed ClusterClient
|
||||
// path the ack Tell merely enqueued, so the dispatcher resolved-and-left in one step.)
|
||||
Receive<TriggerSiteFailover>(msg => Sender.Tell(_dispatcher.HandleFailover(msg)));
|
||||
|
||||
// The seven site→central sends now delegate to the injected transport (ClusterClient by
|
||||
// default, gRPC when configured). Each handler captures the current Sender as the reply
|
||||
// target so central's reply routes straight back to the waiting Ask, not through this
|
||||
// actor — the exact sender-forwarding the ClusterClient path relied on. The per-message
|
||||
// The seven site→central sends delegate to the injected transport (GrpcCentralTransport in
|
||||
// production). Each handler captures the current Sender as the reply target so central's
|
||||
// reply routes straight back to the waiting Ask, not through this actor — preserving the
|
||||
// sender-forwarding the original ClusterClient path relied on. The per-message
|
||||
// "no transport / not-accepted" fallbacks live inside the transport now.
|
||||
|
||||
// Notification Outbox: forward a buffered notification (S&F forwarder's Ask → ack back).
|
||||
@@ -229,9 +230,9 @@ public class SiteCommunicationActor : ReceiveActor, IWithTimers
|
||||
/// Executes a resolved command route within the actor: Forward to the target (preserving the
|
||||
/// central Ask sender so the reply routes straight back to the waiting Ask), or — when a
|
||||
/// null-guarded handler is unregistered — Tell the caller the dispatcher's synthetic reply.
|
||||
/// The fire-and-forget disposition (UnsubscribeDebugView) is a plain Forward here, exactly as
|
||||
/// before: over ClusterClient the site never acked it, so the synthetic ack is a gRPC-only
|
||||
/// concern.
|
||||
/// The fire-and-forget disposition (UnsubscribeDebugView) is a plain Forward here: the site
|
||||
/// never acked it under the original ClusterClient path either, so the synthetic ack is a
|
||||
/// gRPC-only concern.
|
||||
/// </summary>
|
||||
/// <param name="command">The migrated central→site command to route.</param>
|
||||
private void DispatchCommand(object command)
|
||||
|
||||
@@ -13,7 +13,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
|
||||
///
|
||||
/// Direction: SITE is the client, CENTRAL is the server — the inverse of
|
||||
/// SiteStreamService, where central dials the site. That asymmetry is deliberate
|
||||
/// and mirrors the direction the Akka ClusterClient traffic flows today: these
|
||||
/// and mirrors the direction the Akka ClusterClient traffic used to flow: these
|
||||
/// seven calls are exactly the seven messages SiteCommunicationActor sends to
|
||||
/// /user/central-communication.
|
||||
///
|
||||
|
||||
@@ -139,7 +139,7 @@ public class CommunicationService
|
||||
/// Queries a site for the currently-applied deployment
|
||||
/// identity of a single instance. Used by the Deployment Manager before a
|
||||
/// re-deploy to reconcile against the site's actual state. Sent over the
|
||||
/// existing ClusterClient command/control transport; the Ask times out (no
|
||||
/// gRPC command/control transport; the Ask times out (no
|
||||
/// central buffering) if the site is unreachable, and the caller falls
|
||||
/// through to a normal deploy.
|
||||
/// </summary>
|
||||
|
||||
@@ -16,8 +16,8 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc;
|
||||
/// <para>
|
||||
/// The seven pairs mirror, one for one, the seven messages
|
||||
/// <c>SiteCommunicationActor</c> forwards to <c>/user/central-communication</c> over
|
||||
/// Akka <c>ClusterClient</c> today. Both transports carry the SAME message types
|
||||
/// end-to-end — central's handlers are untouched by the migration — so this mapper is
|
||||
/// the control plane. The in-process and wire representations carry the SAME message types
|
||||
/// end-to-end — central's handlers were untouched by the migration — so this mapper is
|
||||
/// the only place the two representations meet, and a field that does not survive a
|
||||
/// round-trip here is a field the gRPC transport silently drops.
|
||||
/// </para>
|
||||
|
||||
@@ -14,7 +14,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc;
|
||||
/// <summary>
|
||||
/// Central-hosted gRPC face of the seven site→central control messages
|
||||
/// (<c>Protos/central_control.proto</c>). Decodes each request onto the SAME in-process
|
||||
/// message type the Akka <c>ClusterClient</c> path already carries, <c>Ask</c>s
|
||||
/// message type the removed Akka <c>ClusterClient</c> path used to carry, <c>Ask</c>s
|
||||
/// <see cref="Actors.CentralCommunicationActor"/>, and encodes the reply back.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
@@ -26,8 +26,8 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc;
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Zero handler logic lives here.</b> Every RPC lands on a receive
|
||||
/// <c>CentralCommunicationActor</c> already implements for the ClusterClient path, so the two
|
||||
/// transports cannot drift in behaviour: the actor is the single implementation, and this class
|
||||
/// <c>CentralCommunicationActor</c> already implements, so transport and handler cannot drift
|
||||
/// in behaviour: the actor is the single implementation, and this class
|
||||
/// is a codec plus an <c>Ask</c>. That is also why the service takes the actor through
|
||||
/// <see cref="SetReady"/> rather than resolving anything from DI — the actor is created by the
|
||||
/// host's Akka bootstrap, not by the container.
|
||||
@@ -36,8 +36,8 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc;
|
||||
/// <b>Fault semantics deliberately differ from <see cref="SiteStreamGrpcServer"/>'s ingest
|
||||
/// RPCs.</b> That server answers a failed audit ingest with an EMPTY <c>IngestAck</c>; this one
|
||||
/// fails the call with a non-OK status. Both leave the site's rows <c>Pending</c> for the next
|
||||
/// drain, so the outcome is the same — but this service replaces the ClusterClient path, whose
|
||||
/// documented behaviour is to propagate the fault (<c>CentralCommunicationActor</c>'s
|
||||
/// drain, so the outcome is the same — but this service replaced the ClusterClient path, whose
|
||||
/// documented behaviour was to propagate the fault (<c>CentralCommunicationActor</c>'s
|
||||
/// <c>HandleIngestAuditEvents</c> pipes a <c>Status.Failure</c> back), and preserving that keeps
|
||||
/// a lost batch visible as a failure rather than as a successful call that acked nothing.
|
||||
/// </para>
|
||||
@@ -94,8 +94,8 @@ public sealed class CentralControlGrpcService : CentralControlService.CentralCon
|
||||
/// actor exists and can receive, NOT that every downstream singleton proxy
|
||||
/// (<c>notification-outbox</c>, <c>audit-log-ingest</c>) has registered itself yet. Those
|
||||
/// register moments later in the same startup path, and the actor already answers a call
|
||||
/// that beats them with the same "not available, retry" reply it gives on the ClusterClient
|
||||
/// path — so gating readiness on them would add nothing but a longer window in which sites
|
||||
/// that beats them with a "not available, retry" reply
|
||||
/// — so gating readiness on them would add nothing but a longer window in which sites
|
||||
/// see <see cref="StatusCode.Unavailable"/>.
|
||||
/// </remarks>
|
||||
/// <param name="centralCommunicationActor">The central communication actor.</param>
|
||||
|
||||
@@ -11,8 +11,8 @@ using AkkaStatus = Akka.Actor.Status;
|
||||
namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc;
|
||||
|
||||
/// <summary>
|
||||
/// The <see cref="ICentralTransport"/> that carries the seven site→central sends over gRPC — the
|
||||
/// migration target for the Akka <c>ClusterClient</c> path. Each method encodes the message with
|
||||
/// The <see cref="ICentralTransport"/> that carries the seven site→central sends over gRPC —
|
||||
/// the replacement for the removed Akka <c>ClusterClient</c> path. Each method encodes the message with
|
||||
/// <see cref="CentralControlDtoMapper"/>, dials <c>CentralControlService</c> through the sticky
|
||||
/// <see cref="CentralChannelProvider"/>, and delivers the decoded reply (or a transient-failure
|
||||
/// signal) to the waiting Ask.
|
||||
|
||||
@@ -81,7 +81,7 @@ public sealed class GrpcSiteTransport : ISiteCommandTransport
|
||||
}
|
||||
catch (SiteChannelUnavailableException)
|
||||
{
|
||||
// Parity with the Akka "no ClusterClient for site" path: warn and drop, so the caller's
|
||||
// Parity with the removed Akka "no ClusterClient for site" path: warn and drop, so the caller's
|
||||
// Ask times out. Central never buffers.
|
||||
_logger.LogWarning(
|
||||
"No gRPC channel for site {SiteId}; dropping {Message} (caller's Ask will time out)",
|
||||
|
||||
@@ -22,7 +22,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc;
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Coexistence — server-side only.</b> This makes the site ALSO listen on gRPC for commands;
|
||||
/// nothing central flips to gRPC here (that is T1B.3). Central still dials sites via ClusterClient.
|
||||
/// this was the server half, landed before central was flipped to dial it (T1B.3).
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Fire-and-forget.</b> The Akka path never acks <c>UnsubscribeDebugView</c>, but a unary RPC
|
||||
|
||||
@@ -9,7 +9,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc;
|
||||
/// <summary>
|
||||
/// Raised when a site has no usable gRPC channel — an unknown site, or a site with neither
|
||||
/// <c>GrpcNodeAAddress</c> nor <c>GrpcNodeBAddress</c> configured. The gRPC transport treats this
|
||||
/// the way the Akka path treats "no ClusterClient for site": warn and drop, so the caller's Ask
|
||||
/// the way the removed Akka path treated "no ClusterClient for site": warn and drop, so the caller's Ask
|
||||
/// times out (central never buffers).
|
||||
/// </summary>
|
||||
public sealed class SiteChannelUnavailableException(string siteId)
|
||||
|
||||
@@ -17,7 +17,7 @@ import "Protos/sitestream.proto";
|
||||
//
|
||||
// Direction: SITE is the client, CENTRAL is the server — the inverse of
|
||||
// SiteStreamService, where central dials the site. That asymmetry is deliberate
|
||||
// and mirrors the direction the Akka ClusterClient traffic flows today: these
|
||||
// and mirrors the direction the Akka ClusterClient traffic used to flow: these
|
||||
// seven calls are exactly the seven messages SiteCommunicationActor sends to
|
||||
// /user/central-communication.
|
||||
//
|
||||
|
||||
@@ -8,8 +8,8 @@ namespace ZB.MOM.WW.ScadaBridge.Communication;
|
||||
|
||||
/// <summary>
|
||||
/// Central-side startup-reconciliation handler. A site node, on startup, reports its
|
||||
/// local deployed inventory via <see cref="ReconcileSiteRequest"/> (delivered over
|
||||
/// ClusterClient to <see cref="Actors.CentralCommunicationActor"/>); this service diffs
|
||||
/// local deployed inventory via <see cref="ReconcileSiteRequest"/> (delivered over the
|
||||
/// control plane to <see cref="Actors.CentralCommunicationActor"/>); this service diffs
|
||||
/// it against central's expected deployed set and replies with fresh fetch tokens for the
|
||||
/// gap — instances the node is missing or has at a stale revision — plus the orphan names
|
||||
/// (present locally but no longer deployed centrally, which the node only logs).
|
||||
|
||||
@@ -4,7 +4,7 @@ namespace ZB.MOM.WW.ScadaBridge.HealthMonitoring;
|
||||
|
||||
/// <summary>
|
||||
/// Abstraction for sending health reports to central.
|
||||
/// In production, implemented over Akka ClusterClient with an acked round-trip
|
||||
/// In production, implemented over the gRPC central transport with an acked round-trip
|
||||
/// (review 01 [Medium]): the transport Asks and awaits a
|
||||
/// <see cref="SiteHealthReportAck"/>, so a lost/rejected report surfaces as a
|
||||
/// fault the sender can observe (and its per-interval counter-restore fires).
|
||||
|
||||
@@ -556,7 +556,7 @@ akka {{
|
||||
_logger);
|
||||
|
||||
// Hand the audit-ingest proxy to the CentralCommunicationActor so audit
|
||||
// ingest commands forwarded by sites over ClusterClient are routed to the
|
||||
// ingest commands forwarded by sites are routed to the
|
||||
// singleton. Mirrors the RegisterNotificationOutbox wiring above.
|
||||
centralCommActor.Tell(new RegisterAuditIngest(auditIngest.Proxy));
|
||||
|
||||
@@ -628,7 +628,7 @@ akka {{
|
||||
|
||||
// Hand the CentralCommunicationActor to the SiteCallAudit
|
||||
// actor so it can relay operator Retry/Discard on parked cached calls to
|
||||
// the owning site (over the per-site ClusterClient via SiteEnvelope).
|
||||
// the owning site (over the per-site gRPC transport via SiteEnvelope).
|
||||
// Mirrors the RegisterAuditIngest / RegisterNotificationOutbox wiring;
|
||||
// the message is sent to the singleton proxy so it reaches whichever
|
||||
// central node currently hosts the singleton.
|
||||
@@ -896,7 +896,7 @@ akka {{
|
||||
// Event log handler — cluster singleton so queries always reach the
|
||||
// active node. The event log is node-local SQLite and is not
|
||||
// replicated; only the active node records events. A per-node handler
|
||||
// would let a ClusterClient query land on the standby and find nothing.
|
||||
// would let a central query land on the standby and find nothing.
|
||||
var eventLogQueryService = _serviceProvider.GetService<SiteEventLogging.IEventLogQueryService>();
|
||||
if (eventLogQueryService != null)
|
||||
{
|
||||
@@ -1048,23 +1048,23 @@ akka {{
|
||||
// collaborators through its constructor, so we resolve them from DI
|
||||
// and pass them in via Props.Create rather than relying on a future
|
||||
// FactoryProvider. The real site→central client is constructed and
|
||||
// wired immediately below: a ClusterClientSiteAuditClient (ClusterClient
|
||||
// transport, not gRPC) replaces the DI-default NoOpSiteStreamAuditClient
|
||||
// wired immediately below: a SiteCommunicationAuditClient (routing through
|
||||
// the SiteCommunicationActor's central transport) replaces the DI-default NoOpSiteStreamAuditClient
|
||||
// for site roles, without disturbing the rest of this wiring.
|
||||
var siteAuditOptions = _serviceProvider
|
||||
.GetRequiredService<IOptions<ZB.MOM.WW.ScadaBridge.AuditLog.Site.Telemetry.SiteAuditTelemetryOptions>>();
|
||||
var siteAuditQueue = _serviceProvider
|
||||
.GetRequiredService<ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services.ISiteAuditQueue>();
|
||||
// Audit Log follow-up: the production site→central audit
|
||||
// push uses the ClusterClient transport via the SiteCommunicationActor,
|
||||
// push routes through the SiteCommunicationActor's central transport,
|
||||
// not the DI-resolved NoOpSiteStreamAuditClient. The NoOp default stays
|
||||
// correct for central/test composition roots (no SiteCommunicationActor);
|
||||
// a site role wires the real ClusterClient-based client here so the
|
||||
// a site role wires the real client here so the
|
||||
// SQLite Pending backlog actually drains to central. The forward Ask
|
||||
// reuses NotificationForwardTimeout — the same site→central command
|
||||
// forward bound notifications already use over this transport.
|
||||
ZB.MOM.WW.ScadaBridge.AuditLog.Site.Telemetry.ISiteStreamAuditClient siteAuditClient =
|
||||
new ZB.MOM.WW.ScadaBridge.AuditLog.Site.Telemetry.ClusterClientSiteAuditClient(
|
||||
new ZB.MOM.WW.ScadaBridge.AuditLog.Site.Telemetry.SiteCommunicationAuditClient(
|
||||
siteCommActor,
|
||||
_communicationOptions.NotificationForwardTimeout);
|
||||
var siteAuditLogger = _serviceProvider.GetRequiredService<ILoggerFactory>()
|
||||
@@ -1100,10 +1100,10 @@ akka {{
|
||||
// SetReady asserts a deliberately narrow contract. By this point the
|
||||
// actor system exists, SiteStreamManager.Initialize has run, and every
|
||||
// role actor (SiteCommunicationActor, deployment-manager singleton,
|
||||
// the ClusterClient) has been created with ActorOf —
|
||||
// the central transport) has been created with ActorOf —
|
||||
// creation and the registration Tells are synchronous and strictly ordered.
|
||||
// What is NOT guaranteed is completion of each actor's PreStart or the
|
||||
// ClusterClient's initial-contact handshake with central: those are
|
||||
// the central transport's initial connection to central: those are
|
||||
// intentionally asynchronous. Gating readiness on the central handshake would
|
||||
// be wrong — a site must come up and stream locally even while central is
|
||||
// briefly unreachable. gRPC readiness therefore guarantees "the site actor
|
||||
|
||||
@@ -160,7 +160,7 @@ try
|
||||
// which would let DI hand the instance back and bypass Grpc.AspNetCore's own activation
|
||||
// path (the shape that once hid a two-public-constructor defect until the rig caught it).
|
||||
// CentralControlGrpcService decodes each request onto the same in-process message the
|
||||
// ClusterClient path carries and Asks CentralCommunicationActor — zero handler logic here.
|
||||
// removed ClusterClient path used to carry, and Asks CentralCommunicationActor — zero handler logic here.
|
||||
builder.Services.AddGrpc(options =>
|
||||
{
|
||||
options.Interceptors.Add<CentralControlAuthInterceptor>();
|
||||
@@ -638,7 +638,7 @@ try
|
||||
builder.Services.AddSingleton<ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamGrpcServer>();
|
||||
// Site command plane (central→site) — the gRPC peer of SiteStreamGrpcServer. Both share
|
||||
// the h2c listener and the ControlPlaneAuthInterceptor PSK gate; both are readiness-gated
|
||||
// (SetReady, below). Central still dials over ClusterClient until T1B.3.
|
||||
// (SetReady, below).
|
||||
builder.Services.AddSingleton<ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteCommandGrpcService>();
|
||||
|
||||
// Existing site service registrations (this is also where LocalDb and its
|
||||
|
||||
@@ -388,7 +388,7 @@ public class RouteTarget
|
||||
|
||||
/// <summary>
|
||||
/// N3: awaits a routed Ask and TYPES unreachability. AskTimeoutException means
|
||||
/// the site never answered (no ClusterClient contact, site down, partition) —
|
||||
/// the site never answered (no route to the site, site down, partition) —
|
||||
/// the communication layer's own typed signal (CentralCommunicationActor drops
|
||||
/// envelopes for contactless sites so the Ask expires caller-side) — and
|
||||
/// surfaces as SiteUnreachableException so the executor maps it to the spec's
|
||||
|
||||
@@ -38,7 +38,7 @@ using ZB.MOM.WW.ScadaBridge.TemplateEngine.Services;
|
||||
namespace ZB.MOM.WW.ScadaBridge.ManagementService;
|
||||
|
||||
/// <summary>
|
||||
/// Central actor that handles all management commands from the CLI (via ClusterClient).
|
||||
/// Central actor that handles all management commands from the CLI (via the HTTP /management endpoint).
|
||||
/// Receives ManagementEnvelope messages, authorizes based on roles, then delegates to
|
||||
/// the appropriate service or repository using scoped DI.
|
||||
/// </summary>
|
||||
@@ -477,7 +477,7 @@ public class ManagementActor : ReceiveActor
|
||||
// NOTE: ResolveRolesCommand is intentionally NOT dispatched. The two-step
|
||||
// "ResolveRoles + command" flow is retired — the HTTP endpoint performs LDAP
|
||||
// auth and role resolution itself before sending a single envelope. Leaving a
|
||||
// handler would expose role-mapping data to any raw ClusterClient sender with
|
||||
// handler would expose role-mapping data to any raw in-cluster sender with
|
||||
// no role requirement; the command now falls through to the default below.
|
||||
_ => throw new NotSupportedException($"Unknown command type: {command.GetType().Name}")
|
||||
};
|
||||
|
||||
@@ -164,7 +164,7 @@ public class SiteCallAuditActor : ReceiveActor
|
||||
/// <summary>
|
||||
/// The central→site command transport — the
|
||||
/// <c>CentralCommunicationActor</c>, which owns the per-site
|
||||
/// <c>ClusterClient</c> map and routes a <see cref="SiteEnvelope"/> to the
|
||||
/// transport map and routes a <see cref="SiteEnvelope"/> to the
|
||||
/// owning site. Set via <see cref="RegisterCentralCommunication"/> by the
|
||||
/// Host after both actors exist (this actor is a cluster singleton; the
|
||||
/// transport actor is created separately). Null until registration
|
||||
@@ -1079,7 +1079,7 @@ public class SiteCallAuditActor : ReceiveActor
|
||||
/// <c>SiteCalls</c> mirror row. It wraps a <see cref="RetryParkedOperation"/>
|
||||
/// in a <see cref="SiteEnvelope"/> addressed to <c>SourceSite</c>, Asks the
|
||||
/// <c>CentralCommunicationActor</c> (which routes it over the per-site
|
||||
/// <c>ClusterClient</c>), and maps the site's
|
||||
/// gRPC transport), and maps the site's
|
||||
/// <see cref="ParkedOperationActionAck"/> — or an Ask timeout — onto a
|
||||
/// <see cref="RetrySiteCallResponse"/>. A timeout / no-route is reported as
|
||||
/// the distinct <see cref="SiteCallRelayOutcome.SiteUnreachable"/> outcome,
|
||||
|
||||
@@ -30,7 +30,7 @@ public class SiteCallAuditOptions
|
||||
/// Ask timeout for the central→site Retry/Discard relay. When
|
||||
/// the owning site does not ack a <c>RetryParkedOperation</c> /
|
||||
/// <c>DiscardParkedOperation</c> within this window — site offline, no
|
||||
/// ClusterClient route, or central buffering deliberately absent — the relay
|
||||
/// route to the site, or central buffering deliberately absent — the relay
|
||||
/// reports a <c>SiteUnreachable</c> outcome. Default 10 seconds: long enough
|
||||
/// to absorb a healthy cross-cluster round-trip, short enough that an
|
||||
/// operator clicking Retry on an offline site gets a fast, honest answer.
|
||||
|
||||
@@ -27,8 +27,8 @@ namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors;
|
||||
/// crashes on these failures.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// The pass runs after a small startup delay (so the central ClusterClient has time to
|
||||
/// register) and is driven entirely off the actor thread: the Ask + fetch + write happen in
|
||||
/// The pass runs after a small startup delay (so the site's central transport has time to
|
||||
/// connect) and is driven entirely off the actor thread: the Ask + fetch + write happen in
|
||||
/// an awaited continuation whose summary is captured in an internal message
|
||||
/// <see cref="ReconcilePassResult"/> piped back to <c>Self</c>. The actor thread never blocks.
|
||||
/// </para>
|
||||
@@ -60,14 +60,14 @@ public sealed class SiteReconciliationActor : ReceiveActor, IWithTimers
|
||||
/// <param name="configFetcher">Fetches a deployment's flattened config JSON from central over HTTP.</param>
|
||||
/// <param name="siteCommunicationActor">
|
||||
/// The site's <c>SiteCommunicationActor</c>; it forwards the
|
||||
/// <see cref="ReconcileSiteRequest"/> over the registered central ClusterClient and routes
|
||||
/// <see cref="ReconcileSiteRequest"/> over the site's central transport and routes
|
||||
/// the <see cref="ReconcileSiteResponse"/> back to this actor's Ask.
|
||||
/// </param>
|
||||
/// <param name="siteIdentifier">This node's site identifier (resolved by central).</param>
|
||||
/// <param name="nodeId">This node's semantic id (e.g. <c>node-a</c>/<c>node-b</c>), for logging/diagnostics.</param>
|
||||
/// <param name="logger">Logger.</param>
|
||||
/// <param name="initialDelay">
|
||||
/// Delay before the single startup pass, giving the central ClusterClient time to register.
|
||||
/// Delay before the single startup pass, giving the site's central transport time to connect.
|
||||
/// Defaults to 5 seconds.
|
||||
/// </param>
|
||||
/// <param name="askTimeout">Round-trip timeout for the reconcile Ask to central. Defaults to 30 seconds.</param>
|
||||
@@ -105,7 +105,7 @@ public sealed class SiteReconciliationActor : ReceiveActor, IWithTimers
|
||||
protected override void PreStart()
|
||||
{
|
||||
base.PreStart();
|
||||
// One-shot pass after a small delay so the central ClusterClient can register first.
|
||||
// One-shot pass after a small delay so the site's central transport can connect first.
|
||||
// Non-blocking: the timer fires RunReconcile back onto this actor's mailbox.
|
||||
Timers.StartSingleTimer(StartupTimerKey, RunReconcile.Instance, _initialDelay);
|
||||
_logger.LogInformation(
|
||||
|
||||
@@ -74,7 +74,7 @@ public class ScriptRuntimeContext
|
||||
|
||||
/// <summary>
|
||||
/// Notification Outbox: the site communication actor that <c>Notify.Status</c>
|
||||
/// queries central through (via the ClusterClient command/control transport).
|
||||
/// queries central through (via the gRPC command/control transport).
|
||||
/// </summary>
|
||||
private readonly ICanTell? _siteCommunicationActor;
|
||||
|
||||
@@ -175,7 +175,7 @@ public class ScriptRuntimeContext
|
||||
/// <param name="externalSystemClient">Optional client for external system API calls.</param>
|
||||
/// <param name="databaseGateway">Optional gateway for database connection and cached write access.</param>
|
||||
/// <param name="storeAndForward">Optional store-and-forward service for notification delivery.</param>
|
||||
/// <param name="siteCommunicationActor">Optional actor for site-to-central communication (ClusterClient).</param>
|
||||
/// <param name="siteCommunicationActor">Optional actor for site-to-central communication.</param>
|
||||
/// <param name="siteId">Identifier of the site where this instance is running.</param>
|
||||
/// <param name="sourceScript">Optional name of the source script for audit trail identification.</param>
|
||||
/// <param name="auditWriter">Optional writer for audit log entries.</param>
|
||||
|
||||
@@ -26,10 +26,10 @@ namespace ZB.MOM.WW.ScadaBridge.StoreAndForward;
|
||||
/// for operator forensics.</description></item>
|
||||
/// </list>
|
||||
///
|
||||
/// The forward travels over the ClusterClient command/control transport: the handler
|
||||
/// The forward travels over the gRPC command/control transport: the handler
|
||||
/// <see cref="ActorRefImplicitSenderExtensions.Ask{T}(ICanTell, object, TimeSpan?)">Asks</see>
|
||||
/// the site communication actor, which wraps the message in a
|
||||
/// <c>ClusterClient.Send("/user/central-communication", …)</c> and routes central's
|
||||
/// <c>SubmitNotification</c> call on <c>CentralControlService</c> and routes central's
|
||||
/// reply straight back to this Ask.
|
||||
/// </summary>
|
||||
public sealed class NotificationForwarder
|
||||
@@ -45,7 +45,7 @@ public sealed class NotificationForwarder
|
||||
/// </summary>
|
||||
/// <param name="siteCommunicationActor">
|
||||
/// The site communication actor. It forwards a <see cref="NotificationSubmit"/> to
|
||||
/// central via the registered ClusterClient and replies with the
|
||||
/// central over the site's central transport and replies with the
|
||||
/// <see cref="NotificationSubmitAck"/>.
|
||||
/// </param>
|
||||
/// <param name="sourceSiteId">This site's identifier, stamped on every submit.</param>
|
||||
|
||||
+10
-10
@@ -11,13 +11,13 @@ using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
|
||||
namespace ZB.MOM.WW.ScadaBridge.AuditLog.Tests.Site.Telemetry;
|
||||
|
||||
/// <summary>
|
||||
/// Tests for <see cref="ClusterClientSiteAuditClient"/> — the production
|
||||
/// Tests for <see cref="SiteCommunicationAuditClient"/> — the production
|
||||
/// <see cref="ISiteStreamAuditClient"/> binding wired by the Host for site
|
||||
/// roles. The client maps the proto-DTO batches produced by
|
||||
/// <see cref="SiteAuditTelemetryActor"/> into the Akka
|
||||
/// <see cref="IngestAuditEventsCommand"/> / <see cref="IngestCachedTelemetryCommand"/>
|
||||
/// messages, Asks the site's <c>SiteCommunicationActor</c> (which forwards over
|
||||
/// ClusterClient to central), and maps the reply back into an
|
||||
/// messages, Asks the site's <c>SiteCommunicationActor</c> (which forwards to
|
||||
/// central over its central transport), and maps the reply back into an
|
||||
/// <see cref="IngestAck"/>.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
@@ -25,7 +25,7 @@ namespace ZB.MOM.WW.ScadaBridge.AuditLog.Tests.Site.Telemetry;
|
||||
/// it lets the tests assert the exact command shape AND drive the reply (or
|
||||
/// withhold one to exercise the Ask-timeout path).
|
||||
/// </remarks>
|
||||
public class ClusterClientSiteAuditClientTests : TestKit
|
||||
public class SiteCommunicationAuditClientTests : TestKit
|
||||
{
|
||||
/// <summary>Short Ask timeout so the timeout test completes quickly.</summary>
|
||||
private static readonly TimeSpan AskTimeout = TimeSpan.FromMilliseconds(500);
|
||||
@@ -65,7 +65,7 @@ public class ClusterClientSiteAuditClientTests : TestKit
|
||||
public async Task IngestAuditEventsAsync_FullAck_MapsAllAcceptedIdsOntoAck()
|
||||
{
|
||||
var probe = CreateTestProbe();
|
||||
var sut = new ClusterClientSiteAuditClient(probe.Ref, AskTimeout);
|
||||
var sut = new SiteCommunicationAuditClient(probe.Ref, AskTimeout);
|
||||
|
||||
var events = Enumerable.Range(0, 3).Select(_ => NewEvent()).ToList();
|
||||
var batch = BatchOf(events);
|
||||
@@ -92,7 +92,7 @@ public class ClusterClientSiteAuditClientTests : TestKit
|
||||
public async Task IngestAuditEventsAsync_PartialAck_OnlyAcceptedIdsAppearOnAck()
|
||||
{
|
||||
var probe = CreateTestProbe();
|
||||
var sut = new ClusterClientSiteAuditClient(probe.Ref, AskTimeout);
|
||||
var sut = new SiteCommunicationAuditClient(probe.Ref, AskTimeout);
|
||||
|
||||
var events = Enumerable.Range(0, 5).Select(_ => NewEvent()).ToList();
|
||||
var accepted = events.Take(3).Select(e => e.EventId).ToList();
|
||||
@@ -114,7 +114,7 @@ public class ClusterClientSiteAuditClientTests : TestKit
|
||||
public async Task IngestAuditEventsAsync_AskTimeout_Throws_SoDrainLoopKeepsRowsPending()
|
||||
{
|
||||
var probe = CreateTestProbe();
|
||||
var sut = new ClusterClientSiteAuditClient(probe.Ref, AskTimeout);
|
||||
var sut = new SiteCommunicationAuditClient(probe.Ref, AskTimeout);
|
||||
|
||||
var batch = BatchOf(new[] { NewEvent() });
|
||||
|
||||
@@ -131,7 +131,7 @@ public class ClusterClientSiteAuditClientTests : TestKit
|
||||
public async Task IngestAuditEventsAsync_FaultedReply_Throws()
|
||||
{
|
||||
var probe = CreateTestProbe();
|
||||
var sut = new ClusterClientSiteAuditClient(probe.Ref, AskTimeout);
|
||||
var sut = new SiteCommunicationAuditClient(probe.Ref, AskTimeout);
|
||||
|
||||
var task = sut.IngestAuditEventsAsync(BatchOf(new[] { NewEvent() }), CancellationToken.None);
|
||||
probe.ExpectMsg<IngestAuditEventsCommand>(TimeSpan.FromSeconds(3));
|
||||
@@ -147,7 +147,7 @@ public class ClusterClientSiteAuditClientTests : TestKit
|
||||
public async Task IngestCachedTelemetryAsync_RoutesCommand_AndMapsReply()
|
||||
{
|
||||
var probe = CreateTestProbe();
|
||||
var sut = new ClusterClientSiteAuditClient(probe.Ref, AskTimeout);
|
||||
var sut = new SiteCommunicationAuditClient(probe.Ref, AskTimeout);
|
||||
|
||||
var events = Enumerable.Range(0, 2).Select(_ => NewEvent()).ToList();
|
||||
var batch = new CachedTelemetryBatch();
|
||||
@@ -182,7 +182,7 @@ public class ClusterClientSiteAuditClientTests : TestKit
|
||||
public async Task IngestCachedTelemetryAsync_AskTimeout_Throws()
|
||||
{
|
||||
var probe = CreateTestProbe();
|
||||
var sut = new ClusterClientSiteAuditClient(probe.Ref, AskTimeout);
|
||||
var sut = new SiteCommunicationAuditClient(probe.Ref, AskTimeout);
|
||||
|
||||
var batch = new CachedTelemetryBatch();
|
||||
batch.Packets.Add(new CachedTelemetryPacket
|
||||
@@ -28,7 +28,7 @@ namespace ZB.MOM.WW.ScadaBridge.CentralUI.PlaywrightTests.Deployment;
|
||||
/// <b>Tolerant terminal-state assertion (validation-behavior protocol):</b> whether
|
||||
/// <c>Connect</c> reaches a <b>Live</b> badge depends on the owning site returning a
|
||||
/// snapshot for the instance. Reading <c>DebugView.razor::Connect</c>: it awaits
|
||||
/// <c>DebugStreamService.StartStreamAsync</c> (a ClusterClient snapshot round-trip);
|
||||
/// <c>DebugStreamService.StartStreamAsync</c> (a site snapshot round-trip);
|
||||
/// on success it flips <c>_connected = true</c> (Live badge + success toast), and on
|
||||
/// any exception it surfaces a <c>Connect failed: …</c> error <c>.toast</c>. Either
|
||||
/// way <c>_connecting</c> resets, so the click always resolves — it never hangs. The
|
||||
|
||||
+3
-3
@@ -19,13 +19,13 @@ namespace ZB.MOM.WW.ScadaBridge.CentralUI.PlaywrightTests.Deployment;
|
||||
/// and would never appear, so the test mints an instance via the shared
|
||||
/// <see cref="DeploymentFixture"/> and drives it to <c>Enabled</c> over the CLI
|
||||
/// (<c>instance deploy</c>) before selecting it. The fixture provisions onto the
|
||||
/// real, running <c>site-a</c> so the ClusterClient snapshot round-trip resolves
|
||||
/// fast (an unknown site has no registered ClusterClient and only times out).
|
||||
/// real, running <c>site-a</c> so the snapshot round-trip resolves
|
||||
/// fast (an unknown site has no reachable gRPC endpoint and only times out).
|
||||
/// </para>
|
||||
///
|
||||
/// <para>
|
||||
/// <b>Tolerant terminal-state wait, then the tabs/trees assertions</b> — Connect
|
||||
/// awaits a ClusterClient snapshot round-trip; on success the page flips to the
|
||||
/// awaits a site snapshot round-trip; on success the page flips to the
|
||||
/// Live badge and renders the tabbed card, on failure it surfaces a
|
||||
/// <c>Connect failed: …</c> <c>.toast</c> (see <c>DebugView.razor::Connect</c>).
|
||||
/// The test first waits on a TERMINAL state (Live badge OR error toast) within a
|
||||
|
||||
+2
-2
@@ -11,8 +11,8 @@ namespace ZB.MOM.WW.ScadaBridge.CentralUI.PlaywrightTests.Deployment;
|
||||
///
|
||||
/// <para>
|
||||
/// <b>Why <c>site-a</c> and not a throwaway site:</b> the deploy / enable / disable
|
||||
/// actions relay to the owning site over the Akka <c>ClusterClient</c>. An unknown
|
||||
/// site identifier has no registered <c>ClusterClient</c>, so the relay only
|
||||
/// actions relay to the owning site over the gRPC site command plane. An unknown
|
||||
/// site identifier has no reachable gRPC endpoint, so the relay only
|
||||
/// resolves on a slow 10-second timeout (and never surfaces a fast failure toast).
|
||||
/// To exercise the real action path, the ephemeral instances must therefore live
|
||||
/// on a <em>real, running</em> site — <c>site-a</c> — rather than a fixture-created
|
||||
|
||||
+3
-3
@@ -282,8 +282,8 @@ public class SiteCallsPageTests
|
||||
// be relayed to the owning site. Unlike the display-only tests above,
|
||||
// this one actually relays to the owning site, so the SourceSite must
|
||||
// be a *real* site identifier from the running cluster (site-a) and
|
||||
// not the cosmetic "plant-a" label: an unknown site has no registered
|
||||
// ClusterClient, so CentralCommunicationActor drops the envelope
|
||||
// not the cosmetic "plant-a" label: an unknown site has no reachable
|
||||
// gRPC endpoint, so CentralCommunicationActor drops the envelope
|
||||
// without replying and the relay only resolves on the 10s inner Ask
|
||||
// timeout — too slow for the toast assertion below. Relayed to a live
|
||||
// site, the site finds no parked S&F message for this freshly-seeded
|
||||
@@ -319,7 +319,7 @@ public class SiteCallsPageTests
|
||||
// We only assert that an outcome toast appears (exactly one — the
|
||||
// single-toast contract), not which one, since the live cluster
|
||||
// state determines the outcome. The wait is generous (15s): the
|
||||
// relay round-trips to the site over ClusterClient, and a worst-case
|
||||
// relay round-trips to the site over gRPC, and a worst-case
|
||||
// path can sit on the 10s inner relay timeout before the response —
|
||||
// and the toast itself auto-dismisses 5s after it appears, so the
|
||||
// assertion must catch it inside that window.
|
||||
|
||||
+4
-4
@@ -8,10 +8,10 @@ using ZB.MOM.WW.ScadaBridge.Commons.Messages.Streaming;
|
||||
namespace ZB.MOM.WW.ScadaBridge.Commons.Tests.Messages;
|
||||
|
||||
/// <summary>
|
||||
/// Contract-lock tests for the highest-traffic ClusterClient record contracts.
|
||||
/// ClusterClient traffic uses Akka.NET's default Newtonsoft-JSON serializer and the
|
||||
/// design supports system-wide artifact version skew across sites, so these records
|
||||
/// must obey additive-only evolution:
|
||||
/// Contract-lock tests for the highest-traffic cross-boundary record contracts.
|
||||
/// These predate the ClusterClient→gRPC migration, when this traffic rode Akka.NET's
|
||||
/// default Newtonsoft-JSON serializer; the design still supports system-wide artifact
|
||||
/// version skew across sites, so these records must still obey additive-only evolution:
|
||||
/// (a) an old-shape payload (missing newer trailing optional fields) still
|
||||
/// deserializes, with the missing fields taking their constructor defaults;
|
||||
/// (b) an unknown field from a *newer* peer is ignored (never throws);
|
||||
|
||||
+1
-1
@@ -94,7 +94,7 @@ public class ManagementCommandRegistryTests
|
||||
|
||||
/// <summary>
|
||||
/// M9-T32c: the schema-library CRUD commands must auto-register by reflection
|
||||
/// (no manual registry entry) so the HTTP / ClusterClient boundary can route them.
|
||||
/// (no manual registry entry) so the HTTP management boundary can route them.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void Resolve_SchemaLibraryCommands_AllRegistered()
|
||||
|
||||
+1
-1
@@ -104,7 +104,7 @@ public class SiteCommunicationActorTransportTests : TestKit
|
||||
// The seam preserves the reply-routing the S&F layer depends on: when the transport
|
||||
// answers the captured replyTo with a Status.Failure (its transient-failure signal on a
|
||||
// non-OK status), that failure reaches the original sender — here the test actor — so the
|
||||
// waiting Ask faults, exactly as it did on the ClusterClient path.
|
||||
// waiting Ask faults, exactly as it did on the removed ClusterClient path.
|
||||
var transport = Substitute.For<ICentralTransport>();
|
||||
transport
|
||||
.When(t => t.SubmitNotification(Arg.Any<NotificationSubmit>(), Arg.Any<IActorRef>()))
|
||||
|
||||
+3
-3
@@ -15,10 +15,10 @@ using ZB.MOM.WW.ScadaBridge.Communication.Actors;
|
||||
namespace ZB.MOM.WW.ScadaBridge.Communication.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Tests for the Audit Log (#23) site→central ClusterClient ingest routing on
|
||||
/// <see cref="CentralCommunicationActor"/>. A site ClusterClient delivers
|
||||
/// Tests for the Audit Log (#23) site→central ingest routing on
|
||||
/// <see cref="CentralCommunicationActor"/>. A site delivers
|
||||
/// <see cref="IngestAuditEventsCommand"/> / <see cref="IngestCachedTelemetryCommand"/>
|
||||
/// to the receptionist-registered actor, which forwards to the registered
|
||||
/// to the actor, which forwards to the registered
|
||||
/// <c>AuditLogIngestActor</c> proxy and routes the reply back to the site.
|
||||
/// Mirrors the NotificationSubmit / RegisterNotificationOutbox pattern.
|
||||
/// </summary>
|
||||
|
||||
+1
-1
@@ -17,7 +17,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Tests;
|
||||
/// Tests that <see cref="CentralCommunicationActor"/> routes a site→central
|
||||
/// <see cref="ReconcileSiteRequest"/> through the scoped <see cref="ReconcileService"/>
|
||||
/// and pipes the resulting <see cref="ReconcileSiteResponse"/> back to the original
|
||||
/// sender (the site's ClusterClient path). Mirrors the audit-ingest routing tests.
|
||||
/// sender (the site's transport path). Mirrors the audit-ingest routing tests.
|
||||
/// </summary>
|
||||
public class CentralCommunicationActorReconcileTests : TestKit
|
||||
{
|
||||
|
||||
@@ -293,7 +293,7 @@ public class DebugStreamBridgeActorTests : TestKit
|
||||
|
||||
ctx.BridgeActor.Tell(new StopDebugStream());
|
||||
|
||||
// Should send ClusterClient unsubscribe
|
||||
// Should send the site-addressed unsubscribe
|
||||
var envelope = ctx.CommProbe.ExpectMsg<SiteEnvelope>();
|
||||
Assert.IsType<UnsubscribeDebugViewRequest>(envelope.Message);
|
||||
|
||||
|
||||
@@ -60,7 +60,7 @@ public class SiteCommunicationActorTests : TestKit
|
||||
public void DeploymentStateQuery_ForwardedToDeploymentManager()
|
||||
{
|
||||
// DeploymentManager-006: the site-before-redeploy query travels over the
|
||||
// ClusterClient command/control transport and is routed to the local
|
||||
// gRPC command/control transport and is routed to the local
|
||||
// Deployment Manager, which owns the deployed-config store.
|
||||
var dmProbe = CreateTestProbe();
|
||||
var siteActor = Sys.ActorOf(Props.Create(() =>
|
||||
|
||||
@@ -24,14 +24,14 @@ namespace ZB.MOM.WW.ScadaBridge.IntegrationTests.AuditLog;
|
||||
|
||||
/// <summary>
|
||||
/// End-to-end integration test for the Audit Log (#23) site→central push path
|
||||
/// introduced by the "real ClusterClient-based site audit push client" follow-up.
|
||||
/// introduced by the "real site audit push client" follow-up.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// Exercises the full production chain in one actor system: the real
|
||||
/// <see cref="SqliteAuditWriter"/> site SQLite hot-path, the real
|
||||
/// <see cref="SiteAuditTelemetryActor"/> drain loop, the real
|
||||
/// <see cref="ClusterClientSiteAuditClient"/>, the real
|
||||
/// <see cref="SiteCommunicationAuditClient"/>, the real
|
||||
/// <see cref="SiteCommunicationActor"/> forward, the real
|
||||
/// <see cref="CentralCommunicationActor"/> routing, and the real
|
||||
/// <c>AuditLogIngestActor</c> ingest — only the cross-cluster gRPC transport itself is
|
||||
@@ -180,7 +180,7 @@ public class SiteAuditPushFlowTests : TestKit
|
||||
new BridgeCentralTransport(centralCommActor))));
|
||||
|
||||
// The production site audit push client — the unit under integration.
|
||||
var auditClient = new ClusterClientSiteAuditClient(
|
||||
var auditClient = new SiteCommunicationAuditClient(
|
||||
siteCommActor, TimeSpan.FromSeconds(5));
|
||||
|
||||
// Real SiteAuditTelemetryActor drains the writer's Pending queue and
|
||||
|
||||
@@ -47,7 +47,7 @@ public class DispatchCoverageTests : TestKit, IDisposable
|
||||
{
|
||||
// The two-step "ResolveRoles + command" flow is retired — the HTTP endpoint
|
||||
// resolves roles itself and sends a single envelope. Leaving a handler would
|
||||
// expose role-mapping data to a raw ClusterClient sender with no role gate.
|
||||
// expose role-mapping data to a raw in-cluster sender with no role gate.
|
||||
// See ManagementActor.DispatchCommand (the comment above the default arm, ~line 472).
|
||||
typeof(ResolveRolesCommand),
|
||||
|
||||
|
||||
@@ -1089,7 +1089,7 @@ public class ManagementActorTests : TestKit, IDisposable
|
||||
//
|
||||
// The two-step ResolveRoles + command flow is retired: the HTTP endpoint does
|
||||
// LDAP auth and role resolution itself. The actor must no longer dispatch
|
||||
// ResolveRolesCommand — a stray ClusterClient sender hitting it gets a uniform
|
||||
// ResolveRolesCommand — a stray in-cluster sender hitting it gets a uniform
|
||||
// ManagementError rather than an unauthenticated role-mapping enumeration.
|
||||
// ========================================================================
|
||||
|
||||
|
||||
@@ -164,7 +164,7 @@ public class SiteCallRelayTests : TestKit
|
||||
public void RetrySiteCall_SiteNeverReplies_ReportsSiteUnreachable()
|
||||
{
|
||||
// A central comm probe that silently drops the relay — models an offline
|
||||
// site / no ClusterClient route: the Ask times out.
|
||||
// site / no route to the site: the Ask times out.
|
||||
var central = CreateTestProbe();
|
||||
var actor = CreateActor(central.Ref);
|
||||
|
||||
|
||||
Reference in New Issue
Block a user