feat(central): one central cache-invalidation topic; role-mapping and external-definition caches follow peer edits (ScadaBridge#100 item 3, ScadaBridge#83)

The #96 site-psk-invalidation topic was the first cross-node invalidation broadcast.
It is generalised into ONE central DistributedPubSub topic, central-cache-invalidation,
carrying typed messages (Commons ICentralCacheInvalidation: SitePskInvalidated,
RoleMappingsInvalidated, ExternalDefinitionsInvalidated, InboundApiMethodInvalidated)
in an envelope stamped with the publisher's process-unique origin id.

- Host: CentralCacheInvalidationBus (ICentralCacheInvalidationPublisher, no-op until
  the mediator is set, never throws) and CentralCacheInvalidationSubscriber, which
  skips its own node's envelopes and hands the peer's to every registered
  ICentralCacheInvalidationHandler (a failing handler does not stop the rest).
  Replaces SitePskInvalidationSubscriber.
- Migration of #96: SitePskRotationInvalidator publishes through the bus and is the
  PSK handler; SitePskInvalidated keeps its type name and is also published bare on
  site-psk-invalidation, which the subscriber still listens to, so a mixed-version
  central pair keeps exchanging PSK invalidations. Remove after one release.
- Audit D1: RoleMappingCache is a handler; ManagementActor's post-commit role-mapping
  invalidation also publishes RoleMappingsInvalidated, so a revocation is immediate on
  the peer instead of after RoleMappingCacheSeconds.
- Audit D3 / #83 (peer half): ExternalDefinitionLookupCache is a handler; the 9
  ManagementActor handlers and the Transport bundle importer publish
  ExternalDefinitionsInvalidated after commit. (#83's local import half was already in.)

TTLs remain the fallback for a lost broadcast or a direct SQL edit. Tests: bus and
subscriber unit tests (origin filter, legacy topic), peer-node tests through the real
caches in ManagementService.Tests, importer publish, composition root, and a real
two-node cluster from the production HOCON (CentralCacheInvalidationTwoNodeTests)
proving every message type crosses with the production serializer. Docs: Host,
Communication, Security, ManagementService, ESG, and the #85 cache audit.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
2026-11-09 15:50:03 -05:00
parent 063fb21815
commit 153609ef8a
27 changed files with 794 additions and 110 deletions
+1 -1
View File
@@ -163,7 +163,7 @@ if (!AuthorizationPolicies.OperationalAuditRoles.Contains(userRole))
| `JwtExpiryMinutes` | `15` | Lifetime of the bearer JWT issued by `/auth/token`. | | `JwtExpiryMinutes` | `15` | Lifetime of the bearer JWT issued by `/auth/token`. |
| `IdleTimeoutMinutes` | `30` | Cookie session idle timeout; set as the cookie `ExpireTimeSpan`. | | `IdleTimeoutMinutes` | `30` | Cookie session idle timeout; set as the cookie `ExpireTimeSpan`. |
| `RoleRefreshThresholdMinutes` | `15` | Age of the role-refresh anchor after which the cookie's roles are re-mapped from the DB (LDAP-free). | | `RoleRefreshThresholdMinutes` | `15` | Age of the role-refresh anchor after which the cookie's roles are re-mapped from the DB (LDAP-free). |
| `RoleMappingCacheSeconds` | `30` | Process-wide `RoleMappingCache` TTL (`0` disables); invalidated on the local node after mapping/scope-rule edits. | | `RoleMappingCacheSeconds` | `30` | Process-wide `RoleMappingCache` TTL (`0` disables); invalidated after mapping/scope-rule edits on the local node and, over the central cache-invalidation topic, on the peer central node (#100). |
| `RequireHttpsCookie` | `true` | When `true`, the cookie is `Secure`-only (HTTPS required). Set `false` for HTTP-only dev deployments; a warning is logged at startup. | | `RequireHttpsCookie` | `true` | When `true`, the cookie is `Secure`-only (HTTPS required). Set `false` for HTTP-only dev deployments; a warning is logged at startup. |
| `CookieName` | `ZB.MOM.WW.ScadaBridge.Auth` | Authentication cookie name. Override per deployment when two ScadaBridge stacks share a hostname — browsers scope cookies by host+path but not by port. | | `CookieName` | `ZB.MOM.WW.ScadaBridge.Auth` | Authentication cookie name. Override per deployment when two ScadaBridge stacks share a hostname — browsers scope cookies by host+path but not by port. |
@@ -23,12 +23,12 @@ node after a change is made through the other one. Same defect class as #83
|---|---|---|---|---| |---|---|---|---|---|
| `InboundScriptExecutor` compiled handlers (InboundAPI) | Compiled method delegates | `InvalidateMethod` on delete; change bus on import | **None** — every request revision-checks the cached handler against the resolved `ApiMethod.Script` and recompiles on mismatch | OK | | `InboundScriptExecutor` compiled handlers (InboundAPI) | Compiled method delegates | `InvalidateMethod` on delete; change bus on import | **None** — every request revision-checks the cached handler against the resolved `ApiMethod.Script` and recompiles on mismatch | OK |
| `ApiMethodCache` (InboundAPI) | `ApiMethod` rows, incl. not-found | Change bus (bundle import); **management create/update/delete — added by #85** | `InboundApi:ApiMethodCacheTtl`, default **5 s** | OK (local gap fixed, see below) | | `ApiMethodCache` (InboundAPI) | `ApiMethod` rows, incl. not-found | Change bus (bundle import); **management create/update/delete — added by #85** | `InboundApi:ApiMethodCacheTtl`, default **5 s** | OK (local gap fixed, see below) |
| `ExternalDefinitionLookupCache` (ESG, central only) | External systems / methods / DB connections (found only) | 9 ManagementActor handlers (#65); bundle import (#83) | `ExternalSystemGateway:DefinitionCacheTtl`, **10 s** on central | Owner decision — #83 | | `ExternalDefinitionLookupCache` (ESG, central only) | External systems / methods / DB connections (found only) | 9 ManagementActor handlers (#65); bundle import (#83) | Edit or import through the other node: **none** (`ExternalDefinitionsInvalidated` on `central-cache-invalidation`, #100); direct SQL: `ExternalSystemGateway:DefinitionCacheTtl`, **10 s** on central | Fixed — #100 item 3 (see D3) |
| `RoleMappingCache` (Security) — the cache at `ManagementActor` `InvalidateRoleMappingCacheAfterCommit` | LDAP group → role map + Deployer site scopes | Mapping / scope-rule commands, post-commit | `Security:RoleMappingCacheSeconds`, default **30 s** (`0` disables) | Owner decision — see D1 | | `RoleMappingCache` (Security) — the cache at `ManagementActor` `InvalidateRoleMappingCacheAfterCommit` | LDAP group → role map + Deployer site scopes | Mapping / scope-rule commands, post-commit | **None** (`RoleMappingsInvalidated` on `central-cache-invalidation`, #100); TTL `Security:RoleMappingCacheSeconds` (**30 s**) only if the broadcast is lost | Fixed — #100 item 3 (see D1) |
| `PendingUpdatesCache` (DeploymentManager, #91) | Fleet pending-updates snapshot | Deployment finished on this node | `DeploymentManager:PendingUpdatesCacheTtl`, default **60 s** | OK — display only | | `PendingUpdatesCache` (DeploymentManager, #91) | Fleet pending-updates snapshot | Deployment finished on this node | `DeploymentManager:PendingUpdatesCacheTtl`, default **60 s** | OK — display only |
| `CentralCommunicationActor` site-address cache (+ `GrpcSiteTransport` channel pairs) | Site gRPC endpoints | Site create/update/delete → `RefreshSiteAddresses` | Periodic DB reload, **60 s** (hard-coded) | OK — rare edit, self-heals | | `CentralCommunicationActor` site-address cache (+ `GrpcSiteTransport` channel pairs) | Site gRPC endpoints | Site create/update/delete → `RefreshSiteAddresses` | Periodic DB reload, **60 s** (hard-coded) | OK — rare edit, self-heals |
| `SiteStreamGrpcClientFactory` clients | Per (site, endpoint) stream clients | Site delete → `RemoveSiteAsync` (local only) | Not stale data: a removed site's client on the peer is reclaimed by idle eviction, `Communication:GrpcClientIdleTtl` (**30 min**) × 1.5 after last use | OK — resource, not correctness | | `SiteStreamGrpcClientFactory` clients | Per (site, endpoint) stream clients | Site delete → `RemoveSiteAsync` (local only) | Not stale data: a removed site's client on the peer is reclaimed by idle eviction, `Communication:GrpcClientIdleTtl` (**30 min**) × 1.5 after last use | OK — resource, not correctness |
| `SitePskProvider` (Host) | Per-site gRPC PSK from config / secret store | Site removal; `SB-GRPC-PSK-{site}` write via `/admin/secrets`; site auth rejection; `SitePsks` reload (#96) | Write via `/admin/secrets`: **none** (peer told over `DistributedPubSub` topic `site-psk-invalidation`); other writes: first rejected call after `Secrets:ResolveCacheTtl` | Fixed — #96 (see D2) | | `SitePskProvider` (Host) | Per-site gRPC PSK from config / secret store | Site removal; `SB-GRPC-PSK-{site}` write via `/admin/secrets`; site auth rejection; `SitePsks` reload (#96) | Write via `/admin/secrets`: **none** (peer told over `DistributedPubSub` topic `central-cache-invalidation`, formerly `site-psk-invalidation`); other writes: first rejected call after `Secrets:ResolveCacheTtl` | Fixed — #96 (see D2) |
| `DefaultSecretResolver` (shared `ZB.MOM.WW.Secrets`) | Decrypted secret values | Lib's own invalidator (UI writes, sync service) | `Secrets:ResolveCacheTtl`, **30 s** | OK — owned by the shared lib | | `DefaultSecretResolver` (shared `ZB.MOM.WW.Secrets`) | Decrypted secret values | Lib's own invalidator (UI writes, sync service) | `Secrets:ResolveCacheTtl`, **30 s** | OK — owned by the shared lib |
| `KpiSnapshotCache` (CentralUI) | KPI query results (operational data) | — | **8 s** | OK — not config | | `KpiSnapshotCache` (CentralUI) | KPI query results (operational data) | — | **8 s** | OK — not config |
| `SharedAlarmSummaryService` (CentralUI) | Per-site alarm summary fan-out | — | **12 s** cold-cache window | OK — not config | | `SharedAlarmSummaryService` (CentralUI) | Per-site alarm summary fan-out | — | **12 s** cold-cache window | OK — not config |
@@ -75,3 +75,16 @@ broadcast was added.
- **D3 — `ExternalDefinitionLookupCache` peer bound (10 s).** Already tracked by #83; the - **D3 — `ExternalDefinitionLookupCache` peer bound (10 s).** Already tracked by #83; the
recommendation there (accept and document) still stands. If D1 gets a broadcast topic, this cache recommendation there (accept and document) still stands. If D1 gets a broadcast topic, this cache
and `ApiMethodCache` should join it rather than add a second mechanism. and `ApiMethodCache` should join it rather than add a second mechanism.
## Follow-up: one central cache-invalidation topic (#100 item 3, 2026-11-09)
D1 and D3 now ride a single `DistributedPubSub` topic, `central-cache-invalidation`, with typed
messages (`ICentralCacheInvalidation`: `SitePskInvalidated`, `RoleMappingsInvalidated`,
`ExternalDefinitionsInvalidated`, `InboundApiMethodInvalidated`) instead of one topic per cache.
The #96 topic was migrated onto it: `SitePskInvalidated` travels on the shared topic, and for one
release is also published bare on `site-psk-invalidation` (still subscribed), so a mixed-version
central pair keeps working. Each envelope carries the publisher's origin id and the subscriber
skips its own. Publishers: ManagementActor post-commit (role mappings, external definitions),
the Transport bundle importer (external definitions) and the PSK invalidator. Every cache keeps its TTL / revision check as the fallback for a
lost broadcast or an out-of-band (SQL) edit. Design: Component-Host.md → *Central cache
invalidation*; tests include a real two-node cluster (`CentralCacheInvalidationTwoNodeTests`).
+3 -2
View File
@@ -207,8 +207,9 @@ site can omit it.
Central caches each resolved key, and drops the entry (Gitea #96) when: the site is removed; the Central caches each resolved key, and drops the entry (Gitea #96) when: the site is removed; the
secret `SB-GRPC-PSK-<siteId>` is written through central's secrets write path (`/admin/secrets`) — secret `SB-GRPC-PSK-<siteId>` is written through central's secrets write path (`/admin/secrets`) —
`SitePskRotationInvalidator` wraps the secrets library's `ISecretCacheInvalidator` and also `SitePskRotationInvalidator` wraps the secrets library's `ISecretCacheInvalidator` and also
broadcasts a `SitePskInvalidated` on the central `DistributedPubSub` topic `site-psk-invalidation` broadcasts a `SitePskInvalidated` on the central cache-invalidation topic `central-cache-invalidation`
so the peer central node drops its copy too; the site rejects a call with `PermissionDenied` / (ScadaBridge#100 item 3; the #96 topic `site-psk-invalidation` is kept for one release for
mixed-version pairs — see Component-Host.md) so the peer central node drops its copy too; the site rejects a call with `PermissionDenied` /
`Unauthenticated` (`PskRejectionRefreshHandler`, wrapped around every per-site channel by `Unauthenticated` (`PskRejectionRefreshHandler`, wrapped around every per-site channel by
`WithSiteCredentials`, so the next call re-reads the key — this covers keys rotated outside the `WithSiteCredentials`, so the next call re-reads the key — this covers keys rotated outside the
process, e.g. by the `secret` CLI, after at most the resolver's `Secrets:ResolveCacheTtl`); a process, e.g. by the `secret` CLI, after at most the resolver's `Secrets:ResolveCacheTtl`); a
@@ -120,7 +120,7 @@ Scripts choose between two call modes per invocation, mirroring the dual-mode da
- Database connections use **standard ADO.NET connection pooling** per named connection. No custom pool management. - Database connections use **standard ADO.NET connection pooling** per named connection. No custom pool management.
- Pool behavior (max pool size, connection lifetime, etc.) can be tuned via connection string parameters in the database connection definition if needed. - Pool behavior (max pool size, connection lifetime, etc.) can be tuned via connection string parameters in the database connection definition if needed.
- **Definition cache (central only).** `ExternalDefinitionLookupCache` holds resolved external-system, method and database-connection definitions for `ExternalSystemGateway:DefinitionCacheTtl` (default 0 = off in the options; the central host sets 10 s because each lookup there is a SQL Server round-trip). Only found definitions are cached, so a new definition is visible immediately; an edit/delete within one TTL. Sites keep it off — `SiteExternalSystemRepository` already serves external systems from the deploy-invalidated `ExternalSystemDefinitionCache` snapshot. - **Definition cache (central only).** `ExternalDefinitionLookupCache` holds resolved external-system, method and database-connection definitions for `ExternalSystemGateway:DefinitionCacheTtl` (default 0 = off in the options; the central host sets 10 s because each lookup there is a SQL Server round-trip). Only found definitions are cached, so a new definition is visible immediately. A ManagementActor edit (#65) or a committed Transport bundle import (#83) drops the cache on the node that made the change and publishes `ExternalDefinitionsInvalidated` on the central cache-invalidation topic so the peer central node drops its copy too (ScadaBridge#100 item 3); an edit that reaches neither path (direct SQL) is visible within one TTL. Sites keep it off — `SiteExternalSystemRepository` already serves external systems from the deploy-invalidated `ExternalSystemDefinitionCache` snapshot.
- GET/DELETE parameters are rendered into the query string with scalars via `ToString()` and collections/objects JSON-serialised (never a CLR type name). - GET/DELETE parameters are rendered into the query string with scalars via `ToString()` and collections/objects JSON-serialised (never a CLR type name).
- Synchronous failures on `Database.Connection()` (e.g., unreachable server) return an error to the calling script, consistent with external system permanent failure handling. - Synchronous failures on `Database.Connection()` (e.g., unreachable server) return an error to the calling script, consistent with external system permanent failure handling.
+2 -1
View File
@@ -351,5 +351,6 @@ built)* section for where the shipped code deviates from the original design).
- **Port options come from the bound `NodeOptions`.** `Program.cs` binds `ScadaBridge:Node` once and reads `GrpcPort` / `CentralGrpcPort` / `MetricsPort` from it; the 8083/8084 defaults live only in `NodeOptions`. Central's dedicated metrics listener remains an explicit *presence* check on the raw key (a defaulted read would bind a new port on every existing central node). - **Port options come from the bound `NodeOptions`.** `Program.cs` binds `ScadaBridge:Node` once and reads `GrpcPort` / `CentralGrpcPort` / `MetricsPort` from it; the 8083/8084 defaults live only in `NodeOptions`. Central's dedicated metrics listener remains an explicit *presence* check on the raw key (a defaulted read would bind a new port on every existing central node).
- **Inbound method warm-up is a hosted service** (`InboundMethodWarmupService`, registered after `AkkaHostedService`), not a pre-`RunAsync` loop — see Component-InboundAPI.md. - **Inbound method warm-up is a hosted service** (`InboundMethodWarmupService`, registered after `AkkaHostedService`), not a pre-`RunAsync` loop — see Component-InboundAPI.md.
- **`SitePskProvider` distinguishes a secret-store fault from a missing secret.** A store fault logs a Warning with the exception and throws `SecretStoreUnavailableException` (not cached); only a genuinely absent/empty/tombstoned secret produces the "seed `SB-GRPC-PSK-{site}`" message. - **`SitePskProvider` distinguishes a secret-store fault from a missing secret.** A store fault logs a Warning with the exception and throws `SecretStoreUnavailableException` (not cached); only a genuinely absent/empty/tombstoned secret produces the "seed `SB-GRPC-PSK-{site}`" message.
- **`SitePskProvider`'s cache follows an in-place rotation (Gitea #96).** Central replaces the secrets library's `ISecretCacheInvalidator` with `SitePskRotationInvalidator` (`AddSitePskRotationInvalidation`), which drops the site's cached key when `SB-GRPC-PSK-{site}` is written and publishes `SitePskInvalidated` on the `site-psk-invalidation` `DistributedPubSub` topic; `AkkaHostedService.RegisterCentralActors` starts the `SitePskInvalidationSubscriber` that applies the peer's broadcasts. A site's auth rejection and a `SitePsks` reload also drop the cache — see Component-Communication.md → *Authentication*. - **`SitePskProvider`'s cache follows an in-place rotation (Gitea #96).** Central replaces the secrets library's `ISecretCacheInvalidator` with `SitePskRotationInvalidator` (`AddSitePskRotationInvalidation`), which drops the site's cached key when `SB-GRPC-PSK-{site}` is written and publishes `SitePskInvalidated` to the peer over the central cache-invalidation topic (below). A site's auth rejection and a `SitePsks` reload also drop the cache — see Component-Communication.md → *Authentication*.
- **Central cache invalidation (ScadaBridge#100 item 3).** One `DistributedPubSub` topic, `central-cache-invalidation`, carries typed `ICentralCacheInvalidation` messages (Commons `Messages/Communication/CentralCacheInvalidations.cs`) between the two central nodes, each wrapped in a `CentralCacheInvalidationEnvelope` stamped with the publishing process's origin id. `AddCentralCacheInvalidation` registers `CentralCacheInvalidationBus` as `ICentralCacheInvalidationPublisher`; `AkkaHostedService.RegisterCentralActors` gives it the mediator and starts `CentralCacheInvalidationSubscriber`, which skips its own node's envelopes and hands the peer's to every registered `ICentralCacheInvalidationHandler`: `SitePskRotationInvalidator` (`SitePskInvalidated`, #96), `RoleMappingCache` (`RoleMappingsInvalidated`, audit D1), and `ExternalDefinitionLookupCache` (`ExternalDefinitionsInvalidated`, audit D3 / #83). Publishers invalidate locally first, post-commit, then publish; delivery is best-effort, so each cache's TTL or revision check stays the fallback. Migration: the #96 topic `site-psk-invalidation` is still published to (bare `SitePskInvalidated`) and subscribed, so a briefly mixed-version central pair keeps exchanging PSK invalidations — remove it once every central node runs this build.
- Central registers `ExternalSystemGatewayOptions.DefinitionCacheTtl = 10 s` ahead of `AddExternalSystemGateway()` so an explicit config value wins; sites leave it off. - Central registers `ExternalSystemGatewayOptions.DefinitionCacheTtl = 10 s` ahead of `AddExternalSystemGateway()` so an explicit config value wins; sites leave it off.
@@ -194,7 +194,7 @@ The two-person authorization workflow for writes through the MxAccess Gateway. B
- **ListApiKeys** / **CreateApiKey** / **UpdateApiKey** / **EnableApiKey** / **DisableApiKey** / **DeleteApiKey**: Manage API keys. - **ListApiKeys** / **CreateApiKey** / **UpdateApiKey** / **EnableApiKey** / **DisableApiKey** / **DeleteApiKey**: Manage API keys.
- **ListRoleMappings** / **CreateRoleMapping** / **UpdateRoleMapping** / **DeleteRoleMapping**: Manage LDAP group-to-role mappings. - **ListRoleMappings** / **CreateRoleMapping** / **UpdateRoleMapping** / **DeleteRoleMapping**: Manage LDAP group-to-role mappings.
- **ListScopeRules** / **AddScopeRule** / **DeleteScopeRule**: Manage site scope rules on role mappings. - **ListScopeRules** / **AddScopeRule** / **DeleteScopeRule**: Manage site scope rules on role mappings.
- Every role-mapping and scope-rule mutation (Create/Update/DeleteRoleMapping, Add/DeleteScopeRule) calls `RoleMappingCache.Invalidate()` as a **post-commit** action, so the change takes effect for the next role mapping on the node that served the command instead of after the `RoleMappingCacheSeconds` TTL (default 30 s); the other central node still converges within that TTL. Post-commit, because invalidating inside the transaction would let a concurrent mapping re-cache the pre-change table. - Every role-mapping and scope-rule mutation (Create/Update/DeleteRoleMapping, Add/DeleteScopeRule) calls `RoleMappingCache.Invalidate()` as a **post-commit** action, so the change takes effect for the next role mapping on the node that served the command instead of after the `RoleMappingCacheSeconds` TTL (default 30 s), and publishes `RoleMappingsInvalidated` on the central cache-invalidation topic so the other central node drops its snapshot too (audit D1, ScadaBridge#100 item 3); the TTL is only the fallback if that best-effort broadcast is lost. The external-definition edit handlers do the same with `ExternalDefinitionsInvalidated` (#65, #83). Post-commit, because invalidating inside the transaction would let a concurrent mapping re-cache the pre-change table.
### Audit Log ### Audit Log
+1 -1
View File
@@ -53,7 +53,7 @@ Set in a local or docker-dev environment via the environment variable `ScadaBrid
- **Idle timeout**: Configurable, default **30 minutes**. Without eligible activity within the idle window, the session is rejected and the user must re-login. Passive `/auth/ping`, activity prevalidation, and Blazor transport requests preserve the activity anchor and suppress sliding renewal; the activity endpoint renews only after authentication and antiforgery validation. Tracked via a `LastActivity` last-activity timestamp claim. The cookie's `ExpireTimeSpan` is set to the idle timeout and `SlidingExpiration` renews it on activity, so the cookie window and the explicit `OnValidatePrincipal` idle check use the **same** value and cannot contradict each other. - **Idle timeout**: Configurable, default **30 minutes**. Without eligible activity within the idle window, the session is rejected and the user must re-login. Passive `/auth/ping`, activity prevalidation, and Blazor transport requests preserve the activity anchor and suppress sliding renewal; the activity endpoint renews only after authentication and antiforgery validation. Tracked via a `LastActivity` last-activity timestamp claim. The cookie's `ExpireTimeSpan` is set to the idle timeout and `SlidingExpiration` renews it on activity, so the cookie window and the explicit `OnValidatePrincipal` idle check use the **same** value and cannot contradict each other.
- **Role-mapping refresh (LDAP-free)**: Configurable, default **15 minutes** (`SecurityOptions.RoleRefreshThresholdMinutes`). At login the session stores the user's raw LDAP groups (one `zb:group` claim each) plus a `zb:lastrolerefresh` anchor. Once the anchor is older than the threshold, `OnValidatePrincipal` re-runs the **DB-backed** `RoleMapper` on the stored groups — **with no LDAP call** — rebuilds the role/scope claims via the shared claim-builder, advances the anchor, and re-issues the cookie. Central role-mapping (DB) changes — including a **revoked** mapping that drops the user's roles, and changed site-scope rules — take effect within this window. Ordinary HTTP validation uses this threshold. A live circuit refreshes mappings in fresh scopes on authentication-state reads — display reads (`GetAuthenticationStateAsync`) at most once per `CookieAuthenticationStateProvider.NonAuthoritativeRefreshInterval` (10 s) per circuit — and mutation entrypoints (`GetAuthoritativeStateAsync`) require a successful current mapping before every dispatch. - **Role-mapping refresh (LDAP-free)**: Configurable, default **15 minutes** (`SecurityOptions.RoleRefreshThresholdMinutes`). At login the session stores the user's raw LDAP groups (one `zb:group` claim each) plus a `zb:lastrolerefresh` anchor. Once the anchor is older than the threshold, `OnValidatePrincipal` re-runs the **DB-backed** `RoleMapper` on the stored groups — **with no LDAP call** — rebuilds the role/scope claims via the shared claim-builder, advances the anchor, and re-issues the cookie. Central role-mapping (DB) changes — including a **revoked** mapping that drops the user's roles, and changed site-scope rules — take effect within this window. Ordinary HTTP validation uses this threshold. A live circuit refreshes mappings in fresh scopes on authentication-state reads — display reads (`GetAuthenticationStateAsync`) at most once per `CookieAuthenticationStateProvider.NonAuthoritativeRefreshInterval` (10 s) per circuit — and mutation entrypoints (`GetAuthoritativeStateAsync`) require a successful current mapping before every dispatch.
- **Role-mapping cache**: `RoleMapper` reads the LDAP-group→role table and the scope rules of every Deployer mapping through a process-wide `RoleMappingCache` snapshot (`SecurityOptions.RoleMappingCacheSeconds`, default **30**; `0` disables). One refresh loads the whole table plus all Deployer scope rules, so a cache hit costs no database round-trip — previously every Basic-authenticated management request and every circuit auth-state read issued `GetAllMappingsAsync` plus one `GetScopeRulesForMappingAsync` per matched Deployer mapping. Without an explicit invalidation a mapping or scope-rule change is therefore visible on every node within the TTL. `RoleMappingCache.Invalidate()` discards the calling node's snapshot so its next mapping re-reads the database (a refresh started before an invalidation is never published over it); other nodes still rely on the TTL. `RoleMappingCacheSeconds = 0` bypasses the snapshot and reads only the matched mappings' scope rules per call. - **Role-mapping cache**: `RoleMapper` reads the LDAP-group→role table and the scope rules of every Deployer mapping through a process-wide `RoleMappingCache` snapshot (`SecurityOptions.RoleMappingCacheSeconds`, default **30**; `0` disables). One refresh loads the whole table plus all Deployer scope rules, so a cache hit costs no database round-trip — previously every Basic-authenticated management request and every circuit auth-state read issued `GetAllMappingsAsync` plus one `GetScopeRulesForMappingAsync` per matched Deployer mapping. Without an explicit invalidation a mapping or scope-rule change is therefore visible on every node within the TTL. `RoleMappingCache.Invalidate()` discards the calling node's snapshot so its next mapping re-reads the database (a refresh started before an invalidation is never published over it); the Management Service's mapping/scope-rule handlers also publish `RoleMappingsInvalidated` on the central cache-invalidation topic, so the peer central node discards its snapshot too (audit D1, ScadaBridge#100 item 3) and the TTL is only the fallback. `RoleMappingCacheSeconds = 0` bypasses the snapshot and reads only the matched mappings' scope rules per call.
#### Session Validation (`OnValidatePrincipal`) #### Session Validation (`OnValidatePrincipal`)
- The cookie principal is built at login by a **single shared claim-builder** (`SessionClaimBuilder`). The `OnValidatePrincipal` role-refresh path rebuilds the principal through the **same** builder, so the login and refresh claim shapes cannot drift. - The cookie principal is built at login by a **single shared claim-builder** (`SessionClaimBuilder`). The `OnValidatePrincipal` role-refresh path rebuilds the principal through the **same** builder, so the login and refresh claim shapes cannot drift.
@@ -0,0 +1,33 @@
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Communication;
namespace ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services;
/// <summary>
/// Tells the peer central node to drop a cache entry this node has just invalidated
/// (ScadaBridge#100 item 3). Central only; resolved optionally, so a host without it (a site
/// node, a plain test host) simply keeps invalidations local.
/// </summary>
/// <remarks>
/// Callers invalidate their own node's cache first and then publish: the publishing node does
/// not apply its own broadcast. Publishing never throws; before the actor system is up it is a
/// no-op, and the peer's cache TTL bounds the staleness exactly as it did before.
/// </remarks>
public interface ICentralCacheInvalidationPublisher
{
/// <summary>Broadcasts <paramref name="invalidation"/> to the other central node.</summary>
/// <param name="invalidation">The typed invalidation.</param>
void Publish(ICentralCacheInvalidation invalidation);
}
/// <summary>
/// Applies a peer's <see cref="ICentralCacheInvalidation"/> to one of this node's caches. Each
/// cache-owning component registers one (as an enumerable service) and ignores the message types
/// it does not own; the central invalidation subscriber hands every received message to all of
/// them.
/// </summary>
public interface ICentralCacheInvalidationHandler
{
/// <summary>Applies <paramref name="invalidation"/> if this handler owns its type.</summary>
/// <param name="invalidation">The received invalidation.</param>
void Apply(ICentralCacheInvalidation invalidation);
}
@@ -0,0 +1,44 @@
namespace ZB.MOM.WW.ScadaBridge.Commons.Messages.Communication;
/// <summary>
/// Marker for a typed cache-invalidation message exchanged between the two central nodes over
/// the single central <c>DistributedPubSub</c> topic <c>central-cache-invalidation</c>
/// (ScadaBridge#100 item 3, #83, #103). Each central node caches configuration-database state
/// in-process; a change made through one node drops that node's cache directly and publishes
/// one of these so the peer drops its copy too, instead of waiting out the cache TTL.
/// </summary>
/// <remarks>
/// Delivery is best-effort and at-most-once: every cache that takes part still has a TTL or a
/// per-request revision check as its correctness fallback. Message contracts follow the
/// additive-only evolution rule — a node that does not know a message type ignores it.
/// </remarks>
public interface ICentralCacheInvalidation;
/// <summary>
/// The LDAP group → role mapping table or a Deployer site-scope rule changed: drop the
/// <c>RoleMappingCache</c> snapshot (audit D1).
/// </summary>
public sealed record RoleMappingsInvalidated : ICentralCacheInvalidation;
/// <summary>
/// An external system, external-system method or database connection definition changed
/// (management edit or Transport bundle import): drop every <c>ExternalDefinitionLookupCache</c>
/// entry (audit D3, #83).
/// </summary>
public sealed record ExternalDefinitionsInvalidated : ICentralCacheInvalidation;
/// <summary>
/// An inbound API method was created, updated, deleted or recompiled: drop the method's
/// compiled handler, its known-bad compile record and its cached row (#103).
/// </summary>
/// <param name="MethodName">The API method's name.</param>
public sealed record InboundApiMethodInvalidated(string MethodName) : ICentralCacheInvalidation;
/// <summary>
/// What travels on the <c>central-cache-invalidation</c> topic: the invalidation plus the
/// publishing node's process-unique origin id, so a node can skip its own broadcast (it already
/// invalidated locally before publishing).
/// </summary>
/// <param name="OriginId">Process-unique id of the publishing node's invalidation bus.</param>
/// <param name="Invalidation">The typed invalidation.</param>
public sealed record CentralCacheInvalidationEnvelope(string OriginId, ICentralCacheInvalidation Invalidation);
@@ -2,6 +2,7 @@ using System.Collections.Concurrent;
using Microsoft.Extensions.Options; using Microsoft.Extensions.Options;
using ZB.MOM.WW.ScadaBridge.Commons.Entities.ExternalSystems; using ZB.MOM.WW.ScadaBridge.Commons.Entities.ExternalSystems;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services; using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Communication;
namespace ZB.MOM.WW.ScadaBridge.ExternalSystemGateway; namespace ZB.MOM.WW.ScadaBridge.ExternalSystemGateway;
@@ -20,7 +21,7 @@ namespace ZB.MOM.WW.ScadaBridge.ExternalSystemGateway;
/// deleted one is visible within one TTL. /// deleted one is visible within one TTL.
/// </para> /// </para>
/// </summary> /// </summary>
public sealed class ExternalDefinitionLookupCache : IExternalDefinitionCacheInvalidator public sealed class ExternalDefinitionLookupCache : IExternalDefinitionCacheInvalidator, ICentralCacheInvalidationHandler
{ {
private sealed record Entry(object? Value, DateTimeOffset ExpiresAtUtc); private sealed record Entry(object? Value, DateTimeOffset ExpiresAtUtc);
@@ -75,10 +76,20 @@ public sealed class ExternalDefinitionLookupCache : IExternalDefinitionCacheInva
/// <summary>Drops every cached definition.</summary> /// <summary>Drops every cached definition.</summary>
/// <remarks> /// <remarks>
/// Called post-commit by the ManagementActor edit handlers (Gitea #65) and by the Transport /// Called post-commit by the ManagementActor edit handlers (Gitea #65) and by the Transport
/// bundle importer through <see cref="IExternalDefinitionCacheInvalidator"/> (Gitea #83). /// bundle importer through <see cref="IExternalDefinitionCacheInvalidator"/> (Gitea #83). Both
/// also publish <see cref="ExternalDefinitionsInvalidated"/> so the peer central node drops its
/// copy through <see cref="Apply"/> (ScadaBridge#100 item 3).
/// </remarks> /// </remarks>
public void InvalidateAll() => _entries.Clear(); public void InvalidateAll() => _entries.Clear();
/// <summary>Applies a peer central node's <see cref="ExternalDefinitionsInvalidated"/>.</summary>
/// <param name="invalidation">The received invalidation; other types are ignored.</param>
public void Apply(ICentralCacheInvalidation invalidation)
{
if (invalidation is ExternalDefinitionsInvalidated)
InvalidateAll();
}
private async Task<T?> GetAsync<T>(string key, Func<Task<T?>> fetch) where T : class private async Task<T?> GetAsync<T>(string key, Func<Task<T?>> fetch) where T : class
{ {
if (!Enabled) if (!Enabled)
@@ -48,6 +48,10 @@ public static class ServiceCollectionExtensions
// importer can drop it after a committed import without referencing this assembly. // importer can drop it after a committed import without referencing this assembly.
services.TryAddSingleton<IExternalDefinitionCacheInvalidator>( services.TryAddSingleton<IExternalDefinitionCacheInvalidator>(
sp => sp.GetRequiredService<ExternalDefinitionLookupCache>()); sp => sp.GetRequiredService<ExternalDefinitionLookupCache>());
// Audit D3 / #83 (ScadaBridge#100 item 3): an edit or import made through the other central
// node reaches this node's cache over the central cache-invalidation topic.
services.TryAddEnumerable(ServiceDescriptor.Singleton<ICentralCacheInvalidationHandler, ExternalDefinitionLookupCache>(
sp => sp.GetRequiredService<ExternalDefinitionLookupCache>()));
services.AddScoped<ExternalSystemClient>(); services.AddScoped<ExternalSystemClient>();
services.AddScoped<IExternalSystemClient>(sp => sp.GetRequiredService<ExternalSystemClient>()); services.AddScoped<IExternalSystemClient>(sp => sp.GetRequiredService<ExternalSystemClient>());
services.AddScoped<DatabaseGateway>(); services.AddScoped<DatabaseGateway>();
@@ -463,16 +463,23 @@ akka {{
"CentralControlGrpcService readiness set (service bound: {Bound})", "CentralControlGrpcService readiness set (service bound: {Bound})",
centralControlGrpc is not null); centralControlGrpc is not null);
// Peer half of site-PSK rotation invalidation (Gitea #96): subscribe to the topic, and give // Peer half of central cache invalidation (ScadaBridge#100 item 3, generalising the #96
// the invalidator the mediator so a rotation written on this node reaches the other one. // site-PSK topic): subscribe this node's cache handlers (site PSK, role mappings, external
var pskRotation = _serviceProvider.GetService<SitePskRotationInvalidator>(); // definitions, inbound API methods) to the shared topic, and give the bus the mediator so
if (pskRotation is not null) // an invalidation made on this node reaches the other one.
var cacheInvalidationBus = _serviceProvider.GetService<CentralCacheInvalidationBus>();
if (cacheInvalidationBus is not null)
{ {
var handlers = _serviceProvider
.GetServices<ICentralCacheInvalidationHandler>()
.ToList();
_actorSystem!.ActorOf( _actorSystem!.ActorOf(
Props.Create(() => new SitePskInvalidationSubscriber(pskRotation)), Props.Create(() => new CentralCacheInvalidationSubscriber(cacheInvalidationBus, handlers)),
"site-psk-invalidation"); "central-cache-invalidation");
pskRotation.SetMediator( cacheInvalidationBus.SetMediator(
Akka.Cluster.Tools.PublishSubscribe.DistributedPubSub.Get(_actorSystem!).Mediator); Akka.Cluster.Tools.PublishSubscribe.DistributedPubSub.Get(_actorSystem!).Mediator);
_logger.LogInformation(
"Central cache invalidation subscribed ({HandlerCount} handlers)", handlers.Count);
} }
// Wire up the CommunicationService with the actor reference // Wire up the CommunicationService with the actor reference
@@ -0,0 +1,185 @@
using Akka.Actor;
using Akka.Cluster.Tools.PublishSubscribe;
using Akka.Event;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.DependencyInjection.Extensions;
using Microsoft.Extensions.Logging;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Communication;
namespace ZB.MOM.WW.ScadaBridge.Host;
/// <summary>
/// Central's cross-node cache-invalidation channel (ScadaBridge#100 item 3): one
/// <c>DistributedPubSub</c> topic, <see cref="Topic"/>, carrying typed
/// <see cref="ICentralCacheInvalidation"/> messages inside a
/// <see cref="CentralCacheInvalidationEnvelope"/>. It generalises the per-cache topic #96 added
/// for site PSK rotation; the role-mapping cache (audit D1), the external-definition cache
/// (audit D3 / #83), the inbound API compile state (#103) and the site PSK cache all ride it.
/// </summary>
/// <remarks>
/// <para>
/// <b>Origin filtering.</b> A publisher has already invalidated its own caches, and a node
/// receives its own broadcast too. Re-applying is harmless for most caches but not free for all
/// of them (the inbound API would drop a handler it has just compiled, and recompile it on the
/// next request), so each envelope carries this bus's process-unique <see cref="OriginId"/> and
/// the subscriber skips its own.
/// </para>
/// <para>
/// <b>Migration from <c>site-psk-invalidation</c>.</b> A <see cref="SitePskInvalidated"/> is also
/// published, bare, on <see cref="LegacySitePskTopic"/>, and the subscriber still listens there,
/// so a central pair that is briefly mixed-version during an upgrade keeps exchanging PSK
/// invalidations. Applying a PSK invalidation twice is harmless. Remove both once every central
/// node runs this build.
/// </para>
/// </remarks>
public sealed class CentralCacheInvalidationBus : ICentralCacheInvalidationPublisher
{
/// <summary>The single central cache-invalidation topic.</summary>
public const string Topic = "central-cache-invalidation";
/// <summary>The #96 per-cache topic, kept for mixed-version central pairs (see remarks).</summary>
public const string LegacySitePskTopic = "site-psk-invalidation";
private readonly ILogger<CentralCacheInvalidationBus> _logger;
private volatile IActorRef? _mediator;
/// <summary>Creates the bus.</summary>
/// <param name="logger">Logger.</param>
public CentralCacheInvalidationBus(ILogger<CentralCacheInvalidationBus> logger)
{
ArgumentNullException.ThrowIfNull(logger);
_logger = logger;
}
/// <summary>This process's origin id, stamped on every envelope it publishes.</summary>
public string OriginId { get; } = Guid.NewGuid().ToString("N");
/// <summary>
/// Supplies the <c>DistributedPubSub</c> mediator once the actor system exists. Until then
/// publishing is a no-op (callers have already invalidated locally).
/// </summary>
/// <param name="mediator">The mediator to publish to.</param>
public void SetMediator(IActorRef mediator) => _mediator = mediator;
/// <inheritdoc />
public void Publish(ICentralCacheInvalidation invalidation)
{
ArgumentNullException.ThrowIfNull(invalidation);
var mediator = _mediator;
if (mediator is null)
{
_logger.LogDebug(
"No pub-sub mediator yet; {Invalidation} stays local to this node.", invalidation.GetType().Name);
return;
}
try
{
mediator.Tell(new Publish(Topic, new CentralCacheInvalidationEnvelope(OriginId, invalidation)));
if (invalidation is SitePskInvalidated psk)
{
mediator.Tell(new Publish(LegacySitePskTopic, psk));
}
}
catch (Exception ex)
{
// Never let a broadcast failure fail the write that triggered it: the peer's TTL
// (or its per-request revision check) still bounds the staleness.
_logger.LogWarning(ex, "Publishing {Invalidation} to the peer central node failed.", invalidation.GetType().Name);
}
}
/// <summary>
/// True when <paramref name="envelope"/> was published by this process (and was therefore
/// already applied locally).
/// </summary>
/// <param name="envelope">The received envelope.</param>
/// <returns><c>true</c> for this node's own broadcast.</returns>
public bool IsOwn(CentralCacheInvalidationEnvelope envelope) =>
string.Equals(envelope.OriginId, OriginId, StringComparison.Ordinal);
}
/// <summary>
/// Central actor that subscribes to <see cref="CentralCacheInvalidationBus.Topic"/> (and the
/// legacy <see cref="CentralCacheInvalidationBus.LegacySitePskTopic"/>) and hands every
/// invalidation published by the peer central node to this node's
/// <see cref="ICentralCacheInvalidationHandler"/>s.
/// </summary>
public sealed class CentralCacheInvalidationSubscriber : ReceiveActor
{
private readonly IReadOnlyList<ICentralCacheInvalidationHandler> _handlers;
/// <summary>Creates the subscriber.</summary>
/// <param name="bus">This node's bus, used to recognise this node's own broadcasts.</param>
/// <param name="handlers">This node's cache handlers.</param>
public CentralCacheInvalidationSubscriber(
CentralCacheInvalidationBus bus, IReadOnlyList<ICentralCacheInvalidationHandler> handlers)
{
ArgumentNullException.ThrowIfNull(bus);
ArgumentNullException.ThrowIfNull(handlers);
_handlers = handlers;
Receive<CentralCacheInvalidationEnvelope>(envelope =>
{
if (!bus.IsOwn(envelope) && envelope.Invalidation is not null)
{
Apply(envelope.Invalidation);
}
});
// Bare message on the #96 topic, from a peer that predates the shared topic.
Receive<SitePskInvalidated>(Apply);
Receive<SubscribeAck>(_ => { });
}
private void Apply(ICentralCacheInvalidation invalidation)
{
foreach (var handler in _handlers)
{
try
{
handler.Apply(invalidation);
}
catch (Exception ex)
{
// One failing cache must not stop the others; its own TTL still bounds it.
Context.GetLogger().Warning(ex, "Applying peer invalidation {0} in {1} failed.",
invalidation.GetType().Name, handler.GetType().Name);
}
}
}
/// <inheritdoc />
protected override void PreStart()
{
try
{
var mediator = DistributedPubSub.Get(Context.System).Mediator;
mediator.Tell(new Subscribe(CentralCacheInvalidationBus.Topic, Self));
mediator.Tell(new Subscribe(CentralCacheInvalidationBus.LegacySitePskTopic, Self));
}
catch (Exception ex)
{
// Non-clustered host (TestKit): peer invalidation is simply unavailable.
Context.GetLogger().Debug("DistributedPubSub not available — peer cache invalidation disabled: {0}", ex.Message);
}
}
}
/// <summary>DI wiring for <see cref="CentralCacheInvalidationBus"/>.</summary>
public static class CentralCacheInvalidationServiceCollectionExtensions
{
/// <summary>
/// Registers the central cache-invalidation bus as <see cref="ICentralCacheInvalidationPublisher"/>.
/// Central only; the actor system wires the mediator and the subscriber at startup.
/// </summary>
/// <param name="services">The service collection.</param>
/// <returns>The same collection.</returns>
public static IServiceCollection AddCentralCacheInvalidation(this IServiceCollection services)
{
services.TryAddSingleton<CentralCacheInvalidationBus>();
services.TryAddSingleton<ICentralCacheInvalidationPublisher>(
sp => sp.GetRequiredService<CentralCacheInvalidationBus>());
return services;
}
}
+4 -1
View File
@@ -413,7 +413,10 @@ try
builder.Services.AddScadaBridgeSecrets(builder.Configuration, SecretsNodeRole.Central); builder.Services.AddScadaBridgeSecrets(builder.Configuration, SecretsNodeRole.Central);
// An in-place rotation of a site's SB-GRPC-PSK-{siteId} through /admin/secrets must drop // An in-place rotation of a site's SB-GRPC-PSK-{siteId} through /admin/secrets must drop
// central's cached copy of that key on BOTH central nodes (Gitea #96). Wraps the library's // central's cached copy of that key on BOTH central nodes (Gitea #96). Wraps the library's
// ISecretCacheInvalidator; the peer half is wired in AkkaHostedService.RegisterCentralActors. // ISecretCacheInvalidator; the peer half rides the central cache-invalidation topic
// (AddCentralCacheInvalidation, ScadaBridge#100 item 3 — also used by the role-mapping,
// external-definition and inbound-API caches), wired in AkkaHostedService.RegisterCentralActors.
builder.Services.AddCentralCacheInvalidation();
builder.Services.AddSitePskRotationInvalidation(); builder.Services.AddSitePskRotationInvalidation();
// Secrets UI authorization: adds the named policies secrets:manage + secrets:reveal // Secrets UI authorization: adds the named policies secrets:manage + secrets:reveal
// (role-based) consumed by the /admin/secrets page. AddSecretsAuthorization only ADDS // (role-based) consumed by the /admin/secrets page. AddSecretsAuthorization only ADDS
@@ -1,9 +1,8 @@
using Akka.Actor;
using Akka.Cluster.Tools.PublishSubscribe;
using Akka.Event;
using Microsoft.Extensions.DependencyInjection; using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.DependencyInjection.Extensions; using Microsoft.Extensions.DependencyInjection.Extensions;
using Microsoft.Extensions.Logging; using Microsoft.Extensions.Logging;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Communication;
using ZB.MOM.WW.ScadaBridge.Communication.Grpc; using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
using ZB.MOM.WW.Secrets; using ZB.MOM.WW.Secrets;
using ZB.MOM.WW.Secrets.Abstractions; using ZB.MOM.WW.Secrets.Abstractions;
@@ -26,9 +25,9 @@ namespace ZB.MOM.WW.ScadaBridge.Host;
/// <para> /// <para>
/// <b>Why the peer needs telling.</b> Both central nodes cache the key in their own /// <b>Why the peer needs telling.</b> Both central nodes cache the key in their own
/// <see cref="SitePskProvider"/> (and their own resolver cache), and the write only ran on one of /// <see cref="SitePskProvider"/> (and their own resolver cache), and the write only ran on one of
/// them. The peer is told over the central <c>DistributedPubSub</c> mediator on /// them. The peer is told with a <see cref="SitePskInvalidated"/> on the central cache-invalidation
/// <see cref="Topic"/> — the same mediator the peer health replication uses. A node receives its /// topic (<see cref="CentralCacheInvalidationBus"/>, ScadaBridge#100 item 3, which generalised
/// own broadcast too; applying it twice is harmless, so there is no origin filtering. /// the #96 <c>site-psk-invalidation</c> topic); this class is also the peer's handler for it.
/// </para> /// </para>
/// <para> /// <para>
/// Writes made outside this process (the <c>secret</c> CLI, a direct SQL write) never reach this /// Writes made outside this process (the <c>secret</c> CLI, a direct SQL write) never reach this
@@ -36,23 +35,22 @@ namespace ZB.MOM.WW.ScadaBridge.Host;
/// <c>ControlPlaneCredentials.WithSiteCredentials</c>). /// <c>ControlPlaneCredentials.WithSiteCredentials</c>).
/// </para> /// </para>
/// </remarks> /// </remarks>
public sealed class SitePskRotationInvalidator : ISecretCacheInvalidator public sealed class SitePskRotationInvalidator : ISecretCacheInvalidator, ICentralCacheInvalidationHandler
{ {
/// <summary>The <c>DistributedPubSub</c> topic the central nodes exchange PSK invalidations on.</summary>
public const string Topic = "site-psk-invalidation";
private readonly ISecretCacheInvalidator _inner; private readonly ISecretCacheInvalidator _inner;
private readonly ISitePskProvider _pskProvider; private readonly ISitePskProvider _pskProvider;
private readonly ICentralCacheInvalidationPublisher? _publisher;
private readonly ILogger<SitePskRotationInvalidator> _logger; private readonly ILogger<SitePskRotationInvalidator> _logger;
private volatile IActorRef? _mediator;
/// <summary>Creates the invalidator.</summary> /// <summary>Creates the invalidator.</summary>
/// <param name="inner">The library's own invalidator (the secret resolver's cache).</param> /// <param name="inner">The library's own invalidator (the secret resolver's cache).</param>
/// <param name="pskProvider">Central's per-site PSK cache.</param> /// <param name="pskProvider">Central's per-site PSK cache.</param>
/// <param name="publisher">The central cache-invalidation publisher; null keeps evictions local.</param>
/// <param name="logger">Logger.</param> /// <param name="logger">Logger.</param>
public SitePskRotationInvalidator( public SitePskRotationInvalidator(
ISecretCacheInvalidator inner, ISecretCacheInvalidator inner,
ISitePskProvider pskProvider, ISitePskProvider pskProvider,
ICentralCacheInvalidationPublisher? publisher,
ILogger<SitePskRotationInvalidator> logger) ILogger<SitePskRotationInvalidator> logger)
{ {
ArgumentNullException.ThrowIfNull(inner); ArgumentNullException.ThrowIfNull(inner);
@@ -60,16 +58,10 @@ public sealed class SitePskRotationInvalidator : ISecretCacheInvalidator
ArgumentNullException.ThrowIfNull(logger); ArgumentNullException.ThrowIfNull(logger);
_inner = inner; _inner = inner;
_pskProvider = pskProvider; _pskProvider = pskProvider;
_publisher = publisher;
_logger = logger; _logger = logger;
} }
/// <summary>
/// Supplies the <c>DistributedPubSub</c> mediator once the actor system exists. Until then
/// (and on a host without one) evictions stay local.
/// </summary>
/// <param name="mediator">The mediator to publish peer invalidations to.</param>
public void SetMediator(IActorRef mediator) => _mediator = mediator;
/// <inheritdoc /> /// <inheritdoc />
public void Invalidate(SecretName name) public void Invalidate(SecretName name)
{ {
@@ -85,12 +77,21 @@ public sealed class SitePskRotationInvalidator : ISecretCacheInvalidator
_logger.LogInformation( _logger.LogInformation(
"gRPC preshared key secret '{SecretName}' was written; dropped the cached key for site " "gRPC preshared key secret '{SecretName}' was written; dropped the cached key for site "
+ "{SiteId} and notified the peer central node.", name.Value, siteId); + "{SiteId} and notified the peer central node.", name.Value, siteId);
_mediator?.Tell(new Publish(Topic, new SitePskInvalidated(siteId))); _publisher?.Publish(new SitePskInvalidated(siteId));
}
/// <inheritdoc />
public void Apply(ICentralCacheInvalidation invalidation)
{
if (invalidation is SitePskInvalidated psk)
{
ApplyPeerInvalidation(psk.SiteId);
}
} }
/// <summary> /// <summary>
/// Applies an invalidation received from a central node (the peer, or this node's own /// Applies an invalidation received from the peer central node: evicts the resolver's cached
/// broadcast): evicts the resolver's cached secret and the PSK cache entry. Never re-publishes. /// secret and the PSK cache entry. Never re-publishes.
/// </summary> /// </summary>
/// <param name="siteId">The site whose key was rotated.</param> /// <param name="siteId">The site whose key was rotated.</param>
public void ApplyPeerInvalidation(string siteId) public void ApplyPeerInvalidation(string siteId)
@@ -107,51 +108,22 @@ public sealed class SitePskRotationInvalidator : ISecretCacheInvalidator
/// <summary> /// <summary>
/// Peer message: the gRPC preshared key of <paramref name="SiteId"/> was rotated on a central node. /// Peer message: the gRPC preshared key of <paramref name="SiteId"/> was rotated on a central node.
/// Rides the central cache-invalidation topic; kept in this namespace because the #96
/// <c>site-psk-invalidation</c> topic carries it bare by this type name (see
/// <see cref="CentralCacheInvalidationBus"/>).
/// </summary> /// </summary>
/// <param name="SiteId">The site whose key changed.</param> /// <param name="SiteId">The site whose key changed.</param>
public sealed record SitePskInvalidated(string SiteId); public sealed record SitePskInvalidated(string SiteId) : ICentralCacheInvalidation;
/// <summary>
/// Central actor that subscribes to <see cref="SitePskRotationInvalidator.Topic"/> and applies
/// each <see cref="SitePskInvalidated"/> to this node's caches.
/// </summary>
public sealed class SitePskInvalidationSubscriber : ReceiveActor
{
private readonly SitePskRotationInvalidator _invalidator;
/// <summary>Creates the subscriber.</summary>
/// <param name="invalidator">This node's invalidator.</param>
public SitePskInvalidationSubscriber(SitePskRotationInvalidator invalidator)
{
_invalidator = invalidator;
Receive<SitePskInvalidated>(m => _invalidator.ApplyPeerInvalidation(m.SiteId));
Receive<SubscribeAck>(_ => { });
}
/// <inheritdoc />
protected override void PreStart()
{
try
{
DistributedPubSub.Get(Context.System).Mediator.Tell(
new Subscribe(SitePskRotationInvalidator.Topic, Self));
}
catch (Exception ex)
{
// Non-clustered host (TestKit): peer invalidation is simply unavailable.
Context.GetLogger().Debug("DistributedPubSub not available — peer PSK invalidation disabled: {0}", ex.Message);
}
}
}
/// <summary>DI wiring for <see cref="SitePskRotationInvalidator"/>.</summary> /// <summary>DI wiring for <see cref="SitePskRotationInvalidator"/>.</summary>
public static class SitePskRotationServiceCollectionExtensions public static class SitePskRotationServiceCollectionExtensions
{ {
/// <summary> /// <summary>
/// Replaces the secrets library's <see cref="ISecretCacheInvalidator"/> with /// Replaces the secrets library's <see cref="ISecretCacheInvalidator"/> with
/// <see cref="SitePskRotationInvalidator"/> wrapping the library's resolver. Central only; /// <see cref="SitePskRotationInvalidator"/> wrapping the library's resolver, and registers it
/// call after <c>AddScadaBridgeSecrets</c> and after <see cref="ISitePskProvider"/> is /// as a central cache-invalidation handler. Central only; call after
/// registered. /// <c>AddScadaBridgeSecrets</c>, after <see cref="ISitePskProvider"/> is registered and after
/// <see cref="CentralCacheInvalidationServiceCollectionExtensions.AddCentralCacheInvalidation"/>.
/// </summary> /// </summary>
/// <param name="services">The service collection.</param> /// <param name="services">The service collection.</param>
/// <returns>The same collection.</returns> /// <returns>The same collection.</returns>
@@ -160,9 +132,12 @@ public static class SitePskRotationServiceCollectionExtensions
services.TryAddSingleton(sp => new SitePskRotationInvalidator( services.TryAddSingleton(sp => new SitePskRotationInvalidator(
sp.GetRequiredService<DefaultSecretResolver>(), sp.GetRequiredService<DefaultSecretResolver>(),
sp.GetRequiredService<ISitePskProvider>(), sp.GetRequiredService<ISitePskProvider>(),
sp.GetService<ICentralCacheInvalidationPublisher>(),
sp.GetRequiredService<ILogger<SitePskRotationInvalidator>>())); sp.GetRequiredService<ILogger<SitePskRotationInvalidator>>()));
services.Replace(ServiceDescriptor.Singleton<ISecretCacheInvalidator>( services.Replace(ServiceDescriptor.Singleton<ISecretCacheInvalidator>(
sp => sp.GetRequiredService<SitePskRotationInvalidator>())); sp => sp.GetRequiredService<SitePskRotationInvalidator>()));
services.AddSingleton<ICentralCacheInvalidationHandler>(
sp => sp.GetRequiredService<SitePskRotationInvalidator>());
return services; return services;
} }
} }
@@ -21,6 +21,7 @@ using ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories; using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Security; using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Security;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services; using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Communication;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.DebugView; using ZB.MOM.WW.ScadaBridge.Commons.Messages.DebugView;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Transport; using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Transport;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Management; using ZB.MOM.WW.ScadaBridge.Commons.Messages.Management;
@@ -2386,15 +2387,22 @@ public class ManagementActor : ReceiveActor
/// call on this node resolves the fresh definition rather than waiting out /// call on this node resolves the fresh definition rather than waiting out
/// <c>DefinitionCacheTtl</c> (ScadaBridge#65). Queued post-commit: invalidating before the commit /// <c>DefinitionCacheTtl</c> (ScadaBridge#65). Queued post-commit: invalidating before the commit
/// would let a concurrent lookup re-cache the pre-edit row. The cache is per-node, so the other /// would let a concurrent lookup re-cache the pre-edit row. The cache is per-node, so the other
/// central node still converges within <c>DefinitionCacheTtl</c>. A no-op when the cache is not /// central node is told with an <see cref="ExternalDefinitionsInvalidated"/> on the central
/// registered (site roles, plain test hosts). /// cache-invalidation topic (audit D3 / #83, ScadaBridge#100 item 3). A no-op when neither the
/// cache nor the publisher is registered (site roles, plain test hosts).
/// </summary> /// </summary>
private static Task InvalidateExternalDefinitionCacheAfterCommit(IServiceProvider sp) private static Task InvalidateExternalDefinitionCacheAfterCommit(IServiceProvider sp)
{ {
var cache = sp.GetService<ExternalDefinitionLookupCache>(); var cache = sp.GetService<ExternalDefinitionLookupCache>();
return cache is null var peers = sp.GetService<ICentralCacheInvalidationPublisher>();
return cache is null && peers is null
? Task.CompletedTask ? Task.CompletedTask
: AfterConfigurationCommit(sp, () => { cache.InvalidateAll(); return Task.CompletedTask; }); : AfterConfigurationCommit(sp, () =>
{
cache?.InvalidateAll();
peers?.Publish(new ExternalDefinitionsInvalidated());
return Task.CompletedTask;
});
} }
private static async Task<object?> HandleListExternalSystems(IServiceProvider sp) private static async Task<object?> HandleListExternalSystems(IServiceProvider sp)
@@ -2902,15 +2910,23 @@ public class ManagementActor : ReceiveActor
/// Drops this node's <see cref="RoleMappingCache"/> snapshot once a mapping / scope-rule /// Drops this node's <see cref="RoleMappingCache"/> snapshot once a mapping / scope-rule
/// change has committed, so the next role mapping on this node sees it immediately rather /// change has committed, so the next role mapping on this node sees it immediately rather
/// than after the cache TTL (lane K review M1). Queued post-commit: invalidating before the /// than after the cache TTL (lane K review M1). Queued post-commit: invalidating before the
/// commit would let a concurrent mapping re-cache the pre-change table. Other central /// commit would let a concurrent mapping re-cache the pre-change table. The peer central node
/// nodes still converge within <c>RoleMappingCacheSeconds</c>. /// is told with a <see cref="RoleMappingsInvalidated"/> on the central cache-invalidation topic
/// (audit D1, ScadaBridge#100 item 3), so a revocation takes effect there at once too;
/// <c>RoleMappingCacheSeconds</c> is only the fallback if that broadcast is lost.
/// </summary> /// </summary>
private static Task InvalidateRoleMappingCacheAfterCommit(IServiceProvider sp) private static Task InvalidateRoleMappingCacheAfterCommit(IServiceProvider sp)
{ {
var cache = sp.GetService<RoleMappingCache>(); var cache = sp.GetService<RoleMappingCache>();
return cache is null var peers = sp.GetService<ICentralCacheInvalidationPublisher>();
return cache is null && peers is null
? Task.CompletedTask ? Task.CompletedTask
: AfterConfigurationCommit(sp, () => { cache.Invalidate(); return Task.CompletedTask; }); : AfterConfigurationCommit(sp, () =>
{
cache?.Invalidate();
peers?.Publish(new RoleMappingsInvalidated());
return Task.CompletedTask;
});
} }
private static async Task<object?> HandleCreateRoleMapping(IServiceProvider sp, CreateRoleMappingCommand cmd, string user) private static async Task<object?> HandleCreateRoleMapping(IServiceProvider sp, CreateRoleMappingCommand cmd, string user)
@@ -1,5 +1,7 @@
using Microsoft.Extensions.Options; using Microsoft.Extensions.Options;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories; using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Communication;
namespace ZB.MOM.WW.ScadaBridge.Security; namespace ZB.MOM.WW.ScadaBridge.Security;
@@ -14,7 +16,10 @@ namespace ZB.MOM.WW.ScadaBridge.Security;
/// an acceptable price: without an explicit <see cref="Invalidate"/>, a mapping or scope-rule /// an acceptable price: without an explicit <see cref="Invalidate"/>, a mapping or scope-rule
/// change is visible on every node within that window. <see cref="Invalidate"/> discards this /// change is visible on every node within that window. <see cref="Invalidate"/> discards this
/// node's snapshot so the next mapping re-reads the database; calling it after a mutation makes /// node's snapshot so the next mapping re-reads the database; calling it after a mutation makes
/// the change immediate on the node that performed it (other nodes still rely on the TTL). /// the change immediate on the node that performed it. The peer central node is told over the
/// central cache-invalidation topic (<see cref="RoleMappingsInvalidated"/>, audit D1,
/// ScadaBridge#100 item 3), so a revoked mapping stops working there at once too; the TTL is
/// only the fallback if that best-effort broadcast is lost.
/// A TTL of <c>0</c> disables the cache: <see cref="Enabled"/> is false and /// A TTL of <c>0</c> disables the cache: <see cref="Enabled"/> is false and
/// <see cref="RoleMapper"/> reads the repository directly on every call. /// <see cref="RoleMapper"/> reads the repository directly on every call.
/// </summary> /// </summary>
@@ -30,7 +35,7 @@ namespace ZB.MOM.WW.ScadaBridge.Security;
/// read of the pre-change table can never overwrite the invalidation. /// read of the pre-change table can never overwrite the invalidation.
/// </para> /// </para>
/// </remarks> /// </remarks>
public sealed class RoleMappingCache public sealed class RoleMappingCache : ICentralCacheInvalidationHandler
{ {
private readonly TimeProvider _timeProvider; private readonly TimeProvider _timeProvider;
private readonly IOptionsMonitor<SecurityOptions>? _options; private readonly IOptionsMonitor<SecurityOptions>? _options;
@@ -65,6 +70,14 @@ public sealed class RoleMappingCache
/// </summary> /// </summary>
public bool Enabled => Ttl > TimeSpan.Zero; public bool Enabled => Ttl > TimeSpan.Zero;
/// <summary>Applies a peer central node's <see cref="RoleMappingsInvalidated"/>.</summary>
/// <param name="invalidation">The received invalidation; other types are ignored.</param>
public void Apply(ICentralCacheInvalidation invalidation)
{
if (invalidation is RoleMappingsInvalidated)
Invalidate();
}
/// <summary> /// <summary>
/// Discards the cached snapshot so the next mapping re-reads the database. Call after any /// Discards the cached snapshot so the next mapping re-reads the database. Call after any
/// LDAP-group-mapping or site-scope-rule create/update/delete. /// LDAP-group-mapping or site-scope-rule create/update/delete.
@@ -65,6 +65,10 @@ public static class ServiceCollectionExtensions
// (review 2026-10-21 05/P1). Singleton so every scoped RoleMapper shares it; the // (review 2026-10-21 05/P1). Singleton so every scoped RoleMapper shares it; the
// repository it reads through is still the caller's scoped one. // repository it reads through is still the caller's scoped one.
services.AddSingleton<RoleMappingCache>(); services.AddSingleton<RoleMappingCache>();
// Audit D1 (ScadaBridge#100 item 3): a mapping change made through the other central node
// reaches this node's snapshot over the central cache-invalidation topic.
services.TryAddEnumerable(ServiceDescriptor.Singleton<ICentralCacheInvalidationHandler, RoleMappingCache>(
sp => sp.GetRequiredService<RoleMappingCache>()));
// The cookie OnValidatePrincipal core. Scoped to match the // The cookie OnValidatePrincipal core. Scoped to match the
// IGroupRoleMapper<string> it depends on (which depends on the Scoped // IGroupRoleMapper<string> it depends on (which depends on the Scoped
@@ -17,6 +17,7 @@ using ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories; using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Scripting; using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Scripting;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services; using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Communication;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Transport; using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Transport;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Scripting; using ZB.MOM.WW.ScadaBridge.Commons.Messages.Scripting;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Transport; using ZB.MOM.WW.ScadaBridge.Commons.Types.Transport;
@@ -98,6 +99,9 @@ public sealed class BundleImporter : IBundleImporter
// Optional (Gitea #83): drops this node's external-definition lookup cache after a // Optional (Gitea #83): drops this node's external-definition lookup cache after a
// committed import. Null on hosts without the External System Gateway. // committed import. Null on hosts without the External System Gateway.
private readonly IExternalDefinitionCacheInvalidator? _definitionCacheInvalidator; private readonly IExternalDefinitionCacheInvalidator? _definitionCacheInvalidator;
// Optional (ScadaBridge#100 item 3 / #83): tells the peer central node to drop its caches of
// what the import wrote. Null on hosts without the central cache-invalidation bus.
private readonly ICentralCacheInvalidationPublisher? _cacheInvalidationPublisher;
private readonly ILogger<BundleImporter>? _logger; private readonly ILogger<BundleImporter>? _logger;
/// <summary> /// <summary>
@@ -125,6 +129,7 @@ public sealed class BundleImporter : IBundleImporter
/// <param name="logger">Optional logger.</param> /// <param name="logger">Optional logger.</param>
/// <param name="watermark">Optional: template-graph version watermark, bumped once per apply attempt so the flatten-session caches and the process-wide staleness memo cannot serve pre-import (or rolled-back) readings. Null on hosts that don't register it.</param> /// <param name="watermark">Optional: template-graph version watermark, bumped once per apply attempt so the flatten-session caches and the process-wide staleness memo cannot serve pre-import (or rolled-back) readings. Null on hosts that don't register it.</param>
/// <param name="definitionCacheInvalidator">Optional (Gitea #83): drops this node's external-system / database-connection definition cache after a committed import, so imported definitions take effect immediately rather than after the cache TTL. Null on hosts without the External System Gateway.</param> /// <param name="definitionCacheInvalidator">Optional (Gitea #83): drops this node's external-system / database-connection definition cache after a committed import, so imported definitions take effect immediately rather than after the cache TTL. Null on hosts without the External System Gateway.</param>
/// <param name="cacheInvalidationPublisher">Optional (ScadaBridge#100 item 3): tells the peer central node to drop its caches of what a committed import wrote. Null on hosts without the central cache-invalidation bus.</param>
public BundleImporter( public BundleImporter(
BundleSerializer bundleSerializer, BundleSerializer bundleSerializer,
ManifestValidator manifestValidator, ManifestValidator manifestValidator,
@@ -147,7 +152,8 @@ public sealed class BundleImporter : IBundleImporter
IScriptArtifactChangeBus? scriptArtifactChangeBus = null, IScriptArtifactChangeBus? scriptArtifactChangeBus = null,
ILogger<BundleImporter>? logger = null, ILogger<BundleImporter>? logger = null,
ITemplateGraphWatermark? watermark = null, ITemplateGraphWatermark? watermark = null,
IExternalDefinitionCacheInvalidator? definitionCacheInvalidator = null) IExternalDefinitionCacheInvalidator? definitionCacheInvalidator = null,
ICentralCacheInvalidationPublisher? cacheInvalidationPublisher = null)
{ {
_bundleSerializer = bundleSerializer ?? throw new ArgumentNullException(nameof(bundleSerializer)); _bundleSerializer = bundleSerializer ?? throw new ArgumentNullException(nameof(bundleSerializer));
_manifestValidator = manifestValidator ?? throw new ArgumentNullException(nameof(manifestValidator)); _manifestValidator = manifestValidator ?? throw new ArgumentNullException(nameof(manifestValidator));
@@ -170,6 +176,7 @@ public sealed class BundleImporter : IBundleImporter
_scriptArtifactChangeBus = scriptArtifactChangeBus; _scriptArtifactChangeBus = scriptArtifactChangeBus;
_watermark = watermark; _watermark = watermark;
_definitionCacheInvalidator = definitionCacheInvalidator; _definitionCacheInvalidator = definitionCacheInvalidator;
_cacheInvalidationPublisher = cacheInvalidationPublisher;
_logger = logger; _logger = logger;
} }
@@ -2498,18 +2505,20 @@ public sealed class BundleImporter : IBundleImporter
/// served the pre-import definition for up to <c>DefinitionCacheTtl</c>. Invalidates on every /// served the pre-import definition for up to <c>DefinitionCacheTtl</c>. Invalidates on every
/// committed import rather than only when an external entity was resolved: dependencies can /// committed import rather than only when an external entity was resolved: dependencies can
/// pull definitions in implicitly, imports are rare, and a cleared cache only costs one /// pull definitions in implicitly, imports are rare, and a cleared cache only costs one
/// re-query per name. Never throws (the transaction has already committed). /// re-query per name. The peer central node is told too (ScadaBridge#100 item 3), since its
/// copy of the cache is just as stale. Never throws (the transaction has already committed).
/// </summary> /// </summary>
private void InvalidateExternalDefinitionCache() private void InvalidateExternalDefinitionCache()
{ {
if (_definitionCacheInvalidator is null) if (_definitionCacheInvalidator is null && _cacheInvalidationPublisher is null)
{ {
return; return;
} }
try try
{ {
_definitionCacheInvalidator.InvalidateAll(); _definitionCacheInvalidator?.InvalidateAll();
_cacheInvalidationPublisher?.Publish(new ExternalDefinitionsInvalidated());
} }
catch (Exception ex) catch (Exception ex)
{ {
@@ -0,0 +1,80 @@
using Akka.Actor;
using Akka.Cluster.Tools.PublishSubscribe;
using Akka.TestKit.Xunit2;
using Microsoft.Extensions.Logging.Abstractions;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Communication;
namespace ZB.MOM.WW.ScadaBridge.Host.Tests;
/// <summary>
/// ScadaBridge#100 item 3: the central cache-invalidation bus publishes typed invalidations on
/// one topic, and the subscriber hands the peer's (never its own) to every handler.
/// </summary>
public class CentralCacheInvalidationBusTests : TestKit
{
private sealed class RecordingHandler : ICentralCacheInvalidationHandler
{
public List<ICentralCacheInvalidation> Applied { get; } = new();
public void Apply(ICentralCacheInvalidation invalidation)
{
lock (Applied) Applied.Add(invalidation);
}
}
private sealed class ThrowingHandler : ICentralCacheInvalidationHandler
{
public void Apply(ICentralCacheInvalidation invalidation) => throw new InvalidOperationException("boom");
}
private readonly CentralCacheInvalidationBus _bus = new(NullLogger<CentralCacheInvalidationBus>.Instance);
/// <summary>Publishing wraps the invalidation with this node's origin id on the shared topic.</summary>
[Fact]
public void Publish_SendsAnEnvelopeOnTheSharedTopic()
{
var mediator = CreateTestProbe();
_bus.SetMediator(mediator.Ref);
_bus.Publish(new RoleMappingsInvalidated());
var publish = mediator.ExpectMsg<Publish>();
Assert.Equal("central-cache-invalidation", publish.Topic);
var envelope = Assert.IsType<CentralCacheInvalidationEnvelope>(publish.Message);
Assert.Equal(_bus.OriginId, envelope.OriginId);
Assert.IsType<RoleMappingsInvalidated>(envelope.Invalidation);
mediator.ExpectNoMsg(TimeSpan.FromMilliseconds(100)); // only PSK also uses the legacy topic
}
/// <summary>Before the mediator exists publishing is a silent no-op.</summary>
[Fact]
public void Publish_WithoutAMediator_DoesNotThrow()
{
_bus.Publish(new ExternalDefinitionsInvalidated());
}
/// <summary>The peer's invalidation reaches every handler; this node's own broadcast is skipped.</summary>
[Fact]
public void Subscriber_AppliesPeerInvalidations_AndSkipsItsOwn()
{
var handler = new RecordingHandler();
var subscriber = Sys.ActorOf(Props.Create(() => new CentralCacheInvalidationSubscriber(
_bus, new ICentralCacheInvalidationHandler[] { new ThrowingHandler(), handler })));
subscriber.Tell(new CentralCacheInvalidationEnvelope(_bus.OriginId, new RoleMappingsInvalidated()));
subscriber.Tell(new CentralCacheInvalidationEnvelope("peer", new ExternalDefinitionsInvalidated()));
subscriber.Tell(new CentralCacheInvalidationEnvelope("peer", new InboundApiMethodInvalidated("M")));
AwaitAssert(() =>
{
lock (handler.Applied)
{
// A throwing handler ahead of it does not stop delivery.
Assert.Equal(
new ICentralCacheInvalidation[] { new ExternalDefinitionsInvalidated(), new InboundApiMethodInvalidated("M") },
handler.Applied);
}
});
}
}
@@ -419,6 +419,26 @@ public class CentralCompositionRootTests : IDisposable
_factory.Services.GetService<ZB.MOM.WW.Secrets.Abstractions.ISecretCacheInvalidator>()); _factory.Services.GetService<ZB.MOM.WW.Secrets.Abstractions.ISecretCacheInvalidator>());
} }
/// <summary>
/// ScadaBridge#100 item 3: central publishes peer invalidations through the shared bus, and
/// every cache that takes part (site PSK, role mappings, external definitions) is registered
/// as a handler for the subscriber.
/// </summary>
[Fact]
public void Central_CacheInvalidation_BusAndHandlersAreRegistered()
{
Assert.IsType<CentralCacheInvalidationBus>(
_factory.Services.GetService<ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services.ICentralCacheInvalidationPublisher>());
var handlers = _factory.Services
.GetServices<ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services.ICentralCacheInvalidationHandler>()
.Select(h => h.GetType())
.ToList();
Assert.Contains(typeof(SitePskRotationInvalidator), handlers);
Assert.Contains(typeof(ZB.MOM.WW.ScadaBridge.Security.RoleMappingCache), handlers);
Assert.Contains(typeof(ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.ExternalDefinitionLookupCache), handlers);
Assert.Equal(handlers.Count, handlers.Distinct().Count());
}
/// <summary>Verifies that the shared audit seam resolves to the central direct-write bridge.</summary> /// <summary>Verifies that the shared audit seam resolves to the central direct-write bridge.</summary>
[Fact] [Fact]
public void Central_SharedAuditSeam_IsTheCentralDirectWriteBridge() public void Central_SharedAuditSeam_IsTheCentralDirectWriteBridge()
@@ -3,6 +3,8 @@ using Akka.Cluster.Tools.PublishSubscribe;
using Akka.TestKit.Xunit2; using Akka.TestKit.Xunit2;
using Microsoft.Extensions.Logging.Abstractions; using Microsoft.Extensions.Logging.Abstractions;
using NSubstitute; using NSubstitute;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Communication;
using ZB.MOM.WW.ScadaBridge.Communication.Grpc; using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
using ZB.MOM.WW.Secrets.Abstractions; using ZB.MOM.WW.Secrets.Abstractions;
@@ -10,35 +12,43 @@ namespace ZB.MOM.WW.ScadaBridge.Host.Tests;
/// <summary> /// <summary>
/// Gitea #96: an in-place rotation of <c>SB-GRPC-PSK-{siteId}</c> written through the secrets /// Gitea #96: an in-place rotation of <c>SB-GRPC-PSK-{siteId}</c> written through the secrets
/// write path must drop central's cached key on this node and on the peer central node. /// write path must drop central's cached key on this node and on the peer central node. Since
/// ScadaBridge#100 item 3 the peer is told over the shared central cache-invalidation topic.
/// </summary> /// </summary>
public class SitePskRotationInvalidatorTests : TestKit public class SitePskRotationInvalidatorTests : TestKit
{ {
private readonly ISecretCacheInvalidator _inner = Substitute.For<ISecretCacheInvalidator>(); private readonly ISecretCacheInvalidator _inner = Substitute.For<ISecretCacheInvalidator>();
private readonly ISitePskProvider _psk = Substitute.For<ISitePskProvider>(); private readonly ISitePskProvider _psk = Substitute.For<ISitePskProvider>();
private readonly CentralCacheInvalidationBus _bus = new(NullLogger<CentralCacheInvalidationBus>.Instance);
private SitePskRotationInvalidator Create() => private SitePskRotationInvalidator Create(ICentralCacheInvalidationPublisher? publisher) =>
new(_inner, _psk, NullLogger<SitePskRotationInvalidator>.Instance); new(_inner, _psk, publisher, NullLogger<SitePskRotationInvalidator>.Instance);
/// <summary> /// <summary>
/// Verifies that writing a site's PSK secret evicts the resolver cache, drops the site's /// Verifies that writing a site's PSK secret evicts the resolver cache, drops the site's
/// cached key and publishes the invalidation to the peer. /// cached key and publishes the invalidation on the shared topic and, for a mixed-version
/// peer, bare on the legacy #96 topic.
/// </summary> /// </summary>
[Fact] [Fact]
public void WritingAPskSecret_DropsTheCachedKey_AndTellsThePeer() public void WritingAPskSecret_DropsTheCachedKey_AndTellsThePeer()
{ {
var mediator = CreateTestProbe(); var mediator = CreateTestProbe();
var invalidator = Create(); _bus.SetMediator(mediator.Ref);
invalidator.SetMediator(mediator.Ref); var invalidator = Create(_bus);
// The secrets library lowercases names, so the site id arrives lowercased too. // The secrets library lowercases names, so the site id arrives lowercased too.
invalidator.Invalidate(new SecretName("SB-GRPC-PSK-site-a")); invalidator.Invalidate(new SecretName("SB-GRPC-PSK-site-a"));
_inner.Received(1).Invalidate(new SecretName("SB-GRPC-PSK-site-a")); _inner.Received(1).Invalidate(new SecretName("SB-GRPC-PSK-site-a"));
_psk.Received(1).Invalidate("site-a"); _psk.Received(1).Invalidate("site-a");
var publish = mediator.ExpectMsg<Publish>(); var shared = mediator.ExpectMsg<Publish>();
Assert.Equal(SitePskRotationInvalidator.Topic, publish.Topic); Assert.Equal(CentralCacheInvalidationBus.Topic, shared.Topic);
Assert.Equal(new SitePskInvalidated("site-a"), publish.Message); var envelope = Assert.IsType<CentralCacheInvalidationEnvelope>(shared.Message);
Assert.Equal(new SitePskInvalidated("site-a"), envelope.Invalidation);
Assert.Equal(_bus.OriginId, envelope.OriginId);
var legacy = mediator.ExpectMsg<Publish>();
Assert.Equal("site-psk-invalidation", legacy.Topic);
Assert.Equal(new SitePskInvalidated("site-a"), legacy.Message);
} }
/// <summary>Negative control: any other secret only evicts the resolver cache.</summary> /// <summary>Negative control: any other secret only evicts the resolver cache.</summary>
@@ -46,8 +56,8 @@ public class SitePskRotationInvalidatorTests : TestKit
public void WritingAnyOtherSecret_OnlyEvictsTheResolverCache() public void WritingAnyOtherSecret_OnlyEvictsTheResolverCache()
{ {
var mediator = CreateTestProbe(); var mediator = CreateTestProbe();
var invalidator = Create(); _bus.SetMediator(mediator.Ref);
invalidator.SetMediator(mediator.Ref); var invalidator = Create(_bus);
invalidator.Invalidate(new SecretName("SMTP-PASSWORD")); invalidator.Invalidate(new SecretName("SMTP-PASSWORD"));
@@ -56,13 +66,15 @@ public class SitePskRotationInvalidatorTests : TestKit
mediator.ExpectNoMsg(TimeSpan.FromMilliseconds(200)); mediator.ExpectNoMsg(TimeSpan.FromMilliseconds(200));
} }
/// <summary>Verifies that before the actor system exists an invalidation still applies locally.</summary> /// <summary>Verifies that without a publisher (or before the actor system exists) an invalidation still applies locally.</summary>
[Fact] [Fact]
public void WithoutAMediator_TheLocalInvalidationStillApplies() public void WithoutAPublisher_TheLocalInvalidationStillApplies()
{ {
Create().Invalidate(new SecretName("SB-GRPC-PSK-site-a")); Create(null).Invalidate(new SecretName("SB-GRPC-PSK-site-a"));
Create(_bus).Invalidate(new SecretName("SB-GRPC-PSK-site-b")); // bus has no mediator yet
_psk.Received(1).Invalidate("site-a"); _psk.Received(1).Invalidate("site-a");
_psk.Received(1).Invalidate("site-b");
} }
/// <summary> /// <summary>
@@ -73,11 +85,12 @@ public class SitePskRotationInvalidatorTests : TestKit
public void Subscriber_AppliesAPeerInvalidation_WithoutRepublishing() public void Subscriber_AppliesAPeerInvalidation_WithoutRepublishing()
{ {
var mediator = CreateTestProbe(); var mediator = CreateTestProbe();
var invalidator = Create(); _bus.SetMediator(mediator.Ref);
invalidator.SetMediator(mediator.Ref); var invalidator = Create(_bus);
var subscriber = Sys.ActorOf(Props.Create(() => new SitePskInvalidationSubscriber(invalidator))); var subscriber = Sys.ActorOf(Props.Create(() => new CentralCacheInvalidationSubscriber(
_bus, new ICentralCacheInvalidationHandler[] { invalidator })));
subscriber.Tell(new SitePskInvalidated("site-b")); subscriber.Tell(new CentralCacheInvalidationEnvelope("peer-node", new SitePskInvalidated("site-b")));
AwaitAssert(() => AwaitAssert(() =>
{ {
@@ -86,4 +99,20 @@ public class SitePskRotationInvalidatorTests : TestKit
}); });
mediator.ExpectNoMsg(TimeSpan.FromMilliseconds(200)); mediator.ExpectNoMsg(TimeSpan.FromMilliseconds(200));
} }
/// <summary>
/// Migration: a bare <see cref="SitePskInvalidated"/> from a peer still on the #96 topic is
/// applied too.
/// </summary>
[Fact]
public void Subscriber_AppliesALegacyBarePskInvalidation()
{
var invalidator = Create(_bus);
var subscriber = Sys.ActorOf(Props.Create(() => new CentralCacheInvalidationSubscriber(
_bus, new ICentralCacheInvalidationHandler[] { invalidator })));
subscriber.Tell(new SitePskInvalidated("site-c"));
AwaitAssert(() => _psk.Received(1).Invalidate("site-c"));
}
} }
@@ -0,0 +1,87 @@
using Akka.Actor;
using Akka.Cluster.Tools.PublishSubscribe;
using Microsoft.Extensions.Logging.Abstractions;
using Xunit;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Communication;
using ZB.MOM.WW.ScadaBridge.Host;
namespace ZB.MOM.WW.ScadaBridge.IntegrationTests.Cluster;
/// <summary>
/// ScadaBridge#100 item 3 / #83 / #103 across a REAL two-node central cluster built from the
/// production HOCON: an invalidation published on one node's bus is delivered, through
/// <c>DistributedPubSub</c> and the production serializer, to the other node's handlers — and
/// not back to the publishing node's own handlers.
/// </summary>
public class CentralCacheInvalidationTwoNodeTests
{
private sealed class RecordingHandler : ICentralCacheInvalidationHandler
{
private readonly List<ICentralCacheInvalidation> _applied = new();
public IReadOnlyList<ICentralCacheInvalidation> Applied
{
get { lock (_applied) return _applied.ToList(); }
}
public void Apply(ICentralCacheInvalidation invalidation)
{
lock (_applied) _applied.Add(invalidation);
}
}
private static (CentralCacheInvalidationBus Bus, RecordingHandler Handler) Wire(ActorSystem node)
{
var bus = new CentralCacheInvalidationBus(NullLogger<CentralCacheInvalidationBus>.Instance);
var handler = new RecordingHandler();
node.ActorOf(Props.Create(() => new CentralCacheInvalidationSubscriber(
bus, new ICentralCacheInvalidationHandler[] { handler })), "central-cache-invalidation");
bus.SetMediator(DistributedPubSub.Get(node).Mediator);
return (bus, handler);
}
/// <summary>
/// Every typed invalidation crosses to the peer intact; the publisher never applies its own.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task Invalidations_ReachThePeerNode_AndNotThePublisher()
{
await using var cluster = await TwoNodeClusterFixture.StartAsync();
var a = Wire(cluster.NodeA);
var b = Wire(cluster.NodeB);
// Subscriptions spread by gossip, so probe until the peer has registered its subscriber.
var deadline = DateTime.UtcNow + TimeSpan.FromSeconds(20);
while (b.Handler.Applied.Count == 0)
{
Assert.True(DateTime.UtcNow < deadline, "node B never received node A's invalidation");
a.Bus.Publish(new RoleMappingsInvalidated());
await Task.Delay(250);
}
a.Bus.Publish(new ExternalDefinitionsInvalidated());
a.Bus.Publish(new InboundApiMethodInvalidated("MesMoveIn"));
a.Bus.Publish(new SitePskInvalidated("site-a"));
deadline = DateTime.UtcNow + TimeSpan.FromSeconds(10);
while (!b.Handler.Applied.Any(i => i is SitePskInvalidated))
{
Assert.True(DateTime.UtcNow < deadline, "node B did not receive the later invalidations");
await Task.Delay(100);
}
var applied = b.Handler.Applied;
Assert.Contains(applied, i => i is RoleMappingsInvalidated);
Assert.Contains(new ExternalDefinitionsInvalidated(), applied);
Assert.Contains(new InboundApiMethodInvalidated("MesMoveIn"), applied);
// The PSK invalidation arrives twice during the migration window (shared + legacy topic).
Assert.Contains(new SitePskInvalidated("site-a"), applied);
// The publisher skips its own envelopes; only the bare legacy PSK message (no origin)
// comes back to it, which is the pre-existing #96 behaviour and idempotent.
await Task.Delay(500);
Assert.All(a.Handler.Applied, i => Assert.IsType<SitePskInvalidated>(i));
}
}
@@ -6,6 +6,7 @@ using NSubstitute;
using ZB.MOM.WW.ScadaBridge.Commons.Entities.ExternalSystems; using ZB.MOM.WW.ScadaBridge.Commons.Entities.ExternalSystems;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories; using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services; using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Communication;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Management; using ZB.MOM.WW.ScadaBridge.Commons.Messages.Management;
using ZB.MOM.WW.ScadaBridge.ExternalSystemGateway; using ZB.MOM.WW.ScadaBridge.ExternalSystemGateway;
using ZB.MOM.WW.ScadaBridge.Security; using ZB.MOM.WW.ScadaBridge.Security;
@@ -23,6 +24,9 @@ public class ExternalDefinitionCacheInvalidationTests : TestKit
private readonly IExternalSystemRepository _repo = Substitute.For<IExternalSystemRepository>(); private readonly IExternalSystemRepository _repo = Substitute.For<IExternalSystemRepository>();
// A TTL far longer than the test: only an explicit InvalidateAll can make the cache re-fetch. // A TTL far longer than the test: only an explicit InvalidateAll can make the cache re-fetch.
private readonly ExternalDefinitionLookupCache _cache = new(TimeSpan.FromHours(1)); private readonly ExternalDefinitionLookupCache _cache = new(TimeSpan.FromHours(1));
// The other central node's cache, reached only through the peer invalidation (audit D3 / #83).
private readonly ExternalDefinitionLookupCache _peerCache = new(TimeSpan.FromHours(1));
private readonly PeerCacheInvalidationPublisher _publisher;
private readonly ServiceCollection _services = new ServiceCollection().AddTestConfigurationTransaction(); private readonly ServiceCollection _services = new ServiceCollection().AddTestConfigurationTransaction();
/// <summary>Initializes the harness with one external system, one method and one DB connection.</summary> /// <summary>Initializes the harness with one external system, one method and one DB connection.</summary>
@@ -37,6 +41,8 @@ public class ExternalDefinitionCacheInvalidationTests : TestKit
_services.AddScoped(_ => _repo); _services.AddScoped(_ => _repo);
_services.AddScoped(_ => Substitute.For<IAuditService>()); _services.AddScoped(_ => Substitute.For<IAuditService>());
_services.AddSingleton(_cache); _services.AddSingleton(_cache);
_publisher = new PeerCacheInvalidationPublisher(_peerCache);
_services.AddSingleton<ICentralCacheInvalidationPublisher>(_publisher);
} }
private IActorRef CreateActor() => private IActorRef CreateActor() =>
@@ -103,4 +109,50 @@ public class ExternalDefinitionCacheInvalidationTests : TestKit
Assert.False(await IsStillCached()); // dropped by the edit, so the lookup re-fetched Assert.False(await IsStillCached()); // dropped by the edit, so the lookup re-fetched
} }
} }
/// <summary>
/// Audit D3 / #83 (ScadaBridge#100 item 3): every definition edit made through this node
/// drops the peer central node's cache too, instead of leaving it to <c>DefinitionCacheTtl</c>.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task EveryDefinitionMutation_InvalidatesThePeerCache()
{
object[] commands =
[
new CreateExternalSystemCommand("New", "http://n", "None", null),
new UpdateExternalSystemCommand(1, "Erp", "http://erp2", "None", null),
new DeleteExternalSystemCommand(1),
new CreateExternalSystemMethodCommand(1, "M", "GET", "/m"),
new UpdateExternalSystemMethodCommand(2, Path: "/y"),
new DeleteExternalSystemMethodCommand(2),
new CreateDatabaseConnectionDefCommand("Db2", "Server=y"),
new UpdateDatabaseConnectionDefCommand(3, "Db", "Server=z"),
new DeleteDatabaseConnectionDefCommand(3),
];
var actor = CreateActor();
foreach (var command in commands)
{
var fetches = 0;
Task<ExternalSystemDefinition?> Fetch()
{
fetches++;
return Task.FromResult<ExternalSystemDefinition?>(new ExternalSystemDefinition("Erp", "http://erp", "None"));
}
_peerCache.InvalidateAll();
await _peerCache.GetSystemAsync("Erp", Fetch);
await _peerCache.GetSystemAsync("Erp", Fetch);
Assert.Equal(1, fetches); // positive control: the peer serves from its cache
actor.Tell(Admin(command));
ExpectMsg<ManagementSuccess>(TimeSpan.FromSeconds(5));
await _peerCache.GetSystemAsync("Erp", Fetch);
Assert.Equal(2, fetches); // the peer re-fetched
}
Assert.Equal(commands.Length, _publisher.Published.Count);
Assert.All(_publisher.Published, i => Assert.IsType<ExternalDefinitionsInvalidated>(i));
}
} }
@@ -0,0 +1,30 @@
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Communication;
namespace ZB.MOM.WW.ScadaBridge.ManagementService.Tests;
/// <summary>
/// Test stand-in for the central cache-invalidation bus (ScadaBridge#100 item 3): records every
/// published invalidation and hands it to the "peer" node's handlers, as the subscriber on the
/// other central node would.
/// </summary>
internal sealed class PeerCacheInvalidationPublisher : ICentralCacheInvalidationPublisher
{
private readonly ICentralCacheInvalidationHandler[] _peerHandlers;
/// <summary>Creates the publisher.</summary>
/// <param name="peerHandlers">The peer node's handlers.</param>
public PeerCacheInvalidationPublisher(params ICentralCacheInvalidationHandler[] peerHandlers) =>
_peerHandlers = peerHandlers;
/// <summary>Gets every invalidation published.</summary>
public List<ICentralCacheInvalidation> Published { get; } = new();
/// <inheritdoc />
public void Publish(ICentralCacheInvalidation invalidation)
{
lock (Published) Published.Add(invalidation);
foreach (var handler in _peerHandlers)
handler.Apply(invalidation);
}
}
@@ -6,6 +6,7 @@ using NSubstitute;
using ZB.MOM.WW.ScadaBridge.Commons.Entities.Security; using ZB.MOM.WW.ScadaBridge.Commons.Entities.Security;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories; using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services; using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Communication;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Management; using ZB.MOM.WW.ScadaBridge.Commons.Messages.Management;
using ZB.MOM.WW.ScadaBridge.Security; using ZB.MOM.WW.ScadaBridge.Security;
@@ -20,6 +21,9 @@ public class RoleMappingCacheInvalidationTests : TestKit
{ {
private readonly ISecurityRepository _repo = Substitute.For<ISecurityRepository>(); private readonly ISecurityRepository _repo = Substitute.For<ISecurityRepository>();
private readonly RoleMappingCache _cache = new(TimeProvider.System); private readonly RoleMappingCache _cache = new(TimeProvider.System);
// The other central node's cache, reached only through the peer invalidation (audit D1).
private readonly RoleMappingCache _peerCache = new(TimeProvider.System);
private readonly PeerCacheInvalidationPublisher _publisher;
private readonly ServiceCollection _services = new ServiceCollection().AddTestConfigurationTransaction(); private readonly ServiceCollection _services = new ServiceCollection().AddTestConfigurationTransaction();
private List<LdapGroupMapping> _mappings = [new("Ops", "Deployer") { Id = 5 }]; private List<LdapGroupMapping> _mappings = [new("Ops", "Deployer") { Id = 5 }];
@@ -33,6 +37,8 @@ public class RoleMappingCacheInvalidationTests : TestKit
_services.AddScoped(_ => _repo); _services.AddScoped(_ => _repo);
_services.AddScoped(_ => Substitute.For<IAuditService>()); _services.AddScoped(_ => Substitute.For<IAuditService>());
_services.AddSingleton(_cache); _services.AddSingleton(_cache);
_publisher = new PeerCacheInvalidationPublisher(_peerCache);
_services.AddSingleton<ICentralCacheInvalidationPublisher>(_publisher);
} }
private IActorRef CreateActor() => private IActorRef CreateActor() =>
@@ -84,4 +90,45 @@ public class RoleMappingCacheInvalidationTests : TestKit
await _repo.Received(1).GetAllMappingsAsync(Arg.Any<CancellationToken>()); // re-read, not served stale await _repo.Received(1).GetAllMappingsAsync(Arg.Any<CancellationToken>()); // re-read, not served stale
} }
} }
/// <summary>
/// Audit D1 (ScadaBridge#100 item 3): a revocation made through this node reaches the peer
/// central node's snapshot at once, instead of after <c>RoleMappingCacheSeconds</c>.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task DeleteRoleMapping_IsVisibleOnThePeerNode()
{
var peerMapper = new RoleMapper(_repo, _peerCache);
Assert.Contains("Deployer", (await peerMapper.MapGroupsToRolesAsync(["Ops"])).Roles); // primes the peer
_mappings = [];
CreateActor().Tell(Admin(new DeleteRoleMappingCommand(5)));
ExpectMsg<ManagementSuccess>(TimeSpan.FromSeconds(5));
Assert.Empty((await peerMapper.MapGroupsToRolesAsync(["Ops"])).Roles);
}
/// <summary>Every mapping / scope-rule mutation tells the peer, once, after commit.</summary>
[Fact]
public void EveryMappingMutation_TellsThePeer()
{
object[] commands =
[
new CreateRoleMappingCommand("Eng", "Designer"),
new UpdateRoleMappingCommand(5, "Ops", "Viewer"),
new DeleteRoleMappingCommand(5),
new AddScopeRuleCommand(5, 2),
new DeleteScopeRuleCommand(9),
];
var actor = CreateActor();
foreach (var command in commands)
{
actor.Tell(Admin(command));
ExpectMsg<ManagementSuccess>(TimeSpan.FromSeconds(5));
}
Assert.Equal(commands.Length, _publisher.Published.Count);
Assert.All(_publisher.Published, i => Assert.IsType<RoleMappingsInvalidated>(i));
}
} }
@@ -13,6 +13,7 @@ using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Scripting; using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Scripting;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services; using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Transport; using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Transport;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Communication;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Scripting; using ZB.MOM.WW.ScadaBridge.Commons.Messages.Scripting;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Transport; using ZB.MOM.WW.ScadaBridge.Commons.Types.Transport;
@@ -48,6 +49,7 @@ public sealed class BundleImporterApplyTests : IDisposable
private readonly ServiceProvider _provider; private readonly ServiceProvider _provider;
private readonly RecordingScriptArtifactChangeBus _artifactBus = new(); private readonly RecordingScriptArtifactChangeBus _artifactBus = new();
private readonly CountingDefinitionCacheInvalidator _definitionCache = new(); private readonly CountingDefinitionCacheInvalidator _definitionCache = new();
private readonly RecordingCacheInvalidationPublisher _peerPublisher = new();
/// <summary>Initializes a new instance of the <c>BundleImporterApplyTests</c> class.</summary> /// <summary>Initializes a new instance of the <c>BundleImporterApplyTests</c> class.</summary>
public BundleImporterApplyTests() public BundleImporterApplyTests()
@@ -103,6 +105,8 @@ public sealed class BundleImporterApplyTests : IDisposable
services.AddSingleton<IScriptArtifactChangeBus>(_artifactBus); services.AddSingleton<IScriptArtifactChangeBus>(_artifactBus);
// Gitea #83: observable post-commit drop of the external-definition lookup cache. // Gitea #83: observable post-commit drop of the external-definition lookup cache.
services.AddSingleton<IExternalDefinitionCacheInvalidator>(_definitionCache); services.AddSingleton<IExternalDefinitionCacheInvalidator>(_definitionCache);
// ScadaBridge#100 item 3: observable post-commit broadcast to the peer central node.
services.AddSingleton<ICentralCacheInvalidationPublisher>(_peerPublisher);
// The stale-instance probe is implemented in DeploymentManager // The stale-instance probe is implemented in DeploymentManager
// and the flatten/hash primitives in TemplateEngine — register both so // and the flatten/hash primitives in TemplateEngine — register both so
// BundleImporter resolves a real IStaleInstanceProbe and ApplyAsync can // BundleImporter resolves a real IStaleInstanceProbe and ApplyAsync can
@@ -1946,6 +1950,8 @@ public sealed class BundleImporterApplyTests : IDisposable
} }
Assert.Equal(1, _definitionCache.Calls); Assert.Equal(1, _definitionCache.Calls);
// ScadaBridge#100 item 3 / #83: the peer central node is told to drop its copy too.
Assert.Contains(_peerPublisher.Published, i => i is ExternalDefinitionsInvalidated);
} }
/// <summary>Verifies that a failed apply publishes no notifications.</summary> /// <summary>Verifies that a failed apply publishes no notifications.</summary>
@@ -1998,6 +2004,20 @@ public sealed class BundleImporterApplyTests : IDisposable
Assert.Empty(_artifactBus.Received); Assert.Empty(_artifactBus.Received);
// Gitea #83: nothing committed, so the definition cache is left alone. // Gitea #83: nothing committed, so the definition cache is left alone.
Assert.Equal(0, _definitionCache.Calls); Assert.Equal(0, _definitionCache.Calls);
Assert.Empty(_peerPublisher.Published);
}
/// <summary>Records peer cache invalidations (ScadaBridge#100 item 3).</summary>
private sealed class RecordingCacheInvalidationPublisher : ICentralCacheInvalidationPublisher
{
/// <summary>Gets the invalidations published.</summary>
public List<ICentralCacheInvalidation> Published { get; } = new();
/// <inheritdoc />
public void Publish(ICentralCacheInvalidation invalidation)
{
lock (Published) Published.Add(invalidation);
}
} }
/// <summary>Counts post-commit external-definition cache invalidations (Gitea #83).</summary> /// <summary>Counts post-commit external-definition cache invalidations (Gitea #83).</summary>