feat(central): one central cache-invalidation topic; role-mapping and external-definition caches follow peer edits (ScadaBridge#100 item 3, ScadaBridge#83)

The #96 site-psk-invalidation topic was the first cross-node invalidation broadcast.
It is generalised into ONE central DistributedPubSub topic, central-cache-invalidation,
carrying typed messages (Commons ICentralCacheInvalidation: SitePskInvalidated,
RoleMappingsInvalidated, ExternalDefinitionsInvalidated, InboundApiMethodInvalidated)
in an envelope stamped with the publisher's process-unique origin id.

- Host: CentralCacheInvalidationBus (ICentralCacheInvalidationPublisher, no-op until
  the mediator is set, never throws) and CentralCacheInvalidationSubscriber, which
  skips its own node's envelopes and hands the peer's to every registered
  ICentralCacheInvalidationHandler (a failing handler does not stop the rest).
  Replaces SitePskInvalidationSubscriber.
- Migration of #96: SitePskRotationInvalidator publishes through the bus and is the
  PSK handler; SitePskInvalidated keeps its type name and is also published bare on
  site-psk-invalidation, which the subscriber still listens to, so a mixed-version
  central pair keeps exchanging PSK invalidations. Remove after one release.
- Audit D1: RoleMappingCache is a handler; ManagementActor's post-commit role-mapping
  invalidation also publishes RoleMappingsInvalidated, so a revocation is immediate on
  the peer instead of after RoleMappingCacheSeconds.
- Audit D3 / #83 (peer half): ExternalDefinitionLookupCache is a handler; the 9
  ManagementActor handlers and the Transport bundle importer publish
  ExternalDefinitionsInvalidated after commit. (#83's local import half was already in.)

TTLs remain the fallback for a lost broadcast or a direct SQL edit. Tests: bus and
subscriber unit tests (origin filter, legacy topic), peer-node tests through the real
caches in ManagementService.Tests, importer publish, composition root, and a real
two-node cluster from the production HOCON (CentralCacheInvalidationTwoNodeTests)
proving every message type crosses with the production serializer. Docs: Host,
Communication, Security, ManagementService, ESG, and the #85 cache audit.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
2026-11-09 15:50:03 -05:00
parent 063fb21815
commit 153609ef8a
27 changed files with 794 additions and 110 deletions
+3 -2
View File
@@ -207,8 +207,9 @@ site can omit it.
Central caches each resolved key, and drops the entry (Gitea #96) when: the site is removed; the
secret `SB-GRPC-PSK-<siteId>` is written through central's secrets write path (`/admin/secrets`) —
`SitePskRotationInvalidator` wraps the secrets library's `ISecretCacheInvalidator` and also
broadcasts a `SitePskInvalidated` on the central `DistributedPubSub` topic `site-psk-invalidation`
so the peer central node drops its copy too; the site rejects a call with `PermissionDenied` /
broadcasts a `SitePskInvalidated` on the central cache-invalidation topic `central-cache-invalidation`
(ScadaBridge#100 item 3; the #96 topic `site-psk-invalidation` is kept for one release for
mixed-version pairs — see Component-Host.md) so the peer central node drops its copy too; the site rejects a call with `PermissionDenied` /
`Unauthenticated` (`PskRejectionRefreshHandler`, wrapped around every per-site channel by
`WithSiteCredentials`, so the next call re-reads the key — this covers keys rotated outside the
process, e.g. by the `secret` CLI, after at most the resolver's `Secrets:ResolveCacheTtl`); a
@@ -120,7 +120,7 @@ Scripts choose between two call modes per invocation, mirroring the dual-mode da
- Database connections use **standard ADO.NET connection pooling** per named connection. No custom pool management.
- Pool behavior (max pool size, connection lifetime, etc.) can be tuned via connection string parameters in the database connection definition if needed.
- **Definition cache (central only).** `ExternalDefinitionLookupCache` holds resolved external-system, method and database-connection definitions for `ExternalSystemGateway:DefinitionCacheTtl` (default 0 = off in the options; the central host sets 10 s because each lookup there is a SQL Server round-trip). Only found definitions are cached, so a new definition is visible immediately; an edit/delete within one TTL. Sites keep it off — `SiteExternalSystemRepository` already serves external systems from the deploy-invalidated `ExternalSystemDefinitionCache` snapshot.
- **Definition cache (central only).** `ExternalDefinitionLookupCache` holds resolved external-system, method and database-connection definitions for `ExternalSystemGateway:DefinitionCacheTtl` (default 0 = off in the options; the central host sets 10 s because each lookup there is a SQL Server round-trip). Only found definitions are cached, so a new definition is visible immediately. A ManagementActor edit (#65) or a committed Transport bundle import (#83) drops the cache on the node that made the change and publishes `ExternalDefinitionsInvalidated` on the central cache-invalidation topic so the peer central node drops its copy too (ScadaBridge#100 item 3); an edit that reaches neither path (direct SQL) is visible within one TTL. Sites keep it off — `SiteExternalSystemRepository` already serves external systems from the deploy-invalidated `ExternalSystemDefinitionCache` snapshot.
- GET/DELETE parameters are rendered into the query string with scalars via `ToString()` and collections/objects JSON-serialised (never a CLR type name).
- Synchronous failures on `Database.Connection()` (e.g., unreachable server) return an error to the calling script, consistent with external system permanent failure handling.
+2 -1
View File
@@ -351,5 +351,6 @@ built)* section for where the shipped code deviates from the original design).
- **Port options come from the bound `NodeOptions`.** `Program.cs` binds `ScadaBridge:Node` once and reads `GrpcPort` / `CentralGrpcPort` / `MetricsPort` from it; the 8083/8084 defaults live only in `NodeOptions`. Central's dedicated metrics listener remains an explicit *presence* check on the raw key (a defaulted read would bind a new port on every existing central node).
- **Inbound method warm-up is a hosted service** (`InboundMethodWarmupService`, registered after `AkkaHostedService`), not a pre-`RunAsync` loop — see Component-InboundAPI.md.
- **`SitePskProvider` distinguishes a secret-store fault from a missing secret.** A store fault logs a Warning with the exception and throws `SecretStoreUnavailableException` (not cached); only a genuinely absent/empty/tombstoned secret produces the "seed `SB-GRPC-PSK-{site}`" message.
- **`SitePskProvider`'s cache follows an in-place rotation (Gitea #96).** Central replaces the secrets library's `ISecretCacheInvalidator` with `SitePskRotationInvalidator` (`AddSitePskRotationInvalidation`), which drops the site's cached key when `SB-GRPC-PSK-{site}` is written and publishes `SitePskInvalidated` on the `site-psk-invalidation` `DistributedPubSub` topic; `AkkaHostedService.RegisterCentralActors` starts the `SitePskInvalidationSubscriber` that applies the peer's broadcasts. A site's auth rejection and a `SitePsks` reload also drop the cache — see Component-Communication.md → *Authentication*.
- **`SitePskProvider`'s cache follows an in-place rotation (Gitea #96).** Central replaces the secrets library's `ISecretCacheInvalidator` with `SitePskRotationInvalidator` (`AddSitePskRotationInvalidation`), which drops the site's cached key when `SB-GRPC-PSK-{site}` is written and publishes `SitePskInvalidated` to the peer over the central cache-invalidation topic (below). A site's auth rejection and a `SitePsks` reload also drop the cache — see Component-Communication.md → *Authentication*.
- **Central cache invalidation (ScadaBridge#100 item 3).** One `DistributedPubSub` topic, `central-cache-invalidation`, carries typed `ICentralCacheInvalidation` messages (Commons `Messages/Communication/CentralCacheInvalidations.cs`) between the two central nodes, each wrapped in a `CentralCacheInvalidationEnvelope` stamped with the publishing process's origin id. `AddCentralCacheInvalidation` registers `CentralCacheInvalidationBus` as `ICentralCacheInvalidationPublisher`; `AkkaHostedService.RegisterCentralActors` gives it the mediator and starts `CentralCacheInvalidationSubscriber`, which skips its own node's envelopes and hands the peer's to every registered `ICentralCacheInvalidationHandler`: `SitePskRotationInvalidator` (`SitePskInvalidated`, #96), `RoleMappingCache` (`RoleMappingsInvalidated`, audit D1), and `ExternalDefinitionLookupCache` (`ExternalDefinitionsInvalidated`, audit D3 / #83). Publishers invalidate locally first, post-commit, then publish; delivery is best-effort, so each cache's TTL or revision check stays the fallback. Migration: the #96 topic `site-psk-invalidation` is still published to (bare `SitePskInvalidated`) and subscribed, so a briefly mixed-version central pair keeps exchanging PSK invalidations — remove it once every central node runs this build.
- Central registers `ExternalSystemGatewayOptions.DefinitionCacheTtl = 10 s` ahead of `AddExternalSystemGateway()` so an explicit config value wins; sites leave it off.
@@ -194,7 +194,7 @@ The two-person authorization workflow for writes through the MxAccess Gateway. B
- **ListApiKeys** / **CreateApiKey** / **UpdateApiKey** / **EnableApiKey** / **DisableApiKey** / **DeleteApiKey**: Manage API keys.
- **ListRoleMappings** / **CreateRoleMapping** / **UpdateRoleMapping** / **DeleteRoleMapping**: Manage LDAP group-to-role mappings.
- **ListScopeRules** / **AddScopeRule** / **DeleteScopeRule**: Manage site scope rules on role mappings.
- Every role-mapping and scope-rule mutation (Create/Update/DeleteRoleMapping, Add/DeleteScopeRule) calls `RoleMappingCache.Invalidate()` as a **post-commit** action, so the change takes effect for the next role mapping on the node that served the command instead of after the `RoleMappingCacheSeconds` TTL (default 30 s); the other central node still converges within that TTL. Post-commit, because invalidating inside the transaction would let a concurrent mapping re-cache the pre-change table.
- Every role-mapping and scope-rule mutation (Create/Update/DeleteRoleMapping, Add/DeleteScopeRule) calls `RoleMappingCache.Invalidate()` as a **post-commit** action, so the change takes effect for the next role mapping on the node that served the command instead of after the `RoleMappingCacheSeconds` TTL (default 30 s), and publishes `RoleMappingsInvalidated` on the central cache-invalidation topic so the other central node drops its snapshot too (audit D1, ScadaBridge#100 item 3); the TTL is only the fallback if that best-effort broadcast is lost. The external-definition edit handlers do the same with `ExternalDefinitionsInvalidated` (#65, #83). Post-commit, because invalidating inside the transaction would let a concurrent mapping re-cache the pre-change table.
### Audit Log
+1 -1
View File
@@ -53,7 +53,7 @@ Set in a local or docker-dev environment via the environment variable `ScadaBrid
- **Idle timeout**: Configurable, default **30 minutes**. Without eligible activity within the idle window, the session is rejected and the user must re-login. Passive `/auth/ping`, activity prevalidation, and Blazor transport requests preserve the activity anchor and suppress sliding renewal; the activity endpoint renews only after authentication and antiforgery validation. Tracked via a `LastActivity` last-activity timestamp claim. The cookie's `ExpireTimeSpan` is set to the idle timeout and `SlidingExpiration` renews it on activity, so the cookie window and the explicit `OnValidatePrincipal` idle check use the **same** value and cannot contradict each other.
- **Role-mapping refresh (LDAP-free)**: Configurable, default **15 minutes** (`SecurityOptions.RoleRefreshThresholdMinutes`). At login the session stores the user's raw LDAP groups (one `zb:group` claim each) plus a `zb:lastrolerefresh` anchor. Once the anchor is older than the threshold, `OnValidatePrincipal` re-runs the **DB-backed** `RoleMapper` on the stored groups — **with no LDAP call** — rebuilds the role/scope claims via the shared claim-builder, advances the anchor, and re-issues the cookie. Central role-mapping (DB) changes — including a **revoked** mapping that drops the user's roles, and changed site-scope rules — take effect within this window. Ordinary HTTP validation uses this threshold. A live circuit refreshes mappings in fresh scopes on authentication-state reads — display reads (`GetAuthenticationStateAsync`) at most once per `CookieAuthenticationStateProvider.NonAuthoritativeRefreshInterval` (10 s) per circuit — and mutation entrypoints (`GetAuthoritativeStateAsync`) require a successful current mapping before every dispatch.
- **Role-mapping cache**: `RoleMapper` reads the LDAP-group→role table and the scope rules of every Deployer mapping through a process-wide `RoleMappingCache` snapshot (`SecurityOptions.RoleMappingCacheSeconds`, default **30**; `0` disables). One refresh loads the whole table plus all Deployer scope rules, so a cache hit costs no database round-trip — previously every Basic-authenticated management request and every circuit auth-state read issued `GetAllMappingsAsync` plus one `GetScopeRulesForMappingAsync` per matched Deployer mapping. Without an explicit invalidation a mapping or scope-rule change is therefore visible on every node within the TTL. `RoleMappingCache.Invalidate()` discards the calling node's snapshot so its next mapping re-reads the database (a refresh started before an invalidation is never published over it); other nodes still rely on the TTL. `RoleMappingCacheSeconds = 0` bypasses the snapshot and reads only the matched mappings' scope rules per call.
- **Role-mapping cache**: `RoleMapper` reads the LDAP-group→role table and the scope rules of every Deployer mapping through a process-wide `RoleMappingCache` snapshot (`SecurityOptions.RoleMappingCacheSeconds`, default **30**; `0` disables). One refresh loads the whole table plus all Deployer scope rules, so a cache hit costs no database round-trip — previously every Basic-authenticated management request and every circuit auth-state read issued `GetAllMappingsAsync` plus one `GetScopeRulesForMappingAsync` per matched Deployer mapping. Without an explicit invalidation a mapping or scope-rule change is therefore visible on every node within the TTL. `RoleMappingCache.Invalidate()` discards the calling node's snapshot so its next mapping re-reads the database (a refresh started before an invalidation is never published over it); the Management Service's mapping/scope-rule handlers also publish `RoleMappingsInvalidated` on the central cache-invalidation topic, so the peer central node discards its snapshot too (audit D1, ScadaBridge#100 item 3) and the TTL is only the fallback. `RoleMappingCacheSeconds = 0` bypasses the snapshot and reads only the matched mappings' scope rules per call.
#### Session Validation (`OnValidatePrincipal`)
- The cookie principal is built at login by a **single shared claim-builder** (`SessionClaimBuilder`). The `OnValidatePrincipal` role-refresh path rebuilds the principal through the **same** builder, so the login and refresh claim shapes cannot drift.