feat(central): one central cache-invalidation topic; role-mapping and external-definition caches follow peer edits (ScadaBridge#100 item 3, ScadaBridge#83)
The #96 site-psk-invalidation topic was the first cross-node invalidation broadcast. It is generalised into ONE central DistributedPubSub topic, central-cache-invalidation, carrying typed messages (Commons ICentralCacheInvalidation: SitePskInvalidated, RoleMappingsInvalidated, ExternalDefinitionsInvalidated, InboundApiMethodInvalidated) in an envelope stamped with the publisher's process-unique origin id. - Host: CentralCacheInvalidationBus (ICentralCacheInvalidationPublisher, no-op until the mediator is set, never throws) and CentralCacheInvalidationSubscriber, which skips its own node's envelopes and hands the peer's to every registered ICentralCacheInvalidationHandler (a failing handler does not stop the rest). Replaces SitePskInvalidationSubscriber. - Migration of #96: SitePskRotationInvalidator publishes through the bus and is the PSK handler; SitePskInvalidated keeps its type name and is also published bare on site-psk-invalidation, which the subscriber still listens to, so a mixed-version central pair keeps exchanging PSK invalidations. Remove after one release. - Audit D1: RoleMappingCache is a handler; ManagementActor's post-commit role-mapping invalidation also publishes RoleMappingsInvalidated, so a revocation is immediate on the peer instead of after RoleMappingCacheSeconds. - Audit D3 / #83 (peer half): ExternalDefinitionLookupCache is a handler; the 9 ManagementActor handlers and the Transport bundle importer publish ExternalDefinitionsInvalidated after commit. (#83's local import half was already in.) TTLs remain the fallback for a lost broadcast or a direct SQL edit. Tests: bus and subscriber unit tests (origin filter, legacy topic), peer-node tests through the real caches in ManagementService.Tests, importer publish, composition root, and a real two-node cluster from the production HOCON (CentralCacheInvalidationTwoNodeTests) proving every message type crosses with the production serializer. Docs: Host, Communication, Security, ManagementService, ESG, and the #85 cache audit. Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
@@ -23,12 +23,12 @@ node after a change is made through the other one. Same defect class as #83
|
||||
|---|---|---|---|---|
|
||||
| `InboundScriptExecutor` compiled handlers (InboundAPI) | Compiled method delegates | `InvalidateMethod` on delete; change bus on import | **None** — every request revision-checks the cached handler against the resolved `ApiMethod.Script` and recompiles on mismatch | OK |
|
||||
| `ApiMethodCache` (InboundAPI) | `ApiMethod` rows, incl. not-found | Change bus (bundle import); **management create/update/delete — added by #85** | `InboundApi:ApiMethodCacheTtl`, default **5 s** | OK (local gap fixed, see below) |
|
||||
| `ExternalDefinitionLookupCache` (ESG, central only) | External systems / methods / DB connections (found only) | 9 ManagementActor handlers (#65); bundle import (#83) | `ExternalSystemGateway:DefinitionCacheTtl`, **10 s** on central | Owner decision — #83 |
|
||||
| `RoleMappingCache` (Security) — the cache at `ManagementActor` `InvalidateRoleMappingCacheAfterCommit` | LDAP group → role map + Deployer site scopes | Mapping / scope-rule commands, post-commit | `Security:RoleMappingCacheSeconds`, default **30 s** (`0` disables) | Owner decision — see D1 |
|
||||
| `ExternalDefinitionLookupCache` (ESG, central only) | External systems / methods / DB connections (found only) | 9 ManagementActor handlers (#65); bundle import (#83) | Edit or import through the other node: **none** (`ExternalDefinitionsInvalidated` on `central-cache-invalidation`, #100); direct SQL: `ExternalSystemGateway:DefinitionCacheTtl`, **10 s** on central | Fixed — #100 item 3 (see D3) |
|
||||
| `RoleMappingCache` (Security) — the cache at `ManagementActor` `InvalidateRoleMappingCacheAfterCommit` | LDAP group → role map + Deployer site scopes | Mapping / scope-rule commands, post-commit | **None** (`RoleMappingsInvalidated` on `central-cache-invalidation`, #100); TTL `Security:RoleMappingCacheSeconds` (**30 s**) only if the broadcast is lost | Fixed — #100 item 3 (see D1) |
|
||||
| `PendingUpdatesCache` (DeploymentManager, #91) | Fleet pending-updates snapshot | Deployment finished on this node | `DeploymentManager:PendingUpdatesCacheTtl`, default **60 s** | OK — display only |
|
||||
| `CentralCommunicationActor` site-address cache (+ `GrpcSiteTransport` channel pairs) | Site gRPC endpoints | Site create/update/delete → `RefreshSiteAddresses` | Periodic DB reload, **60 s** (hard-coded) | OK — rare edit, self-heals |
|
||||
| `SiteStreamGrpcClientFactory` clients | Per (site, endpoint) stream clients | Site delete → `RemoveSiteAsync` (local only) | Not stale data: a removed site's client on the peer is reclaimed by idle eviction, `Communication:GrpcClientIdleTtl` (**30 min**) × 1.5 after last use | OK — resource, not correctness |
|
||||
| `SitePskProvider` (Host) | Per-site gRPC PSK from config / secret store | Site removal; `SB-GRPC-PSK-{site}` write via `/admin/secrets`; site auth rejection; `SitePsks` reload (#96) | Write via `/admin/secrets`: **none** (peer told over `DistributedPubSub` topic `site-psk-invalidation`); other writes: first rejected call after `Secrets:ResolveCacheTtl` | Fixed — #96 (see D2) |
|
||||
| `SitePskProvider` (Host) | Per-site gRPC PSK from config / secret store | Site removal; `SB-GRPC-PSK-{site}` write via `/admin/secrets`; site auth rejection; `SitePsks` reload (#96) | Write via `/admin/secrets`: **none** (peer told over `DistributedPubSub` topic `central-cache-invalidation`, formerly `site-psk-invalidation`); other writes: first rejected call after `Secrets:ResolveCacheTtl` | Fixed — #96 (see D2) |
|
||||
| `DefaultSecretResolver` (shared `ZB.MOM.WW.Secrets`) | Decrypted secret values | Lib's own invalidator (UI writes, sync service) | `Secrets:ResolveCacheTtl`, **30 s** | OK — owned by the shared lib |
|
||||
| `KpiSnapshotCache` (CentralUI) | KPI query results (operational data) | — | **8 s** | OK — not config |
|
||||
| `SharedAlarmSummaryService` (CentralUI) | Per-site alarm summary fan-out | — | **12 s** cold-cache window | OK — not config |
|
||||
@@ -75,3 +75,16 @@ broadcast was added.
|
||||
- **D3 — `ExternalDefinitionLookupCache` peer bound (10 s).** Already tracked by #83; the
|
||||
recommendation there (accept and document) still stands. If D1 gets a broadcast topic, this cache
|
||||
and `ApiMethodCache` should join it rather than add a second mechanism.
|
||||
|
||||
## Follow-up: one central cache-invalidation topic (#100 item 3, 2026-11-09)
|
||||
|
||||
D1 and D3 now ride a single `DistributedPubSub` topic, `central-cache-invalidation`, with typed
|
||||
messages (`ICentralCacheInvalidation`: `SitePskInvalidated`, `RoleMappingsInvalidated`,
|
||||
`ExternalDefinitionsInvalidated`, `InboundApiMethodInvalidated`) instead of one topic per cache.
|
||||
The #96 topic was migrated onto it: `SitePskInvalidated` travels on the shared topic, and for one
|
||||
release is also published bare on `site-psk-invalidation` (still subscribed), so a mixed-version
|
||||
central pair keeps working. Each envelope carries the publisher's origin id and the subscriber
|
||||
skips its own. Publishers: ManagementActor post-commit (role mappings, external definitions),
|
||||
the Transport bundle importer (external definitions) and the PSK invalidator. Every cache keeps its TTL / revision check as the fallback for a
|
||||
lost broadcast or an out-of-band (SQL) edit. Design: Component-Host.md → *Central cache
|
||||
invalidation*; tests include a real two-node cluster (`CentralCacheInvalidationTwoNodeTests`).
|
||||
|
||||
Reference in New Issue
Block a user