The #96 site-psk-invalidation topic was the first cross-node invalidation broadcast. It is generalised into ONE central DistributedPubSub topic, central-cache-invalidation, carrying typed messages (Commons ICentralCacheInvalidation: SitePskInvalidated, RoleMappingsInvalidated, ExternalDefinitionsInvalidated, InboundApiMethodInvalidated) in an envelope stamped with the publisher's process-unique origin id. - Host: CentralCacheInvalidationBus (ICentralCacheInvalidationPublisher, no-op until the mediator is set, never throws) and CentralCacheInvalidationSubscriber, which skips its own node's envelopes and hands the peer's to every registered ICentralCacheInvalidationHandler (a failing handler does not stop the rest). Replaces SitePskInvalidationSubscriber. - Migration of #96: SitePskRotationInvalidator publishes through the bus and is the PSK handler; SitePskInvalidated keeps its type name and is also published bare on site-psk-invalidation, which the subscriber still listens to, so a mixed-version central pair keeps exchanging PSK invalidations. Remove after one release. - Audit D1: RoleMappingCache is a handler; ManagementActor's post-commit role-mapping invalidation also publishes RoleMappingsInvalidated, so a revocation is immediate on the peer instead of after RoleMappingCacheSeconds. - Audit D3 / #83 (peer half): ExternalDefinitionLookupCache is a handler; the 9 ManagementActor handlers and the Transport bundle importer publish ExternalDefinitionsInvalidated after commit. (#83's local import half was already in.) TTLs remain the fallback for a lost broadcast or a direct SQL edit. Tests: bus and subscriber unit tests (origin filter, legacy topic), peer-node tests through the real caches in ManagementService.Tests, importer publish, composition root, and a real two-node cluster from the production HOCON (CentralCacheInvalidationTwoNodeTests) proving every message type crosses with the production serializer. Docs: Host, Communication, Security, ManagementService, ESG, and the #85 cache audit. Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
8.8 KiB
Central per-node cache staleness audit (Gitea #85)
Date: 2026-11-09. Scope: every in-process cache on a central node that holds
configuration-database (or secret-store) state, and how stale it can be on the peer central
node after a change is made through the other one. Same defect class as #83
(ExternalDefinitionLookupCache).
Context that sets the risk
- Traefik routes UI,
/managementand/api/*to the active central node only (/health/active). An edit and a use on different nodes therefore only happen when someone talks to a node directly (CLI pointed at a node's own port) or across a failover. - Every TTL below is measured from when the entry was loaded, so an entry a node cached while it was previously active is still bounded by its TTL after it becomes active again.
- No cache below replicates its invalidation to the peer. Central already has one cross-node
channel: the
DistributedPubSubtopicCentralCommunicationActor.HealthReportTopic(peer health replication). A config-invalidation broadcast, if an owner wants one, should be a second topic on the same mediator, published fromAfterConfigurationCommit.
Inventory
| Cache | Holds | Local invalidation | Peer staleness bound | Verdict |
|---|---|---|---|---|
InboundScriptExecutor compiled handlers (InboundAPI) |
Compiled method delegates | InvalidateMethod on delete; change bus on import |
None — every request revision-checks the cached handler against the resolved ApiMethod.Script and recompiles on mismatch |
OK |
ApiMethodCache (InboundAPI) |
ApiMethod rows, incl. not-found |
Change bus (bundle import); management create/update/delete — added by #85 | InboundApi:ApiMethodCacheTtl, default 5 s |
OK (local gap fixed, see below) |
ExternalDefinitionLookupCache (ESG, central only) |
External systems / methods / DB connections (found only) | 9 ManagementActor handlers (#65); bundle import (#83) | Edit or import through the other node: none (ExternalDefinitionsInvalidated on central-cache-invalidation, #100); direct SQL: ExternalSystemGateway:DefinitionCacheTtl, 10 s on central |
Fixed — #100 item 3 (see D3) |
RoleMappingCache (Security) — the cache at ManagementActor InvalidateRoleMappingCacheAfterCommit |
LDAP group → role map + Deployer site scopes | Mapping / scope-rule commands, post-commit | None (RoleMappingsInvalidated on central-cache-invalidation, #100); TTL Security:RoleMappingCacheSeconds (30 s) only if the broadcast is lost |
Fixed — #100 item 3 (see D1) |
PendingUpdatesCache (DeploymentManager, #91) |
Fleet pending-updates snapshot | Deployment finished on this node | DeploymentManager:PendingUpdatesCacheTtl, default 60 s |
OK — display only |
CentralCommunicationActor site-address cache (+ GrpcSiteTransport channel pairs) |
Site gRPC endpoints | Site create/update/delete → RefreshSiteAddresses |
Periodic DB reload, 60 s (hard-coded) | OK — rare edit, self-heals |
SiteStreamGrpcClientFactory clients |
Per (site, endpoint) stream clients | Site delete → RemoveSiteAsync (local only) |
Not stale data: a removed site's client on the peer is reclaimed by idle eviction, Communication:GrpcClientIdleTtl (30 min) × 1.5 after last use |
OK — resource, not correctness |
SitePskProvider (Host) |
Per-site gRPC PSK from config / secret store | Site removal; SB-GRPC-PSK-{site} write via /admin/secrets; site auth rejection; SitePsks reload (#96) |
Write via /admin/secrets: none (peer told over DistributedPubSub topic central-cache-invalidation, formerly site-psk-invalidation); other writes: first rejected call after Secrets:ResolveCacheTtl |
Fixed — #96 (see D2) |
DefaultSecretResolver (shared ZB.MOM.WW.Secrets) |
Decrypted secret values | Lib's own invalidator (UI writes, sync service) | Secrets:ResolveCacheTtl, 30 s |
OK — owned by the shared lib |
KpiSnapshotCache (CentralUI) |
KPI query results (operational data) | — | 8 s | OK — not config |
SharedAlarmSummaryService (CentralUI) |
Per-site alarm summary fan-out | — | 12 s cold-cache window | OK — not config |
IDeploymentStatusNotifier (DeploymentManager) |
Push of deployment status to UI circuits | In-process event | Peer UI never sees the push, but the UI is only served by the active node and the pages also poll | OK |
ITemplateGraphWatermark / FlattenSession |
Template-chain memo keyed by in-process version | Repository bumps | Session lives for one deploy/validate operation; the watermark is process-local by design (peer starts from 0 → extra work, never stale work) | OK |
ScriptCompileVerdictCache, TrustVerdictCache, AuditRegexCache, ScriptAnalysisService IMemoryCache, OAuth2TokenService |
Pure functions of their key (code hash + surface, pattern, credential hash) | — | Not applicable — a changed input is a different key | OK |
ZB.MOM.WW.Auth.ApiKeys.SqlServer (inbound API keys) |
— | — | No in-process cache in the shared store (checked in the 0.3.0 source) — revocation is immediate on both nodes | OK |
Fixed in #85
ApiMethodCache was not invalidated by the management API even locally. Only the compiled
handler was dropped, so for up to the 5 s TTL on the node that made the change:
- a just-deleted method stayed callable — the executor recompiled it from the cached row;
- a just-created method answered 404 if a not-found had been cached.
ManagementActor now drops the method's cache entry post-commit on create, update and delete
(InvalidateApiMethodCacheAfterCommit), the same post-commit pattern as the role-mapping cache.
The peer is still bounded by the 5 s TTL. Tests: ApiMethodCacheInvalidationTests (3 of 4 fail
without the fix; the fourth is the negative control).
No peer-staleness case was clearly a bug: every config cache with peer staleness is TTL-bounded (5–60 s), its bound is documented at the cache, and the active-only routing means the window is only hit via a direct-to-node call or a failover inside the TTL. So no cross-node broadcast was added.
Needs an owner decision
- D1 —
RoleMappingCacherevocation lag on the peer (30 s). Removing a mapping or narrowing a Deployer's site scope takes effect at once on the node that made the change and within 30 s on the other. It is a security-relevant window (revoked access keeps working briefly), but only on the node the change did not go through. Options: accept and document (current state); lowerRoleMappingCacheSeconds; or add theDistributedPubSubinvalidation topic described above. Recommendation: accept unless the operators' revocation SLA is tighter than 30 s. - D2 —
SitePskProvidercaches a successful PSK resolve forever. Its own remarks sayInvalidatecovers "a key rotated in place", but nothing calls it on rotation — only on site removal. RotatingSB-GRPC-PSK-{site}in the secret store (with the site pair restarted on the new key) leaves both central nodes dialling with the old key until they restart. Options: make central restart part of the rotation runbook; give the cache a TTL; or invalidate on aPermissionDeniedfrom the site. Not peer-specific, so it is out of #85's fix scope — file it separately. Resolved by #96: all three — invalidate on the rotation write (and the peer, via the newsite-psk-invalidationtopic; the first config-invalidation broadcast topic D1/D3 could join), refresh on a site's auth rejection, and drop the cache on aSitePsksreload. - D3 —
ExternalDefinitionLookupCachepeer bound (10 s). Already tracked by #83; the recommendation there (accept and document) still stands. If D1 gets a broadcast topic, this cache andApiMethodCacheshould join it rather than add a second mechanism.
Follow-up: one central cache-invalidation topic (#100 item 3, 2026-11-09)
D1 and D3 now ride a single DistributedPubSub topic, central-cache-invalidation, with typed
messages (ICentralCacheInvalidation: SitePskInvalidated, RoleMappingsInvalidated,
ExternalDefinitionsInvalidated, InboundApiMethodInvalidated) instead of one topic per cache.
The #96 topic was migrated onto it: SitePskInvalidated travels on the shared topic, and for one
release is also published bare on site-psk-invalidation (still subscribed), so a mixed-version
central pair keeps working. Each envelope carries the publisher's origin id and the subscriber
skips its own. Publishers: ManagementActor post-commit (role mappings, external definitions),
the Transport bundle importer (external definitions) and the PSK invalidator. Every cache keeps its TTL / revision check as the fallback for a
lost broadcast or an out-of-band (SQL) edit. Design: Component-Host.md → Central cache
invalidation; tests include a real two-node cluster (CentralCacheInvalidationTwoNodeTests).