fix(central): audit per-node central caches; management API drops ApiMethodCache entries (ScadaBridge#85)

Audit (docs/plans/2026-11-09-central-cache-staleness-audit.md) of every central in-process
cache holding configuration or secret state, with its staleness bound on the peer central
node. Every config cache with peer staleness is TTL-bounded (5-60 s) and Traefik routes to the
active node only, so no peer case was clearly a bug and no cross-node broadcast is added; the
doc names the existing DistributedPubSub mediator (health replication topic) as the mechanism
if one is wanted. Owner decisions listed: RoleMappingCache 30 s revocation lag on the peer,
SitePskProvider caching a PSK forever across an in-place rotation (both nodes), and the #83
ExternalDefinitionLookupCache 10 s bound.

Fixed: ManagementActor API-method create/update/delete invalidated only the compiled handler,
not ApiMethodCache, so the node that made the change kept serving the pre-change row for up to
the 5 s TTL (a deleted method stayed callable; a cached not-found hid a new one). The handlers
now drop the entry post-commit, like the role-mapping cache.

Tests: ApiMethodCacheInvalidationTests (create/update/delete fail without the fix; a
negative control checks other methods stay cached).

(ScadaBridge#85)

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV

docs/plans/2026-11-09-central-cache-staleness-audit.md

@@ -0,0 +1,75 @@
+# Central per-node cache staleness audit (Gitea #85)
+
+**Date:** 2026-11-09. **Scope:** every in-process cache on a **central** node that holds
+configuration-database (or secret-store) state, and how stale it can be on the **peer** central
+node after a change is made through the other one. Same defect class as #83
+(`ExternalDefinitionLookupCache`).
+
+## Context that sets the risk
+
+- Traefik routes UI, `/management` and `/api/*` to the **active** central node only
+  (`/health/active`). An edit and a use on different nodes therefore only happen when someone
+  talks to a node directly (CLI pointed at a node's own port) or across a **failover**.
+- Every TTL below is measured from when the entry was loaded, so an entry a node cached while it
+  was previously active is still bounded by its TTL after it becomes active again.
+- No cache below replicates its invalidation to the peer. Central already has one cross-node
+  channel: the `DistributedPubSub` topic `CentralCommunicationActor.HealthReportTopic` (peer
+  health replication). A config-invalidation broadcast, if an owner wants one, should be a second
+  topic on the same mediator, published from `AfterConfigurationCommit`.
+
+## Inventory
+
+| Cache | Holds | Local invalidation | Peer staleness bound | Verdict |
+|---|---|---|---|---|
+| `InboundScriptExecutor` compiled handlers (InboundAPI) | Compiled method delegates | `InvalidateMethod` on delete; change bus on import | **None** — every request revision-checks the cached handler against the resolved `ApiMethod.Script` and recompiles on mismatch | OK |
+| `ApiMethodCache` (InboundAPI) | `ApiMethod` rows, incl. not-found | Change bus (bundle import); **management create/update/delete — added by #85** | `InboundApi:ApiMethodCacheTtl`, default **5 s** | OK (local gap fixed, see below) |
+| `ExternalDefinitionLookupCache` (ESG, central only) | External systems / methods / DB connections (found only) | 9 ManagementActor handlers (#65); bundle import (#83) | `ExternalSystemGateway:DefinitionCacheTtl`, **10 s** on central | Owner decision — #83 |
+| `RoleMappingCache` (Security) — the cache at `ManagementActor` `InvalidateRoleMappingCacheAfterCommit` | LDAP group → role map + Deployer site scopes | Mapping / scope-rule commands, post-commit | `Security:RoleMappingCacheSeconds`, default **30 s** (`0` disables) | Owner decision — see D1 |
+| `PendingUpdatesCache` (DeploymentManager, #91) | Fleet pending-updates snapshot | Deployment finished on this node | `DeploymentManager:PendingUpdatesCacheTtl`, default **60 s** | OK — display only |
+| `CentralCommunicationActor` site-address cache (+ `GrpcSiteTransport` channel pairs) | Site gRPC endpoints | Site create/update/delete → `RefreshSiteAddresses` | Periodic DB reload, **60 s** (hard-coded) | OK — rare edit, self-heals |
+| `SiteStreamGrpcClientFactory` clients | Per (site, endpoint) stream clients | Site delete → `RemoveSiteAsync` (local only) | Not stale data: a removed site's client on the peer is reclaimed by idle eviction, `Communication:GrpcClientIdleTtl` (**30 min**) × 1.5 after last use | OK — resource, not correctness |
+| `SitePskProvider` (Host) | Per-site gRPC PSK from config / secret store | Site removal (via either gRPC client path, on every node within the 60 s reload) | **Unbounded** for an in-place key rotation — on **both** nodes, not only the peer | Owner decision — see D2 |
+| `DefaultSecretResolver` (shared `ZB.MOM.WW.Secrets`) | Decrypted secret values | Lib's own invalidator (UI writes, sync service) | `Secrets:ResolveCacheTtl`, **30 s** | OK — owned by the shared lib |
+| `KpiSnapshotCache` (CentralUI) | KPI query results (operational data) | — | **8 s** | OK — not config |
+| `SharedAlarmSummaryService` (CentralUI) | Per-site alarm summary fan-out | — | **12 s** cold-cache window | OK — not config |
+| `IDeploymentStatusNotifier` (DeploymentManager) | Push of deployment status to UI circuits | In-process event | Peer UI never sees the push, but the UI is only served by the active node and the pages also poll | OK |
+| `ITemplateGraphWatermark` / `FlattenSession` | Template-chain memo keyed by in-process version | Repository bumps | Session lives for one deploy/validate operation; the watermark is process-local by design (peer starts from 0 → extra work, never stale work) | OK |
+| `ScriptCompileVerdictCache`, `TrustVerdictCache`, `AuditRegexCache`, `ScriptAnalysisService` `IMemoryCache`, `OAuth2TokenService` | Pure functions of their key (code hash + surface, pattern, credential hash) | — | Not applicable — a changed input is a different key | OK |
+| `ZB.MOM.WW.Auth.ApiKeys.SqlServer` (inbound API keys) | — | — | No in-process cache in the shared store (checked in the 0.3.0 source) — revocation is immediate on both nodes | OK |
+
+## Fixed in #85
+
+**`ApiMethodCache` was not invalidated by the management API even locally.** Only the compiled
+handler was dropped, so for up to the 5 s TTL on the node that made the change:
+
+- a just-**deleted** method stayed callable — the executor recompiled it from the cached row;
+- a just-**created** method answered 404 if a not-found had been cached.
+
+`ManagementActor` now drops the method's cache entry post-commit on create, update and delete
+(`InvalidateApiMethodCacheAfterCommit`), the same post-commit pattern as the role-mapping cache.
+The peer is still bounded by the 5 s TTL. Tests: `ApiMethodCacheInvalidationTests` (3 of 4 fail
+without the fix; the fourth is the negative control).
+
+No peer-staleness case was **clearly** a bug: every config cache with peer staleness is
+TTL-bounded (5–60 s), its bound is documented at the cache, and the active-only routing means the
+window is only hit via a direct-to-node call or a failover inside the TTL. So no cross-node
+broadcast was added.
+
+## Needs an owner decision
+
+- **D1 — `RoleMappingCache` revocation lag on the peer (30 s).** Removing a mapping or narrowing a
+  Deployer's site scope takes effect at once on the node that made the change and within 30 s on
+  the other. It is a security-relevant window (revoked access keeps working briefly), but only on
+  the node the change did *not* go through. Options: accept and document (current state); lower
+  `RoleMappingCacheSeconds`; or add the `DistributedPubSub` invalidation topic described above.
+  Recommendation: accept unless the operators' revocation SLA is tighter than 30 s.
+- **D2 — `SitePskProvider` caches a successful PSK resolve forever.** Its own remarks say
+  `Invalidate` covers "a key rotated in place", but nothing calls it on rotation — only on site
+  removal. Rotating `SB-GRPC-PSK-{site}` in the secret store (with the site pair restarted on the
+  new key) leaves **both** central nodes dialling with the old key until they restart. Options: make
+  central restart part of the rotation runbook; give the cache a TTL; or invalidate on a
+  `PermissionDenied` from the site. Not peer-specific, so it is out of #85's fix scope — file it
+  separately.
+- **D3 — `ExternalDefinitionLookupCache` peer bound (10 s).** Already tracked by #83; the
+  recommendation there (accept and document) still stands. If D1 gets a broadcast topic, this cache
+  and `ApiMethodCache` should join it rather than add a second mechanism.