fix(host): follow an in-place site PSK rotation without a central restart (ScadaBridge#96)

SitePskProvider cached each site's gRPC preshared key forever; only site removal
invalidated it, so rotating SB-GRPC-PSK-{site} broke the central<->site link on
both central nodes until they restarted. The cached key is now dropped when:

- the secret is written through /admin/secrets: central replaces the secrets
  library's ISecretCacheInvalidator with SitePskRotationInvalidator, which also
  publishes SitePskInvalidated on a new DistributedPubSub topic
  (site-psk-invalidation) so the peer central node drops its copy and its
  resolver cache entry too (SitePskInvalidationSubscriber);
- a site rejects a central->site call with PermissionDenied/Unauthenticated:
  WithSiteCredentials wraps every per-site channel's handler in
  PskRejectionRefreshHandler, so the next call re-reads the key (covers
  out-of-process writes such as the secret CLI);
- a site->central call presents a token that does not match the cached key:
  CentralControlAuthInterceptor invalidates and re-reads once before refusing;
- ScadaBridge:Communication:SitePsks reloads.

Invalidation is case-insensitive because SecretName lowercases names. Rotation
runbook (topology-guide), Component-Communication/Host and the #85 cache audit
(D2) updated.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
2026-11-09 14:27:14 -05:00
parent 7f479f3a5c
commit 3a31dd2de7
15 changed files with 666 additions and 26 deletions
@@ -28,7 +28,7 @@ node after a change is made through the other one. Same defect class as #83
| `PendingUpdatesCache` (DeploymentManager, #91) | Fleet pending-updates snapshot | Deployment finished on this node | `DeploymentManager:PendingUpdatesCacheTtl`, default **60 s** | OK — display only |
| `CentralCommunicationActor` site-address cache (+ `GrpcSiteTransport` channel pairs) | Site gRPC endpoints | Site create/update/delete → `RefreshSiteAddresses` | Periodic DB reload, **60 s** (hard-coded) | OK — rare edit, self-heals |
| `SiteStreamGrpcClientFactory` clients | Per (site, endpoint) stream clients | Site delete → `RemoveSiteAsync` (local only) | Not stale data: a removed site's client on the peer is reclaimed by idle eviction, `Communication:GrpcClientIdleTtl` (**30 min**) × 1.5 after last use | OK — resource, not correctness |
| `SitePskProvider` (Host) | Per-site gRPC PSK from config / secret store | Site removal (via either gRPC client path, on every node within the 60 s reload) | **Unbounded** for an in-place key rotation — on **both** nodes, not only the peer | Owner decision — see D2 |
| `SitePskProvider` (Host) | Per-site gRPC PSK from config / secret store | Site removal; `SB-GRPC-PSK-{site}` write via `/admin/secrets`; site auth rejection; `SitePsks` reload (#96) | Write via `/admin/secrets`: **none** (peer told over `DistributedPubSub` topic `site-psk-invalidation`); other writes: first rejected call after `Secrets:ResolveCacheTtl` | Fixed — #96 (see D2) |
| `DefaultSecretResolver` (shared `ZB.MOM.WW.Secrets`) | Decrypted secret values | Lib's own invalidator (UI writes, sync service) | `Secrets:ResolveCacheTtl`, **30 s** | OK — owned by the shared lib |
| `KpiSnapshotCache` (CentralUI) | KPI query results (operational data) | — | **8 s** | OK — not config |
| `SharedAlarmSummaryService` (CentralUI) | Per-site alarm summary fan-out | — | **12 s** cold-cache window | OK — not config |
@@ -69,7 +69,9 @@ broadcast was added.
new key) leaves **both** central nodes dialling with the old key until they restart. Options: make
central restart part of the rotation runbook; give the cache a TTL; or invalidate on a
`PermissionDenied` from the site. Not peer-specific, so it is out of #85's fix scope — file it
separately.
separately. **Resolved by #96:** all three — invalidate on the rotation write (and the peer, via
the new `site-psk-invalidation` topic; the first config-invalidation broadcast topic D1/D3 could
join), refresh on a site's auth rejection, and drop the cache on a `SitePsks` reload.
- **D3 — `ExternalDefinitionLookupCache` peer bound (10 s).** Already tracked by #83; the
recommendation there (accept and document) still stands. If D1 gets a broadcast topic, this cache
and `ApiMethodCache` should join it rather than add a second mechanism.