fix(host): follow an in-place site PSK rotation without a central restart (ScadaBridge#96)

SitePskProvider cached each site's gRPC preshared key forever; only site removal
invalidated it, so rotating SB-GRPC-PSK-{site} broke the central<->site link on
both central nodes until they restarted. The cached key is now dropped when:

- the secret is written through /admin/secrets: central replaces the secrets
  library's ISecretCacheInvalidator with SitePskRotationInvalidator, which also
  publishes SitePskInvalidated on a new DistributedPubSub topic
  (site-psk-invalidation) so the peer central node drops its copy and its
  resolver cache entry too (SitePskInvalidationSubscriber);
- a site rejects a central->site call with PermissionDenied/Unauthenticated:
  WithSiteCredentials wraps every per-site channel's handler in
  PskRejectionRefreshHandler, so the next call re-reads the key (covers
  out-of-process writes such as the secret CLI);
- a site->central call presents a token that does not match the cached key:
  CentralControlAuthInterceptor invalidates and re-reads once before refusing;
- ScadaBridge:Communication:SitePsks reloads.

Invalidation is case-insensitive because SecretName lowercases names. Rotation
runbook (topology-guide), Component-Communication/Host and the #85 cache audit
(D2) updated.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV

docs/deployment/topology-guide.md

@@ -557,8 +557,14 @@ One key **per site**, never one for the fleet: a compromised site must not yield
 key. And never share it with `LocalDb:Replication:ApiKey` — that authenticates the *pair partner*
 for database replication, a different trust relationship on the same listener.
 
-**Rotation:** set the new value on both sides, then restart the pair (pairs restart together
-anyway — see above). **Upgrading to a build that has this gate requires seeding the key first**,
+**Rotation:** set the new value on both sides, then restart the site pair (pairs restart together
+anyway — see above; the site reads its key once, at boot). **Central needs no restart** (Gitea #96):
+writing `SB-GRPC-PSK-<siteId>` through `/admin/secrets` on either central node drops the cached key
+on both central nodes at once, and a key changed any other way (the `secret` CLI, a `SitePsks`
+edit) is picked up on the first call the site rejects, or the first site call central's own gate
+cannot match — within `Secrets:ResolveCacheTtl` (30 s) of
+the change. Expect the link to that site to fail between the two halves of the change; doing
+central first and the site straight after keeps that window short. **Upgrading to a build that has this gate requires seeding the key first**,
 including in the on-host `deploy/` overlays.
 
 The key is a bearer token over plaintext h2c, so it is readable and replayable by anyone on the

docs/plans/2026-11-09-central-cache-staleness-audit.md

@@ -28,7 +28,7 @@ node after a change is made through the other one. Same defect class as #83
 | `PendingUpdatesCache` (DeploymentManager, #91) | Fleet pending-updates snapshot | Deployment finished on this node | `DeploymentManager:PendingUpdatesCacheTtl`, default **60 s** | OK — display only |
 | `CentralCommunicationActor` site-address cache (+ `GrpcSiteTransport` channel pairs) | Site gRPC endpoints | Site create/update/delete → `RefreshSiteAddresses` | Periodic DB reload, **60 s** (hard-coded) | OK — rare edit, self-heals |
 | `SiteStreamGrpcClientFactory` clients | Per (site, endpoint) stream clients | Site delete → `RemoveSiteAsync` (local only) | Not stale data: a removed site's client on the peer is reclaimed by idle eviction, `Communication:GrpcClientIdleTtl` (**30 min**) × 1.5 after last use | OK — resource, not correctness |
-| `SitePskProvider` (Host) | Per-site gRPC PSK from config / secret store | Site removal (via either gRPC client path, on every node within the 60 s reload) | **Unbounded** for an in-place key rotation — on **both** nodes, not only the peer | Owner decision — see D2 |
+| `SitePskProvider` (Host) | Per-site gRPC PSK from config / secret store | Site removal; `SB-GRPC-PSK-{site}` write via `/admin/secrets`; site auth rejection; `SitePsks` reload (#96) | Write via `/admin/secrets`: **none** (peer told over `DistributedPubSub` topic `site-psk-invalidation`); other writes: first rejected call after `Secrets:ResolveCacheTtl` | Fixed — #96 (see D2) |
 | `DefaultSecretResolver` (shared `ZB.MOM.WW.Secrets`) | Decrypted secret values | Lib's own invalidator (UI writes, sync service) | `Secrets:ResolveCacheTtl`, **30 s** | OK — owned by the shared lib |
 | `KpiSnapshotCache` (CentralUI) | KPI query results (operational data) | — | **8 s** | OK — not config |
 | `SharedAlarmSummaryService` (CentralUI) | Per-site alarm summary fan-out | — | **12 s** cold-cache window | OK — not config |
@@ -69,7 +69,9 @@ broadcast was added.
   new key) leaves **both** central nodes dialling with the old key until they restart. Options: make
   central restart part of the rotation runbook; give the cache a TTL; or invalidate on a
   `PermissionDenied` from the site. Not peer-specific, so it is out of #85's fix scope — file it
-  separately.
+  separately. **Resolved by #96:** all three — invalidate on the rotation write (and the peer, via
+  the new `site-psk-invalidation` topic; the first config-invalidation broadcast topic D1/D3 could
+  join), refresh on a site's auth rejection, and drop the cache on a `SitePsks` reload.
 - **D3 — `ExternalDefinitionLookupCache` peer bound (10 s).** Already tracked by #83; the
   recommendation there (accept and document) still stands. If D1 gets a broadcast topic, this cache
   and `ApiMethodCache` should join it rather than add a second mechanism.

docs/requirements/Component-Communication.md

@@ -204,6 +204,18 @@ because sites are created at runtime and cannot be enumerated in configuration a
 `ControlPlaneCredentials` binds the key to the channel as `CallCredentials`, so no individual call
 site can omit it.
 
+Central caches each resolved key, and drops the entry (Gitea #96) when: the site is removed; the
+secret `SB-GRPC-PSK-<siteId>` is written through central's secrets write path (`/admin/secrets`) —
+`SitePskRotationInvalidator` wraps the secrets library's `ISecretCacheInvalidator` and also
+broadcasts a `SitePskInvalidated` on the central `DistributedPubSub` topic `site-psk-invalidation`
+so the peer central node drops its copy too; the site rejects a call with `PermissionDenied` /
+`Unauthenticated` (`PskRejectionRefreshHandler`, wrapped around every per-site channel by
+`WithSiteCredentials`, so the next call re-reads the key — this covers keys rotated outside the
+process, e.g. by the `secret` CLI, after at most the resolver's `Secrets:ResolveCacheTtl`); a
+site's inbound call to central presents a token that does not match the cached key
+(`CentralControlAuthInterceptor` drops the entry and re-reads once before refusing); or
+`ScadaBridge:Communication:SitePsks` reloads. An in-place rotation therefore needs no central restart.
+
 This is distinct from `LocalDb:Replication:ApiKey`, which gates
 `/localdb_sync.v1.LocalDbSync/` on the same listener via its own interceptor: that key
 authenticates the *pair partner* for database replication, a different trust relationship, and the

docs/requirements/Component-Host.md

@@ -351,4 +351,5 @@ built)* section for where the shipped code deviates from the original design).
 - **Port options come from the bound `NodeOptions`.** `Program.cs` binds `ScadaBridge:Node` once and reads `GrpcPort` / `CentralGrpcPort` / `MetricsPort` from it; the 8083/8084 defaults live only in `NodeOptions`. Central's dedicated metrics listener remains an explicit *presence* check on the raw key (a defaulted read would bind a new port on every existing central node).
 - **Inbound method warm-up is a hosted service** (`InboundMethodWarmupService`, registered after `AkkaHostedService`), not a pre-`RunAsync` loop — see Component-InboundAPI.md.
 - **`SitePskProvider` distinguishes a secret-store fault from a missing secret.** A store fault logs a Warning with the exception and throws `SecretStoreUnavailableException` (not cached); only a genuinely absent/empty/tombstoned secret produces the "seed `SB-GRPC-PSK-{site}`" message.
+- **`SitePskProvider`'s cache follows an in-place rotation (Gitea #96).** Central replaces the secrets library's `ISecretCacheInvalidator` with `SitePskRotationInvalidator` (`AddSitePskRotationInvalidation`), which drops the site's cached key when `SB-GRPC-PSK-{site}` is written and publishes `SitePskInvalidated` on the `site-psk-invalidation` `DistributedPubSub` topic; `AkkaHostedService.RegisterCentralActors` starts the `SitePskInvalidationSubscriber` that applies the peer's broadcasts. A site's auth rejection and a `SitePsks` reload also drop the cache — see Component-Communication.md → *Authentication*.
 - Central registers `ExternalSystemGatewayOptions.DefinitionCacheTtl = 10 s` ahead of `AddExternalSystemGateway()` so an explicit config value wins; sites leave it off.