SitePskProvider caches site PSKs forever: in-place PSK rotation breaks the central↔site link until restart #96

New Issue

2026-11-09T14:02:08-05:00

dohertj2 commented

2026-11-09 14:02:08 -05:00

SitePskProvider caches each site's inbound PSK for the life of the process. Its comment says the entry is dropped "when a key is rotated in place", but nothing does that — the cache is cleared only when a site is removed. After an in-place PSK rotation both central nodes keep presenting the old key and the central↔site link fails until both centrals restart. Found in the #85 cache audit.

Fix options: add a TTL; invalidate on a site's auth rejection and retry once with a fresh read; or invalidate on the rotation write (+ peer via central pub-sub). At minimum document a restart step in the rotation runbook.

`SitePskProvider` caches each site's inbound PSK for the life of the process. Its comment says the entry is dropped "when a key is rotated in place", but nothing does that — the cache is cleared only when a site is removed. After an in-place PSK rotation both central nodes keep presenting the old key and the central↔site link fails until both centrals restart. Found in the #85 cache audit. Fix options: add a TTL; invalidate on a site's auth rejection and retry once with a fresh read; or invalidate on the rotation write (+ peer via central pub-sub). At minimum document a restart step in the rotation runbook.

dohertj2 added the type:bug priority:p2 labels 2026-11-09 14:02:08 -05:00

dohertj2 commented

2026-11-09 14:38:53 -05:00

Fixed on main 3a31dd2d — central drops a site's cached PSK on: (1) rotation via /admin/secrets — new SitePskRotationInvalidator hooks the write and publishes on new DistributedPubSub topic site-psk-invalidation so the peer central drops its copy + its 30 s secret-resolver entry; (2) a site rejecting a central→site call — each per-site channel re-reads the key on its next call (no in-place retry; callers already retry), covering out-of-process rotations such as the secret CLI; (3) an inbound site→central token mismatch — CentralControlAuthInterceptor re-reads once before refusing, so a site already restarted on its new key isn't locked out; (4) a SitePsks config reload clears the whole cache. Invalidation is case-insensitive (the secrets lib lowercases names, so 'Site-A' was missed). docs/deployment/topology-guide.md now says no central restart is needed and describes the central↔site outage window; Component-Communication/Host and the cache audit D2 updated. 9 new tests fail with the fix off (incl. end-to-end against a real gRPC test server for unary + streaming); controls pass both ways. Evidence (after rebase): slnx 0 warnings/0 errors; Host, Communication, AuditLog, Transport, ConfigurationDatabase and Transport.IntegrationTests (143) passed.

Fixed on main 3a31dd2d — central drops a site's cached PSK on: (1) rotation via /admin/secrets — new SitePskRotationInvalidator hooks the write and publishes on new DistributedPubSub topic `site-psk-invalidation` so the peer central drops its copy + its 30 s secret-resolver entry; (2) a site rejecting a central→site call — each per-site channel re-reads the key on its next call (no in-place retry; callers already retry), covering out-of-process rotations such as the `secret` CLI; (3) an inbound site→central token mismatch — CentralControlAuthInterceptor re-reads once before refusing, so a site already restarted on its new key isn't locked out; (4) a SitePsks config reload clears the whole cache. Invalidation is case-insensitive (the secrets lib lowercases names, so 'Site-A' was missed). docs/deployment/topology-guide.md now says no central restart is needed and describes the central↔site outage window; Component-Communication/Host and the cache audit D2 updated. 9 new tests fail with the fix off (incl. end-to-end against a real gRPC test server for unary + streaming); controls pass both ways. Evidence (after rebase): slnx 0 warnings/0 errors; Host, Communication, AuditLog, Transport, ConfigurationDatabase and Transport.IntegrationTests (143) passed.

dohertj2 closed this issue

2026-11-09 14:38:53 -05:00

dohertj2 referenced this issue

2026-11-09 14:38:53 -05:00

Import: GetIdByNameAsync derived-template ambiguity; intra-record exact-name lookups; reuse the PSK pub-sub invalidation for D1/D3 #100

dohertj2 referenced this issue

2026-11-09 15:32:28 -05:00

Inbound API compile state is per central node; stale InboundAPI doc; locked-override warning name matching #103

dohertj2 referenced this issue from a commit

2026-11-09 16:11:41 -05:00

feat(central): one central cache-invalidation topic; role-mapping and external-definition caches follow peer edits (ScadaBridge#100 item 3, ScadaBridge#83)

Sign in to join this conversation.