fix(security): warn-only compatibility for deployed drivers + upgrade preflight (#815)

User decision: save, Test Connect, live browse and the deploy gate keep
refusing out-of-namespace credential references, but an ALREADY-DEPLOYED
driver must not fault at the upgrade restart or (Galaxy) at the next
gateway reconnect, since ReopenAsync re-resolves Gateway.ApiKeySecretRef.

- LegacyCredentialReferenceTracker (Core.Abstractions): the one, deprecated,
  call-site-chosen compatibility path. Running Galaxy (InitializeAsync,
  ReopenAsync, ack fallback) and OpcUaClient (session open) pass it and keep
  the pre-#815 semantics (case-insensitive prefixes, Galaxy file:, OpcUaClient
  literal pass-through) with a rate-limited Warning (driver/field/allowed
  form, never the value) and a credential-compat:<field> diagnostics entry
  (both drivers now implement IDriverDiagnostics). Browsers and probes pass
  none and stay hard-fail; Heidenhain has none.
- Deploy-gate message names the migration.
- Upgrade preflight: CredentialReferencePreflightService (admin nodes, once
  after ApplicationStarted, ConfigDb only, resolves nothing) logs a Warning
  per Draft row / Deployed-artifact finding; the /raw banner shows them plus
  Running drivers reporting warn-only use (driver-health diagnostics).
- docs/security.md: exact replacement table (incl. the old form default
  env:MX_API_KEY, file:, un-namespaced secret:, casing changes), the
  ReopenAsync reconnect behaviour, the warn-only window and its removal.

Co-Authored-By: Claude <noreply@anthropic.com>

docs/drivers/Galaxy.md

@@ -81,7 +81,7 @@ Project root files:
 
 `DriverConfig` JSON binds to `Config/GalaxyDriverOptions.cs`. The four sections are:
 
-- **`Gateway`** — endpoint, API key secret ref (`secret:galaxy/<name>` · `secret:otopcua/galaxy/<name>` · `env:OTOPCUA_GALAXY_<NAME>` · legacy `env:GALAXY_MXGW_<NAME>` · `dev:KEY` · literal; anything else, incl. `file:`, is refused — #815, [`docs/security.md`](../security.md) §"Credential references are namespaced per driver"), TLS knobs, connect/call/stream timeouts. `StreamTimeoutSeconds = 0` keeps the long-lived `StreamEvents` RPC open for the driver's lifetime.
+- **`Gateway`** — endpoint, API key secret ref (`secret:galaxy/<name>` · `secret:otopcua/galaxy/<name>` · `env:OTOPCUA_GALAXY_<NAME>` · legacy `env:GALAXY_MXGW_<NAME>` · `dev:KEY` · literal; anything else, incl. `file:`, is refused — #815 (an already-deployed driver keeps resolving a pre-#815 ref on init and every gateway reconnect in deprecated warn-only mode until its next deploy), [`docs/security.md`](../security.md) §"Credential references are namespaced per driver"), TLS knobs, connect/call/stream timeouts. `StreamTimeoutSeconds = 0` keeps the long-lived `StreamEvents` RPC open for the driver's lifetime.
 - **`MxAccess`** — `ClientName` (must be unique per OtOpcUa instance — redundancy pairs enforce uniqueness at install time), `PublishingIntervalMs` (forwarded as `buffered_update_interval_ms` on subscribe), `WriteUserId` for ArchestrA secured-write, `EventPumpChannelCapacity` (default 50_000 — one second of headroom at 50k tags / 1Hz; tune via the `galaxy.events.dropped` metric).
 - **`Repository`** — `DiscoverPageSize`, `WatchDeployEvents`.
 - **`Reconnect`** — `InitialBackoffMs`, `MaxBackoffMs`, `ReplayOnSessionLost` (calls the gateway's `ReplaySubscriptions` RPC after reconnect rather than re-issuing subscribe-bulk for every tag).

docs/drivers/OpcUaClient.md

@@ -84,7 +84,8 @@ factory, the Test-Connect probe and the browse picker (unknown keys are skipped)
 **Credentials (#815).** `Password` / `UserCertificatePassword` are a literal, `secret:opcuaclient/<name>` (the node's
 secret store) or `env:OTOPCUA_OPCUACLIENT_<NAME>`. They are resolved on the node and sent to the configured endpoint,
 so any other `secret:`/`env:` reference (and `file:`) is refused at save, deploy (`CredentialReferenceOutOfScope`), Test
-Connect, browse and again at session open, before anything is read. Migrate an existing `secret:<name>` by re-creating
+Connect, browse and (for the AdminUI browser) again before anything is read; an already-deployed driver's session open
+keeps the pre-#815 semantics in deprecated warn-only mode until its next deploy. Migrate an existing `secret:<name>` by re-creating
 the secret as `opcuaclient/<name>`. See [`docs/security.md`](../security.md) §"Credential references are namespaced per driver".
 
 `ProbeTimeoutSeconds` (default 15, AdminUI "Probe timeout") caps the Test-Connect probe's deadline — it can shorten

docs/security.md

@@ -1178,28 +1178,59 @@ Enforced, fail-closed, at every seam — the earlier ones exist for a clear mess
    `HeidenhainCredentialPersisted`).
 3. **Test Connect** — `AdminOperationsActor` refuses before any probe runs (`"Test Connect refused: …"`).
 4. **Live browse** — `BrowserSessionService.OpenAsync` refuses before any bespoke or universal browser opens.
-5. **Resolve time** — `GalaxySecretRef`, `GalaxyAckCredentialResolver`, `OpcUaUserIdentityFactory` (driver +
-   browser) and `HeidenhainCredentials` all resolve through `CredentialReferences`, which re-checks the scope
-   **before** any secret-store or environment read.
+5. **Resolve time** — `GalaxySecretRef`, `GalaxyAckCredentialResolver`, `OpcUaUserIdentityFactory` and
+   `HeidenhainCredentials` all resolve through `CredentialReferences`, which re-checks the scope **before** any
+   secret-store or environment read. This is hard-fail on every admin-node path (the Galaxy and OpcUaClient
+   AdminUI browsers, every Test Connect probe) and everywhere for Heidenhain. The one exception is the deprecated
+   warn-only compatibility path for already-deployed drivers, below.
 
 Messages name the field and the allowed forms, never the value. Out of scope by design: `${secret:}` tokens in
 `appsettings.json` (host config, not operator-authored), and the `Sql` driver's `connectionStringRef`, which is
 already a fixed prefix (`Sql__ConnectionStrings__<ref>`) whose value carries its own server — a Designer can
 choose *which* provisioned database a ref names, but cannot redirect its credentials to another host.
 
-**Migration (existing configs).** After the upgrade, a stored config with an out-of-namespace reference fails
-its next deploy (`CredentialReferenceOutOfScope`), its Test Connect / browse, and — for a running driver — its
-next (re)initialise, which faults the driver with the same message. To migrate:
+**Migration (existing configs).** A stored config with an out-of-namespace reference is listed by the upgrade
+preflight (below), refused by its next deploy (`CredentialReferenceOutOfScope` — the message names the
+migration), and refused by Test Connect / live browse. Values most likely to be in real config databases, with
+the exact replacement (create the new secret / variable first — on every node that runs the driver **and** on
+the admin nodes, which run Test Connect and browse — then edit the field):
 
-- **Galaxy API key** — `env:GALAXY_MXGW_API_KEY` (the docker-dev rig and docs' name) keeps working. Any other
-  variable: rename it to `OTOPCUA_GALAXY_<NAME>` on every node that runs the driver *and* the admin nodes
-  (Test Connect / browse), and set `env:OTOPCUA_GALAXY_<NAME>`. A store secret already named `galaxy/…` keeps
-  working; otherwise re-create it as `galaxy/<inst>/apikey` (`/admin/secrets` or the `secret` CLI). A
-  `file:<path>` key must move into the store or a namespaced variable.
-- **Galaxy fallback ack password** — the documented `secret:otopcua/galaxy/…` name keeps working; `env:` / `file:`
-  references move as above.
-- **OpcUaClient** — re-create `secret:<name>` as `opcuaclient/<name>` (and update the field), or move an
-  environment value to `OTOPCUA_OPCUACLIENT_<NAME>`. Literal passwords are unaffected.
+| Old value | Why it is refused | Replace with |
+|---|---|---|
+| Galaxy `env:MX_API_KEY` — the **old AdminUI Galaxy form default** (pre-#815 `GalaxyDriverForm.razor`), so the likeliest value in a real DB | variable outside the namespace | rename the variable to `OTOPCUA_GALAXY_API_KEY` and set `env:OTOPCUA_GALAXY_API_KEY` (or store it and use `secret:galaxy/<inst>/apikey`) |
+| Galaxy `env:<ANY_OTHER_NAME>` (e.g. `env:MY_API_KEY`) | same | variable `OTOPCUA_GALAXY_<NAME>` → `env:OTOPCUA_GALAXY_<NAME>`. `env:GALAXY_MXGW_API_KEY` (docker-dev, parity docs) needs **no** change |
+| Galaxy `file:<path>` — `docs/Configuration.md` used to recommend it for production | `file:` is refused everywhere | put the file's contents in the store as `galaxy/<inst>/apikey` → `secret:galaxy/<inst>/apikey` (or a variable `OTOPCUA_GALAXY_<NAME>`) |
+| Galaxy `secret:<name>` not under `galaxy/` (e.g. the old `docs/v2/driver-specs.md` example `secret:galaxy-gw-api-key`) | secret outside the namespace | re-create it as `galaxy/<name>` (e.g. `galaxy/gw/apikey`) → `secret:galaxy/gw/apikey`. Names already under `galaxy/` or `otopcua/galaxy/` need no change |
+| Galaxy `Env:X`, `SECRET:x`, `File:x`, … (any non-lower-case prefix) — **pre-#815 resolved these case-insensitively** | prefixes are now exact | the lower-case namespaced form from the rows above |
+| Galaxy `AlarmAck.FallbackPasswordSecretRef` = `env:` / `file:` / `secret:` outside the namespace | same | as above; the documented `secret:otopcua/galaxy/<name>` needs no change |
+| OpcUaClient `Password` / `UserCertificatePassword` = `secret:<name>` not under `opcuaclient/` (e.g. `secret:opcua/plant/pw`) | secret outside the namespace | re-create the secret as `opcuaclient/<name>` → `secret:opcuaclient/<name>` |
+| OpcUaClient `SECRET:x` / `Secret:x` — pre-#815 resolved case-insensitively | prefixes are now exact | `secret:opcuaclient/<name>` |
+| OpcUaClient **literal** password that happens to start with `env:`, `file:`, `secret:`/`${secret:` in any casing (e.g. a real password `Env:Prod1`) — pre-#815 sent it verbatim | it now parses as a reference | store it: `secret:opcuaclient/<name>` (a literal that does not start with one of those prefixes is unaffected) |
+
+**Warn-only compatibility for already-deployed drivers (⚠️ deprecated — a later release fails closed).** Refusing
+at resolve time would fault running plant drivers at the upgrade restart — and, for Galaxy, at **every gateway
+reconnect**: `GalaxyDriver.ReopenAsync` (the `ReconnectSupervisor` callback) rebuilds the client options through
+`BuildClientOptionsAsync`, which re-resolves `Gateway.ApiKeySecretRef` each time, so a running driver would fault
+at its next reconnect, not at a deploy. So, for one compatibility window, the running-driver call sites — Galaxy
+`InitializeAsync` / `ReopenAsync` / the ack-fallback resolver, and OpcUaClient session open — resolve an
+out-of-namespace reference with the **old** semantics (Galaxy: case-insensitive `env:`/`file:`/`dev:`/`secret:`,
+literal otherwise; OpcUaClient: case-insensitive `secret:`, literal otherwise) and log a Warning naming the
+driver, field and allowed form (never the value), at most once per field per hour. The use also rides the
+driver's diagnostics bag (`credential-compat:<field>`) to `/hosts` and to the `/raw` credential banner. This is
+chosen **by call site**, not by a flag (`LegacyCredentialReferenceTracker`): the AdminUI browsers and every probe
+never get the tracker and stay hard-fail, and Heidenhain (new driver, no legacy configs) is hard-fail everywhere.
+It cannot be used to exfiltrate: save, Test Connect, browse and the deploy gate all refuse such a reference, so it
+can only be the reference an administrator deployed **before** the upgrade, and it lives only until that driver's
+next successful deploy (which requires migrating it). The hard-fail release deletes the tracker, the per-driver
+`Legacy…` resolve methods and their `legacy` parameters — nothing else changes.
+
+**Upgrade preflight.** On an admin node, `CredentialReferencePreflightService` runs once, 15 s after
+`ApplicationStarted` (never inside an Akka starter, and it resolves nothing — ConfigDb reads only). It scans
+every credential-resolving driver's stored `DriverConfig` / `DeviceConfig` rows (**Draft**) and the latest
+deployment artifact (**Deployed** — what the running drivers were given) and logs one Warning per finding:
+driver, device, field, allowed form, never the value. The same list, plus running drivers that reported a
+warn-only resolution (**Running**, from driver-health diagnostics, fleet-wide), is shown as a banner on `/raw`,
+with a Re-check button.
 
 Residual (accepted): within one namespace, a Designer can still point one driver instance at another
 instance's secret (e.g. `secret:galaxy/<other-inst>/apikey`) — the value then reaches the endpoint the Designer