fix(inbound-api): API-method recompile/invalidate reaches the peer central node; stale doc; locked-override name rule (ScadaBridge#103)

(a) Inbound API compile state (#75) is per central node: a recompile cleared the
known-bad record only on the node that answered, and the peer kept refusing the
unchanged script with SCRIPT_COMPILE_FAILED (the per-request revision check only
notices a changed script). Create / update / delete (ManagementActor post-commit),
RecompileApiMethod, and API methods written by a Transport bundle import now publish
InboundApiMethodInvalidated on the central cache-invalidation topic (#100 item 3).
The new InboundApiCacheInvalidationHandler drops that method's compiled handler,
known-bad record and cached ApiMethod row on the peer, which compiles the current
text on its next request.

(b) docs/components/InboundAPI.md no longer claims a direct DB edit stays stale
until restart: the per-request revision check picks it up once the node's cached
row expires (ApiMethodCacheTtl, 5 s). Component-InboundAPI, Component-Host and the
#85 cache audit describe the peer invalidation.

(c) Decision: the import's locked-override warnings keep EXACT (ordinal) name
matching, because FlatteningService matches override names ordinally. An override
differing from a locked member only by case never reaches that member's lock check
(the flattener drops it as unresolved), so a "targets a LOCKED member" warning for it
would be false. The comparer is now named (LockedOverrideNameComparer) and documented,
and pinned by tests on both sides (flattener + import).

Tests: the peer's known-bad record is cleared by a recompile (and validate leaves it
alone); create/update/delete and the import publish the invalidation; the handler
drops only the named method; composition root registers the handler; case-only
override gets no lock warning; the flattener matches override names exactly.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
2026-11-09 16:04:07 -05:00
parent 153609ef8a
commit 0c2e852a62
16 changed files with 304 additions and 24 deletions
+3 -3
View File
@@ -223,7 +223,7 @@ The `Route.To().Call()` inherits the method-level timeout automatically. A scrip
### Startup compilation and hot-reload
At startup the Host loads all `ApiMethod` rows from the configuration database and calls `CompileAndRegister` on each. After a method is updated via the Management API or CLI, the Management Service calls `CompileAndRegister` again — the updated script takes effect on the next request, with no node restart. Methods created after startup compile lazily on first invocation. Scripts modified directly in the database do not take effect until the next node restart; always use the Management API, CLI, or Central UI.
At startup the Host loads all `ApiMethod` rows from the configuration database and calls `CompileAndRegister` on each. After a method is updated via the Management API or CLI, the Management Service calls `CompileAndRegister` again — the updated script takes effect on the next request, with no node restart. Methods created after startup compile lazily on first invocation. Each request resolves the method's stored row (through `ApiMethodCache`, `InboundApi:ApiMethodCacheTtl`, default 5 s) and revision-checks the cached handler against its script text, recompiling on a mismatch — so a script edited directly in the database takes effect on each node within that TTL, without a restart. Management API, CLI, Central UI and bundle-import changes also drop the method's cached row and handler on **both** central nodes at once: the node that made the change directly, the peer through `InboundApiMethodInvalidated` on the central cache-invalidation topic (#103, see [Host](./Host.md)).
## Configuration
@@ -252,11 +252,11 @@ The inbound body-capture cap for audit is configured separately under `AuditLog:
### A method always returns 500 after a script update
The in-memory handler cache still holds the previous compiled delegate. If the update went through the Management API or CLI, `CompileAndRegister` should have been called automatically and the new script should be active on the next request. If the script was edited directly in the database, the cached delegate is stale until the next node restart. Check the `ScadaBridge.InboundAPI` log category for a `"script compilation failed"` or `"trust model violation"` warning to distinguish a compile error from a routing failure.
The in-memory handler cache still holds the previous compiled delegate. If the update went through the Management API or CLI, `CompileAndRegister` should have been called automatically and the new script should be active on the next request. If the script was edited directly in the database, the per-request revision check recompiles it once the node's cached `ApiMethod` row expires (`ApiMethodCacheTtl`, default 5 s) — no restart is needed. Check the `ScadaBridge.InboundAPI` log category for a `"script compilation failed"` or `"trust model violation"` warning to distinguish a compile error from a routing failure.
### A method is stuck in the known-bad-methods cache
If a previously broken script is fixed but `ExecuteAsync` still returns `"Script compilation failed for this method"`, the method name is in `_knownBadMethods`. A record only short-circuits while the stored script text is the one that failed, so a changed script always recompiles; a stale record therefore means the script text is unchanged and something outside it made the compile fail. `api-method get --id <N>` shows the node's `isKnownBad` / `lastCompileError` (the real diagnostics; the data plane keeps its generic message), `api-method validate --id <N>` compiles the stored script and returns the diagnostics without touching the cache, and `api-method recompile --id <N>` re-runs `CompileAndRegister` — clearing the entry on success — without re-sending the entity (#75). All three are per central node; the Central UI's API-method edit page exposes the same state and actions.
If a previously broken script is fixed but `ExecuteAsync` still returns `"Script compilation failed for this method"`, the method name is in `_knownBadMethods`. A record only short-circuits while the stored script text is the one that failed, so a changed script always recompiles; a stale record therefore means the script text is unchanged and something outside it made the compile fail. `api-method get --id <N>` shows the node's `isKnownBad` / `lastCompileError` (the real diagnostics; the data plane keeps its generic message), `api-method validate --id <N>` compiles the stored script and returns the diagnostics without touching the cache, and `api-method recompile --id <N>` re-runs `CompileAndRegister` — clearing the entry on success — without re-sending the entity (#75). `get` and `validate` report the answering central node; a `recompile` also drops the method's handler and known-bad record on the peer central node (#103), which compiles the current script on its next request. The Central UI's API-method edit page exposes the same state and actions.
### Routed calls time out but the site is reachable
@@ -22,7 +22,7 @@ node after a change is made through the other one. Same defect class as #83
| Cache | Holds | Local invalidation | Peer staleness bound | Verdict |
|---|---|---|---|---|
| `InboundScriptExecutor` compiled handlers (InboundAPI) | Compiled method delegates | `InvalidateMethod` on delete; change bus on import | **None** — every request revision-checks the cached handler against the resolved `ApiMethod.Script` and recompiles on mismatch | OK |
| `ApiMethodCache` (InboundAPI) | `ApiMethod` rows, incl. not-found | Change bus (bundle import); **management create/update/delete — added by #85** | `InboundApi:ApiMethodCacheTtl`, default **5 s** | OK (local gap fixed, see below) |
| `ApiMethodCache` (InboundAPI) | `ApiMethod` rows, incl. not-found | Change bus (bundle import); **management create/update/delete — added by #85** | Change through the other node: **none** (`InboundApiMethodInvalidated`, #103); direct SQL: `InboundApi:ApiMethodCacheTtl`, default **5 s** | OK (local gap fixed by #85, peer by #103) |
| `ExternalDefinitionLookupCache` (ESG, central only) | External systems / methods / DB connections (found only) | 9 ManagementActor handlers (#65); bundle import (#83) | Edit or import through the other node: **none** (`ExternalDefinitionsInvalidated` on `central-cache-invalidation`, #100); direct SQL: `ExternalSystemGateway:DefinitionCacheTtl`, **10 s** on central | Fixed — #100 item 3 (see D3) |
| `RoleMappingCache` (Security) — the cache at `ManagementActor` `InvalidateRoleMappingCacheAfterCommit` | LDAP group → role map + Deployer site scopes | Mapping / scope-rule commands, post-commit | **None** (`RoleMappingsInvalidated` on `central-cache-invalidation`, #100); TTL `Security:RoleMappingCacheSeconds` (**30 s**) only if the broadcast is lost | Fixed — #100 item 3 (see D1) |
| `PendingUpdatesCache` (DeploymentManager, #91) | Fleet pending-updates snapshot | Deployment finished on this node | `DeploymentManager:PendingUpdatesCacheTtl`, default **60 s** | OK — display only |
@@ -84,7 +84,9 @@ messages (`ICentralCacheInvalidation`: `SitePskInvalidated`, `RoleMappingsInvali
The #96 topic was migrated onto it: `SitePskInvalidated` travels on the shared topic, and for one
release is also published bare on `site-psk-invalidation` (still subscribed), so a mixed-version
central pair keeps working. Each envelope carries the publisher's origin id and the subscriber
skips its own. Publishers: ManagementActor post-commit (role mappings, external definitions),
the Transport bundle importer (external definitions) and the PSK invalidator. Every cache keeps its TTL / revision check as the fallback for a
skips its own. Publishers: ManagementActor post-commit (role mappings, external definitions,
API methods), the Transport bundle importer (external definitions, API methods), the API-method
recompile (#103: a recompile now clears a stale known-bad record on the peer too) and the PSK
invalidator. Every cache keeps its TTL / revision check as the fallback for a
lost broadcast or an out-of-band (SQL) edit. Design: Component-Host.md → *Central cache
invalidation*; tests include a real two-node cluster (`CentralCacheInvalidationTwoNodeTests`).
+1 -1
View File
@@ -352,5 +352,5 @@ built)* section for where the shipped code deviates from the original design).
- **Inbound method warm-up is a hosted service** (`InboundMethodWarmupService`, registered after `AkkaHostedService`), not a pre-`RunAsync` loop — see Component-InboundAPI.md.
- **`SitePskProvider` distinguishes a secret-store fault from a missing secret.** A store fault logs a Warning with the exception and throws `SecretStoreUnavailableException` (not cached); only a genuinely absent/empty/tombstoned secret produces the "seed `SB-GRPC-PSK-{site}`" message.
- **`SitePskProvider`'s cache follows an in-place rotation (Gitea #96).** Central replaces the secrets library's `ISecretCacheInvalidator` with `SitePskRotationInvalidator` (`AddSitePskRotationInvalidation`), which drops the site's cached key when `SB-GRPC-PSK-{site}` is written and publishes `SitePskInvalidated` to the peer over the central cache-invalidation topic (below). A site's auth rejection and a `SitePsks` reload also drop the cache — see Component-Communication.md → *Authentication*.
- **Central cache invalidation (ScadaBridge#100 item 3).** One `DistributedPubSub` topic, `central-cache-invalidation`, carries typed `ICentralCacheInvalidation` messages (Commons `Messages/Communication/CentralCacheInvalidations.cs`) between the two central nodes, each wrapped in a `CentralCacheInvalidationEnvelope` stamped with the publishing process's origin id. `AddCentralCacheInvalidation` registers `CentralCacheInvalidationBus` as `ICentralCacheInvalidationPublisher`; `AkkaHostedService.RegisterCentralActors` gives it the mediator and starts `CentralCacheInvalidationSubscriber`, which skips its own node's envelopes and hands the peer's to every registered `ICentralCacheInvalidationHandler`: `SitePskRotationInvalidator` (`SitePskInvalidated`, #96), `RoleMappingCache` (`RoleMappingsInvalidated`, audit D1), and `ExternalDefinitionLookupCache` (`ExternalDefinitionsInvalidated`, audit D3 / #83). Publishers invalidate locally first, post-commit, then publish; delivery is best-effort, so each cache's TTL or revision check stays the fallback. Migration: the #96 topic `site-psk-invalidation` is still published to (bare `SitePskInvalidated`) and subscribed, so a briefly mixed-version central pair keeps exchanging PSK invalidations — remove it once every central node runs this build.
- **Central cache invalidation (ScadaBridge#100 item 3).** One `DistributedPubSub` topic, `central-cache-invalidation`, carries typed `ICentralCacheInvalidation` messages (Commons `Messages/Communication/CentralCacheInvalidations.cs`) between the two central nodes, each wrapped in a `CentralCacheInvalidationEnvelope` stamped with the publishing process's origin id. `AddCentralCacheInvalidation` registers `CentralCacheInvalidationBus` as `ICentralCacheInvalidationPublisher`; `AkkaHostedService.RegisterCentralActors` gives it the mediator and starts `CentralCacheInvalidationSubscriber`, which skips its own node's envelopes and hands the peer's to every registered `ICentralCacheInvalidationHandler`: `SitePskRotationInvalidator` (`SitePskInvalidated`, #96), `RoleMappingCache` (`RoleMappingsInvalidated`, audit D1), `ExternalDefinitionLookupCache` (`ExternalDefinitionsInvalidated`, audit D3 / #83) and the Inbound API's `InboundApiCacheInvalidationHandler` (`InboundApiMethodInvalidated` — compiled handler, known-bad record and cached row of one method, #103). Publishers invalidate locally first, post-commit, then publish; delivery is best-effort, so each cache's TTL or revision check stays the fallback. Migration: the #96 topic `site-psk-invalidation` is still published to (bare `SitePskInvalidated`) and subscribed, so a briefly mixed-version central pair keeps exchanging PSK invalidations — remove it once every central node runs this build.
- Central registers `ExternalSystemGatewayOptions.DefinitionCacheTtl = 10 s` ahead of `AddExternalSystemGateway()` so an explicit config value wins; sites leave it off.
+1 -1
View File
@@ -181,7 +181,7 @@ API method scripts are pre-compiled in the **background after startup** by `Inbo
- Updating a method via the CLI (`api-method update --id <N> --code '...'`) or Management API triggers immediate recompilation (`CompileAndRegister`). The updated script takes effect on the next API call — no node restart is required.
- Creating a new method after startup: if the method is created but not yet compiled, the first invocation triggers lazy (on-demand) compilation.
- **Operator diagnostics (#75).** The data plane answers a broken method with the generic `Script compilation failed for this method` (the Roslyn text can leak code to an API-key caller). Operators get the real diagnostics through the role-gated management surface instead: `GetApiMethod` carries the node's `IsCompiled` / `IsKnownBad` / `LastCompileError`, `ValidateApiMethod` compiles the stored script and returns the errors without registering anything, and `RecompileApiMethod` re-runs `CompileAndRegister` for one method without a full-replace update (CLI `api-method get|validate|recompile`, Central UI API-method edit page). Compile state is per central node.
- **Operator diagnostics (#75).** The data plane answers a broken method with the generic `Script compilation failed for this method` (the Roslyn text can leak code to an API-key caller). Operators get the real diagnostics through the role-gated management surface instead: `GetApiMethod` carries the node's `IsCompiled` / `IsKnownBad` / `LastCompileError`, `ValidateApiMethod` compiles the stored script and returns the errors without registering anything, and `RecompileApiMethod` re-runs `CompileAndRegister` for one method without a full-replace update (CLI `api-method get|validate|recompile`, Central UI API-method edit page). Compile state is per central node, but a change does not stay local: create / update / delete / recompile through the management surface (and an API method written by a bundle import) publishes `InboundApiMethodInvalidated` on the central cache-invalidation topic, and the peer central node drops that method's compiled handler, known-bad record and cached row (`InboundApiCacheInvalidationHandler`, #103) — so a recompile that clears a stale compile failure clears it on both nodes. The peer compiles lazily on its next request.
> **Compile failures on save are non-fatal but surfaced, and the DB row is authoritative.** The persisted `ApiMethod.Script` row is the single source of truth: on every request `ExecuteAsync` revision-checks the cached delegate against the freshly-fetched script and recompiles in place when it differs, so a saved-but-broken script does **not** leave a previously registered handler silently serving. The change still persists, so the Create/Update Management result carries a top-level **`compileWarning`** (null when the script compiled) listing the Roslyn/trust-model errors. On both **create** and **update**, once a non-compiling script is saved the method returns a compilation error (`HTTP 500`) **consistently on every central node** until a compiling version is saved — there is no node-divergent "old delegate still works here" state. This means a saved script that does not compile is **not silently a no-op** — the caller (CLI / Central UI) sees the warning, and the real diagnostics are also logged centrally.