feat(transport): fleet-wide bundle-unlock throttle window (register row 52)

The per-IP unlock throttle's counters were a process-local ConcurrentDictionary
on each central node, so the effective fleet cap was N x
MaxUnlockAttemptsPerIpPerHour per client (failing over to the other node handed
the client a fresh budget) and a node restart cleared the window outright.

The window now lives in the central MS SQL BundleUnlockAttempt table — the HA
store both central nodes already share, so no new infrastructure, no new port,
no consensus protocol:

- Commons: BundleUnlockAttempt entity + IBundleUnlockAttemptRepository.
- ConfigurationDatabase: entity config (one index,
  IX_BundleUnlockAttempt_Client_Attempted, serving both the windowed count and
  the prune), repository (AsNoTracking count, ExecuteDeleteAsync prune),
  DbSet, DI registration, migration AddBundleUnlockAttemptTable + the
  hand-written idempotent production script under docs/plans/sql/.
- Transport: IBundleUnlockAttemptWindow seam + SqlBundleUnlockAttemptWindow
  (singleton; resolves the scoped repository from its own scope per call, like
  the central purge singletons). BundleUnlockRateLimiter keeps the policy and
  gains TryRegisterAttemptAsync; BundleImporter.LoadAsync calls it.

Behaviour preserved from row 44: locked bundles spend no hourly budget (the
lockout check still runs first), throttle denials burn no per-bundle strike and
are NOT recorded so they cannot roll the window forward, unattributable clients
share the stricter <unattributed> bucket, and the budget counts attempts rather
than failures. Config keys and their semantics are unchanged, so the rig's 500
override stays valid.

Three deliberate trades, documented in both Transport docs:
- FAILS OPEN. A store error logs a Warning naming the client and cap, and
  permits the attempt. This is a best-effort rate cap over an already
  authenticated, Administrator-gated surface — the availability of legitimate
  imports outranks the cap, and a DB blip must not make the import wizard
  unusable. Falling back to the process-local window was considered and
  rejected: it would still block a legitimate operator's 11th import.
- Approximate. Count-then-insert is not atomic across nodes, so simultaneous
  attempts can overshoot by the number in flight — bounded, self-correcting
  within the window, and not worth provider-specific raw SQL.
- Pruned opportunistically on the unlock path (CAS-claimed at most every 15
  min per process, bounded to 1000 rows, retaining 2x the window, best-effort
  try/catch) rather than by a new background service — Transport owns no
  central scheduler. Same shape as DeploymentService's terminal-record sweep.

Tests: Transport.IntegrationTests 116 passed (+10 — two limiter instances over
one store enforcing a single combined window, restart durability, denials
burning no budget, sliding release, bucket isolation, store-failure fail-open,
caller-cancellation not swallowed, no-shared-window fallback, prune and its
negative control); ConfigurationDatabase 395 passed (+5).
This commit is contained in:
Joseph Doherty
2026-08-15 11:52:34 -04:00
parent 372ea4df1b
commit 677263af57
21 changed files with 3157 additions and 30 deletions
+9 -1
View File
@@ -131,6 +131,14 @@ The two levels interact in a specific order, and it is load-bearing (register ro
2. **Per-IP throttle next, before the decrypt.** Over budget throws `BundleUnlockThrottledException` (Commons, so the management surface can catch it and return a curated message). Because it precedes the decrypt, a denial burns **no** per-bundle strike — otherwise the throttle would be a way to lock out a bundle someone is legitimately importing. Every unlock attempt is registered, successful ones included: the documented budget counts attempts, not failures.
3. Only then the AES-GCM decrypt, whose failure increments the per-bundle counter as before.
**The per-IP window is fleet-wide (register row 52).** It used to be a process-local `ConcurrentDictionary` on each central node, which had two consequences worth naming: the effective fleet cap was N × `MaxUnlockAttemptsPerIpPerHour` per client (a client that failed over to the other node got a fresh budget), and a node restart cleared the window outright. The window now lives in the central MS SQL `BundleUnlockAttempt` table — the HA store both central nodes already share, so this needs no new infrastructure, no new port and no consensus protocol. `BundleUnlockRateLimiter` keeps the policy; `SqlBundleUnlockAttemptWindow` is the store, resolving a scoped `IBundleUnlockAttemptRepository` from its own scope per call (it is a singleton, like the limiter). Three properties are load-bearing:
- **It fails OPEN.** A store error logs a Warning naming the client and the cap, and *permits* the attempt. This is a best-effort rate cap layered over a surface that is already authenticated and Administrator-gated, so the availability of legitimate imports outranks the cap — a database blip must not make the import wizard unusable. Falling back to the old process-local window instead was considered and rejected: it would still block a legitimate operator's 11th import during an outage, which is precisely the outcome fail-open exists to prevent. The Warning is the operator's signal that the fleet cap is not currently being enforced.
- **Accuracy is deliberately approximate.** The count-then-insert pair is not atomic across the two nodes, so genuinely simultaneous attempts can overshoot the cap by up to the number in flight. A conditional single-statement insert would close that at the cost of provider-specific raw SQL; the overshoot is bounded and self-corrects within the window.
- **Rows are pruned opportunistically, not by a background service.** Transport owns no central scheduler, so the prune rides the unlock path: claimed by CAS at most once per 15 minutes per process, bounded to 1000 rows per pass, retaining `2 ×` the window so a prune racing a count can never delete a row the count still needs, and wrapped in a try/catch that logs and swallows — retention maintenance must never fail the operator's actual unlock. This is the same shape `DeploymentService` uses for terminal deployment records.
Behaviour the operator sees is otherwise unchanged: same config key, same semantics ("attempts, not failures"), same ordering relative to the per-bundle lockout, same `<unattributed>` bucket.
The client key is the caller's remote IP: the Central UI wizard reads it from `CircuitClientAddressProvider` (a scoped service that snapshots `Connection.RemoteIpAddress` at circuit establishment, since `HttpContext` is gone by the time a circuit submits a passphrase — the same pattern as `CookieAuthenticationStateProvider`), and the management endpoint stamps it on `ManagementEnvelope.ClientIp` for the CLI `transport preview` / `transport import` paths. A caller with no attributable address is counted against a single **shared** bucket rather than skipping the throttle — "unknown client" degrades to a stricter budget, never to none. As with the login throttle (`Component-Security.md`), per-client isolation is only real when ForwardedHeaders is configured for the fronting proxy; behind an untrusted proxy every caller collapses into the proxy's bucket.
```csharp
@@ -264,7 +272,7 @@ After 3 wrong passphrase attempts against the same bundle (keyed by `ContentHash
### Unlock throttled ("Too many bundle unlock attempts")
`BundleUnlockThrottledException` — this **client** (not this bundle) has used its `MaxUnlockAttemptsPerIpPerHour` budget within the trailing hour. Re-uploading does not help; the budget is keyed by IP and rolls forward on a sliding window, and the counters are process-local (a central node restart clears them, deliberately, so an outage doesn't strand legitimate operators). The denial burns no per-bundle strike, so the bundle itself is still importable once the window frees up. Raise `ScadaBridge:Transport:MaxUnlockAttemptsPerIpPerHour` if a legitimate workflow (many encrypted bundles in one sitting) trips it routinely.
`BundleUnlockThrottledException` — this **client** (not this bundle) has used its `MaxUnlockAttemptsPerIpPerHour` budget within the trailing hour. Re-uploading does not help; the budget is keyed by IP, rolls forward on a sliding window, and is **fleet-wide and restart-durable** (register row 52 — see below), so failing over to the other central node does not help either. The denial burns no per-bundle strike, so the bundle itself is still importable once the window frees up. Raise `ScadaBridge:Transport:MaxUnlockAttemptsPerIpPerHour` if a legitimate workflow (many encrypted bundles in one sitting) trips it routinely.
### BundleImportAlarmScriptUnresolved / BundleImportCompositionUnresolved warnings